Why my webhooks no longer return 5xx

Tuesday, 3pm - you're about to ship a small deployment.

Everything looks fine, great; but two hours later, an issue shows up!

Your endpoint /bank/card/webhook throws, under certain conditions, an exception! Because of one of the new business rules!

Not only is that annoying, but the snowball effect is instant:

• a pile-up of errors (and yes... the client/sender - of the endpoint/of the webhook - retries, with delay, on every 500)
• a partially completed process - cards get generated on the client/sender side, then on yours, but the processing doesn't make it to the end of the chain and your end users never receive them... in the end you'll have to write a custom repair dev JUST for the data... not to mention support!
• clients/senders who stop contacting you for a while (circuit-breaker open)

... In short, you're in for a bad week, and you'll almost certainly be unable to perfectly fix some of the operations that already ran.

You've probably already run into this case, and whatever the cause of your error (typo, timeout on a third party, cache issue, missing dependency...), the fix to put in place will be the same.

The core problem is this: your endpoint does a business ack, it should do a technical ack.

And to fix it, we can lean on several patterns!

Introduction: setting up the webhook consumption endpoint

The goal of our ledger is to automate and secure the reception of webhooks, and to list them. The plan I put in place is the following (with the technical topics covered in each phase in parentheses):

1. reception and validation - (idempotency)
2. persistence, dispatch & response - (atomicity, dual-write, transactional outbox)
3. asynchronous processing - (at-least-once, pessimistic locking)

Reception & validation

The message is received via a controller route (Symfony). The whole question is figuring out what it's responsible for, and what it absolutely must not do.

The note above establishes that we need idempotent behavior: having a third party make changes to our system can always go wrong. Also, idempotency protects against duplicate side effects. On our demo project this doesn't matter much, but in prod, it would mean multiplying business executions (e.g. several emails sent for the same event to the same user). In short, better to avoid it.

To do that, we need to determine what defines the uniqueness of a webhook. Stripe, Github and the likes always provide an event id, which we're going to rely on (plus one other attribute ... but let's not get ahead of ourselves!).

Then, since the call is made by a third party, we need to make sure of two things: that the sender is indeed the right one, and that the payload is interpretable on our side (the event id I just mentioned).

Here's a quick outline of what our controller should do:

1. check the signature's validity (via hash_equals)
2. (IF -> invalid signature) we record it (via a dedicated service class) but don't publish it, and respond with a 401
3. (ELSE) validate the format
4. (IF -> invalid format) we respond with a 422
5. we record & dispatch it (via a dedicated class)
6. we respond with a 202

Persistence, dispatch & response

This section might run a bit long, so buckle up. Now that our controller exists, and that we've decided to go with idempotency, we need to build a system that guarantees it.

Earlier, I mentioned defining what makes our webhook_entry unique. To materialize that uniqueness, I decided to set up a SQL constraint via the Doctrine attribute #[ORM\UniqueConstraint].

That constraint is enforced by the combination of external_event_id + source. In my repository, when creating a webhook_entry, I catch any UniqueConstraintViolationException to wrap it in a custom exception, which I re-throw. That way I can guarantee idempotency via a final catch in the controller, while avoiding a direct coupling to Doctrine.

I have my uniqueness, my persistence, my idempotency ... I can now move on to the asynchronous behavior of my app, namely: processing the business logic tied to the webhook - finally!

Unfortunately, it's not that simple... we're about to tackle a new problem: the Dual-Write problem!

What this means in our case: Even though our webhook_entry is now saved to the database, there's still a risk that the business processing after the save never runs.

And to solve that problem, we're going to use the Transactional Outbox - a pattern well suited to our situation.

To make this pattern operational, we need to make sure our database inserts are atomic.

Symfony's messenger component lets you, when using Doctrine, store pending messages in a messenger_messages table. And you can use Doctrine itself as the dsn! So rather than building a full outbox table with its own relay, I use the messenger mechanism directly :).

We then hook a listener onto those events so it updates the status of our webhook_entry to reflect that it was properly dispatched.

• WorkerMessageReceivedEvent → "dispatched"

The last step is to make sure the records are saved within an atomic transaction.

And there we go! By using Doctrine as the dsn, and our listener that hooks into these 3 Symfony events, we get the start of a very simple Transactional Outbox to set up! Just a few more wires and it'll be fully functional!

From there, everything else is handled asynchronously (deferred) since it's taken care of by Symfony's workers. Once the wrap is done, the transaction is COMMIT (or ROLLBACK on error) and our controller regains control to pick the response to return: 202, 401 or 422.

Asynchronous processing

The synchronous part is done, so let's look at the rest. What's missing at this stage is mainly two tasks:

• updating the status of our webhook_entry
• wiring in our business logic (which matters quite a bit, since without it our webhook reception isn't really useful in the end)

Luckily, we'll get there quickly and easily. In fact, we basically just need to enrich our listener so it listens for two additional Symfony events. In the end, we end up with this mapping:

• WorkerMessageReceivedEvent → "dispatched"
• WorkerMessageHandledEvent → "succeeded"
• WorkerMessageFailedEvent → "failed" (or "dead" once retries are exhausted)

Easy: we set up a handler in charge of the worker / business logic handoff. The WorkerMessageHandledEvent will then be dispatched automatically once your code has run successfully. On failure, it goes into retry.

Simple and effective: 5 attempts, a 1000ms delay that doubles between each try, and jitter to avoid potentially saturating the workers.

Oh, and almost forgot: the Symfony worker natively uses pessimistic locking (a SELECT ... FOR UPDATE), so there's no risk of the same message being picked up by several workers at once.

There is, however, one remaining risk: if a worker dies between running your handler and deleting the row, the message will go out again. That's how at-least-once works: delivery is guaranteed at least once.

As a result, and to guarantee full reliability, your handler will also need to be idempotent.

Conclusion

Our endpoint now has no reason left to return a 5xx because of our business logic! Only technical issues (an unreachable DB, for instance) will trigger one, and that's exactly what we were aiming for.

As mentioned earlier, the code is on the repository, and the instance is live. Have fun POSTing the same external_event_id twice, and see what happens ;)