Idempotency is not optional at scale
At enterprise volume, webhook redelivery (from retries, network issues, or a partner's own bugs) is a statistical certainty, not an edge case. Any webhook handler without idempotency protection — processing the same event twice safely — will eventually double-process a real event, and the resulting data corruption is often discovered far later than the event itself.
Idempotency keys checked against a durable store before processing are a small implementation cost that prevents a category of bug that's otherwise very difficult to trace back to its cause.
Design for partner-side failure, not just your own
A sending partner's webhook delivery can fail, retry inconsistently, or simply stop without clear signal. Build explicit monitoring for 'a partner that used to send events regularly has gone quiet' — this silent failure mode is easy to miss because nothing actually errors; data just stops arriving.
This is a genuinely different monitoring problem than typical error-rate alerting, and it needs its own specific detection logic rather than being assumed to be covered by general observability.
Queue and backpressure, don't process inline
Processing webhook payloads synchronously, inline with the incoming HTTP request, creates fragility under partner burst traffic and makes your own downstream outages immediately visible to the partner as failed deliveries. Accepting the event quickly, queueing it, and processing asynchronously decouples your processing capacity from the partner's delivery timing entirely.
This pattern also makes replay and reprocessing far easier during an incident — you can reprocess from the queue without asking the partner to resend anything, which is often not even possible on their side.
A webhook handler that assumes every event arrives exactly once is a handler that has not yet met enterprise scale.MNB Integration Practice
Key actions
- Implement idempotency keys checked against a durable store
- Monitor for partners that have gone silent, not just for errors
- Queue incoming events and process asynchronously
- Make replay from the queue possible without partner resend