Design a webhook consumer with the expectation that a valid event can arrive more than once, later than expected, or out of order. Authenticate the request, record a stable delivery or event identifier, acknowledge receipt quickly, and make the downstream action idempotent before enabling retries.
Define the delivery contract
Record the provider, endpoint, subscribed event types, signature method, expected response codes, retry policy, delivery identifier, event identifier, and retention period for deduplication records. Subscribe only to events the automation actually uses. Keep the signing secret in an approved secret store and never place credentials in the callback URL or logs.
GitHub's webhook guidance recommends using a webhook secret, HTTPS, event-type checks, quick successful responses, and the delivery header to recognize a redelivery. Stripe's webhook documentation likewise requires signature verification against the raw request body and recommends returning a success response before complex processing.
Separate receipt from processing
Validate the signature and basic envelope, then persist the event and enqueue work before returning the provider's expected success status. Do not acknowledge an event that has not been durably recorded unless losing it is acceptable. Keep the receiver small so a slow downstream service does not cause unnecessary delivery retries.
Store the provider's delivery or event identifier with a unique constraint when possible. If the identifier was already completed, return success without repeating the side effect. If it is still processing, use a lease or state transition so two workers cannot perform the same action simultaneously.
Make the side effect idempotent
Choose an idempotency key that represents one intended business action. Reuse that key only for retries of the same parameters. Stripe's idempotent-request documentation explains that a repeated request with the same idempotency key returns the recorded result rather than performing the operation twice, while mismatched parameters are rejected. Other systems may implement idempotency differently, so follow the destination API's rules.
For a database change, encode the uniqueness rule in the database rather than relying only on an in-memory check. For an external API, save the outbound idempotency key and response reference with the event. Do not mark the event complete until the intended state is confirmed.
Retry with boundaries
Retry only transient failures such as timeouts, unavailable services, or documented rate limits. Do not repeatedly retry invalid signatures, unsupported event types, malformed payloads, or permanent permission errors. Use bounded exponential backoff with jitter when the provider or destination does not prescribe another policy. Set a maximum attempt count and move exhausted events to a visible recovery queue.
Expect manual and automatic redelivery to overlap. Stripe documents automatic retries and warns that a manual resend does not cancel automatic retry behavior. It also notes that event order is not guaranteed and recommends tracking event IDs to identify duplicates. Build processing around current resource state when order matters instead of assuming arrival sequence.
Reconcile after recovery
For each failed event, record the authenticated identity, first receipt, attempts, final result, and downstream reference without retaining unnecessary sensitive payload data. Compare the event source with the destination system to confirm that every intended action occurred once. Alert on rising retry counts, old queued events, signature failures, and duplicate side-effect attempts. Test the handler with duplicates, delayed events, out-of-order events, timeouts, and a worker crash between the external action and local completion.
Sources
- Best practices for using webhooks — GitHub Docs
- Receive Stripe events in your webhook endpoint — Stripe
- Idempotent requests — Stripe Documentation