Webhook monitoring should answer two separate questions: did the provider deliver the event, and did your system finish the intended work? An HTTP success can confirm receipt while downstream processing still fails. Track both stages with a shared delivery identifier.
Record the delivery boundary
For each request, capture the provider, event type, delivery identifier, receipt time, signature-verification result, HTTP response, processing status, attempt count, and final outcome. Keep sensitive payload fields out of routine logs. Store enough metadata to investigate without turning the log into another copy of private customer data or secrets.
GitHub's webhook best practices recommend subscribing only to needed events, using a webhook secret, keeping SSL verification enabled, responding promptly, checking the event and action before processing, and using the delivery header to identify an event. Those controls also produce useful monitoring signals: signature failures, unexpected event types, slow acknowledgments, and duplicate identifiers.
Acknowledge quickly, process separately
Validate the sender and the basic envelope before accepting work. Then place accepted events on a durable queue when the actual task might be slow. The receiver can return a timely success while a worker handles the business action. Monitor queue age and processing failures; otherwise a healthy endpoint can hide a growing backlog.
Do not treat every retry as a new business event. Use the provider's stable delivery identifier or a carefully designed idempotency key so replaying a delivery does not create duplicate orders, tickets, messages, or deployments. Record the original and replay attempts under the same event identity.
Monitor four failure classes
Create separate views or alerts for:
- delivery failures: connection, certificate, timeout, or non-success HTTP results;
- validation failures: missing or invalid signatures, unsupported content types, or malformed envelopes;
- processing failures: the request was accepted but downstream work failed; and
- freshness failures: expected events stopped arriving or the queue age exceeded its normal range.
A low error rate is not enough if expected traffic silently drops to zero. Where there is a dependable baseline, add a missing-event or stale-last-delivery check. Keep thresholds specific to the integration; a billing webhook and a low-volume repository webhook have different urgency.
Build a controlled replay path
The GitHub troubleshooting guidance advises checking recent delivery logs when events appear missing and explains that non-success responses are recorded as failures. It also notes that deliveries can arrive out of order, so order-sensitive processing should use timestamps or current resource state rather than arrival order alone.
A replay tool should show the original event, current processing state, intended side effect, and operator. Require authorization for consequential replays. Re-verify the signature or retrieve the event through an authenticated provider interface, and run the same idempotency protections used for normal deliveries.
Test the monitor, not only the handler
Exercise a valid delivery, bad signature, delayed worker, duplicate delivery, out-of-order pair, and downstream outage. Confirm that each condition appears in the expected dashboard or alert and that recovery clears the alert. Review retained failures on a schedule so a permanently stuck event does not become normal background noise.
A trustworthy webhook monitor joins provider delivery evidence to internal processing evidence. It makes a missing side effect visible, explains where the path stopped, and provides a replay method that does not repeat completed work.
Sources
- Best practices for using webhooks — GitHub Docs
- Troubleshooting webhooks — GitHub