Webhook Monitoring: What to Track and How to Alert on Failures
Webhook monitoring in practice: the six signals to track (success rate, failures, latency, retry backlog, dead letters, silence), alert thresholds that don't page you for nothing, and where Stripe, Shopify and GitHub stop helping.
On this page
Webhook monitoring is watching whether the webhooks you receive (or send) are delivered and processed, and getting told when they aren't. The failure mode it exists for is silence: a provider gets a 500, retries for a while, gives up, and nothing on your side notices until a customer asks where their order went. Good webhook monitoring tracks six signals per source and destination, alerts on a few of them, and resolves itself when deliveries recover.
The short version
- Track six signals per source and destination: success rate (alert below ~99% first-attempt), failed deliveries, p95 latency (alert at half the provider's timeout), retry backlog (alert while it grows), dead letters (alert on any) and silence (alert past the source's normal gap).
- Alert on the first failure for money-moving flows and on an error-rate threshold elsewhere.
- Alert when p95 latency gets close to the provider's timeout: that is a failure about to happen.
- Alert on silence from sources that normally send. No error counter catches a provider that stopped sending.
- Keep your own request log. Provider dashboards keep days, not weeks, and only show their side.
The six signals to track
| Signal | What it tells you | Alert when |
|---|---|---|
| Delivery success rate | Share of deliveries your endpoint accepted with a 2xx, on the first attempt and eventually | First-attempt rate drops below ~99%, or any event fails for good |
| Failed deliveries | Count of non-2xx responses, timeouts and connection errors | First failure on critical flows (payments, orders); error rate above a threshold elsewhere |
| Response latency (p95) | How long your endpoint takes to answer | p95 passes half the provider's timeout (1.5 s for Slack, 2.5 s for Shopify, 5 s for GitHub) |
| Retry backlog | Events waiting for another attempt | Backlog grows for more than one retry interval |
| Dead letters | Events that exhausted their retries | Any new entry |
| Silence | Time since the last event from a source | Longer than the source's normal gap (an hour for a busy store, a day for a nightly job) |
The Prometheus rules for all six are in build it yourself below. Split every signal by source (Stripe, Shopify, GitHub) and destination (your API, the warehouse loader, a partner). One healthy average hides one broken integration.
Here is what those signals look like during one real-shaped incident: an endpoint starts returning 503 after a bad deploy, retries pile up, someone ships a fix, and the backlog drains without losing an event.
The alert fired on the first failure, the latency, backlog and success-rate signals told on-call how bad it was, and the automatic resolve closed it without anyone touching it. Durable retries are what turned half an hour of failures into zero lost events. The retries follow the doubling schedule from reliable webhook delivery.
Why latency is a failure signal
Providers measure success by time, not just by status code. A handler that returns 200 after six seconds has succeeded in your logs and failed in Shopify's, which allows five. Slack allows three; GitHub ten. When p95 response time creeps toward those limits, you are hours away from timeouts that you will only see as retries and duplicates. The fix is structural (store the event, acknowledge, process later; see reliable webhook delivery), but the alert is what tells you to do it.
Why silence needs its own alert
Every other signal counts something that happened. Silence is the absence of events, and it covers the failures that never reach your endpoint at all: a subscription that Shopify deleted after repeated failures, a Slack app whose event subscriptions were disabled, an expired token on the sender's side, a filter that matches nothing, a DNS change. Base the threshold on the source's normal rhythm: a store with hundreds of orders a day should not go an hour without orders/create; a nightly export should arrive once a day.
Where Stripe, Shopify and GitHub stop helping
| Provider | Delivery log | How far back you can see or resend | Alerts you get |
|---|---|---|---|
| Stripe | Per event and per endpoint in the Dashboard | Resend from the Dashboard up to 15 days, CLI up to 30 | Not stated in Stripe's webhook docs; retries stop after 3 days |
| Shopify | Dev Dashboard: deliveries, failure rate, p90 response time | 7 days | Emergency developer email for app subscriptions; flags failure rates above 0.5% |
| GitHub | Recent Deliveries per webhook, with request and response | 3 days | None documented; there are no automatic retries either |
| Slack | None documented; retries carry x-slack-retry-num and x-slack-retry-reason headers | — | Event subscriptions disabled above 95% failures in 60 minutes (apps with 1,000+ events an hour) |
From each provider's documentation, checked November 2026.
Provider dashboards are good for one question: did the provider try, and what did your endpoint answer? They don't show what your system did after the 200, they don't page anyone, they cover one provider each, and they forget within days. For anything that matters you need the same data on your side.
Build it yourself
If you run your own receiver, instrument it at the edge and in the worker:
import client from 'prom-client'
const received = new client.Counter({
name: 'webhook_received_total',
help: 'Webhooks received',
labelNames: ['source', 'status', 'attempt'], // attempt: first | retry
})
const latency = new client.Histogram({
name: 'webhook_response_seconds',
help: 'Time to acknowledge',
labelNames: ['source'],
buckets: [0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 7.5, 10],
})
const lastSeen = new client.Gauge({
name: 'webhook_last_received_timestamp',
help: 'Unix time of last webhook',
labelNames: ['source'],
})
// Set these from the worker: events waiting for a retry, and dead letters.
export const backlog = new client.Gauge({
name: 'webhook_retry_backlog',
help: 'Events waiting for another attempt',
labelNames: ['destination'],
})
export const deadLetters = new client.Counter({
name: 'webhook_dead_letters_total',
help: 'Events that exhausted their retries',
labelNames: ['destination'],
})
export function track(source, handler, isRetry = () => false) {
return async (req, res) => {
const attempt = (await isRetry(req)) ? 'retry' : 'first'
const end = latency.startTimer({ source })
res.on('finish', () => {
end()
received.inc({ source, status: String(res.statusCode), attempt })
lastSeen.set({ source }, Date.now() / 1000)
})
return handler(req, res)
}
}
Then alert on the series, not on log lines:
groups:
- name: webhooks
rules:
- alert: WebhookErrorRate
expr: |
sum by (source) (rate(webhook_received_total{status!~"2.."}[10m]))
/ sum by (source) (rate(webhook_received_total[10m])) > 0.01
for: 10m
# Half the provider's timeout: 2.5 s for Shopify, 1.5 s for Slack.
- alert: WebhookSlowAck
expr: |
histogram_quantile(0.95,
sum by (le, source) (rate(webhook_response_seconds_bucket[10m]))
) > 2.5
for: 15m
- alert: WebhookSlowAckSlack
expr: |
histogram_quantile(0.95,
sum by (le) (rate(webhook_response_seconds_bucket{source="slack"}[10m]))
) > 1.5
for: 15m
- alert: WebhookSilence
expr: |
(time() - webhook_last_received_timestamp{source="shopify"} > 3600)
or absent(webhook_last_received_timestamp{source="shopify"})
for: 5m
- alert: WebhookBacklogGrowing
expr: deriv(webhook_retry_backlog[15m]) > 0
for: 15m
- alert: WebhookDeadLetter
expr: increase(webhook_dead_letters_total[5m]) > 0
isRetry decides the attempt label: a delivery is a retry when its id is already in your event store (Slack also sends x-slack-retry-num on retries), which gives you first-attempt success alongside eventual success. The absent() clause matters: without it, a receiver that restarts and never receives another event never alerts. The gauge also resets on restart, so for a silence check that must survive deploys, read the last event time from your event store instead. Prometheus is one option: the same counters work as OpenTelemetry metrics, so you can send them to Datadog, Grafana Cloud or New Relic if that is where your alerts already live. This works well, and it is also a dashboard and five alert rules you now own per receiver.
Alerting that doesn't cry wolf
Noisy webhook alerts get muted, and muted alerts are worse than none. A few rules keep them useful:
- Match sensitivity to the flow. Page on the first failed payment or order webhook. For analytics or notification webhooks, alert only above an error rate, over a window long enough to ignore a single blip.
- Resolve automatically. An incident that closes itself after a clean window means on-call only looks at things that are still broken.
- Alert on the destination, not the payload. A run of failures to one endpoint is one incident, not five hundred messages.
- Say what is affected. "orders/create to fulfilment-api failing since 14:02, 37 failures" beats "webhook error". A good alert carries enough to act on without opening a dashboard:
{
"status": "open",
"title": "Webhook deliveries failing: shopify → fulfilment-api",
"source": "shopify",
"topic": "orders/create",
"destination": "https://fulfilment.internal/webhooks/shopify",
"since": "2026-11-10T14:02:00Z",
"failures": 37,
"success_rate_10m": 0.61,
"last_response": { "status": 503, "body": "upstream connect error" },
"retry_backlog": 37,
"oldest_pending_age_s": 1080,
"runbook": "https://wiki.example.com/runbooks/shopify-webhooks"
}
Send the matching "status": "resolved" message when deliveries recover, so the incident tool closes the ticket on its own.
- Test the alert. Point a destination at a URL that returns
500, for example a Webhook Bin configured to fail, and confirm the alert fires and resolves.
If you send webhooks
Monitoring the sending side is the same idea, one level out: your customers' endpoints are the destinations, and they fail more often than yours.
- Success rate per customer endpoint, not just overall. One customer with a dead endpoint can hide inside a 99.9% average.
- Endpoints that keep failing. Mark them as failing, alert your support team, and auto-disable after a long outage instead of retrying forever.
- Messages that ran out of retries. Each one is data a customer never received; they need a list they can replay.
- Tell the customer. An email or dashboard banner when their endpoint is failing saves a support ticket three days later.
Webhook Relay's Outbound Gateway tracks endpoint health, marks endpoints failing, optionally auto-disables them and lets you recover every failed delivery once the customer is back; see monitor, retry and recover deliveries.
Monitor webhooks with Webhook Relay
If your webhooks pass through a Webhook Relay bucket, the receiving side is logged and measured without code.
- Delivery health across every bucket. Requests, delivery rate, failures, retries and average latency per bucket, ranked by failures, with a failures-over-time sparkline and 7-day to 12-month windows.
- A request log per bucket with the full request, your endpoint's response and every attempt, filterable by status and destination. Resend one delivery, or recover every failure in a window once a fix is out. See webhook logs.
- Alerts on delivery failures, sent to Slack, email, Microsoft Teams, Discord, Telegram, Pushover or any webhook (PagerDuty, Opsgenie). A policy opens an incident on the first failed delivery or above an error-rate threshold, per bucket, and resolves it automatically after a full window without failures.
- Durable retries keep failed deliveries retrying for up to 30 days while you fix the destination, so an alert is a prompt to act, not a countdown to data loss.
Alerts in Webhook Relay cover delivery failures to your destinations. They don't yet alert on silence, so pair them with one scheduled check of your own: a job that reads the time of the last event per source and pages when it is older than that source's normal gap. Your agent can also ask: the MCP server exposes delivery stats and logs, so "which buckets had failures this week?" is one question.
Next steps
- Build the pipeline these signals watch: reliable webhook delivery.
- Shopify specifics, including the 0.5% failure-rate flag: Shopify webhooks.
- Set up an alert policy in Webhook Relay, and test it against a failing Webhook Bin.
Monitoring checklist
This list covers what to watch. For the pipeline itself (storing, retrying, dead-lettering), use the reliability checklist.
- Every source and destination has its own success rate, failure count and p95 latency.
- Critical flows page on the first failed delivery; others alert above an error-rate threshold.
- p95 acknowledgement time alerts at half the strictest provider timeout.
- Retry backlog and dead letters are visible, and any new dead letter alerts.
- Busy sources have a silence alert sized to their normal gap.
- Your own request log keeps longer than the providers' (7 days for Shopify, 3 for GitHub).
- Incidents resolve automatically after a clean window.
- A reconciliation job covers events the provider never sent; see reliable webhook delivery.
- You have fired a test failure through each alert in the last quarter.