Reliable Webhook Delivery: How to Never Miss a Webhook

Reliable webhook delivery means every event a provider sends is eventually processed exactly once by your system, even when your endpoint is slow, deploying or down. Providers don't guarantee that. They retry for a window that ranges from zero (GitHub) to seven days (WhatsApp Cloud API), with Stripe at three days and Shopify at four hours, and then they stop. Reliability is something the receiving side builds out of five stages: receive fast, queue durably, process idempotently, retry with backoff, and dead-letter what still fails. Monitoring and reconciliation wrap around all five.

Why webhooks get lost

Every lost webhook falls into one of six buckets. The first four are your side; the last two are the provider's.

The last row is the uncomfortable one: Shopify's documentation states plainly that delivery "isn't always guaranteed". A receiver that only listens can never be fully reliable. It also has to ask.

How long providers really retry

Before you design anything, know how much slack each sender gives you. All of these are from the providers' own documentation, checked in November 2026.

Two things stand out. The windows are short: half of these give you minutes or hours, not days. And they are all different, so an outage that one provider survives is data loss for another. The fix is to stop depending on each sender's retry policy and own the retry window yourself, which the pipeline below does.

At-least-once is the best anyone offers

Without an acknowledgement, a sender can't tell these four cases apart:

1. The request was lost on the way.
2. It arrived and was processed, but the 200 was lost on the way back.
3. The receiver crashed before processing it.
4. The network is just slow.

Retrying fixes 1 and 3, and turns 2 and 4 into duplicates. That is why every provider above delivers at least once: if it doesn't see a 2xx in time, it sends again, even when your server did the work and only the response was lost. Exactly-once delivery over a network isn't possible; what you can build is exactly-once processing, by making the receiver ignore what it has already seen.

Two decisions make deduplication work:

• Which id to key on. Use the id that stays the same across retries of the same event. That is the event id for Stripe (evt_…) and Standard Webhooks (webhook-id), X-GitHub-Delivery for GitHub, and X-Shopify-Webhook-Id for Shopify. Shopify's X-Shopify-Event-Id is shared by deliveries to different subscriptions, so keying on it would drop legitimate deliveries if you have two.
• How long to remember it. At least the longest retry window you receive from, plus your own retry window. Stripe retries for 3 days, so keep Stripe ids for a week; a day covers Shopify's 4 hours. Then delete them on a schedule so the dedupe table doesn't grow forever, or keep ids in Redis with a TTL at high volume:

delete from webhook_events
 where status = 'done' and received_at < now() - interval '7 days';

# Returns OK the first time, nil if the id was already seen
SET webhook:<provider>:<event_id> 1 NX EX 604800

• Also dedupe on the business key for one-time side effects. One business action can arrive as several different events with different ids: a Stripe payment can produce both payment_intent.succeeded and charge.succeeded. If both trigger "send the receipt", key that side effect on the payment intent or order id, not the event id.
• Prefer natural idempotency where you can. If the event says "order 42 changed", fetch order 42's current state from the provider's API and write that, instead of applying the event as a delta. Duplicates and out-of-order deliveries then converge on the right answer without any bookkeeping.
• Know when you can skip it. Cache invalidation, analytics counters you can reconcile, and handlers that only fetch current state tolerate duplicates as they are.

The five-stage delivery pipeline

1. Receive: verify, store, then acknowledge

The receiving endpoint has one job: get the event somewhere durable and answer before the provider's timeout. Verify the signature against the raw body, write the event with its id, and return 200. Nothing else happens in the request: no API calls, no emails, no business logic.

app.post('/webhooks/:provider', express.raw({ type: '*/*' }), async (req, res) => {
  if (!verifySignature(req.params.provider, req)) return res.sendStatus(401)

  // Unique (provider, event_id): a duplicate delivery is a no-op insert.
  await db.query(
    `insert into webhook_events (provider, event_id, headers, body, status)
     values ($1, $2, $3, $4, 'pending')
     on conflict (provider, event_id) do nothing`,
    [req.params.provider, eventId(req), req.headers, req.body]
  )
  res.sendStatus(200)
})

If storing fails, return 500 so the provider retries. That is the one failure you want the provider to see.

2. Queue: durable storage between receive and process

The stored event is the queue. A database table with a status column is enough for most teams, because the insert and the acknowledgement already happen in the same place. At higher volume, publish to SQS, Pub/Sub or Kafka after the insert. What matters is that a crash, deploy or restart of the worker loses nothing: unprocessed events are still pending when it comes back.

3. Process: idempotent on the event id

Workers pick up pending events and do the real work. Because every stage is at-least-once, the same event will sometimes be processed twice. A handler is idempotent when the second run changes nothing:

• Key on the id that stays the same across retries (see which id to key on), never on a hash of the payload. Standard Webhooks senders put it in webhook-id.
• Put the side effect and the "processed" mark in one transaction, or make the side effect itself idempotent with an idempotency key on the outbound call.
• Guard against out-of-order delivery: compare the object's updated_at with what you have and ignore older versions.

Webhook retries and idempotency goes deeper on duplicate handling.

4. Retry: backoff, jitter and a deadline

When processing fails, retry from storage, not from the provider. Use exponential backoff so a struggling dependency gets room to recover, add jitter so a thousand failed events don't retry in the same second, and set a deadline after which the event moves to the dead-letter stage.

An example schedule with a 30-second base, doubling each time, ±20% jitter and a deadline after the 12th retry:

That is 12 retries over about 34 hours. Cap the wait (a day is common) if you extend the schedule further; otherwise the gaps grow past anything useful.

A worker that claims due events, retries them and dead-letters them at the deadline fits in one query and one loop:

const MAX_RETRIES = 12

async function runOnce() {
  const db = await pool.connect()
  try {
    await db.query('begin') // the row locks below last until commit
    await processBatch(db)
    await db.query('commit')
  } catch (err) {
    await db.query('rollback')
    throw err
  } finally {
    db.release()
  }
}

async function processBatch(db) {
  // Claim due events; SKIP LOCKED lets several workers run safely.
  const { rows } = await db.query(
    `select * from webhook_events
      where status = 'pending' and next_attempt_at <= now()
      order by next_attempt_at limit 50
      for update skip locked`)

  for (const ev of rows) {
    try {
      await handle(ev) // idempotent: safe to run twice
      await db.query(`update webhook_events set status = 'done'
                      where id = $1`, [ev.id])
    } catch (err) {
      const attempts = ev.attempts + 1
      const permanent = err.permanent || attempts > MAX_RETRIES
      const wait = 30 * 2 ** (attempts - 1) * (0.8 + Math.random() * 0.4)
      await db.query(
        `update webhook_events
            set attempts = $2, last_error = $3,
                status = $4,
                next_attempt_at = now() + make_interval(secs => $5)
          where id = $1`,
        [ev.id, attempts, String(err), permanent ? 'dead' : 'pending', wait])
    }
  }
}

Add an index on (status, next_attempt_at). The trade-off of this version is that row locks are held while handle() runs. If handlers are slow, claim with a lease instead and process outside the transaction:

update webhook_events
   set status = 'processing', locked_until = now() + interval '5 min'
 where id in (select id from webhook_events
               where status = 'pending' and next_attempt_at <= now()
               order by next_attempt_at limit 50
               for update skip locked)
returning *;

A periodic job puts rows whose lease expired back to pending, which covers a worker that died mid-batch. SKIP LOCKED is what stops two workers from claiming the same event at the same moment and processing it twice; the idempotent handler covers the remaining case, where a worker crashes after the side effect but before marking the event done.

Separate retryable failures from permanent ones. A 503 from a dependency is worth retrying; a payload that fails validation will fail identically every time and should go straight to the dead-letter stage.

5. Dead-letter: park it, fix it, replay it

A dead-letter queue keeps events that exhausted their retries, with the payload and the last error, so they stop clogging the main queue without being thrown away. The operational loop is: an alert fires, someone reads the last error, ships a fix, and replays the dead-lettered events through the normal handler. Replay is safe because stage 3 is idempotent. Without a replay path, a dead-letter queue is just a slower way to lose data.

Around the pipeline: monitoring and reconciliation

The five stages handle events that arrive. Two more jobs handle the ones that don't, and the ones that fail quietly.

Monitoring. Track success rate, latency and retry backlog per source and destination, and alert on the first failed delivery or on an error-rate threshold. Add a silence alert: if a source that normally sends every few minutes has sent nothing for an hour, something upstream is broken, and no failure counter will tell you. Webhook monitoring covers which metrics to track and how to set the thresholds.

Reconciliation. On a schedule, ask the provider's API for objects that changed since the last run and process any you have not seen. Stripe's GET /v1/events lists events for 30 days. Shopify's Admin API can query orders and products by updated_at. GitHub's API lists recent deliveries per webhook, so you can redeliver failed ones. Reconciliation turns "the provider never sent it" from data loss into a delay.

See durable retries converge

The chart below is a real run: 50 webhooks sent through Webhook Relay to flakey-script, an open-source endpoint that rejects about 80% of requests until an event is an hour old, with durable retries on the medium schedule. Fast retries run for the first 15 minutes, then durable retry takes over; the endpoint starts accepting old events at 60 minutes, and the last webhook lands at 73 minutes. Press play to watch, or switch to Retry attempts to see the 275 attempts behind the curve.

The result: 50 of 50 webhooks delivered, the last at 73 minutes, after 275 attempts. With only a provider's own retries, anything still failing after Shopify's 4-hour window, or immediately for GitHub, would have been dropped.

Build it or use a gateway

There are three ways to get the pipeline. They are not exclusive: most teams that use a gateway still keep idempotent handlers and a reconciliation job.

Webhook Relay sits in the third column. A bucket gives each provider a stable HTTPS URL that answers in milliseconds and stores the request, then durable retries deliver it with exponential backoff for up to 30 days. Alerts open an incident on the first failed delivery, request logs keep every attempt, and one click resends a delivery or recovers every failure after a fix. The relay agent delivers into localhost, private networks and Kubernetes without inbound firewall rules, and throttling stops a backlog from flattening a recovering service.

If you send webhooks

Everything above is the receiving side. If your product sends webhooks to customers, the same rules apply in reverse: sign every message, retry with backoff for a published window, give each customer endpoint its own rate limit so one slow customer doesn't delay the rest, keep a delivery log your support team can search, and let customers replay. Webhook Relay's Outbound Gateway does that with Standard Webhooks signatures and 35 days of history on every plan; Svix and Hookdeck Outpost are the main alternatives.

Next steps

• Apply it to one provider: Shopify webhooks covers the 5-second timeout, 8 retries in 4 hours and deleted subscriptions.
• Decide what to alert on with webhook monitoring.
• Go deeper on duplicates in webhook retries and idempotency.
• Turn on durable retries for a destination.

Reliability checklist

This list covers the pipeline. For what to watch and alert on, use the monitoring checklist.

• The endpoint answers within the strictest provider timeout you receive from (3 s for Slack).
• The raw event is stored before 200 is returned; storage failure returns 500.
• Every event has a unique constraint on provider + event id.
• Handlers ignore payloads older than the stored object.
• Processing failures retry from storage with backoff, jitter and a deadline.
• Permanent failures skip retries and go to the dead-letter queue.
• Dead-lettered events can be replayed through the normal handler.
• Alerts fire on failed deliveries and on silence from busy sources.
• A reconciliation job runs on a schedule for every provider that has a list API.
• Deploys never take the receiving endpoint down for longer than the shortest retry window, or a buffer sits in front of it.