Webhook System Design: How to Build a Webhook Delivery System

A webhook sending system has seven parts: a durable intake (a transactional outbox or an idempotent publish API), a fan-out step that creates one delivery per subscribed endpoint, per-endpoint queues with their own rate and concurrency limits, HTTP workers that sign every attempt, a retry scheduler with backoff and a deadline, an endpoint health state machine, and a delivery log with retry and replay. Each part exists because of a specific failure: a crash between commit and send, one slow customer delaying everyone, a retry storm against an endpoint that is already down, or a customer asking "did you send it?" with no way to answer.

This post walks through each component, the data model behind it, and the trade-offs, then ends with an honest build-versus-buy comparison. If you only need the checklist, see webhook best practices.

The architecture at a glance

 your service                      webhook system
┌──────────────┐   same tx   ┌───────────┐  poll   ┌──────────┐
│ business     │────────────▶│  outbox   │────────▶│ publish  │  idempotency key
│ change       │             │  table    │         │ (intake) │  → 202 + message id
└──────────────┘             └───────────┘         └────┬─────┘
                                                        │ fan-out: one delivery
                                                        │ per subscribed endpoint
                         ┌──────────────────────────────┼───────────────────────┐
                         ▼                              ▼                       ▼
                  ┌─────────────┐               ┌─────────────┐         ┌─────────────┐
                  │ lane: ep_A  │               │ lane: ep_B  │   ...   │ lane: ep_N  │
                  │ 50 rps, 16  │               │ 50 rps, 16  │         │ paused      │
                  │ in flight   │               │ in flight   │         │             │
                  └──────┬──────┘               └──────┬──────┘         └─────────────┘
                         ▼                              ▼
                  ┌─────────────────────────────────────────────┐
                  │ delivery workers: sign attempt, POST,       │
                  │ 15s timeout, no redirects, SSRF guard       │
                  └──────┬───────────────────────┬──────────────┘
                   2xx   │                       │ error / timeout / 429 / 5xx
                         ▼                       ▼
                  ┌─────────────┐        ┌──────────────────┐     ┌──────────────────┐
                  │ delivery    │◀───────│ retry scheduler  │────▶│ endpoint health  │
                  │ log (35 d)  │        │ backoff, deadline│     │ active/failing/  │
                  │ retry,replay│        │ Retry-After      │     │ disabled         │
                  └─────────────┘        └──────────────────┘     └──────────────────┘

Requirements first

Write these down before choosing technology, because they decide the design:

A sizing example helps: 1 million events a day is about 12 per second on average. With peaks at 10x and an average of 2 endpoints per event, design for roughly 250 delivery attempts per second at peak, plus retries. That is well within a single Postgres primary and a small worker pool; you don't need Kafka to start.

Data model

Six tables carry the whole system:

1. Intake: the transactional outbox and an idempotent publish

The first place webhooks get lost is inside the producer. A handler commits UPDATE invoices SET paid = true, then crashes before sending the event. The fix is the transactional outbox: insert the event into an outbox table in the same transaction as the business change. A relay worker reads unpublished rows and hands them to the webhook system.

The handoff must be idempotent, because the relay will sometimes publish and crash before marking the row. Give every publish an idempotency key derived from the event (invoice-paid:inv_123). The intake stores the key with a unique constraint and, on a repeat with identical content, returns the original message instead of creating a second one. A repeat with different content under the same key is a bug and should fail loudly (409 Conflict).

The intake responds 202 Accepted with a durable receipt as soon as the message is committed. Delivery is always asynchronous: never make a producer wait on a customer's server. Enforce a payload limit here (256 KiB is generous) and validate the event type.

2. Fan-out: resolve subscriptions once

For each message, find the consumer's endpoints subscribed to the event type (or to *) and create one deliveries row per endpoint, in the same transaction as the message or in a quick follow-up step. Resolve this at accept time and store it. Two consequences worth documenting for your customers:

• An endpoint added after a message was published does not receive that message.
• Changing an endpoint's subscriptions affects only future messages.

That snapshot makes replays deterministic: "replay message X" always means the same set of endpoints.

3. Per-endpoint lanes, not one global queue

The most common design mistake is a single FIFO queue in front of a worker pool. When one customer's endpoint starts timing out after 30 seconds, it holds workers hostage and every other customer's webhooks queue behind it: head-of-line blocking.

Instead, treat each endpoint as its own lane with:

• a rate limit (a token bucket, for example 50 requests per second by default, configurable per endpoint),
• a concurrency cap (for example at most 16 requests in flight),
• its own retry schedule, independent of other endpoints.

You don't need a physical queue per endpoint. A common implementation is the deliveries table itself as the queue: workers claim due rows with SELECT ... WHERE next_attempt_at <= now() AND status IN ('received','stalled') ORDER BY next_attempt_at LIMIT n FOR UPDATE SKIP LOCKED, skipping endpoints whose token bucket is empty or whose in-flight count is at the cap (tracked in Redis or a counter table). Broker-based designs work too (one stream per endpoint in Redis, or a broker with per-key fairness), but you still need the scheduler for multi-hour retry delays, which brokers are bad at.

4. Delivery workers: sign, send, record

Each attempt:

1. Signs the request fresh, with the current timestamp. With Standard Webhooks, that is webhook-id (the message ID, stable forever), webhook-timestamp (now) and webhook-signature: v1,<base64 HMAC-SHA256 of "id.timestamp.body">. During a secret rotation, include one signature per active secret.
2. Resolves and checks the destination. Resolve DNS and refuse private, loopback and link-local addresses at connect time, not just at registration, so a DNS change can't point an endpoint at 169.254.169.254.
3. Sends with a short timeout (15 seconds is a good default, configurable up to 60) and does not follow redirects; a 3xx is a failure.
4. Reads a bounded response. Keep the status, headers (capped, for example 8 KiB) and the start of the body (for example 4 KiB). Never buffer an unbounded body from a customer server.
5. Records the attempt and classifies the outcome for the scheduler.

5. The retry scheduler

Classify each result:

A schedule that works in practice, and is what Webhook Relay's Outbound Gateway uses: retry after 5 seconds, 5 minutes, 30 minutes, 2 hours, 5 hours, 10 hours and 10 hours, at most 8 attempts and never later than 48 hours after the message was accepted. Add jitter so retries from a shared outage don't arrive as one wave.

6. Endpoint health as a state machine

Per-delivery retries aren't enough on their own: an endpoint that has been down for a day still receives first attempts for every new event. Track state on the endpoint:

            5 failures in a row, or a delivery runs out of retries
  ┌────────┐ ────────────────────────────────────────▶ ┌─────────┐
  │ active │ ◀──────────────── any 2xx ─────────────── │ failing │
  └────────┘                                           └─────────┘
    │    ▲  ▲                                               │
    │    │  └──────────── resume ─────────────┐             │ 410 Gone, or auto-disable
    │    │ resume                             │             │ (100 failures in a row, ≥ 12h)
    ▼    │                                    │             ▼
  ┌────────┐                                  │       ┌──────────┐
  │ paused │                                  └────── │ disabled │
  └────────┘                                          └──────────┘
   pause (down arrow) and resume are manual; paused and disabled endpoints
   skip deliveries, which can be replayed after resume

A 410 Gone disables the endpoint from any state. Expose the state, failing_since and a 24-hour failure count per endpoint, plus an account-wide view of every failing endpoint, so your support team and your customers see problems before the customer notices missing data.

7. Delivery log, retention and recovery

The delivery log is what turns "we never got it" from an investigation into a lookup. Keep every attempt for a fixed window (30 days is common; Webhook Relay keeps 35) and make three recovery operations first-class, running as background tasks:

• Retry one: re-send one message to one endpoint now.
• Recover: re-send every delivery to an endpoint that exhausted its retries since a given time (the customer's outage started at 14:00 Tuesday).
• Replay missing: send what was skipped while the endpoint was paused or disabled.

Every copy carries the original webhook-id, so receivers that deduplicate process it once. Partition the attempts table by day so expiring old data is cheap.

8. Observability

Instrument the pipeline, not just the HTTP client:

• publish rate and intake errors (409, 413, backpressure 429),
• outbox age (oldest unpublished row) and scheduler lag (oldest due delivery),
• attempts per second, success rate and p95 latency, overall and per endpoint,
• count of endpoints by state, and the longest-failing endpoints.

Alert on outbox age and scheduler lag; those mean you are failing. Customer endpoints failing is normal and belongs on a dashboard, not a pager.

9. Security checklist

• A separate signing secret per endpoint, encrypted at rest, revealable only to admins, with a 24-hour dual-signing window on rotation.
• SSRF protection at registration and at connect time; HTTPS only.
• Reserved headers (webhook-*, Host, Content-Length) that customers can't override with custom headers.
• Narrow credentials for producers: a token that can only publish, separate from the one that manages endpoints.
• Payload hygiene: no secrets or full card numbers in events, because they live in the delivery log.

Build vs buy: what it actually costs

Everything above is a realistic first version. Built by an experienced team, expect roughly 6-10 engineer-weeks for intake, fan-out, lanes, scheduler, signing, health, the log, recovery tooling and a minimal admin UI, then ongoing ownership: on-call, Postgres growth, retention jobs, customer support tooling and SDK or docs work. At a fully loaded cost of around $3,000 per engineer-week, that is about $18,000-30,000 up front plus a fraction of an engineer each year, before infrastructure.

Hosted options, at list prices checked on 2026-10-10:

Webhook Relay prices on how many customers you serve (Basic 100, Business 500, Pro 1,000), not on messages; retries and fan-out are free and outbound messages don't count against the inbound webhook quota. The full matrix, including portals, compliance and self-hosting, is in webhooks as a service compared.

Build it yourself when webhooks are your product's core, or you need something hosted services don't offer: strict FIFO ordering, non-HTTP destinations like SQS or Kafka, on-premises deployment, an embeddable customer portal from day one, or more than 1,000 customers on one account. Svix, Hookdeck Outpost and Convoy also have self-hostable open-source servers if you want a head start.

Buy when webhooks are a feature your customers expect rather than the thing you sell. Webhook Relay's Outbound Gateway implements the design in this post as a hosted service: an idempotent publish API with a durable outbox, fan-out, per-endpoint lanes (50 requests per second by default, up to 1,000), Standard Webhooks signing with rotation, the retry schedule above, health states with optional auto-disable, and 35 days of per-attempt history with retry, recover and replay. You call one method from the REST API, the TypeScript or Go SDK, or MCP.

Next steps

• Read the Outbound Gateway docs or jump into a tutorial: Node.js / TypeScript, Go with a transactional outbox, or a Next.js SaaS settings page.
• Already running your own sender? The migration guide for homegrown senders maps outbox tables and retry workers onto the Outbound Gateway with a dual-publish cutover.
• Receiving side: webhook retries and idempotency and how to verify a webhook signature.