[{"data":1,"prerenderedAt":2090},["ShallowReactive",2],{"content-query-BRkhyR651x":3},{"_path":4,"_dir":5,"_draft":6,"_partial":6,"_locale":7,"title":8,"description":9,"layout":10,"head":11,"author":13,"tags":14,"categories":21,"date":23,"image":24,"faq":25,"body":41,"_type":2083,"_id":2084,"_source":2085,"_file":2086,"_stem":2087,"_extension":2088,"sitemap":2089,"readingTime":183},"/blog/webhook-system-design","blog",false,"","Webhook System Design: How to Build a Webhook Delivery System","How to implement webhooks: the architecture of a webhook delivery system (outbox, fan-out, per-endpoint queues, retries, signing, delivery log) and build vs buy costs.","post",{"title":12},"Webhook System Design: Architecture, Retries, Build vs Buy","Karolis Rusenas",[15,16,17,18,19,20],"webhook system design","how to implement webhooks","outbound webhooks","webhooks as a service","architecture","reliability",[22],"guides","2026-10-25 09:00:00","/images/blog/heroes/route.jpg",[26,29,32,35,38],{"q":27,"a":28},"How do you design a webhook delivery system?","Split it into seven parts: a durable intake that accepts events through a transactional outbox or an idempotent publish API, a fan-out step that creates one delivery per subscribed endpoint, per-endpoint queues with their own rate and concurrency limits, HTTP workers that sign each attempt and enforce timeouts, a retry scheduler with exponential backoff and a deadline, an endpoint health state machine, and a delivery log that supports retry and replay.",{"q":30,"a":31},"How do I implement webhooks in my application?","Define event types, let each customer register HTTPS endpoints and subscribe to event types, and give every endpoint its own signing secret. When an event happens, write it to an outbox table in the same transaction as the business change, then have a worker deliver it to each subscribed endpoint with an HMAC signature, retry failures with backoff and record every attempt. Or publish the event to a webhooks as a service API that does the delivery for you.",{"q":33,"a":34},"Should I use Kafka or SQS to send webhooks?","A queue is a useful building block but not a webhook system. A single shared queue causes head-of-line blocking when one customer's endpoint is slow, and queues do not give you per-endpoint rate limits, delayed retries spanning days, endpoint health tracking or a searchable delivery log. Most teams end up adding a scheduler table and per-endpoint lanes on top, or using Postgres alone with SELECT FOR UPDATE SKIP LOCKED.",{"q":36,"a":37},"Can webhooks be delivered exactly once?","No. Over HTTP the sender cannot know whether a receiver processed a request whose response was lost, so every practical system delivers at least once. The sender keeps the event ID stable across retries and replays, and the receiver deduplicates on it, which makes processing effectively once.",{"q":39,"a":40},"Is it cheaper to build or buy webhook delivery?","For most SaaS teams buying is cheaper. A production-grade sender typically takes six to ten engineer-weeks to build and a slice of an engineer to operate, while hosted services start at $9.99 per month on Webhook Relay Basic or $20 per month plus $0.0001 per message after 50,000 on Svix Basic (list prices checked on 2026-10-10). Building makes sense when you need ordering, non-HTTP destinations, on-premises deployment or scale beyond what hosted plans offer.",{"type":42,"children":43,"toc":2067},"root",[44,58,72,79,325,331,336,500,505,511,516,922,928,956,991,1004,1010,1037,1052,1057,1063,1068,1080,1118,1138,1144,1149,1259,1265,1270,1435,1447,1453,1458,1569,1588,1594,1599,1632,1644,1650,1655,1700,1712,1718,1768,1774,1793,1798,1944,1956,1966,1984,1990,2061],{"type":45,"tag":46,"props":47,"children":48},"element","p",{},[49,56],{"type":45,"tag":50,"props":51,"children":52},"strong",{},[53],{"type":54,"value":55},"text","A webhook sending system has seven parts: a durable intake (a transactional outbox or an idempotent publish API), a fan-out step that creates one delivery per subscribed endpoint, per-endpoint queues with their own rate and concurrency limits, HTTP workers that sign every attempt, a retry scheduler with backoff and a deadline, an endpoint health state machine, and a delivery log with retry and replay.",{"type":54,"value":57}," Each part exists because of a specific failure: a crash between commit and send, one slow customer delaying everyone, a retry storm against an endpoint that is already down, or a customer asking \"did you send it?\" with no way to answer.",{"type":45,"tag":46,"props":59,"children":60},{},[61,63,70],{"type":54,"value":62},"This post walks through each component, the data model behind it, and the trade-offs, then ends with an honest build-versus-buy comparison. If you only need the checklist, see ",{"type":45,"tag":64,"props":65,"children":67},"a",{"href":66},"/blog/webhook-best-practices/",[68],{"type":54,"value":69},"webhook best practices",{"type":54,"value":71},".",{"type":45,"tag":73,"props":74,"children":76},"h2",{"id":75},"the-architecture-at-a-glance",[77],{"type":54,"value":78},"The architecture at a glance",{"type":45,"tag":80,"props":81,"children":84},"pre",{"className":82,"code":83,"language":54,"meta":7,"style":7},"language-text shiki shiki-themes github-dark"," your service                      webhook system\n┌──────────────┐   same tx   ┌───────────┐  poll   ┌──────────┐\n│ business     │────────────▶│  outbox   │────────▶│ publish  │  idempotency key\n│ change       │             │  table    │         │ (intake) │  → 202 + message id\n└──────────────┘             └───────────┘         └────┬─────┘\n                                                        │ fan-out: one delivery\n                                                        │ per subscribed endpoint\n                         ┌──────────────────────────────┼───────────────────────┐\n                         ▼                              ▼                       ▼\n                  ┌─────────────┐               ┌─────────────┐         ┌─────────────┐\n                  │ lane: ep_A  │               │ lane: ep_B  │   ...   │ lane: ep_N  │\n                  │ 50 rps, 16  │               │ 50 rps, 16  │         │ paused      │\n                  │ in flight   │               │ in flight   │         │             │\n                  └──────┬──────┘               └──────┬──────┘         └─────────────┘\n                         ▼                              ▼\n                  ┌─────────────────────────────────────────────┐\n                  │ delivery workers: sign attempt, POST,       │\n                  │ 15s timeout, no redirects, SSRF guard       │\n                  └──────┬───────────────────────┬──────────────┘\n                   2xx   │                       │ error / timeout / 429 / 5xx\n                         ▼                       ▼\n                  ┌─────────────┐        ┌──────────────────┐     ┌──────────────────┐\n                  │ delivery    │◀───────│ retry scheduler  │────▶│ endpoint health  │\n                  │ log (35 d)  │        │ backoff, deadline│     │ active/failing/  │\n                  │ retry,replay│        │ Retry-After      │     │ disabled         │\n                  └─────────────┘        └──────────────────┘     └──────────────────┘\n",[85],{"type":45,"tag":86,"props":87,"children":88},"code",{"__ignoreMap":7},[89,100,109,118,127,136,145,154,163,172,181,190,199,208,217,226,235,244,253,262,271,280,289,298,307,316],{"type":45,"tag":90,"props":91,"children":94},"span",{"class":92,"line":93},"line",1,[95],{"type":45,"tag":90,"props":96,"children":97},{},[98],{"type":54,"value":99}," your service                      webhook system\n",{"type":45,"tag":90,"props":101,"children":103},{"class":92,"line":102},2,[104],{"type":45,"tag":90,"props":105,"children":106},{},[107],{"type":54,"value":108},"┌──────────────┐   same tx   ┌───────────┐  poll   ┌──────────┐\n",{"type":45,"tag":90,"props":110,"children":112},{"class":92,"line":111},3,[113],{"type":45,"tag":90,"props":114,"children":115},{},[116],{"type":54,"value":117},"│ business     │────────────▶│  outbox   │────────▶│ publish  │  idempotency key\n",{"type":45,"tag":90,"props":119,"children":121},{"class":92,"line":120},4,[122],{"type":45,"tag":90,"props":123,"children":124},{},[125],{"type":54,"value":126},"│ change       │             │  table    │         │ (intake) │  → 202 + message id\n",{"type":45,"tag":90,"props":128,"children":130},{"class":92,"line":129},5,[131],{"type":45,"tag":90,"props":132,"children":133},{},[134],{"type":54,"value":135},"└──────────────┘             └───────────┘         └────┬─────┘\n",{"type":45,"tag":90,"props":137,"children":139},{"class":92,"line":138},6,[140],{"type":45,"tag":90,"props":141,"children":142},{},[143],{"type":54,"value":144},"                                                        │ fan-out: one delivery\n",{"type":45,"tag":90,"props":146,"children":148},{"class":92,"line":147},7,[149],{"type":45,"tag":90,"props":150,"children":151},{},[152],{"type":54,"value":153},"                                                        │ per subscribed endpoint\n",{"type":45,"tag":90,"props":155,"children":157},{"class":92,"line":156},8,[158],{"type":45,"tag":90,"props":159,"children":160},{},[161],{"type":54,"value":162},"                         ┌──────────────────────────────┼───────────────────────┐\n",{"type":45,"tag":90,"props":164,"children":166},{"class":92,"line":165},9,[167],{"type":45,"tag":90,"props":168,"children":169},{},[170],{"type":54,"value":171},"                         ▼                              ▼                       ▼\n",{"type":45,"tag":90,"props":173,"children":175},{"class":92,"line":174},10,[176],{"type":45,"tag":90,"props":177,"children":178},{},[179],{"type":54,"value":180},"                  ┌─────────────┐               ┌─────────────┐         ┌─────────────┐\n",{"type":45,"tag":90,"props":182,"children":184},{"class":92,"line":183},11,[185],{"type":45,"tag":90,"props":186,"children":187},{},[188],{"type":54,"value":189},"                  │ lane: ep_A  │               │ lane: ep_B  │   ...   │ lane: ep_N  │\n",{"type":45,"tag":90,"props":191,"children":193},{"class":92,"line":192},12,[194],{"type":45,"tag":90,"props":195,"children":196},{},[197],{"type":54,"value":198},"                  │ 50 rps, 16  │               │ 50 rps, 16  │         │ paused      │\n",{"type":45,"tag":90,"props":200,"children":202},{"class":92,"line":201},13,[203],{"type":45,"tag":90,"props":204,"children":205},{},[206],{"type":54,"value":207},"                  │ in flight   │               │ in flight   │         │             │\n",{"type":45,"tag":90,"props":209,"children":211},{"class":92,"line":210},14,[212],{"type":45,"tag":90,"props":213,"children":214},{},[215],{"type":54,"value":216},"                  └──────┬──────┘               └──────┬──────┘         └─────────────┘\n",{"type":45,"tag":90,"props":218,"children":220},{"class":92,"line":219},15,[221],{"type":45,"tag":90,"props":222,"children":223},{},[224],{"type":54,"value":225},"                         ▼                              ▼\n",{"type":45,"tag":90,"props":227,"children":229},{"class":92,"line":228},16,[230],{"type":45,"tag":90,"props":231,"children":232},{},[233],{"type":54,"value":234},"                  ┌─────────────────────────────────────────────┐\n",{"type":45,"tag":90,"props":236,"children":238},{"class":92,"line":237},17,[239],{"type":45,"tag":90,"props":240,"children":241},{},[242],{"type":54,"value":243},"                  │ delivery workers: sign attempt, POST,       │\n",{"type":45,"tag":90,"props":245,"children":247},{"class":92,"line":246},18,[248],{"type":45,"tag":90,"props":249,"children":250},{},[251],{"type":54,"value":252},"                  │ 15s timeout, no redirects, SSRF guard       │\n",{"type":45,"tag":90,"props":254,"children":256},{"class":92,"line":255},19,[257],{"type":45,"tag":90,"props":258,"children":259},{},[260],{"type":54,"value":261},"                  └──────┬───────────────────────┬──────────────┘\n",{"type":45,"tag":90,"props":263,"children":265},{"class":92,"line":264},20,[266],{"type":45,"tag":90,"props":267,"children":268},{},[269],{"type":54,"value":270},"                   2xx   │                       │ error / timeout / 429 / 5xx\n",{"type":45,"tag":90,"props":272,"children":274},{"class":92,"line":273},21,[275],{"type":45,"tag":90,"props":276,"children":277},{},[278],{"type":54,"value":279},"                         ▼                       ▼\n",{"type":45,"tag":90,"props":281,"children":283},{"class":92,"line":282},22,[284],{"type":45,"tag":90,"props":285,"children":286},{},[287],{"type":54,"value":288},"                  ┌─────────────┐        ┌──────────────────┐     ┌──────────────────┐\n",{"type":45,"tag":90,"props":290,"children":292},{"class":92,"line":291},23,[293],{"type":45,"tag":90,"props":294,"children":295},{},[296],{"type":54,"value":297},"                  │ delivery    │◀───────│ retry scheduler  │────▶│ endpoint health  │\n",{"type":45,"tag":90,"props":299,"children":301},{"class":92,"line":300},24,[302],{"type":45,"tag":90,"props":303,"children":304},{},[305],{"type":54,"value":306},"                  │ log (35 d)  │        │ backoff, deadline│     │ active/failing/  │\n",{"type":45,"tag":90,"props":308,"children":310},{"class":92,"line":309},25,[311],{"type":45,"tag":90,"props":312,"children":313},{},[314],{"type":54,"value":315},"                  │ retry,replay│        │ Retry-After      │     │ disabled         │\n",{"type":45,"tag":90,"props":317,"children":319},{"class":92,"line":318},26,[320],{"type":45,"tag":90,"props":321,"children":322},{},[323],{"type":54,"value":324},"                  └─────────────┘        └──────────────────┘     └──────────────────┘\n",{"type":45,"tag":73,"props":326,"children":328},{"id":327},"requirements-first",[329],{"type":54,"value":330},"Requirements first",{"type":45,"tag":46,"props":332,"children":333},{},[334],{"type":54,"value":335},"Write these down before choosing technology, because they decide the design:",{"type":45,"tag":337,"props":338,"children":339},"table",{},[340,359],{"type":45,"tag":341,"props":342,"children":343},"thead",{},[344],{"type":45,"tag":345,"props":346,"children":347},"tr",{},[348,354],{"type":45,"tag":349,"props":350,"children":351},"th",{},[352],{"type":54,"value":353},"Requirement",{"type":45,"tag":349,"props":355,"children":356},{},[357],{"type":54,"value":358},"What it means in practice",{"type":45,"tag":360,"props":361,"children":362},"tbody",{},[363,380,396,412,428,444,460,476],{"type":45,"tag":345,"props":364,"children":365},{},[366,375],{"type":45,"tag":367,"props":368,"children":369},"td",{},[370],{"type":45,"tag":50,"props":371,"children":372},{},[373],{"type":54,"value":374},"At-least-once delivery",{"type":45,"tag":367,"props":376,"children":377},{},[378],{"type":54,"value":379},"An accepted event is never silently dropped, across deploys, crashes and failovers. Duplicates are allowed; the event ID makes them harmless.",{"type":45,"tag":345,"props":381,"children":382},{},[383,391],{"type":45,"tag":367,"props":384,"children":385},{},[386],{"type":45,"tag":50,"props":387,"children":388},{},[389],{"type":54,"value":390},"Tenant isolation",{"type":45,"tag":367,"props":392,"children":393},{},[394],{"type":54,"value":395},"A slow or failing customer endpoint never delays another customer's deliveries.",{"type":45,"tag":345,"props":397,"children":398},{},[399,407],{"type":45,"tag":367,"props":400,"children":401},{},[402],{"type":45,"tag":50,"props":403,"children":404},{},[405],{"type":54,"value":406},"Bounded retries",{"type":45,"tag":367,"props":408,"children":409},{},[410],{"type":54,"value":411},"Retry for long enough to outlast a normal outage (one to three days), then stop and keep the failure replayable.",{"type":45,"tag":345,"props":413,"children":414},{},[415,423],{"type":45,"tag":367,"props":416,"children":417},{},[418],{"type":45,"tag":50,"props":419,"children":420},{},[421],{"type":54,"value":422},"Authenticity",{"type":45,"tag":367,"props":424,"children":425},{},[426],{"type":54,"value":427},"Receivers can verify every request came from you and was not modified.",{"type":45,"tag":345,"props":429,"children":430},{},[431,439],{"type":45,"tag":367,"props":432,"children":433},{},[434],{"type":45,"tag":50,"props":435,"children":436},{},[437],{"type":54,"value":438},"Safety",{"type":45,"tag":367,"props":440,"children":441},{},[442],{"type":54,"value":443},"Customer-supplied URLs can't reach your internal network (SSRF).",{"type":45,"tag":345,"props":445,"children":446},{},[447,455],{"type":45,"tag":367,"props":448,"children":449},{},[450],{"type":45,"tag":50,"props":451,"children":452},{},[453],{"type":54,"value":454},"Observability",{"type":45,"tag":367,"props":456,"children":457},{},[458],{"type":54,"value":459},"Support can see every attempt and response for weeks, and re-send on demand.",{"type":45,"tag":345,"props":461,"children":462},{},[463,471],{"type":45,"tag":367,"props":464,"children":465},{},[466],{"type":45,"tag":50,"props":467,"children":468},{},[469],{"type":54,"value":470},"Latency",{"type":45,"tag":367,"props":472,"children":473},{},[474],{"type":54,"value":475},"Typical delivery within a second or two of the event.",{"type":45,"tag":345,"props":477,"children":478},{},[479,487],{"type":45,"tag":367,"props":480,"children":481},{},[482],{"type":45,"tag":50,"props":483,"children":484},{},[485],{"type":54,"value":486},"Ordering",{"type":45,"tag":367,"props":488,"children":489},{},[490,492,498],{"type":54,"value":491},"Usually ",{"type":45,"tag":493,"props":494,"children":495},"em",{},[496],{"type":54,"value":497},"not",{"type":54,"value":499}," guaranteed. Strict per-customer ordering forces one in-flight request per endpoint and makes one failure block the rest; decide explicitly.",{"type":45,"tag":46,"props":501,"children":502},{},[503],{"type":54,"value":504},"A sizing example helps: 1 million events a day is about 12 per second on average. With peaks at 10x and an average of 2 endpoints per event, design for roughly 250 delivery attempts per second at peak, plus retries. That is well within a single Postgres primary and a small worker pool; you don't need Kafka to start.",{"type":45,"tag":73,"props":506,"children":508},{"id":507},"data-model",[509],{"type":54,"value":510},"Data model",{"type":45,"tag":46,"props":512,"children":513},{},[514],{"type":54,"value":515},"Six tables carry the whole system:",{"type":45,"tag":337,"props":517,"children":518},{},[519,540],{"type":45,"tag":341,"props":520,"children":521},{},[522],{"type":45,"tag":345,"props":523,"children":524},{},[525,530,535],{"type":45,"tag":349,"props":526,"children":527},{},[528],{"type":54,"value":529},"Table",{"type":45,"tag":349,"props":531,"children":532},{},[533],{"type":54,"value":534},"Key columns",{"type":45,"tag":349,"props":536,"children":537},{},[538],{"type":54,"value":539},"Notes",{"type":45,"tag":360,"props":541,"children":542},{},[543,598,638,726,785,839],{"type":45,"tag":345,"props":544,"children":545},{},[546,555,586],{"type":45,"tag":367,"props":547,"children":548},{},[549],{"type":45,"tag":86,"props":550,"children":552},{"className":551},[],[553],{"type":54,"value":554},"event_types",{"type":45,"tag":367,"props":556,"children":557},{},[558,564,566,572,573,579,580],{"type":45,"tag":86,"props":559,"children":561},{"className":560},[],[562],{"type":54,"value":563},"name",{"type":54,"value":565},", ",{"type":45,"tag":86,"props":567,"children":569},{"className":568},[],[570],{"type":54,"value":571},"description",{"type":54,"value":565},{"type":45,"tag":86,"props":574,"children":576},{"className":575},[],[577],{"type":54,"value":578},"example",{"type":54,"value":565},{"type":45,"tag":86,"props":581,"children":583},{"className":582},[],[584],{"type":54,"value":585},"deprecated",{"type":45,"tag":367,"props":587,"children":588},{},[589,591,597],{"type":54,"value":590},"Your public catalog, such as ",{"type":45,"tag":86,"props":592,"children":594},{"className":593},[],[595],{"type":54,"value":596},"invoice.paid",{"type":54,"value":71},{"type":45,"tag":345,"props":599,"children":600},{},[601,610,633],{"type":45,"tag":367,"props":602,"children":603},{},[604],{"type":45,"tag":86,"props":605,"children":607},{"className":606},[],[608],{"type":54,"value":609},"consumers",{"type":45,"tag":367,"props":611,"children":612},{},[613,619,621,626,627],{"type":45,"tag":86,"props":614,"children":616},{"className":615},[],[617],{"type":54,"value":618},"id",{"type":54,"value":620}," (your customer ID), ",{"type":45,"tag":86,"props":622,"children":624},{"className":623},[],[625],{"type":54,"value":563},{"type":54,"value":565},{"type":45,"tag":86,"props":628,"children":630},{"className":629},[],[631],{"type":54,"value":632},"default_rate",{"type":45,"tag":367,"props":634,"children":635},{},[636],{"type":54,"value":637},"One per customer or tenant.",{"type":45,"tag":345,"props":639,"children":640},{},[641,650,721],{"type":45,"tag":367,"props":642,"children":643},{},[644],{"type":45,"tag":86,"props":645,"children":647},{"className":646},[],[648],{"type":54,"value":649},"endpoints",{"type":45,"tag":367,"props":651,"children":652},{},[653,658,659,665,666,672,673,679,680,686,687,693,694,700,701,707,708,714,715],{"type":45,"tag":86,"props":654,"children":656},{"className":655},[],[657],{"type":54,"value":618},{"type":54,"value":565},{"type":45,"tag":86,"props":660,"children":662},{"className":661},[],[663],{"type":54,"value":664},"consumer_id",{"type":54,"value":565},{"type":45,"tag":86,"props":667,"children":669},{"className":668},[],[670],{"type":54,"value":671},"url",{"type":54,"value":565},{"type":45,"tag":86,"props":674,"children":676},{"className":675},[],[677],{"type":54,"value":678},"event_types[]",{"type":54,"value":565},{"type":45,"tag":86,"props":681,"children":683},{"className":682},[],[684],{"type":54,"value":685},"secrets[]",{"type":54,"value":565},{"type":45,"tag":86,"props":688,"children":690},{"className":689},[],[691],{"type":54,"value":692},"rate",{"type":54,"value":565},{"type":45,"tag":86,"props":695,"children":697},{"className":696},[],[698],{"type":54,"value":699},"timeout",{"type":54,"value":565},{"type":45,"tag":86,"props":702,"children":704},{"className":703},[],[705],{"type":54,"value":706},"state",{"type":54,"value":565},{"type":45,"tag":86,"props":709,"children":711},{"className":710},[],[712],{"type":54,"value":713},"consecutive_failures",{"type":54,"value":565},{"type":45,"tag":86,"props":716,"children":718},{"className":717},[],[719],{"type":54,"value":720},"failing_since",{"type":45,"tag":367,"props":722,"children":723},{},[724],{"type":54,"value":725},"Secrets encrypted at rest; two during rotation.",{"type":45,"tag":345,"props":727,"children":728},{},[729,738,780],{"type":45,"tag":367,"props":730,"children":731},{},[732],{"type":45,"tag":86,"props":733,"children":735},{"className":734},[],[736],{"type":54,"value":737},"messages",{"type":45,"tag":367,"props":739,"children":740},{},[741,746,747,752,753,759,760,766,767,773,774],{"type":45,"tag":86,"props":742,"children":744},{"className":743},[],[745],{"type":54,"value":618},{"type":54,"value":565},{"type":45,"tag":86,"props":748,"children":750},{"className":749},[],[751],{"type":54,"value":664},{"type":54,"value":565},{"type":45,"tag":86,"props":754,"children":756},{"className":755},[],[757],{"type":54,"value":758},"event_type",{"type":54,"value":565},{"type":45,"tag":86,"props":761,"children":763},{"className":762},[],[764],{"type":54,"value":765},"payload",{"type":54,"value":565},{"type":45,"tag":86,"props":768,"children":770},{"className":769},[],[771],{"type":54,"value":772},"idempotency_key",{"type":54,"value":565},{"type":45,"tag":86,"props":775,"children":777},{"className":776},[],[778],{"type":54,"value":779},"created_at",{"type":45,"tag":367,"props":781,"children":782},{},[783],{"type":54,"value":784},"Unique on (account, idempotency key).",{"type":45,"tag":345,"props":786,"children":787},{},[788,797,834],{"type":45,"tag":367,"props":789,"children":790},{},[791],{"type":45,"tag":86,"props":792,"children":794},{"className":793},[],[795],{"type":54,"value":796},"deliveries",{"type":45,"tag":367,"props":798,"children":799},{},[800,806,807,813,814,820,821,827,828],{"type":45,"tag":86,"props":801,"children":803},{"className":802},[],[804],{"type":54,"value":805},"message_id",{"type":54,"value":565},{"type":45,"tag":86,"props":808,"children":810},{"className":809},[],[811],{"type":54,"value":812},"endpoint_id",{"type":54,"value":565},{"type":45,"tag":86,"props":815,"children":817},{"className":816},[],[818],{"type":54,"value":819},"status",{"type":54,"value":565},{"type":45,"tag":86,"props":822,"children":824},{"className":823},[],[825],{"type":54,"value":826},"attempt_count",{"type":54,"value":565},{"type":45,"tag":86,"props":829,"children":831},{"className":830},[],[832],{"type":54,"value":833},"next_attempt_at",{"type":45,"tag":367,"props":835,"children":836},{},[837],{"type":54,"value":838},"One per message per endpoint; the scheduler's work queue.",{"type":45,"tag":345,"props":840,"children":841},{},[842,851,910],{"type":45,"tag":367,"props":843,"children":844},{},[845],{"type":45,"tag":86,"props":846,"children":848},{"className":847},[],[849],{"type":54,"value":850},"attempts",{"type":45,"tag":367,"props":852,"children":853},{},[854,860,861,867,868,874,875,881,882,888,889,895,896,902,904],{"type":45,"tag":86,"props":855,"children":857},{"className":856},[],[858],{"type":54,"value":859},"delivery_id",{"type":54,"value":565},{"type":45,"tag":86,"props":862,"children":864},{"className":863},[],[865],{"type":54,"value":866},"attempt_no",{"type":54,"value":565},{"type":45,"tag":86,"props":869,"children":871},{"className":870},[],[872],{"type":54,"value":873},"started_at",{"type":54,"value":565},{"type":45,"tag":86,"props":876,"children":878},{"className":877},[],[879],{"type":54,"value":880},"duration_ms",{"type":54,"value":565},{"type":45,"tag":86,"props":883,"children":885},{"className":884},[],[886],{"type":54,"value":887},"status_code",{"type":54,"value":565},{"type":45,"tag":86,"props":890,"children":892},{"className":891},[],[893],{"type":54,"value":894},"response_headers",{"type":54,"value":565},{"type":45,"tag":86,"props":897,"children":899},{"className":898},[],[900],{"type":54,"value":901},"response_body",{"type":54,"value":903}," (truncated), ",{"type":45,"tag":86,"props":905,"children":907},{"className":906},[],[908],{"type":54,"value":909},"error",{"type":45,"tag":367,"props":911,"children":912},{},[913,915,921],{"type":54,"value":914},"Append-only; partition by day so retention is a ",{"type":45,"tag":86,"props":916,"children":918},{"className":917},[],[919],{"type":54,"value":920},"DROP PARTITION",{"type":54,"value":71},{"type":45,"tag":73,"props":923,"children":925},{"id":924},"_1-intake-the-transactional-outbox-and-an-idempotent-publish",[926],{"type":54,"value":927},"1. Intake: the transactional outbox and an idempotent publish",{"type":45,"tag":46,"props":929,"children":930},{},[931,933,939,941,946,948,954],{"type":54,"value":932},"The first place webhooks get lost is inside the producer. A handler commits ",{"type":45,"tag":86,"props":934,"children":936},{"className":935},[],[937],{"type":54,"value":938},"UPDATE invoices SET paid = true",{"type":54,"value":940},", then crashes before sending the event. The fix is the ",{"type":45,"tag":50,"props":942,"children":943},{},[944],{"type":54,"value":945},"transactional outbox",{"type":54,"value":947},": insert the event into an ",{"type":45,"tag":86,"props":949,"children":951},{"className":950},[],[952],{"type":54,"value":953},"outbox",{"type":54,"value":955}," table in the same transaction as the business change. A relay worker reads unpublished rows and hands them to the webhook system.",{"type":45,"tag":46,"props":957,"children":958},{},[959,961,966,968,974,976,981,983,989],{"type":54,"value":960},"The handoff must be idempotent, because the relay will sometimes publish and crash before marking the row. Give every publish an ",{"type":45,"tag":50,"props":962,"children":963},{},[964],{"type":54,"value":965},"idempotency key",{"type":54,"value":967}," derived from the event (",{"type":45,"tag":86,"props":969,"children":971},{"className":970},[],[972],{"type":54,"value":973},"invoice-paid:inv_123",{"type":54,"value":975},"). The intake stores the key with a unique constraint and, on a repeat with identical content, returns the original message instead of creating a second one. A repeat with ",{"type":45,"tag":493,"props":977,"children":978},{},[979],{"type":54,"value":980},"different",{"type":54,"value":982}," content under the same key is a bug and should fail loudly (",{"type":45,"tag":86,"props":984,"children":986},{"className":985},[],[987],{"type":54,"value":988},"409 Conflict",{"type":54,"value":990},").",{"type":45,"tag":46,"props":992,"children":993},{},[994,996,1002],{"type":54,"value":995},"The intake responds ",{"type":45,"tag":86,"props":997,"children":999},{"className":998},[],[1000],{"type":54,"value":1001},"202 Accepted",{"type":54,"value":1003}," with a durable receipt as soon as the message is committed. Delivery is always asynchronous: never make a producer wait on a customer's server. Enforce a payload limit here (256 KiB is generous) and validate the event type.",{"type":45,"tag":73,"props":1005,"children":1007},{"id":1006},"_2-fan-out-resolve-subscriptions-once",[1008],{"type":54,"value":1009},"2. Fan-out: resolve subscriptions once",{"type":45,"tag":46,"props":1011,"children":1012},{},[1013,1015,1021,1023,1028,1030,1035],{"type":54,"value":1014},"For each message, find the consumer's endpoints subscribed to the event type (or to ",{"type":45,"tag":86,"props":1016,"children":1018},{"className":1017},[],[1019],{"type":54,"value":1020},"*",{"type":54,"value":1022},") and create one ",{"type":45,"tag":86,"props":1024,"children":1026},{"className":1025},[],[1027],{"type":54,"value":796},{"type":54,"value":1029}," row per endpoint, in the same transaction as the message or in a quick follow-up step. Resolve this ",{"type":45,"tag":50,"props":1031,"children":1032},{},[1033],{"type":54,"value":1034},"at accept time",{"type":54,"value":1036}," and store it. Two consequences worth documenting for your customers:",{"type":45,"tag":1038,"props":1039,"children":1040},"ul",{},[1041,1047],{"type":45,"tag":1042,"props":1043,"children":1044},"li",{},[1045],{"type":54,"value":1046},"An endpoint added after a message was published does not receive that message.",{"type":45,"tag":1042,"props":1048,"children":1049},{},[1050],{"type":54,"value":1051},"Changing an endpoint's subscriptions affects only future messages.",{"type":45,"tag":46,"props":1053,"children":1054},{},[1055],{"type":54,"value":1056},"That snapshot makes replays deterministic: \"replay message X\" always means the same set of endpoints.",{"type":45,"tag":73,"props":1058,"children":1060},{"id":1059},"_3-per-endpoint-lanes-not-one-global-queue",[1061],{"type":54,"value":1062},"3. Per-endpoint lanes, not one global queue",{"type":45,"tag":46,"props":1064,"children":1065},{},[1066],{"type":54,"value":1067},"The most common design mistake is a single FIFO queue in front of a worker pool. When one customer's endpoint starts timing out after 30 seconds, it holds workers hostage and every other customer's webhooks queue behind it: head-of-line blocking.",{"type":45,"tag":46,"props":1069,"children":1070},{},[1071,1073,1078],{"type":54,"value":1072},"Instead, treat each endpoint as its own ",{"type":45,"tag":50,"props":1074,"children":1075},{},[1076],{"type":54,"value":1077},"lane",{"type":54,"value":1079}," with:",{"type":45,"tag":1038,"props":1081,"children":1082},{},[1083,1095,1106],{"type":45,"tag":1042,"props":1084,"children":1085},{},[1086,1088,1093],{"type":54,"value":1087},"a ",{"type":45,"tag":50,"props":1089,"children":1090},{},[1091],{"type":54,"value":1092},"rate limit",{"type":54,"value":1094}," (a token bucket, for example 50 requests per second by default, configurable per endpoint),",{"type":45,"tag":1042,"props":1096,"children":1097},{},[1098,1099,1104],{"type":54,"value":1087},{"type":45,"tag":50,"props":1100,"children":1101},{},[1102],{"type":54,"value":1103},"concurrency cap",{"type":54,"value":1105}," (for example at most 16 requests in flight),",{"type":45,"tag":1042,"props":1107,"children":1108},{},[1109,1111,1116],{"type":54,"value":1110},"its own ",{"type":45,"tag":50,"props":1112,"children":1113},{},[1114],{"type":54,"value":1115},"retry schedule",{"type":54,"value":1117},", independent of other endpoints.",{"type":45,"tag":46,"props":1119,"children":1120},{},[1121,1123,1128,1130,1136],{"type":54,"value":1122},"You don't need a physical queue per endpoint. A common implementation is the ",{"type":45,"tag":86,"props":1124,"children":1126},{"className":1125},[],[1127],{"type":54,"value":796},{"type":54,"value":1129}," table itself as the queue: workers claim due rows with ",{"type":45,"tag":86,"props":1131,"children":1133},{"className":1132},[],[1134],{"type":54,"value":1135},"SELECT ... WHERE next_attempt_at \u003C= now() AND status IN ('received','stalled') ORDER BY next_attempt_at LIMIT n FOR UPDATE SKIP LOCKED",{"type":54,"value":1137},", skipping endpoints whose token bucket is empty or whose in-flight count is at the cap (tracked in Redis or a counter table). Broker-based designs work too (one stream per endpoint in Redis, or a broker with per-key fairness), but you still need the scheduler for multi-hour retry delays, which brokers are bad at.",{"type":45,"tag":73,"props":1139,"children":1141},{"id":1140},"_4-delivery-workers-sign-send-record",[1142],{"type":54,"value":1143},"4. Delivery workers: sign, send, record",{"type":45,"tag":46,"props":1145,"children":1146},{},[1147],{"type":54,"value":1148},"Each attempt:",{"type":45,"tag":1150,"props":1151,"children":1152},"ol",{},[1153,1197,1214,1239,1249],{"type":45,"tag":1042,"props":1154,"children":1155},{},[1156,1161,1163,1171,1173,1179,1181,1187,1189,1195],{"type":45,"tag":50,"props":1157,"children":1158},{},[1159],{"type":54,"value":1160},"Signs",{"type":54,"value":1162}," the request fresh, with the current timestamp. With ",{"type":45,"tag":64,"props":1164,"children":1168},{"href":1165,"rel":1166},"https://www.standardwebhooks.com/",[1167],"nofollow",[1169],{"type":54,"value":1170},"Standard Webhooks",{"type":54,"value":1172},", that is ",{"type":45,"tag":86,"props":1174,"children":1176},{"className":1175},[],[1177],{"type":54,"value":1178},"webhook-id",{"type":54,"value":1180}," (the message ID, stable forever), ",{"type":45,"tag":86,"props":1182,"children":1184},{"className":1183},[],[1185],{"type":54,"value":1186},"webhook-timestamp",{"type":54,"value":1188}," (now) and ",{"type":45,"tag":86,"props":1190,"children":1192},{"className":1191},[],[1193],{"type":54,"value":1194},"webhook-signature: v1,\u003Cbase64 HMAC-SHA256 of \"id.timestamp.body\">",{"type":54,"value":1196},". During a secret rotation, include one signature per active secret.",{"type":45,"tag":1042,"props":1198,"children":1199},{},[1200,1205,1207,1213],{"type":45,"tag":50,"props":1201,"children":1202},{},[1203],{"type":54,"value":1204},"Resolves and checks the destination.",{"type":54,"value":1206}," Resolve DNS and refuse private, loopback and link-local addresses at connect time, not just at registration, so a DNS change can't point an endpoint at ",{"type":45,"tag":86,"props":1208,"children":1210},{"className":1209},[],[1211],{"type":54,"value":1212},"169.254.169.254",{"type":54,"value":71},{"type":45,"tag":1042,"props":1215,"children":1216},{},[1217,1222,1224,1229,1231,1237],{"type":45,"tag":50,"props":1218,"children":1219},{},[1220],{"type":54,"value":1221},"Sends",{"type":54,"value":1223}," with a short timeout (15 seconds is a good default, configurable up to 60) and ",{"type":45,"tag":50,"props":1225,"children":1226},{},[1227],{"type":54,"value":1228},"does not follow redirects",{"type":54,"value":1230},"; a ",{"type":45,"tag":86,"props":1232,"children":1234},{"className":1233},[],[1235],{"type":54,"value":1236},"3xx",{"type":54,"value":1238}," is a failure.",{"type":45,"tag":1042,"props":1240,"children":1241},{},[1242,1247],{"type":45,"tag":50,"props":1243,"children":1244},{},[1245],{"type":54,"value":1246},"Reads a bounded response.",{"type":54,"value":1248}," Keep the status, headers (capped, for example 8 KiB) and the start of the body (for example 4 KiB). Never buffer an unbounded body from a customer server.",{"type":45,"tag":1042,"props":1250,"children":1251},{},[1252,1257],{"type":45,"tag":50,"props":1253,"children":1254},{},[1255],{"type":54,"value":1256},"Records the attempt",{"type":54,"value":1258}," and classifies the outcome for the scheduler.",{"type":45,"tag":73,"props":1260,"children":1262},{"id":1261},"_5-the-retry-scheduler",[1263],{"type":54,"value":1264},"5. The retry scheduler",{"type":45,"tag":46,"props":1266,"children":1267},{},[1268],{"type":54,"value":1269},"Classify each result:",{"type":45,"tag":337,"props":1271,"children":1272},{},[1273,1289],{"type":45,"tag":341,"props":1274,"children":1275},{},[1276],{"type":45,"tag":345,"props":1277,"children":1278},{},[1279,1284],{"type":45,"tag":349,"props":1280,"children":1281},{},[1282],{"type":54,"value":1283},"Result",{"type":45,"tag":349,"props":1285,"children":1286},{},[1287],{"type":54,"value":1288},"Action",{"type":45,"tag":360,"props":1290,"children":1291},{},[1292,1317,1341,1381,1416],{"type":45,"tag":345,"props":1293,"children":1294},{},[1295,1304],{"type":45,"tag":367,"props":1296,"children":1297},{},[1298],{"type":45,"tag":86,"props":1299,"children":1301},{"className":1300},[],[1302],{"type":54,"value":1303},"2xx",{"type":45,"tag":367,"props":1305,"children":1306},{},[1307,1309,1315],{"type":54,"value":1308},"Mark ",{"type":45,"tag":86,"props":1310,"children":1312},{"className":1311},[],[1313],{"type":54,"value":1314},"sent",{"type":54,"value":1316},". Reset the endpoint's failure count.",{"type":45,"tag":345,"props":1318,"children":1319},{},[1320,1329],{"type":45,"tag":367,"props":1321,"children":1322},{},[1323],{"type":45,"tag":86,"props":1324,"children":1326},{"className":1325},[],[1327],{"type":54,"value":1328},"410 Gone",{"type":45,"tag":367,"props":1330,"children":1331},{},[1332,1333,1339],{"type":54,"value":1308},{"type":45,"tag":86,"props":1334,"children":1336},{"className":1335},[],[1337],{"type":54,"value":1338},"failed",{"type":54,"value":1340},", disable the endpoint. The receiver asked you to stop.",{"type":45,"tag":345,"props":1342,"children":1343},{},[1344,1369],{"type":45,"tag":367,"props":1345,"children":1346},{},[1347,1353,1355,1361,1363],{"type":45,"tag":86,"props":1348,"children":1350},{"className":1349},[],[1351],{"type":54,"value":1352},"429",{"type":54,"value":1354}," or ",{"type":45,"tag":86,"props":1356,"children":1358},{"className":1357},[],[1359],{"type":54,"value":1360},"503",{"type":54,"value":1362}," with ",{"type":45,"tag":86,"props":1364,"children":1366},{"className":1365},[],[1367],{"type":54,"value":1368},"Retry-After",{"type":45,"tag":367,"props":1370,"children":1371},{},[1372,1374,1379],{"type":54,"value":1373},"Schedule the next attempt no earlier than ",{"type":45,"tag":86,"props":1375,"children":1377},{"className":1376},[],[1378],{"type":54,"value":1368},{"type":54,"value":1380}," (capped, for example at one hour).",{"type":45,"tag":345,"props":1382,"children":1383},{},[1384,1411],{"type":45,"tag":367,"props":1385,"children":1386},{},[1387,1389,1395,1397,1403,1404,1409],{"type":54,"value":1388},"Other ",{"type":45,"tag":86,"props":1390,"children":1392},{"className":1391},[],[1393],{"type":54,"value":1394},"4xx",{"type":54,"value":1396},"/",{"type":45,"tag":86,"props":1398,"children":1400},{"className":1399},[],[1401],{"type":54,"value":1402},"5xx",{"type":54,"value":565},{"type":45,"tag":86,"props":1405,"children":1407},{"className":1406},[],[1408],{"type":54,"value":1236},{"type":54,"value":1410},", timeout, connection error",{"type":45,"tag":367,"props":1412,"children":1413},{},[1414],{"type":54,"value":1415},"Schedule the next attempt on the backoff schedule.",{"type":45,"tag":345,"props":1417,"children":1418},{},[1419,1424],{"type":45,"tag":367,"props":1420,"children":1421},{},[1422],{"type":54,"value":1423},"Deadline or attempt cap reached",{"type":45,"tag":367,"props":1425,"children":1426},{},[1427,1428,1433],{"type":54,"value":1308},{"type":45,"tag":86,"props":1429,"children":1431},{"className":1430},[],[1432],{"type":54,"value":1338},{"type":54,"value":1434},"; keep it replayable.",{"type":45,"tag":46,"props":1436,"children":1437},{},[1438,1440,1445],{"type":54,"value":1439},"A schedule that works in practice, and is what Webhook Relay's Outbound Gateway uses: retry after ",{"type":45,"tag":50,"props":1441,"children":1442},{},[1443],{"type":54,"value":1444},"5 seconds, 5 minutes, 30 minutes, 2 hours, 5 hours, 10 hours and 10 hours",{"type":54,"value":1446},", at most 8 attempts and never later than 48 hours after the message was accepted. Add jitter so retries from a shared outage don't arrive as one wave.",{"type":45,"tag":73,"props":1448,"children":1450},{"id":1449},"_6-endpoint-health-as-a-state-machine",[1451],{"type":54,"value":1452},"6. Endpoint health as a state machine",{"type":45,"tag":46,"props":1454,"children":1455},{},[1456],{"type":54,"value":1457},"Per-delivery retries aren't enough on their own: an endpoint that has been down for a day still receives first attempts for every new event. Track state on the endpoint:",{"type":45,"tag":80,"props":1459,"children":1461},{"className":82,"code":1460,"language":54,"meta":7,"style":7},"            5 failures in a row, or a delivery runs out of retries\n  ┌────────┐ ────────────────────────────────────────▶ ┌─────────┐\n  │ active │ ◀──────────────── any 2xx ─────────────── │ failing │\n  └────────┘                                           └─────────┘\n    │    ▲  ▲                                               │\n    │    │  └──────────── resume ─────────────┐             │ 410 Gone, or auto-disable\n    │    │ resume                             │             │ (100 failures in a row, ≥ 12h)\n    ▼    │                                    │             ▼\n  ┌────────┐                                  │       ┌──────────┐\n  │ paused │                                  └────── │ disabled │\n  └────────┘                                          └──────────┘\n   pause (down arrow) and resume are manual; paused and disabled endpoints\n   skip deliveries, which can be replayed after resume\n",[1462],{"type":45,"tag":86,"props":1463,"children":1464},{"__ignoreMap":7},[1465,1473,1481,1489,1497,1505,1513,1521,1529,1537,1545,1553,1561],{"type":45,"tag":90,"props":1466,"children":1467},{"class":92,"line":93},[1468],{"type":45,"tag":90,"props":1469,"children":1470},{},[1471],{"type":54,"value":1472},"            5 failures in a row, or a delivery runs out of retries\n",{"type":45,"tag":90,"props":1474,"children":1475},{"class":92,"line":102},[1476],{"type":45,"tag":90,"props":1477,"children":1478},{},[1479],{"type":54,"value":1480},"  ┌────────┐ ────────────────────────────────────────▶ ┌─────────┐\n",{"type":45,"tag":90,"props":1482,"children":1483},{"class":92,"line":111},[1484],{"type":45,"tag":90,"props":1485,"children":1486},{},[1487],{"type":54,"value":1488},"  │ active │ ◀──────────────── any 2xx ─────────────── │ failing │\n",{"type":45,"tag":90,"props":1490,"children":1491},{"class":92,"line":120},[1492],{"type":45,"tag":90,"props":1493,"children":1494},{},[1495],{"type":54,"value":1496},"  └────────┘                                           └─────────┘\n",{"type":45,"tag":90,"props":1498,"children":1499},{"class":92,"line":129},[1500],{"type":45,"tag":90,"props":1501,"children":1502},{},[1503],{"type":54,"value":1504},"    │    ▲  ▲                                               │\n",{"type":45,"tag":90,"props":1506,"children":1507},{"class":92,"line":138},[1508],{"type":45,"tag":90,"props":1509,"children":1510},{},[1511],{"type":54,"value":1512},"    │    │  └──────────── resume ─────────────┐             │ 410 Gone, or auto-disable\n",{"type":45,"tag":90,"props":1514,"children":1515},{"class":92,"line":147},[1516],{"type":45,"tag":90,"props":1517,"children":1518},{},[1519],{"type":54,"value":1520},"    │    │ resume                             │             │ (100 failures in a row, ≥ 12h)\n",{"type":45,"tag":90,"props":1522,"children":1523},{"class":92,"line":156},[1524],{"type":45,"tag":90,"props":1525,"children":1526},{},[1527],{"type":54,"value":1528},"    ▼    │                                    │             ▼\n",{"type":45,"tag":90,"props":1530,"children":1531},{"class":92,"line":165},[1532],{"type":45,"tag":90,"props":1533,"children":1534},{},[1535],{"type":54,"value":1536},"  ┌────────┐                                  │       ┌──────────┐\n",{"type":45,"tag":90,"props":1538,"children":1539},{"class":92,"line":174},[1540],{"type":45,"tag":90,"props":1541,"children":1542},{},[1543],{"type":54,"value":1544},"  │ paused │                                  └────── │ disabled │\n",{"type":45,"tag":90,"props":1546,"children":1547},{"class":92,"line":183},[1548],{"type":45,"tag":90,"props":1549,"children":1550},{},[1551],{"type":54,"value":1552},"  └────────┘                                          └──────────┘\n",{"type":45,"tag":90,"props":1554,"children":1555},{"class":92,"line":192},[1556],{"type":45,"tag":90,"props":1557,"children":1558},{},[1559],{"type":54,"value":1560},"   pause (down arrow) and resume are manual; paused and disabled endpoints\n",{"type":45,"tag":90,"props":1562,"children":1563},{"class":92,"line":201},[1564],{"type":45,"tag":90,"props":1565,"children":1566},{},[1567],{"type":54,"value":1568},"   skip deliveries, which can be replayed after resume\n",{"type":45,"tag":46,"props":1570,"children":1571},{},[1572,1574,1579,1581,1586],{"type":54,"value":1573},"A ",{"type":45,"tag":86,"props":1575,"children":1577},{"className":1576},[],[1578],{"type":54,"value":1328},{"type":54,"value":1580}," disables the endpoint from any state. Expose the state, ",{"type":45,"tag":86,"props":1582,"children":1584},{"className":1583},[],[1585],{"type":54,"value":720},{"type":54,"value":1587}," and a 24-hour failure count per endpoint, plus an account-wide view of every failing endpoint, so your support team and your customers see problems before the customer notices missing data.",{"type":45,"tag":73,"props":1589,"children":1591},{"id":1590},"_7-delivery-log-retention-and-recovery",[1592],{"type":54,"value":1593},"7. Delivery log, retention and recovery",{"type":45,"tag":46,"props":1595,"children":1596},{},[1597],{"type":54,"value":1598},"The delivery log is what turns \"we never got it\" from an investigation into a lookup. Keep every attempt for a fixed window (30 days is common; Webhook Relay keeps 35) and make three recovery operations first-class, running as background tasks:",{"type":45,"tag":1038,"props":1600,"children":1601},{},[1602,1612,1622],{"type":45,"tag":1042,"props":1603,"children":1604},{},[1605,1610],{"type":45,"tag":50,"props":1606,"children":1607},{},[1608],{"type":54,"value":1609},"Retry one",{"type":54,"value":1611},": re-send one message to one endpoint now.",{"type":45,"tag":1042,"props":1613,"children":1614},{},[1615,1620],{"type":45,"tag":50,"props":1616,"children":1617},{},[1618],{"type":54,"value":1619},"Recover",{"type":54,"value":1621},": re-send every delivery to an endpoint that exhausted its retries since a given time (the customer's outage started at 14:00 Tuesday).",{"type":45,"tag":1042,"props":1623,"children":1624},{},[1625,1630],{"type":45,"tag":50,"props":1626,"children":1627},{},[1628],{"type":54,"value":1629},"Replay missing",{"type":54,"value":1631},": send what was skipped while the endpoint was paused or disabled.",{"type":45,"tag":46,"props":1633,"children":1634},{},[1635,1637,1642],{"type":54,"value":1636},"Every copy carries the original ",{"type":45,"tag":86,"props":1638,"children":1640},{"className":1639},[],[1641],{"type":54,"value":1178},{"type":54,"value":1643},", so receivers that deduplicate process it once. Partition the attempts table by day so expiring old data is cheap.",{"type":45,"tag":73,"props":1645,"children":1647},{"id":1646},"_8-observability",[1648],{"type":54,"value":1649},"8. Observability",{"type":45,"tag":46,"props":1651,"children":1652},{},[1653],{"type":54,"value":1654},"Instrument the pipeline, not just the HTTP client:",{"type":45,"tag":1038,"props":1656,"children":1657},{},[1658,1685,1690,1695],{"type":45,"tag":1042,"props":1659,"children":1660},{},[1661,1663,1669,1670,1676,1678,1683],{"type":54,"value":1662},"publish rate and intake errors (",{"type":45,"tag":86,"props":1664,"children":1666},{"className":1665},[],[1667],{"type":54,"value":1668},"409",{"type":54,"value":565},{"type":45,"tag":86,"props":1671,"children":1673},{"className":1672},[],[1674],{"type":54,"value":1675},"413",{"type":54,"value":1677},", backpressure ",{"type":45,"tag":86,"props":1679,"children":1681},{"className":1680},[],[1682],{"type":54,"value":1352},{"type":54,"value":1684},"),",{"type":45,"tag":1042,"props":1686,"children":1687},{},[1688],{"type":54,"value":1689},"outbox age (oldest unpublished row) and scheduler lag (oldest due delivery),",{"type":45,"tag":1042,"props":1691,"children":1692},{},[1693],{"type":54,"value":1694},"attempts per second, success rate and p95 latency, overall and per endpoint,",{"type":45,"tag":1042,"props":1696,"children":1697},{},[1698],{"type":54,"value":1699},"count of endpoints by state, and the longest-failing endpoints.",{"type":45,"tag":46,"props":1701,"children":1702},{},[1703,1705,1710],{"type":54,"value":1704},"Alert on outbox age and scheduler lag; those mean ",{"type":45,"tag":493,"props":1706,"children":1707},{},[1708],{"type":54,"value":1709},"you",{"type":54,"value":1711}," are failing. Customer endpoints failing is normal and belongs on a dashboard, not a pager.",{"type":45,"tag":73,"props":1713,"children":1715},{"id":1714},"_9-security-checklist",[1716],{"type":54,"value":1717},"9. Security checklist",{"type":45,"tag":1038,"props":1719,"children":1720},{},[1721,1726,1731,1758,1763],{"type":45,"tag":1042,"props":1722,"children":1723},{},[1724],{"type":54,"value":1725},"A separate signing secret per endpoint, encrypted at rest, revealable only to admins, with a 24-hour dual-signing window on rotation.",{"type":45,"tag":1042,"props":1727,"children":1728},{},[1729],{"type":54,"value":1730},"SSRF protection at registration and at connect time; HTTPS only.",{"type":45,"tag":1042,"props":1732,"children":1733},{},[1734,1736,1742,1743,1749,1750,1756],{"type":54,"value":1735},"Reserved headers (",{"type":45,"tag":86,"props":1737,"children":1739},{"className":1738},[],[1740],{"type":54,"value":1741},"webhook-*",{"type":54,"value":565},{"type":45,"tag":86,"props":1744,"children":1746},{"className":1745},[],[1747],{"type":54,"value":1748},"Host",{"type":54,"value":565},{"type":45,"tag":86,"props":1751,"children":1753},{"className":1752},[],[1754],{"type":54,"value":1755},"Content-Length",{"type":54,"value":1757},") that customers can't override with custom headers.",{"type":45,"tag":1042,"props":1759,"children":1760},{},[1761],{"type":54,"value":1762},"Narrow credentials for producers: a token that can only publish, separate from the one that manages endpoints.",{"type":45,"tag":1042,"props":1764,"children":1765},{},[1766],{"type":54,"value":1767},"Payload hygiene: no secrets or full card numbers in events, because they live in the delivery log.",{"type":45,"tag":73,"props":1769,"children":1771},{"id":1770},"build-vs-buy-what-it-actually-costs",[1772],{"type":54,"value":1773},"Build vs buy: what it actually costs",{"type":45,"tag":46,"props":1775,"children":1776},{},[1777,1779,1784,1786,1791],{"type":54,"value":1778},"Everything above is a realistic first version. Built by an experienced team, expect roughly ",{"type":45,"tag":50,"props":1780,"children":1781},{},[1782],{"type":54,"value":1783},"6-10 engineer-weeks",{"type":54,"value":1785}," for intake, fan-out, lanes, scheduler, signing, health, the log, recovery tooling and a minimal admin UI, then ongoing ownership: on-call, Postgres growth, retention jobs, customer support tooling and SDK or docs work. At a fully loaded cost of around $3,000 per engineer-week, that is about ",{"type":45,"tag":50,"props":1787,"children":1788},{},[1789],{"type":54,"value":1790},"$18,000-30,000 up front plus a fraction of an engineer each year",{"type":54,"value":1792},", before infrastructure.",{"type":45,"tag":46,"props":1794,"children":1795},{},[1796],{"type":54,"value":1797},"Hosted options, at list prices checked on 2026-10-10:",{"type":45,"tag":337,"props":1799,"children":1800},{},[1801,1827],{"type":45,"tag":341,"props":1802,"children":1803},{},[1804],{"type":45,"tag":345,"props":1805,"children":1806},{},[1807,1812,1817,1822],{"type":45,"tag":349,"props":1808,"children":1809},{},[1810],{"type":54,"value":1811},"Option",{"type":45,"tag":349,"props":1813,"children":1814},{},[1815],{"type":54,"value":1816},"1M messages/month",{"type":45,"tag":349,"props":1818,"children":1819},{},[1820],{"type":54,"value":1821},"10M messages/month",{"type":45,"tag":349,"props":1823,"children":1824},{},[1825],{"type":54,"value":1826},"Delivery history",{"type":45,"tag":360,"props":1828,"children":1829},{},[1830,1852,1875,1898,1921],{"type":45,"tag":345,"props":1831,"children":1832},{},[1833,1838,1843,1847],{"type":45,"tag":367,"props":1834,"children":1835},{},[1836],{"type":54,"value":1837},"Build it yourself",{"type":45,"tag":367,"props":1839,"children":1840},{},[1841],{"type":54,"value":1842},"infra + engineering time",{"type":45,"tag":367,"props":1844,"children":1845},{},[1846],{"type":54,"value":1842},{"type":45,"tag":367,"props":1848,"children":1849},{},[1850],{"type":54,"value":1851},"whatever you build",{"type":45,"tag":345,"props":1853,"children":1854},{},[1855,1860,1865,1870],{"type":45,"tag":367,"props":1856,"children":1857},{},[1858],{"type":54,"value":1859},"Webhook Relay Outbound Gateway",{"type":45,"tag":367,"props":1861,"children":1862},{},[1863],{"type":54,"value":1864},"$9.99 (Basic, up to 100 customers)",{"type":45,"tag":367,"props":1866,"children":1867},{},[1868],{"type":54,"value":1869},"$9.99 / $79.99 / $249.99 by customer count",{"type":45,"tag":367,"props":1871,"children":1872},{},[1873],{"type":54,"value":1874},"35 days",{"type":45,"tag":345,"props":1876,"children":1877},{},[1878,1883,1888,1893],{"type":45,"tag":367,"props":1879,"children":1880},{},[1881],{"type":54,"value":1882},"Svix Basic ($20 incl. 50k, then $0.0001/msg)",{"type":45,"tag":367,"props":1884,"children":1885},{},[1886],{"type":54,"value":1887},"about $115",{"type":45,"tag":367,"props":1889,"children":1890},{},[1891],{"type":54,"value":1892},"about $1,015",{"type":45,"tag":367,"props":1894,"children":1895},{},[1896],{"type":54,"value":1897},"30 days",{"type":45,"tag":345,"props":1899,"children":1900},{},[1901,1906,1911,1916],{"type":45,"tag":367,"props":1902,"children":1903},{},[1904],{"type":54,"value":1905},"Hookdeck Outpost Starter ($10 per million)",{"type":45,"tag":367,"props":1907,"children":1908},{},[1909],{"type":54,"value":1910},"about $10",{"type":45,"tag":367,"props":1912,"children":1913},{},[1914],{"type":54,"value":1915},"about $100",{"type":45,"tag":367,"props":1917,"children":1918},{},[1919],{"type":54,"value":1920},"7 days",{"type":45,"tag":345,"props":1922,"children":1923},{},[1924,1929,1934,1939],{"type":45,"tag":367,"props":1925,"children":1926},{},[1927],{"type":54,"value":1928},"Hook0",{"type":45,"tag":367,"props":1930,"children":1931},{},[1932],{"type":54,"value":1933},"€59 or €190 plans with daily caps",{"type":45,"tag":367,"props":1935,"children":1936},{},[1937],{"type":54,"value":1938},"depends on caps",{"type":45,"tag":367,"props":1940,"children":1941},{},[1942],{"type":54,"value":1943},"plan-dependent",{"type":45,"tag":46,"props":1945,"children":1946},{},[1947,1949,1955],{"type":54,"value":1948},"Webhook Relay prices on how many customers you serve (Basic 100, Business 500, Pro 1,000), not on messages; retries and fan-out are free and outbound messages don't count against the inbound webhook quota. The full matrix, including portals, compliance and self-hosting, is in ",{"type":45,"tag":64,"props":1950,"children":1952},{"href":1951},"/blog/webhooks-as-a-service-comparison/",[1953],{"type":54,"value":1954},"webhooks as a service compared",{"type":54,"value":71},{"type":45,"tag":46,"props":1957,"children":1958},{},[1959,1964],{"type":45,"tag":50,"props":1960,"children":1961},{},[1962],{"type":54,"value":1963},"Build it yourself when",{"type":54,"value":1965}," webhooks are your product's core, or you need something hosted services don't offer: strict FIFO ordering, non-HTTP destinations like SQS or Kafka, on-premises deployment, an embeddable customer portal from day one, or more than 1,000 customers on one account. Svix, Hookdeck Outpost and Convoy also have self-hostable open-source servers if you want a head start.",{"type":45,"tag":46,"props":1967,"children":1968},{},[1969,1974,1976,1982],{"type":45,"tag":50,"props":1970,"children":1971},{},[1972],{"type":54,"value":1973},"Buy when",{"type":54,"value":1975}," webhooks are a feature your customers expect rather than the thing you sell. Webhook Relay's ",{"type":45,"tag":64,"props":1977,"children":1979},{"href":1978},"/outbound-webhooks/",[1980],{"type":54,"value":1981},"Outbound Gateway",{"type":54,"value":1983}," implements the design in this post as a hosted service: an idempotent publish API with a durable outbox, fan-out, per-endpoint lanes (50 requests per second by default, up to 1,000), Standard Webhooks signing with rotation, the retry schedule above, health states with optional auto-disable, and 35 days of per-attempt history with retry, recover and replay. You call one method from the REST API, the TypeScript or Go SDK, or MCP.",{"type":45,"tag":73,"props":1985,"children":1987},{"id":1986},"next-steps",[1988],{"type":54,"value":1989},"Next steps",{"type":45,"tag":1038,"props":1991,"children":1992},{},[1993,2028,2041],{"type":45,"tag":1042,"props":1994,"children":1995},{},[1996,1998,2004,2006,2012,2013,2019,2021,2027],{"type":54,"value":1997},"Read the ",{"type":45,"tag":64,"props":1999,"children":2001},{"href":2000},"/docs/outbound/",[2002],{"type":54,"value":2003},"Outbound Gateway docs",{"type":54,"value":2005}," or jump into a tutorial: ",{"type":45,"tag":64,"props":2007,"children":2009},{"href":2008},"/docs/tutorials/outbound/send-webhooks-nodejs/",[2010],{"type":54,"value":2011},"Node.js / TypeScript",{"type":54,"value":565},{"type":45,"tag":64,"props":2014,"children":2016},{"href":2015},"/docs/tutorials/outbound/send-webhooks-go/",[2017],{"type":54,"value":2018},"Go with a transactional outbox",{"type":54,"value":2020},", or ",{"type":45,"tag":64,"props":2022,"children":2024},{"href":2023},"/docs/tutorials/outbound/nextjs-saas-webhooks/",[2025],{"type":54,"value":2026},"a Next.js SaaS settings page",{"type":54,"value":71},{"type":45,"tag":1042,"props":2029,"children":2030},{},[2031,2033,2039],{"type":54,"value":2032},"Already running your own sender? The ",{"type":45,"tag":64,"props":2034,"children":2036},{"href":2035},"/docs/outbound/migrate/diy/",[2037],{"type":54,"value":2038},"migration guide for homegrown senders",{"type":54,"value":2040}," maps outbox tables and retry workers onto the Outbound Gateway with a dual-publish cutover.",{"type":45,"tag":1042,"props":2042,"children":2043},{},[2044,2046,2052,2054,2060],{"type":54,"value":2045},"Receiving side: ",{"type":45,"tag":64,"props":2047,"children":2049},{"href":2048},"/blog/webhook-retries-and-idempotency/",[2050],{"type":54,"value":2051},"webhook retries and idempotency",{"type":54,"value":2053}," and ",{"type":45,"tag":64,"props":2055,"children":2057},{"href":2056},"/blog/verify-webhook-signature/",[2058],{"type":54,"value":2059},"how to verify a webhook signature",{"type":54,"value":71},{"type":45,"tag":2062,"props":2063,"children":2064},"style",{},[2065],{"type":54,"value":2066},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}",{"title":7,"searchDepth":111,"depth":111,"links":2068},[2069,2070,2071,2072,2073,2074,2075,2076,2077,2078,2079,2080,2081,2082],{"id":75,"depth":102,"text":78},{"id":327,"depth":102,"text":330},{"id":507,"depth":102,"text":510},{"id":924,"depth":102,"text":927},{"id":1006,"depth":102,"text":1009},{"id":1059,"depth":102,"text":1062},{"id":1140,"depth":102,"text":1143},{"id":1261,"depth":102,"text":1264},{"id":1449,"depth":102,"text":1452},{"id":1590,"depth":102,"text":1593},{"id":1646,"depth":102,"text":1649},{"id":1714,"depth":102,"text":1717},{"id":1770,"depth":102,"text":1773},{"id":1986,"depth":102,"text":1989},"markdown","content:blog:webhook-system-design.md","content","blog/webhook-system-design.md","blog/webhook-system-design","md",{"loc":4},1792953553435]