[{"data":1,"prerenderedAt":3785},["ShallowReactive",2],{"content-query-rTHS2evdyW":3},{"_path":4,"_dir":5,"_draft":6,"_partial":6,"_locale":7,"title":8,"description":9,"layout":10,"head":11,"author":13,"tags":14,"categories":20,"date":22,"faq":23,"image":42,"body":43,"_type":3778,"_id":3779,"_source":3780,"_file":3781,"_stem":3782,"_extension":3783,"sitemap":3784,"readingTime":2080},"/blog/reliable-webhook-delivery","blog",false,"","Reliable Webhook Delivery: How to Never Miss a Webhook","Why webhooks get lost, how long Stripe, Shopify, GitHub and Slack really retry, and the five-stage pipeline (receive, queue, process, retry, dead-letter) plus monitoring and reconciliation that makes delivery reliable.","post",{"title":12},"Reliable Webhook Delivery: Never Miss a Webhook","Karolis Rusenas",[15,16,17,18,19],"webhook reliability","reliable webhook delivery","webhook retries","dead letter queue","reliability",[21],"guides","2026-11-10 12:30:00",[24,27,30,33,36,39],{"q":25,"a":26},"How do I make sure I never miss a webhook?","No single setting guarantees it. Combine four things: answer within the provider's timeout by storing the event and returning 200 before doing any work; retry your own processing failures from durable storage with backoff; alert on failed deliveries and on unusual silence; and run a scheduled reconciliation job that asks the provider's API for objects changed since the last run, so events that never arrived are still processed.",{"q":28,"a":29},"Which queue ensures reliable webhook delivery?","Any durable queue works for the processing side, for example Amazon SQS with a dead-letter queue, Google Cloud Pub/Sub, RabbitMQ with quorum queues, or a database table used as an outbox. The queue only protects events after your endpoint has accepted them. You still need an endpoint that is up and fast enough to accept every delivery, idempotent consumers, a dead-letter path with replay, and reconciliation for events the provider never delivered.",{"q":31,"a":32},"Are webhooks delivered exactly once?","No. Webhooks are delivered at least once. A provider retries when it does not see a 2xx in time, even if your server already processed the request, so the same event can arrive twice. Make handlers idempotent by recording each event id with a unique constraint and skipping ids you have already processed.",{"q":34,"a":35},"How long do webhook providers retry failed deliveries?","It varies a lot. Stripe retries for up to three days in live mode. Shopify retries eight times over four hours. Slack retries three times within about five minutes. The WhatsApp Cloud API retries for up to seven days. GitHub does not retry automatically at all, though you can redeliver from the last three days by hand or through the API.",{"q":37,"a":38},"What is a webhook dead-letter queue?","A dead-letter queue holds events that failed processing more times than your retry policy allows, so they stop blocking the main queue without being thrown away. Each entry keeps the original payload and the last error. After you fix the cause, you replay the dead-lettered events through the normal handler, which is safe because the handler is idempotent.",{"q":40,"a":41},"Should I build webhook retry infrastructure or use a service?","Build it when webhooks are central to your product, you already run queues and on-call, and you need full control of storage. Use a managed webhook gateway when you want durable retries, logs, alerting and replay without operating them, or when the receiver sits on a private network. Either way, keep idempotent handlers and reconciliation in your own code.","/images/blog/heroes/route.jpg",{"type":44,"children":45,"toc":3760},"root",[46,60,129,136,141,287,292,298,303,494,499,505,510,540,567,648,653,725,833,932,981,987,1012,1029,1036,1048,1342,1355,1361,1382,1388,1393,1442,1453,1459,1464,1469,1776,1781,1786,2663,2684,2900,2920,2933,3156,3162,3167,3173,3178,3196,3221,3227,3255,3259,3264,3270,3275,3463,3516,3530,3536,3570,3576,3625,3631,3643,3754],{"type":47,"tag":48,"props":49,"children":50},"element","p",{},[51,58],{"type":47,"tag":52,"props":53,"children":54},"strong",{},[55],{"type":56,"value":57},"text","Reliable webhook delivery means every event a provider sends is eventually processed exactly once by your system, even when your endpoint is slow, deploying or down.",{"type":56,"value":59}," Providers don't guarantee that. They retry for a window that ranges from zero (GitHub) to seven days (WhatsApp Cloud API), with Stripe at three days and Shopify at four hours, and then they stop. Reliability is something the receiving side builds out of five stages: receive fast, queue durably, process idempotently, retry with backoff, and dead-letter what still fails. Monitoring and reconciliation wrap around all five.",{"type":47,"tag":61,"props":62,"children":64},"callout",{"label":63},"The short version",[65],{"type":47,"tag":66,"props":67,"children":68},"ol",{},[69,89,99,109,119],{"type":47,"tag":70,"props":71,"children":72},"li",{},[73,78,80,87],{"type":47,"tag":52,"props":74,"children":75},{},[76],{"type":56,"value":77},"Acknowledge fast.",{"type":56,"value":79}," Verify the signature, store the raw event, return ",{"type":47,"tag":81,"props":82,"children":84},"code",{"className":83},[],[85],{"type":56,"value":86},"200",{"type":56,"value":88},". Do the work later.",{"type":47,"tag":70,"props":90,"children":91},{},[92,97],{"type":47,"tag":52,"props":93,"children":94},{},[95],{"type":56,"value":96},"Process idempotently.",{"type":56,"value":98}," Record each event id with a unique constraint; skip duplicates.",{"type":47,"tag":70,"props":100,"children":101},{},[102,107],{"type":47,"tag":52,"props":103,"children":104},{},[105],{"type":56,"value":106},"Retry your own failures",{"type":56,"value":108}," from durable storage with exponential backoff and a deadline.",{"type":47,"tag":70,"props":110,"children":111},{},[112,117],{"type":47,"tag":52,"props":113,"children":114},{},[115],{"type":56,"value":116},"Dead-letter, then replay.",{"type":56,"value":118}," Park events that keep failing; replay them after the fix.",{"type":47,"tag":70,"props":120,"children":121},{},[122,127],{"type":47,"tag":52,"props":123,"children":124},{},[125],{"type":56,"value":126},"Watch and reconcile.",{"type":56,"value":128}," Alert on failures and on silence, and periodically ask the provider's API for anything you never received.",{"type":47,"tag":130,"props":131,"children":133},"h2",{"id":132},"why-webhooks-get-lost",[134],{"type":56,"value":135},"Why webhooks get lost",{"type":47,"tag":48,"props":137,"children":138},{},[139],{"type":56,"value":140},"Every lost webhook falls into one of six buckets. The first four are your side; the last two are the provider's.",{"type":47,"tag":142,"props":143,"children":144},"table",{},[145,169],{"type":47,"tag":146,"props":147,"children":148},"thead",{},[149],{"type":47,"tag":150,"props":151,"children":152},"tr",{},[153,159,164],{"type":47,"tag":154,"props":155,"children":156},"th",{},[157],{"type":56,"value":158},"Failure",{"type":47,"tag":154,"props":160,"children":161},{},[162],{"type":56,"value":163},"What happens",{"type":47,"tag":154,"props":165,"children":166},{},[167],{"type":56,"value":168},"Defence",{"type":47,"tag":170,"props":171,"children":172},"tbody",{},[173,192,210,233,251,269],{"type":47,"tag":150,"props":174,"children":175},{},[176,182,187],{"type":47,"tag":177,"props":178,"children":179},"td",{},[180],{"type":56,"value":181},"Endpoint down or deploying",{"type":47,"tag":177,"props":183,"children":184},{},[185],{"type":56,"value":186},"Delivery fails; provider retries for its window, then gives up",{"type":47,"tag":177,"props":188,"children":189},{},[190],{"type":56,"value":191},"Buffer in front; long retries",{"type":47,"tag":150,"props":193,"children":194},{},[195,200,205],{"type":47,"tag":177,"props":196,"children":197},{},[198],{"type":56,"value":199},"Handler too slow",{"type":47,"tag":177,"props":201,"children":202},{},[203],{"type":56,"value":204},"Provider times out (Slack 3 s, Shopify 5 s, GitHub 10 s) and counts a failure, even if you finish the work",{"type":47,"tag":177,"props":206,"children":207},{},[208],{"type":56,"value":209},"Acknowledge first, process async",{"type":47,"tag":150,"props":211,"children":212},{},[213,223,228],{"type":47,"tag":177,"props":214,"children":215},{},[216,218],{"type":56,"value":217},"Handler crashes after ",{"type":47,"tag":81,"props":219,"children":221},{"className":220},[],[222],{"type":56,"value":86},{"type":47,"tag":177,"props":224,"children":225},{},[226],{"type":56,"value":227},"Provider thinks it is delivered; your work never happened",{"type":47,"tag":177,"props":229,"children":230},{},[231],{"type":56,"value":232},"Store before acknowledging; retry from storage",{"type":47,"tag":150,"props":234,"children":235},{},[236,241,246],{"type":47,"tag":177,"props":237,"children":238},{},[239],{"type":56,"value":240},"Bug in processing",{"type":47,"tag":177,"props":242,"children":243},{},[244],{"type":56,"value":245},"Every event of a type fails the same way",{"type":47,"tag":177,"props":247,"children":248},{},[249],{"type":56,"value":250},"Dead-letter and replay after the fix",{"type":47,"tag":150,"props":252,"children":253},{},[254,259,264],{"type":47,"tag":177,"props":255,"children":256},{},[257],{"type":56,"value":258},"Provider stops retrying",{"type":47,"tag":177,"props":260,"children":261},{},[262],{"type":56,"value":263},"Outage outlasts the retry window; events are dropped",{"type":47,"tag":177,"props":265,"children":266},{},[267],{"type":56,"value":268},"Your own durable retries; reconciliation",{"type":47,"tag":150,"props":270,"children":271},{},[272,277,282],{"type":47,"tag":177,"props":273,"children":274},{},[275],{"type":56,"value":276},"Provider never sends",{"type":47,"tag":177,"props":278,"children":279},{},[280],{"type":56,"value":281},"Bugs, disabled or deleted subscriptions, filtering mistakes",{"type":47,"tag":177,"props":283,"children":284},{},[285],{"type":56,"value":286},"Silence alerts; reconciliation",{"type":47,"tag":48,"props":288,"children":289},{},[290],{"type":56,"value":291},"The last row is the uncomfortable one: Shopify's documentation states plainly that delivery \"isn't always guaranteed\". A receiver that only listens can never be fully reliable. It also has to ask.",{"type":47,"tag":130,"props":293,"children":295},{"id":294},"how-long-providers-really-retry",[296],{"type":56,"value":297},"How long providers really retry",{"type":47,"tag":48,"props":299,"children":300},{},[301],{"type":56,"value":302},"Before you design anything, know how much slack each sender gives you. All of these are from the providers' own documentation, checked in November 2026.",{"type":47,"tag":142,"props":304,"children":305},{},[306,332],{"type":47,"tag":146,"props":307,"children":308},{},[309],{"type":47,"tag":150,"props":310,"children":311},{},[312,317,322,327],{"type":47,"tag":154,"props":313,"children":314},{},[315],{"type":56,"value":316},"Provider",{"type":47,"tag":154,"props":318,"children":319},{},[320],{"type":56,"value":321},"Response timeout",{"type":47,"tag":154,"props":323,"children":324},{},[325],{"type":56,"value":326},"Automatic retries",{"type":47,"tag":154,"props":328,"children":329},{},[330],{"type":56,"value":331},"When it gives up",{"type":47,"tag":170,"props":333,"children":334},{},[335,368,399,433,464],{"type":47,"tag":150,"props":336,"children":337},{},[338,353,358,363],{"type":47,"tag":177,"props":339,"children":340},{},[341],{"type":47,"tag":342,"props":343,"children":347},"a",{"href":344,"rel":345},"https://docs.stripe.com/webhooks",[346],"nofollow",[348],{"type":47,"tag":52,"props":349,"children":350},{},[351],{"type":56,"value":352},"Stripe",{"type":47,"tag":177,"props":354,"children":355},{},[356],{"type":56,"value":357},"Not published",{"type":47,"tag":177,"props":359,"children":360},{},[361],{"type":56,"value":362},"Up to 3 days, exponential backoff (live mode); 3 times over a few hours in a sandbox",{"type":47,"tag":177,"props":364,"children":365},{},[366],{"type":56,"value":367},"Event is no longer retried; resend by hand for 15 days in the Dashboard, 30 days with the CLI",{"type":47,"tag":150,"props":369,"children":370},{},[371,384,389,394],{"type":47,"tag":177,"props":372,"children":373},{},[374],{"type":47,"tag":342,"props":375,"children":378},{"href":376,"rel":377},"https://shopify.dev/docs/apps/build/webhooks/troubleshoot",[346],[379],{"type":47,"tag":52,"props":380,"children":381},{},[382],{"type":56,"value":383},"Shopify",{"type":47,"tag":177,"props":385,"children":386},{},[387],{"type":56,"value":388},"1 s to connect, 5 s total",{"type":47,"tag":177,"props":390,"children":391},{},[392],{"type":56,"value":393},"8 retries over 4 hours",{"type":47,"tag":177,"props":395,"children":396},{},[397],{"type":56,"value":398},"Stops; repeatedly failing admin or Admin API subscriptions are deleted",{"type":47,"tag":150,"props":400,"children":401},{},[402,415,420,428],{"type":47,"tag":177,"props":403,"children":404},{},[405],{"type":47,"tag":342,"props":406,"children":409},{"href":407,"rel":408},"https://docs.github.com/en/webhooks/using-webhooks/handling-failed-webhook-deliveries",[346],[410],{"type":47,"tag":52,"props":411,"children":412},{},[413],{"type":56,"value":414},"GitHub",{"type":47,"tag":177,"props":416,"children":417},{},[418],{"type":56,"value":419},"10 s",{"type":47,"tag":177,"props":421,"children":422},{},[423],{"type":47,"tag":52,"props":424,"children":425},{},[426],{"type":56,"value":427},"None",{"type":47,"tag":177,"props":429,"children":430},{},[431],{"type":56,"value":432},"Redeliver by hand or API from the last 3 days",{"type":47,"tag":150,"props":434,"children":435},{},[436,449,454,459],{"type":47,"tag":177,"props":437,"children":438},{},[439],{"type":47,"tag":342,"props":440,"children":443},{"href":441,"rel":442},"https://docs.slack.dev/apis/events-api/",[346],[444],{"type":47,"tag":52,"props":445,"children":446},{},[447],{"type":56,"value":448},"Slack Events API",{"type":47,"tag":177,"props":450,"children":451},{},[452],{"type":56,"value":453},"3 s",{"type":47,"tag":177,"props":455,"children":456},{},[457],{"type":56,"value":458},"3 retries: almost immediately, after 1 minute, after 5 minutes",{"type":47,"tag":177,"props":460,"children":461},{},[462],{"type":56,"value":463},"Subscriptions are temporarily disabled above 95% failures in 60 minutes",{"type":47,"tag":150,"props":465,"children":466},{},[467,480,484,489],{"type":47,"tag":177,"props":468,"children":469},{},[470],{"type":47,"tag":342,"props":471,"children":474},{"href":472,"rel":473},"https://developers.facebook.com/documentation/business-messaging/whatsapp/webhooks/overview",[346],[475],{"type":47,"tag":52,"props":476,"children":477},{},[478],{"type":56,"value":479},"WhatsApp Cloud API",{"type":47,"tag":177,"props":481,"children":482},{},[483],{"type":56,"value":357},{"type":47,"tag":177,"props":485,"children":486},{},[487],{"type":56,"value":488},"With decreasing frequency for up to 7 days",{"type":47,"tag":177,"props":490,"children":491},{},[492],{"type":56,"value":493},"Dropped",{"type":47,"tag":48,"props":495,"children":496},{},[497],{"type":56,"value":498},"Two things stand out. The windows are short: half of these give you minutes or hours, not days. And they are all different, so an outage that one provider survives is data loss for another. The fix is to stop depending on each sender's retry policy and own the retry window yourself, which the pipeline below does.",{"type":47,"tag":130,"props":500,"children":502},{"id":501},"at-least-once-is-the-best-anyone-offers",[503],{"type":56,"value":504},"At-least-once is the best anyone offers",{"type":47,"tag":48,"props":506,"children":507},{},[508],{"type":56,"value":509},"Without an acknowledgement, a sender can't tell these four cases apart:",{"type":47,"tag":66,"props":511,"children":512},{},[513,518,530,535],{"type":47,"tag":70,"props":514,"children":515},{},[516],{"type":56,"value":517},"The request was lost on the way.",{"type":47,"tag":70,"props":519,"children":520},{},[521,523,528],{"type":56,"value":522},"It arrived and was processed, but the ",{"type":47,"tag":81,"props":524,"children":526},{"className":525},[],[527],{"type":56,"value":86},{"type":56,"value":529}," was lost on the way back.",{"type":47,"tag":70,"props":531,"children":532},{},[533],{"type":56,"value":534},"The receiver crashed before processing it.",{"type":47,"tag":70,"props":536,"children":537},{},[538],{"type":56,"value":539},"The network is just slow.",{"type":47,"tag":48,"props":541,"children":542},{},[543,545,550,552,558,560,565],{"type":56,"value":544},"Retrying fixes 1 and 3, and turns 2 and 4 into duplicates. That is why every provider above delivers ",{"type":47,"tag":52,"props":546,"children":547},{},[548],{"type":56,"value":549},"at least once",{"type":56,"value":551},": if it doesn't see a ",{"type":47,"tag":81,"props":553,"children":555},{"className":554},[],[556],{"type":56,"value":557},"2xx",{"type":56,"value":559}," in time, it sends again, even when your server did the work and only the response was lost. Exactly-once delivery over a network isn't possible; what you can build is exactly-once ",{"type":47,"tag":52,"props":561,"children":562},{},[563],{"type":56,"value":564},"processing",{"type":56,"value":566},", by making the receiver ignore what it has already seen.",{"type":47,"tag":142,"props":568,"children":569},{},[570,591],{"type":47,"tag":146,"props":571,"children":572},{},[573],{"type":47,"tag":150,"props":574,"children":575},{},[576,581,586],{"type":47,"tag":154,"props":577,"children":578},{},[579],{"type":56,"value":580},"Guarantee",{"type":47,"tag":154,"props":582,"children":583},{},[584],{"type":56,"value":585},"What it means",{"type":47,"tag":154,"props":587,"children":588},{},[589],{"type":56,"value":590},"Who offers it",{"type":47,"tag":170,"props":592,"children":593},{},[594,612,630],{"type":47,"tag":150,"props":595,"children":596},{},[597,602,607],{"type":47,"tag":177,"props":598,"children":599},{},[600],{"type":56,"value":601},"At most once",{"type":47,"tag":177,"props":603,"children":604},{},[605],{"type":56,"value":606},"Sent once, never retried; may be lost",{"type":47,"tag":177,"props":608,"children":609},{},[610],{"type":56,"value":611},"Fire-and-forget senders, GitHub without manual redelivery",{"type":47,"tag":150,"props":613,"children":614},{},[615,620,625],{"type":47,"tag":177,"props":616,"children":617},{},[618],{"type":56,"value":619},"At least once",{"type":47,"tag":177,"props":621,"children":622},{},[623],{"type":56,"value":624},"Retried until acknowledged; may be duplicated",{"type":47,"tag":177,"props":626,"children":627},{},[628],{"type":56,"value":629},"Stripe, Shopify, Slack, Meta and every webhook gateway",{"type":47,"tag":150,"props":631,"children":632},{},[633,638,643],{"type":47,"tag":177,"props":634,"children":635},{},[636],{"type":56,"value":637},"Exactly once",{"type":47,"tag":177,"props":639,"children":640},{},[641],{"type":56,"value":642},"Never lost, never duplicated",{"type":47,"tag":177,"props":644,"children":645},{},[646],{"type":56,"value":647},"Nobody, over HTTP. You build it on top of at-least-once with idempotent processing",{"type":47,"tag":48,"props":649,"children":650},{},[651],{"type":56,"value":652},"Two decisions make deduplication work:",{"type":47,"tag":654,"props":655,"children":656},"ul",{},[657,715],{"type":47,"tag":70,"props":658,"children":659},{},[660,665,667,673,675,681,683,689,691,697,699,705,707,713],{"type":47,"tag":52,"props":661,"children":662},{},[663],{"type":56,"value":664},"Which id to key on.",{"type":56,"value":666}," Use the id that stays the same across retries of the same event. That is the event id for Stripe (",{"type":47,"tag":81,"props":668,"children":670},{"className":669},[],[671],{"type":56,"value":672},"evt_…",{"type":56,"value":674},") and Standard Webhooks (",{"type":47,"tag":81,"props":676,"children":678},{"className":677},[],[679],{"type":56,"value":680},"webhook-id",{"type":56,"value":682},"), ",{"type":47,"tag":81,"props":684,"children":686},{"className":685},[],[687],{"type":56,"value":688},"X-GitHub-Delivery",{"type":56,"value":690}," for GitHub, and ",{"type":47,"tag":81,"props":692,"children":694},{"className":693},[],[695],{"type":56,"value":696},"X-Shopify-Webhook-Id",{"type":56,"value":698}," for Shopify. Shopify's ",{"type":47,"tag":81,"props":700,"children":702},{"className":701},[],[703],{"type":56,"value":704},"X-Shopify-Event-Id",{"type":56,"value":706}," is shared by deliveries to ",{"type":47,"tag":708,"props":709,"children":710},"em",{},[711],{"type":56,"value":712},"different",{"type":56,"value":714}," subscriptions, so keying on it would drop legitimate deliveries if you have two.",{"type":47,"tag":70,"props":716,"children":717},{},[718,723],{"type":47,"tag":52,"props":719,"children":720},{},[721],{"type":56,"value":722},"How long to remember it.",{"type":56,"value":724}," At least the longest retry window you receive from, plus your own retry window. Stripe retries for 3 days, so keep Stripe ids for a week; a day covers Shopify's 4 hours. Then delete them on a schedule so the dedupe table doesn't grow forever, or keep ids in Redis with a TTL at high volume:",{"type":47,"tag":726,"props":727,"children":729},"code-block",{"label":728},"SQL · nightly cleanup (keep ids longer than every retry window)",[730],{"type":47,"tag":731,"props":732,"children":736},"pre",{"className":733,"code":734,"language":735,"meta":7,"style":7},"language-sql shiki shiki-themes github-dark","delete from webhook_events\n where status = 'done' and received_at \u003C now() - interval '7 days';\n","sql",[737],{"type":47,"tag":81,"props":738,"children":739},{"__ignoreMap":7},[740,763],{"type":47,"tag":741,"props":742,"children":745},"span",{"class":743,"line":744},"line",1,[746,752,757],{"type":47,"tag":741,"props":747,"children":749},{"style":748},"--shiki-default:#F97583",[750],{"type":56,"value":751},"delete",{"type":47,"tag":741,"props":753,"children":754},{"style":748},[755],{"type":56,"value":756}," from",{"type":47,"tag":741,"props":758,"children":760},{"style":759},"--shiki-default:#E1E4E8",[761],{"type":56,"value":762}," webhook_events\n",{"type":47,"tag":741,"props":764,"children":766},{"class":743,"line":765},2,[767,772,777,782,788,793,798,803,808,813,818,823,828],{"type":47,"tag":741,"props":768,"children":769},{"style":748},[770],{"type":56,"value":771}," where",{"type":47,"tag":741,"props":773,"children":774},{"style":748},[775],{"type":56,"value":776}," status",{"type":47,"tag":741,"props":778,"children":779},{"style":748},[780],{"type":56,"value":781}," =",{"type":47,"tag":741,"props":783,"children":785},{"style":784},"--shiki-default:#9ECBFF",[786],{"type":56,"value":787}," 'done'",{"type":47,"tag":741,"props":789,"children":790},{"style":748},[791],{"type":56,"value":792}," and",{"type":47,"tag":741,"props":794,"children":795},{"style":759},[796],{"type":56,"value":797}," received_at ",{"type":47,"tag":741,"props":799,"children":800},{"style":748},[801],{"type":56,"value":802},"\u003C",{"type":47,"tag":741,"props":804,"children":805},{"style":748},[806],{"type":56,"value":807}," now",{"type":47,"tag":741,"props":809,"children":810},{"style":759},[811],{"type":56,"value":812},"() ",{"type":47,"tag":741,"props":814,"children":815},{"style":748},[816],{"type":56,"value":817},"-",{"type":47,"tag":741,"props":819,"children":820},{"style":759},[821],{"type":56,"value":822}," interval ",{"type":47,"tag":741,"props":824,"children":825},{"style":784},[826],{"type":56,"value":827},"'7 days'",{"type":47,"tag":741,"props":829,"children":830},{"style":759},[831],{"type":56,"value":832},";\n",{"type":47,"tag":726,"props":834,"children":836},{"label":835},"Redis · seen-id check with a 7-day TTL",[837],{"type":47,"tag":731,"props":838,"children":842},{"className":839,"code":840,"language":841,"meta":7,"style":7},"language-bash shiki shiki-themes github-dark","# Returns OK the first time, nil if the id was already seen\nSET webhook:\u003Cprovider>:\u003Cevent_id> 1 NX EX 604800\n","bash",[843],{"type":47,"tag":81,"props":844,"children":845},{"__ignoreMap":7},[846,855],{"type":47,"tag":741,"props":847,"children":848},{"class":743,"line":744},[849],{"type":47,"tag":741,"props":850,"children":852},{"style":851},"--shiki-default:#6A737D",[853],{"type":56,"value":854},"# Returns OK the first time, nil if the id was already seen\n",{"type":47,"tag":741,"props":856,"children":857},{"class":743,"line":765},[858,864,869,873,878,883,888,893,897,902,907,911,917,922,927],{"type":47,"tag":741,"props":859,"children":861},{"style":860},"--shiki-default:#B392F0",[862],{"type":56,"value":863},"SET",{"type":47,"tag":741,"props":865,"children":866},{"style":784},[867],{"type":56,"value":868}," webhook:",{"type":47,"tag":741,"props":870,"children":871},{"style":748},[872],{"type":56,"value":802},{"type":47,"tag":741,"props":874,"children":875},{"style":784},[876],{"type":56,"value":877},"provide",{"type":47,"tag":741,"props":879,"children":880},{"style":759},[881],{"type":56,"value":882},"r",{"type":47,"tag":741,"props":884,"children":885},{"style":748},[886],{"type":56,"value":887},">",{"type":47,"tag":741,"props":889,"children":890},{"style":784},[891],{"type":56,"value":892},":",{"type":47,"tag":741,"props":894,"children":895},{"style":748},[896],{"type":56,"value":802},{"type":47,"tag":741,"props":898,"children":899},{"style":784},[900],{"type":56,"value":901},"event_i",{"type":47,"tag":741,"props":903,"children":904},{"style":759},[905],{"type":56,"value":906},"d",{"type":47,"tag":741,"props":908,"children":909},{"style":748},[910],{"type":56,"value":887},{"type":47,"tag":741,"props":912,"children":914},{"style":913},"--shiki-default:#79B8FF",[915],{"type":56,"value":916}," 1",{"type":47,"tag":741,"props":918,"children":919},{"style":784},[920],{"type":56,"value":921}," NX",{"type":47,"tag":741,"props":923,"children":924},{"style":784},[925],{"type":56,"value":926}," EX",{"type":47,"tag":741,"props":928,"children":929},{"style":913},[930],{"type":56,"value":931}," 604800\n",{"type":47,"tag":654,"props":933,"children":934},{},[935,961,971],{"type":47,"tag":70,"props":936,"children":937},{},[938,943,945,951,953,959],{"type":47,"tag":52,"props":939,"children":940},{},[941],{"type":56,"value":942},"Also dedupe on the business key for one-time side effects.",{"type":56,"value":944}," One business action can arrive as several different events with different ids: a Stripe payment can produce both ",{"type":47,"tag":81,"props":946,"children":948},{"className":947},[],[949],{"type":56,"value":950},"payment_intent.succeeded",{"type":56,"value":952}," and ",{"type":47,"tag":81,"props":954,"children":956},{"className":955},[],[957],{"type":56,"value":958},"charge.succeeded",{"type":56,"value":960},". If both trigger \"send the receipt\", key that side effect on the payment intent or order id, not the event id.",{"type":47,"tag":70,"props":962,"children":963},{},[964,969],{"type":47,"tag":52,"props":965,"children":966},{},[967],{"type":56,"value":968},"Prefer natural idempotency where you can.",{"type":56,"value":970}," If the event says \"order 42 changed\", fetch order 42's current state from the provider's API and write that, instead of applying the event as a delta. Duplicates and out-of-order deliveries then converge on the right answer without any bookkeeping.",{"type":47,"tag":70,"props":972,"children":973},{},[974,979],{"type":47,"tag":52,"props":975,"children":976},{},[977],{"type":56,"value":978},"Know when you can skip it.",{"type":56,"value":980}," Cache invalidation, analytics counters you can reconcile, and handlers that only fetch current state tolerate duplicates as they are.",{"type":47,"tag":130,"props":982,"children":984},{"id":983},"the-five-stage-delivery-pipeline",[985],{"type":56,"value":986},"The five-stage delivery pipeline",{"type":47,"tag":988,"props":989,"children":993},"picture",{"className":990},[991,992],"block","dark:hidden",[994,996,1002,1003],{"type":56,"value":995},"\n  ",{"type":47,"tag":997,"props":998,"children":1001},"source",{"media":999,"srcSet":1000},"(max-width: 640px)","/images/blog/delivery/reliable-webhook-delivery-mobile.svg",[],{"type":56,"value":995},{"type":47,"tag":1004,"props":1005,"children":1011},"img",{"src":1006,"width":1007,"height":1008,"loading":1009,"alt":1010},"/images/blog/delivery/reliable-webhook-delivery.svg",1000,534,"lazy","Reliable webhook delivery pipeline: 1 receive (verify the signature, store the event, then return 200), 2 queue in durable storage that survives restarts, 3 process idempotently on the event id, 4 retry with backoff and jitter until a deadline, 5 dead-letter to inspect, fix and replay. Around it, monitoring tracks success rate, latency and retry backlog and alerts on failures and silence, and a reconciliation job asks the provider API for anything that never arrived.",[],{"type":47,"tag":988,"props":1013,"children":1017},{"className":1014},[1015,1016],"hidden","dark:block",[1018,1019,1023,1024],{"type":56,"value":995},{"type":47,"tag":997,"props":1020,"children":1022},{"media":999,"srcSet":1021},"/images/blog/delivery/reliable-webhook-delivery-mobile-dark.svg",[],{"type":56,"value":995},{"type":47,"tag":1004,"props":1025,"children":1028},{"src":1026,"width":1007,"height":1008,"loading":1009,"alt":1027},"/images/blog/delivery/reliable-webhook-delivery-dark.svg","Reliable webhook delivery pipeline: receive, queue, process idempotently, retry with backoff, dead-letter and replay, wrapped by monitoring and reconciliation.",[],{"type":47,"tag":1030,"props":1031,"children":1033},"h3",{"id":1032},"_1-receive-verify-store-then-acknowledge",[1034],{"type":56,"value":1035},"1. Receive: verify, store, then acknowledge",{"type":47,"tag":48,"props":1037,"children":1038},{},[1039,1041,1046],{"type":56,"value":1040},"The receiving endpoint has one job: get the event somewhere durable and answer before the provider's timeout. Verify the signature against the raw body, write the event with its id, and return ",{"type":47,"tag":81,"props":1042,"children":1044},{"className":1043},[],[1045],{"type":56,"value":86},{"type":56,"value":1047},". Nothing else happens in the request: no API calls, no emails, no business logic.",{"type":47,"tag":726,"props":1049,"children":1051},{"label":1050},"receiver.js · acknowledge after storing",[1052],{"type":47,"tag":731,"props":1053,"children":1057},{"className":1054,"code":1055,"language":1056,"meta":7,"style":7},"language-js shiki shiki-themes github-dark","app.post('/webhooks/:provider', express.raw({ type: '*/*' }), async (req, res) => {\n  if (!verifySignature(req.params.provider, req)) return res.sendStatus(401)\n\n  // Unique (provider, event_id): a duplicate delivery is a no-op insert.\n  await db.query(\n    `insert into webhook_events (provider, event_id, headers, body, status)\n     values ($1, $2, $3, $4, 'pending')\n     on conflict (provider, event_id) do nothing`,\n    [req.params.provider, eventId(req), req.headers, req.body]\n  )\n  res.sendStatus(200)\n})\n","js",[1058],{"type":47,"tag":81,"props":1059,"children":1060},{"__ignoreMap":7},[1061,1149,1205,1215,1224,1248,1257,1266,1280,1299,1308,1333],{"type":47,"tag":741,"props":1062,"children":1063},{"class":743,"line":744},[1064,1069,1073,1078,1083,1088,1093,1098,1103,1108,1113,1118,1124,1129,1134,1139,1144],{"type":47,"tag":741,"props":1065,"children":1066},{"style":759},[1067],{"type":56,"value":1068},"app.",{"type":47,"tag":741,"props":1070,"children":1071},{"style":860},[1072],{"type":56,"value":10},{"type":47,"tag":741,"props":1074,"children":1075},{"style":759},[1076],{"type":56,"value":1077},"(",{"type":47,"tag":741,"props":1079,"children":1080},{"style":784},[1081],{"type":56,"value":1082},"'/webhooks/:provider'",{"type":47,"tag":741,"props":1084,"children":1085},{"style":759},[1086],{"type":56,"value":1087},", express.",{"type":47,"tag":741,"props":1089,"children":1090},{"style":860},[1091],{"type":56,"value":1092},"raw",{"type":47,"tag":741,"props":1094,"children":1095},{"style":759},[1096],{"type":56,"value":1097},"({ type: ",{"type":47,"tag":741,"props":1099,"children":1100},{"style":784},[1101],{"type":56,"value":1102},"'*/*'",{"type":47,"tag":741,"props":1104,"children":1105},{"style":759},[1106],{"type":56,"value":1107}," }), ",{"type":47,"tag":741,"props":1109,"children":1110},{"style":748},[1111],{"type":56,"value":1112},"async",{"type":47,"tag":741,"props":1114,"children":1115},{"style":759},[1116],{"type":56,"value":1117}," (",{"type":47,"tag":741,"props":1119,"children":1121},{"style":1120},"--shiki-default:#FFAB70",[1122],{"type":56,"value":1123},"req",{"type":47,"tag":741,"props":1125,"children":1126},{"style":759},[1127],{"type":56,"value":1128},", ",{"type":47,"tag":741,"props":1130,"children":1131},{"style":1120},[1132],{"type":56,"value":1133},"res",{"type":47,"tag":741,"props":1135,"children":1136},{"style":759},[1137],{"type":56,"value":1138},") ",{"type":47,"tag":741,"props":1140,"children":1141},{"style":748},[1142],{"type":56,"value":1143},"=>",{"type":47,"tag":741,"props":1145,"children":1146},{"style":759},[1147],{"type":56,"value":1148}," {\n",{"type":47,"tag":741,"props":1150,"children":1151},{"class":743,"line":765},[1152,1157,1161,1166,1171,1176,1181,1186,1191,1195,1200],{"type":47,"tag":741,"props":1153,"children":1154},{"style":748},[1155],{"type":56,"value":1156},"  if",{"type":47,"tag":741,"props":1158,"children":1159},{"style":759},[1160],{"type":56,"value":1117},{"type":47,"tag":741,"props":1162,"children":1163},{"style":748},[1164],{"type":56,"value":1165},"!",{"type":47,"tag":741,"props":1167,"children":1168},{"style":860},[1169],{"type":56,"value":1170},"verifySignature",{"type":47,"tag":741,"props":1172,"children":1173},{"style":759},[1174],{"type":56,"value":1175},"(req.params.provider, req)) ",{"type":47,"tag":741,"props":1177,"children":1178},{"style":748},[1179],{"type":56,"value":1180},"return",{"type":47,"tag":741,"props":1182,"children":1183},{"style":759},[1184],{"type":56,"value":1185}," res.",{"type":47,"tag":741,"props":1187,"children":1188},{"style":860},[1189],{"type":56,"value":1190},"sendStatus",{"type":47,"tag":741,"props":1192,"children":1193},{"style":759},[1194],{"type":56,"value":1077},{"type":47,"tag":741,"props":1196,"children":1197},{"style":913},[1198],{"type":56,"value":1199},"401",{"type":47,"tag":741,"props":1201,"children":1202},{"style":759},[1203],{"type":56,"value":1204},")\n",{"type":47,"tag":741,"props":1206,"children":1208},{"class":743,"line":1207},3,[1209],{"type":47,"tag":741,"props":1210,"children":1212},{"emptyLinePlaceholder":1211},true,[1213],{"type":56,"value":1214},"\n",{"type":47,"tag":741,"props":1216,"children":1218},{"class":743,"line":1217},4,[1219],{"type":47,"tag":741,"props":1220,"children":1221},{"style":851},[1222],{"type":56,"value":1223},"  // Unique (provider, event_id): a duplicate delivery is a no-op insert.\n",{"type":47,"tag":741,"props":1225,"children":1227},{"class":743,"line":1226},5,[1228,1233,1238,1243],{"type":47,"tag":741,"props":1229,"children":1230},{"style":748},[1231],{"type":56,"value":1232},"  await",{"type":47,"tag":741,"props":1234,"children":1235},{"style":759},[1236],{"type":56,"value":1237}," db.",{"type":47,"tag":741,"props":1239,"children":1240},{"style":860},[1241],{"type":56,"value":1242},"query",{"type":47,"tag":741,"props":1244,"children":1245},{"style":759},[1246],{"type":56,"value":1247},"(\n",{"type":47,"tag":741,"props":1249,"children":1251},{"class":743,"line":1250},6,[1252],{"type":47,"tag":741,"props":1253,"children":1254},{"style":784},[1255],{"type":56,"value":1256},"    `insert into webhook_events (provider, event_id, headers, body, status)\n",{"type":47,"tag":741,"props":1258,"children":1260},{"class":743,"line":1259},7,[1261],{"type":47,"tag":741,"props":1262,"children":1263},{"style":784},[1264],{"type":56,"value":1265},"     values ($1, $2, $3, $4, 'pending')\n",{"type":47,"tag":741,"props":1267,"children":1269},{"class":743,"line":1268},8,[1270,1275],{"type":47,"tag":741,"props":1271,"children":1272},{"style":784},[1273],{"type":56,"value":1274},"     on conflict (provider, event_id) do nothing`",{"type":47,"tag":741,"props":1276,"children":1277},{"style":759},[1278],{"type":56,"value":1279},",\n",{"type":47,"tag":741,"props":1281,"children":1283},{"class":743,"line":1282},9,[1284,1289,1294],{"type":47,"tag":741,"props":1285,"children":1286},{"style":759},[1287],{"type":56,"value":1288},"    [req.params.provider, ",{"type":47,"tag":741,"props":1290,"children":1291},{"style":860},[1292],{"type":56,"value":1293},"eventId",{"type":47,"tag":741,"props":1295,"children":1296},{"style":759},[1297],{"type":56,"value":1298},"(req), req.headers, req.body]\n",{"type":47,"tag":741,"props":1300,"children":1302},{"class":743,"line":1301},10,[1303],{"type":47,"tag":741,"props":1304,"children":1305},{"style":759},[1306],{"type":56,"value":1307},"  )\n",{"type":47,"tag":741,"props":1309,"children":1311},{"class":743,"line":1310},11,[1312,1317,1321,1325,1329],{"type":47,"tag":741,"props":1313,"children":1314},{"style":759},[1315],{"type":56,"value":1316},"  res.",{"type":47,"tag":741,"props":1318,"children":1319},{"style":860},[1320],{"type":56,"value":1190},{"type":47,"tag":741,"props":1322,"children":1323},{"style":759},[1324],{"type":56,"value":1077},{"type":47,"tag":741,"props":1326,"children":1327},{"style":913},[1328],{"type":56,"value":86},{"type":47,"tag":741,"props":1330,"children":1331},{"style":759},[1332],{"type":56,"value":1204},{"type":47,"tag":741,"props":1334,"children":1336},{"class":743,"line":1335},12,[1337],{"type":47,"tag":741,"props":1338,"children":1339},{"style":759},[1340],{"type":56,"value":1341},"})\n",{"type":47,"tag":48,"props":1343,"children":1344},{},[1345,1347,1353],{"type":56,"value":1346},"If storing fails, return ",{"type":47,"tag":81,"props":1348,"children":1350},{"className":1349},[],[1351],{"type":56,"value":1352},"500",{"type":56,"value":1354}," so the provider retries. That is the one failure you want the provider to see.",{"type":47,"tag":1030,"props":1356,"children":1358},{"id":1357},"_2-queue-durable-storage-between-receive-and-process",[1359],{"type":56,"value":1360},"2. Queue: durable storage between receive and process",{"type":47,"tag":48,"props":1362,"children":1363},{},[1364,1366,1372,1374,1380],{"type":56,"value":1365},"The stored event is the queue. A database table with a ",{"type":47,"tag":81,"props":1367,"children":1369},{"className":1368},[],[1370],{"type":56,"value":1371},"status",{"type":56,"value":1373}," column is enough for most teams, because the insert and the acknowledgement already happen in the same place. At higher volume, publish to SQS, Pub/Sub or Kafka after the insert. What matters is that a crash, deploy or restart of the worker loses nothing: unprocessed events are still ",{"type":47,"tag":81,"props":1375,"children":1377},{"className":1376},[],[1378],{"type":56,"value":1379},"pending",{"type":56,"value":1381}," when it comes back.",{"type":47,"tag":1030,"props":1383,"children":1385},{"id":1384},"_3-process-idempotent-on-the-event-id",[1386],{"type":56,"value":1387},"3. Process: idempotent on the event id",{"type":47,"tag":48,"props":1389,"children":1390},{},[1391],{"type":56,"value":1392},"Workers pick up pending events and do the real work. Because every stage is at-least-once, the same event will sometimes be processed twice. A handler is idempotent when the second run changes nothing:",{"type":47,"tag":654,"props":1394,"children":1395},{},[1396,1424,1429],{"type":47,"tag":70,"props":1397,"children":1398},{},[1399,1401,1407,1409,1415,1417,1422],{"type":56,"value":1400},"Key on the id that stays the same across retries (see ",{"type":47,"tag":342,"props":1402,"children":1404},{"href":1403},"#at-least-once-is-the-best-anyone-offers",[1405],{"type":56,"value":1406},"which id to key on",{"type":56,"value":1408},"), never on a hash of the payload. ",{"type":47,"tag":342,"props":1410,"children":1412},{"href":1411},"/glossary/standard-webhooks/",[1413],{"type":56,"value":1414},"Standard Webhooks",{"type":56,"value":1416}," senders put it in ",{"type":47,"tag":81,"props":1418,"children":1420},{"className":1419},[],[1421],{"type":56,"value":680},{"type":56,"value":1423},".",{"type":47,"tag":70,"props":1425,"children":1426},{},[1427],{"type":56,"value":1428},"Put the side effect and the \"processed\" mark in one transaction, or make the side effect itself idempotent with an idempotency key on the outbound call.",{"type":47,"tag":70,"props":1430,"children":1431},{},[1432,1434,1440],{"type":56,"value":1433},"Guard against out-of-order delivery: compare the object's ",{"type":47,"tag":81,"props":1435,"children":1437},{"className":1436},[],[1438],{"type":56,"value":1439},"updated_at",{"type":56,"value":1441}," with what you have and ignore older versions.",{"type":47,"tag":48,"props":1443,"children":1444},{},[1445,1451],{"type":47,"tag":342,"props":1446,"children":1448},{"href":1447},"/blog/webhook-retries-and-idempotency/",[1449],{"type":56,"value":1450},"Webhook retries and idempotency",{"type":56,"value":1452}," goes deeper on duplicate handling.",{"type":47,"tag":1030,"props":1454,"children":1456},{"id":1455},"_4-retry-backoff-jitter-and-a-deadline",[1457],{"type":56,"value":1458},"4. Retry: backoff, jitter and a deadline",{"type":47,"tag":48,"props":1460,"children":1461},{},[1462],{"type":56,"value":1463},"When processing fails, retry from storage, not from the provider. Use exponential backoff so a struggling dependency gets room to recover, add jitter so a thousand failed events don't retry in the same second, and set a deadline after which the event moves to the dead-letter stage.",{"type":47,"tag":48,"props":1465,"children":1466},{},[1467],{"type":56,"value":1468},"An example schedule with a 30-second base, doubling each time, ±20% jitter and a deadline after the 12th retry:",{"type":47,"tag":142,"props":1470,"children":1471},{},[1472,1498],{"type":47,"tag":146,"props":1473,"children":1474},{},[1475],{"type":47,"tag":150,"props":1476,"children":1477},{},[1478,1483,1488,1493],{"type":47,"tag":154,"props":1479,"children":1480},{},[1481],{"type":56,"value":1482},"Retry",{"type":47,"tag":154,"props":1484,"children":1485},{},[1486],{"type":56,"value":1487},"Wait before it",{"type":47,"tag":154,"props":1489,"children":1490},{},[1491],{"type":56,"value":1492},"Elapsed since first failure",{"type":47,"tag":154,"props":1494,"children":1495},{},[1496],{"type":56,"value":1497},"What it covers",{"type":47,"tag":170,"props":1499,"children":1500},{},[1501,1523,1546,1569,1592,1615,1638,1661,1684,1706,1729,1751],{"type":47,"tag":150,"props":1502,"children":1503},{},[1504,1509,1514,1518],{"type":47,"tag":177,"props":1505,"children":1506},{},[1507],{"type":56,"value":1508},"1",{"type":47,"tag":177,"props":1510,"children":1511},{},[1512],{"type":56,"value":1513},"30 s",{"type":47,"tag":177,"props":1515,"children":1516},{},[1517],{"type":56,"value":1513},{"type":47,"tag":177,"props":1519,"children":1520},{},[1521],{"type":56,"value":1522},"Network blip, connection reset",{"type":47,"tag":150,"props":1524,"children":1525},{},[1526,1531,1536,1541],{"type":47,"tag":177,"props":1527,"children":1528},{},[1529],{"type":56,"value":1530},"2",{"type":47,"tag":177,"props":1532,"children":1533},{},[1534],{"type":56,"value":1535},"1 min",{"type":47,"tag":177,"props":1537,"children":1538},{},[1539],{"type":56,"value":1540},"1.5 min",{"type":47,"tag":177,"props":1542,"children":1543},{},[1544],{"type":56,"value":1545},"Pod restart",{"type":47,"tag":150,"props":1547,"children":1548},{},[1549,1554,1559,1564],{"type":47,"tag":177,"props":1550,"children":1551},{},[1552],{"type":56,"value":1553},"3",{"type":47,"tag":177,"props":1555,"children":1556},{},[1557],{"type":56,"value":1558},"2 min",{"type":47,"tag":177,"props":1560,"children":1561},{},[1562],{"type":56,"value":1563},"3.5 min",{"type":47,"tag":177,"props":1565,"children":1566},{},[1567],{"type":56,"value":1568},"Rolling deploy",{"type":47,"tag":150,"props":1570,"children":1571},{},[1572,1577,1582,1587],{"type":47,"tag":177,"props":1573,"children":1574},{},[1575],{"type":56,"value":1576},"4",{"type":47,"tag":177,"props":1578,"children":1579},{},[1580],{"type":56,"value":1581},"4 min",{"type":47,"tag":177,"props":1583,"children":1584},{},[1585],{"type":56,"value":1586},"7.5 min",{"type":47,"tag":177,"props":1588,"children":1589},{},[1590],{"type":56,"value":1591},"Slow deploy, brief dependency outage",{"type":47,"tag":150,"props":1593,"children":1594},{},[1595,1600,1605,1610],{"type":47,"tag":177,"props":1596,"children":1597},{},[1598],{"type":56,"value":1599},"5",{"type":47,"tag":177,"props":1601,"children":1602},{},[1603],{"type":56,"value":1604},"8 min",{"type":47,"tag":177,"props":1606,"children":1607},{},[1608],{"type":56,"value":1609},"15.5 min",{"type":47,"tag":177,"props":1611,"children":1612},{},[1613],{"type":56,"value":1614},"Dependency outage that outlasts a deploy",{"type":47,"tag":150,"props":1616,"children":1617},{},[1618,1623,1628,1633],{"type":47,"tag":177,"props":1619,"children":1620},{},[1621],{"type":56,"value":1622},"6",{"type":47,"tag":177,"props":1624,"children":1625},{},[1626],{"type":56,"value":1627},"16 min",{"type":47,"tag":177,"props":1629,"children":1630},{},[1631],{"type":56,"value":1632},"31.5 min",{"type":47,"tag":177,"props":1634,"children":1635},{},[1636],{"type":56,"value":1637},"Incident someone is already fixing",{"type":47,"tag":150,"props":1639,"children":1640},{},[1641,1646,1651,1656],{"type":47,"tag":177,"props":1642,"children":1643},{},[1644],{"type":56,"value":1645},"7",{"type":47,"tag":177,"props":1647,"children":1648},{},[1649],{"type":56,"value":1650},"32 min",{"type":47,"tag":177,"props":1652,"children":1653},{},[1654],{"type":56,"value":1655},"~1 h",{"type":47,"tag":177,"props":1657,"children":1658},{},[1659],{"type":56,"value":1660},"Regional cloud incident",{"type":47,"tag":150,"props":1662,"children":1663},{},[1664,1669,1674,1679],{"type":47,"tag":177,"props":1665,"children":1666},{},[1667],{"type":56,"value":1668},"8",{"type":47,"tag":177,"props":1670,"children":1671},{},[1672],{"type":56,"value":1673},"64 min",{"type":47,"tag":177,"props":1675,"children":1676},{},[1677],{"type":56,"value":1678},"~2 h",{"type":47,"tag":177,"props":1680,"children":1681},{},[1682],{"type":56,"value":1683},"Hours-long outage",{"type":47,"tag":150,"props":1685,"children":1686},{},[1687,1692,1696,1701],{"type":47,"tag":177,"props":1688,"children":1689},{},[1690],{"type":56,"value":1691},"9",{"type":47,"tag":177,"props":1693,"children":1694},{},[1695],{"type":56,"value":1678},{"type":47,"tag":177,"props":1697,"children":1698},{},[1699],{"type":56,"value":1700},"~4 h",{"type":47,"tag":177,"props":1702,"children":1703},{},[1704],{"type":56,"value":1705},"Half-day outage",{"type":47,"tag":150,"props":1707,"children":1708},{},[1709,1714,1719,1724],{"type":47,"tag":177,"props":1710,"children":1711},{},[1712],{"type":56,"value":1713},"10",{"type":47,"tag":177,"props":1715,"children":1716},{},[1717],{"type":56,"value":1718},"~4.3 h",{"type":47,"tag":177,"props":1720,"children":1721},{},[1722],{"type":56,"value":1723},"~8.5 h",{"type":47,"tag":177,"props":1725,"children":1726},{},[1727],{"type":56,"value":1728},"Overnight outage",{"type":47,"tag":150,"props":1730,"children":1731},{},[1732,1737,1741,1746],{"type":47,"tag":177,"props":1733,"children":1734},{},[1735],{"type":56,"value":1736},"11",{"type":47,"tag":177,"props":1738,"children":1739},{},[1740],{"type":56,"value":1723},{"type":47,"tag":177,"props":1742,"children":1743},{},[1744],{"type":56,"value":1745},"~17 h",{"type":47,"tag":177,"props":1747,"children":1748},{},[1749],{"type":56,"value":1750},"Outage longer than a working day",{"type":47,"tag":150,"props":1752,"children":1753},{},[1754,1759,1763,1771],{"type":47,"tag":177,"props":1755,"children":1756},{},[1757],{"type":56,"value":1758},"12",{"type":47,"tag":177,"props":1760,"children":1761},{},[1762],{"type":56,"value":1745},{"type":47,"tag":177,"props":1764,"children":1765},{},[1766],{"type":47,"tag":52,"props":1767,"children":1768},{},[1769],{"type":56,"value":1770},"~34 h, then dead-letter",{"type":47,"tag":177,"props":1772,"children":1773},{},[1774],{"type":56,"value":1775},"A day and a half, then a human looks",{"type":47,"tag":48,"props":1777,"children":1778},{},[1779],{"type":56,"value":1780},"That is 12 retries over about 34 hours. Cap the wait (a day is common) if you extend the schedule further; otherwise the gaps grow past anything useful.",{"type":47,"tag":48,"props":1782,"children":1783},{},[1784],{"type":56,"value":1785},"A worker that claims due events, retries them and dead-letters them at the deadline fits in one query and one loop:",{"type":47,"tag":726,"props":1787,"children":1789},{"label":1788},"worker.js · claim, process, retry or dead-letter",[1790],{"type":47,"tag":731,"props":1791,"children":1793},{"className":1054,"code":1792,"language":1056,"meta":7,"style":7},"const MAX_RETRIES = 12\n\nasync function runOnce() {\n  const db = await pool.connect()\n  try {\n    await db.query('begin') // the row locks below last until commit\n    await processBatch(db)\n    await db.query('commit')\n  } catch (err) {\n    await db.query('rollback')\n    throw err\n  } finally {\n    db.release()\n  }\n}\n\nasync function processBatch(db) {\n  // Claim due events; SKIP LOCKED lets several workers run safely.\n  const { rows } = await db.query(\n    `select * from webhook_events\n      where status = 'pending' and next_attempt_at \u003C= now()\n      order by next_attempt_at limit 50\n      for update skip locked`)\n\n  for (const ev of rows) {\n    try {\n      await handle(ev) // idempotent: safe to run twice\n      await db.query(`update webhook_events set status = 'done'\n                      where id = $1`, [ev.id])\n    } catch (err) {\n      const attempts = ev.attempts + 1\n      const permanent = err.permanent || attempts > MAX_RETRIES\n      const wait = 30 * 2 ** (attempts - 1) * (0.8 + Math.random() * 0.4)\n      await db.query(\n        `update webhook_events\n            set attempts = $2, last_error = $3,\n                status = $4,\n                next_attempt_at = now() + make_interval(secs => $5)\n          where id = $1`,\n        [ev.id, attempts, String(err), permanent ? 'dead' : 'pending', wait])\n    }\n  }\n}\n",[1794],{"type":47,"tag":81,"props":1795,"children":1796},{"__ignoreMap":7},[1797,1819,1826,1848,1885,1897,1931,1948,1976,1994,2022,2035,2051,2069,2078,2087,2095,2125,2134,2178,2187,2196,2205,2218,2226,2258,2271,2295,2320,2334,2351,2384,2425,2525,2545,2554,2563,2572,2581,2594,2638,2647,2655],{"type":47,"tag":741,"props":1798,"children":1799},{"class":743,"line":744},[1800,1805,1810,1814],{"type":47,"tag":741,"props":1801,"children":1802},{"style":748},[1803],{"type":56,"value":1804},"const",{"type":47,"tag":741,"props":1806,"children":1807},{"style":913},[1808],{"type":56,"value":1809}," MAX_RETRIES",{"type":47,"tag":741,"props":1811,"children":1812},{"style":748},[1813],{"type":56,"value":781},{"type":47,"tag":741,"props":1815,"children":1816},{"style":913},[1817],{"type":56,"value":1818}," 12\n",{"type":47,"tag":741,"props":1820,"children":1821},{"class":743,"line":765},[1822],{"type":47,"tag":741,"props":1823,"children":1824},{"emptyLinePlaceholder":1211},[1825],{"type":56,"value":1214},{"type":47,"tag":741,"props":1827,"children":1828},{"class":743,"line":1207},[1829,1833,1838,1843],{"type":47,"tag":741,"props":1830,"children":1831},{"style":748},[1832],{"type":56,"value":1112},{"type":47,"tag":741,"props":1834,"children":1835},{"style":748},[1836],{"type":56,"value":1837}," function",{"type":47,"tag":741,"props":1839,"children":1840},{"style":860},[1841],{"type":56,"value":1842}," runOnce",{"type":47,"tag":741,"props":1844,"children":1845},{"style":759},[1846],{"type":56,"value":1847},"() {\n",{"type":47,"tag":741,"props":1849,"children":1850},{"class":743,"line":1217},[1851,1856,1861,1865,1870,1875,1880],{"type":47,"tag":741,"props":1852,"children":1853},{"style":748},[1854],{"type":56,"value":1855},"  const",{"type":47,"tag":741,"props":1857,"children":1858},{"style":913},[1859],{"type":56,"value":1860}," db",{"type":47,"tag":741,"props":1862,"children":1863},{"style":748},[1864],{"type":56,"value":781},{"type":47,"tag":741,"props":1866,"children":1867},{"style":748},[1868],{"type":56,"value":1869}," await",{"type":47,"tag":741,"props":1871,"children":1872},{"style":759},[1873],{"type":56,"value":1874}," pool.",{"type":47,"tag":741,"props":1876,"children":1877},{"style":860},[1878],{"type":56,"value":1879},"connect",{"type":47,"tag":741,"props":1881,"children":1882},{"style":759},[1883],{"type":56,"value":1884},"()\n",{"type":47,"tag":741,"props":1886,"children":1887},{"class":743,"line":1226},[1888,1893],{"type":47,"tag":741,"props":1889,"children":1890},{"style":748},[1891],{"type":56,"value":1892},"  try",{"type":47,"tag":741,"props":1894,"children":1895},{"style":759},[1896],{"type":56,"value":1148},{"type":47,"tag":741,"props":1898,"children":1899},{"class":743,"line":1250},[1900,1905,1909,1913,1917,1922,1926],{"type":47,"tag":741,"props":1901,"children":1902},{"style":748},[1903],{"type":56,"value":1904},"    await",{"type":47,"tag":741,"props":1906,"children":1907},{"style":759},[1908],{"type":56,"value":1237},{"type":47,"tag":741,"props":1910,"children":1911},{"style":860},[1912],{"type":56,"value":1242},{"type":47,"tag":741,"props":1914,"children":1915},{"style":759},[1916],{"type":56,"value":1077},{"type":47,"tag":741,"props":1918,"children":1919},{"style":784},[1920],{"type":56,"value":1921},"'begin'",{"type":47,"tag":741,"props":1923,"children":1924},{"style":759},[1925],{"type":56,"value":1138},{"type":47,"tag":741,"props":1927,"children":1928},{"style":851},[1929],{"type":56,"value":1930},"// the row locks below last until commit\n",{"type":47,"tag":741,"props":1932,"children":1933},{"class":743,"line":1259},[1934,1938,1943],{"type":47,"tag":741,"props":1935,"children":1936},{"style":748},[1937],{"type":56,"value":1904},{"type":47,"tag":741,"props":1939,"children":1940},{"style":860},[1941],{"type":56,"value":1942}," processBatch",{"type":47,"tag":741,"props":1944,"children":1945},{"style":759},[1946],{"type":56,"value":1947},"(db)\n",{"type":47,"tag":741,"props":1949,"children":1950},{"class":743,"line":1268},[1951,1955,1959,1963,1967,1972],{"type":47,"tag":741,"props":1952,"children":1953},{"style":748},[1954],{"type":56,"value":1904},{"type":47,"tag":741,"props":1956,"children":1957},{"style":759},[1958],{"type":56,"value":1237},{"type":47,"tag":741,"props":1960,"children":1961},{"style":860},[1962],{"type":56,"value":1242},{"type":47,"tag":741,"props":1964,"children":1965},{"style":759},[1966],{"type":56,"value":1077},{"type":47,"tag":741,"props":1968,"children":1969},{"style":784},[1970],{"type":56,"value":1971},"'commit'",{"type":47,"tag":741,"props":1973,"children":1974},{"style":759},[1975],{"type":56,"value":1204},{"type":47,"tag":741,"props":1977,"children":1978},{"class":743,"line":1282},[1979,1984,1989],{"type":47,"tag":741,"props":1980,"children":1981},{"style":759},[1982],{"type":56,"value":1983},"  } ",{"type":47,"tag":741,"props":1985,"children":1986},{"style":748},[1987],{"type":56,"value":1988},"catch",{"type":47,"tag":741,"props":1990,"children":1991},{"style":759},[1992],{"type":56,"value":1993}," (err) {\n",{"type":47,"tag":741,"props":1995,"children":1996},{"class":743,"line":1301},[1997,2001,2005,2009,2013,2018],{"type":47,"tag":741,"props":1998,"children":1999},{"style":748},[2000],{"type":56,"value":1904},{"type":47,"tag":741,"props":2002,"children":2003},{"style":759},[2004],{"type":56,"value":1237},{"type":47,"tag":741,"props":2006,"children":2007},{"style":860},[2008],{"type":56,"value":1242},{"type":47,"tag":741,"props":2010,"children":2011},{"style":759},[2012],{"type":56,"value":1077},{"type":47,"tag":741,"props":2014,"children":2015},{"style":784},[2016],{"type":56,"value":2017},"'rollback'",{"type":47,"tag":741,"props":2019,"children":2020},{"style":759},[2021],{"type":56,"value":1204},{"type":47,"tag":741,"props":2023,"children":2024},{"class":743,"line":1310},[2025,2030],{"type":47,"tag":741,"props":2026,"children":2027},{"style":748},[2028],{"type":56,"value":2029},"    throw",{"type":47,"tag":741,"props":2031,"children":2032},{"style":759},[2033],{"type":56,"value":2034}," err\n",{"type":47,"tag":741,"props":2036,"children":2037},{"class":743,"line":1335},[2038,2042,2047],{"type":47,"tag":741,"props":2039,"children":2040},{"style":759},[2041],{"type":56,"value":1983},{"type":47,"tag":741,"props":2043,"children":2044},{"style":748},[2045],{"type":56,"value":2046},"finally",{"type":47,"tag":741,"props":2048,"children":2049},{"style":759},[2050],{"type":56,"value":1148},{"type":47,"tag":741,"props":2052,"children":2054},{"class":743,"line":2053},13,[2055,2060,2065],{"type":47,"tag":741,"props":2056,"children":2057},{"style":759},[2058],{"type":56,"value":2059},"    db.",{"type":47,"tag":741,"props":2061,"children":2062},{"style":860},[2063],{"type":56,"value":2064},"release",{"type":47,"tag":741,"props":2066,"children":2067},{"style":759},[2068],{"type":56,"value":1884},{"type":47,"tag":741,"props":2070,"children":2072},{"class":743,"line":2071},14,[2073],{"type":47,"tag":741,"props":2074,"children":2075},{"style":759},[2076],{"type":56,"value":2077},"  }\n",{"type":47,"tag":741,"props":2079,"children":2081},{"class":743,"line":2080},15,[2082],{"type":47,"tag":741,"props":2083,"children":2084},{"style":759},[2085],{"type":56,"value":2086},"}\n",{"type":47,"tag":741,"props":2088,"children":2090},{"class":743,"line":2089},16,[2091],{"type":47,"tag":741,"props":2092,"children":2093},{"emptyLinePlaceholder":1211},[2094],{"type":56,"value":1214},{"type":47,"tag":741,"props":2096,"children":2098},{"class":743,"line":2097},17,[2099,2103,2107,2111,2115,2120],{"type":47,"tag":741,"props":2100,"children":2101},{"style":748},[2102],{"type":56,"value":1112},{"type":47,"tag":741,"props":2104,"children":2105},{"style":748},[2106],{"type":56,"value":1837},{"type":47,"tag":741,"props":2108,"children":2109},{"style":860},[2110],{"type":56,"value":1942},{"type":47,"tag":741,"props":2112,"children":2113},{"style":759},[2114],{"type":56,"value":1077},{"type":47,"tag":741,"props":2116,"children":2117},{"style":1120},[2118],{"type":56,"value":2119},"db",{"type":47,"tag":741,"props":2121,"children":2122},{"style":759},[2123],{"type":56,"value":2124},") {\n",{"type":47,"tag":741,"props":2126,"children":2128},{"class":743,"line":2127},18,[2129],{"type":47,"tag":741,"props":2130,"children":2131},{"style":851},[2132],{"type":56,"value":2133},"  // Claim due events; SKIP LOCKED lets several workers run safely.\n",{"type":47,"tag":741,"props":2135,"children":2137},{"class":743,"line":2136},19,[2138,2142,2147,2152,2157,2162,2166,2170,2174],{"type":47,"tag":741,"props":2139,"children":2140},{"style":748},[2141],{"type":56,"value":1855},{"type":47,"tag":741,"props":2143,"children":2144},{"style":759},[2145],{"type":56,"value":2146}," { ",{"type":47,"tag":741,"props":2148,"children":2149},{"style":913},[2150],{"type":56,"value":2151},"rows",{"type":47,"tag":741,"props":2153,"children":2154},{"style":759},[2155],{"type":56,"value":2156}," } ",{"type":47,"tag":741,"props":2158,"children":2159},{"style":748},[2160],{"type":56,"value":2161},"=",{"type":47,"tag":741,"props":2163,"children":2164},{"style":748},[2165],{"type":56,"value":1869},{"type":47,"tag":741,"props":2167,"children":2168},{"style":759},[2169],{"type":56,"value":1237},{"type":47,"tag":741,"props":2171,"children":2172},{"style":860},[2173],{"type":56,"value":1242},{"type":47,"tag":741,"props":2175,"children":2176},{"style":759},[2177],{"type":56,"value":1247},{"type":47,"tag":741,"props":2179,"children":2181},{"class":743,"line":2180},20,[2182],{"type":47,"tag":741,"props":2183,"children":2184},{"style":784},[2185],{"type":56,"value":2186},"    `select * from webhook_events\n",{"type":47,"tag":741,"props":2188,"children":2190},{"class":743,"line":2189},21,[2191],{"type":47,"tag":741,"props":2192,"children":2193},{"style":784},[2194],{"type":56,"value":2195},"      where status = 'pending' and next_attempt_at \u003C= now()\n",{"type":47,"tag":741,"props":2197,"children":2199},{"class":743,"line":2198},22,[2200],{"type":47,"tag":741,"props":2201,"children":2202},{"style":784},[2203],{"type":56,"value":2204},"      order by next_attempt_at limit 50\n",{"type":47,"tag":741,"props":2206,"children":2208},{"class":743,"line":2207},23,[2209,2214],{"type":47,"tag":741,"props":2210,"children":2211},{"style":784},[2212],{"type":56,"value":2213},"      for update skip locked`",{"type":47,"tag":741,"props":2215,"children":2216},{"style":759},[2217],{"type":56,"value":1204},{"type":47,"tag":741,"props":2219,"children":2221},{"class":743,"line":2220},24,[2222],{"type":47,"tag":741,"props":2223,"children":2224},{"emptyLinePlaceholder":1211},[2225],{"type":56,"value":1214},{"type":47,"tag":741,"props":2227,"children":2229},{"class":743,"line":2228},25,[2230,2235,2239,2243,2248,2253],{"type":47,"tag":741,"props":2231,"children":2232},{"style":748},[2233],{"type":56,"value":2234},"  for",{"type":47,"tag":741,"props":2236,"children":2237},{"style":759},[2238],{"type":56,"value":1117},{"type":47,"tag":741,"props":2240,"children":2241},{"style":748},[2242],{"type":56,"value":1804},{"type":47,"tag":741,"props":2244,"children":2245},{"style":913},[2246],{"type":56,"value":2247}," ev",{"type":47,"tag":741,"props":2249,"children":2250},{"style":748},[2251],{"type":56,"value":2252}," of",{"type":47,"tag":741,"props":2254,"children":2255},{"style":759},[2256],{"type":56,"value":2257}," rows) {\n",{"type":47,"tag":741,"props":2259,"children":2261},{"class":743,"line":2260},26,[2262,2267],{"type":47,"tag":741,"props":2263,"children":2264},{"style":748},[2265],{"type":56,"value":2266},"    try",{"type":47,"tag":741,"props":2268,"children":2269},{"style":759},[2270],{"type":56,"value":1148},{"type":47,"tag":741,"props":2272,"children":2274},{"class":743,"line":2273},27,[2275,2280,2285,2290],{"type":47,"tag":741,"props":2276,"children":2277},{"style":748},[2278],{"type":56,"value":2279},"      await",{"type":47,"tag":741,"props":2281,"children":2282},{"style":860},[2283],{"type":56,"value":2284}," handle",{"type":47,"tag":741,"props":2286,"children":2287},{"style":759},[2288],{"type":56,"value":2289},"(ev) ",{"type":47,"tag":741,"props":2291,"children":2292},{"style":851},[2293],{"type":56,"value":2294},"// idempotent: safe to run twice\n",{"type":47,"tag":741,"props":2296,"children":2298},{"class":743,"line":2297},28,[2299,2303,2307,2311,2315],{"type":47,"tag":741,"props":2300,"children":2301},{"style":748},[2302],{"type":56,"value":2279},{"type":47,"tag":741,"props":2304,"children":2305},{"style":759},[2306],{"type":56,"value":1237},{"type":47,"tag":741,"props":2308,"children":2309},{"style":860},[2310],{"type":56,"value":1242},{"type":47,"tag":741,"props":2312,"children":2313},{"style":759},[2314],{"type":56,"value":1077},{"type":47,"tag":741,"props":2316,"children":2317},{"style":784},[2318],{"type":56,"value":2319},"`update webhook_events set status = 'done'\n",{"type":47,"tag":741,"props":2321,"children":2323},{"class":743,"line":2322},29,[2324,2329],{"type":47,"tag":741,"props":2325,"children":2326},{"style":784},[2327],{"type":56,"value":2328},"                      where id = $1`",{"type":47,"tag":741,"props":2330,"children":2331},{"style":759},[2332],{"type":56,"value":2333},", [ev.id])\n",{"type":47,"tag":741,"props":2335,"children":2337},{"class":743,"line":2336},30,[2338,2343,2347],{"type":47,"tag":741,"props":2339,"children":2340},{"style":759},[2341],{"type":56,"value":2342},"    } ",{"type":47,"tag":741,"props":2344,"children":2345},{"style":748},[2346],{"type":56,"value":1988},{"type":47,"tag":741,"props":2348,"children":2349},{"style":759},[2350],{"type":56,"value":1993},{"type":47,"tag":741,"props":2352,"children":2354},{"class":743,"line":2353},31,[2355,2360,2365,2369,2374,2379],{"type":47,"tag":741,"props":2356,"children":2357},{"style":748},[2358],{"type":56,"value":2359},"      const",{"type":47,"tag":741,"props":2361,"children":2362},{"style":913},[2363],{"type":56,"value":2364}," attempts",{"type":47,"tag":741,"props":2366,"children":2367},{"style":748},[2368],{"type":56,"value":781},{"type":47,"tag":741,"props":2370,"children":2371},{"style":759},[2372],{"type":56,"value":2373}," ev.attempts ",{"type":47,"tag":741,"props":2375,"children":2376},{"style":748},[2377],{"type":56,"value":2378},"+",{"type":47,"tag":741,"props":2380,"children":2381},{"style":913},[2382],{"type":56,"value":2383}," 1\n",{"type":47,"tag":741,"props":2385,"children":2387},{"class":743,"line":2386},32,[2388,2392,2397,2401,2406,2411,2416,2420],{"type":47,"tag":741,"props":2389,"children":2390},{"style":748},[2391],{"type":56,"value":2359},{"type":47,"tag":741,"props":2393,"children":2394},{"style":913},[2395],{"type":56,"value":2396}," permanent",{"type":47,"tag":741,"props":2398,"children":2399},{"style":748},[2400],{"type":56,"value":781},{"type":47,"tag":741,"props":2402,"children":2403},{"style":759},[2404],{"type":56,"value":2405}," err.permanent ",{"type":47,"tag":741,"props":2407,"children":2408},{"style":748},[2409],{"type":56,"value":2410},"||",{"type":47,"tag":741,"props":2412,"children":2413},{"style":759},[2414],{"type":56,"value":2415}," attempts ",{"type":47,"tag":741,"props":2417,"children":2418},{"style":748},[2419],{"type":56,"value":887},{"type":47,"tag":741,"props":2421,"children":2422},{"style":913},[2423],{"type":56,"value":2424}," MAX_RETRIES\n",{"type":47,"tag":741,"props":2426,"children":2428},{"class":743,"line":2427},33,[2429,2433,2438,2442,2447,2452,2457,2462,2467,2471,2475,2479,2484,2488,2493,2498,2503,2508,2512,2516,2521],{"type":47,"tag":741,"props":2430,"children":2431},{"style":748},[2432],{"type":56,"value":2359},{"type":47,"tag":741,"props":2434,"children":2435},{"style":913},[2436],{"type":56,"value":2437}," wait",{"type":47,"tag":741,"props":2439,"children":2440},{"style":748},[2441],{"type":56,"value":781},{"type":47,"tag":741,"props":2443,"children":2444},{"style":913},[2445],{"type":56,"value":2446}," 30",{"type":47,"tag":741,"props":2448,"children":2449},{"style":748},[2450],{"type":56,"value":2451}," *",{"type":47,"tag":741,"props":2453,"children":2454},{"style":913},[2455],{"type":56,"value":2456}," 2",{"type":47,"tag":741,"props":2458,"children":2459},{"style":748},[2460],{"type":56,"value":2461}," **",{"type":47,"tag":741,"props":2463,"children":2464},{"style":759},[2465],{"type":56,"value":2466}," (attempts ",{"type":47,"tag":741,"props":2468,"children":2469},{"style":748},[2470],{"type":56,"value":817},{"type":47,"tag":741,"props":2472,"children":2473},{"style":913},[2474],{"type":56,"value":916},{"type":47,"tag":741,"props":2476,"children":2477},{"style":759},[2478],{"type":56,"value":1138},{"type":47,"tag":741,"props":2480,"children":2481},{"style":748},[2482],{"type":56,"value":2483},"*",{"type":47,"tag":741,"props":2485,"children":2486},{"style":759},[2487],{"type":56,"value":1117},{"type":47,"tag":741,"props":2489,"children":2490},{"style":913},[2491],{"type":56,"value":2492},"0.8",{"type":47,"tag":741,"props":2494,"children":2495},{"style":748},[2496],{"type":56,"value":2497}," +",{"type":47,"tag":741,"props":2499,"children":2500},{"style":759},[2501],{"type":56,"value":2502}," Math.",{"type":47,"tag":741,"props":2504,"children":2505},{"style":860},[2506],{"type":56,"value":2507},"random",{"type":47,"tag":741,"props":2509,"children":2510},{"style":759},[2511],{"type":56,"value":812},{"type":47,"tag":741,"props":2513,"children":2514},{"style":748},[2515],{"type":56,"value":2483},{"type":47,"tag":741,"props":2517,"children":2518},{"style":913},[2519],{"type":56,"value":2520}," 0.4",{"type":47,"tag":741,"props":2522,"children":2523},{"style":759},[2524],{"type":56,"value":1204},{"type":47,"tag":741,"props":2526,"children":2528},{"class":743,"line":2527},34,[2529,2533,2537,2541],{"type":47,"tag":741,"props":2530,"children":2531},{"style":748},[2532],{"type":56,"value":2279},{"type":47,"tag":741,"props":2534,"children":2535},{"style":759},[2536],{"type":56,"value":1237},{"type":47,"tag":741,"props":2538,"children":2539},{"style":860},[2540],{"type":56,"value":1242},{"type":47,"tag":741,"props":2542,"children":2543},{"style":759},[2544],{"type":56,"value":1247},{"type":47,"tag":741,"props":2546,"children":2548},{"class":743,"line":2547},35,[2549],{"type":47,"tag":741,"props":2550,"children":2551},{"style":784},[2552],{"type":56,"value":2553},"        `update webhook_events\n",{"type":47,"tag":741,"props":2555,"children":2557},{"class":743,"line":2556},36,[2558],{"type":47,"tag":741,"props":2559,"children":2560},{"style":784},[2561],{"type":56,"value":2562},"            set attempts = $2, last_error = $3,\n",{"type":47,"tag":741,"props":2564,"children":2566},{"class":743,"line":2565},37,[2567],{"type":47,"tag":741,"props":2568,"children":2569},{"style":784},[2570],{"type":56,"value":2571},"                status = $4,\n",{"type":47,"tag":741,"props":2573,"children":2575},{"class":743,"line":2574},38,[2576],{"type":47,"tag":741,"props":2577,"children":2578},{"style":784},[2579],{"type":56,"value":2580},"                next_attempt_at = now() + make_interval(secs => $5)\n",{"type":47,"tag":741,"props":2582,"children":2584},{"class":743,"line":2583},39,[2585,2590],{"type":47,"tag":741,"props":2586,"children":2587},{"style":784},[2588],{"type":56,"value":2589},"          where id = $1`",{"type":47,"tag":741,"props":2591,"children":2592},{"style":759},[2593],{"type":56,"value":1279},{"type":47,"tag":741,"props":2595,"children":2597},{"class":743,"line":2596},40,[2598,2603,2608,2613,2618,2623,2628,2633],{"type":47,"tag":741,"props":2599,"children":2600},{"style":759},[2601],{"type":56,"value":2602},"        [ev.id, attempts, ",{"type":47,"tag":741,"props":2604,"children":2605},{"style":860},[2606],{"type":56,"value":2607},"String",{"type":47,"tag":741,"props":2609,"children":2610},{"style":759},[2611],{"type":56,"value":2612},"(err), permanent ",{"type":47,"tag":741,"props":2614,"children":2615},{"style":748},[2616],{"type":56,"value":2617},"?",{"type":47,"tag":741,"props":2619,"children":2620},{"style":784},[2621],{"type":56,"value":2622}," 'dead'",{"type":47,"tag":741,"props":2624,"children":2625},{"style":748},[2626],{"type":56,"value":2627}," :",{"type":47,"tag":741,"props":2629,"children":2630},{"style":784},[2631],{"type":56,"value":2632}," 'pending'",{"type":47,"tag":741,"props":2634,"children":2635},{"style":759},[2636],{"type":56,"value":2637},", wait])\n",{"type":47,"tag":741,"props":2639,"children":2641},{"class":743,"line":2640},41,[2642],{"type":47,"tag":741,"props":2643,"children":2644},{"style":759},[2645],{"type":56,"value":2646},"    }\n",{"type":47,"tag":741,"props":2648,"children":2650},{"class":743,"line":2649},42,[2651],{"type":47,"tag":741,"props":2652,"children":2653},{"style":759},[2654],{"type":56,"value":2077},{"type":47,"tag":741,"props":2656,"children":2658},{"class":743,"line":2657},43,[2659],{"type":47,"tag":741,"props":2660,"children":2661},{"style":759},[2662],{"type":56,"value":2086},{"type":47,"tag":48,"props":2664,"children":2665},{},[2666,2668,2674,2676,2682],{"type":56,"value":2667},"Add an index on ",{"type":47,"tag":81,"props":2669,"children":2671},{"className":2670},[],[2672],{"type":56,"value":2673},"(status, next_attempt_at)",{"type":56,"value":2675},". The trade-off of this version is that row locks are held while ",{"type":47,"tag":81,"props":2677,"children":2679},{"className":2678},[],[2680],{"type":56,"value":2681},"handle()",{"type":56,"value":2683}," runs. If handlers are slow, claim with a lease instead and process outside the transaction:",{"type":47,"tag":726,"props":2685,"children":2687},{"label":2686},"SQL · claim a batch with a 5-minute lease",[2688],{"type":47,"tag":731,"props":2689,"children":2691},{"className":733,"code":2690,"language":735,"meta":7,"style":7},"update webhook_events\n   set status = 'processing', locked_until = now() + interval '5 min'\n where id in (select id from webhook_events\n               where status = 'pending' and next_attempt_at \u003C= now()\n               order by next_attempt_at limit 50\n               for update skip locked)\nreturning *;\n",[2692],{"type":47,"tag":81,"props":2693,"children":2694},{"__ignoreMap":7},[2695,2707,2758,2797,2839,2861,2884],{"type":47,"tag":741,"props":2696,"children":2697},{"class":743,"line":744},[2698,2703],{"type":47,"tag":741,"props":2699,"children":2700},{"style":748},[2701],{"type":56,"value":2702},"update",{"type":47,"tag":741,"props":2704,"children":2705},{"style":759},[2706],{"type":56,"value":762},{"type":47,"tag":741,"props":2708,"children":2709},{"class":743,"line":765},[2710,2715,2719,2723,2728,2733,2737,2741,2745,2749,2753],{"type":47,"tag":741,"props":2711,"children":2712},{"style":748},[2713],{"type":56,"value":2714},"   set",{"type":47,"tag":741,"props":2716,"children":2717},{"style":748},[2718],{"type":56,"value":776},{"type":47,"tag":741,"props":2720,"children":2721},{"style":748},[2722],{"type":56,"value":781},{"type":47,"tag":741,"props":2724,"children":2725},{"style":784},[2726],{"type":56,"value":2727}," 'processing'",{"type":47,"tag":741,"props":2729,"children":2730},{"style":759},[2731],{"type":56,"value":2732},", locked_until ",{"type":47,"tag":741,"props":2734,"children":2735},{"style":748},[2736],{"type":56,"value":2161},{"type":47,"tag":741,"props":2738,"children":2739},{"style":748},[2740],{"type":56,"value":807},{"type":47,"tag":741,"props":2742,"children":2743},{"style":759},[2744],{"type":56,"value":812},{"type":47,"tag":741,"props":2746,"children":2747},{"style":748},[2748],{"type":56,"value":2378},{"type":47,"tag":741,"props":2750,"children":2751},{"style":759},[2752],{"type":56,"value":822},{"type":47,"tag":741,"props":2754,"children":2755},{"style":784},[2756],{"type":56,"value":2757},"'5 min'\n",{"type":47,"tag":741,"props":2759,"children":2760},{"class":743,"line":1207},[2761,2765,2770,2775,2779,2784,2788,2793],{"type":47,"tag":741,"props":2762,"children":2763},{"style":748},[2764],{"type":56,"value":771},{"type":47,"tag":741,"props":2766,"children":2767},{"style":759},[2768],{"type":56,"value":2769}," id ",{"type":47,"tag":741,"props":2771,"children":2772},{"style":748},[2773],{"type":56,"value":2774},"in",{"type":47,"tag":741,"props":2776,"children":2777},{"style":759},[2778],{"type":56,"value":1117},{"type":47,"tag":741,"props":2780,"children":2781},{"style":748},[2782],{"type":56,"value":2783},"select",{"type":47,"tag":741,"props":2785,"children":2786},{"style":759},[2787],{"type":56,"value":2769},{"type":47,"tag":741,"props":2789,"children":2790},{"style":748},[2791],{"type":56,"value":2792},"from",{"type":47,"tag":741,"props":2794,"children":2795},{"style":759},[2796],{"type":56,"value":762},{"type":47,"tag":741,"props":2798,"children":2799},{"class":743,"line":1217},[2800,2805,2809,2813,2817,2821,2826,2831,2835],{"type":47,"tag":741,"props":2801,"children":2802},{"style":748},[2803],{"type":56,"value":2804},"               where",{"type":47,"tag":741,"props":2806,"children":2807},{"style":748},[2808],{"type":56,"value":776},{"type":47,"tag":741,"props":2810,"children":2811},{"style":748},[2812],{"type":56,"value":781},{"type":47,"tag":741,"props":2814,"children":2815},{"style":784},[2816],{"type":56,"value":2632},{"type":47,"tag":741,"props":2818,"children":2819},{"style":748},[2820],{"type":56,"value":792},{"type":47,"tag":741,"props":2822,"children":2823},{"style":759},[2824],{"type":56,"value":2825}," next_attempt_at ",{"type":47,"tag":741,"props":2827,"children":2828},{"style":748},[2829],{"type":56,"value":2830},"\u003C=",{"type":47,"tag":741,"props":2832,"children":2833},{"style":748},[2834],{"type":56,"value":807},{"type":47,"tag":741,"props":2836,"children":2837},{"style":759},[2838],{"type":56,"value":1884},{"type":47,"tag":741,"props":2840,"children":2841},{"class":743,"line":1226},[2842,2847,2851,2856],{"type":47,"tag":741,"props":2843,"children":2844},{"style":748},[2845],{"type":56,"value":2846},"               order by",{"type":47,"tag":741,"props":2848,"children":2849},{"style":759},[2850],{"type":56,"value":2825},{"type":47,"tag":741,"props":2852,"children":2853},{"style":748},[2854],{"type":56,"value":2855},"limit",{"type":47,"tag":741,"props":2857,"children":2858},{"style":913},[2859],{"type":56,"value":2860}," 50\n",{"type":47,"tag":741,"props":2862,"children":2863},{"class":743,"line":1250},[2864,2869,2874,2879],{"type":47,"tag":741,"props":2865,"children":2866},{"style":748},[2867],{"type":56,"value":2868},"               for",{"type":47,"tag":741,"props":2870,"children":2871},{"style":748},[2872],{"type":56,"value":2873}," update",{"type":47,"tag":741,"props":2875,"children":2876},{"style":748},[2877],{"type":56,"value":2878}," skip",{"type":47,"tag":741,"props":2880,"children":2881},{"style":759},[2882],{"type":56,"value":2883}," locked)\n",{"type":47,"tag":741,"props":2885,"children":2886},{"class":743,"line":1259},[2887,2892,2896],{"type":47,"tag":741,"props":2888,"children":2889},{"style":759},[2890],{"type":56,"value":2891},"returning ",{"type":47,"tag":741,"props":2893,"children":2894},{"style":748},[2895],{"type":56,"value":2483},{"type":47,"tag":741,"props":2897,"children":2898},{"style":759},[2899],{"type":56,"value":832},{"type":47,"tag":48,"props":2901,"children":2902},{},[2903,2905,2910,2912,2918],{"type":56,"value":2904},"A periodic job puts rows whose lease expired back to ",{"type":47,"tag":81,"props":2906,"children":2908},{"className":2907},[],[2909],{"type":56,"value":1379},{"type":56,"value":2911},", which covers a worker that died mid-batch. ",{"type":47,"tag":81,"props":2913,"children":2915},{"className":2914},[],[2916],{"type":56,"value":2917},"SKIP LOCKED",{"type":56,"value":2919}," is what stops two workers from claiming the same event at the same moment and processing it twice; the idempotent handler covers the remaining case, where a worker crashes after the side effect but before marking the event done.",{"type":47,"tag":48,"props":2921,"children":2922},{},[2923,2925,2931],{"type":56,"value":2924},"Separate retryable failures from permanent ones. A ",{"type":47,"tag":81,"props":2926,"children":2928},{"className":2927},[],[2929],{"type":56,"value":2930},"503",{"type":56,"value":2932}," from a dependency is worth retrying; a payload that fails validation will fail identically every time and should go straight to the dead-letter stage.",{"type":47,"tag":142,"props":2934,"children":2935},{},[2936,2957],{"type":47,"tag":146,"props":2937,"children":2938},{},[2939],{"type":47,"tag":150,"props":2940,"children":2941},{},[2942,2947,2952],{"type":47,"tag":154,"props":2943,"children":2944},{},[2945],{"type":56,"value":2946},"Response or error",{"type":47,"tag":154,"props":2948,"children":2949},{},[2950],{"type":56,"value":2951},"Retry?",{"type":47,"tag":154,"props":2953,"children":2954},{},[2955],{"type":56,"value":2956},"Why",{"type":47,"tag":170,"props":2958,"children":2959},{},[2960,2978,2999,3027,3048,3070,3099,3127],{"type":47,"tag":150,"props":2961,"children":2962},{},[2963,2968,2973],{"type":47,"tag":177,"props":2964,"children":2965},{},[2966],{"type":56,"value":2967},"Timeout, connection refused, DNS failure",{"type":47,"tag":177,"props":2969,"children":2970},{},[2971],{"type":56,"value":2972},"Yes",{"type":47,"tag":177,"props":2974,"children":2975},{},[2976],{"type":56,"value":2977},"Transient by nature",{"type":47,"tag":150,"props":2979,"children":2980},{},[2981,2990,2994],{"type":47,"tag":177,"props":2982,"children":2983},{},[2984],{"type":47,"tag":81,"props":2985,"children":2987},{"className":2986},[],[2988],{"type":56,"value":2989},"5xx",{"type":47,"tag":177,"props":2991,"children":2992},{},[2993],{"type":56,"value":2972},{"type":47,"tag":177,"props":2995,"children":2996},{},[2997],{"type":56,"value":2998},"Server-side, usually temporary",{"type":47,"tag":150,"props":3000,"children":3001},{},[3002,3011,3022],{"type":47,"tag":177,"props":3003,"children":3004},{},[3005],{"type":47,"tag":81,"props":3006,"children":3008},{"className":3007},[],[3009],{"type":56,"value":3010},"429 Too Many Requests",{"type":47,"tag":177,"props":3012,"children":3013},{},[3014,3016],{"type":56,"value":3015},"Yes, after ",{"type":47,"tag":81,"props":3017,"children":3019},{"className":3018},[],[3020],{"type":56,"value":3021},"Retry-After",{"type":47,"tag":177,"props":3023,"children":3024},{},[3025],{"type":56,"value":3026},"Back off as asked",{"type":47,"tag":150,"props":3028,"children":3029},{},[3030,3039,3043],{"type":47,"tag":177,"props":3031,"children":3032},{},[3033],{"type":47,"tag":81,"props":3034,"children":3036},{"className":3035},[],[3037],{"type":56,"value":3038},"408 Request Timeout",{"type":47,"tag":177,"props":3040,"children":3041},{},[3042],{"type":56,"value":2972},{"type":47,"tag":177,"props":3044,"children":3045},{},[3046],{"type":56,"value":3047},"Transient",{"type":47,"tag":150,"props":3049,"children":3050},{},[3051,3060,3065],{"type":47,"tag":177,"props":3052,"children":3053},{},[3054],{"type":47,"tag":81,"props":3055,"children":3057},{"className":3056},[],[3058],{"type":56,"value":3059},"3xx",{"type":47,"tag":177,"props":3061,"children":3062},{},[3063],{"type":56,"value":3064},"No; fix the URL",{"type":47,"tag":177,"props":3066,"children":3067},{},[3068],{"type":56,"value":3069},"Shopify counts a redirect as a failure, and most senders do not follow one",{"type":47,"tag":150,"props":3071,"children":3072},{},[3073,3089,3094],{"type":47,"tag":177,"props":3074,"children":3075},{},[3076,3082,3083],{"type":47,"tag":81,"props":3077,"children":3079},{"className":3078},[],[3080],{"type":56,"value":3081},"400",{"type":56,"value":1128},{"type":47,"tag":81,"props":3084,"children":3086},{"className":3085},[],[3087],{"type":56,"value":3088},"422",{"type":47,"tag":177,"props":3090,"children":3091},{},[3092],{"type":56,"value":3093},"No; dead-letter",{"type":47,"tag":177,"props":3095,"children":3096},{},[3097],{"type":56,"value":3098},"The payload won't change between attempts",{"type":47,"tag":150,"props":3100,"children":3101},{},[3102,3117,3122],{"type":47,"tag":177,"props":3103,"children":3104},{},[3105,3110,3111],{"type":47,"tag":81,"props":3106,"children":3108},{"className":3107},[],[3109],{"type":56,"value":1199},{"type":56,"value":1128},{"type":47,"tag":81,"props":3112,"children":3114},{"className":3113},[],[3115],{"type":56,"value":3116},"403",{"type":47,"tag":177,"props":3118,"children":3119},{},[3120],{"type":56,"value":3121},"No; dead-letter and alert",{"type":47,"tag":177,"props":3123,"children":3124},{},[3125],{"type":56,"value":3126},"A credential or permission is broken",{"type":47,"tag":150,"props":3128,"children":3129},{},[3130,3146,3151],{"type":47,"tag":177,"props":3131,"children":3132},{},[3133,3139,3140],{"type":47,"tag":81,"props":3134,"children":3136},{"className":3135},[],[3137],{"type":56,"value":3138},"404",{"type":56,"value":1128},{"type":47,"tag":81,"props":3141,"children":3143},{"className":3142},[],[3144],{"type":56,"value":3145},"410 Gone",{"type":47,"tag":177,"props":3147,"children":3148},{},[3149],{"type":56,"value":3150},"No; alert",{"type":47,"tag":177,"props":3152,"children":3153},{},[3154],{"type":56,"value":3155},"The endpoint moved or was removed on purpose",{"type":47,"tag":1030,"props":3157,"children":3159},{"id":3158},"_5-dead-letter-park-it-fix-it-replay-it",[3160],{"type":56,"value":3161},"5. Dead-letter: park it, fix it, replay it",{"type":47,"tag":48,"props":3163,"children":3164},{},[3165],{"type":56,"value":3166},"A dead-letter queue keeps events that exhausted their retries, with the payload and the last error, so they stop clogging the main queue without being thrown away. The operational loop is: an alert fires, someone reads the last error, ships a fix, and replays the dead-lettered events through the normal handler. Replay is safe because stage 3 is idempotent. Without a replay path, a dead-letter queue is just a slower way to lose data.",{"type":47,"tag":130,"props":3168,"children":3170},{"id":3169},"around-the-pipeline-monitoring-and-reconciliation",[3171],{"type":56,"value":3172},"Around the pipeline: monitoring and reconciliation",{"type":47,"tag":48,"props":3174,"children":3175},{},[3176],{"type":56,"value":3177},"The five stages handle events that arrive. Two more jobs handle the ones that don't, and the ones that fail quietly.",{"type":47,"tag":48,"props":3179,"children":3180},{},[3181,3186,3188,3194],{"type":47,"tag":52,"props":3182,"children":3183},{},[3184],{"type":56,"value":3185},"Monitoring.",{"type":56,"value":3187}," Track success rate, latency and retry backlog per source and destination, and alert on the first failed delivery or on an error-rate threshold. Add a silence alert: if a source that normally sends every few minutes has sent nothing for an hour, something upstream is broken, and no failure counter will tell you. ",{"type":47,"tag":342,"props":3189,"children":3191},{"href":3190},"/blog/webhook-monitoring/",[3192],{"type":56,"value":3193},"Webhook monitoring",{"type":56,"value":3195}," covers which metrics to track and how to set the thresholds.",{"type":47,"tag":48,"props":3197,"children":3198},{},[3199,3204,3206,3212,3214,3219],{"type":47,"tag":52,"props":3200,"children":3201},{},[3202],{"type":56,"value":3203},"Reconciliation.",{"type":56,"value":3205}," On a schedule, ask the provider's API for objects that changed since the last run and process any you have not seen. Stripe's ",{"type":47,"tag":81,"props":3207,"children":3209},{"className":3208},[],[3210],{"type":56,"value":3211},"GET /v1/events",{"type":56,"value":3213}," lists events for 30 days. Shopify's Admin API can query orders and products by ",{"type":47,"tag":81,"props":3215,"children":3217},{"className":3216},[],[3218],{"type":56,"value":1439},{"type":56,"value":3220},". GitHub's API lists recent deliveries per webhook, so you can redeliver failed ones. Reconciliation turns \"the provider never sent it\" from data loss into a delay.",{"type":47,"tag":130,"props":3222,"children":3224},{"id":3223},"see-durable-retries-converge",[3225],{"type":56,"value":3226},"See durable retries converge",{"type":47,"tag":48,"props":3228,"children":3229},{},[3230,3232,3239,3241,3246,3248,3253],{"type":56,"value":3231},"The chart below is a real run: 50 webhooks sent through Webhook Relay to ",{"type":47,"tag":342,"props":3233,"children":3236},{"href":3234,"rel":3235},"https://github.com/webhookrelay/flakey-script",[346],[3237],{"type":56,"value":3238},"flakey-script",{"type":56,"value":3240},", an open-source endpoint that rejects about 80% of requests until an event is an hour old, with durable retries on the medium schedule. Fast retries run for the first 15 minutes, then durable retry takes over; the endpoint starts accepting old events at 60 minutes, and the last webhook lands at 73 minutes. Press ",{"type":47,"tag":52,"props":3242,"children":3243},{},[3244],{"type":56,"value":3245},"play",{"type":56,"value":3247}," to watch, or switch to ",{"type":47,"tag":52,"props":3249,"children":3250},{},[3251],{"type":56,"value":3252},"Retry attempts",{"type":56,"value":3254}," to see the 275 attempts behind the curve.",{"type":47,"tag":3256,"props":3257,"children":3258},"delivery-chart",{},[],{"type":47,"tag":48,"props":3260,"children":3261},{},[3262],{"type":56,"value":3263},"The result: 50 of 50 webhooks delivered, the last at 73 minutes, after 275 attempts. With only a provider's own retries, anything still failing after Shopify's 4-hour window, or immediately for GitHub, would have been dropped.",{"type":47,"tag":130,"props":3265,"children":3267},{"id":3266},"build-it-or-use-a-gateway",[3268],{"type":56,"value":3269},"Build it or use a gateway",{"type":47,"tag":48,"props":3271,"children":3272},{},[3273],{"type":56,"value":3274},"There are three ways to get the pipeline. They are not exclusive: most teams that use a gateway still keep idempotent handlers and a reconciliation job.",{"type":47,"tag":142,"props":3276,"children":3277},{},[3278,3302],{"type":47,"tag":146,"props":3279,"children":3280},{},[3281],{"type":47,"tag":150,"props":3282,"children":3283},{},[3284,3287,3292,3297],{"type":47,"tag":154,"props":3285,"children":3286},{},[],{"type":47,"tag":154,"props":3288,"children":3289},{},[3290],{"type":56,"value":3291},"Build it yourself",{"type":47,"tag":154,"props":3293,"children":3294},{},[3295],{"type":56,"value":3296},"Provider-native queues",{"type":47,"tag":154,"props":3298,"children":3299},{},[3300],{"type":56,"value":3301},"Managed webhook gateway",{"type":47,"tag":170,"props":3303,"children":3304},{},[3305,3328,3351,3372,3395,3418,3441],{"type":47,"tag":150,"props":3306,"children":3307},{},[3308,3313,3318,3323],{"type":47,"tag":177,"props":3309,"children":3310},{},[3311],{"type":56,"value":3312},"What it is",{"type":47,"tag":177,"props":3314,"children":3315},{},[3316],{"type":56,"value":3317},"Your endpoint + a queue (SQS, Pub/Sub, a table) + workers + DLQ",{"type":47,"tag":177,"props":3319,"children":3320},{},[3321],{"type":56,"value":3322},"Provider delivers straight into your cloud queue (Shopify → Pub/Sub or EventBridge, Stripe → EventBridge)",{"type":47,"tag":177,"props":3324,"children":3325},{},[3326],{"type":56,"value":3327},"A hosted endpoint that stores, retries, logs and forwards (Webhook Relay, Hookdeck)",{"type":47,"tag":150,"props":3329,"children":3330},{},[3331,3336,3341,3346],{"type":47,"tag":177,"props":3332,"children":3333},{},[3334],{"type":56,"value":3335},"Retry window",{"type":47,"tag":177,"props":3337,"children":3338},{},[3339],{"type":56,"value":3340},"Whatever you build",{"type":47,"tag":177,"props":3342,"children":3343},{},[3344],{"type":56,"value":3345},"Your queue's retention",{"type":47,"tag":177,"props":3347,"children":3348},{},[3349],{"type":56,"value":3350},"Configurable; up to 30 days on Webhook Relay",{"type":47,"tag":150,"props":3352,"children":3353},{},[3354,3359,3363,3368],{"type":47,"tag":177,"props":3355,"children":3356},{},[3357],{"type":56,"value":3358},"Works for every provider",{"type":47,"tag":177,"props":3360,"children":3361},{},[3362],{"type":56,"value":2972},{"type":47,"tag":177,"props":3364,"children":3365},{},[3366],{"type":56,"value":3367},"Only providers that support it",{"type":47,"tag":177,"props":3369,"children":3370},{},[3371],{"type":56,"value":2972},{"type":47,"tag":150,"props":3373,"children":3374},{},[3375,3380,3385,3390],{"type":47,"tag":177,"props":3376,"children":3377},{},[3378],{"type":56,"value":3379},"Logs, alerting, replay",{"type":47,"tag":177,"props":3381,"children":3382},{},[3383],{"type":56,"value":3384},"You build them",{"type":47,"tag":177,"props":3386,"children":3387},{},[3388],{"type":56,"value":3389},"Your cloud's tooling",{"type":47,"tag":177,"props":3391,"children":3392},{},[3393],{"type":56,"value":3394},"Included",{"type":47,"tag":150,"props":3396,"children":3397},{},[3398,3403,3408,3413],{"type":47,"tag":177,"props":3399,"children":3400},{},[3401],{"type":56,"value":3402},"Private-network receivers",{"type":47,"tag":177,"props":3404,"children":3405},{},[3406],{"type":56,"value":3407},"Needs inbound access or a tunnel",{"type":47,"tag":177,"props":3409,"children":3410},{},[3411],{"type":56,"value":3412},"Yes, via the queue",{"type":47,"tag":177,"props":3414,"children":3415},{},[3416],{"type":56,"value":3417},"Webhook Relay delivers through an outbound agent",{"type":47,"tag":150,"props":3419,"children":3420},{},[3421,3426,3431,3436],{"type":47,"tag":177,"props":3422,"children":3423},{},[3424],{"type":56,"value":3425},"Operational cost",{"type":47,"tag":177,"props":3427,"children":3428},{},[3429],{"type":56,"value":3430},"Highest",{"type":47,"tag":177,"props":3432,"children":3433},{},[3434],{"type":56,"value":3435},"Medium",{"type":47,"tag":177,"props":3437,"children":3438},{},[3439],{"type":56,"value":3440},"Lowest",{"type":47,"tag":150,"props":3442,"children":3443},{},[3444,3449,3453,3458],{"type":47,"tag":177,"props":3445,"children":3446},{},[3447],{"type":56,"value":3448},"Lock-in",{"type":47,"tag":177,"props":3450,"children":3451},{},[3452],{"type":56,"value":427},{"type":47,"tag":177,"props":3454,"children":3455},{},[3456],{"type":56,"value":3457},"Per provider and cloud",{"type":47,"tag":177,"props":3459,"children":3460},{},[3461],{"type":56,"value":3462},"Per gateway; endpoints are plain HTTPS",{"type":47,"tag":48,"props":3464,"children":3465},{},[3466,3468,3474,3476,3482,3484,3490,3492,3498,3500,3506,3508,3514],{"type":56,"value":3467},"Webhook Relay sits in the third column. A ",{"type":47,"tag":342,"props":3469,"children":3471},{"href":3470},"/webhooks/",[3472],{"type":56,"value":3473},"bucket",{"type":56,"value":3475}," gives each provider a stable HTTPS URL that answers in milliseconds and stores the request, then ",{"type":47,"tag":342,"props":3477,"children":3479},{"href":3478},"/features/durable-retries/",[3480],{"type":56,"value":3481},"durable retries",{"type":56,"value":3483}," deliver it with exponential backoff for up to 30 days. ",{"type":47,"tag":342,"props":3485,"children":3487},{"href":3486},"/features/alerts/",[3488],{"type":56,"value":3489},"Alerts",{"type":56,"value":3491}," open an incident on the first failed delivery, ",{"type":47,"tag":342,"props":3493,"children":3495},{"href":3494},"/features/webhook-logs/",[3496],{"type":56,"value":3497},"request logs",{"type":56,"value":3499}," keep every attempt, and one click resends a delivery or recovers every failure after a fix. The relay agent delivers into ",{"type":47,"tag":342,"props":3501,"children":3503},{"href":3502},"/features/webhook-to-internal-server/",[3504],{"type":56,"value":3505},"localhost, private networks and Kubernetes",{"type":56,"value":3507}," without inbound firewall rules, and ",{"type":47,"tag":342,"props":3509,"children":3511},{"href":3510},"/features/throttling/",[3512],{"type":56,"value":3513},"throttling",{"type":56,"value":3515}," stops a backlog from flattening a recovering service.",{"type":47,"tag":3517,"props":3518,"children":3524},"inline-cta",{"label":3519,"secondary-label":3520,"secondary-to":3521,"title":3522,"to":3523},"Create a free bucket","Read the durable retries docs","/docs/webhooks/durable-webhooks/","Put a durable buffer in front of every webhook","https://my.webhookrelay.com/register",[3525],{"type":47,"tag":48,"props":3526,"children":3527},{},[3528],{"type":56,"value":3529},"Point any provider at a bucket, turn on durable delivery and add an alert. It takes a few minutes and works with Stripe, Shopify, GitHub and anything else that sends HTTP.",{"type":47,"tag":130,"props":3531,"children":3533},{"id":3532},"if-you-send-webhooks",[3534],{"type":56,"value":3535},"If you send webhooks",{"type":47,"tag":48,"props":3537,"children":3538},{},[3539,3541,3547,3549,3553,3555,3561,3562,3568],{"type":56,"value":3540},"Everything above is the receiving side. If your product sends webhooks to customers, the same rules apply in reverse: sign every message, retry with backoff for a published window, give each customer endpoint its own rate limit so one slow customer doesn't delay the rest, keep a delivery log your support team can search, and let customers replay. Webhook Relay's ",{"type":47,"tag":342,"props":3542,"children":3544},{"href":3543},"/outbound-webhooks/",[3545],{"type":56,"value":3546},"Outbound Gateway",{"type":56,"value":3548}," does that with ",{"type":47,"tag":342,"props":3550,"children":3551},{"href":1411},[3552],{"type":56,"value":1414},{"type":56,"value":3554}," signatures and 35 days of history on every plan; ",{"type":47,"tag":342,"props":3556,"children":3558},{"href":3557},"/blog/svix-alternative/",[3559],{"type":56,"value":3560},"Svix",{"type":56,"value":952},{"type":47,"tag":342,"props":3563,"children":3565},{"href":3564},"/blog/hookdeck-alternative/",[3566],{"type":56,"value":3567},"Hookdeck Outpost",{"type":56,"value":3569}," are the main alternatives.",{"type":47,"tag":130,"props":3571,"children":3573},{"id":3572},"next-steps",[3574],{"type":56,"value":3575},"Next steps",{"type":47,"tag":654,"props":3577,"children":3578},{},[3579,3592,3603,3614],{"type":47,"tag":70,"props":3580,"children":3581},{},[3582,3584,3590],{"type":56,"value":3583},"Apply it to one provider: ",{"type":47,"tag":342,"props":3585,"children":3587},{"href":3586},"/blog/shopify-webhooks/",[3588],{"type":56,"value":3589},"Shopify webhooks",{"type":56,"value":3591}," covers the 5-second timeout, 8 retries in 4 hours and deleted subscriptions.",{"type":47,"tag":70,"props":3593,"children":3594},{},[3595,3597,3602],{"type":56,"value":3596},"Decide what to alert on with ",{"type":47,"tag":342,"props":3598,"children":3599},{"href":3190},[3600],{"type":56,"value":3601},"webhook monitoring",{"type":56,"value":1423},{"type":47,"tag":70,"props":3604,"children":3605},{},[3606,3608,3613],{"type":56,"value":3607},"Go deeper on duplicates in ",{"type":47,"tag":342,"props":3609,"children":3610},{"href":1447},[3611],{"type":56,"value":3612},"webhook retries and idempotency",{"type":56,"value":1423},{"type":47,"tag":70,"props":3615,"children":3616},{},[3617,3619,3623],{"type":56,"value":3618},"Turn on ",{"type":47,"tag":342,"props":3620,"children":3621},{"href":3521},[3622],{"type":56,"value":3481},{"type":56,"value":3624}," for a destination.",{"type":47,"tag":130,"props":3626,"children":3628},{"id":3627},"reliability-checklist",[3629],{"type":56,"value":3630},"Reliability checklist",{"type":47,"tag":48,"props":3632,"children":3633},{},[3634,3636,3642],{"type":56,"value":3635},"This list covers the pipeline. For what to watch and alert on, use the ",{"type":47,"tag":342,"props":3637,"children":3639},{"href":3638},"/blog/webhook-monitoring/#monitoring-checklist",[3640],{"type":56,"value":3641},"monitoring checklist",{"type":56,"value":1423},{"type":47,"tag":654,"props":3644,"children":3647},{"className":3645},[3646],"contains-task-list",[3648,3660,3682,3691,3700,3709,3718,3727,3736,3745],{"type":47,"tag":70,"props":3649,"children":3652},{"className":3650},[3651],"task-list-item",[3653,3658],{"type":47,"tag":3654,"props":3655,"children":3657},"input",{"disabled":1211,"type":3656},"checkbox",[],{"type":56,"value":3659}," The endpoint answers within the strictest provider timeout you receive from (3 s for Slack).",{"type":47,"tag":70,"props":3661,"children":3663},{"className":3662},[3651],[3664,3667,3669,3674,3676,3681],{"type":47,"tag":3654,"props":3665,"children":3666},{"disabled":1211,"type":3656},[],{"type":56,"value":3668}," The raw event is stored before ",{"type":47,"tag":81,"props":3670,"children":3672},{"className":3671},[],[3673],{"type":56,"value":86},{"type":56,"value":3675}," is returned; storage failure returns ",{"type":47,"tag":81,"props":3677,"children":3679},{"className":3678},[],[3680],{"type":56,"value":1352},{"type":56,"value":1423},{"type":47,"tag":70,"props":3683,"children":3685},{"className":3684},[3651],[3686,3689],{"type":47,"tag":3654,"props":3687,"children":3688},{"disabled":1211,"type":3656},[],{"type":56,"value":3690}," Every event has a unique constraint on provider + event id.",{"type":47,"tag":70,"props":3692,"children":3694},{"className":3693},[3651],[3695,3698],{"type":47,"tag":3654,"props":3696,"children":3697},{"disabled":1211,"type":3656},[],{"type":56,"value":3699}," Handlers ignore payloads older than the stored object.",{"type":47,"tag":70,"props":3701,"children":3703},{"className":3702},[3651],[3704,3707],{"type":47,"tag":3654,"props":3705,"children":3706},{"disabled":1211,"type":3656},[],{"type":56,"value":3708}," Processing failures retry from storage with backoff, jitter and a deadline.",{"type":47,"tag":70,"props":3710,"children":3712},{"className":3711},[3651],[3713,3716],{"type":47,"tag":3654,"props":3714,"children":3715},{"disabled":1211,"type":3656},[],{"type":56,"value":3717}," Permanent failures skip retries and go to the dead-letter queue.",{"type":47,"tag":70,"props":3719,"children":3721},{"className":3720},[3651],[3722,3725],{"type":47,"tag":3654,"props":3723,"children":3724},{"disabled":1211,"type":3656},[],{"type":56,"value":3726}," Dead-lettered events can be replayed through the normal handler.",{"type":47,"tag":70,"props":3728,"children":3730},{"className":3729},[3651],[3731,3734],{"type":47,"tag":3654,"props":3732,"children":3733},{"disabled":1211,"type":3656},[],{"type":56,"value":3735}," Alerts fire on failed deliveries and on silence from busy sources.",{"type":47,"tag":70,"props":3737,"children":3739},{"className":3738},[3651],[3740,3743],{"type":47,"tag":3654,"props":3741,"children":3742},{"disabled":1211,"type":3656},[],{"type":56,"value":3744}," A reconciliation job runs on a schedule for every provider that has a list API.",{"type":47,"tag":70,"props":3746,"children":3748},{"className":3747},[3651],[3749,3752],{"type":47,"tag":3654,"props":3750,"children":3751},{"disabled":1211,"type":3656},[],{"type":56,"value":3753}," Deploys never take the receiving endpoint down for longer than the shortest retry window, or a buffer sits in front of it.",{"type":47,"tag":3755,"props":3756,"children":3757},"style",{},[3758],{"type":56,"value":3759},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}",{"title":7,"searchDepth":1207,"depth":1207,"links":3761},[3762,3763,3764,3765,3772,3773,3774,3775,3776,3777],{"id":132,"depth":765,"text":135},{"id":294,"depth":765,"text":297},{"id":501,"depth":765,"text":504},{"id":983,"depth":765,"text":986,"children":3766},[3767,3768,3769,3770,3771],{"id":1032,"depth":1207,"text":1035},{"id":1357,"depth":1207,"text":1360},{"id":1384,"depth":1207,"text":1387},{"id":1455,"depth":1207,"text":1458},{"id":3158,"depth":1207,"text":3161},{"id":3169,"depth":765,"text":3172},{"id":3223,"depth":765,"text":3226},{"id":3266,"depth":765,"text":3269},{"id":3532,"depth":765,"text":3535},{"id":3572,"depth":765,"text":3575},{"id":3627,"depth":765,"text":3630},"markdown","content:blog:reliable-webhook-delivery.md","content","blog/reliable-webhook-delivery.md","blog/reliable-webhook-delivery","md",{"loc":4},1794413980500]