[{"data":1,"prerenderedAt":3008},["ShallowReactive",2],{"content-query-cXxPnZ9NvG":3},{"_path":4,"_dir":5,"_draft":6,"_partial":6,"_locale":7,"title":8,"description":9,"layout":10,"head":11,"author":13,"tags":14,"categories":19,"date":21,"faq":22,"image":41,"body":42,"_type":3001,"_id":3002,"_source":3003,"_file":3004,"_stem":3005,"_extension":3006,"sitemap":3007,"readingTime":965},"/blog/webhook-monitoring","blog",false,"","Webhook Monitoring: What to Track and How to Alert on Failures","Webhook monitoring in practice: the six signals to track (success rate, failures, latency, retry backlog, dead letters, silence), alert thresholds that don't page you for nothing, and where Stripe, Shopify and GitHub stop helping.","post",{"title":12},"Webhook Monitoring: Metrics, Alerts & Tools","Karolis Rusenas",[15,16,17,18],"webhook monitoring","webhook alerts","observability","reliability",[20],"guides","2026-11-10 13:00:00",[23,26,29,32,35,38],{"q":24,"a":25},"What is webhook monitoring?","Webhook monitoring is tracking whether webhooks you receive or send are delivered and processed: success rate, failed deliveries, response latency, retry backlog, dead-lettered events and unexpected silence, with alerts when any of them crosses a threshold. It answers the question every webhook integration eventually gets asked: did that event arrive, and if not, why?",{"q":27,"a":28},"What should I alert on for webhooks?","Alert on failed deliveries to a destination (the first failure for critical flows, an error-rate threshold for noisy ones), on a growing retry backlog or any dead-lettered event, on p95 response time approaching the provider's timeout, and on silence: no events for longer than normal from a source that usually sends regularly. Resolve alerts automatically once deliveries recover.",{"q":30,"a":31},"How do I know if a webhook was not delivered?","Check three places: the provider's delivery log (Stripe Dashboard, Shopify Dev Dashboard, GitHub Recent Deliveries) for failed attempts, your own request log for what actually arrived, and the provider's API for objects that changed without a matching event. Silence alerts and a scheduled reconciliation job catch the case where the provider never sent anything.",{"q":33,"a":34},"Does Stripe or Shopify alert me when my webhook endpoint fails?","Partly. Shopify emails the app's emergency developer address when deliveries to an app's subscription keep failing, and its Dev Dashboard flags failure rates above 0.5% but keeps only seven days of logs. Webhooks a merchant creates in the Shopify admin trigger no email. Stripe shows failed deliveries per event in its Dashboard. Neither tells you about events that were never sent, and none of them page your on-call.",{"q":36,"a":37},"What is the difference between webhook monitoring and webhook observability?","Monitoring answers whether webhooks are being delivered: success rate, failures, latency, backlog and silence, with alerts when they cross a threshold. Observability answers why one particular delivery failed: the full request and response, every attempt and its timing, searchable after the fact. You need monitoring to know something is wrong and an observable delivery log to fix it.",{"q":39,"a":40},"What is a good webhook success rate?","For a healthy endpoint, effectively 100% of deliveries should eventually succeed, and first-attempt success should stay above 99%. Shopify flags apps whose failure rate goes above 0.5%. A first-attempt rate that drops while the eventual rate stays at 100% means retries are hiding a problem worth fixing before it outlasts the retry window.","/images/blog/heroes/route.jpg",{"type":43,"children":44,"toc":2988},"root",[45,68,178,185,359,387,400,424,441,454,461,481,487,500,506,679,688,700,706,711,1755,1760,2204,2237,2243,2248,2292,2595,2608,2636,2642,2647,2690,2718,2724,2737,2795,2808,2822,2828,2870,2876,2888,2982],{"type":46,"tag":47,"props":48,"children":49},"element","p",{},[50,57,59,66],{"type":46,"tag":51,"props":52,"children":53},"strong",{},[54],{"type":55,"value":56},"text","Webhook monitoring is watching whether the webhooks you receive (or send) are delivered and processed, and getting told when they aren't.",{"type":55,"value":58}," The failure mode it exists for is silence: a provider gets a ",{"type":46,"tag":60,"props":61,"children":63},"code",{"className":62},[],[64],{"type":55,"value":65},"500",{"type":55,"value":67},", retries for a while, gives up, and nothing on your side notices until a customer asks where their order went. Good webhook monitoring tracks six signals per source and destination, alerts on a few of them, and resolves itself when deliveries recover.",{"type":46,"tag":69,"props":70,"children":72},"callout",{"label":71},"The short version",[73],{"type":46,"tag":74,"props":75,"children":76},"ol",{},[77,125,144,155,166],{"type":46,"tag":78,"props":79,"children":80},"li",{},[81,83,88,90,95,97,102,104,109,111,116,118,123],{"type":55,"value":82},"Track six signals per source and destination: ",{"type":46,"tag":51,"props":84,"children":85},{},[86],{"type":55,"value":87},"success rate",{"type":55,"value":89}," (alert below ~99% first-attempt), ",{"type":46,"tag":51,"props":91,"children":92},{},[93],{"type":55,"value":94},"failed deliveries",{"type":55,"value":96},", ",{"type":46,"tag":51,"props":98,"children":99},{},[100],{"type":55,"value":101},"p95 latency",{"type":55,"value":103}," (alert at half the provider's timeout), ",{"type":46,"tag":51,"props":105,"children":106},{},[107],{"type":55,"value":108},"retry backlog",{"type":55,"value":110}," (alert while it grows), ",{"type":46,"tag":51,"props":112,"children":113},{},[114],{"type":55,"value":115},"dead letters",{"type":55,"value":117}," (alert on any) and ",{"type":46,"tag":51,"props":119,"children":120},{},[121],{"type":55,"value":122},"silence",{"type":55,"value":124}," (alert past the source's normal gap).",{"type":46,"tag":78,"props":126,"children":127},{},[128,130,135,137,142],{"type":55,"value":129},"Alert on the ",{"type":46,"tag":51,"props":131,"children":132},{},[133],{"type":55,"value":134},"first failure",{"type":55,"value":136}," for money-moving flows and on an ",{"type":46,"tag":51,"props":138,"children":139},{},[140],{"type":55,"value":141},"error-rate threshold",{"type":55,"value":143}," elsewhere.",{"type":46,"tag":78,"props":145,"children":146},{},[147,149,153],{"type":55,"value":148},"Alert when ",{"type":46,"tag":51,"props":150,"children":151},{},[152],{"type":55,"value":101},{"type":55,"value":154}," gets close to the provider's timeout: that is a failure about to happen.",{"type":46,"tag":78,"props":156,"children":157},{},[158,160,164],{"type":55,"value":159},"Alert on ",{"type":46,"tag":51,"props":161,"children":162},{},[163],{"type":55,"value":122},{"type":55,"value":165}," from sources that normally send. No error counter catches a provider that stopped sending.",{"type":46,"tag":78,"props":167,"children":168},{},[169,171,176],{"type":55,"value":170},"Keep your own ",{"type":46,"tag":51,"props":172,"children":173},{},[174],{"type":55,"value":175},"request log",{"type":55,"value":177},". Provider dashboards keep days, not weeks, and only show their side.",{"type":46,"tag":179,"props":180,"children":182},"h2",{"id":181},"the-six-signals-to-track",[183],{"type":55,"value":184},"The six signals to track",{"type":46,"tag":186,"props":187,"children":188},"table",{},[189,213],{"type":46,"tag":190,"props":191,"children":192},"thead",{},[193],{"type":46,"tag":194,"props":195,"children":196},"tr",{},[197,203,208],{"type":46,"tag":198,"props":199,"children":200},"th",{},[201],{"type":55,"value":202},"Signal",{"type":46,"tag":198,"props":204,"children":205},{},[206],{"type":55,"value":207},"What it tells you",{"type":46,"tag":198,"props":209,"children":210},{},[211],{"type":55,"value":212},"Alert when",{"type":46,"tag":214,"props":215,"children":216},"tbody",{},[217,247,275,296,317,338],{"type":46,"tag":194,"props":218,"children":219},{},[220,229,242],{"type":46,"tag":221,"props":222,"children":223},"td",{},[224],{"type":46,"tag":51,"props":225,"children":226},{},[227],{"type":55,"value":228},"Delivery success rate",{"type":46,"tag":221,"props":230,"children":231},{},[232,234,240],{"type":55,"value":233},"Share of deliveries your endpoint accepted with a ",{"type":46,"tag":60,"props":235,"children":237},{"className":236},[],[238],{"type":55,"value":239},"2xx",{"type":55,"value":241},", on the first attempt and eventually",{"type":46,"tag":221,"props":243,"children":244},{},[245],{"type":55,"value":246},"First-attempt rate drops below ~99%, or any event fails for good",{"type":46,"tag":194,"props":248,"children":249},{},[250,258,270],{"type":46,"tag":221,"props":251,"children":252},{},[253],{"type":46,"tag":51,"props":254,"children":255},{},[256],{"type":55,"value":257},"Failed deliveries",{"type":46,"tag":221,"props":259,"children":260},{},[261,263,268],{"type":55,"value":262},"Count of non-",{"type":46,"tag":60,"props":264,"children":266},{"className":265},[],[267],{"type":55,"value":239},{"type":55,"value":269}," responses, timeouts and connection errors",{"type":46,"tag":221,"props":271,"children":272},{},[273],{"type":55,"value":274},"First failure on critical flows (payments, orders); error rate above a threshold elsewhere",{"type":46,"tag":194,"props":276,"children":277},{},[278,286,291],{"type":46,"tag":221,"props":279,"children":280},{},[281],{"type":46,"tag":51,"props":282,"children":283},{},[284],{"type":55,"value":285},"Response latency (p95)",{"type":46,"tag":221,"props":287,"children":288},{},[289],{"type":55,"value":290},"How long your endpoint takes to answer",{"type":46,"tag":221,"props":292,"children":293},{},[294],{"type":55,"value":295},"p95 passes half the provider's timeout (1.5 s for Slack, 2.5 s for Shopify, 5 s for GitHub)",{"type":46,"tag":194,"props":297,"children":298},{},[299,307,312],{"type":46,"tag":221,"props":300,"children":301},{},[302],{"type":46,"tag":51,"props":303,"children":304},{},[305],{"type":55,"value":306},"Retry backlog",{"type":46,"tag":221,"props":308,"children":309},{},[310],{"type":55,"value":311},"Events waiting for another attempt",{"type":46,"tag":221,"props":313,"children":314},{},[315],{"type":55,"value":316},"Backlog grows for more than one retry interval",{"type":46,"tag":194,"props":318,"children":319},{},[320,328,333],{"type":46,"tag":221,"props":321,"children":322},{},[323],{"type":46,"tag":51,"props":324,"children":325},{},[326],{"type":55,"value":327},"Dead letters",{"type":46,"tag":221,"props":329,"children":330},{},[331],{"type":55,"value":332},"Events that exhausted their retries",{"type":46,"tag":221,"props":334,"children":335},{},[336],{"type":55,"value":337},"Any new entry",{"type":46,"tag":194,"props":339,"children":340},{},[341,349,354],{"type":46,"tag":221,"props":342,"children":343},{},[344],{"type":46,"tag":51,"props":345,"children":346},{},[347],{"type":55,"value":348},"Silence",{"type":46,"tag":221,"props":350,"children":351},{},[352],{"type":55,"value":353},"Time since the last event from a source",{"type":46,"tag":221,"props":355,"children":356},{},[357],{"type":55,"value":358},"Longer than the source's normal gap (an hour for a busy store, a day for a nightly job)",{"type":46,"tag":47,"props":360,"children":361},{},[362,364,371,373,378,380,385],{"type":55,"value":363},"The Prometheus rules for all six are in ",{"type":46,"tag":365,"props":366,"children":368},"a",{"href":367},"#build-it-yourself",[369],{"type":55,"value":370},"build it yourself",{"type":55,"value":372}," below. Split every signal by ",{"type":46,"tag":51,"props":374,"children":375},{},[376],{"type":55,"value":377},"source",{"type":55,"value":379}," (Stripe, Shopify, GitHub) and ",{"type":46,"tag":51,"props":381,"children":382},{},[383],{"type":55,"value":384},"destination",{"type":55,"value":386}," (your API, the warehouse loader, a partner). One healthy average hides one broken integration.",{"type":46,"tag":47,"props":388,"children":389},{},[390,392,398],{"type":55,"value":391},"Here is what those signals look like during one real-shaped incident: an endpoint starts returning ",{"type":46,"tag":60,"props":393,"children":395},{"className":394},[],[396],{"type":55,"value":397},"503",{"type":55,"value":399}," after a bad deploy, retries pile up, someone ships a fix, and the backlog drains without losing an event.",{"type":46,"tag":401,"props":402,"children":406},"picture",{"className":403},[404,405],"block","dark:hidden",[407,409,414,415],{"type":55,"value":408},"\n  ",{"type":46,"tag":377,"props":410,"children":413},{"media":411,"srcSet":412},"(max-width: 640px)","/images/blog/delivery/webhook-monitoring-timeline-mobile.svg",[],{"type":55,"value":408},{"type":46,"tag":416,"props":417,"children":423},"img",{"src":418,"width":419,"height":420,"loading":421,"alt":422},"/images/blog/delivery/webhook-monitoring-timeline.svg",1000,680,"lazy","Webhook monitoring timeline for one incident. 14:02 delivery #1 returns 503 and an alert opens in Slack on the first failure. Retries follow a doubling schedule: 30 seconds (timeout, p95 latency jumps to 5 seconds), 1.5 minutes (503, 37 events in the retry backlog), 3.5 minutes (503, 10-minute success rate 61%), 7.5 minutes (502) and 15.5 minutes (502, on-call reads the failing response). 14:20 a fix is deployed. 14:33 retry 6, 31.5 minutes after the first failure, returns 200 and the backlog drains with nothing lost. 14:38 after a clean five-minute window the incident resolves automatically; it was open for 36 minutes.",[],{"type":46,"tag":401,"props":425,"children":429},{"className":426},[427,428],"hidden","dark:block",[430,431,435,436],{"type":55,"value":408},{"type":46,"tag":377,"props":432,"children":434},{"media":411,"srcSet":433},"/images/blog/delivery/webhook-monitoring-timeline-mobile-dark.svg",[],{"type":55,"value":408},{"type":46,"tag":416,"props":437,"children":440},{"src":438,"width":419,"height":420,"loading":421,"alt":439},"/images/blog/delivery/webhook-monitoring-timeline-dark.svg","Webhook monitoring timeline: a 503 opens an alert, retries fail while latency and backlog rise, a fix is deployed, a retry succeeds and the incident resolves automatically after a clean window.",[],{"type":46,"tag":47,"props":442,"children":443},{},[444,446,452],{"type":55,"value":445},"The alert fired on the first failure, the latency, backlog and success-rate signals told on-call how bad it was, and the automatic resolve closed it without anyone touching it. Durable retries are what turned half an hour of failures into zero lost events. The retries follow the doubling schedule from ",{"type":46,"tag":365,"props":447,"children":449},{"href":448},"/blog/reliable-webhook-delivery/#4-retry-backoff-jitter-and-a-deadline",[450],{"type":55,"value":451},"reliable webhook delivery",{"type":55,"value":453},".",{"type":46,"tag":455,"props":456,"children":458},"h3",{"id":457},"why-latency-is-a-failure-signal",[459],{"type":55,"value":460},"Why latency is a failure signal",{"type":46,"tag":47,"props":462,"children":463},{},[464,466,472,474,479],{"type":55,"value":465},"Providers measure success by time, not just by status code. A handler that returns ",{"type":46,"tag":60,"props":467,"children":469},{"className":468},[],[470],{"type":55,"value":471},"200",{"type":55,"value":473}," after six seconds has succeeded in your logs and failed in Shopify's, which allows five. Slack allows three; GitHub ten. When p95 response time creeps toward those limits, you are hours away from timeouts that you will only see as retries and duplicates. The fix is structural (store the event, acknowledge, process later; see ",{"type":46,"tag":365,"props":475,"children":477},{"href":476},"/blog/reliable-webhook-delivery/",[478],{"type":55,"value":451},{"type":55,"value":480},"), but the alert is what tells you to do it.",{"type":46,"tag":455,"props":482,"children":484},{"id":483},"why-silence-needs-its-own-alert",[485],{"type":55,"value":486},"Why silence needs its own alert",{"type":46,"tag":47,"props":488,"children":489},{},[490,492,498],{"type":55,"value":491},"Every other signal counts something that happened. Silence is the absence of events, and it covers the failures that never reach your endpoint at all: a subscription that Shopify deleted after repeated failures, a Slack app whose event subscriptions were disabled, an expired token on the sender's side, a filter that matches nothing, a DNS change. Base the threshold on the source's normal rhythm: a store with hundreds of orders a day should not go an hour without ",{"type":46,"tag":60,"props":493,"children":495},{"className":494},[],[496],{"type":55,"value":497},"orders/create",{"type":55,"value":499},"; a nightly export should arrive once a day.",{"type":46,"tag":179,"props":501,"children":503},{"id":502},"where-stripe-shopify-and-github-stop-helping",[504],{"type":55,"value":505},"Where Stripe, Shopify and GitHub stop helping",{"type":46,"tag":186,"props":507,"children":508},{},[509,535],{"type":46,"tag":190,"props":510,"children":511},{},[512],{"type":46,"tag":194,"props":513,"children":514},{},[515,520,525,530],{"type":46,"tag":198,"props":516,"children":517},{},[518],{"type":55,"value":519},"Provider",{"type":46,"tag":198,"props":521,"children":522},{},[523],{"type":55,"value":524},"Delivery log",{"type":46,"tag":198,"props":526,"children":527},{},[528],{"type":55,"value":529},"How far back you can see or resend",{"type":46,"tag":198,"props":531,"children":532},{},[533],{"type":55,"value":534},"Alerts you get",{"type":46,"tag":214,"props":536,"children":537},{},[538,570,601,632],{"type":46,"tag":194,"props":539,"children":540},{},[541,555,560,565],{"type":46,"tag":221,"props":542,"children":543},{},[544],{"type":46,"tag":365,"props":545,"children":549},{"href":546,"rel":547},"https://docs.stripe.com/webhooks",[548],"nofollow",[550],{"type":46,"tag":51,"props":551,"children":552},{},[553],{"type":55,"value":554},"Stripe",{"type":46,"tag":221,"props":556,"children":557},{},[558],{"type":55,"value":559},"Per event and per endpoint in the Dashboard",{"type":46,"tag":221,"props":561,"children":562},{},[563],{"type":55,"value":564},"Resend from the Dashboard up to 15 days, CLI up to 30",{"type":46,"tag":221,"props":566,"children":567},{},[568],{"type":55,"value":569},"Not stated in Stripe's webhook docs; retries stop after 3 days",{"type":46,"tag":194,"props":571,"children":572},{},[573,586,591,596],{"type":46,"tag":221,"props":574,"children":575},{},[576],{"type":46,"tag":365,"props":577,"children":580},{"href":578,"rel":579},"https://shopify.dev/docs/apps/build/webhooks/troubleshoot",[548],[581],{"type":46,"tag":51,"props":582,"children":583},{},[584],{"type":55,"value":585},"Shopify",{"type":46,"tag":221,"props":587,"children":588},{},[589],{"type":55,"value":590},"Dev Dashboard: deliveries, failure rate, p90 response time",{"type":46,"tag":221,"props":592,"children":593},{},[594],{"type":55,"value":595},"7 days",{"type":46,"tag":221,"props":597,"children":598},{},[599],{"type":55,"value":600},"Emergency developer email for app subscriptions; flags failure rates above 0.5%",{"type":46,"tag":194,"props":602,"children":603},{},[604,617,622,627],{"type":46,"tag":221,"props":605,"children":606},{},[607],{"type":46,"tag":365,"props":608,"children":611},{"href":609,"rel":610},"https://docs.github.com/en/webhooks/using-webhooks/handling-failed-webhook-deliveries",[548],[612],{"type":46,"tag":51,"props":613,"children":614},{},[615],{"type":55,"value":616},"GitHub",{"type":46,"tag":221,"props":618,"children":619},{},[620],{"type":55,"value":621},"Recent Deliveries per webhook, with request and response",{"type":46,"tag":221,"props":623,"children":624},{},[625],{"type":55,"value":626},"3 days",{"type":46,"tag":221,"props":628,"children":629},{},[630],{"type":55,"value":631},"None documented; there are no automatic retries either",{"type":46,"tag":194,"props":633,"children":634},{},[635,648,669,674],{"type":46,"tag":221,"props":636,"children":637},{},[638],{"type":46,"tag":365,"props":639,"children":642},{"href":640,"rel":641},"https://docs.slack.dev/apis/events-api/",[548],[643],{"type":46,"tag":51,"props":644,"children":645},{},[646],{"type":55,"value":647},"Slack",{"type":46,"tag":221,"props":649,"children":650},{},[651,653,659,661,667],{"type":55,"value":652},"None documented; retries carry ",{"type":46,"tag":60,"props":654,"children":656},{"className":655},[],[657],{"type":55,"value":658},"x-slack-retry-num",{"type":55,"value":660}," and ",{"type":46,"tag":60,"props":662,"children":664},{"className":663},[],[665],{"type":55,"value":666},"x-slack-retry-reason",{"type":55,"value":668}," headers",{"type":46,"tag":221,"props":670,"children":671},{},[672],{"type":55,"value":673},"—",{"type":46,"tag":221,"props":675,"children":676},{},[677],{"type":55,"value":678},"Event subscriptions disabled above 95% failures in 60 minutes (apps with 1,000+ events an hour)",{"type":46,"tag":47,"props":680,"children":681},{},[682],{"type":46,"tag":683,"props":684,"children":685},"em",{},[686],{"type":55,"value":687},"From each provider's documentation, checked November 2026.",{"type":46,"tag":47,"props":689,"children":690},{},[691,693,698],{"type":55,"value":692},"Provider dashboards are good for one question: did the provider try, and what did your endpoint answer? They don't show what your system did after the ",{"type":46,"tag":60,"props":694,"children":696},{"className":695},[],[697],{"type":55,"value":471},{"type":55,"value":699},", they don't page anyone, they cover one provider each, and they forget within days. For anything that matters you need the same data on your side.",{"type":46,"tag":179,"props":701,"children":703},{"id":702},"build-it-yourself",[704],{"type":55,"value":705},"Build it yourself",{"type":46,"tag":47,"props":707,"children":708},{},[709],{"type":55,"value":710},"If you run your own receiver, instrument it at the edge and in the worker:",{"type":46,"tag":712,"props":713,"children":715},"code-block",{"label":714},"metrics.js · Prometheus counters at the receiver",[716],{"type":46,"tag":717,"props":718,"children":722},"pre",{"className":719,"code":720,"language":721,"meta":7,"style":7},"language-js shiki shiki-themes github-dark","import client from 'prom-client'\n\nconst received = new client.Counter({\n  name: 'webhook_received_total',\n  help: 'Webhooks received',\n  labelNames: ['source', 'status', 'attempt'], // attempt: first | retry\n})\nconst latency = new client.Histogram({\n  name: 'webhook_response_seconds',\n  help: 'Time to acknowledge',\n  labelNames: ['source'],\n  buckets: [0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 7.5, 10],\n})\nconst lastSeen = new client.Gauge({\n  name: 'webhook_last_received_timestamp',\n  help: 'Unix time of last webhook',\n  labelNames: ['source'],\n})\n// Set these from the worker: events waiting for a retry, and dead letters.\nexport const backlog = new client.Gauge({\n  name: 'webhook_retry_backlog',\n  help: 'Events waiting for another attempt',\n  labelNames: ['destination'],\n})\nexport const deadLetters = new client.Counter({\n  name: 'webhook_dead_letters_total',\n  help: 'Events that exhausted their retries',\n  labelNames: ['destination'],\n})\n\nexport function track(source, handler, isRetry = () => false) {\n  return async (req, res) => {\n    const attempt = (await isRetry(req)) ? 'retry' : 'first'\n    const end = latency.startTimer({ source })\n    res.on('finish', () => {\n      end()\n      received.inc({ source, status: String(res.statusCode), attempt })\n      lastSeen.set({ source }, Date.now() / 1000)\n    })\n    return handler(req, res)\n  }\n}\n","js",[723],{"type":46,"tag":60,"props":724,"children":725},{"__ignoreMap":7},[726,755,765,806,825,843,886,895,929,946,963,980,1070,1078,1112,1129,1146,1162,1170,1179,1218,1235,1252,1269,1277,1314,1331,1348,1364,1372,1380,1450,1497,1554,1586,1622,1636,1665,1709,1718,1737,1746],{"type":46,"tag":727,"props":728,"children":731},"span",{"class":729,"line":730},"line",1,[732,738,744,749],{"type":46,"tag":727,"props":733,"children":735},{"style":734},"--shiki-default:#F97583",[736],{"type":55,"value":737},"import",{"type":46,"tag":727,"props":739,"children":741},{"style":740},"--shiki-default:#E1E4E8",[742],{"type":55,"value":743}," client ",{"type":46,"tag":727,"props":745,"children":746},{"style":734},[747],{"type":55,"value":748},"from",{"type":46,"tag":727,"props":750,"children":752},{"style":751},"--shiki-default:#9ECBFF",[753],{"type":55,"value":754}," 'prom-client'\n",{"type":46,"tag":727,"props":756,"children":758},{"class":729,"line":757},2,[759],{"type":46,"tag":727,"props":760,"children":762},{"emptyLinePlaceholder":761},true,[763],{"type":55,"value":764},"\n",{"type":46,"tag":727,"props":766,"children":768},{"class":729,"line":767},3,[769,774,780,785,790,795,801],{"type":46,"tag":727,"props":770,"children":771},{"style":734},[772],{"type":55,"value":773},"const",{"type":46,"tag":727,"props":775,"children":777},{"style":776},"--shiki-default:#79B8FF",[778],{"type":55,"value":779}," received",{"type":46,"tag":727,"props":781,"children":782},{"style":734},[783],{"type":55,"value":784}," =",{"type":46,"tag":727,"props":786,"children":787},{"style":734},[788],{"type":55,"value":789}," new",{"type":46,"tag":727,"props":791,"children":792},{"style":740},[793],{"type":55,"value":794}," client.",{"type":46,"tag":727,"props":796,"children":798},{"style":797},"--shiki-default:#B392F0",[799],{"type":55,"value":800},"Counter",{"type":46,"tag":727,"props":802,"children":803},{"style":740},[804],{"type":55,"value":805},"({\n",{"type":46,"tag":727,"props":807,"children":809},{"class":729,"line":808},4,[810,815,820],{"type":46,"tag":727,"props":811,"children":812},{"style":740},[813],{"type":55,"value":814},"  name: ",{"type":46,"tag":727,"props":816,"children":817},{"style":751},[818],{"type":55,"value":819},"'webhook_received_total'",{"type":46,"tag":727,"props":821,"children":822},{"style":740},[823],{"type":55,"value":824},",\n",{"type":46,"tag":727,"props":826,"children":828},{"class":729,"line":827},5,[829,834,839],{"type":46,"tag":727,"props":830,"children":831},{"style":740},[832],{"type":55,"value":833},"  help: ",{"type":46,"tag":727,"props":835,"children":836},{"style":751},[837],{"type":55,"value":838},"'Webhooks received'",{"type":46,"tag":727,"props":840,"children":841},{"style":740},[842],{"type":55,"value":824},{"type":46,"tag":727,"props":844,"children":846},{"class":729,"line":845},6,[847,852,857,861,866,870,875,880],{"type":46,"tag":727,"props":848,"children":849},{"style":740},[850],{"type":55,"value":851},"  labelNames: [",{"type":46,"tag":727,"props":853,"children":854},{"style":751},[855],{"type":55,"value":856},"'source'",{"type":46,"tag":727,"props":858,"children":859},{"style":740},[860],{"type":55,"value":96},{"type":46,"tag":727,"props":862,"children":863},{"style":751},[864],{"type":55,"value":865},"'status'",{"type":46,"tag":727,"props":867,"children":868},{"style":740},[869],{"type":55,"value":96},{"type":46,"tag":727,"props":871,"children":872},{"style":751},[873],{"type":55,"value":874},"'attempt'",{"type":46,"tag":727,"props":876,"children":877},{"style":740},[878],{"type":55,"value":879},"], ",{"type":46,"tag":727,"props":881,"children":883},{"style":882},"--shiki-default:#6A737D",[884],{"type":55,"value":885},"// attempt: first | retry\n",{"type":46,"tag":727,"props":887,"children":889},{"class":729,"line":888},7,[890],{"type":46,"tag":727,"props":891,"children":892},{"style":740},[893],{"type":55,"value":894},"})\n",{"type":46,"tag":727,"props":896,"children":898},{"class":729,"line":897},8,[899,903,908,912,916,920,925],{"type":46,"tag":727,"props":900,"children":901},{"style":734},[902],{"type":55,"value":773},{"type":46,"tag":727,"props":904,"children":905},{"style":776},[906],{"type":55,"value":907}," latency",{"type":46,"tag":727,"props":909,"children":910},{"style":734},[911],{"type":55,"value":784},{"type":46,"tag":727,"props":913,"children":914},{"style":734},[915],{"type":55,"value":789},{"type":46,"tag":727,"props":917,"children":918},{"style":740},[919],{"type":55,"value":794},{"type":46,"tag":727,"props":921,"children":922},{"style":797},[923],{"type":55,"value":924},"Histogram",{"type":46,"tag":727,"props":926,"children":927},{"style":740},[928],{"type":55,"value":805},{"type":46,"tag":727,"props":930,"children":932},{"class":729,"line":931},9,[933,937,942],{"type":46,"tag":727,"props":934,"children":935},{"style":740},[936],{"type":55,"value":814},{"type":46,"tag":727,"props":938,"children":939},{"style":751},[940],{"type":55,"value":941},"'webhook_response_seconds'",{"type":46,"tag":727,"props":943,"children":944},{"style":740},[945],{"type":55,"value":824},{"type":46,"tag":727,"props":947,"children":949},{"class":729,"line":948},10,[950,954,959],{"type":46,"tag":727,"props":951,"children":952},{"style":740},[953],{"type":55,"value":833},{"type":46,"tag":727,"props":955,"children":956},{"style":751},[957],{"type":55,"value":958},"'Time to acknowledge'",{"type":46,"tag":727,"props":960,"children":961},{"style":740},[962],{"type":55,"value":824},{"type":46,"tag":727,"props":964,"children":966},{"class":729,"line":965},11,[967,971,975],{"type":46,"tag":727,"props":968,"children":969},{"style":740},[970],{"type":55,"value":851},{"type":46,"tag":727,"props":972,"children":973},{"style":751},[974],{"type":55,"value":856},{"type":46,"tag":727,"props":976,"children":977},{"style":740},[978],{"type":55,"value":979},"],\n",{"type":46,"tag":727,"props":981,"children":983},{"class":729,"line":982},12,[984,989,994,998,1003,1007,1012,1016,1021,1025,1030,1034,1039,1043,1048,1052,1057,1061,1066],{"type":46,"tag":727,"props":985,"children":986},{"style":740},[987],{"type":55,"value":988},"  buckets: [",{"type":46,"tag":727,"props":990,"children":991},{"style":776},[992],{"type":55,"value":993},"0.05",{"type":46,"tag":727,"props":995,"children":996},{"style":740},[997],{"type":55,"value":96},{"type":46,"tag":727,"props":999,"children":1000},{"style":776},[1001],{"type":55,"value":1002},"0.1",{"type":46,"tag":727,"props":1004,"children":1005},{"style":740},[1006],{"type":55,"value":96},{"type":46,"tag":727,"props":1008,"children":1009},{"style":776},[1010],{"type":55,"value":1011},"0.25",{"type":46,"tag":727,"props":1013,"children":1014},{"style":740},[1015],{"type":55,"value":96},{"type":46,"tag":727,"props":1017,"children":1018},{"style":776},[1019],{"type":55,"value":1020},"0.5",{"type":46,"tag":727,"props":1022,"children":1023},{"style":740},[1024],{"type":55,"value":96},{"type":46,"tag":727,"props":1026,"children":1027},{"style":776},[1028],{"type":55,"value":1029},"1",{"type":46,"tag":727,"props":1031,"children":1032},{"style":740},[1033],{"type":55,"value":96},{"type":46,"tag":727,"props":1035,"children":1036},{"style":776},[1037],{"type":55,"value":1038},"2.5",{"type":46,"tag":727,"props":1040,"children":1041},{"style":740},[1042],{"type":55,"value":96},{"type":46,"tag":727,"props":1044,"children":1045},{"style":776},[1046],{"type":55,"value":1047},"5",{"type":46,"tag":727,"props":1049,"children":1050},{"style":740},[1051],{"type":55,"value":96},{"type":46,"tag":727,"props":1053,"children":1054},{"style":776},[1055],{"type":55,"value":1056},"7.5",{"type":46,"tag":727,"props":1058,"children":1059},{"style":740},[1060],{"type":55,"value":96},{"type":46,"tag":727,"props":1062,"children":1063},{"style":776},[1064],{"type":55,"value":1065},"10",{"type":46,"tag":727,"props":1067,"children":1068},{"style":740},[1069],{"type":55,"value":979},{"type":46,"tag":727,"props":1071,"children":1073},{"class":729,"line":1072},13,[1074],{"type":46,"tag":727,"props":1075,"children":1076},{"style":740},[1077],{"type":55,"value":894},{"type":46,"tag":727,"props":1079,"children":1081},{"class":729,"line":1080},14,[1082,1086,1091,1095,1099,1103,1108],{"type":46,"tag":727,"props":1083,"children":1084},{"style":734},[1085],{"type":55,"value":773},{"type":46,"tag":727,"props":1087,"children":1088},{"style":776},[1089],{"type":55,"value":1090}," lastSeen",{"type":46,"tag":727,"props":1092,"children":1093},{"style":734},[1094],{"type":55,"value":784},{"type":46,"tag":727,"props":1096,"children":1097},{"style":734},[1098],{"type":55,"value":789},{"type":46,"tag":727,"props":1100,"children":1101},{"style":740},[1102],{"type":55,"value":794},{"type":46,"tag":727,"props":1104,"children":1105},{"style":797},[1106],{"type":55,"value":1107},"Gauge",{"type":46,"tag":727,"props":1109,"children":1110},{"style":740},[1111],{"type":55,"value":805},{"type":46,"tag":727,"props":1113,"children":1115},{"class":729,"line":1114},15,[1116,1120,1125],{"type":46,"tag":727,"props":1117,"children":1118},{"style":740},[1119],{"type":55,"value":814},{"type":46,"tag":727,"props":1121,"children":1122},{"style":751},[1123],{"type":55,"value":1124},"'webhook_last_received_timestamp'",{"type":46,"tag":727,"props":1126,"children":1127},{"style":740},[1128],{"type":55,"value":824},{"type":46,"tag":727,"props":1130,"children":1132},{"class":729,"line":1131},16,[1133,1137,1142],{"type":46,"tag":727,"props":1134,"children":1135},{"style":740},[1136],{"type":55,"value":833},{"type":46,"tag":727,"props":1138,"children":1139},{"style":751},[1140],{"type":55,"value":1141},"'Unix time of last webhook'",{"type":46,"tag":727,"props":1143,"children":1144},{"style":740},[1145],{"type":55,"value":824},{"type":46,"tag":727,"props":1147,"children":1149},{"class":729,"line":1148},17,[1150,1154,1158],{"type":46,"tag":727,"props":1151,"children":1152},{"style":740},[1153],{"type":55,"value":851},{"type":46,"tag":727,"props":1155,"children":1156},{"style":751},[1157],{"type":55,"value":856},{"type":46,"tag":727,"props":1159,"children":1160},{"style":740},[1161],{"type":55,"value":979},{"type":46,"tag":727,"props":1163,"children":1165},{"class":729,"line":1164},18,[1166],{"type":46,"tag":727,"props":1167,"children":1168},{"style":740},[1169],{"type":55,"value":894},{"type":46,"tag":727,"props":1171,"children":1173},{"class":729,"line":1172},19,[1174],{"type":46,"tag":727,"props":1175,"children":1176},{"style":882},[1177],{"type":55,"value":1178},"// Set these from the worker: events waiting for a retry, and dead letters.\n",{"type":46,"tag":727,"props":1180,"children":1182},{"class":729,"line":1181},20,[1183,1188,1193,1198,1202,1206,1210,1214],{"type":46,"tag":727,"props":1184,"children":1185},{"style":734},[1186],{"type":55,"value":1187},"export",{"type":46,"tag":727,"props":1189,"children":1190},{"style":734},[1191],{"type":55,"value":1192}," const",{"type":46,"tag":727,"props":1194,"children":1195},{"style":776},[1196],{"type":55,"value":1197}," backlog",{"type":46,"tag":727,"props":1199,"children":1200},{"style":734},[1201],{"type":55,"value":784},{"type":46,"tag":727,"props":1203,"children":1204},{"style":734},[1205],{"type":55,"value":789},{"type":46,"tag":727,"props":1207,"children":1208},{"style":740},[1209],{"type":55,"value":794},{"type":46,"tag":727,"props":1211,"children":1212},{"style":797},[1213],{"type":55,"value":1107},{"type":46,"tag":727,"props":1215,"children":1216},{"style":740},[1217],{"type":55,"value":805},{"type":46,"tag":727,"props":1219,"children":1221},{"class":729,"line":1220},21,[1222,1226,1231],{"type":46,"tag":727,"props":1223,"children":1224},{"style":740},[1225],{"type":55,"value":814},{"type":46,"tag":727,"props":1227,"children":1228},{"style":751},[1229],{"type":55,"value":1230},"'webhook_retry_backlog'",{"type":46,"tag":727,"props":1232,"children":1233},{"style":740},[1234],{"type":55,"value":824},{"type":46,"tag":727,"props":1236,"children":1238},{"class":729,"line":1237},22,[1239,1243,1248],{"type":46,"tag":727,"props":1240,"children":1241},{"style":740},[1242],{"type":55,"value":833},{"type":46,"tag":727,"props":1244,"children":1245},{"style":751},[1246],{"type":55,"value":1247},"'Events waiting for another attempt'",{"type":46,"tag":727,"props":1249,"children":1250},{"style":740},[1251],{"type":55,"value":824},{"type":46,"tag":727,"props":1253,"children":1255},{"class":729,"line":1254},23,[1256,1260,1265],{"type":46,"tag":727,"props":1257,"children":1258},{"style":740},[1259],{"type":55,"value":851},{"type":46,"tag":727,"props":1261,"children":1262},{"style":751},[1263],{"type":55,"value":1264},"'destination'",{"type":46,"tag":727,"props":1266,"children":1267},{"style":740},[1268],{"type":55,"value":979},{"type":46,"tag":727,"props":1270,"children":1272},{"class":729,"line":1271},24,[1273],{"type":46,"tag":727,"props":1274,"children":1275},{"style":740},[1276],{"type":55,"value":894},{"type":46,"tag":727,"props":1278,"children":1280},{"class":729,"line":1279},25,[1281,1285,1289,1294,1298,1302,1306,1310],{"type":46,"tag":727,"props":1282,"children":1283},{"style":734},[1284],{"type":55,"value":1187},{"type":46,"tag":727,"props":1286,"children":1287},{"style":734},[1288],{"type":55,"value":1192},{"type":46,"tag":727,"props":1290,"children":1291},{"style":776},[1292],{"type":55,"value":1293}," deadLetters",{"type":46,"tag":727,"props":1295,"children":1296},{"style":734},[1297],{"type":55,"value":784},{"type":46,"tag":727,"props":1299,"children":1300},{"style":734},[1301],{"type":55,"value":789},{"type":46,"tag":727,"props":1303,"children":1304},{"style":740},[1305],{"type":55,"value":794},{"type":46,"tag":727,"props":1307,"children":1308},{"style":797},[1309],{"type":55,"value":800},{"type":46,"tag":727,"props":1311,"children":1312},{"style":740},[1313],{"type":55,"value":805},{"type":46,"tag":727,"props":1315,"children":1317},{"class":729,"line":1316},26,[1318,1322,1327],{"type":46,"tag":727,"props":1319,"children":1320},{"style":740},[1321],{"type":55,"value":814},{"type":46,"tag":727,"props":1323,"children":1324},{"style":751},[1325],{"type":55,"value":1326},"'webhook_dead_letters_total'",{"type":46,"tag":727,"props":1328,"children":1329},{"style":740},[1330],{"type":55,"value":824},{"type":46,"tag":727,"props":1332,"children":1334},{"class":729,"line":1333},27,[1335,1339,1344],{"type":46,"tag":727,"props":1336,"children":1337},{"style":740},[1338],{"type":55,"value":833},{"type":46,"tag":727,"props":1340,"children":1341},{"style":751},[1342],{"type":55,"value":1343},"'Events that exhausted their retries'",{"type":46,"tag":727,"props":1345,"children":1346},{"style":740},[1347],{"type":55,"value":824},{"type":46,"tag":727,"props":1349,"children":1351},{"class":729,"line":1350},28,[1352,1356,1360],{"type":46,"tag":727,"props":1353,"children":1354},{"style":740},[1355],{"type":55,"value":851},{"type":46,"tag":727,"props":1357,"children":1358},{"style":751},[1359],{"type":55,"value":1264},{"type":46,"tag":727,"props":1361,"children":1362},{"style":740},[1363],{"type":55,"value":979},{"type":46,"tag":727,"props":1365,"children":1367},{"class":729,"line":1366},29,[1368],{"type":46,"tag":727,"props":1369,"children":1370},{"style":740},[1371],{"type":55,"value":894},{"type":46,"tag":727,"props":1373,"children":1375},{"class":729,"line":1374},30,[1376],{"type":46,"tag":727,"props":1377,"children":1378},{"emptyLinePlaceholder":761},[1379],{"type":55,"value":764},{"type":46,"tag":727,"props":1381,"children":1383},{"class":729,"line":1382},31,[1384,1388,1393,1398,1403,1408,1412,1417,1421,1426,1430,1435,1440,1445],{"type":46,"tag":727,"props":1385,"children":1386},{"style":734},[1387],{"type":55,"value":1187},{"type":46,"tag":727,"props":1389,"children":1390},{"style":734},[1391],{"type":55,"value":1392}," function",{"type":46,"tag":727,"props":1394,"children":1395},{"style":797},[1396],{"type":55,"value":1397}," track",{"type":46,"tag":727,"props":1399,"children":1400},{"style":740},[1401],{"type":55,"value":1402},"(",{"type":46,"tag":727,"props":1404,"children":1406},{"style":1405},"--shiki-default:#FFAB70",[1407],{"type":55,"value":377},{"type":46,"tag":727,"props":1409,"children":1410},{"style":740},[1411],{"type":55,"value":96},{"type":46,"tag":727,"props":1413,"children":1414},{"style":1405},[1415],{"type":55,"value":1416},"handler",{"type":46,"tag":727,"props":1418,"children":1419},{"style":740},[1420],{"type":55,"value":96},{"type":46,"tag":727,"props":1422,"children":1423},{"style":797},[1424],{"type":55,"value":1425},"isRetry",{"type":46,"tag":727,"props":1427,"children":1428},{"style":734},[1429],{"type":55,"value":784},{"type":46,"tag":727,"props":1431,"children":1432},{"style":740},[1433],{"type":55,"value":1434}," () ",{"type":46,"tag":727,"props":1436,"children":1437},{"style":734},[1438],{"type":55,"value":1439},"=>",{"type":46,"tag":727,"props":1441,"children":1442},{"style":776},[1443],{"type":55,"value":1444}," false",{"type":46,"tag":727,"props":1446,"children":1447},{"style":740},[1448],{"type":55,"value":1449},") {\n",{"type":46,"tag":727,"props":1451,"children":1453},{"class":729,"line":1452},32,[1454,1459,1464,1469,1474,1478,1483,1488,1492],{"type":46,"tag":727,"props":1455,"children":1456},{"style":734},[1457],{"type":55,"value":1458},"  return",{"type":46,"tag":727,"props":1460,"children":1461},{"style":734},[1462],{"type":55,"value":1463}," async",{"type":46,"tag":727,"props":1465,"children":1466},{"style":740},[1467],{"type":55,"value":1468}," (",{"type":46,"tag":727,"props":1470,"children":1471},{"style":1405},[1472],{"type":55,"value":1473},"req",{"type":46,"tag":727,"props":1475,"children":1476},{"style":740},[1477],{"type":55,"value":96},{"type":46,"tag":727,"props":1479,"children":1480},{"style":1405},[1481],{"type":55,"value":1482},"res",{"type":46,"tag":727,"props":1484,"children":1485},{"style":740},[1486],{"type":55,"value":1487},") ",{"type":46,"tag":727,"props":1489,"children":1490},{"style":734},[1491],{"type":55,"value":1439},{"type":46,"tag":727,"props":1493,"children":1494},{"style":740},[1495],{"type":55,"value":1496}," {\n",{"type":46,"tag":727,"props":1498,"children":1500},{"class":729,"line":1499},33,[1501,1506,1511,1515,1519,1524,1529,1534,1539,1544,1549],{"type":46,"tag":727,"props":1502,"children":1503},{"style":734},[1504],{"type":55,"value":1505},"    const",{"type":46,"tag":727,"props":1507,"children":1508},{"style":776},[1509],{"type":55,"value":1510}," attempt",{"type":46,"tag":727,"props":1512,"children":1513},{"style":734},[1514],{"type":55,"value":784},{"type":46,"tag":727,"props":1516,"children":1517},{"style":740},[1518],{"type":55,"value":1468},{"type":46,"tag":727,"props":1520,"children":1521},{"style":734},[1522],{"type":55,"value":1523},"await",{"type":46,"tag":727,"props":1525,"children":1526},{"style":797},[1527],{"type":55,"value":1528}," isRetry",{"type":46,"tag":727,"props":1530,"children":1531},{"style":740},[1532],{"type":55,"value":1533},"(req)) ",{"type":46,"tag":727,"props":1535,"children":1536},{"style":734},[1537],{"type":55,"value":1538},"?",{"type":46,"tag":727,"props":1540,"children":1541},{"style":751},[1542],{"type":55,"value":1543}," 'retry'",{"type":46,"tag":727,"props":1545,"children":1546},{"style":734},[1547],{"type":55,"value":1548}," :",{"type":46,"tag":727,"props":1550,"children":1551},{"style":751},[1552],{"type":55,"value":1553}," 'first'\n",{"type":46,"tag":727,"props":1555,"children":1557},{"class":729,"line":1556},34,[1558,1562,1567,1571,1576,1581],{"type":46,"tag":727,"props":1559,"children":1560},{"style":734},[1561],{"type":55,"value":1505},{"type":46,"tag":727,"props":1563,"children":1564},{"style":776},[1565],{"type":55,"value":1566}," end",{"type":46,"tag":727,"props":1568,"children":1569},{"style":734},[1570],{"type":55,"value":784},{"type":46,"tag":727,"props":1572,"children":1573},{"style":740},[1574],{"type":55,"value":1575}," latency.",{"type":46,"tag":727,"props":1577,"children":1578},{"style":797},[1579],{"type":55,"value":1580},"startTimer",{"type":46,"tag":727,"props":1582,"children":1583},{"style":740},[1584],{"type":55,"value":1585},"({ source })\n",{"type":46,"tag":727,"props":1587,"children":1589},{"class":729,"line":1588},35,[1590,1595,1600,1604,1609,1614,1618],{"type":46,"tag":727,"props":1591,"children":1592},{"style":740},[1593],{"type":55,"value":1594},"    res.",{"type":46,"tag":727,"props":1596,"children":1597},{"style":797},[1598],{"type":55,"value":1599},"on",{"type":46,"tag":727,"props":1601,"children":1602},{"style":740},[1603],{"type":55,"value":1402},{"type":46,"tag":727,"props":1605,"children":1606},{"style":751},[1607],{"type":55,"value":1608},"'finish'",{"type":46,"tag":727,"props":1610,"children":1611},{"style":740},[1612],{"type":55,"value":1613},", () ",{"type":46,"tag":727,"props":1615,"children":1616},{"style":734},[1617],{"type":55,"value":1439},{"type":46,"tag":727,"props":1619,"children":1620},{"style":740},[1621],{"type":55,"value":1496},{"type":46,"tag":727,"props":1623,"children":1625},{"class":729,"line":1624},36,[1626,1631],{"type":46,"tag":727,"props":1627,"children":1628},{"style":797},[1629],{"type":55,"value":1630},"      end",{"type":46,"tag":727,"props":1632,"children":1633},{"style":740},[1634],{"type":55,"value":1635},"()\n",{"type":46,"tag":727,"props":1637,"children":1639},{"class":729,"line":1638},37,[1640,1645,1650,1655,1660],{"type":46,"tag":727,"props":1641,"children":1642},{"style":740},[1643],{"type":55,"value":1644},"      received.",{"type":46,"tag":727,"props":1646,"children":1647},{"style":797},[1648],{"type":55,"value":1649},"inc",{"type":46,"tag":727,"props":1651,"children":1652},{"style":740},[1653],{"type":55,"value":1654},"({ source, status: ",{"type":46,"tag":727,"props":1656,"children":1657},{"style":797},[1658],{"type":55,"value":1659},"String",{"type":46,"tag":727,"props":1661,"children":1662},{"style":740},[1663],{"type":55,"value":1664},"(res.statusCode), attempt })\n",{"type":46,"tag":727,"props":1666,"children":1668},{"class":729,"line":1667},38,[1669,1674,1679,1684,1689,1694,1699,1704],{"type":46,"tag":727,"props":1670,"children":1671},{"style":740},[1672],{"type":55,"value":1673},"      lastSeen.",{"type":46,"tag":727,"props":1675,"children":1676},{"style":797},[1677],{"type":55,"value":1678},"set",{"type":46,"tag":727,"props":1680,"children":1681},{"style":740},[1682],{"type":55,"value":1683},"({ source }, Date.",{"type":46,"tag":727,"props":1685,"children":1686},{"style":797},[1687],{"type":55,"value":1688},"now",{"type":46,"tag":727,"props":1690,"children":1691},{"style":740},[1692],{"type":55,"value":1693},"() ",{"type":46,"tag":727,"props":1695,"children":1696},{"style":734},[1697],{"type":55,"value":1698},"/",{"type":46,"tag":727,"props":1700,"children":1701},{"style":776},[1702],{"type":55,"value":1703}," 1000",{"type":46,"tag":727,"props":1705,"children":1706},{"style":740},[1707],{"type":55,"value":1708},")\n",{"type":46,"tag":727,"props":1710,"children":1712},{"class":729,"line":1711},39,[1713],{"type":46,"tag":727,"props":1714,"children":1715},{"style":740},[1716],{"type":55,"value":1717},"    })\n",{"type":46,"tag":727,"props":1719,"children":1721},{"class":729,"line":1720},40,[1722,1727,1732],{"type":46,"tag":727,"props":1723,"children":1724},{"style":734},[1725],{"type":55,"value":1726},"    return",{"type":46,"tag":727,"props":1728,"children":1729},{"style":797},[1730],{"type":55,"value":1731}," handler",{"type":46,"tag":727,"props":1733,"children":1734},{"style":740},[1735],{"type":55,"value":1736},"(req, res)\n",{"type":46,"tag":727,"props":1738,"children":1740},{"class":729,"line":1739},41,[1741],{"type":46,"tag":727,"props":1742,"children":1743},{"style":740},[1744],{"type":55,"value":1745},"  }\n",{"type":46,"tag":727,"props":1747,"children":1749},{"class":729,"line":1748},42,[1750],{"type":46,"tag":727,"props":1751,"children":1752},{"style":740},[1753],{"type":55,"value":1754},"}\n",{"type":46,"tag":47,"props":1756,"children":1757},{},[1758],{"type":55,"value":1759},"Then alert on the series, not on log lines:",{"type":46,"tag":712,"props":1761,"children":1763},{"label":1762},"alerts.yml · Prometheus rules",[1764],{"type":46,"tag":717,"props":1765,"children":1769},{"className":1766,"code":1767,"language":1768,"meta":7,"style":7},"language-yaml shiki shiki-themes github-dark","groups:\n  - name: webhooks\n    rules:\n      - alert: WebhookErrorRate\n        expr: |\n          sum by (source) (rate(webhook_received_total{status!~\"2..\"}[10m]))\n            / sum by (source) (rate(webhook_received_total[10m])) > 0.01\n        for: 10m\n      # Half the provider's timeout: 2.5 s for Shopify, 1.5 s for Slack.\n      - alert: WebhookSlowAck\n        expr: |\n          histogram_quantile(0.95,\n            sum by (le, source) (rate(webhook_response_seconds_bucket[10m]))\n          ) > 2.5\n        for: 15m\n      - alert: WebhookSlowAckSlack\n        expr: |\n          histogram_quantile(0.95,\n            sum by (le) (rate(webhook_response_seconds_bucket{source=\"slack\"}[10m]))\n          ) > 1.5\n        for: 15m\n      - alert: WebhookSilence\n        expr: |\n          (time() - webhook_last_received_timestamp{source=\"shopify\"} > 3600)\n            or absent(webhook_last_received_timestamp{source=\"shopify\"})\n        for: 5m\n      - alert: WebhookBacklogGrowing\n        expr: deriv(webhook_retry_backlog[15m]) > 0\n        for: 15m\n      - alert: WebhookDeadLetter\n        expr: increase(webhook_dead_letters_total[5m]) > 0\n","yaml",[1770],{"type":46,"tag":60,"props":1771,"children":1772},{"__ignoreMap":7},[1773,1787,1810,1822,1844,1861,1869,1877,1894,1902,1922,1937,1945,1953,1961,1977,1997,2012,2019,2027,2035,2050,2070,2085,2093,2101,2117,2137,2153,2168,2188],{"type":46,"tag":727,"props":1774,"children":1775},{"class":729,"line":730},[1776,1782],{"type":46,"tag":727,"props":1777,"children":1779},{"style":1778},"--shiki-default:#85E89D",[1780],{"type":55,"value":1781},"groups",{"type":46,"tag":727,"props":1783,"children":1784},{"style":740},[1785],{"type":55,"value":1786},":\n",{"type":46,"tag":727,"props":1788,"children":1789},{"class":729,"line":757},[1790,1795,1800,1805],{"type":46,"tag":727,"props":1791,"children":1792},{"style":740},[1793],{"type":55,"value":1794},"  - ",{"type":46,"tag":727,"props":1796,"children":1797},{"style":1778},[1798],{"type":55,"value":1799},"name",{"type":46,"tag":727,"props":1801,"children":1802},{"style":740},[1803],{"type":55,"value":1804},": ",{"type":46,"tag":727,"props":1806,"children":1807},{"style":751},[1808],{"type":55,"value":1809},"webhooks\n",{"type":46,"tag":727,"props":1811,"children":1812},{"class":729,"line":767},[1813,1818],{"type":46,"tag":727,"props":1814,"children":1815},{"style":1778},[1816],{"type":55,"value":1817},"    rules",{"type":46,"tag":727,"props":1819,"children":1820},{"style":740},[1821],{"type":55,"value":1786},{"type":46,"tag":727,"props":1823,"children":1824},{"class":729,"line":808},[1825,1830,1835,1839],{"type":46,"tag":727,"props":1826,"children":1827},{"style":740},[1828],{"type":55,"value":1829},"      - ",{"type":46,"tag":727,"props":1831,"children":1832},{"style":1778},[1833],{"type":55,"value":1834},"alert",{"type":46,"tag":727,"props":1836,"children":1837},{"style":740},[1838],{"type":55,"value":1804},{"type":46,"tag":727,"props":1840,"children":1841},{"style":751},[1842],{"type":55,"value":1843},"WebhookErrorRate\n",{"type":46,"tag":727,"props":1845,"children":1846},{"class":729,"line":827},[1847,1852,1856],{"type":46,"tag":727,"props":1848,"children":1849},{"style":1778},[1850],{"type":55,"value":1851},"        expr",{"type":46,"tag":727,"props":1853,"children":1854},{"style":740},[1855],{"type":55,"value":1804},{"type":46,"tag":727,"props":1857,"children":1858},{"style":734},[1859],{"type":55,"value":1860},"|\n",{"type":46,"tag":727,"props":1862,"children":1863},{"class":729,"line":845},[1864],{"type":46,"tag":727,"props":1865,"children":1866},{"style":751},[1867],{"type":55,"value":1868},"          sum by (source) (rate(webhook_received_total{status!~\"2..\"}[10m]))\n",{"type":46,"tag":727,"props":1870,"children":1871},{"class":729,"line":888},[1872],{"type":46,"tag":727,"props":1873,"children":1874},{"style":751},[1875],{"type":55,"value":1876},"            / sum by (source) (rate(webhook_received_total[10m])) > 0.01\n",{"type":46,"tag":727,"props":1878,"children":1879},{"class":729,"line":897},[1880,1885,1889],{"type":46,"tag":727,"props":1881,"children":1882},{"style":1778},[1883],{"type":55,"value":1884},"        for",{"type":46,"tag":727,"props":1886,"children":1887},{"style":740},[1888],{"type":55,"value":1804},{"type":46,"tag":727,"props":1890,"children":1891},{"style":751},[1892],{"type":55,"value":1893},"10m\n",{"type":46,"tag":727,"props":1895,"children":1896},{"class":729,"line":931},[1897],{"type":46,"tag":727,"props":1898,"children":1899},{"style":882},[1900],{"type":55,"value":1901},"      # Half the provider's timeout: 2.5 s for Shopify, 1.5 s for Slack.\n",{"type":46,"tag":727,"props":1903,"children":1904},{"class":729,"line":948},[1905,1909,1913,1917],{"type":46,"tag":727,"props":1906,"children":1907},{"style":740},[1908],{"type":55,"value":1829},{"type":46,"tag":727,"props":1910,"children":1911},{"style":1778},[1912],{"type":55,"value":1834},{"type":46,"tag":727,"props":1914,"children":1915},{"style":740},[1916],{"type":55,"value":1804},{"type":46,"tag":727,"props":1918,"children":1919},{"style":751},[1920],{"type":55,"value":1921},"WebhookSlowAck\n",{"type":46,"tag":727,"props":1923,"children":1924},{"class":729,"line":965},[1925,1929,1933],{"type":46,"tag":727,"props":1926,"children":1927},{"style":1778},[1928],{"type":55,"value":1851},{"type":46,"tag":727,"props":1930,"children":1931},{"style":740},[1932],{"type":55,"value":1804},{"type":46,"tag":727,"props":1934,"children":1935},{"style":734},[1936],{"type":55,"value":1860},{"type":46,"tag":727,"props":1938,"children":1939},{"class":729,"line":982},[1940],{"type":46,"tag":727,"props":1941,"children":1942},{"style":751},[1943],{"type":55,"value":1944},"          histogram_quantile(0.95,\n",{"type":46,"tag":727,"props":1946,"children":1947},{"class":729,"line":1072},[1948],{"type":46,"tag":727,"props":1949,"children":1950},{"style":751},[1951],{"type":55,"value":1952},"            sum by (le, source) (rate(webhook_response_seconds_bucket[10m]))\n",{"type":46,"tag":727,"props":1954,"children":1955},{"class":729,"line":1080},[1956],{"type":46,"tag":727,"props":1957,"children":1958},{"style":751},[1959],{"type":55,"value":1960},"          ) > 2.5\n",{"type":46,"tag":727,"props":1962,"children":1963},{"class":729,"line":1114},[1964,1968,1972],{"type":46,"tag":727,"props":1965,"children":1966},{"style":1778},[1967],{"type":55,"value":1884},{"type":46,"tag":727,"props":1969,"children":1970},{"style":740},[1971],{"type":55,"value":1804},{"type":46,"tag":727,"props":1973,"children":1974},{"style":751},[1975],{"type":55,"value":1976},"15m\n",{"type":46,"tag":727,"props":1978,"children":1979},{"class":729,"line":1131},[1980,1984,1988,1992],{"type":46,"tag":727,"props":1981,"children":1982},{"style":740},[1983],{"type":55,"value":1829},{"type":46,"tag":727,"props":1985,"children":1986},{"style":1778},[1987],{"type":55,"value":1834},{"type":46,"tag":727,"props":1989,"children":1990},{"style":740},[1991],{"type":55,"value":1804},{"type":46,"tag":727,"props":1993,"children":1994},{"style":751},[1995],{"type":55,"value":1996},"WebhookSlowAckSlack\n",{"type":46,"tag":727,"props":1998,"children":1999},{"class":729,"line":1148},[2000,2004,2008],{"type":46,"tag":727,"props":2001,"children":2002},{"style":1778},[2003],{"type":55,"value":1851},{"type":46,"tag":727,"props":2005,"children":2006},{"style":740},[2007],{"type":55,"value":1804},{"type":46,"tag":727,"props":2009,"children":2010},{"style":734},[2011],{"type":55,"value":1860},{"type":46,"tag":727,"props":2013,"children":2014},{"class":729,"line":1164},[2015],{"type":46,"tag":727,"props":2016,"children":2017},{"style":751},[2018],{"type":55,"value":1944},{"type":46,"tag":727,"props":2020,"children":2021},{"class":729,"line":1172},[2022],{"type":46,"tag":727,"props":2023,"children":2024},{"style":751},[2025],{"type":55,"value":2026},"            sum by (le) (rate(webhook_response_seconds_bucket{source=\"slack\"}[10m]))\n",{"type":46,"tag":727,"props":2028,"children":2029},{"class":729,"line":1181},[2030],{"type":46,"tag":727,"props":2031,"children":2032},{"style":751},[2033],{"type":55,"value":2034},"          ) > 1.5\n",{"type":46,"tag":727,"props":2036,"children":2037},{"class":729,"line":1220},[2038,2042,2046],{"type":46,"tag":727,"props":2039,"children":2040},{"style":1778},[2041],{"type":55,"value":1884},{"type":46,"tag":727,"props":2043,"children":2044},{"style":740},[2045],{"type":55,"value":1804},{"type":46,"tag":727,"props":2047,"children":2048},{"style":751},[2049],{"type":55,"value":1976},{"type":46,"tag":727,"props":2051,"children":2052},{"class":729,"line":1237},[2053,2057,2061,2065],{"type":46,"tag":727,"props":2054,"children":2055},{"style":740},[2056],{"type":55,"value":1829},{"type":46,"tag":727,"props":2058,"children":2059},{"style":1778},[2060],{"type":55,"value":1834},{"type":46,"tag":727,"props":2062,"children":2063},{"style":740},[2064],{"type":55,"value":1804},{"type":46,"tag":727,"props":2066,"children":2067},{"style":751},[2068],{"type":55,"value":2069},"WebhookSilence\n",{"type":46,"tag":727,"props":2071,"children":2072},{"class":729,"line":1254},[2073,2077,2081],{"type":46,"tag":727,"props":2074,"children":2075},{"style":1778},[2076],{"type":55,"value":1851},{"type":46,"tag":727,"props":2078,"children":2079},{"style":740},[2080],{"type":55,"value":1804},{"type":46,"tag":727,"props":2082,"children":2083},{"style":734},[2084],{"type":55,"value":1860},{"type":46,"tag":727,"props":2086,"children":2087},{"class":729,"line":1271},[2088],{"type":46,"tag":727,"props":2089,"children":2090},{"style":751},[2091],{"type":55,"value":2092},"          (time() - webhook_last_received_timestamp{source=\"shopify\"} > 3600)\n",{"type":46,"tag":727,"props":2094,"children":2095},{"class":729,"line":1279},[2096],{"type":46,"tag":727,"props":2097,"children":2098},{"style":751},[2099],{"type":55,"value":2100},"            or absent(webhook_last_received_timestamp{source=\"shopify\"})\n",{"type":46,"tag":727,"props":2102,"children":2103},{"class":729,"line":1316},[2104,2108,2112],{"type":46,"tag":727,"props":2105,"children":2106},{"style":1778},[2107],{"type":55,"value":1884},{"type":46,"tag":727,"props":2109,"children":2110},{"style":740},[2111],{"type":55,"value":1804},{"type":46,"tag":727,"props":2113,"children":2114},{"style":751},[2115],{"type":55,"value":2116},"5m\n",{"type":46,"tag":727,"props":2118,"children":2119},{"class":729,"line":1333},[2120,2124,2128,2132],{"type":46,"tag":727,"props":2121,"children":2122},{"style":740},[2123],{"type":55,"value":1829},{"type":46,"tag":727,"props":2125,"children":2126},{"style":1778},[2127],{"type":55,"value":1834},{"type":46,"tag":727,"props":2129,"children":2130},{"style":740},[2131],{"type":55,"value":1804},{"type":46,"tag":727,"props":2133,"children":2134},{"style":751},[2135],{"type":55,"value":2136},"WebhookBacklogGrowing\n",{"type":46,"tag":727,"props":2138,"children":2139},{"class":729,"line":1350},[2140,2144,2148],{"type":46,"tag":727,"props":2141,"children":2142},{"style":1778},[2143],{"type":55,"value":1851},{"type":46,"tag":727,"props":2145,"children":2146},{"style":740},[2147],{"type":55,"value":1804},{"type":46,"tag":727,"props":2149,"children":2150},{"style":751},[2151],{"type":55,"value":2152},"deriv(webhook_retry_backlog[15m]) > 0\n",{"type":46,"tag":727,"props":2154,"children":2155},{"class":729,"line":1366},[2156,2160,2164],{"type":46,"tag":727,"props":2157,"children":2158},{"style":1778},[2159],{"type":55,"value":1884},{"type":46,"tag":727,"props":2161,"children":2162},{"style":740},[2163],{"type":55,"value":1804},{"type":46,"tag":727,"props":2165,"children":2166},{"style":751},[2167],{"type":55,"value":1976},{"type":46,"tag":727,"props":2169,"children":2170},{"class":729,"line":1374},[2171,2175,2179,2183],{"type":46,"tag":727,"props":2172,"children":2173},{"style":740},[2174],{"type":55,"value":1829},{"type":46,"tag":727,"props":2176,"children":2177},{"style":1778},[2178],{"type":55,"value":1834},{"type":46,"tag":727,"props":2180,"children":2181},{"style":740},[2182],{"type":55,"value":1804},{"type":46,"tag":727,"props":2184,"children":2185},{"style":751},[2186],{"type":55,"value":2187},"WebhookDeadLetter\n",{"type":46,"tag":727,"props":2189,"children":2190},{"class":729,"line":1382},[2191,2195,2199],{"type":46,"tag":727,"props":2192,"children":2193},{"style":1778},[2194],{"type":55,"value":1851},{"type":46,"tag":727,"props":2196,"children":2197},{"style":740},[2198],{"type":55,"value":1804},{"type":46,"tag":727,"props":2200,"children":2201},{"style":751},[2202],{"type":55,"value":2203},"increase(webhook_dead_letters_total[5m]) > 0\n",{"type":46,"tag":47,"props":2205,"children":2206},{},[2207,2212,2214,2220,2222,2227,2229,2235],{"type":46,"tag":60,"props":2208,"children":2210},{"className":2209},[],[2211],{"type":55,"value":1425},{"type":55,"value":2213}," decides the ",{"type":46,"tag":60,"props":2215,"children":2217},{"className":2216},[],[2218],{"type":55,"value":2219},"attempt",{"type":55,"value":2221}," label: a delivery is a retry when its id is already in your event store (Slack also sends ",{"type":46,"tag":60,"props":2223,"children":2225},{"className":2224},[],[2226],{"type":55,"value":658},{"type":55,"value":2228}," on retries), which gives you first-attempt success alongside eventual success. The ",{"type":46,"tag":60,"props":2230,"children":2232},{"className":2231},[],[2233],{"type":55,"value":2234},"absent()",{"type":55,"value":2236}," clause matters: without it, a receiver that restarts and never receives another event never alerts. The gauge also resets on restart, so for a silence check that must survive deploys, read the last event time from your event store instead. Prometheus is one option: the same counters work as OpenTelemetry metrics, so you can send them to Datadog, Grafana Cloud or New Relic if that is where your alerts already live. This works well, and it is also a dashboard and five alert rules you now own per receiver.",{"type":46,"tag":179,"props":2238,"children":2240},{"id":2239},"alerting-that-doesnt-cry-wolf",[2241],{"type":55,"value":2242},"Alerting that doesn't cry wolf",{"type":46,"tag":47,"props":2244,"children":2245},{},[2246],{"type":55,"value":2247},"Noisy webhook alerts get muted, and muted alerts are worse than none. A few rules keep them useful:",{"type":46,"tag":2249,"props":2250,"children":2251},"ul",{},[2252,2262,2272,2282],{"type":46,"tag":78,"props":2253,"children":2254},{},[2255,2260],{"type":46,"tag":51,"props":2256,"children":2257},{},[2258],{"type":55,"value":2259},"Match sensitivity to the flow.",{"type":55,"value":2261}," Page on the first failed payment or order webhook. For analytics or notification webhooks, alert only above an error rate, over a window long enough to ignore a single blip.",{"type":46,"tag":78,"props":2263,"children":2264},{},[2265,2270],{"type":46,"tag":51,"props":2266,"children":2267},{},[2268],{"type":55,"value":2269},"Resolve automatically.",{"type":55,"value":2271}," An incident that closes itself after a clean window means on-call only looks at things that are still broken.",{"type":46,"tag":78,"props":2273,"children":2274},{},[2275,2280],{"type":46,"tag":51,"props":2276,"children":2277},{},[2278],{"type":55,"value":2279},"Alert on the destination, not the payload.",{"type":55,"value":2281}," A run of failures to one endpoint is one incident, not five hundred messages.",{"type":46,"tag":78,"props":2283,"children":2284},{},[2285,2290],{"type":46,"tag":51,"props":2286,"children":2287},{},[2288],{"type":55,"value":2289},"Say what is affected.",{"type":55,"value":2291}," \"orders/create to fulfilment-api failing since 14:02, 37 failures\" beats \"webhook error\". A good alert carries enough to act on without opening a dashboard:",{"type":46,"tag":712,"props":2293,"children":2295},{"label":2294},"JSON · a useful webhook alert",[2296],{"type":46,"tag":717,"props":2297,"children":2301},{"className":2298,"code":2299,"language":2300,"meta":7,"style":7},"language-json shiki shiki-themes github-dark","{\n  \"status\": \"open\",\n  \"title\": \"Webhook deliveries failing: shopify → fulfilment-api\",\n  \"source\": \"shopify\",\n  \"topic\": \"orders/create\",\n  \"destination\": \"https://fulfilment.internal/webhooks/shopify\",\n  \"since\": \"2026-11-10T14:02:00Z\",\n  \"failures\": 37,\n  \"success_rate_10m\": 0.61,\n  \"last_response\": { \"status\": 503, \"body\": \"upstream connect error\" },\n  \"retry_backlog\": 37,\n  \"oldest_pending_age_s\": 1080,\n  \"runbook\": \"https://wiki.example.com/runbooks/shopify-webhooks\"\n}\n","json",[2302],{"type":46,"tag":60,"props":2303,"children":2304},{"__ignoreMap":7},[2305,2313,2334,2355,2376,2397,2418,2439,2460,2481,2530,2550,2571,2588],{"type":46,"tag":727,"props":2306,"children":2307},{"class":729,"line":730},[2308],{"type":46,"tag":727,"props":2309,"children":2310},{"style":740},[2311],{"type":55,"value":2312},"{\n",{"type":46,"tag":727,"props":2314,"children":2315},{"class":729,"line":757},[2316,2321,2325,2330],{"type":46,"tag":727,"props":2317,"children":2318},{"style":776},[2319],{"type":55,"value":2320},"  \"status\"",{"type":46,"tag":727,"props":2322,"children":2323},{"style":740},[2324],{"type":55,"value":1804},{"type":46,"tag":727,"props":2326,"children":2327},{"style":751},[2328],{"type":55,"value":2329},"\"open\"",{"type":46,"tag":727,"props":2331,"children":2332},{"style":740},[2333],{"type":55,"value":824},{"type":46,"tag":727,"props":2335,"children":2336},{"class":729,"line":767},[2337,2342,2346,2351],{"type":46,"tag":727,"props":2338,"children":2339},{"style":776},[2340],{"type":55,"value":2341},"  \"title\"",{"type":46,"tag":727,"props":2343,"children":2344},{"style":740},[2345],{"type":55,"value":1804},{"type":46,"tag":727,"props":2347,"children":2348},{"style":751},[2349],{"type":55,"value":2350},"\"Webhook deliveries failing: shopify → fulfilment-api\"",{"type":46,"tag":727,"props":2352,"children":2353},{"style":740},[2354],{"type":55,"value":824},{"type":46,"tag":727,"props":2356,"children":2357},{"class":729,"line":808},[2358,2363,2367,2372],{"type":46,"tag":727,"props":2359,"children":2360},{"style":776},[2361],{"type":55,"value":2362},"  \"source\"",{"type":46,"tag":727,"props":2364,"children":2365},{"style":740},[2366],{"type":55,"value":1804},{"type":46,"tag":727,"props":2368,"children":2369},{"style":751},[2370],{"type":55,"value":2371},"\"shopify\"",{"type":46,"tag":727,"props":2373,"children":2374},{"style":740},[2375],{"type":55,"value":824},{"type":46,"tag":727,"props":2377,"children":2378},{"class":729,"line":827},[2379,2384,2388,2393],{"type":46,"tag":727,"props":2380,"children":2381},{"style":776},[2382],{"type":55,"value":2383},"  \"topic\"",{"type":46,"tag":727,"props":2385,"children":2386},{"style":740},[2387],{"type":55,"value":1804},{"type":46,"tag":727,"props":2389,"children":2390},{"style":751},[2391],{"type":55,"value":2392},"\"orders/create\"",{"type":46,"tag":727,"props":2394,"children":2395},{"style":740},[2396],{"type":55,"value":824},{"type":46,"tag":727,"props":2398,"children":2399},{"class":729,"line":845},[2400,2405,2409,2414],{"type":46,"tag":727,"props":2401,"children":2402},{"style":776},[2403],{"type":55,"value":2404},"  \"destination\"",{"type":46,"tag":727,"props":2406,"children":2407},{"style":740},[2408],{"type":55,"value":1804},{"type":46,"tag":727,"props":2410,"children":2411},{"style":751},[2412],{"type":55,"value":2413},"\"https://fulfilment.internal/webhooks/shopify\"",{"type":46,"tag":727,"props":2415,"children":2416},{"style":740},[2417],{"type":55,"value":824},{"type":46,"tag":727,"props":2419,"children":2420},{"class":729,"line":888},[2421,2426,2430,2435],{"type":46,"tag":727,"props":2422,"children":2423},{"style":776},[2424],{"type":55,"value":2425},"  \"since\"",{"type":46,"tag":727,"props":2427,"children":2428},{"style":740},[2429],{"type":55,"value":1804},{"type":46,"tag":727,"props":2431,"children":2432},{"style":751},[2433],{"type":55,"value":2434},"\"2026-11-10T14:02:00Z\"",{"type":46,"tag":727,"props":2436,"children":2437},{"style":740},[2438],{"type":55,"value":824},{"type":46,"tag":727,"props":2440,"children":2441},{"class":729,"line":897},[2442,2447,2451,2456],{"type":46,"tag":727,"props":2443,"children":2444},{"style":776},[2445],{"type":55,"value":2446},"  \"failures\"",{"type":46,"tag":727,"props":2448,"children":2449},{"style":740},[2450],{"type":55,"value":1804},{"type":46,"tag":727,"props":2452,"children":2453},{"style":776},[2454],{"type":55,"value":2455},"37",{"type":46,"tag":727,"props":2457,"children":2458},{"style":740},[2459],{"type":55,"value":824},{"type":46,"tag":727,"props":2461,"children":2462},{"class":729,"line":931},[2463,2468,2472,2477],{"type":46,"tag":727,"props":2464,"children":2465},{"style":776},[2466],{"type":55,"value":2467},"  \"success_rate_10m\"",{"type":46,"tag":727,"props":2469,"children":2470},{"style":740},[2471],{"type":55,"value":1804},{"type":46,"tag":727,"props":2473,"children":2474},{"style":776},[2475],{"type":55,"value":2476},"0.61",{"type":46,"tag":727,"props":2478,"children":2479},{"style":740},[2480],{"type":55,"value":824},{"type":46,"tag":727,"props":2482,"children":2483},{"class":729,"line":948},[2484,2489,2494,2499,2503,2507,2511,2516,2520,2525],{"type":46,"tag":727,"props":2485,"children":2486},{"style":776},[2487],{"type":55,"value":2488},"  \"last_response\"",{"type":46,"tag":727,"props":2490,"children":2491},{"style":740},[2492],{"type":55,"value":2493},": { ",{"type":46,"tag":727,"props":2495,"children":2496},{"style":776},[2497],{"type":55,"value":2498},"\"status\"",{"type":46,"tag":727,"props":2500,"children":2501},{"style":740},[2502],{"type":55,"value":1804},{"type":46,"tag":727,"props":2504,"children":2505},{"style":776},[2506],{"type":55,"value":397},{"type":46,"tag":727,"props":2508,"children":2509},{"style":740},[2510],{"type":55,"value":96},{"type":46,"tag":727,"props":2512,"children":2513},{"style":776},[2514],{"type":55,"value":2515},"\"body\"",{"type":46,"tag":727,"props":2517,"children":2518},{"style":740},[2519],{"type":55,"value":1804},{"type":46,"tag":727,"props":2521,"children":2522},{"style":751},[2523],{"type":55,"value":2524},"\"upstream connect error\"",{"type":46,"tag":727,"props":2526,"children":2527},{"style":740},[2528],{"type":55,"value":2529}," },\n",{"type":46,"tag":727,"props":2531,"children":2532},{"class":729,"line":965},[2533,2538,2542,2546],{"type":46,"tag":727,"props":2534,"children":2535},{"style":776},[2536],{"type":55,"value":2537},"  \"retry_backlog\"",{"type":46,"tag":727,"props":2539,"children":2540},{"style":740},[2541],{"type":55,"value":1804},{"type":46,"tag":727,"props":2543,"children":2544},{"style":776},[2545],{"type":55,"value":2455},{"type":46,"tag":727,"props":2547,"children":2548},{"style":740},[2549],{"type":55,"value":824},{"type":46,"tag":727,"props":2551,"children":2552},{"class":729,"line":982},[2553,2558,2562,2567],{"type":46,"tag":727,"props":2554,"children":2555},{"style":776},[2556],{"type":55,"value":2557},"  \"oldest_pending_age_s\"",{"type":46,"tag":727,"props":2559,"children":2560},{"style":740},[2561],{"type":55,"value":1804},{"type":46,"tag":727,"props":2563,"children":2564},{"style":776},[2565],{"type":55,"value":2566},"1080",{"type":46,"tag":727,"props":2568,"children":2569},{"style":740},[2570],{"type":55,"value":824},{"type":46,"tag":727,"props":2572,"children":2573},{"class":729,"line":1072},[2574,2579,2583],{"type":46,"tag":727,"props":2575,"children":2576},{"style":776},[2577],{"type":55,"value":2578},"  \"runbook\"",{"type":46,"tag":727,"props":2580,"children":2581},{"style":740},[2582],{"type":55,"value":1804},{"type":46,"tag":727,"props":2584,"children":2585},{"style":751},[2586],{"type":55,"value":2587},"\"https://wiki.example.com/runbooks/shopify-webhooks\"\n",{"type":46,"tag":727,"props":2589,"children":2590},{"class":729,"line":1080},[2591],{"type":46,"tag":727,"props":2592,"children":2593},{"style":740},[2594],{"type":55,"value":1754},{"type":46,"tag":47,"props":2596,"children":2597},{},[2598,2600,2606],{"type":55,"value":2599},"Send the matching ",{"type":46,"tag":60,"props":2601,"children":2603},{"className":2602},[],[2604],{"type":55,"value":2605},"\"status\": \"resolved\"",{"type":55,"value":2607}," message when deliveries recover, so the incident tool closes the ticket on its own.",{"type":46,"tag":2249,"props":2609,"children":2610},{},[2611],{"type":46,"tag":78,"props":2612,"children":2613},{},[2614,2619,2621,2626,2628,2634],{"type":46,"tag":51,"props":2615,"children":2616},{},[2617],{"type":55,"value":2618},"Test the alert.",{"type":55,"value":2620}," Point a destination at a URL that returns ",{"type":46,"tag":60,"props":2622,"children":2624},{"className":2623},[],[2625],{"type":55,"value":65},{"type":55,"value":2627},", for example a ",{"type":46,"tag":365,"props":2629,"children":2631},{"href":2630},"/webhook-bin/",[2632],{"type":55,"value":2633},"Webhook Bin",{"type":55,"value":2635}," configured to fail, and confirm the alert fires and resolves.",{"type":46,"tag":179,"props":2637,"children":2639},{"id":2638},"if-you-send-webhooks",[2640],{"type":55,"value":2641},"If you send webhooks",{"type":46,"tag":47,"props":2643,"children":2644},{},[2645],{"type":55,"value":2646},"Monitoring the sending side is the same idea, one level out: your customers' endpoints are the destinations, and they fail more often than yours.",{"type":46,"tag":2249,"props":2648,"children":2649},{},[2650,2660,2670,2680],{"type":46,"tag":78,"props":2651,"children":2652},{},[2653,2658],{"type":46,"tag":51,"props":2654,"children":2655},{},[2656],{"type":55,"value":2657},"Success rate per customer endpoint",{"type":55,"value":2659},", not just overall. One customer with a dead endpoint can hide inside a 99.9% average.",{"type":46,"tag":78,"props":2661,"children":2662},{},[2663,2668],{"type":46,"tag":51,"props":2664,"children":2665},{},[2666],{"type":55,"value":2667},"Endpoints that keep failing.",{"type":55,"value":2669}," Mark them as failing, alert your support team, and auto-disable after a long outage instead of retrying forever.",{"type":46,"tag":78,"props":2671,"children":2672},{},[2673,2678],{"type":46,"tag":51,"props":2674,"children":2675},{},[2676],{"type":55,"value":2677},"Messages that ran out of retries.",{"type":55,"value":2679}," Each one is data a customer never received; they need a list they can replay.",{"type":46,"tag":78,"props":2681,"children":2682},{},[2683,2688],{"type":46,"tag":51,"props":2684,"children":2685},{},[2686],{"type":55,"value":2687},"Tell the customer.",{"type":55,"value":2689}," An email or dashboard banner when their endpoint is failing saves a support ticket three days later.",{"type":46,"tag":47,"props":2691,"children":2692},{},[2693,2695,2701,2703,2709,2711,2717],{"type":55,"value":2694},"Webhook Relay's ",{"type":46,"tag":365,"props":2696,"children":2698},{"href":2697},"/outbound-webhooks/",[2699],{"type":55,"value":2700},"Outbound Gateway",{"type":55,"value":2702}," tracks endpoint health, marks endpoints ",{"type":46,"tag":60,"props":2704,"children":2706},{"className":2705},[],[2707],{"type":55,"value":2708},"failing",{"type":55,"value":2710},", optionally auto-disables them and lets you recover every failed delivery once the customer is back; see ",{"type":46,"tag":365,"props":2712,"children":2714},{"href":2713},"/docs/outbound/operating/",[2715],{"type":55,"value":2716},"monitor, retry and recover deliveries",{"type":55,"value":453},{"type":46,"tag":179,"props":2719,"children":2721},{"id":2720},"monitor-webhooks-with-webhook-relay",[2722],{"type":55,"value":2723},"Monitor webhooks with Webhook Relay",{"type":46,"tag":47,"props":2725,"children":2726},{},[2727,2729,2735],{"type":55,"value":2728},"If your webhooks pass through a ",{"type":46,"tag":365,"props":2730,"children":2732},{"href":2731},"/webhooks/",[2733],{"type":55,"value":2734},"Webhook Relay",{"type":55,"value":2736}," bucket, the receiving side is logged and measured without code.",{"type":46,"tag":2249,"props":2738,"children":2739},{},[2740,2750,2767,2781],{"type":46,"tag":78,"props":2741,"children":2742},{},[2743,2748],{"type":46,"tag":51,"props":2744,"children":2745},{},[2746],{"type":55,"value":2747},"Delivery health across every bucket.",{"type":55,"value":2749}," Requests, delivery rate, failures, retries and average latency per bucket, ranked by failures, with a failures-over-time sparkline and 7-day to 12-month windows.",{"type":46,"tag":78,"props":2751,"children":2752},{},[2753,2758,2760,2766],{"type":46,"tag":51,"props":2754,"children":2755},{},[2756],{"type":55,"value":2757},"A request log per bucket",{"type":55,"value":2759}," with the full request, your endpoint's response and every attempt, filterable by status and destination. Resend one delivery, or recover every failure in a window once a fix is out. See ",{"type":46,"tag":365,"props":2761,"children":2763},{"href":2762},"/features/webhook-logs/",[2764],{"type":55,"value":2765},"webhook logs",{"type":55,"value":453},{"type":46,"tag":78,"props":2768,"children":2769},{},[2770,2779],{"type":46,"tag":51,"props":2771,"children":2772},{},[2773],{"type":46,"tag":365,"props":2774,"children":2776},{"href":2775},"/features/alerts/",[2777],{"type":55,"value":2778},"Alerts",{"type":55,"value":2780}," on delivery failures, sent to Slack, email, Microsoft Teams, Discord, Telegram, Pushover or any webhook (PagerDuty, Opsgenie). A policy opens an incident on the first failed delivery or above an error-rate threshold, per bucket, and resolves it automatically after a full window without failures.",{"type":46,"tag":78,"props":2782,"children":2783},{},[2784,2793],{"type":46,"tag":51,"props":2785,"children":2786},{},[2787],{"type":46,"tag":365,"props":2788,"children":2790},{"href":2789},"/features/durable-retries/",[2791],{"type":55,"value":2792},"Durable retries",{"type":55,"value":2794}," keep failed deliveries retrying for up to 30 days while you fix the destination, so an alert is a prompt to act, not a countdown to data loss.",{"type":46,"tag":47,"props":2796,"children":2797},{},[2798,2800,2806],{"type":55,"value":2799},"Alerts in Webhook Relay cover delivery failures to your destinations. They don't yet alert on silence, so pair them with one scheduled check of your own: a job that reads the time of the last event per source and pages when it is older than that source's normal gap. Your agent can also ask: the ",{"type":46,"tag":365,"props":2801,"children":2803},{"href":2802},"/docs/mcp/",[2804],{"type":55,"value":2805},"MCP server",{"type":55,"value":2807}," exposes delivery stats and logs, so \"which buckets had failures this week?\" is one question.",{"type":46,"tag":2809,"props":2810,"children":2816},"inline-cta",{"label":2811,"secondary-label":2812,"secondary-to":2813,"title":2814,"to":2815},"Create a free account","Set up an alert policy","/docs/webhooks/alerts/","Know about failed webhooks before your customers do","https://my.webhookrelay.com/register",[2817],{"type":46,"tag":47,"props":2818,"children":2819},{},[2820],{"type":55,"value":2821},"Route a provider through a bucket, attach an alert policy and pick Slack or email. Delivery health and the request log are on from the first webhook.",{"type":46,"tag":179,"props":2823,"children":2825},{"id":2824},"next-steps",[2826],{"type":55,"value":2827},"Next steps",{"type":46,"tag":2249,"props":2829,"children":2830},{},[2831,2841,2853],{"type":46,"tag":78,"props":2832,"children":2833},{},[2834,2836,2840],{"type":55,"value":2835},"Build the pipeline these signals watch: ",{"type":46,"tag":365,"props":2837,"children":2838},{"href":476},[2839],{"type":55,"value":451},{"type":55,"value":453},{"type":46,"tag":78,"props":2842,"children":2843},{},[2844,2846,2852],{"type":55,"value":2845},"Shopify specifics, including the 0.5% failure-rate flag: ",{"type":46,"tag":365,"props":2847,"children":2849},{"href":2848},"/blog/shopify-webhooks/",[2850],{"type":55,"value":2851},"Shopify webhooks",{"type":55,"value":453},{"type":46,"tag":78,"props":2854,"children":2855},{},[2856,2858,2863,2865,2869],{"type":55,"value":2857},"Set up an ",{"type":46,"tag":365,"props":2859,"children":2860},{"href":2813},[2861],{"type":55,"value":2862},"alert policy",{"type":55,"value":2864}," in Webhook Relay, and test it against a failing ",{"type":46,"tag":365,"props":2866,"children":2867},{"href":2630},[2868],{"type":55,"value":2633},{"type":55,"value":453},{"type":46,"tag":179,"props":2871,"children":2873},{"id":2872},"monitoring-checklist",[2874],{"type":55,"value":2875},"Monitoring checklist",{"type":46,"tag":47,"props":2877,"children":2878},{},[2879,2881,2887],{"type":55,"value":2880},"This list covers what to watch. For the pipeline itself (storing, retrying, dead-lettering), use the ",{"type":46,"tag":365,"props":2882,"children":2884},{"href":2883},"/blog/reliable-webhook-delivery/#reliability-checklist",[2885],{"type":55,"value":2886},"reliability checklist",{"type":55,"value":453},{"type":46,"tag":2249,"props":2889,"children":2892},{"className":2890},[2891],"contains-task-list",[2893,2905,2914,2923,2932,2941,2950,2959,2973],{"type":46,"tag":78,"props":2894,"children":2897},{"className":2895},[2896],"task-list-item",[2898,2903],{"type":46,"tag":2899,"props":2900,"children":2902},"input",{"disabled":761,"type":2901},"checkbox",[],{"type":55,"value":2904}," Every source and destination has its own success rate, failure count and p95 latency.",{"type":46,"tag":78,"props":2906,"children":2908},{"className":2907},[2896],[2909,2912],{"type":46,"tag":2899,"props":2910,"children":2911},{"disabled":761,"type":2901},[],{"type":55,"value":2913}," Critical flows page on the first failed delivery; others alert above an error-rate threshold.",{"type":46,"tag":78,"props":2915,"children":2917},{"className":2916},[2896],[2918,2921],{"type":46,"tag":2899,"props":2919,"children":2920},{"disabled":761,"type":2901},[],{"type":55,"value":2922}," p95 acknowledgement time alerts at half the strictest provider timeout.",{"type":46,"tag":78,"props":2924,"children":2926},{"className":2925},[2896],[2927,2930],{"type":46,"tag":2899,"props":2928,"children":2929},{"disabled":761,"type":2901},[],{"type":55,"value":2931}," Retry backlog and dead letters are visible, and any new dead letter alerts.",{"type":46,"tag":78,"props":2933,"children":2935},{"className":2934},[2896],[2936,2939],{"type":46,"tag":2899,"props":2937,"children":2938},{"disabled":761,"type":2901},[],{"type":55,"value":2940}," Busy sources have a silence alert sized to their normal gap.",{"type":46,"tag":78,"props":2942,"children":2944},{"className":2943},[2896],[2945,2948],{"type":46,"tag":2899,"props":2946,"children":2947},{"disabled":761,"type":2901},[],{"type":55,"value":2949}," Your own request log keeps longer than the providers' (7 days for Shopify, 3 for GitHub).",{"type":46,"tag":78,"props":2951,"children":2953},{"className":2952},[2896],[2954,2957],{"type":46,"tag":2899,"props":2955,"children":2956},{"disabled":761,"type":2901},[],{"type":55,"value":2958}," Incidents resolve automatically after a clean window.",{"type":46,"tag":78,"props":2960,"children":2962},{"className":2961},[2896],[2963,2966,2968,2972],{"type":46,"tag":2899,"props":2964,"children":2965},{"disabled":761,"type":2901},[],{"type":55,"value":2967}," A reconciliation job covers events the provider never sent; see ",{"type":46,"tag":365,"props":2969,"children":2970},{"href":476},[2971],{"type":55,"value":451},{"type":55,"value":453},{"type":46,"tag":78,"props":2974,"children":2976},{"className":2975},[2896],[2977,2980],{"type":46,"tag":2899,"props":2978,"children":2979},{"disabled":761,"type":2901},[],{"type":55,"value":2981}," You have fired a test failure through each alert in the last quarter.",{"type":46,"tag":2983,"props":2984,"children":2985},"style",{},[2986],{"type":55,"value":2987},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}",{"title":7,"searchDepth":767,"depth":767,"links":2989},[2990,2994,2995,2996,2997,2998,2999,3000],{"id":181,"depth":757,"text":184,"children":2991},[2992,2993],{"id":457,"depth":767,"text":460},{"id":483,"depth":767,"text":486},{"id":502,"depth":757,"text":505},{"id":702,"depth":757,"text":705},{"id":2239,"depth":757,"text":2242},{"id":2638,"depth":757,"text":2641},{"id":2720,"depth":757,"text":2723},{"id":2824,"depth":757,"text":2827},{"id":2872,"depth":757,"text":2875},"markdown","content:blog:webhook-monitoring.md","content","blog/webhook-monitoring.md","blog/webhook-monitoring","md",{"loc":4},1794413898478]