On-call for AI agents
Your agent said 200 OK.It was wrong.
AI agents fail without crashing: a polished answer built on a tool call that quietly failed. Antibody catches it in production, finds the commit that caused it, proves a fix on a family of variants, and calls you to approve it, by voice.
Patient · company-research agent
LIVE
Every response is 200 OK. One in five is wrong. Logs can't tell the difference.
Incidents fixed
—
Families guarding main
—
Deploy gate
—
Runs · last 10 min
—
01 · The failure you can't see
The brief looked perfect. The data behind it never arrived.
A real run from our sales-research agent on stripe.com, captured on Oct 11. No exception, no error log, a 200 response. Every dashboard showed green.
What the user saw
Stripe
200 OKStripe is a large fintech company with over 17,000 employees. Recent job postings indicate active hiring for partnership development, sales, and engineering roles, suggesting growth in partnership and product sales initiatives. No decision makers were identified because the people-search step returned no results.
Decision makers
None found
“People-search returned an empty result set; no contacts were available.”
Confident, specific, plausible. A sales rep would skip this account.
What actually happened
- Look up the companycrustdata.company_enrich
- Confirm it's the right companyllm.identify
- Search for decision makersHTTP_429
- Read hiring signalscrustdata.jobs
- Write the briefllm.brief
Antibody · tw_empty_after_error
people search failed (HTTP_429) but the brief reports no decision makers instead of saying the data is unavailable.
The people search call failed with HTTP_429. The agent swallowed the error and told the user there was nobody to contact.
02 · Replay a real incident
From silent failure to immune, in one loop.
RECORDEDReal data from Antibody's first incident on 2026-10-11: replayed, not live. Incident inc_20261011_1102_tw_empty_after_error.
11:02 UTC · production
A tripwire keeps firing on 200 OK responses.
No crash, no error-rate spike. The check saw the agent return a confident answer right after a tool call failed, again and again.
Detection
tw_empty_after_error
Fired in 10 min
11×
Opened by
detection run log
Incident opened automatically: inc_20261011_1102_tw_empty_after_error
03 · Why believe it
Measured on real failures, misses included.
We planted three bugs in the agent, each from a separate coding-agent session, then let real traffic find them. The backtest replays every saved family against the commit that caused it and the commit before it.
/3
planted bugs the gate would have blocked
/1
false alarms on clean commits
$
Antibody's LLM cost per incident
minutes from incident to proven fix
Predicted before it broke
“crustdata.people_search errors rose from 7% to 12% over the last 20 minutes (fit r2=0.85)”
A straight-line fit over tool error rates per window. Not a black box: the forecast shows its fit (r²).
The honest miss
invented contacts (prompt v2: at least 3 decision makers, no exceptions)
Never fired on real traffic: the model never actually invented anyone, so there was no incident to learn from. We list it instead of hiding it. 2 of 3 planted bugs were caught in production.
04 · What exists today
Monitoring tells you it's failing. Antibody closes the loop.
| LLM monitoring tools | LangSmith Engine (beta) | Sentry Seer | Antibody | |
|---|---|---|---|---|
| Flags bad runs in production | exceptions only | |||
| Finds the commit that caused a silent failure | needs revision tags | exceptions only | ||
| Quotes why the change was made | ||||
| Breeds a family of variants from one incident | reuses failing traces | |||
| Proves the fix on the whole family before merge | replays after the PR | |||
| Turns the incident into a deploy gate | CI evals you write | re-opens after it recurs | ||
| Approval by voice |
From each vendor's public documentation, October 2026. Monitoring tools: LangSmith, Braintrust, Arize, Langfuse, Datadog LLM Observability.
Built with
neatlogs
traces every run; detections open incidents
Entire
the reasoning behind the commit that broke it
Crustdata
real company data the agent researches
TypeSafe Jev
grounding score on every brief
Telegram + Sarvam
the on-call voice call
Watch it live.
The console shows the agent's real traffic, open incidents, the gate and the trust report, updating every two seconds.