Optimizing AI Incident Triage: Slow, Vague, or Confidently Wrong

When an AI triage agent falls short on a hard incident, it’s usually in one of three ways. It’s slow, because much of its time goes to relearning your environment. It’s vague, because nobody told it what the answer should look like. Or it’s confidently wrong, because nothing checked the answer before you acted on it. Each has its own fix, and none of them is waiting on the next model.

The goal isn’t a faster answer. It’s the fastest right one.

That is the target worth optimizing: the time from alert to an answer backed by evidence. Speed and accuracy aren’t separate goals here, because a wrong answer doesn’t stop the clock. The on-call engineer follows it, finds nothing, comes back, and the minutes start over.

This post is about the local shape of the job — the engineer and the agent on one screen, the pair-programming mode where the agent is free to explore and the engineer can redirect it. The server-side hot path is a different job with a different answer: fix the route in code. And “triage” means what it meant in TTC-Aware: scope and impact, which layer, what to check next, whether to escalate. Not the full causal chain — that comes after.

It also assumes the agent can already reach your systems. An agent without access produces vague analysis and hallucinated root causes for a simpler reason. This post starts where that one ends.

An experienced on-call engineer triages fast because of what’s already in their head: which metrics matter for this service, how the internal tools work, where latency problems usually come from, which checks can run at once, and when the evidence is enough. The agent starts the incident with none of that. Every fix below hands the agent one piece of what’s in your head.

The running example

One alert carries through the whole post. It’s illustrative, not a specific incident.

A p99 latency alert fires on checkout-api. Before anything else, one lookup: is this already known? An open incident in PagerDuty, or an outage posted on the status page of something the service depends on — the cloud region, the payments provider. If it hits, the investigation is mostly over. It’s a shortcut, not a cause.

After that, the latency can come from four places:

1. Load — more requests, or heavier ones: a traffic spike, a new request mix, bigger payloads, larger responses.
2. Change — something we shipped: a deploy, a config change, a feature flag.
3. Capacity — the service’s own resources: CPU, memory, GC, lost instances, throttling.
4. Dependencies — something the service waits on: the database, the cache, an internal service, the payments provider, and the network in between.

More work, different work, less room to do it, or waiting on someone else. Each branch has its own data source, and each one alone could explain the alert.

One twist, which matters later: a deploy went out at 13:40, and the new version leaks memory. Memory usage on every pod climbs slowly from that minute on; garbage collection runs more and more often, and CPU climbs with it. The latency alert fires at 16:45, three hours after the deploy — well outside the window anyone checks first. The CPU graph is real. It’s a symptom.

At a glance

The naive path

Hand that alert to an agent with tools and nothing else, and a typical run goes like this:

1. It reads the alert and learns the service name. It doesn’t know which of thousands of metrics belong to that service, so it lists them and greps for “latency.”
2. It picks a histogram that looks right, queries it, and gets a label wrong. Second try.
3. It wants to know whether any of the hosts behind the service are unhealthy. The internal fleet CLI can check them all at once, but the agent doesn’t know how: it runs --help, tries a flag the help text implied, and gets an error.
4. It remembers a CPU incident on this service last month and goes to CPU first. CPU is up. It never looks at memory usage.
5. It checks PagerDuty, then the deploy log for the last hour, then the logs — one after another, although none of them depends on the others. The deploy log is empty for that hour; the deploy that matters went out three hours ago.
6. A few minutes in, the screen is thirty tool calls long and there is still nothing the engineer can act on.
7. It answers. Either “latency appears elevated, potentially related to resource utilization or recent traffic patterns,” or “root cause: CPU saturation — scale up.”

Steps 1 through 6 are where the time goes: the agent is rediscovering what an engineer on this team already knew — which metrics matter, how the CLI works, which branch to try first, which checks can run at once.

Step 7 is where it goes wrong, for two different reasons. The first answer is vague because nothing told the agent what the answer should look like, so it covered every branch. The second is confidently wrong because of steps 4 and 5: the agent’s memory of last month sent it to CPU first, CPU was high, and a one-hour deploy window came back empty. Memory usage — the one signal that would have explained the CPU — was never checked, and nothing in the answer says so. The bet on where to look became the conclusion.

Every step is plausible. None of them is the agent being bad at reasoning.

Slow

There are two kinds of discovery in that run, and only one of them is waste.

Discovering the incident’s path — which of the four branches this incident wants — is the job. Locally, that exploration is the feature, and it stays.

Rediscovering the environment — which metrics exist, how the CLI works — produces the same answer on every incident, re-derived from scratch each time. That is the time to take back.

Relearning the environment and the tools

Two things in the naive run get re-derived on every incident: which metrics this service exposes, and how the internal CLI works. Neither has a reliable answer outside the source that defines them. The docs cover some metrics and miss the rest. --help covers some flags, or doesn’t exist at all. So the agent lists, greps, guesses, and tries again — every time.

The source is the one place with the complete answer. Every metric the service emits is registered somewhere in its code, with its name and labels; whatever the monitoring pipeline renames or derives is in that pipeline’s config. Every flag the CLI accepts is parsed somewhere in its code, including the ones nobody documented. That completeness is the point: the source knows things the docs and --help never will.

But nobody can read the source on every incident. It’s slow, and the answer is the same each time. So read it once, ahead of time, and keep the result as a cache the agent can look up in seconds — a skill.

Don’t make the agent learn your environment during the incident.

When the source changes, regenerate the cache. Hook it into the release of whatever it was built from — the service, the CLI, the monitoring config — so every version ships with a skill that matches it.

That trigger also sets the one rule for what goes in. The skill is refreshed when its source changes, so anything that can change without its source changing doesn’t belong in it: what latency usually is, a threshold set in runtime config, which hosts have been flaky lately, what caused the last incident. The skill can say where each of those lives — which metric, which config key — but not what it says today. The current value comes from a live query; the history comes from the agent’s memory. Write the value into the skill, and it becomes the stale report The Accidental Middle Layer warns about: quoted fluently, and out of date.

Not knowing what to check next

Give the agent every skill it needs and it still has to choose where to start. Load, change, capacity, dependencies — with nothing to go on, the order is luck.

Skills answer “how.” Playbooks answer “what next.”

A playbook for latency is a strategy, not a script: which branch usually carries the most information first, what rules a branch out, when to stop. For example:

• Check whether it’s already known before anything else. The lookup is cheap, and it can end the investigation.
• Check for a recent change next, for the same reason. “Recent” means since the symptoms began building, not the last hour — a leak can take hours to show.
• Separate the service’s own latency from its dependencies’ before going deeper into either.
• When CPU is up, ask what it’s spending it on.
• Treat a single metric moving as a lead, not a verdict.

The agent’s memory steers the order too. A CPU incident on this service last month puts CPU first this month, and when this incident differs, that memory points the wrong way. That’s consistent with memory being read, not run: it changes where the agent looks, never what it’s allowed to do. But where it looks is exactly what the playbook should decide — so the playbook ranks the branches and the agent’s memory breaks ties, not the other way round.

Checking in series what could run in parallel

The four branches don’t depend on each other, and neither does the lookup before them. Reading the deploy log doesn’t need the metrics; checking the payments provider’s status page doesn’t need the logs. Run one after another, their times add up. Run at once, the slowest one sets the pace.

Not everything splits. “Which service is affected” has to come before “which cluster’s logs,” and fanning out before that’s answered means fanning out in the wrong direction. Split the branches that are genuinely independent; keep the rest in order.

This isn’t the parallelism of the co-oncall post — that was many incidents at once, one thread each. This is one incident split into branches. Whether those branches run in one agent or several is an implementation choice, not what makes the investigation work. And fan-out has costs of its own: more tokens, duplicated queries, branches that come back with conflicting conclusions, and a synthesis step that has to reconcile them.

Tool calls on screen aren’t findings

Locally, every step is on screen. But a transcript of tool calls is not a finding. Five minutes of queries with nothing interpreted is still five minutes of silence — five minutes in which the engineer can’t redirect anything, because there’s nothing to react to.

Fan-out makes it worse. A subagent is out of view until it returns, so its whole runtime is time you can’t see.

That suggests a split: fan out the lookups, keep the deep dive. Subagents get the short, bounded tasks — check the vendors’ status pages, look for an open incident, list the last day’s deploys — a minute or two each, as a rule of thumb. The branch that needs digging stays in the main agent, where each finding prints as it lands.

Often the engineer knows before the agent does — the deploy list comes back with a 13:40 release, and they remember what was in it. They can prune, too: “skip the database and the cache — they’re on their own cluster, and the dashboards are flat.” That’s the pair-programming mode doing what it’s for.

A “return after a few queries” line in a subagent’s instructions is a bound only as strong as the agent’s compliance — a wish, not a fact. Keeping subagent tasks small enough that the bound rarely matters is the real control.

Keep the findings terse, one line each. Ambient presence still applies: a wall of prose per step is its own kind of silence.

Vague

Run the same alert three times and you get three different answers: a narrative, a list of five possible causes, a paragraph about CPU. None of them is exactly wrong. None of them answers what the on-call engineer wanted to know: which of the four branches is it, and what’s the evidence?

The agent answers in whatever format the run happens to land on, because nobody told it the format. So tell it. Write down the question and what the answer should look like — an output contract — and every run answers the same question the same way:

Checkout is slow on every pod; errors haven’t risen. It’s the 13:40 deploy: memory usage has climbed since v2.14 went out, and the CPU is garbage collection, not load.

Roll back v2.14. Memory usage started climbing with that deploy. Scaling up or restarting buys time, because fresh pods are fast — but they’ll climb too, so use it to cover the rollback, not instead of it.

That’s the whole report, in three parts: the answer up top, saying what’s hurting and which branch it’s in; the evidence, row by row; and what to do, including the move that only looks like a fix. The action gives its reason from the table, so the engineer can see how the agent got there, not just where it ended up. A branch the agent didn’t look at still gets a row, marked not checked, so it can’t quietly disappear the way memory usage did in the naive run.

Every row also says where its evidence came from, as something the engineer can open: the panel, set to the incident’s time range; the status page; for a CLI, the exact command, so anyone can run it again. “Checked the deploys, nothing recent” can’t be verified by anyone — and in the naive run it was wrong, because “recent” meant one hour. A link to the deploy history shows its time range at a glance. An agent that ran twenty queries and hands back only its conclusions is asking to be believed.

The format and the playbook are separate things. The playbook decides where the agent looks; the format decides what it hands back. Without a playbook, the agent may take a different path every time — but it can still report in the same format.

Put the format in a skill, with one filled-in report as an example — an example tends to work better than a list of formatting rules. And add an “other” row for anything that doesn’t fit a branch.

Confidently wrong

CPU is up. Is that a capacity problem, or a symptom of something else?

An agent that stops at the first metric that moved will say capacity — scale up — and say it confidently. The fix starts with carrying competing hypotheses — each with evidence for and against — instead of one story built from whatever turned up first. Then something has to check the answer before the engineer acts on it.

Asking the same model whether its answer is right doesn’t help much: a check that shares the agent’s blind spots tends to agree with it. But some checks don’t ask for an opinion. They ask a question the data answers.

A check the data answers can run on the same model. A check that needs judgment is where blind spots get shared.

So most checks can run on the same model:

• Every claim cites a tool result from this run. A sentence with no query behind it is dropped or marked unverified. The links come from the tools that ran the queries, not from the model’s own writing — and that part needs no model at all: code can check that every URL in the report came out of a tool result. A URL the model wrote is just another claim, and invented citation URLs are a common failure.
• Timeline. A cause comes before its effect. Memory usage started climbing at 13:40, the minute v2.14 went out, and not before; CPU followed hours later.
• Ask what else would be true. If it were load, the traffic would have changed — more requests, or heavier ones — and against the same hour last week, it hasn’t. If something is piling up in memory, pods that just restarted should be fast again — and they are. Each of those is one query.

The last one still leans on judgment: the model chooses what to predict, and it can read a result the way it wanted to. Asking questions the data can answer narrows that. It doesn’t close it.

Only when those checks can’t settle it — two branches both well supported, and the answer turns on which story fits better — is a different model family worth bringing in. It’s rarely needed, and it isn’t free: a second model means another full pass over the evidence, on the path, while the engineer waits. Keep it for the cases the data genuinely can’t decide.

These checks run inside one investigation, while the engineer waits for the answer. That’s different from the self-evolving loop, which audits past runs afterwards and fixes the rules for next time.

It also reframes a familiar complaint. An agent that declares a root cause after one query looks lazy. What looks like laziness is usually a missing stop condition — nothing told it what “enough evidence” meant, so the first plausible story was enough. The report sets that condition. It isn’t finished until every branch has a status and every claim has a source — and not checked is only acceptable for a branch that couldn’t explain what was found. If an unchecked branch could, the report doesn’t name a cause yet. It says what to check next.

A check is cheaper than a wrong answer

Every check costs time. The timeline, the what-else-would-be-true queries — each one adds to the run.

A wrong answer costs more. The engineer follows it and scales up. The new pods are fast, latency drops, and because the answer said CPU, the incident looks resolved — until the new pods climb too and the alert fires again. Scaling up wasn’t the mistake; stopping there was. The hour it bought went to waiting instead of rolling back, and the investigation starts over. That’s why the target is the fastest right answer — and a check that prevents a wrong one belongs on the fast path, not off it.

Measuring it

Re-run a past alert over its original time window, with and without each change. A past alert has one advantage: you already know what the answer turned out to be.

It also has a trap: today’s tools know how it ended. The PagerDuty incident now has a resolution note, and the agent’s memory may hold the postmortem. Give the agent only what it could have seen at the time — tool responses captured then, memory from before — or the run measures what it read, not what it worked out.

Then track two numbers:

• Time to a finished report — finished by the condition above, not by the agent deciding it’s done.
• Wrong answers — how often the report pointed at the wrong branch, or stopped short when the evidence at the time was enough to name one.

They’re the goal from the top of the post — the fastest right answer — split in two. Watch them together: a change that makes the report faster by making it wrong more often hasn’t made anything faster.

Where this doesn’t fit

The server-side hot path. Nobody is watching, so streaming findings and pruning branches don’t apply — and the path for a known alert type can be written down. Fix the route in code, pre-fetch, keep context tight: the other half of the split, with the funnel in front of it.

Novel incidents. The skills still help, and so does the contract. The playbook doesn’t, because nobody has seen this shape before — and a novel failure still needs a human at the front.

Small environments. If the agent can list every metric and every command in seconds, rediscovery is cheap, and skills are maintenance you don’t need yet.

Branches that aren’t independent. If each check depends on the answer to the last, fan-out doesn’t save time. It just runs the wrong queries in parallel.

Closing

An agent that’s slow, vague, or confidently wrong usually isn’t short of intelligence. It’s short of what the on-call engineer already knew, and of anything checking what it says.

Each failure mode has its own fix:

1. Slow — hand over what’s in your head before the incident: service skills and tool skills generated from source, and a playbook that ranks the branches. Fan out the lookups; keep the deep dive where you can see it.
2. Vague — tell the agent what the answer should look like: the answer first, saying what’s hurting and which branch; one row per branch, each with a status and a link to its evidence; and “not checked” counts as a status.
3. Confidently wrong — check every claim against the data on the same model, and bring in a different one only when the data can’t decide.

The goal was never a faster answer. It’s the fastest right one — and most of the minutes between the alert and that answer went to finding out what you already knew.