The same triage agent running on your laptop and running server-side off an alert webhook looks like the same agent — but how much agency you give the LLM should be completely different on each side.

Locally, you should let the LLM stretch out: hand it the MCPs you’ve set up for investigation, let it pick which tool to call, let it choose its own path, let it backtrack and try a different hypothesis. Discovery is a feature. You’re standing right there — every step shows up on your screen, and you can redirect at any time.

Server-side, you should deliberately take some of that agency back — not as a limit on the LLM, but as a fit to a different job. The local job is deep investigation: root-cause something nobody has seen before, take as long as the case takes. The server-side job is preliminary triage on a hot path: classify the incoming event quickly, pull the obvious data, post a first answer that lets the on-call decide what to do next. Different jobs, different tool surfaces. A free-roaming exploratory agent is the right shape for the first; a narrow, predictable, fast agent is the right shape for the second.

This post is about that split. Same reasoning core, loose on one side, tight on the other. The LLM’s agency shouldn’t look the same on both sides.

The cleanest way to see the split is in terms most engineers already live in: same agent, two modes of teamwork.

Local is pair programming. You and the LLM share one screen, the incident spread between you. It tries a query, you nudge; you notice a thread, it pulls. Either of you catches the other before a wrong path runs too far. The output is better than either of you alone because you’re in sync — discovery is fast precisely because two minds are reacting to the same screen.

Server is solo-on-runbook. Same agent, no peer at the screen. It executes the SOP you and it agreed on last week, when there was time to think. Inside the runbook it moves fast, decisive, capable — same skill, just no improvisation. Outside the runbook it stops and pages a human. The intelligence didn’t shrink; the scope of discretion did.

Local: let the LLM be the LLM

Local triage is the Solo mode — you drive, you see the output. The agent’s job here isn’t to “execute the steps you wrote.” Its job is to work alongside you: SSH, query, browse, stitch sequences of steps together. The thing that makes it useful is that it picks the right tool for the moment, on its own.

The design rule is simple: err on the side of more tools, not fewer, and let the agent choose.

  • Install the MCPs you’d actually reach for during an investigation. PagerDuty, Grafana, internal admin, log store, ticket system. The discovery cost is worth paying — you don’t know which system the next incident will pull you into, so don’t pre-decide for the agent.
  • Tool overlap is fine. When the same data has both an MCP and a CLI, keep both. The agent picks what it reaches for first; you’ll see the choice in the transcript afterward.
  • Don’t economize on context. Local inference latency is part of your reading speed. Adding ten more tool results to the context doesn’t feel slower to you, but it gives the agent another hypothesis to test.

The point of the local loadout is agency. You don’t know which path this incident wants, so you hand the LLM a wide tool surface and let it pick. You watch the path it takes; if a thread doesn’t pan out, you redirect.

Server: a different job, not a constrained agent

Server-side is the Headless mode — event-triggered, no human in the loop. The job here isn’t the same job as local: local is “investigate this novel thing carefully,” server is “classify this incoming event quickly, pull the canonical data, post a first answer fast enough that the on-call can decide what to do next.” Preliminary analysis at speed — that’s the brief. Match the agent’s surface area to that brief, not to the LLM’s potential.

Replace MCP with a fixed API where you can

MCP is the right tool for the local job — discovery, exploration, finding the path you didn’t know you’d need. On a hot-path server side — alert webhooks where seconds matter — the picture flips: you already know which APIs need to be called for this alert type, and MCP’s discovery layer becomes overhead the brief doesn’t ask for. Two distinct costs are worth separating:

The framework tax — paid for indirection you no longer need.

  1. Tool schema listing eats context. Ten MCP servers attached burns thousands of tokens describing tools before the agent does anything.
  2. Retries and timeouts compound. A flaky MCP server retrying transient errors adds dead seconds to every invocation; without tight timeouts and a circuit breaker, those seconds stack regardless of what the agent is doing.
  3. Round-trips compound. Every MCP call is a process boundary. Five tool calls is five round-trips you wouldn’t pay if the calls were inline.

The exploration tax — paid for letting the agent pick a path the brief already knows.

  1. Discovery cuts both ways. A buffet of tools means the agent samples. Locally that finds paths you wouldn’t have; server-side, where the path is known, sampling is latency the on-call waits through.
  2. Flexibility implies variability. Same task takes a different path each run — a feature locally, a debt server-side.

Framework tax wants fewer tool layers; exploration tax wants fewer choices — both converge on the same move.

Concretely: locally, the LLM drives a tool-call loop — picks a tool, MCP routes, the API responds, the LLM iterates until it has enough. Server-side: pre-fetch whatever can be pre-fetched. Your code knows which APIs this alert type needs; it fetches them up front and hands the results to the LLM as context. The LLM still calls MCP for whatever genuinely needs a runtime decision — but the surface is small and pre-picked. Most of what was tool-call iteration locally moves into plain code that runs before the LLM is invoked.

Notice what got moved: the LLM’s choice of which tools to use is gone, but its choice of how to read the data those tools return is still there. The LLM hasn’t left the loop — it’s still doing the interpretation, still deciding what the metrics mean, still writing the Slack update. Judgment shifted from tool selection to data interpretation — exactly the part of the work the server-side brief asks for, and exactly where the LLM is strongest.

Tight context is a latency and cost budget

The rule on this side is inverted from local: feed the LLM only what it absolutely needs to answer this alert, and nothing else.

A five-minute local run is fine. A five-minute server-side run means nobody is reading the alert in time. And every server-side run costs tokens — local runs you fire a few times a week; server-side runs fire on every alert. At ten thousand alerts a month, the difference between a 5k-token transcript and a 25k-token one scales linearly into real money.

Holding the budget means every tool result is squeezed before the agent sees it. Grafana doesn’t return the full series — it returns a summary plus a few salient samples. Log search doesn’t dump 1000 lines — it returns the top five clusters with counts. That compression isn’t the agent’s job; it’s the server-side client’s job, because that client knows up front what shape it should hand back.

Restricted commands are the production firewall

SSH wide open server-side is an incident waiting to happen. Locally it’s more workable — you’re watching, you can intervene — but it’s not zero risk either. Prompt injection, polluted tool descriptions, and a misread context don’t care that you’re at the keyboard. Even on the local side, destructive-command confirmation, clear prod/staging labels, secrets kept out of the model’s view, and an audit transcript are worth the small friction.

The defaults that matter:

  • Read-only is the default. Query metrics, tail logs, look up config — all green.
  • Writes go on an allow-list. Restart a service? Fine, but only the service named in the alert. Disable a feature flag? Fine, but only flags from a list that’s been vetted.
  • Anything global is hard-blocked. Drop table, kill -9 across hosts, deploy, scale to zero — these aren’t “needs confirmation” problems. The server-side agent should not have the capability at all. If it has to happen, a human presses the button.

The spirit of the line: the server-side agent’s blast radius is bounded by code, not by prompt. A prompt that says “don’t do dangerous things” is a wish. An allow-list is a fact. The first one breaks; the second one doesn’t.

Why you can’t just pick one

The two failure modes that show up most:

“It works great locally, let’s just plug it into the webhook.” The first production alert lands and the agent does what it does best locally — explores. Several minutes listing tools, more time on a tangentially-related host, more cross-referencing staging against prod. (The minute counts are illustrative — measure your own, the shape is the point.) None of those moves are wrong; they’re the local job done correctly. They’re just not the server-side job, which was “classify this in 30 seconds and post a first answer.”

“Server-side is so reliable, let’s just use that loadout locally too.” Now you’re debugging a novel incident and the agent can only run the few paths someone predefined. The shape that was a perfect fit for fast triage becomes a self-imposed cage on a job that calls for exploration.

One real edge case worth naming. When alerts are noisy or the system is too chaotic to predict the path ahead of time, hardcoding just freezes in misclassifications, and some server-side exploration earns its place — the trade becomes fast-but-wrong vs. slow-but-investigative. Most ops teams should still pick the former, since most alerts are known shapes; but for green-field instrumentation or systems still being mapped, the calculus genuinely flips.

It’s not a “which side is right” question. The two modes optimize for genuinely different things:

  • Local is built for deep investigation — hand the LLM a wide tool surface, let it explore.
  • Server is built for fast preliminary triage — pre-pick the path, keep blast radius small, let the LLM do the reading and summarizing.

What carries across both sides is the prompt, the model, the domain knowledge of this system. What doesn’t carry across is the job — and so the agent’s surface shouldn’t either.

How to design the split

In practice, slice the agent into three layers:

  1. Reasoning core — prompt, model, domain knowledge. Shared across both sides.
  2. Tool layer — MCP locally (the right shape for discovery), direct API client server-side (the right shape for known-path triage). Same reasoning core, different jobs.
  3. Output layer — terminal locally, Slack thread / ticket / metric server-side.

Once that split is clean, the workflow for adding a new alert type is:

  • Identify the APIs that should be called for this alert type. For most alert types you already know which ones — that’s exactly the engineering judgment that doesn’t need an LLM in the loop.
  • Write a server-side fixed client that calls those APIs directly, returning compressed shapes — just what the LLM needs to interpret, no more.
  • Pin the path’s command scope to an allow-list.

The whole goal on the server side is minimize MCP calls and minimize context — both spend the same precious time budget. Locally those costs don’t matter; you’re reading along, and exploration is the point. Server-side, you already know the answer for this alert shape, so cut anything that exists only to help discover it.

Closing

Same agent code, two deployment shapes.

Local: broad tool surface, generous context, commands wide open, human in the loop. The LLM is doing discovery, and breadth is what lets it find the path. Server: direct APIs instead of MCP, context tight, commands on an allow-list. The LLM is doing fast triage, and narrowness is what lets it be fast.

You need both — and you need to see that they shouldn’t look the same.