Most “agentic analytics” projects we’ve inherited share the same shape: a single LLM call wrapped in a function, dressed up with tool definitions, and pointed at a warehouse. They work in the demo. They fall over in week three. The teams whose agents make it to a steady-state production cadence are doing three things differently, and none of them are exotic. They route deliberately, they evaluate continuously, and they keep a human on the loop in places where the cost of being wrong is higher than the cost of waiting.

Routing patterns

The first mistake is using a single model for every task in the agent’s repertoire. Schema introspection, SQL drafting, narrative summarization, and chart-spec generation have wildly different latency, cost, and accuracy profiles. A serious routing layer treats them as separate jobs.

Intent classification first, model selection second. A small classifier — often a fine-tuned encoder or a cheap model with a tight prompt — decides what kind of request just came in: lookup, exploration, diagnostic, narrative, transformation. The classifier’s output drives both the prompt template and the model choice downstream. Lookups go to a fast, cheap model with a structured-output schema. Diagnostics get a stronger model and a longer context window. Narrative summaries get a model tuned for prose. The routing layer is boring code, not AI, and that’s the point.

Tool routing inside the agent. Once the request is classified, the agent shouldn’t see every tool in the catalog. Give it the three to seven tools relevant to the intent and nothing else. We’ve measured a 30–50% drop in spurious tool calls just from filtering the tool list at routing time. The agent that doesn’t see drop_table cannot accidentally call it.

Fallback chains, not retries. If the first model fails — schema mismatch, validation error, timeout — the next attempt should be a different model or a different prompt, not the same call repeated. Retry-the-same-thing is the most common cause of cost runaway we see in production agent logs.

Evaluation patterns

Evals are the part everyone agrees they should do and almost nobody actually does. The teams that ship treat evaluation as a build-time gate and a runtime monitor, with deliberately different scopes for each.

Build-time: golden datasets per intent. For each intent class the agent handles, you maintain a small set of canonical inputs with known-good outputs. Twenty to fifty examples per intent is enough to start. The eval suite runs on every prompt change, every model change, and every tool-definition change. It blocks deploys when accuracy drops below a threshold the team agreed on in advance — typically 90% on lookups, 75% on diagnostics, 60% on open-ended exploration. The thresholds are conservative on purpose; you want a noisy alarm when a prompt regression slips in.

Runtime: structural checks, not output grading. Grading agent outputs in production with another LLM is expensive and noisy. Instead, run cheap structural checks: did the SQL parse, did the result set return rows in the expected shape, did the tool call validate against its schema, did the narrative include the entities the user asked about. These checks catch 80% of regressions for 5% of the cost of LLM-as-judge.

Trace every call, sample for review. Every tool call, model invocation, and intermediate output gets logged with a stable trace ID. You don’t review all of them — you sample 1–5% for human review weekly, weighted toward sessions where structural checks failed or the user gave negative feedback. This is where you find the failure modes the golden dataset missed.

Human-in-loop patterns

The right question is not “should there be a human” but “where in the loop, doing what.” Agents that make every action interruptible become unusable. Agents that make no action interruptible become liabilities. The pattern that works is graduated trust by action class.

Read-only by default. Querying, summarizing, charting, exploring — these run autonomously. The agent does not ask permission to run a SELECT.

Confirmation before write. Anything that mutates state — writing back to the warehouse, sending a Slack message, updating a dashboard, creating a ticket — pauses for human confirmation. The confirmation surface should show the exact action in plain language plus the underlying tool call. We’ve found that surfacing both the natural-language intent and the structured payload catches roughly half of the agent errors that would otherwise reach production data.

Override channels for irreversibles. A small set of actions — schema changes, data deletions, customer-facing publishes — require an explicit override token from a named human, not just a click. The friction is the feature.

Replay-ready transcripts. Every human-in-loop checkpoint includes the full prior context as a replayable transcript. When an analyst overrides an agent decision, they can hand the replay to a teammate or to a future eval run. The override becomes training signal, not lost institutional knowledge.

The thread connecting all three pattern families is the same: agentic analytics is a systems-engineering problem dressed up as an AI problem. The teams that win treat the LLM as one component among several, and they spend their architecture budget on the boring scaffolding around it.