When the small model knows to call for help: building model escalation into the agent

Our Barnacle Intel chat runs on small, EU-hosted open-weight models by default — that's the point of it. The model doing most of the work is Nvidia's Nemotron Lightning, and the economics are a big part of why: per token it costs roughly one twentieth of Claude Sonnet for input and one fourtieth for output. At those prices you can afford to let people actually use the thing.

Most of the time, a small model is all you need. Given the right harness — search tools over a curated corpus, and skills that tell it how to work through each type of question — Nemotron answers the majority of questions well. I wrote previously about what it took to get an agent running reliably on a small model; this post picks up where that one left off.

But Nomotron Lightning is still a small model, with a small model's limitations. Some questions genuinely need the deeper reasoning of a frontier model, and I didn't want those questions to get a mediocre answer just because the default model is cheap. So I wanted the agent to recognise those moments itself and hand the conversation over to Sonnet. This post explains how I built that, including where the signal to make the decision comes from and where the switching lives.

The signal

My agent starts every research answer by declaring which skill it's about to use. The skills explain to the agent how to navigate the underlying knowledge graph of data for a specific question type. e.g. a question might be around tracking a specific entity (company, product, technology), comparing two companies, explaining how a story has developed, mapping the landscape of a subject like coding agents, etc. Each of these different question types can be answered by undertaking a set of tool calls and these are described in the skill.

I realised that the selection of the right tool is actually a question categorisation and that this could also include an escape hatch — returning “other” when the question doesn’t fit any of the pre-determined types.

I don’t actually need a way to assess the complexity of a question (which is hard), I just needed to assess that the question doesn’t fit one of the pre-determined categories. Small models are poor at evaluating complexity, but decent at recognition: matching a question against seven named patterns is a recognition task. When the model declares other, it isn't saying “I’m not smart enough”; it's saying this question has a shape I have no procedure for, which conveniently happens to be the same thing.

The mechanics

The declaration happens on the first step of the agent loop. My router (a ~90-line module) watches for it: when the declared skill is “other”, every remaining step of that answer runs on the frontier model — the Vercel AI SDK’s prepareStep hook lets you swap models between steps of a single response. Anything the small model had already done stays done and the expensive model takes over for the remaining reasoning and synthesis.

I made escalation budget-guarded, so I don’t get any surprise bills. Each escalated answer consumes a slot from a tight hourly rate limit. The answer carries a visible banner — "escalated to frontier model (non-EU) — question beyond small model capability." When the budget is exhausted, that's shown as: "a small model answered — treat with care."

The policy itself is four lines of code in the router:

export const ROUTING_POLICY = {
  escalate: {
    whenSkillIn: ['other'],
    unlessAlreadyFrontier: true,
    budget: 'frontier-slot',
  },
} as const;

This logic makes it especially easy to change — if I decide that one of the existing skills really needs the extra intelligence of a frontier model, I just add its name alongside “other” in that piece of code.

One design decision worth calling out: escalation is per-answer, not per-conversation. The router works on every request, so every new question in a conversation go through the whole process again — declare a skill, route accordingly. If the follow-up to an escalated answer matches one of the normal skills, the small model answers it. There's a nice bonus hiding in that: the small model gets to read the frontier model's work. A follow-up like “expand on the second point” is answered by the small model with the frontier answer sitting right there in its context — the expensive reasoning was bought once and gets reused for free.

This is the opposite of how most routing layers behave. Sticky routing — Anthropic's own server-side fallbacks work this way — keeps a conversation on the switched-to model once it has moved. Stickiness optimises for consistency of voice; per-answer routing optimises for cost, and it matches what my trigger actually means: “other” is a property of the question, not of the conversation. A conversation that needed one hard, escalated answer drops straight back to cheap EU-hosted inference for its follow-ups. The flip side is that a user riffing on genuinely hard questions escalates on each one and can exhaust the hourly budget — at which point they get the “treat with care” notice rather than a silent downgrade, which is the behaviour I want. But that's my design choice — I could just as easily relax the rate limit. In practice, very few questions get escalated and so I don't think that is necessary.

Why not a routing proxy?

There's a genre of infrastructure for exactly this problem and I did consider it, because at first glance it seemed the obvious answer. NVIDIA's Switchyard is a nice example: a Rust proxy that sits between your app and model backends, translating API formats and routing requests — including an escalation mode where every request runs on a weak model first and a judge decides whether to re-run it on a strong one. I read it closely and took the escalation idea — and deliberately didn't take the rest. There were two reasons.

Firstly, the routing signal naturally lives inside the agent loop. A proxy like Switchyard sees a raw request and must manufacture its own signal — an extra classifier call, or a judge model evaluating finished answers. Our signal is an LLM call the agent has already made, step one of the loop. In effect, I’m getting the decision to escalate for free. By the time a proxy could form an opinion, my router has already acted on better information.

Secondly, my solution means there’s one less thing to run. A proxy is another independent service that needs to be deployed, monitored, versioned. It’s also another hop on every request and something else that might break. My chat runs as Vercel serverless functions — there's nowhere for a daemon to live, and standing up a VM to host one would have added a dependency I don’t need.

The counterpoint: a proxy is the right answer when many applications share one routing policy, when you can't modify the applications, or when you need org-level cost enforcement in one place. We have one app whose harness we own end to end and for that case, routing is most naturally a feature of the agent.

What I’d tell you to steal

Categorisation of an incoming request using an LLM can give your agent a lot of information about how process the request — which skill to load and whether to escalate to a more capable model. That categorisation needs to be performed by a small and fast model, something like Nvidia Nemotron Lightning, in order to minimise the latency added. There are better models, but better equates to slower. Nemotron Lightning is a nice mixture of “good enough for this task” and “fast enough to minimise latency”.

The other thing worth stealing is to log everything. The telemetry I’d stored included every conversation's declared skill and tool trace. The data helped me identify that more complex escalation solutions, for example escalating potentially on any LLM call, wasn’t worth the extra cost and latency. Keeping it simple, with a single decision point at the start of the agent loop was all that was needed.

The results

If my agent escalated most of its questions to the frontier model, that would invalidate the whole point of the design. That design is there to answer most queries using a small model and that’s exactly what it does. Since introducing the escalation feature, less than one percent of questions have been escalated. That proves that the skill-based solution, that provides graph navigation instructions for different question types, works. The escalation described in this post provides a useful escape valve and ensures even questions that are outside of my expectations get a good answer. Over time, I can scrutinise the logs and probably identify more question types that can have a skill written for them, further reducing the proportion that get escalated. Using this approach I’ve been able to serve almost everything from a model that costs one thirtieth of the frontier for input and one sixtieth for output — that feels like a win.