Running AI agents on small models — production lessons with the Vercel AI SDK

Claude Sonnet costs $2 per million input tokens and $10 per million output. Nvidia Nemotron 3.5 Lightning costs $0.10 and $0.25. Twenty times cheaper on input, forty times on output, and since price follows compute reasonably closely, it’s a similar story on the energy consumed. You can prefer a small model because it’s cheaper, or because it uses less energy — there probably isn’t anyone who cares about neither.

Our Barnacle Intel chat agent now runs on the small model. Getting it to work took a bit of work, but none of it required a smarter model.

Barnacle Intel is a set of agents that collect and categorise AI news into a knowledge graph, with a chat agent sitting on top so you can ask questions of it. Answering a question properly means navigating that graph, which can take a dozen tool calls in sequence. It was originally built using Claude Sonnet for the chat agent.

When I swapped in a small model, it failed immediately. The agent would fire off a single search query and stop, rather than working through the graph. When it did attempt something more involved, it fell over at random. My first read was the obvious one: small models cannot sustain multi-step tool use, that’s why we pay for frontier models. Frustrating, but maybe that’s just how life is.

But that was wrong, or at least incomplete. Going through the traces, almost every failure turned out to be mechanical and repeatable. Wrong tool name. Wrong parameter name. A skipped step. Odd typography in the output. The model's intent was nearly always right and nearly always legible. I could see there was hope.

That changed the problem. I did not need a cleverer model, I needed a harness that expected the model to be “almost right” and that could handle errors gracefully. This post is what that took. Some of the detail is specific to the Vercel AI SDK, which we are using. Other harnesses may have similar concepts, although the Vercel one appears like it was designed by someone who understands these realities better than most. I recommend it.

First, give it a strategy to follow

The biggest single failure was the one that looked most like a lack of intelligence: the model could not work out how to navigate the graph to answer a question. It would reach for a search, get something back, and stop.

Thinking about it, though, people do not ask random questions, they ask questions that fit a small number of patterns. For example, something about a specific company or product. Or a comparison between two things. Or a broad landscape question like "what is happening with coding agents?". Each pattern maps to a different way of navigating the graph. A comparison means resolving the entity name for each side, then querying news items against each entity. A landscape question means something else entirely. Once I realised there were patterns for the question types, it was obvious that each question type needed a common pattern for how to navigate the graph.

So rather than asking the model to invent a navigation strategy, I wrote the strategies down. Each one is a skill: a set of instructions for how to work through the graph to answer one type of question. The model's job is reduced to categorising the question and calling use_skill, which loads the right instructions.

This worked far better than expected, and the reason is interesting. I replaced an open-ended reasoning task, which small models are bad at, with a classification task, which they are good at. As a bonus, the randomness that’s evident even with large models, went away. Two identical questions now get answered the same way.

It left one problem: making sure the model actually calls use_skill before doing anything else. Big models follow that instruction, but small models frequently skipped it. I tried three enforcement designs, and the two that failed are as instructive as the one that worked.

Attempt one, force the call. The Vercel AI SDK lets you force the model's next action with toolChoice, so we forced use_skill as the first call of every answer. But some messages need no tools at all, and when a user simply said "thanks", the model, forced to call a tool anyway, spiralled into six thousand tokens of confusion. Forcing removes the model's ability to do the sensible thing.

Attempt two, hide the other tools. The SDK's prepareStep hook lets you adjust each step of the loop, including which tools are visible. So until use_skill had been called, I made it the only tool available. Elegant, and it broke the best models. A well-behaved model declares its skill and starts working in the same step, as a batch of parallel tool calls. The SDK validates all of a step's calls before running any of them, so at validation time the declaration had not yet happened, and the legitimate follow-on work got rejected. The structural lesson is that a rule about ordering cannot be enforced at a stage that processes things in batches.

Attempt three, enforce at execution. In the AI SDK each tool is an object with an execute function, plain code you wrote, so I wrapped every tool except use_skill in a three-line guard. Until the declaration has run, they do not do their work. They return the message "declare your skill first, then retry this exact call". Well-behaved models never see it. A model that jumps ahead gets that sentence back as the tool result, reads it, declares, retries, and carries on. That is the version that worked.

Then, a harness that expects almost right (based on Vercel AI SDK)

With strategy handled, what remained was a long tail of small mechanical failures. Every one of them is predictable, which means every fix is mechanical too.

Repairing input

Lesson 1: small models blend tool names, so repair the call rather than failing it.

An agent works by giving the model a list of tools it can call. Ours has hybrid_search, search_items, get_item and others. A big model calls them by their exact names. A small model, having seen that list, will sooner or later call search, query_items or fetch_item. Names that do not exist, but which are obvious blends of the naming scheme it was shown. The intent is completely clear and only the label is wrong. Failing the call wastes a round trip and confuses the model.

The fix is to intercept the call and correct the name before it errors. The AI SDK has a hook for exactly this, repairToolCall, which fires whenever the model requests a tool that does not exist and can return a corrected version. Ours checks a hand-maintained alias map first, so query_items becomes search_items, then falls back to closest match by spelling, and gives up if nothing is close, letting the error reach the model. One subtlety once you are doing anything clever here: repair against the tools available right now, which the error object carries, not against your full catalogue. The two can differ.

Lesson 2: they blend parameter names across tools too.

Each tool takes named inputs, like function arguments. When nearly every search tool takes a query parameter, a small model will call the entity-lookup tool with {query: "Mistral"}, even though that one takes name. Obvious intent, wrong label, again.

The same repairToolCall hook fires when a tool is called with inputs that fail validation, so I keep a small per-tool table of wrong name to right name and rename the keys before validation runs again. One hard-won detail: the SDK gives you one repair attempt per call. If you fix a tool's name, clean up its inputs in the same pass, because if the repaired call then fails input validation you do not get a second chance.

Lesson 3: expect Python-isms and case slips in structured input.

Small models fill optional fields with the literal string "None", which is Python's null and which then fails a date format check. They capitalise things that should be lowercase, offering "Tracker" where the allowed value is "tracker".

In the same repair pass, treat null-ish strings, meaning "None", "null" and empty, as "field not provided" and delete them. Lowercase values that belong to a fixed set. "None" does not mean the model wants the word None. It means not set. Honour the intent.

The principle underneath

When a small model gets something wrong, there are two kinds of failure you can hand back. A generic protocol error, "invalid tool call". Or a plain sentence saying what to do instead. The difference in recovery is dramatic. Small models are surprisingly good at following a correction and bad at inferring what a generic failure meant.

Write your error messages for the model the way you would write them for a junior colleague.

Repairing output

Lesson 4: sanitise output on the server, because models typeset in ways your parsers do not expect.

Our answers cite sources inline as [2026-08-13#2], which the UI turns into a clickable button. Small models produce that citation with a non-breaking hyphen instead of a normal one, U+2011, which looks identical and is not. Or wrapped in CJK brackets, 【…】. Or fullwidth brackets, ［…］. Every variant looks right to a human and fails a naive pattern match, so the citation renders as dead text.

The fix is not to demand the model match your format. Our experience was that there’s always another edge case with a new model. Instead, the answer is to transform whatever the model produces, server side, before it reaches the browser. The SDK lets you pipe streamed output through experimental_transform. Ours normalises look-alike hyphens inside citation dates, wraps any valid citation in the correct brackets wherever and however it appears, which makes the model's bracket choice irrelevant, removes leftover decorative brackets, and deletes citations that do not point at a real source. Stream processing has one tricky requirement, in that you cannot un-send text you have already streamed, so the transform holds back any still-forming citation until it is complete. The principle is simple though. One transformer on the server beats format instructions in the prompt.

Not fooling yourself

Lesson 5: test the guard, not just the output.

This was a humbling one. A library upgrade broke that output-cleaning transform and it was silently dead for a week. The upgrade had renamed the field our transform read, delta became text. Our code read the old name, got undefined, and cleaned nothing, while the original text flowed through a field we were not touching. TypeScript would have caught it, but the code had opted out of type checking at exactly that spot.

Nothing looked broken, because models often emit clean citations anyway. That is the trap. A guardrail against an intermittent failure also fails intermittently, which is indistinguishable from working.

Two fixes. Test the guard directly by feeding it deliberately malformed input and watching it make the correction, rather than eyeballing good output. And at the boundary between your code and a fast-moving library, use the library's real types rather than any, so the next renamed field breaks your build instead of silently disabling a safety layer. When a feature is prefixed experimental_, pay attention to library upgrades.

Lesson 6: you only find any of this if you can see the traces.

Every fix above started the same way. A user said an answer looked off, and I looked up exactly what the model did. Every tool call, its exact inputs, its exact result. I store all of it. The SDK's onEnd callback hands you the full step history when an answer finishes, and it is worth handling onAbort too, or the turns users abandon mid-answer vanish from your records entirely.

Without traces, all of these bugs present identically as "small models are flaky". With traces, each problem becomes specific, mechanical and fixable in an afternoon. Observability is not an add-on for agent systems, it is the only way to debug them. Amusingly, Claude Code kept trying to convince me it was a lost cause trying to make a small model work and it was only my perseverence and frequent rebukes that forced it to help me find a solution to each problem.

The takeaway

None of this made the model smarter. It made the system forgiving of the specific, predictable ways small models are clumsy. Wrong labels, wrong fields, skipped steps, odd typography.

That is the difference between "small models are flaky" and an agent running comfortably on a model a fraction of the size and price of the frontier. The intelligence you need is usually already there, what is missing is a harness that tolerates predictable mistakes.

You get to choose your motivation for thinking this is interesting. Either you want to save a ton of money, or you want to save a ton of energy. Either is a worthy cause.

There's now a follow-up to this post: When the small model knows to call for help covers what happens when a question genuinely is beyond the small model — and how the agent recognises those moments and escalates them to a frontier model.