This week we discussed Contrastive Language Models: A System One Model for Fast and Generalizable Decision-Making, from Stanford and NVIDIA Research. We came to it through Akshay Pachaar's explainer on Daily Dose of Data Science.
Agents make lots of small, repeated decisions: which tool to call, which patch to keep, which move to make next. Asking an LLM to generate each answer works, but it is slow when the options are already known. CLM treats these decisions as retrieval rather than generation: embed the state, embed each candidate, and pick the closest. The authors report that it matches Jev (TypeSafe's System One model) on computer-use, gaming and tool-calling tasks with up to 9× lower latency, and that a fine-tuned CLM sets a new state of the art as a verifier on DeepSWE and Terminal-Bench 2.1.
The setup
We agreed the architecture is the most straightforward part of the work. Two projection heads of about 20M parameters each, one for states and one for actions, sit on a frozen Qwen3-8B and are trained with a two-way InfoNCE loss, the same objective as CLIP. The formula is in the authors' Training Algorithm section. A softmax over the scores gives probabilities over the candidates, so an application can take the winner or escalate when the model is unsure.
Because states and actions are encoded separately, action embeddings can be cached. If an agent keeps choosing between the same moves, as in the T-Rex game, only the state needs encoding. The authors also report that with around 1,000 candidates, CLM is 13× faster than Jev.
Three stages, each a little harder
The Data Recipe section describes three training stages: plain Q&A pairs, then synthetic hard negatives (plausible but wrong answers), then real agent trajectories. Each stage asks for a finer distinction than the last.
The last stage turns CLM into a specialist, which risks catastrophic forgetting of what the earlier stages taught. The authors guard against it by replaying Q&A data, which makes up 40% of the post-training mix. With replay, accuracy on hard negatives barely moves (69% to 68.5%). Without it, accuracy falls to 56.2%.
Reading the charts more carefully than the headline
The speedups in the zero-shot chart are real but uneven. The 9× is the T-Rex game. Mario is about 4×, WikiRacing about 2.8× and tool calling about 1.6×.
Accuracy is where we pushed back hardest. CLM ties Jev on T-Rex and Mario, but on tool calling it scores 95.2% against 99.2%. That looks close until you compare error rates of 4.8% and 0.8%: roughly six times as many mistakes. On WikiRacing it succeeds 26 times out of 30, against 30 out of 30 for Jev. For a system that makes thousands of small decisions per task, that's a huge difference.
An uneven comparison
CLM's strongest results come from using it as a verifier. A frontier model writes several candidate solutions to a coding task, and the verifier picks which one to submit. CLM wins clearly here, but the comparison isn't even: CLM was fine-tuned on each benchmark, while Jev was used as-is. Jev actually does worse than just submitting the first candidate. Before giving CLM's architecture credit for the gap, we would want to see Jev fine-tuned the same way, especially since CLM is behind Jev on tool calling and WikiRacing when neither model is fine-tuned.
The fair counterpoint is that Jev can't be fine-tuned at the moment, so that may be part of the point. CLM's heads are cheap to fine-tune: only the projections train, on precomputed embeddings the authors publish. A model you can adapt to your own task is useful, even if the comparison flatters it.
We also noticed that fine-tuned results only appear for the verifier benchmarks. One guess was that the games are too simple to gain anything from fine-tuning.
Why not use hard negatives from the start?
The ablation that got us talking most is a small one, at the end of the Data Recipe section. Pre-training alone reaches 52.1% at ranking the right answer first, and a mid-training stage on hard negatives lifts that to 69.2%. Using hard negatives from the start peaks at 62.4% and then overfits. The loss is the same in both setups and only the timing differs, yet the authors don't really say why the order matters this much.
After the call, we found this is a whole research area, curriculum learning, from Curriculum Learning (Bengio et al., 2009) to When Do Curricula Work? (Wu et al., 2021). What follows are our own guesses from the call, without input from that field.
The first is a curriculum. In-batch negatives are mostly answers to unrelated questions, so they teach something coarse: this answer is on the right topic. Hard negatives are all on topic, so they teach which of two related answers is correct. Starting with them forces the model to learn both at once. You don't learn to run before you can walk. It reminded us of reinforcement-learning experiments where a simulated robot learns to stand before it learns to walk. Our first objection was that the staged run should simply get there faster, not end somewhere better. But if you cram everything into someone's head at once, they may never get there.
The second is noise. The hard negatives come from Gemini 2.5 Flash-Lite, and asking for a plausible wrong answer sometimes produces a right one, or one better than the reference. Early in training, that noise shapes what the model learns about the world. Later, the model already understands the domain and can take what is useful without being pulled off course. We found this convincing, and it fits the overfitting the authors see.
Are the probabilities worth trusting?
"Escalate if confidence is below 70%" only works if 70% means something, so the probabilities need to be well calibrated. Some of us were sceptical. Models trained this way tend to be overconfident, and Jev puts a lot of emphasis on calibration where CLM does not. The model card's Limitations section notes that the probabilities are relative to the candidates you supply, but we didn't find a calibration evaluation. The API returns a probability for every candidate, so this is easy to check, and possibly a post of its own.
Running it ourselves
The code is open and the weights are Apache 2.0, so in principle there isn't much to replicate. The catch is that the heads are encoder-locked to Qwen3-8B's embeddings, so a quantized or different encoder won't reproduce the reported numbers. The latencies were measured on an H100, so a Mac Studio will be slower. Our bet is that it would still be fast enough, and it would keep the decision loop on hardware we control. We wouldn't need to repeat the expensive stages either: fine-tuning only touches the heads, and synthetic data could fill in where we have no labels.
We left the call with two ideas. The first is to look at what other models of this kind exist and replicate the most interesting one on the Mac Studio. The second is to test whether the probabilities are calibrated.
The takeaway
Choosing between known options is what any Jev-like System One model does. What CLM changes is how it chooses. States and actions are encoded separately into the same space, so the options can be embedded once and reused, and each decision costs one embedding pass and a few dot products. That pays off most when the same options keep coming back.
We were less convinced by how the results are framed. The 9× is the best case, the zero-shot accuracy gap is larger than it looks, and the verifier headline compares a fine-tuned model against a zero-shot one.
The main intuition we took away came from the ablation rather than the benchmarks. Hard negatives helped most when they came second, and getting the order right was worth 7 points.