LLMs are built to answer questions. We needed one that asks them.

A language model is built to produce the most probable continuation. It answers confidently, explains thoroughly and stays on the topic it was given. But a good hypothesis is rarely the most probable thing to say: many discoveries started somewhere unlikely, with a method borrowed from another field or a question framed in a way nobody expected.

This is something we wanted to address while building Parlium, a platform that helps scientific researchers develop their ideas and think about difficult problems. One of its two modes, night science, is there to help researchers come up with ideas nobody has had yet, without looking anything up. And that’s where a model built to answer questions falls short.

Our first version of night science simply asked. The prompt listed the conversational moves that stimulate creativity and when each one was useful. It didn’t work: in the conversations we read, most turns used none of these moves. The model kept answering and explaining, and rarely strayed from what the scientist had said.

So we stopped asking: we split the agent into four parts, each with one job.

• An observer reads the conversation and picks a move
• A writer turns that move into a short reply
• A checker sends the draft back if the move is missing, and
• A memory keeps track of what the scientist has already established. The writer no longer decides what to do, only how to do it.

We call this architecture Troupe: as in a theater company, the audience only sees the actor, here the writer, while the rest of the troupe works backstage.

In this post we explain how each part works, how we tested the whole thing, and why we think Troupe works well beyond our particular use case.

What are day science and night science?

Ask any scientist how research works and they will probably describe the scientific method: form a hypothesis, make predictions, test them against data, drop the hypothesis if the data disagree. That answer is correct, but it skips a step: where did the hypothesis come from?

François Jacob, who shared the 1965 Nobel Prize in Physiology or Medicine, called the testing day science and the part that comes before it night science. Itai Yanai and Martin Lercher, who have since founded the Night Science Institute, put the distinction in one sentence: “In day science, we falsify hypotheses and observe which are left standing; in night science, we create them.” [1]

The two forms of science also sound different. Day science is precise, planned and critical. Night science uses loose language and metaphors [2], and follows the “yes, and” rule from improv theatre where you build on an idea before criticising it. [3]

Parlium has a mode for each. In night mode, the agent doesn’t look anything up or judge the idea. It helps the researcher find and develop one of their own. Day mode then checks that idea against the literature and existing grants. A cell biologist might start in night mode with a hunch that a scaffolding protein matters more than people think, turn it into a hypothesis (maybe the protein is a drug target), then switch to day mode to see what papers and funding already exist.

Night science can be done alone, but Yanai and Lercher argue it works best in a pair [4]. Parlium’s night mode is built around this idea, with the agent playing the second person in the pair.

Evaluating a night science chatbot

We started from Yanai and Lercher’s papers and the Night Science Institute’s material [5], which describe what a good night-science partner does, and extracted two kinds of rules from them: principles a good partner must respect, and moves it can make.

What we call principles are rules the agent must follow in every reply, whatever the conversation. They describe the attitude of a good partner. For example, the agent should build on the scientist’s idea rather than dismiss it, and it shouldn’t pull the conversation into day science by asking for evidence or proposing experiments.

On the other hand, moves are specific things the agent can do in a reply to push the scientist’s thinking somewhere new, such as reframing the question or borrowing a concept from another field. No single reply can make them all, so a good partner picks the ones that fit the moment.

That gives two things to measure. The first is quantity and diversity: how many moves the agent makes, how varied they are, and how often the agent breaks a principle. The second is quality: how well the moves are made and the principles followed. A benchmark can count the first, but the second needs researchers’ feedback, so we started with quantity and diversity.

We turned each move and each principle into its own question, such as “Does the reply bring, in its own voice, a competing hypothesis?” for the move “raise a competing hypothesis”. For every agent reply, an LLM judge answers each of the 19 questions with Yes, No or N/A.

We grouped the 19 questions into seven families:

The two passing rules exist to keep the distinction between moves and principles. A principle family passes only if every one of its questions passes, since every reply must respect every principle. A move family passes if at least one of its questions does, and we don’t expect every move family to pass on every turn.

The last piece we’re missing is conversations for the judge to score. To generate them, we created a second agent that plays the role of a scientist and talks with the agent for 15 turns. Both the agent and the simulated scientists run on GLM 5.3 Flash, and the judge is Sonnet 5.5. We created 10 different scientist personas, each with their own problem and style. The judge scores every turn, giving two numbers per agent: the share of families passed and the share of individual questions passed, each averaged over all turns. We then compute an overall score which is the product of the two. The ten conversations and 19 questions together form our diversity benchmark.

How the diversity benchmark scores an agent, from simulated conversations to a single overall score.

Finally, we needed to know how much these numbers can move by chance. We ran the same agent three times on the same ten scientists and had the judge score the same transcripts twice. The result: the overall score varied by about one point, and the judge gave the same answer on 97% of the judgments. Using the minimal detectable change at 95% confidence, we established that below 3 points on the family score, or 2 points on the question score, a difference between two agents is noise.

Simple prompt optimization is not enough

Our first night-science agent was a single system prompt. It set the tone we wanted, listed the moves a good partner makes and when each one helps, and asked for short replies.

The model followed the instructions on tone and presentation, but not those on moves or length. We asked for 80 words at most, yet the median reply ran 135 words at turn 1 and 158 by turn 3. The prompt told the agent to use the moves, yet most turns used none.

We concluded that the problem was not the moves themselves but the choice we left to the model. A system prompt sets a general attitude well, such as being encouraging and not dismissing the user’s idea. But when the prompt offered a menu of moves and asked the model to pick the right one, it mostly picked none. Length is a separate problem: like any specific, checkable instruction, the limit was followed less as the prompt grew and the conversation went on.

Both scores fell turn after turn:

The averages also hide that some moves almost never appeared: when the model did choose, it fell back on the most generic ones.

Architecture after step 1: one agent does everything.

The observer: choosing the move for the model

To address this issue, we took the choice away from the model. This is the core of Troupe: an observer reads the conversation and picks a move, and a writer replies to the user by applying it. The writer no longer decides what to do, only how to do it well.

We wrote about our journey to make the writer actually listen to the observer in our blogpost “Your AI ignores your prompt (and what fixed it for us)”, and that’s when we added the checker, the third part of our agent. Here, we will only focus on how the observer chooses the move.

This observer then raised a new question: how should it choose the moves? We tested several configurations:

Jev picks the move directly

Our first observer was Jev, TypeSafe AI’s classifier model that picks from fixed options instead of generating text [6]. That seemed a natural fit: before each reply, Jev picked two moves from the list for the writer to apply. We chose two because it felt like the right balance: enough to make each reply rich, but few enough to leave room to develop them properly. It will be adapted later on.

We dropped it quickly, because it ran into the same problem as the single prompt: Jev kept picking the same few generic moves. Our guess is that it played it safe: asked for the best move, Jev picks the one that fits the most situations, not the ones that would actually help the user at that moment. Others have seen the same thing in LLMs that choose conversation strategies for emotional support [7]: they fall back on a few favorites. Their fix was a planner (their equivalent to our observer) trained on real data, so it learned how often humans actually use each strategy. Night science has no such dataset, so we needed another way to stop the observer playing it safe. As you’ll see, we ended up bringing Jev back later, but for a task it is better suited to.

The move is picked at random

So we tried the opposite: each turn, we drew two moves at random from those the agent doesn’t already make on its own, such as asking questions or using metaphors. A random draw has no favourites, so every move gets used. And since night science thrives on the unexpected, a surprising move might be a feature rather than a bug.

Random selection gave our best benchmark results so far: Parlium used every move, and diversity was nearly perfect, as you’d expect from random draws.

The risk was that a random move wouldn’t always fit. We don’t want Parlium playing devil’s advocate with a user who doesn’t have an idea yet. For a first estimate, over 150 turns, we asked Sonnet 5.5 whether each move drawn by the random observer would be irrelevant on the next turn. Depending on the move, it judged 10 to 30% of them irrelevant, mostly because the move had just been made: a repeated move was judged irrelevant 59% of the time on average.

That suggests avoiding repeats would solve most of the problem. But in our own tests, some drawn moves clashed with the conversation for reasons that had nothing to do with repetition. To choose well, the observer needed to know where the conversation stood.

Picking moves according to the context of the conversation

Both observers so far chose the moves directly. Jev always gave us the same safe few; the random draw used them all but ignored the conversation. Our intuition was that the best moves depend on the “state” of the conversation, where the scientist stands: e.g. are they still looking for an idea, or do they have one to develop? That’s what we call the conversation “state”.

Recognizing the state was a task we felt more confident handing to Jev, since classifying text into categories is what it was built for. To keep the benefits of randomness, we kept the random draw, but made it depend on the state: instead of a uniform draw over all moves, each state gets its own distribution over the moves that make sense there.

We started with eleven states, but that was too many. The boundaries between them were blurry, and many of them called for the same moves anyway. Since we only use states for their move distribution, two states that call for the same moves should be combined. We merged them until every state changed what Parlium should do next, which left four, following how an idea develops:

State	The scientist…
Searching	Is searching for an idea: they have no explanation of their own yet, or one that ran out of road
Forming	Has a forming idea: a hunch or a mechanism that is still open for expansion
Solid	Has a solid idea: they present a settled explanation, or ask for a verdict.
Housekeeping	Only greets, signs off, or wants to capture an idea

In the first three states, each move gets a weight from 0 to 3, which we set by hand, and is drawn with a probability proportional to it. For instance, a searching scientist has no hypothesis of their own yet. So the move “competing hypothesis” gets 0 in searching, while “narrowing their question” gets 3 and comes up about a quarter of the time. Rarer moves like raising a bias still appear now and then, which keeps the surprise of the random draw. A move weighted 0 can’t be drawn, nor can the two moves used on the previous turn, which removes the repetition problem we measured earlier.

The last state, housekeeping, draws no moves, since there’s nothing for night science to do.

This became the quasi-final version of our observer: it scored as well as full random sampling on the benchmark, without drawing moves out of place.

Architecture after step 2: an observer chooses the move.

Using feedback to polish the agent for actual users

At this point we were confident enough in the agent to put it in front of an expert. Dr Oliver Bogler, scientific advisor at Barnacle Labs and involved in the Night Science Institute from its early days, agreed to try it.

He made three remarks. First, the agent kept steering toward experiments and how an idea could be proven, instead of developing the idea itself. Second, it was hyperactive: too many metaphors, too many questions, too many leads at once. Third, it lost track of the constraints the scientist had set and proposed leads that were out of bounds. Together, these made him feel like he was talking to “someone who wants me to focus on their thoughts, rather than helping me develop mine.” He spent more effort working out what the agent meant and where it was going than thinking about his problem. This was the half of the evaluation our benchmark couldn’t do: it rewarded making moves, however they were made, so it couldn’t see any of this. From then on, we improved the agent by hand, using Oliver’s feedback and our own.

Each remark led to a change. To calm the agent down, we cut replies to about 40 words, and the observer now asks for one move per turn instead of two. To stop it sounding like a reviewer who sets itself up as the scientist’s equal, or even superior, we added a section on tone to the system prompt. To help it keep track of the problem, we added a memory: after each turn, an additional model (DeepSeek V4 Flash) updates a record of the scientist’s constraints and the metaphors already in play, so the agent builds on an existing metaphor rather than inventing a new one each turn.

The last question was how a session should end. It isn’t for the agent to decide when the scientist has a good enough idea. The signal is a change in what the scientist does: once they are happy with an idea, they start checking it, drafting experiments and protocols or asking for papers and data. That is day science, where Parlium’s other mode takes over. So we added a fifth state to the observer, “wants day science”. It detects this shift, and makes Parlium point the scientist to the day-science mode after offering to save the idea so they can take it across.

The final architecture, as used in Parlium’s night science mode.

Results

The result is an agent we are confident in, one that helps people come up with new ideas rather than handing them an answer. The main lesson we learned was that asking nicely wasn’t enough: given a choice, a model falls back on what it does by default, or on the safest answer.

What worked was our Troupe architecture: not trusting the model with everything but splitting it into four parts, each with one job. The observer uses Jev to classify where the scientist stands, then draws a move from that state’s weight table. The memory keeps track of the scientist’s constraints and the metaphors already in play. The writer turns all of this into a short reply that makes the chosen move in the right tone, and the checker sends the draft back if it missed. The writer never decides what to do, only how to do it well.

The idea isn’t entirely new. Before LLMs, chatbots such as booking assistants already separated understanding the user, choosing an action and writing the reply, which usually meant filling in a pre-written template. LLM chatbots mostly dropped that separation and let one model do everything. Troupe brings it back, with an LLM to write what no template could, and a checker to make sure it does what it was asked.

Nothing in Troupe is specific to night science. The only fixed rule is that the model writing the reply doesn’t decide what to do; the other parts can change. A sales team, for instance, could build a chatbot that guides customers without sounding pushy. They would decide that a customer is either browsing or ready to buy, and that checkout is only suggested once they’re ready. In each conversation, the observer then tells which stage the customer is at, and the chatbot follows the team’s plan.

References

1. Yanai, I., Lercher, M. Night science. Genome Biol 20, 179 (2019). https://doi.org/10.1186/s13059-019-1800-6
2. Yanai, I., Lercher, M. The two languages of science. Genome Biol 21, 147 (2020). https://doi.org/10.1186/s13059-020-02057-5
3. Yanai, I., Lercher, M. Improvisational science. Genome Biol 23, 4 (2022). https://doi.org/10.1186/s13059-021-02575-w
4. Yanai, I., Lercher, M.J. It takes two to think. Nat Biotechnol 42, 18–19 (2024). https://doi.org/10.1038/s41587-023-02074-2
5. Night Science Institute, “Read”. https://night-science.org/home/learn/read/
6. TypeSafe AI, “Jev documentation”. https://docs.typesafe.ai
7. Kang, D., Kim, S., Kwon, T., et al. Can Large Language Models be Good Emotional Supporter? Mitigating Preference Bias on Emotional Support Conversation. ACL 2024. https://arxiv.org/abs/2402.13211

Acknowledgments

Thanks to Dr Oliver Bogler for testing the agent and for the feedback that shaped its final version.