Runs a multi-turn conversation with the model, where the conversation structure is expressed via a configurable sequence of message builders. Use this for dialogue evaluation, agentic tasks, and any scenario that requires iterative model interaction.
Examples
Example: Multiturn Clarification. A system prompt instructs the model to ask for confirmation before answering questions about European countries. A judge model answers clarifying questions automatically until the evaluated model gives its final answer.
Multi Turn Solver
# ...definition: # ...solver:type:"multi_turn_solver"message_builders:-type:"chat_message"role:"system" content: | You are a helpful assistant. When asked about a question about countries in Europe, always repeat the question back to the user and ask for confirmation that you have understood the question. When asked a question about non-European countries, answer directly without asking for confirmation. In both cases, when providing the factual answer, do not include any additional explanations or punctuation, but only answer with a single word.-type:"chat_message"role:"user"content:"{{ sample.question }}"-type:"generate"-type:"generate_message"model_key:"<< config.judge_model >>"extra_input_messages:-type:"chat_message"role:"user" content: | If the previous message is a question, answer it using the information in the other messages. If the question is asking for confirmation, always answer 'Yes'. Otherwise, answer with '<done>'.output_role:"user"terminate_if:includes:"<done>"keep_iteration:false-type:"generate"
For a sample {"question": "What is the capital city of Austria?", "target": "Vienna"}, the solver builds the following conversation:
Turn
Role
Content
1
system
You are a helpful assistant. When asked about a question about countries in Europe, always repeat the question back...
2
user
What is the capital city of Austria?
3
assistant
Just to confirm - you're asking about the capital city of Austria, correct?
4
user
Yes(judge model response)
5
assistant
Vienna
Configuration
Properties
typeLiteral “multi_turn_solver”required
Type
message_formatenum TraceFormat
Default: open_responses
NotePossible TraceFormat values
The message format used by the solver for recording its output.
When set to open_responses (default), the solver converts input and output messages into Open Responses trace items and produces a SolverTrace wrapping a structured Trace object.
When set to latticeflow, the solver produces a SingleSolverOutput with LFMessage objects. This format is deprecated and will be removed in a future version.
Example: Static and Templated Messages. Inserts fixed system and templated user messages into the conversation before the first model call.
Chat Message Builder
type:"chat_message"role:"system"content: | You are a helpful assistant. When asked about a question about countries in Europe, always repeat the question back to the user and ask for confirmation that you have understood the question. When asked a question about non-European countries, answer directly without asking for confirmation. In both cases, when providing the factual answer, do not include any additional explanations or punctuation, but only answer with a single word.
Properties
typeLiteral “chat_message”required
Type
contentstringrequired
The chat message content. The prompt can refer to the following variable dynamically (using { } syntax):
In all scenarios:
sample: Sample attributes (ex: { sample.answer })
messages: The list of messages in the conversation (ex: { messages[0].content })
loop_index: (if used in a GenerateLoop) The index of the current iteration (ex: { loop_index })
roleenum ChatCompletionRolerequired
NotePossible ChatCompletionRole values
The role of the message sender, can be system, user, or assistant.
Generate a message using the specified model (and custom instructions, specified via extra input messages).
Example: Generate Message (Judge). Calls a secondary judge model to produce the next user turn. The conversation continues until the judge outputs <done>, at which point the loop terminates and that turn is discarded.
Generate Message
type:"generate_message"model_key:"<< config.judge_model >>"extra_input_messages:-type:"chat_message"role:"user" content: | If the previous message is a question, answer it using the information in the other messages. If the question is asking for confirmation, always answer 'Yes'. Otherwise, answer with '<done>'.output_role:"user"terminate_if:includes:"<done>"keep_iteration:false
Given the conversation so far ending with the model asking a clarifying question, the judge receives the full history plus an extra instruction message and replies:
Turn
Role
Content
…
assistant
Just to confirm - you're asking about the capital city of Austria?
extra
user
If the previous message is a question, answer it... Otherwise, answer with '<done>'.
new
user
Yes(appended as output_role: user)
On the next iteration, once the model gives its final answer, the judge outputs <done> and the loop terminates without appending that turn.
Additional input messages (in addition to the existing messages) given as input to the model.
include_conversationboolean
Whether to include the existing conversation in the model input. When true (the default) the conversation so far is prepended to the extra_input_messages. When false, the model starts from scratch and only receives the extra_input_messages; the conversation so far is still available for templating via the messages Jinja variable.
Default: True
output_roleenum ChatCompletionRole
The role of the output message.
Default: None
NotePossible ChatCompletionRole values
The role of the message sender, can be system, user, or assistant.
Execute a loop, where at each iteration the sequence of message builders is executed. Each message builder can optionally specify when to terminate the loop. There is a maximum number of iterations.
Example: Loop. Repeats a nested sequence of message builders up to max_iterations times. Here a judge answers confirmation questions until the model produces a final answer or the iteration cap is reached.
Generate Loop
type:"loop"max_iterations:3on_max_iterations:"continue"message_builders:-type:"generate_message"model_key:"<< config.judge_model >>"extra_input_messages:-type:"chat_message"role:"user" content: | If the previous message is a question asking for confirmation, answer 'Yes'. Otherwise, answer with '<done>'.output_role:"user"terminate_if:includes:"<done>"keep_iteration:false-type:"generate"
For a sample {"question": "What is the capital city of Austria?", "target": "Vienna"}, the loop runs at most 3 iterations:
Turn
Role
Content
1
system
You are a helpful assistant. When answering a question, always repeat it back...
2
user
What is the capital city of Austria?
3
assistant
Just to confirm - you're asking about the capital of Austria, correct?
Behavior when max_iterations is reached: continue stops the loop and moves to the next message builder, while error raises an error. This option is only meaningful when at least one nested message_builder has a termination condition. If no nested message_builder has a termination condition, the enum value is ignored and the behavior is always treated as continue (since the loop is expected to reach max_iterations, so raising an error is typically not useful).
Allowed Values:
continue
error
LoadTrace
Loads a conversation trace from a dataset column and injects it into the multi-turn conversation. The trace items become part of the message history for subsequent message builders.
Example: Load Trace. Injects a pre-recorded conversation from the dataset into the message history before further message builders run. Use this to continue or score an existing conversation without replaying it.
Load Trace
# tasks/task.yaml...definition: solver: type: "multi_turn_solver" message_builders: # Inject a pre-recorded conversation from the dataset into the message history. - type: "load_trace" trace_column: "conversation_trace" # Continue the conversation from where the loaded trace left off. - type: "generate"
Given a dataset row whose conversation_trace column contains a two-turn exchange:
Turn
Role
Content
1
user
What is the current temperature in Seoul?
2
assistant
The current temperature in Seoul is 11 C with partly cloudy skies.
3
user
And what about Berlin?
4
assistant
In Berlin it is currently 4 C and overcast.
load_trace replays all four turns verbatim; the subsequent generate step then calls the model to continue from turn 4.
Properties
typeLiteral “load_trace”required
Type
trace_columnstringrequired
The name of the dataset column that contains the trace to load. The value must be deserializable as a Trace object.
TerminationCondition
Properties
includesstringrequired
If the messages includes this string, the termination condition is met.
keep_iterationboolean
Within a loop, determines if all messages in the current iteration are kept or not, if the termination condition is met. Outside of a loop, decides if the message is kept or not if the termination condition is met.