Pass Through Solver

Reads pre-recorded trace / conversation history from a dataset column instead of calling a model. Use this to evaluate outputs that were generated outside of LF AI Platform, or to replay a previous conversation.

Examples

Example: Pre-recorded QA Traces. Reads a full conversation trace from the trace column of the dataset and presents it to the scorers without calling any model. Useful for scoring outputs generated outside LF AI Platform or replaying a previous run.

Pass Through Solver

# ...
definition:
  # ...
  solver:
    type: "pass_through_solver"
    message_format: "open_responses"
    trace_column: "trace"

Given a dataset row whose trace column contains a pre-recorded single-turn exchange:

Turn	Role	Content
1	system	`You are a helpful assistant. Answer concisely with a single word.`
2	user	`What is the capital city of Japan?`
3	assistant	`Tokyo`

The solver replays all three turns verbatim. The last assistant message (Tokyo) becomes solver_output.output passed to the scorers.

Multi-turn traces with tool calls are also supported:

Turn	Role	Content
1	user	`What is the current temperature in Seoul?`
2	function_call	`get_current_weather({"city": "Seoul", "unit": "celsius"})`
3	function_call_output	`{"temperature": 11, "condition": "partly cloudy"}`
4	assistant	`The current temperature in Seoul is 11°C with partly cloudy skies.`
5	user	`And what about Berlin?`
6	function_call	`get_current_weather({"city": "Berlin", "unit": "celsius"})`
7	function_call_output	`{"temperature": 4, "condition": "overcast"}`
8	assistant	`In Berlin it is currently 4°C and overcast.`

Configuration

Properties

postprocessor PythonPostprocessorTemplate

An optional postprocessor applied to solver outputs before scoring.

Default: None

type Literal “pass_through_solver” required

The type of the solver.

trace_column string required

The name of the dataset column that contains the trace. The value must be deserializable as a Trace object.

message_format string

The message format used by the solver for recording its output. Defaults to open_responses; latticeflow is deprecated.

Default: open_responses

Related types

`PythonPostprocessorTemplate`

Properties

type Literal “python” required

The type of the postprocessor.

postprocess_snippet string required

Python source code defining a postprocess function with the following signature:

from latticeflow.core.dtypes import RawSample, SolverOutput


def postprocess(
    sample: RawSample,
    solver_output: SolverOutput,
) -> list[tuple[RawSample, SolverOutput]]:

Both def and async def are supported.