Overview

The solver decides how a benchmark task turns one dataset sample into something scoreable. It is the only part of the task that talks to the model under test, and every task defines exactly one solver inline under definition > solver.

Pick the solver that matches the interaction you need, then follow it through to its own page.

Multi Turn

multi_turn_solver - later turns depend on earlier ones.

Single Turn

single_turn_solver - one request per sample, one response.

Python

python - none of the declarative types can express the interaction.

Pass Through

pass_through_solver - the response already exists in the dataset.

Post-processor

postprocessor - optional on any solver, splits one output into several scored rows.

Once the solver produces output, Scoring turns it into scores and metrics.