Overview

Scoring turns what the solver produced into numbers. A benchmark task lists one or more scorers under definition > scorers; each scorer looks at a sample (and, if the task has a solver, the solver’s output) and emits a small dictionary of score values for that sample. Metrics then aggregate those per-sample values across the dataset.

The Scoring Pipeline

sample ─▶ solver ─▶ solver_output ─┬─▶ scorer A ─▶ scores ─▶ metrics ─▶ evaluation run results
                                   └─▶ scorer B ─▶ scores ─▶ metrics ─┘
                                                      │
                                            actions ──┘  (drop rows from the aggregation)

Three things follow from this shape, and they explain most scoring mistakes:

Scorers are independent. Each one runs on every sample and produces its own scores. They cannot read each other’s output.
Metrics belong to a scorer, not to the task. A metric aggregates the score values of the scorer it is nested under, so field must name a key that that scorer emits.
Actions run last. An action rule can exclude a row from aggregation based on its scores, which is how quality filtering works.

Choosing a Scorer

Pick by how the correct answer is defined, not by what the task is called. Each scorer emits named score values, and a metric’s field selects one of them by name - so the last column is what you write metrics against.

Scorer	The correct answer is…	Emits → default metrics
String Equality `string_equals`	A known string that must match exactly.	`is_correct` → mean, named `string_equality_mean`
Multiple Choice `string_equals_mcqa`	One of a fixed set of choices.	`is_correct` → mean, named `accuracy`
Text Similarity `bleu`	A reference text the output should resemble.	`bleu_score` → mean, named `bleu_score`
Function Call Coverage `function_call_coverage`	A set of tool calls the agent had to make.	`all_required_calls_made`, `required_calls_coverage` and four call counts → mean of the first two
RAG Checker `rag_checker`	Faithfulness to retrieved context.	The scores named in `scores_to_compute`, by default all eleven (`recall`, `precision`, `faithfulness`, …) → mean of each
Model as a Judge `model_as_a_judge_classifier`	A verdict only a human - or an LLM - can reach.	`is_correct` → mean, named `accuracy`
Model as a Judge `model_as_a_judge_scorer`	A graded judgement on a scale.	`score` → mean, named `score_mean`
Labeler `labeler_via_model`	A label that filters rows rather than scores them.	`label` → none, the label is recorded but not aggregated
Python Scorers `python`, `python_all_samples`	Whatever your code says it is.	Whatever the snippet returns → none, `metrics` is mandatory

Important

Omitting metrics on a python or python_all_samples scorer fails the task with No metrics specified and there are no default metrics for this scorer. Every other scorer supplies a default. A field that names a value the scorer does not emit is not caught at authoring time - it produces a metric over a field that does not exist.

Three rules of thumb:

Prefer deterministic scorers. They are free, instant and perfectly repeatable. A judge model costs money on every sample and introduces variance you then have to measure with trials.
Prefer a built-in over Python. Built-ins are validated when the task is created; a Python snippet fails at run time.
Combine them. Several scorers on one task is normal and cheap: a deterministic scorer for correctness, a judge for tone, a qa labeler to flag rows you do not trust.

Every field of every scorer is documented in the Scorers reference.

Keys Must Be Unique

Both scorers and metrics get an implicit key: if you do not set key, it defaults to the type. That makes duplicates easy to hit and the failure is a hard error, not a warning.

# Fails: two scorers, both keyed `string_equals`.
definition:
  scorers:
    - type: string_equals
      ground_truth: "{{ sample.expected_short }}"
    - type: string_equals
      ground_truth: "{{ sample.expected_long }}"

# Fails: two metrics on one scorer, both keyed `mean`.
definition:
  scorers:
    - type: python
      compute_scores_snippet: !include score.py
      metrics:
        - type: mean
          field: precision
        - type: mean
          field: recall

Give each one an explicit key:

definition:
  scorers:
    - key: short_answer
      type: string_equals
      ground_truth: "{{ sample.expected_short }}"
    - key: long_answer
      type: string_equals
      ground_truth: "{{ sample.expected_long }}"
    - key: classification
      type: python
      compute_scores_snippet: !include score.py
      metrics:
        - key: precision
          type: mean
          field: precision
        - key: recall
          type: mean
          field: recall

Keys must match ^[a-zA-Z0-9_\-]+$. They are what you see in the evaluation run results and what action rules reference, so choose names that will still make sense in a report.

What Templates Can See

Most scorer fields are Jinja templates. What is in scope depends on the solver, and getting this wrong renders an empty string rather than raising an error.

Variable	Available	Holds
`sample`	Always	The dataset row, e.g. `{{ sample.question }}`.
`trace`	Tasks with a single-trace solver	The conversation, e.g. `{{ trace.get_last_assistant_text() }}`.
`model_outputs`	Tasks with a single-trace solver	Raw model outputs, one per call, e.g. `{{ model_outputs[-1] }}`.
`solver_output`	Tasks with a solver	The full solver output object.
`scores`	Action rule filters only	The scores produced for the row.

Warning

trace and model_outputs are not available when the solver is a grouped single turn solver, because there is no single trace. Use {{ solver_output.solver_outputs["key"].trace… }} instead.

For dataset-only tasks (evaluated_entity_type: dataset) there is no solver at all, so only sample is in scope. A scorer that asks for solver output in such a task fails with an explicit error.

Configuration parameters are substituted before Jinja runs, using the distinct << config.key >> syntax:

config_spec:
  - type: dataset_column
    key: answer_column
    display_name: "Column holding the expected answer"
definition:
  scorers:
    - type: string_equals
      ground_truth: "{{ sample.<< config.answer_column >> }}"

Scoring vs. Quality Assurance

A scorer can declare purpose: qa to say “this measures whether the row is trustworthy”, as opposed to purpose: score (the default) which measures the model. QA scorers are the input to action rules:

definition:
  scorers:
    - key: language_labeler
      purpose: qa
      type: labeler_via_model
      model_key: "<< config.labeler_model >>"
      valid_labels: ["english", "non-english"]
      user_prompt: |
        Is the following text English? Answer `english` or `non-english` and nothing else.

        <text>{{ trace.get_last_assistant_text() }}</text>
  actions:
    - key: exclude_non_english
      action: exclude_from_metrics
      filter:
        op: equals
        expression: "{{ scores.language_labeler.label }}"
        value: "non-english"

See Action Rules for the filter syntax and the exclusion semantics.

Next Steps

Metrics

Choose an aggregation for the values a scorer emits.

Action Rules

Exclude rows from the metric aggregation.

Testing Tasks

Inspect scores and metrics before running an evaluation.