Benchmark Tasks

A benchmark task runs a model or a dataset through a repeatable pipeline: take a sample, interact with the model, score the result, aggregate the scores. Tasks are defined declaratively in YAML so that an evaluation run is reproducible and auditable.

This page walks through building one end to end, then explains the anatomy of the definition. The individual stages have their own pages:

Stage	Decides	Page
Dataset	Which samples the task runs on	Below
Solver	How each sample reaches the model	Solver Types
Scorers	How the response is judged	Scoring
Metrics	How scores are aggregated	Metrics
Actions	Which rows are kept out of the aggregation	Action Rules

Define a Task

We will build a task that measures whether a model follows a simple instruction.

First, create a model to evaluate:

lf add model -p 'openai/gpt-5-nano'

Then download and create a dataset. Each sample looks like {"options": "[a, b, c]", "first": "a", "last": "c"}:

curl 'https://cdn.latticeflow.cloud/_aigo/demos/docs/1.1.0/jsonl_instructions_categorical_options.yaml' -o 'jsonl_instructions_categorical_options.yaml'
curl 'https://cdn.latticeflow.cloud/_aigo/demos/docs/1.1.0/simple_instructions_options.jsonl' -o 'simple_instructions_options.jsonl'
lf add dataset -f 'jsonl_instructions_categorical_options.yaml'

Now the task. For each sample the model is asked to return the first option from the list in the options column, and the response is compared against the first column:

task.yaml

key: "simple-instructions-following"
display_name: "Simple Instructions Following"
description: "Evaluates how good the model is at following instructions."
config_spec:
  - type: "dataset"
    key: "dataset_key"
    display_name: "Dataset"
    default_value: "instructions-categorical-options"
definition:
  dataset:
    key: "<< config.dataset_key >>"
  solver:
    type: "single_turn_solver"
    input_builder:
      type: "chat_completion"
      input_messages:
        - role: "user"
          content: >
            Example: If asked for first option in [A, B, C], correct response is: A.

            Instructions: The list of allowed responses is {{ sample.options }}.
            Pick the first option. No punctuation, quotes or explanations.
  scorers:
    - type: "string_equals"
      ground_truth: "{{ sample.first }}"

The scorer needs no metrics block: string_equals defaults to the mean of its is_correct score.

Create the task, then describe how to run it. A run config pairs the task with the model and dataset it should execute against:

run.yaml

evaluation:
  key: "instructions-eval"
  display_name: "Instruction Following"
  config_spec:
    - key: "model_key"
      type: "model"
      display_name: "Model"
      description: "Model to be used for the task."
  task_specifications:
    - task_key: "simple-instructions-following"
      model_key: "<< config.model_key >>"

config:
  model_key: "gpt-5-nano"

lf add task -f 'task.yaml'
lf test task -f 'run.yaml' --spec-key 'simple-instructions-following' -n 1

That is a working task 🚀

Note

--spec-key identifies a task specification inside the run config, not the task itself. A specification with no explicit key gets one derived from its task_key - the task key alone if it appears once, otherwise <task_key>_1, <task_key>_2, and so on, with a warning telling you which key was generated. Set key explicitly on each specification when a run config uses the same task more than once.

Make It Configurable

Hard-coding “the first option” makes the task rigid. Exposing a parameter in config_spec lets each run choose whether the model should return the first or the last option, and both the solver and the scorer can read that parameter with << config.key >>:

task.yaml

key: "simple-instructions-following"
display_name: "Simple Instructions Following"
description: "Evaluates how good the model is at following instructions."
config_spec:
  - type: "categorical"
    key: "ground_truth"
    display_name: "Ground Truth Option"
    description: >
      Controls whether the model is expected to return the first or the last option.
    allowed_values: ["first", "last"]
  - type: "dataset"
    key: "dataset_key"
    display_name: "Dataset"
    default_value: "instructions-categorical-options"
definition:
  dataset:
    key: "<< config.dataset_key >>"
  solver:
    type: "single_turn_solver"
    input_builder:
      type: "chat_completion"
      input_messages:
        - role: "user"
          content: >
            Example: If asked for first option in [A, B, C], correct response is: A.
            Example: If asked for last option in [A, B, C], correct response is: C.

            Instructions: The list of allowed responses is {{ sample.options }}.
            Pick the << config.ground_truth >> option. No punctuation, quotes or
            explanations.
  scorers:
    - type: "string_equals"
      ground_truth: >
        {{ sample.first if "<< config.ground_truth >>" == "first" else sample.last }}

Because the task now takes a parameter, the run config must supply a value for it under task_config. It has to cover every parameter in the task’s config_spec:

run.yaml

evaluation:
  key: "instructions-eval"
  display_name: "Instruction Following"
  config_spec:
    - key: "model_key"
      type: "model"
      display_name: "Model"
      description: "Model to be used for the task."
  task_specifications:
    - task_key: "simple-instructions-following"
      model_key: "<< config.model_key >>"
      task_config:
        ground_truth: "last"

config:
  model_key: "gpt-5-nano"

Important

model_key is the model’s key you set when creating the model, not the provider-qualified string you passed to lf add model. lf add model -p 'openai/gpt-5-nano' creates a model keyed gpt-5-nano; writing openai/gpt-5-nano here fails validation, because keys may only contain letters, digits, _ and -. Run lf list model to see the actual keys.

Update the task and run it against that config:

lf add task -f 'task.yaml'
lf test task -f 'run.yaml' --spec-key 'simple-instructions-following'

Note

<< config.key >> substitution happens before Jinja rendering, and it is textual. That is why the scorer above wraps the placeholder in quotes - "<< config.ground_truth >>" == "first" - so that the result is a Jinja string comparison rather than an undefined variable.

For the full run config format, see the Evaluations reference.

Anatomy of a Definition

Note

This page covers the fields you reach for most often. The complete specification is in the Tasks reference.

key: "<key>"
display_name: "<display_name>"
description: "<description>"
config_spec: <config_spec>
definition:
  evaluated_entity_type: "model"   # or "dataset"
  dataset: <dataset>
  solver: <solver>
  scorers: <scorers>
  actions: <actions>

• Metadata - key, display_name and description are required. key must match ^[a-zA-Z0-9_\-]+$.
• Configuration specification - parameters the task exposes, filled in per run.
• Evaluated entity type - model (the default) evaluates a model’s behaviour; dataset evaluates the quality of data.
• Dataset - the collection of test scenarios.
• Solver - how each sample is turned into an interaction. See Solver Types.
• Scorers - how the result is judged, and the metrics aggregating those judgements. See Scoring.
• Actions - post-scoring rules, such as excluding untrustworthy rows. See Action Rules.

Warning

evaluated_entity_type goes inside the definition block. Placed at the top level it is silently ignored, since unknown top-level fields are dropped rather than rejected.

The two entity types have mirrored requirements:

evaluated_entity_type	dataset	solver
model (default)	Required	Required
dataset	Must be omitted - supplied by the evaluation	Must be omitted

Getting this wrong is caught at creation time, with an explicit error.

Configuration Specification

config_spec lists the parameters a task exposes. Each parameter has a type, a key and a display_name, and its value is read in the definition as << config.<key> >>.

config_spec:
  - type: "model"
    key: "judge_model"
    display_name: "Judge Model"
  - type: "dataset_column"
    key: "question_column"
    display_name: "Column holding the question"

A task that exposes nothing sets config_spec: []. Parameter types and their options are in the Config Specification reference.

Configuration is what makes a task reusable across datasets and models: rather than naming a column, name a dataset_column parameter and let each run point it at the right column.

Dataset

Reference a dataset by key:

config_spec:
  - type: "dataset"
    key: "dataset_key"
    display_name: "Dataset"
    default_value: "my-dataset"
definition:
  dataset:
    key: "<< config.dataset_key >>"

Or expose it as a parameter so the task is not tied to one dataset:

config_spec:
  - type: "dataset"
    key: "my_dataset"
    display_name: "My Dataset"
definition:
  dataset:
    key: "<< config.my_dataset >>"

See Integrate a Dataset for creating datasets in the first place.

Solver

The solver defines how each sample reaches the model - one request, a conversation, a group of parallel requests, or replayed traces.

definition:
  solver:
    type: "single_turn_solver"
    input_builder:
      type: "chat_completion"
      input_messages:
        - role: "user"
          content: "{{ sample.question }}"

All five solver types, the two input builders and their trade-offs are covered in Solver Types, with dedicated pages for multi-turn conversations, the Python solver and the post-processor.

Scorers and Metrics

Each scorer emits named score values for a sample; its metrics aggregate those values across the dataset.

definition:
  scorers:
    - key: "answer_correctness"
      type: "string_equals"
      ground_truth: "{{ sample.expected_answer }}"
      metrics:
        - type: "mean"
          field: "is_correct"
          name: "Accuracy"

Important

field must name a value the scorer actually emits - string_equals emits is_correct, bleu emits bleu_score, and a Python scorer emits whatever your snippet returns. A field that matches nothing is not a validation error; it shows up as a missing metric after the evaluation has run.

If you omit metrics, most built-in scorers fall back to a sensible default, but python, python_all_samples and rag_checker do not and will fail. See Scoring for the chooser, the score field names and the key-uniqueness rules, and Metrics for the aggregations.

Actions

Action rules run after scoring and decide which rows are excluded from the metrics - the mechanism for keeping untrustworthy samples out of your numbers while leaving them visible in the evidence.

definition:
  actions:
    - key: "exclude_non_english_answers"
      action: "exclude_from_metrics"
      filter:
        op: "equals"
        expression: "{{ scores.language_labeler.label }}"
        value: "non-english"

See Action Rules.

Trials

A benchmark task can run each sample several times and aggregate across the repeats - useful when a judge scorer is noisy, or when you want to measure the model’s own consistency. Trials are specific to benchmark tasks:

definition:
  trials:
    num_trials: 3

See Per-sample Trials. For repeating a whole task run instead, see Repeatability.

Managing Tasks

Creating a task from YAML either inserts it or updates the existing task with that key:

lf add task -f 'task.yaml'     # one task
lf add task -f 'tasks/*.yaml'  # every task in a directory

lf list task                            # list all tasks
lf export task 'my-task'                # export to stdout
lf export task 'my-task' -o 'task.yaml' # export to a file
lf delete task 'my-task'                # delete by key

Examples

Single-turn Task

A task with a single-turn model interaction and a generic input builder.

Multi-turn Task

A task driving a multi-turn conversation.

Data Quality Task

A dataset task scoring the data itself with Python scorers.