Tasks

A task defines the execution flow of a single evaluation and its run: which dataset provides the samples, how the evaluated entity is exercised, how each output is scored, and how the scores are aggregated. A task is defined in its own YAML file and referenced by an evaluation.

# tasks/hp_trivia.yaml
key: "hp-trivia"
display_name: "Harry Potter Trivia Task"
config_spec:
  - type: "dataset"
    key: "dataset_key"
    display_name: "Dataset"
    default_value: "hp-trivia-dataset"
definition:
  evaluated_entity_type: "model"
  dataset:
    key: "<< config.dataset_key >>"
  solver:
    type: "single_turn_solver"
    input_builder:
      type: "chat_completion"
      input_messages:
        - role: "user"
          content: "{{ sample.question }}"
  scorers:
    - type: "string_equals"
      ground_truth: "{{ sample.gt_answer }}"

Execution flow

For every sample of the task’s dataset, LF AI Platform runs the four stages below. Each stage is configured inline in the task’s definition.

Stage Key What it does
Dataset dataset Names the dataset whose samples are evaluated.
Solver solver Produces the output under evaluation, e.g. by prompting the model once or driving a multi-turn conversation.
Scorers scorers Judge each sample’s output and emit named scores, which can be numerical, boolean and string scores.
Metrics scorers[].metrics Aggregate the per-sample scores into the reported result.

Task kinds

The definition.type field selects what the task evaluates:

  • Benchmark tasks (benchmark_task, the default) run the four stages above against a model or a dataset. Set evaluated_entity_type to model to evaluate an inference endpoint, or to dataset to score dataset samples directly (no solver runs). See the benchmark tasks guide.
  • System tasks (system_task) check the AI system’s configuration rather than its outputs, so they define no dataset, solver or scorers. See the system tasks guide.

Working with tasks

Add the task with lf add task, dry-run it on a few samples with lf test task, and list the tasks already registered with lf list task. Tasks are executed as part of an evaluation run, usually through lf run.

Two options apply across samples rather than to a single one: trials repeats every sample to measure output stability (see the per-sample trials guide), and actions matches samples with a filter expression to exclude them from the reported metrics.

Configuration

Properties

key string required

Unique identifier assigned to the entity in AI Platform.

Pattern: ^[a-zA-Z0-9_\-]+$
Max Length: 250

display_name string required

The task’s name displayed to the user.

description string required

Short description of the task.

long_description string

Long description of the task. Supports Markdown formatting.

Default: None

tasks array[enum MLTask]

ML tasks supported by the task.

Default: []


The type of machine learning task to be performed.

Allowed Values:

  • chat_completion
  • embeddings
  • custom

config_spec array[FloatParameterSpec, IntParameterSpec, BooleanParameterSpec, StringParameterSpec, ModelParameterSpec, DatasetParameterSpec, DatasetColumnParameterSpec, ListParameterSpec, DictParameterSpec, CategoricalParameterSpec]

Configuration specification of the task.

Default: []

definition SDKBenchmarkTaskDefinitionTemplate, SDKSystemTaskDefinitionTemplate required

Definition of the task.

tags array[string]

Tags associated with the task.

Default: []

Single-turn Generic Input Model Task
display_name: "Single-turn Solver Generic Input"
key: "singleturn-generic-input"
description: "Example task that uses a single-turn solver with generic input builder."
config_spec:
  - key: "dataset_key"
    type: "dataset"
    display_name: "Dataset"
    default_value: "qa-single-answers"
    description: "Dataset containing questions and expected single-word answers."
definition:
  dataset:
    key: "<< config.dataset_key >>"
  solver:
    type: "single_turn_solver"
    input_builder:
      type: generic
      template: >
        {
            "messages": [
            {
              "role": "system",
              "content": "You are a helpful assistant. Answer with no punctuation."
            },
            {
              "role": "user",
              "content": "{{ sample.question }}"
            }
          ]
        }
  scorers:
    - type: "string_equals"
      ground_truth: "{{ sample.target }}"
display_name: "Uniqueness"
key: "uniqueness"
description: >
  Evaluates the uniqueness rate of values in a field across samples in a dataset.
tags: ["Data Quality"]
config_spec:
  - type: "string"
    key: "field"
    display_name: "Field"
    description: "Dataset field to check for uniqueness."
definition:
  type: "benchmark_task"
  evaluated_entity_type: "dataset"
  scorers:
    - type: "python_all_samples"
      compute_scores_snippet: !include "uniqueness_scorer.py"
      metrics:
        - type: "mean"
          field: "is_unique"
          name: "Uniqueness Rate"
from __future__ import annotations

from collections import Counter
from typing import Any

from latticeflow.core.dtypes import RawSample


def compute_scores(samples: list[RawSample]) -> list[dict[str, Any]]:
    field_name = "<< config.field >>"
    values = [
        sample[field_name] if field_name in sample else None for sample in samples
    ]
    counter = Counter(values)

    return [
        {"is_unique": counter[value] == 1 if value is not None else True}
        for value in values
    ]