Overview
An evaluation runs a set of configured tasks against a model or a dataset and produces technical evidence with full provenance - every score traceable back to the sample, the prompt and the response that produced it.
Every evaluation is built from tasks. A task is either a benchmark task, which runs samples through a model and scores the responses, or a system task, which probes a system-level property with a single Python snippet. Most work is benchmark tasks; system tasks cover the checks that have nothing to sample and nothing to score.
You do not have to write tasks to get started - AI Atlas ships packaged evaluations that only need a model and a few config values.
Inside a Task
A benchmark task is a pipeline, and each stage is a decision you make in the task YAML:
dataset ─▶ solver ─▶ scorers ─▶ metrics ─▶ actions ─▶ resultsThe Benchmark Tasks guide builds a task in that order, and every stage has its own page: datasets, solvers, scorers, metrics and action rules. Then verify with lf test task before running anything at scale: templates, snippets and metric fields are not checked until they execute.
A system task replaces this entire pipeline with one snippet that returns metrics directly, so none of the stage pages apply to it.
Running at Scale
Once a task is correct, the run config decides how much of the dataset it sees, how often it repeats and what gets reused:
- Caching - reuse earlier task results instead of paying for the same model calls twice.
- Repeatability - run a task specification several times to quantify how stable the whole evaluation is.
- Per-sample Trials - repeat individual samples to smooth out a noisy scorer or measure the model’s own consistency.
- Download and Upload Evidence - move evidence in and out of the LF AI Platform.
The full run config format - task_specifications, num_samples, subsampling, cache_policy and the per-specification overrides - is documented in the Evaluations reference.