Core Concepts

Note

Read this page once and the rest of the guides will make sense: it explains the entities you work with, how they fit together, and the workflow that produces an evaluation result.

LF AI Platform is declarative. You describe what to evaluate - models, datasets, tasks, evaluations - in YAML specifications, and the lf CLI creates the corresponding entities on the server. The server executes the evaluation and stores the results as technical evidence with full provenance.

Two consequences shape everything else:

The Entity Model

An evaluation is assembled from a handful of entities. Each one is defined declaratively and referenced by key:

flowchart LR
    M["Model<br/>what you evaluate"] --> T["Task<br/>solver → scorers → metrics"]
    D["Dataset<br/>what you evaluate it on"] --> T
    T --> E["Evaluation<br/>tasks × models"]
    E --> R["Results<br/>metrics, evidence, traces"]
    R --> P["Policies<br/>pass/fail gates"]

    classDef definition fill:#e8f0fe,stroke:#1a73e8,stroke-width:2px,color:#174ea6
    classDef outcome fill:#f1f3f4,stroke:#9aa0a6,stroke-width:2px,color:#3c4043

    class M,D,T,E definition
    class R,P outcome

Everything in this diagram lives inside an AI app, the workspace for one AI use case.

Entity What it is Reference
AI App The workspace that groups every entity for one AI use case. AI Apps
Model An inference endpoint - a provider API, your own HTTP service, or a Python snippet. Models
Model Adapter Translates between your endpoint’s request/response format and the platform’s schema. Model Adapters
Dataset The samples an evaluation runs on: local files, a URL, HuggingFace, or imported traces. Datasets
Dataset Generator Produces a dataset synthetically, from a source dataset or from scratch. Dataset Generators
Task The unit of evaluation logic: a solver interacts with the model, scorers score each sample, metrics aggregate the scores. Tasks
Evaluation A set of configured tasks run against models and datasets. Evaluations
Run config The run.yaml entry point that declares an evaluation together with every entity it depends on. Run
Task Result Log The full output of one task execution: metrics, per-sample evidence, traces, and errors. Task Result Logs
Trace The recorded conversation for one sample: messages, function calls, and their outputs. Trace
Policies Quality gates that check whether evaluation metrics meet the thresholds you defined. Policies
Risk Policies Continuous monitoring that converts metrics into a risk level for the AI app. Risk Policies

Inside a task, the three stages are worth naming explicitly, because most authoring work happens there:

  • Solver - how a sample is turned into model interaction: a single prompt, a multi-turn conversation, a Python function, or no interaction at all when you only score existing data. See Solvers.
  • Scorers - how each sample’s outcome is judged: string comparison, a model-as-a-judge, or custom Python. See Scorers.
  • Metrics - how per-sample scores are aggregated into the numbers you report. See Metrics.

AI Apps Are Workspaces

An AI app behaves much like a Git branch. It is identified by a key, it must exist before you can add anything to it, and you switch into it before doing work:

lf switch my-app

All entities and evaluations live inside the active AI app and are isolated from other AI apps. One AI app should hold everything required to evaluate one AI system, so it can be exported, reviewed, and reproduced as a whole.

How a Project Flows

Whether you drive the CLI yourself or hand the work to a coding agent, a project moves through the same six stages. Each one has a cheap feedback loop, so mistakes surface early instead of during a long run.

  1. Observe - establish where you are before changing anything.

    lf status   # active AI app, backend URL, where the config came from
lf list     # entities that already exist

  2. Research - decide what to evaluate. AI Atlas publishes packaged evaluations covering safety, robustness, and performance; if one matches your goal, start from it instead of writing specs from scratch.

    lf init --atlas <template_key>

  3. Define - write the specs and register them on the server. lf add creates or updates definitions without running anything, so you can iterate freely. Validate first:

    lf run -f run.yaml -v   # validate only, creates nothing
lf add -f run.yaml      # create or update every entity in the run config

  4. Test - verify each component against the server before spending a full run on it.

    lf test model <key>                          # is the endpoint reachable and well-formed?
lf test dataset -f dataset.yaml              # preview generated samples
lf test task -f run.yaml --spec-key <key>    # end-to-end on one sample

  5. Run - start small, then scale. Set num_samples in the evaluation spec to a low number, confirm the results look sane, then remove it and run on the full dataset.

    lf run -f run.yaml

  6. Inspect - review the evidence in the UI, or pull it locally.

    lf overview eval --id <id>          # metrics and status in the terminal
lf export eval --id <id> -o ./out   # metrics, per-sample evidence, and traces

The Quickstart walks this loop end to end for the first time.

Specs, Configuration, and Templates

A project typically keeps one spec per entity and ties them together in a single run.yaml, so that lf run -f run.yaml reproduces the whole evaluation. To keep specs short and free of duplicated or hard-coded values, the CLI extends YAML with a few mechanics you will meet in almost every guide:

YAML
Inline other files and expand environment variables.
Models
Add a provider model without a full model spec.
Templating
Fill prompts at runtime and configure specs before the run.
Secrets
Keep credentials out of specs and reference them instead.

Next Steps

Install the CLI
Install lf and point it at your deployment.
Quickstart
Run your first evaluation from a packaged AI Atlas template.
AI Apps
Bring your own models and datasets into a workspace.