Evaluation Plans

An evaluation plan groups several run configs under one name so they can be created and executed together. Each entry in evaluations is a complete run config - it declares an evaluation together with every entity it depends on - and produces one evaluation when the plan is applied. Use a plan to run a whole suite in one step, for example every task in a benchmark, or the same evaluation across several models.

# eval-plan.yaml
key: "hp-trivia-model-comparison-plan"
display_name: "Harry Potter Trivia Model Comparison Plan"
evaluations:
  # An entry can be written inline...
  - datasets:
      - $ref: "./datasets/hp_trivia.yaml"
    models:
      - $provider: "openai/gpt-4.1-nano"
    tasks:
      - $ref: "./tasks/hp_trivia.yaml"
    evaluation:
      key: "hp-trivia-gpt-4-1-nano-eval"
      display_name: "Harry Potter Trivia - GPT-4.1 Nano"
      task_specifications:
        - key: "hp-trivia-gpt-4-1-nano"
          task_key: "hp-trivia"
          model_key: "openai$gpt-4-1-nano"
  # ...or pulled in from its own reusable run config file.
  - $ref: "./evaluations/hp_trivia.yaml"

How a plan is applied

A file is treated as an evaluation plan when it has a top-level evaluations list (a plain run config has a single evaluation instead). Passing such a file to lf add applies each run config in turn - creating or updating its entities and starting its evaluation - so the whole plan runs from one command. Individual run configs can be written inline or pulled in with $ref: "./path.yaml", which keeps each one reusable on its own.

Tracking a plan

Applying a plan produces an evaluation plan run that groups the resulting evaluations. lf list eval-plan shows the plan runs of the current app together with their progress, and lf overview eval-plan --id <id> reports the computed metrics of every evaluation in one plan run.

Configuration

Properties

---

key string required

Unique identifier assigned to the entity in AI Platform.

Pattern: ^[a-zA-Z0-9_\-]+$
Max Length: 250

---

display_name string required

The evaluation plan’s name displayed to the user.

---

evaluations array[SDKRunConfig] required

List of run configs, each producing one evaluation to be run as part of this plan.

Model Comparison Plan

key: "hp-trivia-model-comparison-plan"
display_name: "Harry Potter Trivia Model Comparison Plan"
evaluations:
  # Option 1: declare the run config inline.
  - datasets:
      - $ref: "./datasets/hp_trivia.yaml"
    models:
      - $provider: "openai/gpt-4.1-nano"
    tasks:
      - $ref: "./tasks/hp_trivia.yaml"
    evaluation:
      key: "hp-trivia-gpt-4-1-nano-eval"
      display_name: "Harry Potter Trivia - GPT-4.1 Nano"
      task_specifications:
        - key: "hp-trivia-gpt-4-1-nano"
          task_key: "hp-trivia"
          display_name: "Harry Potter Trivia GPT-4.1 Nano"
          task_config:
            judge_model_key: "openai$gpt-4-1-nano"
          model_key: "openai$gpt-4-1-nano"
  # Option 2: pull the run config in from its own file with `$ref`, so it can be
  # reused and run on its own with `lf run -f ./evaluations/hp_trivia.yaml`.
  - $ref: "./evaluations/hp_trivia.yaml"

Referenced Run Config (evaluations/hp_trivia.yaml)

datasets:
  - $ref: "../datasets/hp_trivia.yaml"
models:
  - $provider: "openai/gpt-4.1-mini"
  - $provider: "openai/gpt-4.1-nano"
tasks:
  - $ref: "../tasks/hp_trivia.yaml"
evaluation:
  key: "hp-trivia-gpt-4-1-mini-eval"
  display_name: "Harry Potter Trivia - GPT-4.1 Mini"
  task_specifications:
    - key: "hp-trivia-gpt-4-1-mini"
      task_key: "hp-trivia"
      display_name: "Harry Potter Trivia GPT-4.1 Mini"
      task_config:
        judge_model_key: "openai$gpt-4-1-nano"
      model_key: "openai$gpt-4-1-mini"

Note

Each entry in evaluations accepts the same fields as a standalone run config, so an existing run.yaml can be reused inside a plan unchanged via $ref.