Assessment Plans

An assessment plan groups several evaluations under one name so they can be run together, producing an assessment. Each entry in evaluation_specifications references an evaluation by its key and provides the evaluation_config to fill its config_spec parameters. Assessment plans, similarly to evaluation runs or assessments, belong to an AI app. Use an assessment plan to run a whole suite in one step.

# assessment-plan.yaml
key: "hp-trivia-model-comparison-plan"
display_name: "Harry Potter Trivia Model Comparison Plan"
evaluation_specifications:
  - key: "hp-trivia-with-gpt-nano"
    evaluation_key: "hp-trivia"
    evaluation_config:
      model_key: "openai$gpt-4-1-nano"
      judge_model_key: "openai$gpt-4-1-nano"
  - key: "hp-trivia-with-gpt-mini"
    evaluation_key: "hp-trivia"
    evaluation_config:
      model_key: "openai$gpt-4-1-mini"
      judge_model_key: "openai$gpt-4-1-nano"

Managing plans

Create or update an assessment plan with lf add assessment-plan:

lf add assessment-plan -f assessment-plan.yaml

Running a plan

Start all evaluation runs in the plan with lf run assessment-plan. You can pass the plan file directly (which creates/updates the plan entity and then runs it) or reference an already-created plan by key:

# From a file (creates/updates the plan, then runs it)
lf run assessment-plan -f assessment-plan.yaml

# By key (plan must already exist)
lf run assessment-plan -k hp-trivia-model-comparison-plan

# Run only specific evaluation specifications from the plan
lf run assessment-plan -k hp-trivia-model-comparison-plan -sk hp-trivia-with-gpt-nano

Tracking a plan

Running an Assessment Plan produces an Assessment that groups the resulting evaluation runs. lf list assessment shows the assessments together with their progress, and lf overview assessment --id <id> reports the computed metrics of every evaluation run in one Assessment.

Rerunning a plan

Rerun a previous Assessment with lf rerun assessment --id <id>. Pass --use-latest-entities to use the latest versions of all entities instead of the versions from the original run.

Configuration

Properties

---

key string required

Unique identifier assigned to the entity in AI Platform.

Pattern: ^[a-zA-Z0-9_\-]+$
Max Length: 250

---

display_name string required

The assessment plan name displayed to the user.

---

evaluation_specifications array[SDKEvaluationSpecification] required

List of evaluation specifications, each referring to an existing evaluation by key and configuring it through its evaluation_config and evaluation_run_config.

Assessment Plan

key: "hp-trivia-model-comparison-plan"
display_name: "Harry Potter Trivia Model Comparison Plan"
evaluation_specifications:
  - key: "hp-trivia-with-gpt-nano"
    evaluation_key: "hp-trivia"
    evaluation_config:
      model_key: "openai$gpt-4-1-nano"
      judge_model_key: "openai$gpt-4-1-nano"
  - key: "hp-trivia-with-gpt-mini"
    evaluation_key: "hp-trivia"
    evaluation_config:
      model_key: "openai$gpt-4-1-mini"
      judge_model_key: "openai$gpt-4-1-nano"

Run Config (creates evaluation and dependencies)

key: "hp-trivia-model-comparison-plan"
display_name: "Harry Potter Trivia Model Comparison Plan"
evaluation_specifications:
  - key: "hp-trivia-with-gpt-nano"
    evaluation_key: "hp-trivia"
    evaluation_config:
      model_key: "openai$gpt-4-1-nano"
      judge_model_key: "openai$gpt-4-1-nano"
  - key: "hp-trivia-with-gpt-mini"
    evaluation_key: "hp-trivia"
    evaluation_config:
      model_key: "openai$gpt-4-1-mini"
      judge_model_key: "openai$gpt-4-1-nano"

Evaluation (evaluations/hp_trivia.yaml)

key: "hp-trivia-model-comparison-plan"
display_name: "Harry Potter Trivia Model Comparison Plan"
evaluation_specifications:
  - key: "hp-trivia-with-gpt-nano"
    evaluation_key: "hp-trivia"
    evaluation_config:
      model_key: "openai$gpt-4-1-nano"
      judge_model_key: "openai$gpt-4-1-nano"
  - key: "hp-trivia-with-gpt-mini"
    evaluation_key: "hp-trivia"
    evaluation_config:
      model_key: "openai$gpt-4-1-mini"
      judge_model_key: "openai$gpt-4-1-nano"

Note

The evaluations referenced by evaluation_key must exist before the plan is run. You can use a run config with lf add to create them along with their dependencies (models, datasets, tasks).