Assessment Plans

An assessment plan groups several evaluations under one name so they can be run together. Running a plan produces an assessment that tracks the progress and results of every evaluation run it started.

Prerequisites

An assessment plan references evaluations and their dependencies (models, datasets, and tasks) that must exist on the platform before it is run. Add them with lf add -f run.yaml.

Writing an Assessment Plan

An assessment plan YAML contains a key, display name, and evaluation_specifications:

assessment-plan.yaml

key: "hp-trivia-model-comparison-plan"
display_name: "Harry Potter Trivia Model Comparison Plan"
evaluation_specifications:
  - key: "hp-trivia-with-gpt-nano"
    evaluation_key: "hp-trivia"
    evaluation_config:
      model_key: "openai$gpt-4-1-nano"
      judge_model_key: "openai$gpt-4-1-nano"
  - key: "hp-trivia-with-gpt-mini"
    evaluation_key: "hp-trivia"
    evaluation_config:
      model_key: "openai$gpt-4-1-mini"
      judge_model_key: "openai$gpt-4-1-nano"

Creating an Assessment Plan

Create or update the plan with lf add assessment-plan:

lf add assessment-plan -f assessment-plan.yaml

Running an Assessment Plan

Run all evaluations with lf run assessment-plan:

lf run assessment-plan -f assessment-plan.yaml
lf run assessment-plan -k hp-trivia-model-comparison-plan
lf run assessment-plan -k hp-trivia-model-comparison-plan -sk hp-trivia-with-gpt-nano

Tracking an Assessment

List assessments for the current app:

lf list assessment
lf overview assessment --id <assessment-id>

Rerunning an Assessment

lf rerun assessment --id <assessment-id>
lf rerun assessment --id <assessment-id> --use-latest-entities

End-to-End Example

lf add app -f app.yaml
lf switch assessment-plan
lf integration add --provider openai --api-key $OPENAI_API_KEY
lf add -f run.yaml
lf run assessment-plan -f assessment-plan.yaml
lf overview assessment --id <assessment-id>