Run

A run config is the single entry point for executing an evaluation run. It declares the evaluation to run together with everything it depends on - models, datasets, tasks and their supporting entities - so that lf run creates or updates all of them and then starts the evaluation run in one step.

# run.yaml
datasets:
  - $ref: "./datasets/hp_trivia.yaml"
models:
  - $provider: "openai/gpt-4.1-nano"
tasks:
  - $ref: "./tasks/hp_trivia.yaml"
evaluation:
  key: "hp-trivia-evaluation"
  display_name: "Harry Potter Trivia Evaluation"
  config_spec:
    - type: "model"
      key: "model_key"
      display_name: "Model"
      description: "The model to use for the evaluation."
  task_specifications:
    - key: "hp-trivia-gpt-4-1-nano"
      task_key: "hp-trivia"
      model_key: "<< config.model_key >>"
config:
  model_key: "openai$gpt-4-1-nano"

Declaring dependencies

Each top-level section holds the entities of one kind. They are applied in dependency order - model_adapters, models, then dataset_generators and datasets sorted by how they reference each other, then tasks and finally evaluation - so a generated dataset and the generator that produces it can be declared in the same file in any order. Entities can be written inline or, as above, pulled in from their own files with $ref: "./path.yaml", which keeps each definition reusable across run configs. For a model served by an integrated provider, $provider: "openai/gpt-4.1-nano" replaces a full model definition. The secrets section declares the values that entities reference as << secrets.NAME >> (see the secrets guide), and policies applies quality gates to the app.

Running

lf run applies the config to the app you are currently switched to. --validate checks the config without creating anything and --wait blocks until the evaluation run finishes. Ready-made run configs for published evaluations can be fetched with lf init --atlas; see Run Packaged Evaluations from Atlas. An existing evaluation run can be re-run by providing its ID as --id to lf rerun, either with the original entities (default) or with the current ones when --use-latest-entities is given.

Configuration

Properties

---

config object

Configuration for the evaluation’s config_spec parameters.

Default: {}

---

run_config EvaluationConfig

The run configuration for the evaluation run.

---

secrets object

Secrets which can be used to reference secret values in designated places.

Default: None

---

model_adapters array[SDKModelAdapter]

List of all model adapters to be created/updated before running the evaluation run.

Default: []

---

models array[SDKModel]

List of all models to be created/updated before running the evaluation run. Can be either a full model definition or a string representing a third-party provider and model key in the format $provider: <provider>/<model_key>.

Default: []

---

dataset_generators array[SDKDatasetGenerator]

List of all dataset generators to be created/updated before running the evaluation run.

Default: []

---

datasets array[SDKDataset]

List of all datasets to be created/updated before running the evaluation run.

Default: []

---

tasks array[SDKTask]

List of all tasks to be created/updated before running the evaluation run.

Default: []

---

evaluation SDKEvaluation

The evaluation to be created/updated before running the evaluation run.

Default: None

---

policies array[Policy]

List of all policies to apply to the AI app.

Default: []

Example with Provider Model

datasets:
  - $ref: "./datasets/hp_trivia.yaml"

models:
  - $provider: "openai/gpt-4.1-nano"

tasks:
  - $ref: "./tasks/hp_trivia.yaml"

evaluation:
  key: "hp-trivia"
  display_name: "Harry Potter Trivia"
  description: >-
    This evaluation checks that the model can answer trivia questions about
    the Harry Potter universe.
  config_spec:
    - key: "model_key"
      type: "model"
      display_name: "Model"
      description: "Model to be used for the task."
    - key: "judge_model_key"
      type: "model"
      display_name: "Judge Model"
      description: "Judge model to be used for the task."
  task_specifications:
    - key: "hp-trivia"
      task_key: "hp-trivia"
      display_name: "Harry Potter Trivia"
      task_config:
        judge_model_key: "<< config.judge_model_key >>"
      model_key: "<< config.model_key >>"

config:
  model_key: "openai$gpt-4-1-nano"
  judge_model_key: "openai$gpt-4-1-nano"

Example with Custom Model

models:
  - $ref: "./models/model.yaml"

datasets:
  - $ref: "./datasets/conversation_memory_questions.yaml"

tasks:
  - $ref: "./tasks/conversation_memory.yaml"

evaluation:
  key: "conversation_memory"
  display_name: "Conversation Memory"
  tags: ["General Purpose AI", "Chat Completion"]
  description: >-
    This evaluation checks that the model can remember information from
    previous interactions in a conversation.
  config_spec:
    - key: "model_key"
      type: "model"
      display_name: "Model"
      description: "Model to be used for the task."
  task_specifications:
    - key: "conversation_memory"
      display_name: "Conversation Memory"
      task_key: "conversation_memory"
      model_key: "<< config.model_key >>"

config:
  model_key: "openai-gpt-4-1-nano"