Introduction of Evaluation Configuration

Breaking change in AI Platform 3.14.0

Starting with AI Platform version 3.14.0, evaluation definitions were extended such that they can explicitly expose their configuration parameters, instead of relying on environment variables or hardcoded values. More importantly, they must declare their dependencies on other entities explicitly via this configuration. Creating or updating a definition that hardcodes entity keys is now rejected with an informative error, and must be adjusted as described in this guide.

Evaluations refer to other entities in their task specifications. Each task specification contains the evaluated entity (model or dataset) and can refer to further models and datasets as part of the task configuration.

These dependencies must not be hardcoded. Every key that refers to another entity (a model or a dataset) must be a << config.NAME >> template string, and each such parameter must be declared in config_spec. The dependency is then resolved at run time, when you supply a value for each declared parameter.

What to Change

Wherever your definition holds a literal model or dataset key (or an environment vairable), replace that value with a << config.NAME >> template string and add a matching entry to config_spec.

# Before
task_specifications:
  - task_key: "prompt_injection"
    model_key: "openai$gpt-4-1-nano"  # <- not allowed anymore
    task_config:
      judge_model: "anthropic$claude-haiku-5"  # <- not allowed anymore

# After
config_spec:
  - type: "model"
    key: "model_under_test"
    display_name: "Model under test"
  - type: "model"
    key: "judge_model"
    display_name: "Judge model"
task_specifications:
  - task_key: "prompt_injection"
    model_key: "<< config.model_under_test >>"
    task_config:
      judge_model: "<< config.judge_model >>"

Config Spec Entry Types

Each entry in config_spec uses type to specify what kind of value it expects. Use "model" for model keys, "dataset" for dataset keys. Besides the required type, key, and display_name, an entry can set a default_value:

config_spec:
  - type: "model"
    key: "judge_model"
    display_name: "Judge Model"
    default_value: "openai$gpt-4-1-nano" # optional: used when no value is supplied

See the Config Specification reference for all available types and their options.

A default_value is optional but can be leveraged to define an evaluation that can be executed without the need to provide an explicit configuration. When a default_value is set, the dependency is still resolved at run time — the platform verifies that an entity with that key exists before the run starts, so replacing an entity with a new one under the same key keeps the definition runnable. Note that the entity is validated at create time as well.

Supplying Values at Runtime

When using a run file (e.g. lf run -f run.yaml), the values can be provided via a config section in the run file.

# run.yaml
tasks: ...
evaluation: ...
config:
  model_under_test: "openai$gpt-4-1-nano"
  judge_model: "anthropic$claude-haiku-5"

When running evaluations directly (e.g. lf run eval -k my-evaluation -c config.yaml), the values can be provided via a separate config file.

# config.yaml
config:
  model_under_test: "openai$gpt-4-1-nano"
  judge_model: "anthropic$claude-haiku-5"