Migrating to Explicit Dependencies

ImportantBreaking change in AI Platform 3.13.0

Starting with AI Platform version 3.13.0, task and dataset generator definitions must declare their dependencies on other entities explicitly. Creating or updating a definition that hardcodes entity keys is now rejected with an informative error, and must be adjusted as described in this guide.

Task and dataset generators refer to other entities in their definitions. For example, a dataset is used as an input to a benchmark task or a dataset generator, a model synthesizer is used to generate required content, or a model judge is used to score the output of the solver.

These dependencies must not be hardcoded. Every key that refers to another entity (a model or a dataset) must be a << config.NAME >> template string, and each such parameter must be declared in config_spec. The dependency is then resolved at run time, when you supply a value for each declared parameter.

What to Change

Wherever your definition holds a literal model or dataset key, replace that value with a << config.NAME >> template string and add a matching entry to config_spec. This applies to any field that names another entity, such as dataset_key on a data source or model_key on a synthesizer or scorer.

For a dataset key:

# Before
data_source:
  type: "dataset_samples"
  dataset_key: "my-dataset"

# After
config_spec:
  - type: "dataset"
    key: "dataset"
data_source:
  type: "dataset_samples"
  dataset_key: "<< config.dataset >>"

For a model key:

# Before
scorers:
  - type: "model_as_a_judge_classifier"
    model_key: "openai$gpt-4-1-nano"

# After
config_spec:
  - type: "model"
    key: "judge_model"
scorers:
  - type: "model_as_a_judge_classifier"
    model_key: "<< config.judge_model >>"

Each template string must reference a key that is declared in config_spec. For complete, up-to-date definition examples, see the Tasks and Dataset Generators reference pages.

Config Spec Entry Types

Each entry in config_spec uses type to specify what kind of value it expects. Use "model" for model keys, "dataset" for dataset keys. Besides the required type and key, an entry can set a display_name and a default_value:

config_spec:
  - type: "model"
    key: "judge_model"
    display_name: "Judge Model"          # optional: label shown in the UI
    default_value: "openai$gpt-4-1-nano" # optional: used when no value is supplied

See the Config Specification reference for all available types and their options.

A default_value is optional but can be leveraged to define a task or a dataset generator that can be executed without the need to provide an explicit configuration. When a default_value is set, the dependency is still resolved at run time — the platform verifies that an entity with that key exists before the run starts, so replacing an entity with a new one under the same key keeps the definition runnable.

Supplying Values at Runtime

When a dataset references a generator, its dataset_generator_config supplies a value for each key declared in the generator’s config_spec:

# datasets/my-dataset.yaml
generator_specification:
  dataset_generator_key: "question-generator"
  num_samples: 50
  dataset_generator_config:
    dataset: "my-seed-dataset"  # <-- After
    synthesizer_model: "openai$gpt-4-1-nano"  # <-- After

When a task specification references a task, its task_config supplies a value for each key declared in the task’s config_spec:

task_specifications:
  - task_key: "my-task"
    task_config:
      dataset: "my-dataset"  # <-- After
      judge_model: "openai$gpt-4-1-nano"  # <-- After

NoteExisting definitions

Definitions stored before AI Platform 3.13.0 that hardcode entity keys can still be used to run and export evaluations. However, their dependencies are not declared, so the platform cannot guarantee that all dependencies are met — in particular when running evaluations from the UI. Creating or updating such a definition is no longer allowed: the CLI raises an informative error and asks you to move the hardcoded keys into config_spec. Migrate them to declared dependencies as described above to get these guarantees.