Config Specification
A config specification is the parameter list an entity accepts. A task, dataset generator, or evaluation declares one in its config_spec field, refers to the values with << config.key >>, and is then instantiated with different values instead of being copied and edited.
# excerpt of tasks/yes_no_qa.yaml
config_spec:
- type: "int"
key: "max_words"
display_name: "Max Words"
description: "Maximum number of words the model should use in its answer."
default_value: 1
min: 1
max: 5
definition:
solver:
type: "single_turn_solver"
input_builder:
type: "chat_completion"
input_messages:
- role: "system"
content: "Answer in at most << config.max_words >> word(s)."Declaring and supplying values
Every parameter has a type, a key - the name used in << config.key >> - and a display_name. A parameter with a default_value may be omitted by the caller; one marked nullable may be left empty. Note the two distinct substitutions: << config.x >> is replaced with a configuration value before the task runs, whereas { sample.x } is resolved per sample from the dataset.
Who supplies the values depends on the entity: an evaluation sets a task’s parameters in task_specifications[].task_config, a dataset sets a generator’s in generator_specification.dataset_generator_config, and a risk scorer sets a risk scoring methodology’s in config. As for the evaluation and its configuration specification, it is provided in a config field either inside a run config, or as a separate configuration file. Values are validated against the specification, so a missing or out-of-range parameter fails before anything runs.
Parameter types
type |
Value | Type-specific fields |
|---|---|---|
int, float |
A number | min, max |
boolean |
true or false |
|
string |
Text | string_kind (freeform, python or jinja), examples |
categorical |
One or, with multiple, several of allowed_values |
allowed_values, multiple, values_mapping |
list, dict |
A collection of string, integer, float or boolean values |
dtype / value_dtype |
model |
The key of a model | |
dataset |
The key of a dataset | |
dataset_column |
The name of a dataset column | |
task_result |
The key of a task result, e.g. of a sibling task in the same evaluation |
The model and dataset types are what make a task portable: a judge model or an auxiliary dataset is named by the evaluation that runs the task, not hardcoded in it.
Parameter Types
IntParameterSpec
Properties
type Literal “int” required
The type of the parameter.
key string required
The key of the parameter.
display_name string required
The display name of the parameter.
description string
The description of the parameter.
Default: None
min integer
The minimum value of the parameter.
Default: None
max integer
The maximum value of the parameter.
Default: None
default_value integer
The default value to use.
Default: None
nullable boolean
Whether this parameter is nullable.
Default: False
Int Parameter Definition
# ...
config_spec:
# ...
- type: "int"
key: "max_words"
display_name: "Max Words"
description: "Maximum number of words the model should use in its answer."
default_value: 1
min: 1
max: 5BooleanParameterSpec
Properties
type Literal “boolean” required
The type of the parameter.
key string required
The key of the parameter.
display_name string required
The display name of the parameter.
description string
The description of the parameter.
Default: None
default_value boolean
The default value to use.
Default: None
nullable boolean
Whether this parameter is nullable.
Default: False
Boolean Parameter Definition
# ...
config_spec:
# ...
- type: "boolean"
key: "include_context"
display_name: "Include Context"
description: "Whether to include supporting context in the user message."FloatParameterSpec
Properties
type Literal “float” required
The type of the parameter.
key string required
The key of the parameter.
display_name string required
The display name of the parameter.
description string
The description of the parameter.
Default: None
min number
The minimum value of the parameter.
Default: None
max number
The maximum value of the parameter.
Default: None
default_value number
The default value to use.
Default: None
nullable boolean
Whether this parameter is nullable.
Default: False
StringParameterSpec
Properties
type Literal “string” required
The type of the parameter.
key string required
The key of the parameter.
display_name string required
The display name of the parameter.
description string
The description of the parameter.
Default: None
default_value string
The default value of the parameter.
Default: None
default_file_content string
The contents of the default file for this parameter. Serves similar purpose as the default_value, but with the additional signal that the provided value is not a real sensible default value to be used, but rather a placeholder/set of instructions that can be prefilled in a file the user has to fill out.
Default: None
nullable boolean
Whether this parameter is nullable.
Default: False
string_kind enum StringKind
Default: freeform
Specifies the kind of string parameter.
Allowed Values:
freeformpythonjinja
examples null
Examples for the string parameter.
Default: None
String Parameter Definition
# ...
config_spec:
- type: "dataset"
key: "tabular_dataset"
display_name: "Dataset"
description: "Dataset to evaluate for completeness."
- type: "string"
key: "field"
display_name: "Field"
description: "Dataset field to check for completeness."ModelParameterSpec
Properties
type Literal “model” required
The type of the parameter.
key string required
The key of the parameter.
display_name string required
The display name of the parameter.
description string
The description of the parameter.
Default: None
default_value string
The default value to use.
Default: None
nullable boolean
Whether this parameter is nullable.
Default: False
usage enum SystemModelUsage, enum Usage
The intended usage of the model parameter. Helps with pre-filling from the models set on the platform and AI-app level.
Default: None
A supported platform-wide model usage.
Allowed Values:
productivity_assistantjudge_modelsynthesizer_model
The intended usage of the model parameter. Helps with pre-filling from the models set on the platform and AI-app level.
Allowed Values:
model_under_test
Model Parameter Definition
# ...
config_spec:
# ...
- type: "model"
key: "judge_model_key"
display_name: "Judge Model"
description: "Model used to respond to clarifying questions."DatasetParameterSpec
Properties
type Literal “dataset” required
The type of the parameter.
key string required
The key of the parameter.
display_name string required
The display name of the parameter.
description string
The description of the parameter.
Default: None
default_value string
The default value to use.
Default: None
nullable boolean
Whether this parameter is nullable.
Default: False
Dataset Parameter Definition
# ...
config_spec:
- type: "dataset"
key: "dataset_key"
display_name: "Dataset"
description: "The dataset to evaluate against."DatasetColumnParameterSpec
Properties
type Literal “dataset_column” required
The type of the parameter.
key string required
The key of the parameter.
display_name string required
The display name of the parameter.
description string
The description of the parameter.
Default: None
default_value string
The default value to use.
Default: None
nullable boolean
Whether this parameter is nullable.
Default: False
Dataset Column Parameter Definition
# ...
config_spec:
# ...
- type: "dataset_column"
key: "answer_column"
display_name: "Answer Column"
description: "Dataset column containing the correct single-character answer choice."TaskResultParameterSpec
Properties
type Literal “task_result” required
The type of the parameter.
key string required
The key of the parameter.
display_name string required
The display name of the parameter.
description string
The description of the parameter.
Default: None
default_value string
The default value to use (a task result key, resolved to an ID at evaluation run creation time).
Default: None
nullable boolean
Whether this parameter is nullable.
Default: False
Task Result Parameter Definition
# ...
config_spec:
- type: "task_result"
key: "task_result_key"
display_name: "Task Result Key"
description: "Key of the task result whose per-sample correctness scores are compared across languages."
default_value: "multilingual-trivia"ListParameterSpec
Properties
type Literal “list” required
The type of the parameter.
dtype enum ScalarDtype required
The data type of the elements in the list.
The scalar data type.
Allowed Values:
stringintegerfloatboolean
key string required
The key of the parameter.
display_name string required
The display name of the parameter.
description string
The description of the parameter.
Default: None
default_value array[Any]
The default value to use.
Default: None
nullable boolean
Whether this parameter is nullable.
Default: False
DictParameterSpec
Properties
type Literal “dict” required
The type of the parameter.
value_dtype enum ScalarDtype required
The data type of the values in the dict.
The scalar data type.
Allowed Values:
stringintegerfloatboolean
key string required
The key of the parameter.
display_name string required
The display name of the parameter.
description string
The description of the parameter.
Default: None
default_value object
The default value to use.
Default: None
nullable boolean
Whether this parameter is nullable.
Default: False
CategoricalParameterSpec
Properties
type Literal “categorical” required
The type of parameter.
key string required
The key of the parameter.
display_name string required
The display name of the parameter.
description string
The description of the parameter.
Default: None
allowed_values array[string] required
Allowed Values
multiple boolean
Whether the parameter can have multiple values.
Default: False
default_value string
The default value to use.
Default: None
nullable boolean
Whether this parameter is nullable.
Default: False
values_mapping object
A mapping over the categorical values.
Default: None
Categorical Parameter Definition
# ...
config_spec:
# ...
- type: "categorical"
key: "evaluation_dimension"
display_name: "Evaluation Dimension"
description: "The aspect of the model response to evaluate."
allowed_values: ["groundedness", "relevance"]