Config Specification
A config specification allows defining allowed configuration options for a given task or a dataset generator.
Definitions
IntParameterSpec
Properties
type Literal “int” required
The type of the parameter.
key string required
The key of the parameter.
display_name string required
The display name of the parameter.
description string
The description of the parameter.
Default: None
min integer
The minimum value of the parameter.
Default: None
max integer
The maximum value of the parameter.
Default: None
default_value integer
The default value to use.
Default: None
nullable boolean
Whether this parameter is nullable.
Default: False
Int Parameter Definition
# ...
config_spec:
- type: "int"
key: "max_words"
display_name: "Max Words"
description: "Maximum number of words the model should use in its answer."
default_value: 1
min: 1
max: 5
- type: "boolean"
key: "include_context"
display_name: "Include Context"
description: "Whether to include supporting context in the user message."BooleanParameterSpec
Properties
type Literal “boolean” required
The type of the parameter.
key string required
The key of the parameter.
display_name string required
The display name of the parameter.
description string
The description of the parameter.
Default: None
default_value boolean
The default value to use.
Default: None
nullable boolean
Whether this parameter is nullable.
Default: False
Boolean Parameter Definition
# ...
config_spec:
- type: "int"
key: "max_words"
display_name: "Max Words"
description: "Maximum number of words the model should use in its answer."
default_value: 1
min: 1
max: 5
- type: "boolean"
key: "include_context"
display_name: "Include Context"
description: "Whether to include supporting context in the user message."FloatParameterSpec
Properties
type Literal “float” required
The type of the parameter.
key string required
The key of the parameter.
display_name string required
The display name of the parameter.
description string
The description of the parameter.
Default: None
min number
The minimum value of the parameter.
Default: None
max number
The maximum value of the parameter.
Default: None
default_value number
The default value to use.
Default: None
nullable boolean
Whether this parameter is nullable.
Default: False
StringParameterSpec
Properties
type Literal “string” required
The type of the parameter.
key string required
The key of the parameter.
display_name string required
The display name of the parameter.
description string
The description of the parameter.
Default: None
default_value string
The default value of the parameter.
Default: None
nullable boolean
Whether this parameter is nullable.
Default: False
string_kind enum StringKind
Default: freeform
Specifies the kind of string parameter.
Allowed Values:
freeformpythonjinja
examples null
Examples for the string parameter.
Default: None
String Parameter Definition
# ...
config_spec:
- type: "string"
key: "field"
display_name: "Field"ModelParameterSpec
Properties
type Literal “model” required
The type of the parameter.
key string required
The key of the parameter.
display_name string required
The display name of the parameter.
description string
The description of the parameter.
Default: None
default_value string
The default value to use.
Default: None
nullable boolean
Whether this parameter is nullable.
Default: False
Model Parameter Definition
# ...
config_spec:
- type: "model"
key: "judge_model"
display_name: "Judge Model"DatasetParameterSpec
Properties
type Literal “dataset” required
The type of the parameter.
key string required
The key of the parameter.
display_name string required
The display name of the parameter.
description string
The description of the parameter.
Default: None
default_value string
The default value to use.
Default: None
nullable boolean
Whether this parameter is nullable.
Default: False
Dataset Parameter Definition
# ...
config_spec:
- type: "dataset"
key: "qa_dataset"
display_name: "QA Dataset"
description: "The dataset to evaluate against."
- type: "model"
key: "judge_model"
display_name: "Judge Model"
- type: "categorical"
key: "evaluation_dimension"
display_name: "Evaluation Dimension"
description: "The aspect of the model response to evaluate."
allowed_values: ["groundedness", "relevance"]DatasetColumnParameterSpec
Properties
type Literal “dataset_column” required
The type of the parameter.
key string required
The key of the parameter.
display_name string required
The display name of the parameter.
description string
The description of the parameter.
Default: None
default_value string
The default value to use.
Default: None
nullable boolean
Whether this parameter is nullable.
Default: False
Dataset Column Parameter Definition
# ...
config_spec:
- type: "dataset_column"
key: "answer_column"
display_name: "Answer Column"
description: "Dataset column containing the correct single-character answer choice."ListParameterSpec
Properties
type Literal “list” required
The type of the parameter.
dtype enum ScalarDtype required
The data type of the elements in the list.
The scalar data type.
Allowed Values:
stringintegerfloatboolean
key string required
The key of the parameter.
display_name string required
The display name of the parameter.
description string
The description of the parameter.
Default: None
default_value array[Any]
The default value to use.
Default: None
nullable boolean
Whether this parameter is nullable.
Default: False
DictParameterSpec
Properties
type Literal “dict” required
The type of the parameter.
value_dtype enum ScalarDtype required
The data type of the values in the dict.
The scalar data type.
Allowed Values:
stringintegerfloatboolean
key string required
The key of the parameter.
display_name string required
The display name of the parameter.
description string
The description of the parameter.
Default: None
default_value object
The default value to use.
Default: None
nullable boolean
Whether this parameter is nullable.
Default: False
CategoricalParameterSpec
Properties
type Literal “categorical” required
The type of parameter.
key string required
The key of the parameter.
display_name string required
The display name of the parameter.
description string
The description of the parameter.
Default: None
allowed_values array[string] required
Allowed Values
multiple boolean
Whether the parameter can have multiple values.
Default: False
default_value string
The default value to use.
Default: None
nullable boolean
Whether this parameter is nullable.
Default: False
values_mapping object
A mapping over the categorical values.
Default: None
Categorical Parameter Definition
# ...
config_spec:
- type: "dataset"
key: "qa_dataset"
display_name: "QA Dataset"
description: "The dataset to evaluate against."
- type: "model"
key: "judge_model"
display_name: "Judge Model"
- type: "categorical"
key: "evaluation_dimension"
display_name: "Evaluation Dimension"
description: "The aspect of the model response to evaluate."
allowed_values: ["groundedness", "relevance"]