Assessment
ActionRecord
A record of an action taken for a sample.
Properties
action ActionRuleAction required
rule_key string required
The key of the action rule that created the action record.
Pattern: ^[a-zA-Z0-9_\-]+$
Max Length: 250
CustomTaskInputMessage
A custom task input item carrying an opaque user-defined payload.
Properties
type Literal “custom_task_input_message”
Type
Default: custom_task_input_message
content Any required
The opaque user-defined payload.
CustomTaskOutputMessage
A custom task output item carrying an opaque user-defined payload.
Properties
type Literal “custom_task_output_message”
Type
Default: custom_task_output_message
content Any required
The opaque user-defined payload.
DirectModelIO
Raw model endpoint I/O before/after adapter conversion.
Properties
direct_input ModelEndpointInput required
The raw request sent to the model endpoint.
direct_output ModelEndpointOutput required
The raw response received from the model endpoint.
FieldMetadata
Metadata for a single field in a tabular evidence table.
Properties
display_name string
Display name for the field.
Default: None
description string
Description of the field semantics.
Default: None
primary boolean
Whether the field is directly relevant to the understanding of the main correctness result.
Default: True
MetricData
An object that contains the metric scores.
Properties
values object required
Values
metric_key string required
The key of the metric.
metric_type string
The type of the metric. Present for newly computed results and may be missing for legacy results.
Default: None
scorer_name string
The display name of the scorer to which the metric belongs. Present only for benchmark tasks.
Default: None
scorer_key string
The key of the scorer to which the metric belongs. Present only for benchmark tasks.
Default: None
scorer_purpose ScorerPurpose
The purpose of the scorer. Present only for benchmark tasks.
Default: score
reason string
A freeform explanation for the metric value. Present only for system tasks.
Default: None
ModelEndpointInput
The raw request body sent to the model endpoint after adapter conversion.
Properties
body string required
The raw request body as a JSON string.
ModelEndpointOutput
The raw response received from the model endpoint before adapter conversion.
Properties
body string required
The raw response body as a JSON string.
status_code integer required
The HTTP status code of the response.
headers object
The HTTP response headers.
Default: None
SampleData
The raw sample data for a single evaluated sample.
Properties
data object required
The sample’s field values.
ScoresData
The score values produced by a single scorer for a single sample.
Properties
scorer_key string required
The key identifying the scorer.
scorer_purpose ScorerPurpose
Default: score
scorer_name string
Optional display name of the scorer.
Default: None
values object required
The score values produced by the scorer.
metadata object
Optional metadata associated with the scorer output.
Default: None
direct_ios array[DirectModelIO]
Raw model endpoint I/O for each prediction call (if scorer uses a model).
Default: []
Secret
An object representing a secret.
Properties
name string required
The name of the secret.
Pattern: ^[A-Za-z0-9_-]+$
SystemTaskMetricEntry
A single metric entry produced by a system task’s compute_evidence function.
Properties
value number, integer required
The numeric value of the metric.
reason string
A freeform explanation for the metric value. Mapped to the reason field in MetricData.
Default: None
SystemTaskOutput
The expected return value of the compute_evidence function in a system task snippet.
Properties
metrics object required
A mapping from metric key to the metric entry.
TLSContext
Defines the TLS context.
Properties
validation_context CertificateValidationContext
Settings for validating server certificates.
Default: None
TaskExecution
Timing and resource usage information for a task execution.
Properties
runtime number required
The runtime of the task in seconds.
started_at integer required
A Unix timestamp in seconds.
ended_at integer required
A Unix timestamp in seconds.
model_usage ModelUsageStats
Default: None
TaskResultError
Properties
error_type string required
The type of the error.
message string required
The specific error message that occurred during evaluation.
hint string
The suggestion to try out to fix the issue.
Default: None
stage TaskResultErrorStage
Default: None
TaskResultFailures
Properties
num_errors integer required
Num Errors
num_total integer required
Num Total
TaskResultSpecification
The task specification stored inside a TaskResultLog, capturing what was evaluated and how.
Properties
display_name string required
The display name of the evaluation.
task StoredTask required
config object required
Task configuration used for this evaluation.
evaluated_entity StoredDataset, StoredModel
The dataset or model that was evaluated. Present only for benchmark tasks.
Default: None
run_config EvaluationConfig required
repeatability_config RepeatabilityConfig
Default: None