Evidence
ModelTokenUsage
Token usage for a model, tagged with the model and its purpose.
Properties
model_id integer required
Model Id
purpose ModelTokenUsagePurpose required
prompt_tokens integer required
Prompt Tokens
completion_tokens integer required
Completion Tokens
RunEvidence
Properties
index integer required
Index
metrics array[MetricData] required
The metrics. If an error occurred, the metrics will be None.
samples array[SampleEvidence]
The sample evidence (as produced by the given run of a task).
Default: None
errors array[TaskResultError]
A list of task-level errors.
Default: []
failures TaskResultFailures
Default: None
SampleEvidence
Properties
sample_id string, integer required
Sample Id
sample SampleData required
Sample data. Only present for legacy evidence or when computing repeatability, otherwise present in the trials.
solver SolverData required
Solver data. Only present for legacy evidence, otherwise present in the trials.
scores array[ScoresData] required
Scores data. Only present for legacy evidence, or when score aggregation occurred (such as when using multiple trials with score aggregation or when assessing repeatability).
action_records array[ActionRecord]
Action Records
Default: None
errors array[TaskResultError] required
Errors
trials array[SampleTrialEvidence]
Trials
Methods
build classmethod
build(model_input: LFModelInput | None, model_output: LFModelOutput | None, score_values: LFBaseModel | ScoreValues, sample_id: int | str, score_metadata: LFBaseModel | dict[str, Any] | None = None, sample_data: dict[str, Any] | None = None, scorer_key: str | None = None, solver_model_direct_input: ModelEndpointInput | None = None, solver_model_direct_output: ModelEndpointOutput | None = None, scorer_model_direct_input: ModelEndpointInput | None = None, scorer_model_direct_output: ModelEndpointOutput | None = None, message_format: TraceFormat = 'open_responses') -> SampleEvidencebuild_with_1_trial classmethod
build_with_1_trial(*, sample_id: str | int, sample: SampleData | None, solver: SolverData | None, scores: list[ScoresData] | None, errors: list[TaskResultError]) -> SampleEvidenceSampleTrialEvidence
Properties
index integer required
Index
sample_id string, integer required
Sample Id
sample SampleData required
solver SolverData required
scores array[ScoresData] required
Scores
errors array[TaskResultError] required
Errors
SolverData
Properties
output SingleSolverOutput, GroupedSolverOutput, SolverTrace, GroupedSolverTrace required
Output
TaskResultEvidence
Properties
metrics array[MetricData] required
The metrics. If an error occurred, the metrics will be None.
samples array[SampleEvidence]
The sample evidence (as produced by tasks).
Default: None
runs array[RunEvidence]
Per-run evidence for repeatability task results. None when repeatability is not assessed.
Default: None
errors array[TaskResultError]
A list of task-level errors.
Default: []
failures TaskResultFailures
Default: None
model_token_usages array[ModelTokenUsage]
Model Token Usages
Default: []
Methods
adapt_metrics_if_needed classmethod
adapt_metrics_if_needed(value: Any) -> Anybuild_flat_metrics_dict method
build_flat_metrics_dict() -> MetricValuesTaskResultLog
Properties
format_version Literal “v1” required
Format Version
app_version string required
The version of AI GO that computed this task result log.
status string required
Status
evidence TaskResultEvidence required
specification TaskResultSpecification required
execution TaskExecution required
errors array[TaskResultError] required
Errors
Enums
ModelTokenUsagePurpose
Allowed Values:
testjudgegenerate