Task Result Logs
A task result log is the complete, self-contained record of running one task specification: the aggregated metrics, the evidence for every sample, the configuration that produced them, and any errors along the way. It is what makes a reported number auditable - every metric can be traced back to the samples, outputs and scores it came from.
What a log contains
| Field | Contents |
|---|---|
status |
Whether the task run succeeded (success) or failed (failure). |
evidence |
The results themselves: metrics for the aggregated values, samples for the per-sample evidence, runs for repeated runs, plus errors, failures and token usage. |
specification |
The task, its configuration, the evaluated entity and the run config used - everything needed to interpret or reproduce the run. |
execution |
When the run started and ended, its runtime and model usage. |
errors |
Errors that prevented the task from completing. |
Each entry in evidence.samples holds one sample’s input, the solver output - including the full trace of the conversation - and the values every scorer produced for it. Metrics reported at the top are the aggregation of exactly these per-sample scores.
Working with task result logs
lf export eval-run downloads the logs of an evaluation for local inspection or downstream analysis. Feeding one back in is equally useful: setting task_result_log_path on a task specification uploads an existing log instead of running the task, which reproduces results in another AI app without repeating inference. See Download and Upload Evidence.
The log is also available as a typed object in the SDK, which is the recommended way to process it programmatically:
from latticeflow.core.dtypes import TaskResultLogConfiguration
Properties
format_version Literal “v1” required
Format Version
app_version string required
The version of AI Platform that computed this task result log.
status string required
Status
evidence TaskResultEvidence required
specification TaskResultSpecification required
execution TaskExecution required
errors array[TaskResultError] required
Errors
Types
TaskResultEvidence
Properties
metrics array[MetricData] required
The metrics. If an error occurred, the metrics will be None.
samples array[SampleEvidence]
The sample evidence (as produced by tasks).
Default: None
runs array[RunEvidence]
Per-run evidence for repeatability task results. None when repeatability is not assessed.
Default: None
errors array[TaskResultError]
A list of task-level errors.
Default: []
failures TaskResultFailures
Default: None
model_token_usages array[ModelTokenUsage]
Model Token Usages
Default: []
SampleEvidence
Properties
sample_id string, integer required
Sample Id
sample SampleData required
Sample data. Only present for legacy evidence or when computing repeatability, otherwise present in the trials.
solver SolverData required
Solver data. Only present for legacy evidence, otherwise present in the trials.
scores array[ScoresData] required
Scores data. Only present for legacy evidence, or when score aggregation occurred (such as when using multiple trials with score aggregation or when assessing repeatability).
action_records null
Action Records
Default: None
errors array[TaskResultError] required
Errors
trials array[SampleTrialEvidence]
Trials
SolverData
Properties
output null required
Output
SampleData
The raw sample data for a single evaluated sample.
Properties
data object required
The sample’s field values.
ScoresData
The score values produced by a single scorer for a single sample.
Properties
scorer_key string required
The key identifying the scorer.
scorer_purpose enum ScorerPurpose
Default: score
Allowed Values:
scoreqa
scorer_name string
Optional display name of the scorer.
Default: None
values object required
The score values produced by the scorer.
metadata object
Optional metadata associated with the scorer output.
Default: None
direct_ios array[null]
Raw model endpoint I/O for each prediction call (if scorer uses a model).
Default: []
MetricData
An object that contains the metric scores.
Properties
values object required
Values
metric_key string required
The key of the metric.
metric_type string
The type of the metric. Present for newly computed results and may be missing for legacy results.
Default: None
scorer_name string
The display name of the scorer to which the metric belongs. Present only for benchmark tasks.
Default: None
scorer_key string
The key of the scorer to which the metric belongs. Present only for benchmark tasks.
Default: None
scorer_purpose enum ScorerPurpose
The purpose of the scorer. Present only for benchmark tasks.
Default: score
Allowed Values:
scoreqa
reason string
A freeform explanation for the metric value. Present only for system tasks.
Default: None
TaskResultError
Properties
error_type string required
The type of the error.
message string required
The specific error message that occurred during evaluation.
hint string
The suggestion to try out to fix the issue.
Default: None
stage enum TaskResultErrorStage
Default: None
Allowed Values:
configurationdatasetsolverscoremetricaction
TaskResultFailures
Properties
num_errors integer required
Num Errors
num_total integer required
Num Total
TaskResultSpecification
The task specification stored inside a TaskResultLog, capturing what was evaluated and how.
Properties
display_name string required
The display name of the evaluation.
task StoredTask required
config object required
Task configuration used for this evaluation run.
evaluated_entity StoredDataset, StoredModel
The dataset or model that was evaluated. Present only for benchmark tasks.
Default: None
run_config null required
repeatability_config RepeatabilityConfig
Default: None
TaskExecution
Timing and resource usage information for a task execution.
Properties
runtime number required
The runtime of the task in seconds.
started_at integer required
A Unix timestamp in seconds.
ended_at integer required
A Unix timestamp in seconds.
model_usage null
Default: None
Trace
Represents a conversation trace between a user and an agent.
A trace stores a sequence of items in the Open Responses format, including user messages, assistant messages, function calls, and function call outputs. It provides helper methods to extract individual turns, find function calls, and inspect the conversation.
The preamble property exposes everything before the first user message (system messages, assistant greetings, initial function calls, etc.). Everything from the first user message onward is accessible via turns.
For multi-agent traces, an optional events field provides a richer execution record with span markers encoding agent hierarchy. Use Trace.from_events() to construct event-based traces; items is derived automatically.
Properties
FORMAT string
Format
Default: open_responses
items array[null] required
Items
metadata TraceMetadata
Default: None
events null
Events
Default: None
span_id string
Span Id
Default: None
span_name string
Span Name
Default: None
span_type string
Span Type
Default: None
TraceMetadata
Trace-level metadata capturing identity, provenance, and summary information.
All fields are optional. Only the trace data itself (items/events) is required. Metadata enriches the trace for filtering, grouping, and analysis.
Properties
trace_id string
Trace Id
Default: None
source_type string
Source Type
Default: None
source_uri string
Source Uri
Default: None
agent string
Agent
Default: None
model string
Model
Default: None
tags array[string]
Tags
Default: None
created_at string
Created At
Default: None
total_time number
Total Time
Default: None
total_tokens integer
Total Tokens
Default: None
message_count integer
Message Count
Default: None
error string
Error
Default: None
extra object
Extra
Default: None
RunEvidence
Properties
index integer required
Index
metrics array[MetricData] required
The metrics. If an error occurred, the metrics will be None.
samples array[SampleEvidence]
The sample evidence (as produced by the given run of a task).
Default: None
errors array[TaskResultError]
A list of task-level errors.
Default: []
failures TaskResultFailures
Default: None
RepeatabilityConfig
Configuration for a repeatability task result.
Properties
num_runs integer required
Number of times to run the task.
TrialsConfig
Configuration for trials in a task result/specification. Only relevant for benchmark tasks.
Properties
num_trials integer required
Number of trials to run per sample.
TrialsDefinitionTemplate
Configuration for trials in a benchmark task definition template.
Properties
num_trials integer, string required
Number of trials to run per sample.
score_aggregators array[ScoreAggregator]
Score aggregators that compute an aggregated score given the score values for the different trials. Scores with no matching aggregator default to mean for numeric and boolean values (for other dtypes, no default aggregation is computed).
Default: None
SampleTrialEvidence
Properties
index integer required
Index
sample_id string, integer required
Sample Id
sample SampleData required
solver SolverData required
scores array[ScoresData] required
Scores
errors array[TaskResultError] required
Errors
ScoreAggregator
Aggregation configuration for one or more score keys.
Properties
score_name string required
The name of the score that will be aggregated.
aggregator MeanAggregator, MinAggregator, MaxAggregator, PassAtKAggregator, PassPowerKAggregator required
Aggregator
MinAggregator
Aggregates numeric scores by taking the minimum value.
Properties
function Literal “min” required
Function
score_name string
The name to give to the aggregated score.
Default: None
MaxAggregator
Aggregates numeric scores by taking the maximum value.
Properties
function Literal “max” required
Function
score_name string
The name to give to the aggregated score.
Default: None
MeanAggregator
Aggregates numeric scores by computing the mean.
Properties
function Literal “mean” required
Function
score_name string
The name to give to the aggregated score.
Default: None
PassAtKAggregator
Aggregates binary (True/False) scores using the pass@k estimator. Estimates the probability that at least one of k independent attempts will succeed, computed as 1 - (1 - p)^k where p is the empirical pass rate across trials.
Properties
function Literal “pass@k” required
Function
k integer required
The number of independent attempts in the scenario being modelled.
score_name string
The name to give to the aggregated score.
Default: None
PassPowerKAggregator
Aggregates binary (True/False) scores using the pass^k estimator. Estimates the probability that an agent would succeed on all k independent attempts, computed as p^k where p is the empirical pass rate across trials.
Properties
function Literal “pass^k” required
Function
k integer required
The number of independent attempts in the scenario being modelled.
score_name string
The name to give to the aggregated score.
Default: None
AwsBedrockConnectionConfig
Connection configuration for an agent deployed as an AWS Bedrock AgentCore harness.
Properties
connection_type Literal “aws_bedrock” required
The type of connection config.
region string required
The AWS region in which the AgentCore harness is deployed.
harness_arn string required
The ARN of the AWS Bedrock AgentCore harness to invoke.
access_key_id string required
The AWS access key ID. Provide a raw string or reference a secret.
secret_access_key string required
The AWS secret access key. Provide a raw string or reference a secret.
session_token string
The optional AWS session token. Provide a raw string or reference a secret.
Default: None
PlaceholderConnectionConfig
Connection configuration for a placeholder external model that performs no inference and returns an empty completion.
Properties
connection_type Literal “placeholder” required
The type of connection config.
DifyConnectionConfig
Connection configuration for an agent deployed on Dify.
Properties
connection_type Literal “dify” required
The type of connection config.
url string required
The base URL of the Dify deployment.
api_key string required
The Dify API key. Provide a raw string or reference a secret.
AzureFoundryConnectionConfig
Connection configuration for an agent deployed on Azure AI Foundry.
Properties
connection_type Literal “azure_foundry” required
The type of connection config.
project_endpoint string required
The endpoint URL of the Azure AI Foundry project.
agent_name string required
The name of the Azure AI Foundry agent to invoke.
tenant_id string required
The Azure service-principal tenant ID. Provide a raw string or reference a secret.
client_id string required
The Azure service-principal client ID. Provide a raw string or reference a secret.
client_secret string required
The Azure service-principal client secret. Provide a raw string or reference a secret.
ClaudeManagedAgentsConnectionConfig
Connection configuration for an agent deployed as a Claude Managed Agent.
Properties
connection_type Literal “claude_managed_agents” required
The type of connection config.
agent_id string required
The identifier of the Claude managed agent to invoke.
environment_id string required
The identifier of the Claude managed-agent environment.
vault_id string required
The identifier of the Claude managed-agent vault.
api_key string required
The Anthropic API key. Provide a raw string or reference a secret.
LangSmithConnectionConfig
Connection configuration for an agent deployed on LangSmith / LangGraph Platform.
Properties
connection_type Literal “langsmith” required
The type of connection config.
deploy_url string required
The base URL of the LangGraph deployment.
assistant_id string required
The identifier of the LangGraph assistant to invoke.
api_key string required
The LangSmith API key. Provide a raw string or reference a secret.
ModelTokenUsage
Token usage for a model, tagged with the model and its purpose.
Properties
model_id integer required
Model Id
purpose enum ModelTokenUsagePurpose required
Allowed Values:
testjudgegenerateauxiliary
prompt_tokens integer required
Prompt Tokens
completion_tokens integer required
Completion Tokens
TokenUsage
Properties
prompt_tokens integer required
Prompt Tokens
completion_tokens integer required
Completion Tokens