Import Traces

Your agent has already had thousands of conversations in production. Importing them as a dataset lets you evaluate what actually happened, rather than what happens when you prompt the system again: no inference, no non-determinism, no cost per sample.

The LF AI Platform ships connectors for four sources of recorded conversations. Each one produces a dataset of traces, which a task then scores with a pass-through solver - a solver that reads the recorded conversation instead of calling a model.

Install the Connectors

The trace connectors are an optional extra of the CLI, because they pull in the client libraries of the observability platforms:

uv pip install 'latticeflow-go-sdk[traces]'

Choose a Connector

Connector	`type`	Pulls	Credentials
LangSmith	`langsmith`	Traces from a `project`, or examples from a `dataset`.	`api_key`, optional `api_url`
Phoenix	`phoenix`	Traces from an Arize Phoenix `project`.	`api_key`, `base_url`
Claude Code	`claude_code`	Local Claude Code session files.	none - reads the local filesystem
Inspect AI	`inspect_ai`	Local Inspect AI `.eval` logs or a transcript database.	none - reads the local filesystem

Warning

Environment variable names. When api_key or the endpoint is omitted, the underlying client falls back to its own environment variables: LANGSMITH_API_KEY and LANGSMITH_ENDPOINT for LangSmith, PHOENIX_API_KEY and PHOENIX_COLLECTOR_ENDPOINT for Phoenix.

Those are the fallback names. If you reference a variable explicitly in the spec - api_key: $MY_LANGSMITH_KEY - the name is yours to choose, and the CLI expands it from .env. Mixing the two conventions is the most common reason an import silently returns zero traces.

Step 1: Declare the Dataset

A trace dataset is an ordinary dataset with a trace source. Narrow the import at the source with the connector’s own options - from_time/to_time, tags, limit - and start with a small limit while you get the shape right.

datasets/langsmith_traces.yaml

key: "langsmith-traces"
display_name: "LangSmith Traces"
description: "Production traces imported from a LangSmith project."
source:
  type: "langsmith"
  project: $LANGSMITH_PROJECT
  api_key: $LANGSMITH_API_KEY
  from_time: "2025-01-01T00:00:00Z"
  tags: ["production"]
  limit: 100

Set dataset: instead of project: to import the examples of a LangSmith dataset rather than the traces of a project.

datasets/phoenix_traces.yaml

key: "phoenix-traces"
display_name: "Phoenix Traces"
description: "Agent traces imported from an Arize Phoenix project."
source:
  type: "phoenix"
  project: "my-agent"
  base_url: $PHOENIX_BASE_URL
  api_key: $PHOENIX_API_KEY
  metadata:
    environment: "production"
  limit: 100

metadata entries must all match for a trace to be imported. For a self-hosted Phoenix instance without authentication, any non-empty api_key will do.

datasets/claude_code_sessions.yaml

key: "claude-code-sessions"
display_name: "Claude Code Sessions"
description: "Local Claude Code coding sessions."
source:
  type: "claude_code"
  path: "~/.claude/projects/my-project"   # optional; defaults to ~/.claude/projects/
  from_time: "2025-01-01T00:00:00Z"
  limit: 50

datasets/inspect_transcripts.yaml

key: "inspect-transcripts"
display_name: "Inspect AI Transcripts"
description: "Transcripts read from Inspect AI eval logs."
source:
  type: "inspect_ai"
  path: "./logs/"   # a directory of .eval files, a single .eval file, or a transcript DB
  limit: 200

Step 2: Create the Dataset

lf add dataset -f datasets/phoenix_traces.yaml

The import happens in the CLI, not on the server: the connector runs locally, converts each transcript, and uploads the result. That has two practical consequences. Your workstation needs network access to the observability platform - the LF AI Platform deployment does not. And for claude_code and inspect_ai, the files are read from the machine running the CLI, so paths are local paths.

Step 3: Know What You Got

Whatever the connector, the resulting dataset has exactly two columns:

Column	Contents
`trace`	The conversation as a serialized `Trace` - messages, function calls, function call outputs and, for event-based imports, the full event stream with agent spans.
`source_format`	The constant `inspect_ai`, recording that this dataset was produced by a trace import.

The originating platform, trace id and agent metadata survive inside the trace itself, under trace.metadata (trace_id, source_type, source_uri, agent, model, total_tokens, …). See Traces.

Confirm the import before building a task on it:

lf export dataset 'phoenix-traces' -do ./traces.jsonl

Important

Two limitations follow from that two-column shape:

No ground truth. There is no target or expected_answer column, so scorers that compare against a reference - string_equals and friends - have nothing to compare to. Score imported traces by inspecting them (a Python scorer) or by judging them (a model-as-a-judge scorer).
No filters. Unlike file-based sources, trace sources cannot be filtered after loading. Narrow the import with the connector’s tags, from_time/to_time, metadata and limit options instead.

Bring Your Own Log Format

If your conversations live somewhere without a connector - a database, a bespoke JSON log, a vendor with no export API - convert them yourself into the trace format and load the result as a local dataset. The tutorial Convert Your Logs into LF AI Platform Datasets walks through a complete converter, and everything from Step 4 above applies unchanged afterwards.

Next Steps

Score traces per sample

Write a scorer that applies different assertions to each recorded conversation.

Trace types

The items, events and metadata that make up a trace.