Quickstart

Note

Run your first evaluation in a few minutes. You will point the CLI at your deployment, create a workspace, integrate a model, then pull a packaged evaluation from AI Atlas and run it.

Before you begin, make sure you have:

If the entities below are new to you, read Core Concepts first.

Step 1: Configure the CLI

lf configure stores the backend URL, your API key, and the active AI app. Run it interactively, or copy the pre-filled command from the CLI Reference icon in the UI:

lf configure

Confirm the CLI is talking to the right deployment:

$ lf status
                              CLI configuration (4 rows)
โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”“
โ”ƒ Name         โ”ƒ Value                                     โ”ƒ Source                   โ”ƒ
โ”กโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ฉ
โ”‚ URL          โ”‚ https://app.latticeflow.cloud             โ”‚ '~/.latticeflow.yaml'    โ”‚
โ”‚ API Key      โ”‚ ****61c1                                  โ”‚ '~/.latticeflow.yaml'    โ”‚
โ”‚ Verify SSL   โ”‚ True                                      โ”‚ '~/.latticeflow.yaml'    โ”‚
โ”‚ AI App Key   โ”‚ -                                         โ”‚ '~/.latticeflow.yaml'    โ”‚
โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜

Step 2: Create an AI App

An AI app is the workspace that holds every entity for one use case. Define it, create it, and switch into it:

app.yaml
key: "quickstart"
display_name: "Quickstart"
lf add app -f app.yaml
lf switch quickstart

Every command from now on runs inside the quickstart app.

Step 3: Integrate the Model Under Test

Register your provider API key once, then add the model. The provider shorthand needs no model spec:

lf integration add --provider openai --api-key $OPENAI_API_KEY
lf add model -p openai/gpt-4.1-mini

The resulting model key uses a $ separator - here openai$gpt-4-1-mini. Verify the endpoint before you rely on it:

lf test model openai\$gpt-4-1-mini
Tip

lf list model prints the exact key of every model in the app. Escape or quote the $ so your shell does not expand it.

Step 4: Pull an Evaluation from AI Atlas

AI Atlas publishes packaged evaluations - each one bundles its datasets, tasks, and a run.yaml entry point. This example runs a harmful content evaluation:

lf init --atlas harmful_content
$ tree -L 1 harmful_content
harmful_content
โ”œโ”€โ”€ config.env
โ”œโ”€โ”€ datasets
โ”œโ”€โ”€ evaluation.yaml
โ”œโ”€โ”€ README.md
โ”œโ”€โ”€ RUN.md
โ”œโ”€โ”€ run.yaml
โ”œโ”€โ”€ tasks
โ””โ”€โ”€ utils

Step 5: Configure the Evaluation

Each package exposes its configuration through config.env. This one needs the model under test and a judge model - both can be the model you just added:

harmful_content/config.env
# The key of the model to be evaluated.
MODEL_KEY="openai$gpt-4-1-mini"
# The key of the model to be used as judge.
JUDGE_MODEL_KEY="openai$gpt-4-1-mini"
Note

RUN.md in the extracted directory documents the configuration options and prerequisites of that specific evaluation. Read it whenever a value is unclear.

This package runs 22 tasks over roughly 1,200 samples, each scored by the judge model. For a first run, cap the sample count by adding num_samples to the evaluation spec - then remove it once you trust the setup:

harmful_content/evaluation.yaml
key: "harmful_content"
display_name: "Harmful Content"
num_samples: 5
# ... the rest of the spec stays unchanged

Step 6: Validate and Run

Validate first - this checks the specs without creating entities or calling a model:

lf --env harmful_content/config.env run -f harmful_content/run.yaml -v

Once validation passes, drop the -v to create the entities and start the evaluation:

$ lf --env harmful_content/config.env run -f harmful_content/run.yaml
On AI app 'quickstart'.
[Dataset(key="ailuminate_hate")] Created successfully
[Task(key="ailuminate_hate")] Created successfully
[Evaluation(ID="1")] Created successfully
[Evaluation(ID="1")] Started successfully.
----------------------------------------------------------------------------------
Evaluation overview available at:

https://app.latticeflow.cloud/ai-apps/1/evaluations

Or in the CLI using:

lf overview eval --id 1
Tip

To smoke-test the pipeline on a single sample before running everything, use lf test task -f harmful_content/run.yaml --spec-key <task_key>.

Step 7: Explore the Results

Open the printed link to review aggregate metrics in the UI and drill into individual model responses, scores, and traces. From the terminal:

lf overview eval --id 1

Download the full evidence - metrics, per-sample scores, and traces - for archiving or offline analysis:

lf export eval --id 1 -o ./results

Next Steps

You ran an evaluation somebody else authored. Next, make it yours:

Bring your own building blocks
Integrate your own models and datasets instead of a provider model.
Author your own task
Define solvers, scorers, and metrics for your own evaluation logic.
Connect your Agent
Let your coding agent write and run the specs for you.

Also worth a look:

Run packaged evaluations from Atlas
Configure packaged evaluations in more depth, including ZIP packages.
CLI Quickstart
The same end-to-end flow, authoring every spec from scratch.