Quickstart

Note

Run your first evaluation in a few minutes. You will point the CLI at your deployment, create a workspace, integrate a model, then pull a packaged evaluation from AI Atlas and run it.

Before you begin, make sure you have:

• a live LF AI Platform deployment,
• the CLI installed in a Python environment,
• an OpenAI API key available as OPENAI_API_KEY (any supported provider works - adjust the model key accordingly).

If the entities below are new to you, read Core Concepts first.

Step 1: Configure the CLI

lf configure stores the backend URL, your API key, and the active AI app. Run it interactively, or copy the pre-filled command from the CLI Reference icon in the UI:

lf configure

Confirm the CLI is talking to the right deployment:

$ lf status
                              CLI configuration (4 rows)
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Name         ┃ Value                                     ┃ Source                   ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ URL          │ https://app.latticeflow.cloud             │ '~/.latticeflow.yaml'    │
│ API Key      │ ****61c1                                  │ '~/.latticeflow.yaml'    │
│ Verify SSL   │ True                                      │ '~/.latticeflow.yaml'    │
│ AI App Key   │ -                                         │ '~/.latticeflow.yaml'    │
└──────────────┴───────────────────────────────────────────┴──────────────────────────┘

Step 2: Create an AI App

An AI app is the workspace that holds all evaluation runs for a single AI use case. Define it, create it, and switch into it:

app.yaml

key: "quickstart"
display_name: "Quickstart"

lf add app -f app.yaml
lf switch quickstart

Every command that works in an AI-app-scoped manner from now on runs inside the quickstart app.

Step 3: Integrate the Model Under Test

Register your provider API key once, then add the model. The provider shorthand needs no model spec:

lf integration add --provider openai --api-key $OPENAI_API_KEY
lf add model -p openai/gpt-4.1-mini

The resulting model key uses a $ separator - here openai$gpt-4-1-mini. Verify the endpoint before you rely on it:

lf test model openai\$gpt-4-1-mini

Tip

lf list model prints the exact key of every integrated model. Escape or quote the $ so your shell does not expand it.

Step 4: Pull an Evaluation from AI Atlas

AI Atlas publishes packaged evaluations - each one bundles its datasets, tasks, and a run.yaml entry point. This example runs a harmful content evaluation:

lf init eval --key atlas-harmful_content

$ tree -L 1 atlas-harmful_content
atlas-harmful_content
├── README.md
├── config.yaml
├── datasets
├── evaluation.yaml
└── run.yaml

Step 5: Configure the Evaluation Run

Each package exposes its configuration through config.yaml. This one needs the model under test and a judge model - both can be the model you just added:

atlas-harmful_content/config.yaml

config:
  # Evaluated Model: The model to be evaluated.
  model_key: openai$gpt-4-1-mini
  # Judge Model: The key of the judge model used to evaluate model responses.
  judge_model_key: openai$gpt-4-1-mini

Note

Run lf overview eval in the extracted directory to see which values still need to be set and the next command to run.

This package runs 22 tasks over roughly 1,200 samples, each scored by the judge model. For a first run, cap the sample count by adding num_samples to the evaluation run’s run_config - then remove it once you trust the setup:

atlas-harmful_content/config.yaml

run_config:
  num_samples: 5

Step 6: Add Dependencies and Run

Add the dependencies declared in the package’s run.yaml.

lf add -f atlas-harmful_content/run.yaml

You will see output similar to:

[Dataset(key="ailuminate_defamation")] Created successfully
Creating 1 missing tag(s).
Successfully created tag with value 'misinformation'.
Newly linked tags:
+ harmful_content
+ misinformation
+ safety
...

Run the evaluation, passing config.yaml via -c.

lf run eval -k atlas-harmful_content -c atlas-harmful_content/config.yaml

You will see output similar to:

AI App Key: my-app
[EvaluationRun(ID="1", evaluation_key="atlas-harmful_content")] Started successfully.
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Evaluation run overview available at:

https://<your-app-url>/ai-apps/1/evaluate/evaluations/1

Or in the CLI using:

lf overview eval-run --id 1

Step 7: Explore the Results

Open the printed link to review aggregate metrics in the UI and drill into individual model responses, scores, and traces. From the terminal:

lf overview eval-run --id 1

Download the full evidence - metrics, per-sample scores, and traces - for archiving or offline analysis:

lf export eval-run --id 1 -o ./results

Next Steps

You ran an evaluation somebody else authored. Next, make it yours:

Bring your own building blocks

Integrate your own models and datasets instead of a provider model.

Author your own task

Define solvers, scorers, and metrics for your own evaluation logic.

Connect your Agent

Let your coding agent write and run the specs for you.

Also worth a look:

Run packaged evaluations from Atlas

Configure packaged evaluations in more depth, including ZIP packages.

CLI Quickstart

The same end-to-end flow, authoring every spec from scratch.