Quickstart

Note

This quickstart takes you from an empty terminal to a running evaluation. You will install the CLI, create an AI app, define a model, dataset, and task from scratch, then run an evaluation that scores a model with an LLM judge.

You will need a live AI Platform deployment and an OpenAI API key.

Step 1: Install the CLI

The AI Platform CLI ships as a lightweight Python package. Install it into a virtual environment:

uv pip install latticeflow-go-sdk
pip install latticeflow-go-sdk

The CLI and the AI Platform application must run the same version. See the CLI installation guide for supported Python versions and how to find your application version.

Step 2: Configure the CLI

lf configure stores the backend URL, your API key, and the active AI app. Run it interactively, or copy the pre-filled command from the CLI Reference icon in the UI:

lf configure

Confirm your context at any time:

$ lf status
                              CLI configuration (4 rows)                               
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Name         ┃ Value                                     ┃ Source                   ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ URL          │ https://app.latticeflow.cloud             │ '~/.latticeflow.yaml'    │
│ API Key      │ ****61c1                                  │ '~/.latticeflow.yaml'    │
│ Verify SSL   │ True                                      │ '~/.latticeflow.yaml'    │
│ AI App Key   │ my-app                                    │ '~/.latticeflow.yaml'    │
└──────────────┴───────────────────────────────────────────┴──────────────────────────┘

See the lf configure reference for all options.

Step 3: Create an AI App

An AI app is the workspace that groups all of your entities. Create one from a small spec:

app.yaml
key: quickstart
display_name: Quickstart
lf add app -f app.yaml
lf switch quickstart
TipIn a hurry?

lf init --atlas <key> scaffolds a complete, runnable project — model, dataset, task, and run.yaml — from a curated template. See Run an evaluation from AI Atlas and the lf init reference.

Step 4: Add Your Model Credentials

Register your OpenAI API key once so any OpenAI model can use it:

lf integration add --provider openai --api-key $OPENAI_API_KEY

You will reference the model itself with the $provider shorthand in the next step — no full model spec required.

Step 5: Define Your Entities

A single run.yaml can declare every entity an evaluation needs. Create a project folder with a small dataset alongside it:

quickstart/
├── run.yaml
└── data/
    └── samples.csv
data/samples.csv
question,expected_answer
What is the capital of Japan?,Tokyo
What is the capital of France?,Paris
What is the capital of Brazil?,Brasília

The run.yaml below wires together four pieces:

run.yaml
models:
  - $provider: "openai/gpt-5-nano-2025-08-07"

datasets:
  - key: capitals
    display_name: World Capitals
    source:
      type: local
      file_path: "./data/samples.csv"

tasks:
  - key: capitals-qa
    display_name: Capitals QA
    description: "Checks whether the model names the correct capital city."
    config_spec:
      - type: model
        key: judge_model
        display_name: Judge Model
    definition:
      evaluated_entity_type: model
      dataset:
        key: capitals
      solver:
        type: single_turn_solver
        input_builder:
          type: chat_completion
          input_messages:
            - role: system
              content: "Answer with the city name only."
            - role: user
              content: "{{ sample.question }}"
      scorers:
        - type: model_as_a_judge_classifier
          model_key: "<< config.judge_model >>"
          system_prompt: |
            Decide whether the response gives the correct answer.
            Output exactly PASS or FAIL.
          user_prompt: |
            Question: {{ sample.question }}
            Expected: {{ sample.expected_answer }}
            Response: {{ trace.get_last_assistant_text() }}
          correct_labels: ["PASS"]
          incorrect_labels: ["FAIL"]
          use_structured_outputs: true
          metrics:
            - type: mean
              field: is_correct
              name: accuracy

evaluation:
  key: capitals-eval
  display_name: Capitals Evaluation
  task_specifications:
    - task_key: capitals-qa
      model_key: "openai$gpt-5-nano-2025-08-07"
      task_config:
        judge_model: "openai$gpt-5-nano-2025-08-07"

The same model serves as both the evaluated model and the judge here; in practice you can point judge_model at any other model key.

Step 6: Validate and Run

Always validate first — this checks the spec without creating anything or calling a model:

lf run -f run.yaml -v

Once validation passes, create the entities and start the evaluation:

lf run -f run.yaml
On AI app 'quickstart'.
[Model(key="openai$gpt-5-nano-2025-08-07")] Integrated successfully
[Dataset(key="capitals")] Created successfully
[Task(key="capitals-qa")] Created successfully
[Evaluation(ID="1")] Created successfully
[Evaluation(ID="1")] Started successfully.
----------------------------------------------------------------------------------
Evaluation overview available at:

https://app.latticeflow.cloud/ai-apps/1/evaluations
Tip

To smoke-test the full pipeline on a single sample before running everything, use lf test task -f run.yaml --spec-key capitals-qa.

See the lf run reference for all flags.

Step 7: View Results

Open the link printed in the terminal, or inspect results from the CLI:

lf overview eval --id 1

Download the full results — metrics, per-sample evidence, and traces — to a local folder:

lf export eval --id 1 -o ./results

Next Steps