Run Evaluation From AI Atlas

Note

In this tutorial, we will run a harmful content evaluation for an OpenAI model using lf init --atlas to download the evaluation package from AI Atlas.

Before You Begin

  • You need a live LF AI Platform deployment.
  • You need a Python environment with the LF AI Platform CLI installed. Follow the CLI installation page.
  • You need a configured CLI. Follow the CLI configuration steps.
  • You need an OpenAI API key set as OPENAI_API_KEY in your environment.

Step 1: Create an AI App

Create an AI app to use as a workspace for the evaluation.

  1. Define the app in a YAML file.
app.yaml
display_name: "My App"
key: "my-app"
  1. Create the app and switch to it.
lf add app -f app.yaml
lf switch my-app
  1. Confirm the app is active.
$ lf status
Working on AI app with key 'my-app'.

Step 2: Add the Model Under Test

Define and add the model you want to evaluate. This example uses OpenAI GPT-4.1 Nano.

  1. Define the model in a YAML file.
model.yaml
display_name: "OpenAI GPT-4.1 Nano"
key: "openai-gpt-4-1-nano"
task: "chat_completion"
config:
  connection_type: "custom_connection"
  adapter:
    key: "latticeflow$openai_chat_completion"
  url: "https://api.openai.com/v1/chat/completions"
  api_key: $OPENAI_API_KEY
  model_key: "gpt-4.1-nano"
  1. Add the model.
lf add model -f model.yaml

Step 3: Initialize the Evaluation from AI Atlas

Download the malformed_inputs evaluation package from AI Atlas into your working directory.

lf init --atlas atlas-malformed_inputs

This creates a atlas-malformed_inputs/ directory containing the datasets and dataset generators, a config.env and a config.yaml file, and a RUN.md with evaluation-specific instructions. There is also an INTENDED_USE.md file to be filled out.

Step 4: Configure the Dependencies and the Evaluation

Open atlas-malformed_inputs/config.env and set the values required for dataset generation.

atlas-malformed_inputs/config.env
# The key of the model used to generate data.
SYNTHESIZER_MODEL_KEY="openai-gpt-4-1-nano"

Open atlas-malformed_inputs/INTENDED_USE.md and adjust the intended use of the system. We choose a hypothetical scenario here.

atlas-malformed_inputs/INTENDED_USE.md
A helpful assistant that answers customer questions about our phone subscription plans.
The assistant can discuss quotas, signal reception, phone models, extra features, currently running promotions, and pricing.
The assistant must not negotiate the price or give discounts outside of existing running promotions.
The assistant must not answer legal questions.

Open atlas-malformed_inputs/config.yaml and set the values required for dataset generation.

atlas-malformed_inputs/config.yaml
config:
  # Evaluated Model: The model to be evaluated.
  model_key: openai-gpt-4-1-nano
  # Judge Model: The key of the judge model used to score how the system handled the degenerate input.
  judge_model_key: openai-gpt-4-1-nano

Step 5: Add the Dependencies and Run the Evaluation

Add the dependencies, passing config.env via --env.

lf --env atlas-malformed_inputs/config.env add -f atlas-malformed_inputs/run.yaml

You will see output similar to:

[Dataset generator(key="malformed_oversized_truncated_generator")] Created successfully
[Dataset generator(key="malformed_nontext_noise_generator")] Created successfully
[Dataset generator(key="malformed_language_codeswitch_generator")] Created successfully
[Dataset generator(key="malformed_structured_noise_generator")] Created successfully
[Dataset(key="malformed_empty_degenerate")] Created successfully
[Dataset(key="malformed_gibberish")] Created successfully
[Dataset(key="malformed_oversized_truncated")] Created successfully
[Dataset(key="malformed_nontext_noise")] Created successfully
[Dataset(key="malformed_language_codeswitch")] Created successfully
[Dataset(key="malformed_structured_noise")] Created successfully

Run the evaluation, passing config.yaml via -c.

lf run eval -k atlas-malformed_inputs -c atlas-malformed_inputs/config.yaml

You will see output similar to:

AI App Key: my-app
[EvaluationRun(ID="1", evaluation_key="atlas-malformed_inputs")] Started successfully.
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Evaluation run overview available at:

https://<your-app-url>/ai-apps/1/evaluate/evaluations/1

Or in the CLI using:

lf overview eval-run --id 1

Step 6: Explore Results

  1. Check the evaluation status in the CLI.
lf overview eval-run --id 1

  1. Open the evaluations page in the UI to see all evaluation runs and aggregate metrics.

  2. Drill into individual model responses and scores via the task result sidebar.