Datasets

A dataset is the collection of samples a task evaluates. Each sample is a row whose columns are addressable from task templates as {{ sample.<column> }}, so the dataset determines both the inputs sent to the evaluated entity and the ground truth its output is scored against.

# datasets/hp_trivia.yaml
key: "hp-trivia-dataset"
display_name: "Harry Potter Trivia"
description: "Trivia questions with reference answers."
source:
  type: "local"
  file_path: "./hp_trivia.csv"

Where samples come from

A dataset takes its samples from exactly one of two places:

source - an existing collection of samples, e.g. a local CSV/JSONL file, a URL, or a Hugging Face, LangSmith, Phoenix, Claude Code or Inspect AI export. See Dataset Sources for the available types and the integrate a dataset guide.
generator_specification - a dataset generator that synthesizes the samples, optionally parameterized per evaluation. See the dataset generation guide.

Working with datasets

Register a dataset with lf add dataset, inspect the samples that LF AI Platform actually resolved with lf test dataset, and list the registered datasets with lf list dataset. For a generated dataset, lf regenerate dataset produces a fresh set of samples from the same generator.

Configuration

Properties

key string required

Unique identifier assigned to the entity in AI Platform.

Pattern: ^[a-zA-Z0-9_\-]+$
Max Length: 250

display_name string required

The dataset’s name displayed to the user.

description string

Short description of the dataset.

Default: None

long_description string

Long description of the dataset. Supports Markdown formatting.

Default: None

source LocalDatasetSource, FolderDatasetSource, URLDatasetSource, HuggingFaceDatasetSource, LangSmithDatasetSource, PhoenixDatasetSource, ClaudeCodeDatasetSource, InspectAIDatasetSource

Dataset source configuration. Required if dataset generator is not used.

Default: None

generator_specification SDKDatasetGeneratorSpecification

Config for the dataset generator that will be used to generate the dataset. Required if the data file is not provided.

Default: None

tags array[string]

Tags associated with the dataset.

Default: []

display_name: "Legal Questions"
key: "legal-questions"
description: "Generated using 'My Question Generator'"
generator_specification:
  dataset_generator_key: "question-generator"
  num_samples: 5
  dataset_generator_config:
    topic: "Legal"
    synthesizer_model_key: "openai$gpt-4-1-nano"

display_name: "My Question Generator"
key: "question-generator"
description: "Generates questions about arbitrary topics."
config_spec:
  - key: "synthesizer_model_key"
    type: "model"
    display_name: "Synthesizer Model"
    description: "Model used to generate questions."
  - key: "topic"
    type: "string"
    display_name: "Topic"
    description: "Topic for which questions will be generated."
definition:
  type: "declarative_dataset_generator"
  data_source:
    type: "empty"
  synthesizers:
    - type: "llm"
      model_key: "<< config.synthesizer_model_key >>"
      use_structured_outputs: true
      system_prompt: "You are a helpful dataset generator."
      user_prompt_template: >
        Produce 5 deep knowledge questions about << config.topic >>, including the
        correct answer.
      sample_properties:
        question:
          type: "string"
        answer:
          type: "string"

Related types

`SDKDatasetGeneratorSpecification`

Properties

dataset_generator_config object required

The configuration used by the dataset generator.

num_samples integer required

The number of samples to generate. At least 1 sample must be requested.

debug DatasetGenerationDebugOptions

Default: None

dataset_generator_key string required

Key of the dataset generator to use.

Dataset Generator Configuration

# ...
generator_specification:
  dataset_generator_key: "question-generator"
  num_samples: 5
  dataset_generator_config:
    topic: "Legal"
    synthesizer_model_key: "openai$gpt-4-1-nano"

Note

The dataset_generator_config allows instantiating the same dataset generator template in different ways. Always refer to the dataset generator definition for the available configuration options.

`DatasetGenerationDebugOptions`

Properties

enabled boolean required

When true, the response will include a full pipeline trace for each source sample, which contains the source sample itself and the input and output at each synthesizer stage.

include_io boolean required

When true, the model input and output are included in the trace for each synthesizer call that produced I/O. Has no effect when enabled is false.