Dataset Generation

The dataset your evaluation needs often does not exist: no public benchmark covers your domain, your production traces are too thin or too sensitive, and the ground truth was never written down. A dataset generator synthesizes that data instead - declaratively, as a pipeline of a data source that produces source samples and one or more synthesizers that transform them into the final dataset samples.

flowchart LR
    DS["Data Source"] -- "Source Samples" --> S1["Synthesizer 1"]
    S1 -- "Samples" --> S2["Synthesizer 2"]
    S2 --> D["Dataset Samples"]

    classDef pipeline fill:#e8f0fe,stroke:#1a73e8,stroke-width:2px,color:#174ea6
    classDef payload fill:#f1f3f4,stroke:#9aa0a6,stroke-width:2px,color:#3c4043

    class S1,S2 pipeline
    class DS,D payload

A generator is configuration, not a script you run and keep the output of: it lives next to your specs, is versioned with them, and takes parameters - so one generator can back many datasets, and any of them can be regenerated on demand.

A First Generator

Three seed topics in, question-and-answer rows out. The seeds are written inline, so this is the whole input:

dataset_generators/qa.yaml

key: "qa-generator"
display_name: "Q&A Generator"
description: "Turns a topic into a question with its reference answer."
config_spec:
  - type: "model"
    key: "synthesizer_model"
    display_name: "Synthesizer Model"
    default_value: "openai$gpt-4-1-nano"
definition:
  data_source:
    type: "inline_samples"
    samples:
      - topic: "Photosynthesis"
      - topic: "Black holes"
      - topic: "The water cycle"
  synthesizers:
    - type: "llm"
      model_key: "<< config.synthesizer_model >>"
      user_prompt_template: "Ask one factual question about {{ sample.topic }}, and answer it in one sentence."
      sample_properties:
        question:
          type: string
        answer:
          type: string

A dataset then declares how many samples it wants from that generator:

datasets/qa.yaml

key: "qa"
display_name: "Q&A"
generator_specification:
  dataset_generator_key: "qa-generator"
  num_samples: 3

Register the generator and preview the dataset. lf test dataset runs the pipeline and prints the samples without persisting anything:

lf add model -p openai/gpt-4.1-nano       # the model the synthesizer calls
lf add dataset-generator -f 'dataset_generators/qa.yaml'
lf test dataset -f 'datasets/qa.yaml' -n 3

Generated Samples

| topic            | question                                        | answer                                                                  |
| :--------------- | :---------------------------------------------- | :---------------------------------------------------------------------- |
| Photosynthesis   | What gas do plants release during photosynthesis? | Plants release oxygen as a by-product of photosynthesis.                |
| Black holes      | What is the event horizon of a black hole?        | It is the boundary beyond which nothing, not even light, can escape.    |
| The water cycle  | What drives evaporation in the water cycle?       | Solar energy heats surface water until it turns into vapour.            |

The source sample’s topic column survives into the output, and the synthesizer adds the columns it declared under sample_properties. Everything else on this page is a variation on those two blocks: a different data source, more synthesizers, or parameters instead of hard-coded values.

Once the samples look right, lf add dataset -f 'datasets/qa.yaml' generates and stores them.

Do You Need a Generator?

Generating data costs model calls and needs review, so use it only when the data cannot come from somewhere else.

Your situation	What to do
The data already exists as a file, a URL, a folder of documents, or a HuggingFace dataset.	Declare a dataset with a `source` - see Integrate a Dataset.
Your system already had the conversations you want to evaluate on.	Import them from your observability platform - see Import Traces.
You need data for your own domain, your own policies, or a scenario that has no benchmark.	Write a generator - continue with this guide.
You need to cover a grid of dimensions, such as every persona × every scenario.	Write a generator over the `dataset_sample_combinations` data source.
You have a seed dataset but it is missing a column, such as an answer, a label, or extracted claims.	Write a generator over the `dataset_samples` data source.

Building Blocks

Data Source

The data source determines what the pipeline starts from. It produces source samples, which act as the input to the first synthesizer.

A source sample may be one or several dataset rows, one or several documents, or nothing at all - for synthesizers that generate samples from scratch.

Data source	`type`	Starts from
Empty	`empty`	Nothing. Emits `num_samples` empty source samples (default `1`). Use it when the synthesizers generate everything, driven only by the generator’s configuration.
Dataset samples	`dataset_samples`	The samples of an existing dataset, selected by `dataset_key`. Set `random_seed` for a reproducible shuffle.
Inline samples	`inline_samples`	A literal list of samples written into the spec under `samples`. Handy for a handful of seeds you do not want to keep in a separate file.
Dataset sample combinations	`dataset_sample_combinations`	The cross product of several datasets, listed in `dataset_keys`. Use it to combine dimensions, e.g. every persona × every scenario.

See the Data Sources reference for the full configuration of each.

Important

Make sure the source can supply enough samples. Generation pulls source samples one at a time until the requested number of output samples exists. If the source is exhausted first, you get fewer samples than you asked for and a warning - not an error.

So a dataset asking for num_samples: 100 needs either a source that yields at least 100 samples (empty with num_samples: 100, or a seed dataset with 100 rows), or a synthesizer that produces several output samples per source sample. At most twice the requested count of source samples is consumed, which is the headroom for synthesizers that occasionally return nothing.

Note

A data source is not the same as a dataset source. A dataset source populates a standalone dataset; a data source seeds the generation pipeline inside a single dataset generator.

Synthesizers

A synthesizer takes a source sample and produces one or more output samples. definition.synthesizers is a list, and the synthesizers run in order - each one consumes the samples produced by the previous one, so a pipeline can generate text, then label it, then drop the intermediate columns.

Synthesizer	`type`	Produces
LLM	`llm`	New columns written by a chat-completion model from a prompt template.
Template	`template`	Samples rendered from a Jinja template with a grid of placeholder values.
Python	`python`	Whatever a custom `synthesize` function returns.
Question Answering	`question_answering`	Multiple-choice or open-ended QA pairs from source text.
Drop Columns	`drop_columns`	The same samples, minus the named `columns`.
Empty	`empty`	No output samples. Used when the data source alone defines the dataset.
Claim Extraction	`claim_extraction`	Atomic claims extracted from an answer.
Claim Labeling	`claim_labeling`	Each claim labelled as required or optional.
Link Claim Quotes	`link_claim_quotes`	Each claim linked to the verbatim quotes supporting it.

The last three are the building blocks for RAG claim graphs and are normally chained in that order.

A chained pipeline looks like this - generate an answer, then remove the scratch column it used:

dataset_generators/answers.yaml

config_spec:
  - type: "model"
    key: "synthesizer_model"
    display_name: "Synthesizer Model"
    default_value: "openai$gpt-4-1-nano"
  - type: "dataset"
    key: "dataset_key"
    display_name: "Dataset"
    default_value: "questions"
definition:
  data_source:
    type: "dataset_samples"
    dataset_key: "<< config.dataset_key >>"
  synthesizers:
    - type: "llm"
      model_key: "<< config.synthesizer_model >>"
      user_prompt_template: "Answer this question: {{ sample.question }}"
      sample_properties:
        answer:
          type: string
    - type: "drop_columns"
      columns: ["internal_notes"]

Parameterize a Generator

Hard-coded seeds get you one dataset. config_spec declares the parameters a generator accepts, using the same config specification as tasks. The definition refers to their values with << config.key >>, and each dataset supplies the values in its generator_specification.dataset_generator_config. That is what lets one generator serve many datasets:

dataset_generators/questions.yaml

key: "question-generator"
display_name: "Question Generator"
description: "Generates deep knowledge questions about a topic."
config_spec:
  - key: "topic"
    type: "string"
    display_name: "Topic"
  - type: "model"
    key: "synthesizer_model"
    display_name: "Synthesizer Model"
    default_value: "openai$gpt-4-1-nano"
definition:
  data_source:
    type: "empty"
    num_samples: 100
  synthesizers:
    - type: "llm"
      model_key: "<< config.synthesizer_model >>"
      system_prompt: "You are a helpful dataset generator."
      user_prompt_template: >
        Produce a deep knowledge question about << config.topic >>,
        including the correct answer.
      sample_properties:
        question:
          type: string
        answer:
          type: string

datasets/finance_questions.yaml

key: "finance-questions"
display_name: "Finance Questions"
generator_specification:
  dataset_generator_key: "question-generator"
  num_samples: 10
  dataset_generator_config:
    topic: "finance"

<< config.topic >> is resolved before the generator runs, from the dataset’s config. {{ sample.question }} is resolved while it runs, per sample. Credentials work the same way as elsewhere: declare them under secrets and reference them as << secrets.NAME >> - see Manage secrets.

Manage Generators and Datasets

Preview Before Generating

Generation costs model calls, so preview it first. Unlike a dataset backed by a source, a generated dataset can be sampled without being persisted:

lf test dataset -f dataset.yaml -n 5              # generate 5 samples, print them
lf test dataset -f dataset.yaml -n 5 -v           # also show each source sample and each synthesizer's input/output
lf test dataset -f dataset.yaml -n 5 -v --show-io # also show the model I/O of the synthesizers
lf test dataset -f run.yaml -k my-dataset         # pick one dataset out of a run config

-n accepts at most 10. --show-io is only meaningful together with -v, and is the fastest way to find out why a prompt is producing the wrong shape.

Register and Generate

If no generator with that key exists it is created; if one exists it is updated when the definition has changed. Adding the dataset is what actually runs the pipeline and stores the samples:

lf add dataset-generator -f 'dataset_generators/qa.yaml'  # A single generator.
lf add dataset-generator -f 'dataset_generators/*.yaml'   # Every generator in a directory.
lf add dataset -f 'datasets/qa.yaml'

Every property is documented in the Dataset Generator reference, and the remaining commands (lf list, lf export, lf delete) in the CLI reference.

Regenerate a Dataset

Regeneration is automatic: lf add and lf run re-run the generator whenever they detect that its configuration changed or that the dataset was never generated in its current form.

Force it with lf regenerate dataset when nothing in the config changed but you want fresh data anyway - because the samples were unsatisfactory, or because you want a different draw. The dataset must already exist in the AI app; this command never creates one.

lf regenerate dataset --file dataset.yaml               # a standalone dataset spec
lf regenerate dataset --file run.yaml --key my-dataset  # a dataset inside a run config, --key is required

Next Steps

Dataset Generator Reference

Every property of a dataset generator.

Data Sources Reference

All the ways a pipeline can start.

Synthesizer Reference

All available synthesizers and their configuration.

Integrate a Dataset

Use data you already have instead.