Dataset Generation

A dataset generator takes input data and produces a dataset from it. For example: turn a folder of PDF or Markdown documents into a question-and-answer test set, add reference answers to a list of questions you already have, or generate red-teaming prompts specific to your application.

A generator is a pipeline: a single data source supplies the input rows, and a sequence of synthesizers process each row into the final dataset samples.

flowchart LR
    DS["Data Source"]

    S1["Synthesizer 1"]
    S2["Synthesizer 2"]
    S3["Synthesizer 3"]
    OUT["Dataset Samples"]

    DS -- "source samples" --> S1 --> S2 --> S3 --> OUT

    classDef pipeline fill:#e8f0fe,stroke:#1a73e8,stroke-width:2px,color:#174ea6
    classDef payload fill:#f1f3f4,stroke:#9aa0a6,stroke-width:2px,color:#3c4043

    class S1,S2,S3 pipeline
    class DS,OUT payload

A data source can be, for example:

• PDF or Markdown documents
• An existing dataset (CSV, JSONL, or any registered dataset)
• Recorded traces from a production system
• A small set of seeds written inline
• No input at all, when everything is generated from configuration alone

Synthesizers are the processing steps of the pipeline. Each synthesizer receives one sample at a time from the previous step and can generate new content, expand it into many output samples, or drop and reshape columns. Some examples of what a synthesizer can be:

• A language model call - to generate text, answer questions, label samples, or extract information
• A built-in processor - such as question answering or claim extraction
• Custom Python logic - for any transformation or expansion the built-in synthesizers cannot express

Generators are versioned and reusable - multiple datasets can share the same generator, and any of them can be regenerated at any time.

Do You Need a Generator?

Generating data costs model calls and needs review, so use it only when the data cannot come from somewhere else.

Your situation	What to do
The data already exists as a file, a URL, a folder of documents, or a HuggingFace dataset.	Use a dataset source instead - see Integrate a Dataset.
Your system already had the conversations you want to evaluate on.	Import them from your observability platform - see Import Traces.
You need data for your own domain, your own policies, or a scenario that has no benchmark.	A generator is the right tool - continue with this guide.
You need to cover a grid of dimensions, such as every persona x every scenario.	Write a generator - use the dataset_sample_combinations data source (covered in Providing Input Data: Data Sources).
You have a seed dataset but it is missing a column, such as an answer, a label, or extracted claims.	Write a generator - use the dataset_samples data source (covered in Providing Input Data: Data Sources).

Example: Documents to Question-Answer Pairs

Say you have a folder of Markdown documentation files and want to generate a question-and-answer dataset from them. The documents are registered as a dataset first, then a generator reads them and produces question-answer pairs.

Step 1: Register the input dataset - Each Markdown file becomes one row with a content column:

datasets/product_docs.yaml

key: "product-docs"
display_name: "Product Documentation"
source:
  type: "folder"
  directory_path: "./documents"
  extension: "md"

lf add dataset -f datasets/product_docs.yaml

Step 2: Define the generator - The question_answering synthesizer reads the content column of each document and produces a question and answer:

dataset_generators/docs_qa.yaml

key: "docs-qa-generator"
display_name: "Docs QA Generator"
config_spec:
  - type: "model"
    key: "synthesizer_model"
    display_name: "Synthesizer Model"
    default_value: "openai$gpt-4-1-nano"
definition:
  dataset:
    type: "dataset_samples"
    key: "product-docs"
  synthesizers:
    - type: "question_answering"
      model_key: "<< config.synthesizer_model >>"
      qa_type: "open_ended"
      content_column: "content"

lf add model -p openai/gpt-4.1-nano
lf add dataset-generator -f 'dataset_generators/docs_qa.yaml'

Step 3: Generate the dataset - Declare a dataset that uses the generator. lf test dataset previews the samples without persisting anything:

datasets/docs_qa.yaml

key: "docs-qa"
display_name: "Docs QA"
generator_specification:
  dataset_generator_key: "docs-qa-generator"
  num_samples: 50

lf test dataset -f 'datasets/docs_qa.yaml' -n 3

Generated Samples

| file_name    | content                                              | question                          | answer                            |
| :----------- | :--------------------------------------------------- | :-------------------------------- | :-------------------------------- |
| refunds.md   | Refunds are issued within 5 business days...         | When are refunds issued?          | Within 5 business days.           |
| billing.md   | Invoices are generated on the first of each month... | When are invoices generated?      | On the first of each month.       |

Once the samples look right, lf add dataset -f 'datasets/docs_qa.yaml' generates and stores them. The sections below explain each piece in detail.

Providing Input Data: Data Sources

The data source is where the pipeline starts. It determines what each source sample contains before the synthesizers run. Synthesizers - covered in the next section - can call models, but data sources only supply the starting rows; they do not call models or access external services themselves.

Documents: Markdown and Text Files

A folder of Markdown or plain text files is the most common starting point for document-based dataset generation. Each file becomes one source sample with three columns: file_name, sample_id, and content. If a Markdown file has a YAML front matter block, its keys become additional columns.

datasets/product_docs.yaml

key: "product-docs"
display_name: "Product Documentation"
source:
  type: "folder"
  directory_path: "./documents"
  extension: "md"   # omit to include all file types

lf add dataset -f datasets/product_docs.yaml

The generator then uses dataset_samples to reference this dataset by key. The question_answering and llm synthesizers both work well with a content column as input.

Documents: PDFs

PDFs cannot be read directly - the folder source skips them because their binary content cannot be treated as text. There are two options:

Option A - Convert to text. Export the PDFs to Markdown or plain text using a tool such as pymupdf4llm or marker, then use the folder source as described above. This works with all synthesizers but loses PDF formatting.

Option B - Reference PDFs as model file inputs. Some model providers (including OpenAI) support uploading files directly to their API and referencing them by ID in model calls. The model then reads the PDF natively, preserving tables, figures, and complex formatting that text conversion would lose. This requires uploading the PDFs to the provider’s API first and storing the resulting file IDs in a dataset column.

Step 1 - Upload the PDFs to OpenAI and write the file IDs to a JSONL file:

upload_pdfs.py

import json
from pathlib import Path
from openai import OpenAI

client = OpenAI()
records = []

for pdf_path in Path("./documents").glob("*.pdf"):
    with open(pdf_path, "rb") as f:
        response = client.files.create(file=f, purpose="assistants")
    records.append({"file_name": pdf_path.name, "file_ids": [response.id]})

with open("pdf_files.jsonl", "w") as f:
    for record in records:
        f.write(json.dumps(record) + "\n")

Step 2 - Register the JSONL as a dataset:

datasets/pdf_files.yaml

key: "pdf-files"
display_name: "PDF Files"
source:
  type: "local"
  file_path: "./pdf_files.jsonl"

lf add dataset -f datasets/pdf_files.yaml

Step 3 - Use file_ids_field in the llm synthesizer to pass each PDF to the model:

dataset_generators/pdf_qa.yaml

definition:
  dataset:
    type: "dataset_samples"
    key: "pdf-files"
  synthesizers:
    - type: "llm"
      model_key: "<< config.synthesizer_model >>"
      system_prompt: "Generate a question and its answer from the provided document."
      file_ids_field: "file_ids"
      sample_properties:
        question:
          type: string
        answer:
          type: string

Tabular Data: CSV and JSONL

Use this when you already have structured data - a spreadsheet, a database export, or an existing benchmark - and want to add a generated column, such as a reference answer or extracted claims.

datasets/questions.yaml

key: "questions"
display_name: "Questions"
source:
  type: "local"
  file_path: "./questions.csv"

Use JSONL when a column holds structured data (nested objects or lists) - CSV would flatten them into strings.

Traces

Traces from LangSmith, Phoenix, Claude Code, or Inspect AI can be imported as a dataset and used as input to a generator. Each trace becomes one source sample with a trace column containing the full conversation.

datasets/production_traces.yaml

key: "production-traces"
display_name: "Production Traces"
source:
  type: "langsmith"
  project: $LANGSMITH_PROJECT
  api_key: $LANGSMITH_API_KEY
  limit: 200

See Import Traces for the full connector options.

A Small Fixed Set of Samples

When you have a handful of known inputs - a few topics, personas, or scenarios - you can write them directly into the generator spec using inline_samples. No separate dataset file is needed:

definition:
  dataset:
    type: "inline_samples"
    samples:
      - topic: "Photosynthesis"
      - topic: "Black holes"
      - topic: "The water cycle"

Combination of Datasets

When you want to generate samples that cover every combination of two or more dimensions - for example every persona paired with every scenario - use dataset_sample_combinations. It takes the Cartesian product of several datasets and emits one source sample per combination, with columns merged from all datasets:

definition:
  dataset:
    type: "dataset_sample_combinations"
    keys:
      - "personas"
      - "scenarios"

A generator over a personas dataset with 5 rows and a scenarios dataset with 10 rows produces 50 source samples, one per persona-scenario pair.

No Input - Fully Generated

When the synthesizer generates everything from a prompt and configuration alone, use empty. It emits a fixed number of blank source samples that the synthesizer fills:

definition:
  dataset:
    type: "empty"
    num_samples: 50

Reference

The four dataset types are:

Dataset	type	Starts from
Empty	empty	Nothing. Emits num_samples empty source samples (default 1).
Dataset samples	dataset_samples	The samples of an existing dataset, selected by key. Set random_seed for a reproducible shuffle.
Inline samples	inline_samples	A literal list of samples written into the spec under samples.
Dataset sample combinations	dataset_sample_combinations	The Cartesian product of several datasets, listed in keys. Use it to combine dimensions, e.g. every persona x every scenario.

See the Datasets reference for the full configuration of each.

Important

Make sure the source can supply enough samples. Generation pulls source samples one at a time until the requested number of output samples exists. If the source is exhausted first, you get fewer samples than you asked for and a warning - not an error.

A dataset asking for num_samples: 100 needs either a source that yields at least 100 samples (empty with num_samples: 100, or a seed dataset with 100 rows), or a synthesizer that produces several output samples per source sample. At most twice the requested count of source samples is consumed, which is the headroom for synthesizers that occasionally return nothing.

Note

A data source is not the same as a dataset source. A dataset source populates a standalone dataset; a data source seeds the generation pipeline inside a single dataset generator.

Transforming Samples: Synthesizers

A synthesizer takes a source sample and produces one or more output samples. definition.synthesizers is a list, and the synthesizers run in order - each one receives the full set of samples produced by the previous step, with all columns accumulated so far. A later synthesizer can read any column a previous one wrote, add its own, and pass the result on. drop_columns is typically placed last to remove scratch columns before the dataset is stored.

Synthesizer	type	Produces
LLM	llm	New columns written by a chat-completion model from a prompt template.
Template	template	Samples rendered from a Jinja template with a grid of placeholder values.
Python	python	Whatever a custom synthesize function returns.
Question Answering	question_answering	Multiple-choice or open-ended question-answer pairs from source text.
Drop Columns	drop_columns	The same samples, minus the named columns.
Empty	empty	No output samples. Used when the data source alone defines the dataset.
Claim Extraction	claim_extraction	Atomic claims extracted from an answer.
Claim Labeling	claim_labeling	Each claim labelled as required or optional.
Link Claim Quotes	link_claim_quotes	Each claim linked to the verbatim quotes supporting it.

The last three are the building blocks for RAG claim graphs and are normally chained in that order.

Warning

The Python synthesizer is stateless across samples. The synthesize function is called once per source sample. Each call is independent - do not use global variables or any other state that accumulates across calls. State set during one invocation is not carried over to the next. Pass everything a call needs through its source argument or via generator_config.

Parameterize a Generator

config_spec declares the parameters a generator accepts. Each dataset then supplies values for them in dataset_generator_config, letting one generator serve many datasets. Parameters are referenced as << config.key >> in the definition - resolved before the generator runs - while { sample.field } is resolved per sample at runtime:

dataset_generators/questions.yaml

key: "question-generator"
config_spec:
  - key: "topic"
    type: "string"
    display_name: "Topic"
  - type: "model"
    key: "synthesizer_model"
    display_name: "Synthesizer Model"
    default_value: "openai$gpt-4-1-nano"
definition:
  dataset:
    type: "empty"
    num_samples: 10
  synthesizers:
    - type: "llm"
      model_key: "<< config.synthesizer_model >>"
      user_prompt_template: "Produce a question about << config.topic >> and its answer."
      sample_properties:
        question:
          type: string
        answer:
          type: string

datasets/finance_questions.yaml

key: "finance-questions"
display_name: "Finance Questions"
generator_specification:
  dataset_generator_key: "question-generator"
  num_samples: 10
  dataset_generator_config:
    topic: "finance"

Credentials follow the same pattern - declare them under secrets and reference them as << secrets.NAME >> - see Manage secrets.

Manage Generators and Datasets

Preview Before Generating

Generation costs model calls, so preview it first. Unlike a dataset backed by a source, a generated dataset can be sampled without being persisted:

lf test dataset -f dataset.yaml -n 5              # generate 5 samples, print them
lf test dataset -f dataset.yaml -n 5 -v           # also show each source sample and each synthesizer's input/output
lf test dataset -f dataset.yaml -n 5 -v --show-io # also show the model I/O of the synthesizers
lf test dataset -f run.yaml -k my-dataset         # pick one dataset out of a run config

-n accepts at most 10. --show-io is only meaningful together with -v, and is the fastest way to find out why a prompt is producing the wrong shape.

Register and Generate

If no generator with that key exists it is created; if one exists it is updated when the definition has changed. Adding the dataset is what actually runs the pipeline and stores the samples:

lf add dataset-generator -f 'dataset_generators/qa.yaml'  # A single generator.
lf add dataset-generator -f 'dataset_generators/*.yaml'   # Every generator in a directory.
lf add dataset -f 'datasets/qa.yaml'

Every property is documented in the Dataset Generator reference, and the remaining commands (lf list, lf export, lf delete) in the CLI reference.

Regenerate a Dataset

Regeneration is automatic: lf add and lf run re-run the generator whenever they detect that its configuration changed or that the dataset was never generated in its current form.

Force it with lf regenerate dataset when nothing in the config changed but you want fresh data anyway - because the samples were unsatisfactory, or because you want a different draw. The dataset must already exist; this command never creates one.

lf regenerate dataset --file dataset.yaml               # a standalone dataset spec
lf regenerate dataset --file run.yaml --key my-dataset  # a dataset inside a run config, --key is required

Next Steps

Dataset Generator Reference

Every property of a dataset generator.

Datasets Reference

All the ways a pipeline can start.

Synthesizer Reference

All available synthesizers and their configuration.

Integrate a Dataset

Use data you already have instead.