Dataset Generators

A dataset generator produces the samples of a dataset declaratively instead of reading them from a file. It reads seed samples from a data source and passes them through a chain of synthesizers, each of which adds, rewrites or drops columns, until the final samples are reached.

# dataset_generators/unit_conversion_qa_generator.yaml
key: "unit-conversion-qa-generator"
display_name: "Unit Conversion QA Generator"
description: "Turns each seed temperature into two conversion questions."
config_spec:
  - type: "dataset"
    key: "dataset_key"
    display_name: "Dataset"
    default_value: "city-temperatures"
definition:
  data_source:
    type: "dataset_samples"
    dataset_key: "<< config.dataset_key >>"
  synthesizers:
    - type: "python"
      synthesize_snippet: !include "./synthesize.py"

The generation pipeline

definition.data_source decides what the pipeline starts from - nothing at all (empty), the samples of an existing dataset (dataset_samples), samples written inline in the config (inline_samples), or the cross product of several datasets (dataset_sample_combinations). See Data Sources.

definition.synthesizers then lists the transformation steps, applied in order. A synthesizer can prompt a model to write new columns (llm, question_answering), run arbitrary Python (python), render a template (template), remove columns (drop_columns), or pass the source samples through unchanged (empty). See Synthesizers for the available types and the dataset generation guide for worked examples.

Parameterizing a generator

config_spec declares the parameters a generator accepts, using the same config specification as tasks, and the definition refers to their values with << config.key >> - including in data_source, so even the seed dataset can be chosen per use. Each dataset supplies the values in its generator_specification.dataset_generator_config, which lets one generator serve several datasets. Model-backed synthesizers need credentials, which are declared in secrets and referenced as << secrets.NAME >> (see the secrets guide).

Working with dataset generators

Register a generator with lf add dataset-generator and list the registered ones with lf list dataset-generator. A dataset points at a generator through its generator_specification; lf test dataset previews the samples it produces, and lf regenerate dataset generates a fresh set of samples for an existing dataset.

Configuration

Properties

secrets object

Secrets which can be used to reference secret values in designated places.

Default: None

key string required

Unique identifier assigned to the entity in AI Platform.

Pattern: ^[a-zA-Z0-9_\-]+$
Max Length: 250

display_name string required

The dataset generator’s name displayed to the user.

description string required

Short description of the dataset generator.

long_description string

Long description of the dataset generator. Supports Markdown formatting.

Default: None

config_spec array[FloatParameterSpec, IntParameterSpec, BooleanParameterSpec, StringParameterSpec, ModelParameterSpec, DatasetParameterSpec, DatasetColumnParameterSpec, ListParameterSpec, DictParameterSpec, CategoricalParameterSpec]

Configuration specification for the dataset generator.

Default: []

definition SDKDeclarativeDatasetGeneratorDefinitionTemplate required

Declarative dataset generator definition. It must be of type SDKDeclarativeDatasetGeneratorDefinitionTemplate.

Generator Definition
display_name: "My Question Generator"
key: "question-generator"
description: "Generates questions about arbitrary topics."
config_spec:
  - key: "synthesizer_model_key"
    type: "model"
    display_name: "Synthesizer Model"
    description: "Model used to generate questions."
  - key: "topic"
    type: "string"
    display_name: "Topic"
    description: "Topic for which questions will be generated."
definition:
  type: "declarative_dataset_generator"
  data_source:
    type: "empty"
  synthesizers:
    - type: "llm"
      model_key: "<< config.synthesizer_model_key >>"
      use_structured_outputs: true
      system_prompt: "You are a helpful dataset generator."
      user_prompt_template: >
        Produce 5 deep knowledge questions about << config.topic >>, including the
        correct answer.
      sample_properties:
        question:
          type: "string"
        answer:
          type: "string"