Datasets

A dataset is a collection of data used to train, validate, and/or test a model. In AI GO!, datasets are used as a source of samples for AI model evaluation.

To interact with a dataset using the CLI, use the lf dataset command.

Dataset Overview

Properties

key string required

Unique identifier assigned to the entity in AI GO!.

Pattern: ^[a-zA-Z0-9_\-]+$
Max Length: 250

display_name string required

The dataset’s name displayed to the user.

description string

Short description of the dataset.

Default: None

long_description string

Long description of the dataset. Supports Markdown formatting.

Default: None

source LocalDatasetSource, FolderDatasetSource, URLDatasetSource, HuggingFaceDatasetSource, LangSmithDatasetSource, PhoenixDatasetSource, ClaudeCodeDatasetSource, InspectAIDatasetSource

Dataset source configuration. Required if dataset generator is not used.

Default: None

generator_specification SDKDatasetGeneratorSpecification

Config for the dataset generator that will be used to generate the dataset. Required if the data file is not provided.

Default: None

tags array[string]

Tags associated with the dataset.

Default: []

display_name: "Airline User Authentication"
key: "airline-user-authentication"
description: "Dataset of valid and invalid passenger email and booking ID pairs."
source:
  type: "local"
  file_path: "./test_cases.csv"

Email Address,Booking Reference,Complete (True / False),Frequent Flyer Number,Seat Number,Departure Date
alexander.hill@example.com,BKGIKAS4,False,LX872246,16B,2025-07-02
ronald.montgomery@example.com,BKGHBZK0,True,LX329258,3A,2025-07-29
austin.gentry@example.com,BKGMTOQ6,True,LX781453,36B,2025-07-04
christina.ryan@example.com,BKGSUOO0,False,LX543143,1B,2025-08-02
angelica.tucker@example.com,BKGHCBK0,True,LX498382,22A,2025-07-04
david.caldwell@example.com,BKGOHCO4,True,LX865179,39C,2025-07-15
gregory.carter@example.com,BKGFBQC8,False,LX969693,6E,2025-08-15
jeremy.wilson@example.com,BKGIELC8,False,LX910620,3F,2025-07-18
jaclyn.moore@example.com,BKGRADC6,True,LX575435,7D,2025-08-20
melissa.hernandez@example.com,BKGPHGO0,True,LX835911,14F,2025-08-04

Folder Dataset

display_name: "Airline Policy Documents"
key: "airline-policy-documents"
description: "Dataset built from the airline's Markdown policy documents."
source:
  type: "folder"
  directory_path: "../documents"
  extension: "md"

Remote Dataset

display_name: "Judge Comparison"
key: "jailbreakbench-judge-comparison"
description: >
  Jailbreakbench is an open-source robustness benchmark for jailbreaking large language
  models (LLMs).
source:
  type: "url"
  url: "https://huggingface.co/datasets/JailbreakBench/JBB-Behaviors/raw/main/data/judge-comparison.csv"

HuggingFace Dataset

key: "harmbench_illegal"
display_name: "HarmBench: Illegal Activities"
description: >-
  HarmBench is a standardized evaluation framework for automated red teaming methods
  targeting large language models (LLMs).
source:
  type: "huggingface"
  path: "allenai/tulu-3-harmbench-eval"
  split: "test"
  filters:
    - op: "equals"
      expression: "{{ sample.SemanticCategory }}"
      value: "illegal"

display_name: "Legal Questions"
key: "legal-questions"
description: "Generated using 'My Question Generator'"
generator_specification:
  dataset_generator_key: "question-generator"
  num_samples: 5
  dataset_generator_config:
    topic: "Legal"

display_name: "My Question Generator"
key: "question-generator"
description: "Generates questions about arbitrary topics."
config_spec:
  - key: "topic"
    type: "string"
    display_name: "Topic"
definition:
  type: "declarative_dataset_generator"
  data_source:
    type: "empty"
  synthesizers:
    - type: "llm"
      model_key: "openai$gpt-4-1-nano"
      use_structured_outputs: true
      system_prompt: "You are a helpful dataset generator."
      user_prompt_template: >
        Produce 5 deep knowledge questions about << config.topic >>, including the
        correct answer.
      sample_properties:
        question:
          type: "string"
        answer:
          type: "string"

Definitions

`LocalDatasetSource`

Dataset from a local file path.

Properties

type Literal “local” required

Local file dataset source.

file_path string required

Path to a local CSV or JSONL file.

filters array[FilterComparison, FilterMembership, FilterUnary]

Optional list of filters to apply after loading (AND-combined).

Local Dataset Source

# ...
source:
  type: "local"
  file_path: "./test_cases.csv"

`FolderDatasetSource`

Dataset built from all files in a local folder (recursively).

Properties

type Literal “folder” required

Local folder dataset source.

directory_path string required

Path to a local directory to scan recursively for files.

extension string

Optional file extension (without leading dot, e.g. ‘md’) to filter by. When omitted, all files are included.

Default: None

filters array[FilterComparison, FilterMembership, FilterUnary]

Optional list of filters to apply after loading (AND-combined). Filters can reference the sample_id, file_name and content columns as well as any front matter metadata columns.

Folder Dataset Source

# ...
source:
  type: "folder"
  directory_path: "../documents"
  extension: "md"

Note

The folder is scanned recursively. Each file becomes a row with a sample_id (content hash), a file_name (path relative to directory_path, keeping nested subdirectories) and its content. Set the optional extension to only include files with that extension. PDF files are always skipped since their binary content cannot be loaded as text.

Files with a valid YAML front matter header (a --- delimited block at the top, as commonly used in Markdown) contribute one extra column per metadata key, and their content excludes the header. If a header is present but not valid YAML, a warning is logged and the file is still included with its original content but without metadata columns.

Filters are AND-combined and applied after loading, and may reference the sample_id, file_name and content columns as well as any front matter metadata columns.

`URLDatasetSource`

Dataset from a remote URL.

Properties

type Literal “url” required

Remote URL dataset source.

url string required

URL pointing to a CSV or JSONL file. Must end with .csv or .jsonl extension.

filters array[FilterComparison, FilterMembership, FilterUnary]

Optional list of filters to apply after loading (AND-combined).

URL Dataset Source

# ...
source:
  type: "url"
  url: "https://huggingface.co/datasets/JailbreakBench/JBB-Behaviors/raw/main/data/judge-comparison.csv"

Note

The URL must point directly to a .csv or .jsonl file. Filters are AND-combined and applied after the file is downloaded.

`HuggingFaceDatasetSource`

Dataset from HuggingFace dataset hub.

Properties

type Literal “huggingface” required

HuggingFace dataset source.

path string required

HuggingFace dataset path (e.g., ‘squad’, ‘glue’).

split string required

Dataset split to load (e.g., ‘train’, ‘test’, ‘validation’).

load_dataset_kwargs object

Additional keyword arguments to pass to datasets.load_dataset().

filters array[FilterComparison, FilterMembership, FilterUnary]

Optional list of filters to apply after loading (AND-combined).

HuggingFace Dataset

# ...
source:
  type: "huggingface"
  path: "allenai/tulu-3-harmbench-eval"
  split: "test"
  filters:
    - op: "equals"
      expression: "{{ sample.SemanticCategory }}"
      value: "illegal"

Note

The huggingface source requires the datasets library (uv pip install datasets). Use load_dataset_kwargs to pass any additional arguments accepted by datasets.load_dataset().

`ClaudeCodeDatasetSource`

Dataset source from Claude Code local session files.

Properties

type Literal “claude_code” required

Claude Code trace source.

path string

Path to Claude Code session file or project directory. Defaults to ~/.claude/projects/.

Default: None

session_id string

Specific session ID to import.

Default: None

from_time string

Start time filter for sessions.

Default: None

to_time string

End time filter for sessions.

Default: None

limit integer

Maximum number of sessions to import.

Default: None

`LangSmithDatasetSource`

Dataset source from LangSmith observability traces.

Properties

type Literal “langsmith” required

LangSmith trace source.

project string

LangSmith project name to import traces from.

Default: None

dataset string

LangSmith dataset name to import examples from.

Default: None

api_key string

LangSmith API key. Falls back to LANGSMITH_API_KEY env var.

Default: None

api_url string

LangSmith API URL. Falls back to LANGSMITH_ENDPOINT env var.

Default: None

from_time string

Start time filter for traces.

Default: None

to_time string

End time filter for traces.

Default: None

tags array[string]

Filter traces by tags.

Default: None

filter string

LangSmith filter string for traces.

Default: None

limit integer

Maximum number of traces to import.

Default: None

`PhoenixDatasetSource`

Dataset source from Arize Phoenix observability traces.

Properties

type Literal “phoenix” required

Phoenix trace source.

project string required

Phoenix project name.

from_time string

Start time filter for traces.

Default: None

to_time string

End time filter for traces.

Default: None

trace_id string, array[string]

Specific trace ID(s) to import.

Default: None

session_id string, array[string]

Filter traces by session ID(s).

Default: None

tags array[string]

Filter traces by tags.

Default: None

metadata object

Filter traces by metadata key-value pairs on the root span (all must match).

Default: None

limit integer

Maximum number of traces to import.

Default: None

api_key string

Phoenix API key. Falls back to PHOENIX_API_KEY env var.

Default: None

base_url string

Phoenix base URL. Falls back to PHOENIX_COLLECTOR_ENDPOINT env var.

Default: None

`InspectAIDatasetSource`

Dataset source from Inspect AI eval log files.

Properties

type Literal “inspect_ai” required

Inspect AI eval log source.

path string required

Path to a directory containing .eval log files, a single .eval file, or a transcript database directory.

limit integer

Maximum number of transcripts to import.

Default: None

`SDKDatasetGeneratorSpecification`

Properties

dataset_generator_config object required

The configuration used by the dataset generator.

num_samples integer required

The number of samples to generate. At least 1 sample must be requested.

debug DatasetGenerationDebugOptions

Default: None

dataset_generator_key string required

Key of the dataset generator to use.

Dataset Generator Configuration

# ...
generator_specification:
  dataset_generator_key: "question-generator"
  num_samples: 5
  dataset_generator_config:
    topic: "Legal"

Note

The dataset_generator_config allows instantiating the same dataset generator template in different ways. Always refer to the dataset generator definition for the available configuration options.

`FilterComparison`

Properties

op enum FilterComparisonOp required

Possible FilterComparisonOp values

The comparison operator to apply.

Allowed Values:

equals
not_equals
greater_than
less_than
greater_or_equal
less_or_equal

expression string required

An expression encoding what to compare against the value.

Depending on the context, it can refer to different variables:

When filtering a dataset: it can refer to the sample and use dot or bracket notation to access the columns. If filtering a dataset with column names that are illegal under jinja substitution rules (e.g. containing spaces), use bracket notation to access the column.
When used within a task action: it can refer to the sample, the solver_output or the scores (which is a mapping between scorer keys and their corresponding score values dict).

value string, number, integer, boolean required

The value against which the expression is compared.

Filter by Column Comparing to Value

# ...
source:
  # ...
  filters:
    - op: "equals"
      expression: "{{ sample.SemanticCategory }}"
      value: "illegal"

`FilterMembership`

Properties

op enum FilterMembershipOp required

Possible FilterMembershipOp values

The membership operator to apply.

Allowed Values:

in
not_in

expression string required

An expression encoding what to check membership against the values.

Depending on the context, it can refer to different variables:

When filtering a dataset: it can refer to the sample and use dot or bracket notation to access the columns. If filtering a dataset with column names that are illegal under jinja substitution rules (e.g. containing spaces), use bracket notation to access the column.
When used within a task action: it can refer to the sample, the solver_output or the scores (which is a mapping between scorer keys and their corresponding score values dict).

values array[string, number, boolean] required

The set of values to test membership against.

Filter by Column Value Membership

# ...
source:
  # ...
  filters:
    - op: "not_in"
      expression: "{{ sample.SemanticCategory }}"
      values:
        - "copyright"
        - "chemical_biological"

`FilterUnary`

Properties

op enum FilterUnaryOp required

Possible FilterUnaryOp values

The unary operator to apply.

Allowed Values:

exists
not_exists
is_true
is_false

expression string required

An expression encoding what to apply the unary operator to.

Depending on the context, it can refer to different variables:

When filtering a dataset: it can refer to the sample and use dot or bracket notation to access the columns. If filtering a dataset with column names that are illegal under jinja substitution rules (e.g. containing spaces), use bracket notation to access the column.
When used within a task action: it can refer to the sample, the solver_output or the scores (which is a mapping between scorer keys and their corresponding score values dict).

Filter by Unary Expression

# ...
source:
  # ...
  filters:
    - op: "is_true"
      expression: "{{ sample.answer }}"

`DatasetGenerationDebugOptions`

Properties

enabled boolean required

When true, the response will include a full pipeline trace for each source sample, which contains the source sample itself and the input and output at each synthesizer stage.

include_io boolean required

When true, the model input and output are included in the trace for each synthesizer call that produced I/O. Has no effect when enabled is false.