Overview

A dataset source tells AI GO! where the samples of a dataset come from. It is configured under the source key of a dataset, with the concrete source selected by its type discriminator.

# datasets/my_dataset.yaml
key: "my-dataset"
display_name: "My Dataset"
source:
  type: "local"
  file_path: "test_cases.csv"

A dataset must define either a source or a generator_specification (see Dataset Generators). Sources that load tabular data (local, folder, url, huggingface) additionally support filters, which are AND-combined and applied after loading.

Note

Dataset sources are not the same as data sources. A dataset source populates a standalone dataset; a data source seeds the generation pipeline of a single dataset generator.

Available Dataset Sources

Dataset Source	Type value	Description
Local	local	Loads a local CSV or JSONL file.
Folder	folder	Loads all files in a local folder, recursively.
URL	url	Downloads a remote CSV or JSONL file.
HuggingFace	huggingface	Loads a dataset from the HuggingFace hub.
LangSmith	langsmith	Imports traces or examples from LangSmith.
Phoenix	phoenix	Imports traces from Arize Phoenix.
Claude Code	claude_code	Imports local Claude Code session files.
Inspect AI	inspect_ai	Imports Inspect AI eval log files.