Folder

Builds a dataset from every file in a local folder, scanned recursively. Each file becomes one sample. Use it to turn a directory of documents into a dataset.

Behaviour

The folder is scanned recursively. Each file becomes a row with a sample_id (content hash), a file_name (path relative to directory_path, keeping nested subdirectories) and its content. Set the optional extension to only include files with that extension. PDF files are always skipped since their binary content cannot be loaded as text.

Files with a valid YAML front matter header (a --- delimited block at the top, as commonly used in Markdown) contribute one extra column per metadata key, and their content excludes the header. If a header is present but not valid YAML, a warning is logged and the file is still included with its original content but without metadata columns.

Filters are AND-combined and applied after loading, and may reference the sample_id, file_name and content columns as well as any front matter metadata columns.

Examples

Example: Folder of files. Loads every file under a directory as a sample.

Folder Dataset Source

# ...
source:
  type: "folder"
  directory_path: "../documents"
  extension: "md"

The scanned folder holds two Markdown documents, each starting with a YAML front matter header that declares a title and a category:

---
title: Baggage Policy
category: baggage
---

# Baggage Policy

Each passenger may check one bag up to 23 kg free of charge. Additional or
overweight bags incur a fee, which is charged per bag at check-in.

---
title: Refund Policy
category: refunds
---

# Refund Policy

Refundable fares can be cancelled for a full refund up to 24 hours before
departure. Non-refundable fares are eligible for travel credit only.

Loading the folder yields one row per file, with title and category added as columns from the front matter headers:

Resulting Dataset

| sample_id   | file_name                | content                                         | title          | category |
| :---------- | :----------------------- | :---------------------------------------------- | :------------- | :------- |
| 3249fbcd... | baggage_policy.md        | "\n# Baggage Policy\n\nEach passenger may ..."  | Baggage Policy | baggage  |
| d93894c0... | refunds/refund_policy.md | "\n# Refund Policy\n\nRefundable fares can ..." | Refund Policy  | refunds  |

Note

sample_id is the SHA-256 hash of the file’s bytes, abbreviated above. A file without a front matter header contributes only the sample_id, file_name and content columns.

Configuration

Properties

type Literal “folder” required

Local folder dataset source.

directory_path string required

Path to a local directory to scan recursively for files.

extension string

Optional file extension (without leading dot, e.g. ‘md’) to filter by. When omitted, all files are included.

Default: None

filters array[FilterComparison, FilterMembership, FilterUnary]

Optional list of filters to apply after loading (AND-combined). Filters can reference the sample_id, file_name and content columns as well as any front matter metadata columns.

Related types

`FilterComparison`

Filter by Column Comparing to Value

# ...
source:
  # ...
  filters:
    - op: "equals"
      expression: "{{ sample.SemanticCategory }}"
      value: "illegal"

Properties

op enum FilterComparisonOp required

Possible FilterComparisonOp values

The comparison operator to apply.

Allowed Values:

equals
not_equals
greater_than
less_than
greater_or_equal
less_or_equal

expression string required

An expression encoding what to compare against the value.

Depending on the context, it can refer to different variables:

When filtering a dataset: it can refer to the sample and use dot or bracket notation to access the columns. If filtering a dataset with column names that are illegal under jinja substitution rules (e.g. containing spaces), use bracket notation to access the column.
When used within a task action: it can refer to the sample, the solver_output or the scores (which is a mapping between scorer keys and their corresponding score values dict).

value string, number, integer, boolean required

The value against which the expression is compared.

`FilterMembership`

Filter by Column Value Membership

# ...
source:
  # ...
  filters:
    - op: "not_in"
      expression: "{{ sample.SemanticCategory }}"
      values:
        - "copyright"
        - "chemical_biological"

Properties

op enum FilterMembershipOp required

Possible FilterMembershipOp values

The membership operator to apply.

Allowed Values:

in
not_in

expression string required

An expression encoding what to check membership against the values.

Depending on the context, it can refer to different variables:

When filtering a dataset: it can refer to the sample and use dot or bracket notation to access the columns. If filtering a dataset with column names that are illegal under jinja substitution rules (e.g. containing spaces), use bracket notation to access the column.
When used within a task action: it can refer to the sample, the solver_output or the scores (which is a mapping between scorer keys and their corresponding score values dict).

values array[string, number, boolean] required

The set of values to test membership against.

`FilterUnary`

Filter by Unary Expression

# ...
source:
  # ...
  filters:
    - op: "is_true"
      expression: "{{ sample.answer }}"

Properties

op enum FilterUnaryOp required

Possible FilterUnaryOp values

The unary operator to apply.

Allowed Values:

exists
not_exists
is_true
is_false

expression string required

An expression encoding what to apply the unary operator to.

Depending on the context, it can refer to different variables:

When filtering a dataset: it can refer to the sample and use dot or bracket notation to access the columns. If filtering a dataset with column names that are illegal under jinja substitution rules (e.g. containing spaces), use bracket notation to access the column.
When used within a task action: it can refer to the sample, the solver_output or the scores (which is a mapping between scorer keys and their corresponding score values dict).