HuggingFace

Loads a dataset from the HuggingFace dataset hub. Use it to evaluate against public benchmarks and community datasets.

Requirements

The huggingface source requires the datasets library (uv pip install datasets). Use load_dataset_kwargs to pass any additional arguments accepted by datasets.load_dataset().

Examples

Example: HuggingFace benchmark. Loads a split from a hub dataset and filters it down to the relevant rows.

HuggingFace Dataset Source

# ...
source:
  type: "huggingface"
  path: "allenai/tulu-3-harmbench-eval"
  split: "test"
  filters:
    - op: "equals"
      expression: "{{ sample.SemanticCategory }}"
      value: "illegal"

Configuration

Properties

type Literal “huggingface” required

HuggingFace dataset source.

path string required

HuggingFace dataset path (e.g., ‘squad’, ‘glue’).

split string required

Dataset split to load (e.g., ‘train’, ‘test’, ‘validation’).

load_dataset_kwargs object

Additional keyword arguments to pass to datasets.load_dataset().

filters array[FilterComparison, FilterMembership, FilterUnary]

Optional list of filters to apply after loading (AND-combined).

Related types

`FilterComparison`

Filter by Column Comparing to Value

# ...
source:
  # ...
  filters:
    - op: "equals"
      expression: "{{ sample.SemanticCategory }}"
      value: "illegal"

Properties

op enum FilterComparisonOp required

Possible FilterComparisonOp values

The comparison operator to apply.

Allowed Values:

equals
not_equals
greater_than
less_than
greater_or_equal
less_or_equal

expression string required

An expression encoding what to compare against the value.

Depending on the context, it can refer to different variables:

When filtering a dataset: it can refer to the sample and use dot or bracket notation to access the columns. If filtering a dataset with column names that are illegal under jinja substitution rules (e.g. containing spaces), use bracket notation to access the column.
When used within a task action: it can refer to the sample, the solver_output or the scores (which is a mapping between scorer keys and their corresponding score values dict).

value string, number, integer, boolean required

The value against which the expression is compared.

`FilterMembership`

Filter by Column Value Membership

# ...
source:
  # ...
  filters:
    - op: "not_in"
      expression: "{{ sample.SemanticCategory }}"
      values:
        - "copyright"
        - "chemical_biological"

Properties

op enum FilterMembershipOp required

Possible FilterMembershipOp values

The membership operator to apply.

Allowed Values:

in
not_in

expression string required

An expression encoding what to check membership against the values.

Depending on the context, it can refer to different variables:

When filtering a dataset: it can refer to the sample and use dot or bracket notation to access the columns. If filtering a dataset with column names that are illegal under jinja substitution rules (e.g. containing spaces), use bracket notation to access the column.
When used within a task action: it can refer to the sample, the solver_output or the scores (which is a mapping between scorer keys and their corresponding score values dict).

values array[string, number, boolean] required

The set of values to test membership against.

`FilterUnary`

Filter by Unary Expression

# ...
source:
  # ...
  filters:
    - op: "is_true"
      expression: "{{ sample.answer }}"

Properties

op enum FilterUnaryOp required

Possible FilterUnaryOp values

The unary operator to apply.

Allowed Values:

exists
not_exists
is_true
is_false

expression string required

An expression encoding what to apply the unary operator to.

Depending on the context, it can refer to different variables:

When filtering a dataset: it can refer to the sample and use dot or bracket notation to access the columns. If filtering a dataset with column names that are illegal under jinja substitution rules (e.g. containing spaces), use bracket notation to access the column.
When used within a task action: it can refer to the sample, the solver_output or the scores (which is a mapping between scorer keys and their corresponding score values dict).