Action Rules

An action rule decides what happens to a sample after it has been scored. Rules live under definition > actions, and each one pairs a filter - which samples it applies to - with an action to take on them.

Today there is exactly one action, exclude_from_metrics, so in practice action rules are the mechanism for keeping untrustworthy rows out of your numbers without hiding them. The sample is still evaluated, still scored, and still visible in the evidence; it just does not contribute to the aggregation.

Why Exclude Rather Than Filter

You could instead fix the dataset, or filter it before the task runs. Prefer an action rule when the reason to drop a row is only discoverable after the model has responded:

  • The model answered in the wrong language, so the correctness judge cannot be trusted.
  • The retrieved context was empty, so a faithfulness score is meaningless.
  • The model refused, and a refusal is neither a right nor a wrong answer for this metric.

In each case the row is legitimate input - the problem is that scoring it would corrupt the metric. Excluding it keeps the metric honest and leaves an auditable record of what was dropped and why.

Anatomy

tasks/qa/task.yaml
definition:
  scorers:
    - key: language_labeler
      purpose: qa
      type: labeler_via_model
      model_key: "<< config.labeler_model >>"
      valid_labels: ["english", "non-english"]
      user_prompt: |
        Is the following text English? Answer `english` or `non-english` and nothing else.

        <text>{{ trace.get_last_assistant_text() }}</text>
  actions:
    - key: exclude_non_english_answers
      action: exclude_from_metrics
      filter:
        op: equals
        expression: "{{ scores.language_labeler.label }}"
        value: "non-english"
  • key identifies the rule in the results. Required, 1–250 characters, matching ^[a-zA-Z0-9_\-]+$.
  • action is exclude_from_metrics.
  • filter selects the samples. Same filter grammar used elsewhere in LF AI Platform.

A task may define several rules; they are evaluated independently and a sample matching any of them is excluded.

Filters

A filter has an op, an expression (a Jinja template) and - depending on the operator - a comparison value. The expression is rendered against the sample’s evidence and the result is compared.

What the Expression Can See

Variable Holds
sample The dataset row, e.g. { sample.category }.
solver_output The solver’s output, when the task has a solver.
scores Scores keyed by scorer key, then by score field: {{ scores.<scorer_key>.<field> }}.
Warning

trace is not in scope in an action filter, even though scorer prompts can use it. Go through solver_output instead, or - better - have a qa scorer compute the condition and filter on its score. Filters are string expressions with no error reporting worth speaking of; a scorer gives you a value you can inspect with lf test task.

The expression resolves to a typed Python value, not a string, which is what makes numeric and boolean comparisons work.

Operators

Comparison - equals, not_equals, greater_than, less_than, greater_or_equal, less_or_equal. Takes a single value.

filter:
  op: less_than
  expression: "{{ scores.retrieval_quality.num_documents }}"
  value: 1

The ordering operators require both sides to be numeric or boolean; comparing against a string fails with Value '…' is a string, but a non-string value is expected for operator 'greater_than'.

Membership - in, not_in. Takes a list under values.

filter:
  op: in
  expression: "{{ scores.refusal_labeler.label }}"
  values: ["refused", "deflected", "off_topic"]

Unary - exists, not_exists, is_true, is_false. No value at all.

filter:
  op: is_true
  expression: "{{ scores.quality_check.is_malformed }}"

is_true and is_false require the expression to resolve to a genuine boolean - a string "true" raises The expression resolved to a non-boolean value. exists and not_exists test whether the expression resolves at all, which is how you handle score fields that a scorer only emits sometimes.

Exclusion Is Task-Wide

Important

An excluded sample is excluded from every metric in the task, not only from the metrics of the scorer the filter referenced.

So a rule that drops non-English answers to protect a correctness metric also removes those rows from your tone metric, your latency metric and everything else. That is usually what you want - the row is untrustworthy as a whole - but it means one over-broad rule can quietly shrink every number in the task. Check how many rows survived before reading the results.

Samples that errored are already left out of metric aggregation regardless of any action rule; action rules apply to samples that completed and were scored.

The QA Pattern

The idiomatic use is a pair: a scorer marked purpose: qa that measures data or output quality, and an action rule that acts on it.

definition:
  solver:
    type: single_turn_solver
    input_builder:
      type: chat_completion
      input_messages:
        - role: user
          content: "{{ sample.question }}"
  scorers:
    - key: correctness
      type: model_as_a_judge_classifier
      model_key: "<< config.judge_model >>"
      correct_labels: ["correct"]
      incorrect_labels: ["incorrect"]
      user_prompt: |
        Expected: {{ sample.expected_answer }}
        Actual: {{ trace.get_last_assistant_text() }}
        Answer `correct` or `incorrect`.

    - key: answered_at_all
      purpose: qa
      type: python
      compute_scores_snippet: |
        def compute_scores(sample, solver_output):
            text = solver_output.trace.get_last_assistant_text() or ""
            return {"is_empty": len(text.strip()) == 0}
      metrics:
        - type: frequency
          field: is_empty

  actions:
    - key: exclude_empty_answers
      action: exclude_from_metrics
      filter:
        op: is_true
        expression: "{{ scores.answered_at_all.is_empty }}"

purpose: qa is a label, not behaviour - it marks the scorer as measuring trustworthiness rather than model quality, so the results distinguish the two. Give the QA scorer its own metric (a frequency works well) so you can see how many rows the rule is removing.