Metrics

A metric aggregates the per-sample values a scorer emitted into the numbers that end up in the evaluation results. Metrics are nested under the scorer they aggregate:

definition:
  scorers:
    - type: string_equals
      ground_truth: "{{ sample.expected_answer }}"
      metrics:
        - type: mean
          field: is_correct
          name: "Accuracy"

This page is about choosing an aggregation. Every field of every metric type is documented in the Metrics reference.

Choosing a Metric

Choose by the shape of the score value, not by the name of the number you want.

Metric	The score field holds…
Mean	A boolean or 0/1 correctness flag - the mean is the rate - or a continuous score.
Std Dev, Min, Max	A continuous score (judge score, BLEU, latency) whose spread matters, alongside its mean.
Frequency	A category or label. Emits the distribution over labels.
Weighted Average	A category whose values have different importance.
Precision, Recall, F1 Score	Per-sample TP / FP / FN counts.
Binary Classification	A ground-truth and a predicted label, two classes. Emits the whole report at once.
Multiclass Classification	A ground-truth and a predicted label, many classes. Emits the whole report, per class.
Python	Something none of the above expresses.

Two families need highlighting because they are easy to mix up:

Counting metrics (precision, recall, f1_score) do not take a field. They take count fields - num_true_positives_field, num_false_positives_field, num_false_negatives_field - and sum them across samples before dividing. Use them when each sample contributes several decisions, for example an extraction task where one sample yields many entities.

metrics:
  - key: entity_precision
    type: precision
    num_true_positives_field: num_correct_entities
    num_false_positives_field: num_spurious_entities

Classification report metrics (binary-classification, multiclass-classification) take a pair of label fields instead, and emit a whole report - accuracy, precision, recall, F1 and the confusion matrix - from one entry.

metrics:
  - key: sentiment_report
    type: binary-classification
    field_gt: expected_label
    field_pred: predicted_label
    positive_answer: "positive"
    negative_answer: "negative"

If each sample is one classification decision, reach for these before hand-rolling precision and recall.

key vs name

Two different identifiers, and the distinction matters:

• key identifies the metric within its scorer. It defaults to the metric’s type and must be unique among that scorer’s metrics.
• name is the human-readable label shown in results. It is optional and purely cosmetic.

metrics:
  - key: accuracy          # identity - must be unique per scorer
    type: mean
    field: is_correct
    name: "Answer Accuracy"  # what a reader sees

ImportantTwo metrics of the same type need explicit keys

Because key defaults to type, two mean metrics under one scorer collide and the task fails with Metrics keys for scorer with key '<scorer>' must be unique.

# Fails - both keys default to `mean`.
metrics:
  - type: mean
    field: precision
  - type: mean
    field: recall

# Works.
metrics:
  - key: precision
    type: mean
    field: precision
  - key: recall
    type: mean
    field: recall

Setting name does not resolve the collision; only key does.

Note that frequency takes no name at all - it emits one value per distinct label, so the labels are the names.

Python Metrics

Use type: python when the aggregation is not a per-field reduction - when it needs to look at several score fields together, or compute something over the whole set at once.

The snippet defines compute_metrics, receives every SampleScore the scorer produced, and returns a flat dictionary of metric names to numbers.

tasks/classification/compute_metrics.py

from __future__ import annotations

from latticeflow.core.dtypes import SampleScore


def compute_metrics(scores: list[SampleScore]) -> dict[str, int | float]:
    if len(scores) == 0:
        raise ValueError("Cannot compute metrics: received 0 sample scores.")

    num_true_positives = 0
    num_false_positives = 0
    num_positives = 0

    for sample_score in scores:
        ground_truth = sample_score.values["gt"]
        prediction = sample_score.values["pred"]
        if ground_truth:
            num_positives += 1
            if prediction:
                num_true_positives += 1
        elif prediction:
            num_false_positives += 1

    predicted_positives = num_true_positives + num_false_positives
    return {
        "precision": num_true_positives / predicted_positives if predicted_positives > 0 else 1.0,
        "recall": num_true_positives / num_positives if num_positives > 0 else 1.0,
    }

definition:
  scorers:
    - key: binary_classification
      type: python
      compute_scores_snippet: !include score_sample.py  # emits `gt` and `pred`
      metrics:
        - type: python
          compute_metrics_snippet: !include compute_metrics.py

Read score values through sample_score.values["<field>"]. Every key the metric returns becomes a metric in the results, so one python metric can emit several numbers - which is also a way to sidestep the key-collision problem.

The snippet runs in the standard server-side runtime; see Python Snippets.