Overview

Scorers evaluate the output produced by a solver (or a dataset sample directly) and assign one or more numeric scores. Each scorer is defined inline in your task YAML file under the scorers key and produces score values that are aggregated into metrics.

# tasks/my_task.yaml
definition:
  scorers:
    - type: string_equals
      ground_truth: "{{ sample.expected_answer }}"

A task can define multiple scorers. Each scorer runs independently and contributes its own named scores.

Available Scorers

Scorer	Type value	Description
String Equality	string_equals	Exact string match against a ground-truth answer.
String Equality (Multiple Choice)	string_equals_mcqa	Exact match on the first character of the output against a multiple-choice ground-truth.
Model As A Judge Classifier	model_as_a_judge_classifier	Uses an LLM to classify the output and checks it against a set of correct labels.
Labeler (via Model)	labeler_via_model	Uses an LLM to attach a free-form label to the sample or solver output.
Model Scorer	model_as_a_judge_scorer	Uses an LLM to produce a numeric score for the output.
Python Scorer	python	Runs a custom Python function to score each sample individually.
Python Batch Scorer	python_all_samples	Runs a custom Python function over all samples at once, enabling cross-sample metrics.
Text Similarity (BLEU Score)	bleu	Computes BLEU n-gram overlap between the output and a ground-truth string.
RAG Checker	rag_checker	Evaluates RAG outputs for faithfulness and relevance using the RAGChecker framework.
Function Call Coverage	function_call_coverage	Checks whether an agent made all required function calls in the correct order.