Claim Labeling

Labels each previously extracted claim as REQUIRED or OPTIONAL for answering the question, using a chat-completion model. A claim is REQUIRED when the answer would be incomplete without it, and OPTIONAL when it only adds context or background. Reads the question, answer, and claims from the source columns configured in columns, and writes the labels to output_field. Run it after the Claim Extraction synthesizer.

Output

One output sample per source sample with a list of REQUIRED/OPTIONAL labels written to output_field, one label per claim and in the same order as the claims.

What the Labels Are For

The labels let the RAG Checker scorer distinguish claims that the answer must contain from claims that are merely nice to have. Point its target_labels_column or response_labels_column at the output_field column, and set its mode to required_only (metrics over the required claims only) or required_and_optional (both sets, the required ones under required_-prefixed keys).

Labeling as part of dataset generation is optional: the scorer labels the claims itself when no label column is given. Doing it here makes the labels part of the dataset, so they are computed once, can be reviewed, and stay fixed across evaluations.

Examples

Example: Claim labeling. Labels the claims of the response_claims column, writing the labels to the response_claims_labels column.

Claim Labeling Synthesizer
# ...
config_spec:
  - key: "synthesizer_model_key"
    type: "model"
    display_name: "Synthesizer Model"
    description: "Model used to label extracted claims."
definition:
  # ...
  synthesizers:
    - type: claim_labeling
      model_key: "<< config.synthesizer_model_key >>"
      columns:
        question: question
        answer: response
        claims: response_claims
      output_field: response_claims_labels

The claims are read from the response_claims column, together with the question and the answer they were extracted from:

Source Sample
{
  "question": "What is the capital of France?",
  "response": "The capital of France is Paris. Paris is well known for its Eiffel Tower.",
  "response_claims": [
    {"claim": {"subject": "Paris", "predicate": "is the capital of", "object": "France"}},
    {"claim": {"subject": "Paris", "predicate": "is well known for its", "object": "Eiffel Tower"}}
  ]
}

The output sample keeps the source columns and adds one label per claim: the capital claim answers the question, the Eiffel Tower claim only adds context.

Output Sample
{
  "question": "What is the capital of France?",
  "response": "The capital of France is Paris. Paris is well known for its Eiffel Tower.",
  "response_claims": [
    {"claim": {"subject": "Paris", "predicate": "is the capital of", "object": "France"}},
    {"claim": {"subject": "Paris", "predicate": "is well known for its", "object": "Eiffel Tower"}}
  ],
  "response_claims_labels": ["REQUIRED", "OPTIONAL"]
}

Configuration

Properties

type Literal “claim_labeling” required

The type of the synthesizer.

model_key TemplateValue required

The key of the chat completion model to be used as a synthesizer.

output_field string, TemplateValue

Name of the structured output field written to each generated sample.

Default: claims_labels

instructions string, TemplateValue

Optional domain-specific instructions appended to the built-in system prompt.

Default: None

columns ClaimLabelingColumnBindings

Mapping from built-in inputs to source sample columns.