Question Answering

Generates question-answering samples from source text using a chat-completion model. Supports multiple_choice and open_ended question types via qa_type, reading the source text (and optional title/summary) from the configured columns. Use it for building QA datasets from documents. For fully custom prompting, use the LLM synthesizer.

Output

One question-answering sample per source document, holding the question, its answer (answer or choices plus answer_index), the question type, the estimated difficulty, the supporting citations and the source document.

Examples

Example: Multiple-choice QA from documents. Generates multiple-choice questions from the text column of each source document.

config_spec:
  - type: "model"
    key: "synthesizer_model"
    display_name: "Synthesizer Model
    default_value: "openai$gpt-4-1-nano
  - type: "dataset"
    key: "dataset_key"
    display_name: "Dataset"
    default_value: "documents"
definition:
  data_source:
    type: "dataset_samples"
    dataset_key: "<< config.dataset_key >>"
  synthesizers:
    - type: "question_answering"
      model_key: "<< config.synthesizer_model >>"
      qa_type: "multiple_choice"
      content_column: "text"
      title_column: "document_title"

Every source document yields exactly one output sample: the model proposes several questions per document, and the one with the highest estimated_difficulty is kept.

{
  "sample_id": "9f2c41ab",
  "question": "Within which period can a refundable fare be cancelled for a full refund?",
  "choices": ["Up to 2 hours before departure", "Up to 24 hours before departure", "Any time after departure"],
  "answer_index": 1,
  "question_mode": "multiple_choice",
  "question_type": "factual",
  "estimated_difficulty": 6,
  "thought_process": "The refund section states the 24-hour cancellation window.",
  "citations": ["Refundable fares can be cancelled up to 24 hours before departure."],
  "document": {"document_title": "Refund Policy", "text": "..."}
}

Note

answer_index is a 0-based index into choices, which the synthesizer shuffles. With qa_type: "open_ended" the sample carries an answer string instead of choices and answer_index, and question_mode is "open_ended".

The columns of the source sample are kept next to the generated ones, and document holds the whole source sample. The generated values come from the model, so they differ on every generation.

Configuration

Properties

---

type Literal “question_answering” required

The type of the synthesizer.

---

model_key TemplateValue required

The key of the chat completion model to be used as a synthesizer.

---

qa_type string required

The type of question answering synthesizer. Supports ‘multiple_choice’ and ‘open_ended’.

---

content_column string, TemplateValue

The name of the text content column in the dataset. This defines the source text when generating QA dataset. Defaults to ‘text’.

Default: text

---

title_column string, TemplateValue

The name of the title column in the dataset. This is supplementary information and not required for generating QA dataset. Defaults to ‘document_title’.

Default: document_title

---

summary_column string, TemplateValue

The name of the summary column in the dataset. This is supplementary information and not required for generating QA dataset. Defaults to ‘document_summary’.

Default: document_summary

---

system_prompt string, TemplateValue

The system prompt to use. If not provided, a default prompt will be used.

Default: None

---

user_prompt string, TemplateValue

The user prompt to use. If not provided, a default prompt will be used.

Default: None

---

additional_instructions string, TemplateValue

Additional instructions to provide to the generator model.

Default: none