Measure Whether Your RAG Chatbot Answers Completely

Ask an airline support chatbot a real question and you rarely ask just one thing. “How can I take my musical instrument on board, and what happens if it is too big for the cabin?” is two questions wearing one coat. A chatbot can answer the first half, sound confident, wish you a nice day, and leave the second half on the floor. Nobody notices, because the answer reads well. The customer just never learns they needed to book an extra seat.

That gap has a name: recall. It is the fraction of the facts a correct answer should contain that the chatbot actually reproduced. It is a different failure from hallucination. A hallucinating bot says something false; a low-recall bot says nothing false and still fails you, by leaving things out.

This tutorial runs one evaluation from the LF AI Platform Atlas, rag_recall_text_documents, against a live airline support chatbot hosted on Dify. It generates question-answer pairs straight from the airline’s own help documents, sends the questions to the chatbot, and then uses a judge to check how much of each expected answer came back. You end up with a recall score per question type and a stack of transcripts that show you exactly what got dropped.

What you will build

• A dedicated app on LF AI Platform for this work
• Your Dify chatbot connected as the model under test
• The airline’s help documents registered as a knowledge base dataset
• The rag_recall_text_documents evaluation, generating its own test questions from those documents
• A recall score for two kinds of question, plus the transcripts behind it

Before you start

• The lf CLI, configured against your platform. Run lf status; it should print a URL and an app key.
• A Dify chatbot you can call: its chat-messages URL and an API key.
• An OpenAI API key. The evaluation needs a general-purpose model for two jobs, writing the test questions and judging the answers. We use OpenAI for both.
• A folder of the documents your chatbot is supposed to know. Here that is the airline’s public help pages saved as Markdown.

How the evaluation works

LF AI Platform organizes work into a few entities:

Entity	What it is
app	A workspace that holds your evaluation runs
model	An endpoint you call: your chatbot, or a general-purpose model
dataset	The input samples, here generated from your documents
task	How a sample is run and scored
evaluation	Defines how the models, dataset, and task are tied together
evaluation run	A set of results of the defined evaluation

You do not have to write the test set. The evaluation builds one from your documents:

• A synthesizer model reads each help document and writes question-answer pairs grounded in that text. The answer it writes is the ground truth.
• Your chatbot answers each question on its own, using its own retrieval.
• A judge model breaks the ground-truth answer into atomic claims and checks, one by one, whether the chatbot’s response reproduces each claim. Recall is the fraction that survived.

This claim-by-claim scoring is what makes recall meaningful. Instead of asking “does the answer roughly match,” the judge asks “of the five things the correct answer says, how many did the chatbot actually say.” That is the difference between a chatbot that answers your whole question and one that answers the easy part.

The evaluation runs this in two modes:

• Fact retrieval: questions answerable by copying one explicit fact from a document, a number or a short phrase. “What is the checked baggage allowance for SWISS Business?” This tests literal grounding.
• General QA: open-ended questions whose answers carry several claims and need some reasoning across the document. This is where completeness gets hard.

Step 1: Set up a clean workspace

Create a new app so this work does not tangle with your other evaluation runs, then switch to it. Save this as app.yaml:

app.yaml

display_name: "Airline Chatbot"
key: "airline_chatbot"

lf add app -f app.yaml
lf switch airline_chatbot

lf status should now show airline_chatbot as the active app.

Step 2: Connect your Dify chatbot

The chatbot under test speaks Dify’s chat-messages API, which does not look like a standard chat-completion endpoint. You bridge the gap with a model adapter: a small pair of templates that reshapes each request on the way out and each response on the way back.

Save the adapter as adapter.yaml:

adapter.yaml

display_name: "Dify Chat Completion"
key: "adapter-dify"
long_description: >
  Adapter for Dify chat messages. Maps a LatticeFlow chat-completion request to
  Dify's `/v1/chat-messages` blocking call, then unwraps the answer (and any
  `metadata.retriever_resources`) back into chat-completion shape.

  Multi-turn: `conversation_id` is lifted from the last assistant message so
  subsequent turns thread into the same Dify conversation.
process_input:
  language: "jinja"
  source_code: |
    {% set assistants = input.messages | selectattr("role", "equalto", "assistant") | list %}
    {
        "inputs": {},
        "query": {{ input.messages[-1].content | tojson }},
        "response_mode": "blocking",
        "conversation_id": {{ (assistants[-1].conversation_id | default("", true) if assistants else "") | tojson }},
        "user": "latticeflow",
        "files": []
    }
process_output:
  language: "jinja"
  source_code: |
    {% set body = body | fromjson %}
    {
        "choices": [
        {
            "message": {
                "role": "assistant",
                "content": {{ body.answer | tojson }},
                "conversation_id": {{ body.conversation_id | tojson }}
            },
            "references": {{ ((body.metadata | default({}, true)).retriever_resources | default([], true)) | tojson }}
        }
        ]
    }

One detail here pays off later: the output template lifts metadata.retriever_resources into a references field. That carries the chunks Dify’s retriever pulled for each answer, and those chunks become your best diagnostic when a score comes back low.

Now describe the chatbot itself in model.yaml, pointing it at the adapter:

model.yaml

display_name: "Airline Customer Support Chatbot"
key: "model-dify"
description: >
  Chatbot hosted on Dify.
rate_limit: 60
task: "chat_completion"
config:
  adapter:
    key: "adapter-dify"
  connection_type: "custom_connection"
  url: $DIFY_URL
  api_key: "<< secrets.DIFY_API_KEY >>"

The API key goes through a secrets reference, so it lives in server-side secret storage rather than in plain text. Put the key there now, and add your OpenAI provider so the judge and synthesizer have a model to run on:

lf secret add --name DIFY_API_KEY --value <your-dify-api-key>
lf integration add --provider openai --api-key $OPENAI_API_KEY

Register the adapter, the chatbot, and a general-purpose OpenAI model:

lf add model-adapter -f adapter.yaml
lf add model -f model.yaml
lf add model -p openai/gpt-5.4-mini

A provider model added with -p gets a key with a $ in it, so gpt-5.4-mini becomes openai$gpt-5-4-mini.

Before you spend time on anything else, confirm the chatbot actually answers:

lf test model model-dify

A healthy result ends with Successfully tested and prints the chatbot’s reply. The command walks you through the whole round trip: your input, the adapter reshaping it into Dify’s format, the raw Dify response, and the adapter unwrapping it back. If the adapter has a typo, this is where you catch it, not five minutes into a run.

Step 3: Register the knowledge base

The evaluation writes its test questions from documents, so it needs the documents. Drop the airline’s help pages into a folder as Markdown, one file per topic:

data/knowledge_base/
  animals-travelling.md
  baggage.md
  booking-tariffs.md
  change-cancel-booking.md
  check-in.md
  ...

Then point a dataset at that folder. Save data/knowledge_base.yaml:

data/knowledge_base.yaml

key: airline-knowledge-base
display_name: Airline Knowledge Base
description: Airline support knowledge-base documents used by the chatbot, one row per document.
source:
  type: folder
  directory_path: knowledge_base

A folder source scans the directory and turns each file into one row, with the file contents in a content column and the file name in a title column. Those column names matter in the next step.

lf add dataset -f data/knowledge_base.yaml

Step 4: Pull the evaluation from Atlas

Atlas is a library of ready-made evaluations. Download this one:

lf init --atlas atlas-rag_recall_text_documents

You get a folder with everything the evaluation needs:

atlas-rag_recall_text_documents/
  run.yaml                 # declares the dataset generators and datasets the eval depends on
  evaluation.yaml          # the evaluation definition, two task specs
  config.env               # the generator knobs you fill in
  config.yaml              # the model keys you fill in
  datasets/                # the QA generators and the datasets they produce
  GENERIC_FACT_RETRIEVAL_INSTRUCTIONS.md
  RUN.md                   # the template's own notes

Two dataset generators come in the box, one per question mode. The general generator uses a question_answering synthesizer over your documents; the fact retrieval generator uses an LLM synthesizer with a prompt that insists every question be answerable by copying one explicit fact. You do not need to touch either. The datasets they produce, general_qa and fact_retrieval, are created on the fly when you run.

The only files you edit are config.env and config.yaml.

Step 5: Point the evaluation at your entities

Open atlas-rag_recall_text_documents/config.env. It ships with placeholders. Fill in the synthesizer model and the column names from your knowledge base:

atlas-rag_recall_text_documents/config.env

SYNTHESIZER_MODEL_KEY="openai$gpt-5-4-mini"

DOCUMENTS_DATASET_KEY="airline-knowledge-base"
DOCUMENTS_DATASET_CONTENT_COLUMN="content"
DOCUMENTS_DATASET_TITLE_COLUMN="title"
DOCUMENTS_DATASET_SUMMARY_COLUMN=

NUM_SAMPLES_TO_GENERATE=10

The model under test and the judge are set in config.yaml instead:

atlas-rag_recall_text_documents/config.yaml

config:
  # Evaluated Model: The model to be evaluated.
  model_key: model-dify
  # Judge Model: The key of the judge model used to evaluate model responses.
  judge_model_key: openai$gpt-5-4-mini

Three roles, three model keys. The chatbot under test is model-dify. The judge and the synthesizer can share one general-purpose model, and here they do. The document columns tell the generators where to read; leave the summary column empty if your documents do not have one. NUM_SAMPLES_TO_GENERATE is per mode, so 10 here means 10 general and 10 fact-retrieval questions.

Your Dify URL still needs to reach the model YAML. Put it, and anything else the specs read from the environment, in a .env file at the project root:

.env

DIFY_URL=https://api.dify.ai/v1/chat-messages

The CLI loads .env automatically.

Step 6: Run the evaluation

Register the dataset generators and datasets first, passing config.env via --env so the generator placeholders resolve:

lf --env atlas-rag_recall_text_documents/config.env add -f atlas-rag_recall_text_documents/run.yaml

Then run the evaluation by key, passing config.yaml via -c. The -w flag waits and prints results when it finishes:

lf run eval -k atlas-rag_recall_text_documents -c atlas-rag_recall_text_documents/config.yaml -w

Between them the two commands generate both QA datasets, run your chatbot over all 20 questions, score each with the judge, and print an evaluation run ID and a link.

Step 7: Read the results

When the both tasks finished with no errors, we can view their recall scores. The recall scores tell two different stories:

Question mode	Recall
Fact Retrieval	1.00
General QA	0.83

Fact retrieval is perfect. When a customer asks for one concrete number, the chatbot finds it and states it. Baggage allowances, size limits, phone numbers, all reproduced exactly.

General QA at 0.83 is where completeness slips. These answers carry several claims each, and the chatbot reproduced most but not all of them.

What got dropped, and why

Reading the general-QA transcripts against the ground-truth answers, three patterns explain every miss:

Pattern	What happened
Retrieval miss	The retriever returned nothing, so the chatbot answered from general knowledge with no grounding
Dropped qualifier	The chatbot answered the main point but omitted an exception or condition it had retrieved
Detail collapse	The chatbot summarized a multi-part answer and lost the enumerated specifics

The scorer records the chunks the retriever returned for each answer, so you can tell a retrieval failure from a generation failure without guessing.

A closer look at the worst answer

The lowest score, 0.50, went to a question about fare classes:

Question: How are fare classes identified across the different service classes, and what are these letters used for?

Ground truth: The categories are identified by letters that depend on the service class: B, E, G, H, K, L, M, Q… for Economy, A and F for First, and so on. The letters indicate the mileage credit Miles & More members receive and act as abbreviations for the fare the conditions are based on.

Chatbot: Fare classes are typically identified by specific letters that correspond to different service classes. These letters help in categorizing the fare types… If you need more detailed information regarding specific fare classes for Swiss Airlines, please let me know.

The chatbot retrieved zero references for this question. With nothing grounded in front of it, it fell back on a generic, textbook description of how fare classes work in general, and offered to help further. It never named a single letter. The information sits in the knowledge base; the retriever simply did not surface it.

That is a different problem from the other misses. Where the retriever did its job but the chatbot dropped a qualifier, for instance answering that automated check-in happens 20 hours before departure but omitting that it is available only for bookings made on swiss.com, the fix lives in the generation prompt. Where the retriever returns nothing, the fix lives in the retrieval pipeline. Recall on its own says “0.83, good.” The transcripts tell you which knob to turn.

Where to go next

• Chase the retrieval miss. The fare-class document exists but was not retrieved. Check the chunking and the retriever settings in the Dify workflow, then re-run and watch that score move.
• Turn up the volume. Set NUM_SAMPLES_TO_GENERATE higher and run again to see whether the retrieval miss was a one-off or a pattern.
• Pair recall with faithfulness. Recall tells you what the chatbot left out; faithfulness tells you whether what it did say is grounded. The Atlas has evaluations for both, and reading them together is how you separate a retrieval problem from a generation one.

Command recap

lf add app -f app.yaml
lf switch airline_chatbot

lf secret add --name DIFY_API_KEY --value <your-dify-api-key>
lf integration add --provider openai --api-key $OPENAI_API_KEY
lf add model-adapter -f adapter.yaml
lf add model -f model.yaml
lf add model -p openai/gpt-5.4-mini
lf test model model-dify

lf add dataset -f data/knowledge_base.yaml

lf init --atlas atlas-rag_recall_text_documents
# edit atlas-rag_recall_text_documents/config.env, config.yaml and .env

lf --env atlas-rag_recall_text_documents/config.env add -f atlas-rag_recall_text_documents/run.yaml
lf run eval -k atlas-rag_recall_text_documents -c atlas-rag_recall_text_documents/config.yaml -w
lf overview eval-run --id <id>
lf export eval-run --id <id> -o results