Faithfulness and relevance are related, but they are not the same.
Relevance means the answer is useful for the user’s question.
Example:
A relevant answer addresses the question.
Faithfulness means the answer is supported by the provided context or source.
Example source:
Faithful answer:
Unfaithful answer:
The unfaithful answer may still sound relevant because it talks about refunds, but it is not supported by the source.
Best case.
This is both relevant and faithful.
Dangerous case.
The answer addresses the question but gives unsupported information.
This is a hallucination or grounding failure.
The answer is supported by the document but does not answer the user’s question.
The answer is faithful to the document but irrelevant to the question.
Worst case.
This is both unsupported and not useful.
In RAG systems, relevance alone is not enough.
An answer can sound useful but still be unsupported.
Example:
That is risky.
Faithfulness ensures the model does not invent facts beyond the given context.
Relevance measures whether the answer addresses the user’s question, while faithfulness measures whether the answer is supported by the provided context. In RAG, an answer can be relevant but unfaithful if it answers the question using unsupported or invented information. It can also be faithful but irrelevant if it quotes correct context but does not answer the question. A strong RAG system needs both: the answer should be useful to the user and grounded in the retrieved sources.
RAG can reduce hallucination, but it can still fail.
Common RAG failure modes include:
The correct document exists, but the retriever does not fetch it.
Reasons:
If chunks are too small, they may miss context. If chunks are too large, they may add noise.
The retriever may return semantically similar but answer-irrelevant chunks.
Example:
Even if the right chunk is retrieved, the model may ignore it due to long context or competing information.
Multiple documents may provide different answers.
Example:
The model may combine retrieved facts incorrectly.
The answer may be correct-looking but not traceable to sources.
The vector database may contain outdated documents.
The retriever may fetch documents the user should not access.
A RAG pipeline should be evaluated in two parts:
Because a RAG system can fail even before the LLM generates anything. If the retriever brings the wrong chunks, the model may either answer incorrectly or hallucinate.
Retrieval evaluation checks whether the system fetched the right documents or chunks.
Important metrics:
Recall@k checks whether the correct answer-containing chunk appears in the top-k retrieved results.
Example:
This is important because if the correct chunk is not retrieved, the LLM has no reliable source to answer from.
Precision@k checks how many of the retrieved chunks are actually relevant.
Example:
High precision means the context is clean and not noisy.
MRR means Mean Reciprocal Rank. It checks how high the first correct result appears.
If the correct chunk appears at rank 1, that is better than rank 8.
Hit rate checks whether at least one relevant document was retrieved.
It is simpler than recall but useful for high-level monitoring.
Generation evaluation checks whether the final answer is correct, relevant, complete, and grounded.
Important metrics:
Does the answer actually address the user’s question?
Example:
Does the answer stay supported by the retrieved context?
If the retrieved document says:
But the model answers:
That is unfaithful.
If the system provides citations, the cited chunk should actually support the claim.
A common failure is:
That is still a RAG quality issue.
A production RAG system should be evaluated end-to-end using a golden dataset.
A golden dataset contains:
Then you test:
You should evaluate both offline and online.
Used before deployment.
Used after deployment.
A strong RAG evaluation should answer:
A RAG pipeline should be evaluated at both retrieval and generation stages. For retrieval, I would measure whether the correct chunks are retrieved and ranked well using metrics like recall@k, precision@k, MRR, hit rate, and context relevance. For generation, I would evaluate answer relevance, faithfulness, groundedness, citation accuracy, completeness, and hallucination rate. I would use a golden dataset containing user questions, expected answers, and relevant source chunks for offline evaluation, and then monitor production feedback, latency, cost, and failure cases online. The key point is that RAG can fail at retrieval, ranking, context construction, or generation, so evaluation must cover the full pipeline.