A RAG answer can fail even when the model behaves perfectly. The right document may never have reached it.
Evaluate the pipeline in stages. First measure retrieval: did the search return the required source, and how highly was it ranked? Then inspect context assembly: were useful passages preserved, deduplicated, and kept within budget? Finally evaluate generation for groundedness, completeness, relevance, and citation accuracy.
Build cases with verified source documents and required facts. Include queries with no answer in the knowledge base; a good system should abstain rather than invent. Add conflicting and outdated documents to test precedence.
Track results by corpus, query type, language, and retrieval configuration. An average score can hide one collection that never retrieves correctly.
Keep the full trace. A low groundedness score only says the final answer failed. Retrieval and context evidence tell you whether to change chunking, search, reranking, prompt behavior, or the underlying documents.