RAGAS vs TruLens vs DeepEval

LLMs are getting stronger every day, and building a RAG pipeline has never been easier. Knowing whether it actually works is not. Most teams ship a RAG system, see decent-looking answers, and call it done, until users hit hallucination, missing context, or irrelevant chunks.

That’s where evaluation frameworks come in. RAGAS, TruLens, and DeepEval are three of the most widely used tools for measuring RAG quality. In this article, I’ll break down how each one works and when to reach for it.

Why RAG Needs Its Own Evaluation Approach

A RAG system has two moving parts: the retriever, which fetches context, and the generator, which writes the answer using that context. If either half fails, the final answer fails, but they fail in different ways. A bad retriever pulls irrelevant or incomplete chunks. A bad generator ignores good context and hallucinates anyway, or writes something technically correct but unhelpful.

Traditional NLP metrics like BLEU or ROUGE don’t capture any of this. They compare word overlap between the generated answer and a reference answer, useful for translation, not for judging whether an LLM stayed grounded in facts or whether the retriever did its job. RAG evaluation needs metrics that check both halves separately and together, without always requiring a handwritten correct answer for every query. This is exactly the gap RAGAS, TruLens, and DeepEval were built to fill, and each one approaches it with a different philosophy.

The Three Layers of RAG Evaluation

Before diving into each tool, it helps to know that most RAG metrics fall into three buckets: 

1. Retrieval quality: Did the system fetch the right chunks? 
2. Generation quality: Did the model use those chunks correctly, without hallucination. 
3. End-to-End quality: Does the final answer satisfy the user’s questions? 

Every framework below scores some combination of these three. The difference is in how automated scoring is whether it needs ground truth answers, and if the tool is meant for one-time evaluation or continuous monitoring. 

Retrieval Metrics You Need to Know: Precision@K, Recall@K, MRR, NDCG

It is worth understanding the classic information – retrieval metrics that all three build on. These come from traditional search and recommendation systems, and they measure retrieval quality directly using labeled relevance data, no LLM judge needed. 

Precision@K 

What it measures: Out of the top K chunks your retriever returned, how many are actually relevant? 

Formula: Precision@K = (Relevant chunks in top K) / K 

Example: You retrieve the top 5 chunks for a query. 3 of them are relevant to answering it. 

Then, Precision@5 = 3 / 5 = 0.6 

A high Precision@K means your retriever isn’t pulling in noise. A low score means the LLM must sift through junk context, which increases hallucination risk since irrelevant chunks can confuse the generator. 

Recall@K 

What it measures: Out of all the relevant chunks that exist in your knowledge base, how many did you actually retrieve in the top K? 

Formula: Recall@K = (Relevant chunks retrieved in top K) / (Total relevant chunks in the corpus) 

Example: There are 4 relevant chunks in your entire corpus for a query. Your top 5 retrieved chunks contain 3 of them. 

Then, Recall@5 = 3 / 4 = 0.75 

Precision and Recall usually trade off against each other. Increasing K i.e increasing more chunks tends to raise Recall but lower Precision, since you’re pulling in more noise along with the useful chunks. This is why RAGAS reports context precision and context recall as two separate scores instead of one. 

Mean Reciprocal Rank (MRR) 

What it measures: How high up was the first relevant chuck in your ranked results? This matters because LLMs tend to pay more attention to earlier context, so burying the one useful chunk at position 8 is worse than having it at position 1 

Formula: MRR = (1 / N) × Σ (1 / rank of first relevant chunk for each query) 

Example: For Query A, the first relevant chunk is at rank 1 is  

reciprocal rank = 1 / 1 = 1  

For Query B, the first relevant chunk is at rank 3 then, the reciprocal rank = 1 / 3 = 0.3333 

MRR = (1.0 + 0.33) / 2 = 0.665 

Higher MRR means your retriever consistently surfaces the best chunk near the top, not buried deep in the results. 

Normalized Discounted Cumulative Gain (NDCG) 

What it measures: A more nuanced version of the above, it accounts for graded relevance, i.e some chunks are more relevant than others, not just relevant/irrelevant, and penalize relevant chunks that appear lower in the ranking. 

How it works: 

1. DCG (Discounted Cumulative Gain): It sums up relevance scores of retrieved chunks, but discounts chunks that appear lower in the ranking (using a log based penalty). 
2. NDCG: Normalizes that DCG value against the ideal possible ranking i.e, if the retriever had perfectly ordered chunks from most to least relevant, producing a score between 0 and 1. 

Formula: 

An NDCG of 1 means your retriever ranked chunks exactly as well as theoretically possible. This metric matters more when relevance isn’t binary. For example: A chunk might be highly relevant, somewhat relevant or irrelevant rather than just yes or no.

How do these connect to the frameworks? 

Keep these four in mind as you read the next sections, each framework re-implements a version of them, just scored differently: 

1. Precision@K: It reappears as RAGAS’s Context Precision and DeepEval’s Contextual Precision but scored by an LLM judge instead of exact-match relevance labels. 
2. Recall@K: Reappears as RAGAS’s Context recall and DeepEval’s Contextual Recall, checking whether all necessary information was retrieved. 
3. MRR and NDCG: these don’t have a direct named equivalent in any of the three frameworks, but TruLen’s per chunk Context Relevance scores gives you the raw data to compute them yourself. 

The main difference is that classic Information Retrieval (IR) metrics require pre-established relevance labels (“which chunks are relevant“) before you can compute anything. RAGAS, TruLens, and DeepEval swap in an LLM for that relevance judgment, avoiding the requirement. This needs little setup but adds more noise than a proper IR benchmark. To check each method’s trustworthiness, label 200 to 300 query/chunk pairs and compute Precision@K, Recall@K, and NDCG as a baseline. 

Read more: 12 Important Model Evaluation Metrics

RAG Assessment 

Here we’ll put the 3 frameworks to test:

RAGAS

RAGAS is a python framework built specifically for RAG pipelines. Its biggest selling point is that most of its metrics don’t need human labelled ground truth, it uses an LLM as a judge to score outputs against the retrieval context itself, which makes it fast to set up on an existing pipeline. 

Core metrics: 

1. Faithfulness: Is the response consistent with the fetched context, or is there any fabricated information? RAGAS breaks the response into individual statements and checks to see if every individual statement can be confirmed against the source chunks. 
2. Answer Relevance: Does the response directly answer the question, or does it provide indirect answers that would still make factual sense?  
3. Context Precision: Are the fetched chunks consistent with the answer, or is there any unrelated content added to the useful content?  
4. Context Recall: Did the retrieval contain complete and required information, or was there any missing critical pieces? This requires an answer for reference.  
5. Answer Correctness: How closely does the generated output resemble an established answer? This will use a combination of fact similarities and semantic similarities and will require a source answer to use for comparison.  
6. Semantic Similarity: A lower intensity version of the previous, it verifies that the generated output and source output can be compared using the premise of embedding, as opposed to verifying individual claims against each respective output. 

How it works in practice:  

You pass RAGAS a dataset of {questions, retrieved_contexts, generated_answer} and optionally a reference answer, and it runs each metric using an LLM under the hood. 

A typical run looks like: 

Code: 

from ragas import evaluate 
from ragas.metrics import faithfulness, answer_relevancy, context_precision 

results = evaluate(dataset, metrics=[faithfulness, answer_relevancy, context_precision])

Each metrics returns a score between 0 and 1, and RAGAS aggregates them into a summary table you can log or compare across pipeline version. 

Synthetic test set generation: One of RAGAS’s most useful features that often gets overlooked, it can auto generate a test dataset directly from your documents instead of you writing questions and reference answers by hand. It samples chunks from your knowledge base, then uses an LLM to generate realistic questions, reference answers, and even harder variants. This solves the biggest bottleneck in RAG evaluation: building a labeled test set is normally the slowest part, and this cuts most of that manual work. 

from ragas.testset import TestsetGenerator

generator = TestsetGenerator.from_langchain(llm, embeddings)

testset = generator.generate_with_langchain_docs(
    documents,
    testset_size=50
)

This gives you a ready-made {question, reference_answer, contexts} dataset to run every metric above against, without hand-labeling a single row. 

Strengths: Minimal setup, works well for quick benchmarking, plays nicely with LangChain and LlamaIndex out of the box. 

Limitations: Since most metrics rely on an LLM judge, scores can vary slightly run to run, and you’re paying for extra LLM calls on every evaluation pass. 

Best for: Teams wanting fast, automated, reference-free scoring right after building a RAG prototype, or for A/B testing two versions of a retriever or prompt. 

TruLens

TruLens takes a different angle, it’s built for observability, not just one-off scoring. Instead of running an evaluation once and getting a report, TruLens instruments your RAG app so it logs every step: what was retrieved, what the LLM saw, and what it generated. That log becomes the basis for its scores. 

Core concept: the RAG Triad 

1. Context Relevance:how relevant is the retrieved context to the query? Scored chunk by chunk, not just for the whole retrieved set. 
2. Groundedness: is the answer supported by the context, or does it introduce claims that aren’t backed by any retrieved chunk? 
3. Answer Relevance: does the answer address the actual question the user asked? 

What sets TruLens apart is the tooling around these three scores. It ships with a dashboard which is built on Streamlit where you can: 

1. Compare multiple experiment runs side by side 
2. Drill into a single failed example and see exactly which chunk was irrelevant or which claim wasn’t grounded 
3. Track how scores drift over time as you change prompts, chunk sizes, or embedding models 

A basic setup looks like wrapping your existing RAG chain with a TruChain or TruLlama recorder, then letting it log automatically as real queries comes in. 

Code: 

from trulens.apps.langchain import TruChain

tru_recorder = TruChain(
    rag_chain,
    app_id="rag_v1",
    feedbacks=[groundedness, context_relevance]
)

with tru_recorder as recording:
    response = rag_chain.invoke(query)

Strengths: Best-in-class visibility into why a RAG system failed on a specific query, not just that it failed. Great for ongoing monitoring rather than a single benchmark run. 

Limitations: More setup overhead than RAGAS if you just want a quick score. The value really shows up once you’re running many queries over time, not on a single evaluation batch. 

Best for: Teams running RAG in production who need to track quality over time, not just at build time. If you’re iterating on prompts or retrievers and want to see exactly what changed and why, TruLens’s tracing is the stronger fit. 

DeepEval

DeepEval is a testing-first framework, it treats RAG evaluation like unit testing for LLM outputs, and plugs directly into Pytest. If your team already runs automated tests before every deployment, DeepEval slots RAG quality checks into that same pipeline instead of living as a separate notebook or dashboard. 

Core metrics: 

1. Answer Relevancy: does the response actually answer the question asked? 
2. Faithfulness: are all claims in the answer backed by retrieved context? 
3. Contextual Precision & Recall: separate scores for whether the retrieved chunks were both relevant and complete. 
4. Hallucination detection: flags claims that appear nowhere in the source material. 
5. Custom metrics via G-Eval:  lets you define your own scoring criteria in plain language and have an LLM judge apply it consistently, useful when the built-in metrics don’t match your domain’s definition of good. 

How it works in practice: You define test cases the same way you would write a unit test, set a pass/fail threshold, and run it through Pytest: 

Code: 

from deepeval import assert_test
from deepeval.metrics import FaithfulnessMetric
from deepeval.test_case import LLMTestCase

test_case = LLMTestCase(
    input=query,
    actual_output=generated_answer,
    retrieval_context=retrieved_chunks
)

faithfulness_metric = FaithfulnessMetric(threshold=0.7)

assert_test(test_case, [faithfulness_metric])

If faithfulness drops below the threshold, the test fails, the same way a broken function would fail a unit test. This makes it straightforward to block a deploy if a change to your retriever or prompt quietly degrades RAG quality. 

Strengths: Fits naturally into existing CI/CD and testing workflows. Clear pass/fail gates instead of just a score to interpret. Custom metrics via G-Eval add flexibility for niche use cases. 

Limitations: Being test-oriented, it’s less suited for open-ended exploration or visual debugging, for that, TruLens’s dashboard is a better fit. 

Best for: Engineering teams that want RAG evaluation baked into their existing test suite and deployment pipeline, with pass/fail gates rather than exploratory dashboards. 

Choosing Based on Where You Are

Case 1:  

Just built your first RAG prototype and want quick scores use RAGAS. Run it once against a sample set of queries, get faithfulness and relevance scores, and know within minutes if your retriever or prompt needs work. 

Case 2:

Running RAG in production and need to catch quality drift use TruLens. Instrument your app once, then keep an eye on the dashboard as real traffic flows through. If groundedness starts dropping after a model or embedding change, you’ll see it immediately. 

Case 3:

Want RAG quality checks inside your deployment pipeline use DeepEval. Write it once as a test suite, and every PR that touches your retriever or prompt gets automatically checked before it ships. 

Many teams don’t pick just one. A common pattern is using RAGAS or DeepEval for structured scoring during development, and TruLens for live monitoring once the system is in production. They solve overlapping problems but at different points in the lifecycle, prototype, ship, and monitor. 

Common Pitfalls Across All Three

A few things to watch for regardless of which framework you pick: 

1. LLM-judge variance: since all three rely on an LLM to score outputs, results can shift slightly between runs. Don’t treat a single score as final truth, look at trends across multiple runs instead. 
2. Cost of evaluation calls:  every metric that uses an LLM judge is an extra API call. Running these on every single query in production gets expensive fast, sample instead of evaluating everything. 
3. Metrics without context:  a high faithfulness score means nothing if context precision is low. Always look at retrieval and generation metrics together, not in isolation. 
4. Quality scores ignore latency and cost:  none of the metrics above tell you how long a query took or what it cost to run. A pipeline that scores higher on faithfulness but takes 8 seconds and makes 3 extra LLM calls per query might not be the better system for production. Track latency and per-query cost alongside quality scores, not as an afterthought. 

Conclusion

RAG evaluation isn’t optional once you move past a demo. RAGAS gives you fast, reference-free metrics for quick iteration. TruLens gives you visibility into production, query by query. DeepEval gives you testing discipline that fits a normal engineering workflow.

None of them is objectively better; the right pick depends on whether you’re prototyping, monitoring, or shipping. Start with RAGAS for a baseline, add DeepEval once you have a CI/CD pipeline worth protecting, and bring in TruLens once real users are hitting your system. That combination will catch quality issues long before your users do.

Frequently Asked Questions