DeepEval: LLM Testing for QA Engineers (A Practical 2026 Guide)
+ + +For a decade, test automation had a comfortable assumption baked into it: given the same input, the system
+ under test returns the same output. Assert on that output, and you have a test. Large language models broke
+ that assumption. Ask an LLM the same question twice and you can get two differently-worded answers that are
+ both correct — or two that are both subtly wrong. As a QA engineer, "assert
+ response == expected" simply doesn't work anymore.
+
This is the gap DeepEval fills, and it's + become the tool I reach for first when a project ships anything AI-powered.
+ +What DeepEval actually is
+DeepEval is an open-source LLM evaluation framework from Confident AI. The single most important thing to
+ understand about it — especially if you come from a Selenium/Cypress/Playwright background — is that it's
+ a pytest plugin. If you've ever written a unit test in Python, DeepEval feels instantly familiar. You
+ write test cases, you run deepeval test run, and you get pass/fail results you can gate a pipeline
+ on.
The difference is how it decides pass or fail. Instead of comparing strings byte-for-byte, DeepEval + scores outputs using a mix of LLM-as-a-judge evaluation, natural language inference (NLI) + models, and statistical methods. That lets it measure fuzzy, human qualities — relevancy, faithfulness, + hallucination, bias, tool-calling correctness — the things that actually matter for an LLM feature.
+ +The metric that changes everything: G-Eval
+Out of the 50+ metrics DeepEval ships in 2026, the one worth learning first is G-Eval. G-Eval + uses an LLM-as-a-judge with chain-of-thought reasoning to evaluate output against any custom criteria you + describe in plain English.
+That is a genuinely new capability for a QA engineer. You can write an acceptance criterion the same way a + product owner would phrase it — "the answer must not promise a refund timeline" — and turn it into an automated, + repeatable check.
+ +from deepeval import assert_test
+from deepeval.metrics import GEval
+from deepeval.test_case import LLMTestCase, LLMTestCaseParams
+
+correctness = GEval(
+ name="Correctness",
+ criteria="Determine whether the actual output is factually "
+ "correct and does not contradict the expected answer.",
+ evaluation_params=[
+ LLMTestCaseParams.INPUT,
+ LLMTestCaseParams.ACTUAL_OUTPUT,
+ LLMTestCaseParams.EXPECTED_OUTPUT,
+ ],
+ threshold=0.5,
+)
+
+def test_billing_answer():
+ test_case = LLMTestCase(
+ input="How long do refunds take?",
+ actual_output="Refunds are processed within 5-7 business days.",
+ expected_output="Refunds take 5 to 7 business days.",
+ )
+ assert_test(test_case, [correctness])Run that with deepeval test run test_billing.py and it behaves like any other pytest suite — green
+ when the criterion holds, red (with a reasoned explanation) when it doesn't.
The metrics QA teams lean on most
+Beyond G-Eval, a handful of built-in metrics cover the majority of real-world LLM bugs:
+-
+
- Answer Relevancy — is the response actually addressing the question, or wandering off? +
- Faithfulness — for RAG systems, does the answer stay true to the retrieved context, or is + the model inventing things? +
- Hallucination — is the model stating facts that aren't supported by any provided source? + +
- Contextual Precision / Recall — is your retrieval step surfacing the right + documents in the right order? +
- Bias & Toxicity — safety gates you genuinely want failing loudly in CI. +
For a Retrieval-Augmented Generation app, Faithfulness plus Contextual Recall together tell you whether a bad + answer is the retriever's fault or the model's fault — a distinction that used to take hours of + manual debugging.
+ +Wiring it into CI/CD
+Because DeepEval is pytest-native, the CI story is boring in the best possible way. It's the same pattern I + already use for Cypress and Playwright suites:
+-
+
- Store a dataset of representative inputs (a "golden set") as test cases. +
- Run
deepeval test runas a stage in GitHub Actions or Jenkins.
+ - Fail the build if any metric drops below its threshold. +
That converts "the AI feels worse this week" — an unfalsifiable complaint — into a red pipeline with a number + attached. It's the difference between vibes and a quality gate.
+ +The mental shift for a QA engineer isn't the syntax — it's accepting that a score above a + threshold, not an exact match, is a legitimate pass condition.+ +
Where DeepEval fits in a modern QA stack
+I don't think of DeepEval as replacing anything. Your Playwright tests still verify the app renders and the + button works. DeepEval sits one layer deeper, verifying that the content the LLM produced is relevant, + grounded, and safe. On an AI product, you need both — and a QA engineer who can speak fluently about + hallucination rate and faithfulness scores is suddenly a lot more valuable than one who only knows locators.
+If you're a test automation engineer watching AI features land in your product and wondering where you fit, + this is the answer: LLM evaluation is QA. DeepEval is the most approachable on-ramp I've found. Next, + see how Playwright + MCP brings AI agents into + browser testing.
+ +Need a real quality gate for your AI feature?
+I'm a Senior QA Automation Engineer specialising in AI test automation, Playwright, and CI/CD.
+ Let's connect on + LinkedIn +
+