A retrieval-augmented generation pipeline in Python with a rigorous offline evaluation harness. Chunks and embeds documents, retrieves by vector similarity, and generates grounded answers — with pluggable LLM providers (including a deterministic local fake for tests) and metrics for retrieval quality and answer faithfulness. No API key required.

Why a passing benchmark isn't safe to ship: a free 2-stage (benchmark + replay) validation recipe for LLM model swaps & prompt changes, run on flat-rate coding-agent subagents — no eval API bill.

Catch LLM quality regressions in CI: a golden-set regression gate with calibrated graders (grounding, hallucination, tool-call, LLM-judge) that fails the PR when answer quality drops.

Agentic diagnostic assistant for distributed-system incidents: multi-turn RAG, hypothesis updates, evidence packing, golden evals, and failure-attributed run reports.

RAG evaluation harness with RAGAS, TruLens, LLM-as-Judge, golden sets, Recall@k, nDCG, and faithfulness.

RAG evaluation framework: hit-rate, MRR, faithfulness scoring, and async batch evaluation with golden question datasets

Production evaluation harness for AI-driven interactive entertainment. 7 deterministic evaluators, LLM-as-Judge, golden-set regression, red-team probes. Extracted from Dead Signal, a live game running Gemini dialogue generation.

Portfolio-grade AI quality evaluation lab with golden datasets, prompt regression, groundedness checks, hallucination tests and CI thresholds.