diff --git a/docs/examples/colab.md b/docs/examples/colab.md new file mode 100644 index 0000000..a6338ed --- /dev/null +++ b/docs/examples/colab.md @@ -0,0 +1,37 @@ +# Colab Quickstart Tutorial + +[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/OpenAgentHQ/openagent-eval/blob/main/examples/openagent_eval_colab_tutorial.ipynb) + +A zero-setup, end-to-end tour of OpenAgent Eval that runs entirely in your browser. +It uses the built-in **mock** LLM and retriever providers, so it needs **no API keys** +and makes no network calls — click the badge above and run every cell. + +## What you'll learn + +- Installing OpenAgent Eval and verifying it with `oaeval doctor` +- Building a fully offline evaluation config with the `mock` providers +- Running an evaluation from both the CLI (`oaeval run`) and the Python SDK (`Engine`) +- Reading the terminal, Markdown, HTML and JSON reports +- Generating synthetic test cases with `oaeval synth` +- Optionally plugging in a real provider (OpenAI) through a guarded, key-optional cell + +## Run it + +- **In the browser:** [open in Google Colab](https://colab.research.google.com/github/OpenAgentHQ/openagent-eval/blob/main/examples/openagent_eval_colab_tutorial.ipynb) + and choose *Runtime → Run all*. +- **Locally:** download + [`openagent_eval_colab_tutorial.ipynb`](https://github.com/OpenAgentHQ/openagent-eval/blob/main/examples/openagent_eval_colab_tutorial.ipynb) + and open it with `jupyter notebook`. + +## Prerequisites + +None for the offline walkthrough — the notebook installs its own dependencies in the +first cell: + +```bash +pip install -q openagent-eval pytest +``` + +The optional final section runs against a real provider when an `OPENAI_API_KEY` is +present (via Colab Secrets or an environment variable); without a key it skips itself +and the notebook stays fully offline. diff --git a/docs/examples/corpus.md b/docs/examples/corpus.md new file mode 100644 index 0000000..018d334 --- /dev/null +++ b/docs/examples/corpus.md @@ -0,0 +1,39 @@ +# Corpus Health Auditor Tutorial + +A Jupyter notebook tour of OpenAgent Eval's corpus, diagnosis and synthesis modules: + +- Auditing a document corpus for **staleness**, **duplicates** and **coverage** gaps +- Detecting **contradictions** with an LLM-as-judge +- Combining the checks into a single corpus health report +- Diagnosing evaluation failures with `DiagnosisAnalyzer` +- Generating synthetic and adversarial test cases + +**Download:** [`corpus_and_related_modules.ipynb`](https://github.com/OpenAgentHQ/openagent-eval/blob/main/examples/corpus_and_related_modules.ipynb) + +### What you'll learn + +| Section | Topic | +|---------|-------| +| 1–3 | Installation and preparing a sample corpus | +| 4 | Corpus health: staleness, duplicate, coverage and contradiction detectors | +| 5 | A combined corpus audit report | +| 6 | Failure diagnosis with `DiagnosisAnalyzer` | +| 7 | Synthetic and adversarial test-case generation | + +### Prerequisites + +```bash +pip install openagent-eval jupyter +``` + +The staleness, duplicate, coverage and diagnosis sections run fully offline. The +contradiction detector and the synthesis sections use an LLM-as-judge — set a +`GROQ_API_KEY` (a free key from [Groq](https://console.groq.com)) to run them; without +a key those cells are skipped. + +### Quick start + +```bash +cd examples/ +jupyter notebook corpus_and_related_modules.ipynb +``` diff --git a/docs/examples/index.md b/docs/examples/index.md index 5620933..4dd5f38 100644 --- a/docs/examples/index.md +++ b/docs/examples/index.md @@ -1,46 +1,31 @@ # Examples -Worked examples showing how to use OpenAgent Eval in practice. +Worked examples showing how to use OpenAgent Eval in practice. Each tutorial is a +runnable Jupyter notebook in the +[`examples/`](https://github.com/OpenAgentHQ/openagent-eval/tree/main/examples) +directory of the repository. -## RAG Evaluation Tutorial +| Tutorial | What it covers | Run it | +|----------|----------------|--------| +| [Colab Quickstart](colab.md) | Zero-setup, end-to-end evaluation that runs in the browser — no API keys required | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/OpenAgentHQ/openagent-eval/blob/main/examples/openagent_eval_colab_tutorial.ipynb) | +| [RAG Evaluation](rag.md) | A complete RAG pipeline scored with all 18 retrieval, generation, performance and cost metrics | Local Jupyter | +| [Corpus Health Auditor](corpus.md) | Corpus health checks (staleness, duplicates, coverage, contradiction), failure diagnosis and synthetic test-case generation | Local Jupyter | -A hands-on Jupyter notebook that walks through a complete RAG evaluation: - -- Loading a dataset -- Configuring an LLM and retriever -- Running retrieval and generation metrics -- Interpreting the results - -**Download:** [`rag_evaluation_tutorial.ipynb`](https://github.com/OpenAgentHQ/openagent-eval/blob/main/examples/rag_evaluation_tutorial.ipynb) - -### What you'll learn - -| Section | Topic | -|---------|-------| -| 1 | Setting up the environment and config | -| 2 | Loading and inspecting a dataset | -| 3 | Configuring LLM providers (OpenAI, Ollama, Mock) | -| 4 | Configuring retriever providers (Chroma, Memory, BM25) | -| 5 | Running the evaluation pipeline | -| 6 | Understanding retrieval metrics (precision, recall, MRR, NDCG) | -| 7 | Understanding generation metrics (faithfulness, relevancy, hallucination) | -| 8 | Running all 18 metrics together | -| 9 | Interpreting the report output | - -### Prerequisites +## Running the notebooks locally ```bash pip install openagent-eval jupyter +git clone https://github.com/OpenAgentHQ/openagent-eval +cd openagent-eval/examples +jupyter notebook ``` -### Quick start - -```bash -cd examples/ -jupyter notebook rag_evaluation_tutorial.ipynb -``` +The [Colab Quickstart](colab.md) needs no local setup at all — open it directly in +your browser. ## More examples -See the [scripts/](https://github.com/OpenAgentHQ/openagent-eval/tree/main/scripts) directory -in the repository for additional runnable examples. +For copy-paste snippets covering common workflows, see the +[Quickstart](../quickstart.md). The +[`scripts/`](https://github.com/OpenAgentHQ/openagent-eval/tree/main/scripts) +directory in the repository holds additional runnable examples. diff --git a/docs/examples/rag.md b/docs/examples/rag.md new file mode 100644 index 0000000..d4b50c8 --- /dev/null +++ b/docs/examples/rag.md @@ -0,0 +1,41 @@ +# RAG Evaluation Tutorial + +A hands-on Jupyter notebook that walks through a complete RAG evaluation using +OpenAgent Eval's offline **mock** providers — no API keys required: + +- Building a minimal RAG pipeline (chunking, a local keyword retriever, a mock LLM) +- Configuring an evaluation both from YAML and programmatically +- Running all 18 retrieval, generation, performance and cost metrics +- Diagnosing failures and interpreting the report output +- Writing a custom metric and comparing experiments + +**Download:** [`rag_evaluation_tutorial.ipynb`](https://github.com/OpenAgentHQ/openagent-eval/blob/main/examples/rag_evaluation_tutorial.ipynb) + +### What you'll learn + +| Section | Topic | +|---------|-------| +| 1 | Introduction: what OpenAgent Eval is | +| 2 | Installing and verifying the package | +| 3 | Building a minimal RAG pipeline (retriever + mock LLM) | +| 4 | Evaluating the pipeline (YAML and programmatic config) | +| 5 | A deep dive on all 18 metrics | +| 6 | Interpreting results and diagnosing failures | +| 7 | Advanced usage: custom metrics, batch comparison, LLM judges | +| 8 | Best practices for production RAG evaluation | + +### Prerequisites + +```bash +pip install openagent-eval jupyter +``` + +### Quick start + +```bash +cd examples/ +jupyter notebook rag_evaluation_tutorial.ipynb +``` + +The notebook runs entirely offline with the `mock` providers, so you can work through +every cell without an API key. diff --git a/examples/corpus_and_related_modules.ipynb b/examples/corpus_and_related_modules.ipynb index c614e27..3de61b2 100644 --- a/examples/corpus_and_related_modules.ipynb +++ b/examples/corpus_and_related_modules.ipynb @@ -5,7 +5,7 @@ "id": "cell-000", "metadata": {}, "source": [ - "# OpenAgent Eval v0.4.4: Corpus Health Auditor & Related Modules\n", + "# OpenAgent Eval v0.4.8: Corpus Health Auditor & Related Modules\n", "\n", "---\n", "\n", @@ -42,7 +42,7 @@ "metadata": {}, "outputs": [], "source": [ - "%pip install --upgrade openagent-eval==0.4.4" + "%pip install --upgrade openagent-eval==0.4.8" ] }, { @@ -63,19 +63,26 @@ "import os\n", "from openagent_eval.providers.llm.groq import Groq\n", "\n", - "# Set your Groq API key (get yours free at https://console.groq.com)\n", - "groq_api_key = os.environ.get(\"GROQ_API_KEY\", \"your-api-key-here\")\n", + "# The LLM-as-judge sections below (contradiction detection and synthetic\n", + "# data generation) call a real language model. Set GROQ_API_KEY to run them\n", + "# against Groq (free key at https://console.groq.com); without a key those\n", + "# cells are skipped so the rest of the notebook still runs offline.\n", + "groq_api_key = os.environ.get(\"GROQ_API_KEY\", \"\")\n", "\n", - "if groq_api_key == \"your-api-key-here\":\n", - " print(\"Please set GROQ_API_KEY environment variable or edit this cell with your key\")\n", - "else:\n", + "if groq_api_key:\n", " llm = Groq(\n", " api_key=groq_api_key,\n", " model=\"llama-3.3-70b-versatile\",\n", " temperature=0.0,\n", " max_tokens=1024,\n", " )\n", - " print(f\"LLM initialized: {llm.model}\")" + " USING_REAL_LLM = True\n", + " print(f\"Using Groq LLM: {llm.model}\")\n", + "else:\n", + " llm = None\n", + " USING_REAL_LLM = False\n", + " print(\"No GROQ_API_KEY set — LLM-as-judge sections will be skipped.\")\n", + " print(\"Set GROQ_API_KEY (https://console.groq.com) to enable them.\")" ] }, { @@ -299,15 +306,18 @@ "source": [ "from openagent_eval.corpus.contradiction import ContradictionDetector\n", "\n", - "detector = ContradictionDetector(\n", - " llm_provider=llm,\n", - " max_pairs=10,\n", - ")\n", - "report = await detector.analyze(documents)\n", + "if USING_REAL_LLM:\n", + " detector = ContradictionDetector(\n", + " llm_provider=llm,\n", + " max_pairs=10,\n", + " )\n", + " report = await detector.analyze(documents)\n", "\n", - "print(f\"Health score: {report.health_score:.2f}\")\n", - "print(f\"Issues found: {len(report.issues)}\")\n", - "print(f\"Summary: {report.summary}\")" + " print(f\"Health score: {report.health_score:.2f}\")\n", + " print(f\"Issues found: {len(report.issues)}\")\n", + " print(f\"Summary: {report.summary}\")\n", + "else:\n", + " print(\"Skipped — contradiction detection needs a real LLM. Set GROQ_API_KEY to run.\")" ] }, { @@ -444,22 +454,23 @@ "source": [ "from openagent_eval.synthesis import QuestionGenerator\n", "\n", - "q_gen = QuestionGenerator(llm_provider=llm)\n", - "\n", "context = \"RAG (Retrieval-Augmented Generation) combines a retrieval step with a generative model to ground answers in retrieved documents.\"\n", "\n", - "test_cases = await q_gen.generate(\n", - " context=context,\n", - " count=3,\n", - " source_document=\"rag1\",\n", - " chunk_index=0,\n", - ")\n", - "\n", - "print(f\"Generated {len(test_cases)} test cases:\")\n", - "for i, tc in enumerate(test_cases, 1):\n", - " print(f\"{i}. [{tc.test_type.value}] Q: {tc.question}\")\n", - " print(f\" A: {tc.ground_truth}\")\n", - " print()" + "if USING_REAL_LLM:\n", + " q_gen = QuestionGenerator(llm_provider=llm)\n", + " test_cases = await q_gen.generate(\n", + " context=context,\n", + " count=3,\n", + " source_document=\"rag1\",\n", + " chunk_index=0,\n", + " )\n", + " print(f\"Generated {len(test_cases)} test cases:\")\n", + " for i, tc in enumerate(test_cases, 1):\n", + " print(f\"{i}. [{tc.test_type.value}] Q: {tc.question}\")\n", + " print(f\" A: {tc.ground_truth}\")\n", + " print()\n", + "else:\n", + " print(\"Skipped — synthesis needs a real LLM to return structured JSON. Set GROQ_API_KEY to run.\")" ] }, { @@ -480,22 +491,24 @@ "from openagent_eval.synthesis import AdversarialTestCaseGenerator\n", "from openagent_eval.synthesis.models import TestCaseType\n", "\n", - "adv_gen = AdversarialTestCaseGenerator(llm_provider=llm)\n", - "\n", "context = \"A vector database stores high-dimensional embeddings and supports similarity search via cosine or dot-product distance.\"\n", "\n", - "for adv_type in [TestCaseType.UNANSWERABLE, TestCaseType.MISLEADING, TestCaseType.COUNTERFACTUAL]:\n", - " cases = await adv_gen.generate(\n", - " context=context,\n", - " test_type=adv_type,\n", - " count=1,\n", - " source_document=\"vec1\",\n", - " )\n", - " if cases:\n", - " tc = cases[0]\n", - " print(f\"[{tc.test_type.value}] Q: {tc.question}\")\n", - " print(f\" A: {tc.ground_truth}\")\n", - " print()" + "if USING_REAL_LLM:\n", + " adv_gen = AdversarialTestCaseGenerator(llm_provider=llm)\n", + " for adv_type in [TestCaseType.UNANSWERABLE, TestCaseType.MISLEADING, TestCaseType.COUNTERFACTUAL]:\n", + " cases = await adv_gen.generate(\n", + " context=context,\n", + " test_type=adv_type,\n", + " count=1,\n", + " source_document=\"vec1\",\n", + " )\n", + " if cases:\n", + " tc = cases[0]\n", + " print(f\"[{tc.test_type.value}] Q: {tc.question}\")\n", + " print(f\" A: {tc.ground_truth}\")\n", + " print()\n", + "else:\n", + " print(\"Skipped — adversarial synthesis needs a real LLM. Set GROQ_API_KEY to run.\")" ] }, { @@ -515,34 +528,35 @@ "source": [ "from openagent_eval.synthesis import SyntheticDataGenerator\n", "\n", - "generator = SyntheticDataGenerator(\n", - " llm_provider=llm,\n", - " chunk_size=2000,\n", - " chunk_overlap=200,\n", - " max_concurrent=3,\n", - ")\n", - "\n", "text = (\n", " \"Python is a high-level programming language known for its readability. \"\n", " \"It supports multiple paradigms including object-oriented, functional, \"\n", " \"and procedural programming.\"\n", ")\n", "\n", - "dataset = await generator.generate_from_text(\n", - " text=text,\n", - " count=5,\n", - " adversarial=True,\n", - " adversarial_count_per_type=1,\n", - " source_name=\"python_intro\",\n", - ")\n", - "\n", - "print(f\"Total test cases: {dataset.total_count}\")\n", - "print(f\"Type breakdown: {dataset.type_counts}\")\n", - "print()\n", - "for i, tc in enumerate(dataset.test_cases, 1):\n", - " print(f\"{i}. [{tc.test_type.value}] Q: {tc.question[:80]}\")\n", - " print(f\" A: {tc.ground_truth[:80]}\")\n", - " print()" + "if USING_REAL_LLM:\n", + " generator = SyntheticDataGenerator(\n", + " llm_provider=llm,\n", + " chunk_size=2000,\n", + " chunk_overlap=200,\n", + " max_concurrent=3,\n", + " )\n", + " dataset = await generator.generate_from_text(\n", + " text=text,\n", + " count=5,\n", + " adversarial=True,\n", + " adversarial_count_per_type=1,\n", + " source_name=\"python_intro\",\n", + " )\n", + " print(f\"Total test cases: {dataset.total_count}\")\n", + " print(f\"Type breakdown: {dataset.type_counts}\")\n", + " print()\n", + " for i, tc in enumerate(dataset.test_cases, 1):\n", + " print(f\"{i}. [{tc.test_type.value}] Q: {tc.question[:80]}\")\n", + " print(f\" A: {tc.ground_truth[:80]}\")\n", + " print()\n", + "else:\n", + " print(\"Skipped — full synthesis pipeline needs a real LLM. Set GROQ_API_KEY to run.\")" ] }, { @@ -577,7 +591,7 @@ "\n", "---\n", "\n", - "*Generated by OpenAgent Eval v0.4.4*" + "*Generated by OpenAgent Eval v0.4.8*" ] } ], @@ -595,8 +609,9 @@ "file_extension": ".py", "mimetype": "text/x-python", "name": "python", + "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.12.13" + "version": "3.13.14" } }, "nbformat": 4, diff --git a/examples/openagent_eval_colab_tutorial.ipynb b/examples/openagent_eval_colab_tutorial.ipynb new file mode 100644 index 0000000..2bc4290 --- /dev/null +++ b/examples/openagent_eval_colab_tutorial.ipynb @@ -0,0 +1,2163 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "30eaedc4", + "metadata": {}, + "source": [ + "# 🚀 OpenAgent Eval — Complete Colab Tutorial\n", + "\n", + "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/OpenAgentHQ/openagent-eval/blob/main/examples/openagent_eval_colab_tutorial.ipynb)\n", + "[![PyPI Version](https://img.shields.io/pypi/v/openagent-eval?color=blue&logo=pypi&logoColor=white)](https://pypi.org/project/openagent-eval/)\n", + "[![GitHub Stars](https://img.shields.io/github/stars/OpenAgentHQ/openagent-eval?style=social)](https://github.com/OpenAgentHQ/openagent-eval/stargazers)\n", + "[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://github.com/OpenAgentHQ/openagent-eval/blob/main/LICENSE)\n", + "\n", + "**The open-source evaluation framework for RAG systems and AI Agents — learn it end-to-end, right in your browser.**\n", + "\n", + "---\n", + "\n", + "## 👋 Welcome\n", + "\n", + "[OpenAgent Eval](https://github.com/OpenAgentHQ/openagent-eval) brings **pytest-level simplicity** to\n", + "AI evaluation. It measures how well a Retrieval-Augmented Generation (RAG) system retrieves the right\n", + "context and generates faithful, relevant answers — with 18+ metrics, corpus auditing, failure\n", + "diagnosis, and synthetic test-data generation.\n", + "\n", + "This notebook is a **zero-setup, click-*Run all* tutorial**. It runs **completely offline with the\n", + "built-in `mock` providers** — so **you do not need any API keys** to complete every core section.\n", + "When you are ready to point it at a real LLM, the clearly-marked **Optional** cells show you how.\n", + "\n", + "## ⏱️ What to expect\n", + "- **Estimated time:** ~30 minutes\n", + "- **Prerequisites:** none — just a Google account (or any Jupyter runtime)\n", + "- **Cost:** free (no paid Colab features, no API keys required for the core walkthrough)\n", + "- **Runtime:** the whole notebook finishes in well under a minute of compute\n", + "\n", + "## 🗺️ Table of contents\n", + "1. [Installation & environment setup](#sec1)\n", + "2. [Terminal basics for Colab users](#sec2)\n", + "3. [The `oaeval` CLI at a glance](#sec3)\n", + "4. [Initialize your first configuration](#sec4)\n", + "5. [Prepare sample data](#sec5)\n", + "6. [Run your first evaluation — *no API key needed*](#sec6)\n", + "7. [Corpus health audit](#sec7)\n", + "8. [Failure diagnosis (blame attribution)](#sec8)\n", + "9. [Synthetic test-data generation](#sec9)\n", + "10. [Comparing experiments](#sec10)\n", + "11. [SDK usage (the Python API)](#sec11)\n", + "12. [CI/CD gating](#sec12)\n", + "13. [Advanced — custom metrics](#sec13)\n", + "14. [Tips, tricks & troubleshooting](#sec14)\n", + "15. [Next steps & resources](#sec15)\n", + "16. [Feedback & credit](#sec16)\n", + "17. [Optional — using real API keys](#sec17)\n" + ] + }, + { + "cell_type": "markdown", + "id": "1d3d9f30", + "metadata": {}, + "source": [ + "\n", + "## 1. Installation & environment setup\n", + "\n", + "Install OpenAgent Eval straight from PyPI. In Colab/Jupyter the `%pip` magic installs into the\n", + "kernel that is actually running this notebook, which is exactly what we want.\n", + "\n", + "> **Why also `pytest`?** In release `0.4.8` the `oaeval` command-line tool imports `pytest` at\n", + "> start-up (it powers the `oaeval test` CI/CD command). Installing it alongside keeps every CLI\n", + "> command working. It is a tiny, pure-Python package, so the install stays fast." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "649cb2f4", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-23T20:01:55.134708Z", + "iopub.status.busy": "2026-07-23T20:01:55.134422Z", + "iopub.status.idle": "2026-07-23T20:01:56.360961Z", + "shell.execute_reply": "2026-07-23T20:01:56.359844Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Note: you may need to restart the kernel to use updated packages.\n" + ] + } + ], + "source": [ + "%pip install -q openagent-eval pytest" + ] + }, + { + "cell_type": "markdown", + "id": "830ac0cc", + "metadata": {}, + "source": [ + "Keep the tool's output tidy by turning its debug logging down to warnings. Setting this in\n", + "`os.environ` means it is inherited by every `!` shell command we run later, too." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "4094f75f", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-23T20:01:56.387255Z", + "iopub.status.busy": "2026-07-23T20:01:56.386914Z", + "iopub.status.idle": "2026-07-23T20:01:56.398685Z", + "shell.execute_reply": "2026-07-23T20:01:56.397872Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Environment ready.\n" + ] + } + ], + "source": [ + "import os\n", + "\n", + "# Quiet the library's debug logs so the tutorial output stays readable.\n", + "os.environ[\"LOGURU_LEVEL\"] = \"WARNING\"\n", + "\n", + "print(\"Environment ready.\")" + ] + }, + { + "cell_type": "markdown", + "id": "ef8d7685", + "metadata": {}, + "source": [ + "Confirm the version and run the built-in environment doctor. `oaeval doctor` checks your Python\n", + "version, the installed dependencies, and which provider API keys are visible in the environment\n", + "(all *Not set* here — and that is fine, we will use the offline `mock` provider)." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "4afe121b", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-23T20:01:56.412315Z", + "iopub.status.busy": "2026-07-23T20:01:56.411653Z", + "iopub.status.idle": "2026-07-23T20:01:57.354526Z", + "shell.execute_reply": "2026-07-23T20:01:57.353580Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "openagent-eval 0.4.8\r\n" + ] + } + ], + "source": [ + "!oaeval --version" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "cb265eef", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-23T20:01:57.359265Z", + "iopub.status.busy": "2026-07-23T20:01:57.358951Z", + "iopub.status.idle": "2026-07-23T20:01:58.488047Z", + "shell.execute_reply": "2026-07-23T20:01:58.485058Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\u001b[1;34mOpenAgent Eval\u001b[0m - Environment Check\r\n", + "\r\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\u001b[3m Environment Status \u001b[0m\r\n", + "┏━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━┓\r\n", + "┃\u001b[1m \u001b[0m\u001b[1mComponent \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mStatus\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mDetails \u001b[0m\u001b[1m \u001b[0m┃\r\n", + "┡━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━┩\r\n", + "│\u001b[36m \u001b[0m\u001b[36mPython \u001b[0m\u001b[36m \u001b[0m│\u001b[1m \u001b[0m\u001b[1;32mOK\u001b[0m\u001b[1m \u001b[0m\u001b[1m \u001b[0m│\u001b[2m \u001b[0m\u001b[2mv3.13.14 \u001b[0m\u001b[2m \u001b[0m│\r\n", + "│\u001b[36m \u001b[0m\u001b[36mopenagent-eval\u001b[0m\u001b[36m \u001b[0m│\u001b[1m \u001b[0m\u001b[1;32mOK\u001b[0m\u001b[1m \u001b[0m\u001b[1m \u001b[0m│\u001b[2m \u001b[0m\u001b[2mv0.4.8 \u001b[0m\u001b[2m \u001b[0m│\r\n", + "│\u001b[36m \u001b[0m\u001b[36mtyper \u001b[0m\u001b[36m \u001b[0m│\u001b[1m \u001b[0m\u001b[1;32mOK\u001b[0m\u001b[1m \u001b[0m\u001b[1m \u001b[0m│\u001b[2m \u001b[0m\u001b[2mCLI framework \u001b[0m\u001b[2m \u001b[0m│\r\n", + "│\u001b[36m \u001b[0m\u001b[36mrich \u001b[0m\u001b[36m \u001b[0m│\u001b[1m \u001b[0m\u001b[1;32mOK\u001b[0m\u001b[1m \u001b[0m\u001b[1m \u001b[0m│\u001b[2m \u001b[0m\u001b[2mTerminal UI \u001b[0m\u001b[2m \u001b[0m│\r\n", + "│\u001b[36m \u001b[0m\u001b[36mpydantic \u001b[0m\u001b[36m \u001b[0m│\u001b[1m \u001b[0m\u001b[1;32mOK\u001b[0m\u001b[1m \u001b[0m\u001b[1m \u001b[0m│\u001b[2m \u001b[0m\u001b[2mData validation\u001b[0m\u001b[2m \u001b[0m│\r\n", + "│\u001b[36m \u001b[0m\u001b[36myaml \u001b[0m\u001b[36m \u001b[0m│\u001b[1m \u001b[0m\u001b[1;32mOK\u001b[0m\u001b[1m \u001b[0m\u001b[1m \u001b[0m│\u001b[2m \u001b[0m\u001b[2mConfiguration \u001b[0m\u001b[2m \u001b[0m│\r\n", + "│\u001b[36m \u001b[0m\u001b[36mloguru \u001b[0m\u001b[36m \u001b[0m│\u001b[1m \u001b[0m\u001b[1;32mOK\u001b[0m\u001b[1m \u001b[0m\u001b[1m \u001b[0m│\u001b[2m \u001b[0m\u001b[2mLogging \u001b[0m\u001b[2m \u001b[0m│\r\n", + "│\u001b[36m \u001b[0m\u001b[36mjinja2 \u001b[0m\u001b[36m \u001b[0m│\u001b[1m \u001b[0m\u001b[1;32mOK\u001b[0m\u001b[1m \u001b[0m\u001b[1m \u001b[0m│\u001b[2m \u001b[0m\u001b[2mHTML templates \u001b[0m\u001b[2m \u001b[0m│\r\n", + "│\u001b[36m \u001b[0m\u001b[36mhttpx \u001b[0m\u001b[36m \u001b[0m│\u001b[1m \u001b[0m\u001b[1;32mOK\u001b[0m\u001b[1m \u001b[0m\u001b[1m \u001b[0m│\u001b[2m \u001b[0m\u001b[2mHTTP client \u001b[0m\u001b[2m \u001b[0m│\r\n", + "└────────────────┴────────┴─────────────────┘\r\n", + "\u001b[3m API Key Availability \u001b[0m\r\n", + "┏━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┓\r\n", + "┃\u001b[1m \u001b[0m\u001b[1mProvider \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mEnvironment Variable\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mStatus \u001b[0m\u001b[1m \u001b[0m┃\r\n", + "┡━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━┩\r\n", + "│\u001b[36m \u001b[0m\u001b[36mOpenAI \u001b[0m\u001b[36m \u001b[0m│\u001b[33m \u001b[0m\u001b[33mOPENAI_API_KEY \u001b[0m\u001b[33m \u001b[0m│\u001b[1m \u001b[0m\u001b[1;2mNot set\u001b[0m\u001b[1m \u001b[0m│\r\n", + "│\u001b[36m \u001b[0m\u001b[36mGemini \u001b[0m\u001b[36m \u001b[0m│\u001b[33m \u001b[0m\u001b[33mGEMINI_API_KEY \u001b[0m\u001b[33m \u001b[0m│\u001b[1m \u001b[0m\u001b[1;2mNot set\u001b[0m\u001b[1m \u001b[0m│\r\n", + "│\u001b[36m \u001b[0m\u001b[36mAnthropic \u001b[0m\u001b[36m \u001b[0m│\u001b[33m \u001b[0m\u001b[33mANTHROPIC_API_KEY \u001b[0m\u001b[33m \u001b[0m│\u001b[1m \u001b[0m\u001b[1;2mNot set\u001b[0m\u001b[1m \u001b[0m│\r\n", + "│\u001b[36m \u001b[0m\u001b[36mGroq \u001b[0m\u001b[36m \u001b[0m│\u001b[33m \u001b[0m\u001b[33mGROQ_API_KEY \u001b[0m\u001b[33m \u001b[0m│\u001b[1m \u001b[0m\u001b[1;2mNot set\u001b[0m\u001b[1m \u001b[0m│\r\n", + "│\u001b[36m \u001b[0m\u001b[36mOpenRouter\u001b[0m\u001b[36m \u001b[0m│\u001b[33m \u001b[0m\u001b[33mOPENROUTER_API_KEY \u001b[0m\u001b[33m \u001b[0m│\u001b[1m \u001b[0m\u001b[1;2mNot set\u001b[0m\u001b[1m \u001b[0m│\r\n", + "└────────────┴──────────────────────┴─────────┘\r\n", + "\r\n", + "\u001b[1mConfiguration:\u001b[0m\r\n", + " \u001b[2mNo config file in current directory\u001b[0m\r\n", + " \u001b[2mRun \u001b[0m\u001b[2;32m'oaeval init'\u001b[0m\u001b[2m to create one\u001b[0m\r\n", + "\r\n", + "\u001b[1mSummary:\u001b[0m\r\n", + "\u001b[32mOK\u001b[0m Python version is compatible\r\n", + "\u001b[33mWARNING\u001b[0m No API keys configured\r\n", + "\r\n", + "\u001b[1mRecommendations:\u001b[0m\r\n", + " \u001b[33m- Set at least one API key \u001b[0m\u001b[1;33m(\u001b[0m\u001b[33me.g., OPENAI_API_KEY\u001b[0m\u001b[1;33m)\u001b[0m\r\n", + " \u001b[33m- Run \u001b[0m\u001b[33m'oaeval doctor --check-api'\u001b[0m\u001b[33m to test connectivity\u001b[0m\r\n" + ] + } + ], + "source": [ + "!oaeval doctor" + ] + }, + { + "cell_type": "markdown", + "id": "0dd9b582", + "metadata": {}, + "source": [ + "\n", + "## 2. Terminal basics for Colab users\n", + "\n", + "Notebook-first? No problem. A **notebook cell can also run terminal (shell) commands** — you just\n", + "prefix the line with `!`. This section is a quick primer so the rest of the tutorial feels natural.\n", + "\n", + "- A **plain cell** runs **Python**.\n", + "- A cell line starting with **`!`** runs a **shell command** (like a terminal).\n", + "- A cell starting with **`%%writefile `** *writes the rest of the cell to a file*.\n", + "- A line starting with **`%`** is a **magic** command (notebook helper), e.g. `%pip`.\n", + "\n", + "Let's try the basics." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "e4aa9f01", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-23T20:01:58.501697Z", + "iopub.status.busy": "2026-07-23T20:01:58.501335Z", + "iopub.status.idle": "2026-07-23T20:01:58.834231Z", + "shell.execute_reply": "2026-07-23T20:01:58.831897Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "/content\r\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "total 108\r\n", + "drwxr-xr-x 2 root root 4096 Jul 23 22:01 .\r\n", + "drwxr-xr-x 7 root root 4096 Jul 23 22:01 ..\r\n", + "-rw-r--r-- 1 root root 100013 Jul 23 22:01 nb.ipynb\r\n" + ] + } + ], + "source": [ + "!pwd # print the current working directory\n", + "!ls -la # list files here (there may be none yet)" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "b9bc6ba1", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-23T20:01:58.837932Z", + "iopub.status.busy": "2026-07-23T20:01:58.837569Z", + "iopub.status.idle": "2026-07-23T20:01:59.116245Z", + "shell.execute_reply": "2026-07-23T20:01:59.110099Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "total 8\r\n", + "drwxr-xr-x 2 root root 4096 Jul 23 22:01 .\r\n", + "drwxr-xr-x 3 root root 4096 Jul 23 22:01 ..\r\n" + ] + } + ], + "source": [ + "!mkdir -p my_project\n", + "!ls -la my_project # a freshly created (empty) directory" + ] + }, + { + "cell_type": "markdown", + "id": "206eaae3", + "metadata": {}, + "source": [ + "### Writing and reading files\n", + "\n", + "There is no `nano`/`vim` text editor in a notebook, but you don't need one. Use the `%%writefile`\n", + "magic to create a file, then `!cat` (or Python) to read it back." + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "2df43599", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-23T20:01:59.122683Z", + "iopub.status.busy": "2026-07-23T20:01:59.122190Z", + "iopub.status.idle": "2026-07-23T20:01:59.152005Z", + "shell.execute_reply": "2026-07-23T20:01:59.142241Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Writing my_project/hello.txt\n" + ] + } + ], + "source": [ + "%%writefile my_project/hello.txt\n", + "Hello from OpenAgent Eval!\n", + "This file was written straight from a notebook cell." + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "6b73c6b0", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-23T20:01:59.156674Z", + "iopub.status.busy": "2026-07-23T20:01:59.155680Z", + "iopub.status.idle": "2026-07-23T20:01:59.499353Z", + "shell.execute_reply": "2026-07-23T20:01:59.495121Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Hello from OpenAgent Eval!\r\n", + "This file was written straight from a notebook cell.\r\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "---\n", + "Hello from OpenAgent Eval!\r\n" + ] + } + ], + "source": [ + "!cat my_project/hello.txt # view the whole file\n", + "print(\"---\")\n", + "!head -1 my_project/hello.txt # view just the first line" + ] + }, + { + "cell_type": "markdown", + "id": "90955cc0", + "metadata": {}, + "source": [ + "> **Saving files locally:** in Colab you can download any file from the file browser (📁 icon in\n", + "> the left sidebar) → right-click → *Download*. To keep files between sessions, mount Google Drive\n", + "> (shown in the [troubleshooting section](#sec14))." + ] + }, + { + "cell_type": "markdown", + "id": "cc769d72", + "metadata": {}, + "source": [ + "\n", + "## 3. The `oaeval` CLI at a glance\n", + "\n", + "Everything in OpenAgent Eval is available through the `oaeval` command. Here is the full command map:\n", + "\n", + "| Command | What it does |\n", + "|---|---|\n", + "| `oaeval init` | Create a configuration file (wizard or defaults) |\n", + "| `oaeval validate ` | Validate a config without running |\n", + "| `oaeval run ` | Run the evaluation pipeline |\n", + "| `oaeval report ` | View an evaluation report (`latest` for the newest) |\n", + "| `oaeval list` | List previous evaluation runs |\n", + "| `oaeval compare ` | Compare two experiments side by side |\n", + "| `oaeval audit ` | Audit corpus health before you wire up RAG |\n", + "| `oaeval diagnose ` | Attribute failures to retrieval / generation / chunking |\n", + "| `oaeval synth` | Generate synthetic test cases from a corpus or text |\n", + "| `oaeval test -t ...` | Threshold-gated run for CI/CD |\n", + "| `oaeval doctor` | Environment & dependency check |\n", + "\n", + "Any command's options are one `--help` away:" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "2fcb2e03", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-23T20:01:59.511704Z", + "iopub.status.busy": "2026-07-23T20:01:59.511297Z", + "iopub.status.idle": "2026-07-23T20:02:00.587562Z", + "shell.execute_reply": "2026-07-23T20:02:00.579678Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\u001b[1m \u001b[0m\r\n", + "\u001b[1m \u001b[0m\u001b[1;33mUsage: \u001b[0m\u001b[1moaeval [OPTIONS] COMMAND [ARGS]...\u001b[0m\u001b[1m \u001b[0m\u001b[1m \u001b[0m\r\n", + "\u001b[1m \u001b[0m\r\n", + " Open-source CLI framework for evaluating RAG systems and AI Agents. \r\n", + " \r\n", + "\u001b[2m╭─\u001b[0m\u001b[2m Options \u001b[0m\u001b[2m───────────────────────────────────────────────────────────────────\u001b[0m\u001b[2m─╮\u001b[0m\r\n", + "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-version\u001b[0m \u001b[1;32m-V\u001b[0m Show the application's version and exit. \u001b[2m│\u001b[0m\r\n", + "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-quiet\u001b[0m \u001b[1;32m-q\u001b[0m Suppress non-essential output. \u001b[2m│\u001b[0m\r\n", + "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-json\u001b[0m Output machine-readable JSON. \u001b[2m│\u001b[0m\r\n", + "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-no\u001b[0m\u001b[1;36m-color\u001b[0m Disable color output. \u001b[2m│\u001b[0m\r\n", + "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-verbose\u001b[0m \u001b[1;32m-v\u001b[0m Enable verbose output. \u001b[2m│\u001b[0m\r\n", + "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-install\u001b[0m\u001b[1;36m-completion\u001b[0m Install completion for the current shell. \u001b[2m│\u001b[0m\r\n", + "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-show\u001b[0m\u001b[1;36m-completion\u001b[0m Show completion for the current shell, to \u001b[2m│\u001b[0m\r\n", + "\u001b[2m│\u001b[0m copy it or customize the installation. \u001b[2m│\u001b[0m\r\n", + "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-help\u001b[0m Show this message and exit. \u001b[2m│\u001b[0m\r\n", + "\u001b[2m╰──────────────────────────────────────────────────────────────────────────────╯\u001b[0m\r\n", + "\u001b[2m╭─\u001b[0m\u001b[2m Commands \u001b[0m\u001b[2m──────────────────────────────────────────────────────────────────\u001b[0m\u001b[2m─╮\u001b[0m\r\n", + "\u001b[2m│\u001b[0m \u001b[1;36minit \u001b[0m\u001b[1;36m \u001b[0m Create a new evaluation configuration file interactively or \u001b[2m│\u001b[0m\r\n", + "\u001b[2m│\u001b[0m \u001b[1;36m \u001b[0m using defaults. \u001b[2m│\u001b[0m\r\n", + "\u001b[2m│\u001b[0m \u001b[1;36mrun \u001b[0m\u001b[1;36m \u001b[0m Run evaluation pipeline with the specified configuration. \u001b[2m│\u001b[0m\r\n", + "\u001b[2m│\u001b[0m \u001b[1;36mreport \u001b[0m\u001b[1;36m \u001b[0m View and display evaluation reports in various formats. \u001b[2m│\u001b[0m\r\n", + "\u001b[2m│\u001b[0m \u001b[1;36mcompare \u001b[0m\u001b[1;36m \u001b[0m Compare two evaluation experiments side by side. \u001b[2m│\u001b[0m\r\n", + "\u001b[2m│\u001b[0m \u001b[1;36mlist \u001b[0m\u001b[1;36m \u001b[0m List previous evaluation runs with filtering and sorting \u001b[2m│\u001b[0m\r\n", + "\u001b[2m│\u001b[0m \u001b[1;36m \u001b[0m options. \u001b[2m│\u001b[0m\r\n", + "\u001b[2m│\u001b[0m \u001b[1;36mdoctor \u001b[0m\u001b[1;36m \u001b[0m Check environment, dependencies, and API connectivity for \u001b[2m│\u001b[0m\r\n", + "\u001b[2m│\u001b[0m \u001b[1;36m \u001b[0m OpenAgent Eval. \u001b[2m│\u001b[0m\r\n", + "\u001b[2m│\u001b[0m \u001b[1;36mvalidate \u001b[0m\u001b[1;36m \u001b[0m Validate configuration file syntax, schema, datasets, and \u001b[2m│\u001b[0m\r\n", + "\u001b[2m│\u001b[0m \u001b[1;36m \u001b[0m providers. \u001b[2m│\u001b[0m\r\n", + "\u001b[2m│\u001b[0m \u001b[1;36mdelete \u001b[0m\u001b[1;36m \u001b[0m Delete a specific evaluation report or all reports. \u001b[2m│\u001b[0m\r\n", + "\u001b[2m│\u001b[0m \u001b[1;36mdiagnose \u001b[0m\u001b[1;36m \u001b[0m Diagnose evaluation failures and attribute blame to specific \u001b[2m│\u001b[0m\r\n", + "\u001b[2m│\u001b[0m \u001b[1;36m \u001b[0m components. \u001b[2m│\u001b[0m\r\n", + "\u001b[2m│\u001b[0m \u001b[1;36maudit \u001b[0m\u001b[1;36m \u001b[0m Audit the health of a RAG document corpus for quality and \u001b[2m│\u001b[0m\r\n", + "\u001b[2m│\u001b[0m \u001b[1;36m \u001b[0m coherence. \u001b[2m│\u001b[0m\r\n", + "\u001b[2m│\u001b[0m \u001b[1;36msynth \u001b[0m\u001b[1;36m \u001b[0m Generate synthetic evaluation datasets from a corpus or inline \u001b[2m│\u001b[0m\r\n", + "\u001b[2m│\u001b[0m \u001b[1;36m \u001b[0m text. \u001b[2m│\u001b[0m\r\n", + "\u001b[2m│\u001b[0m \u001b[1;36mtest \u001b[0m\u001b[1;36m \u001b[0m Run evaluation as a CI/CD test with threshold gating. \u001b[2m│\u001b[0m\r\n", + "\u001b[2m│\u001b[0m \u001b[1;36mcompletion\u001b[0m\u001b[1;36m \u001b[0m Generate shell completion script for oaeval. \u001b[2m│\u001b[0m\r\n", + "\u001b[2m╰──────────────────────────────────────────────────────────────────────────────╯\u001b[0m\r\n", + "\r\n" + ] + } + ], + "source": [ + "!oaeval --help" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "id": "bf24e160", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-23T20:02:00.616302Z", + "iopub.status.busy": "2026-07-23T20:02:00.608790Z", + "iopub.status.idle": "2026-07-23T20:02:01.886984Z", + "shell.execute_reply": "2026-07-23T20:02:01.884701Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\u001b[1m \u001b[0m\r\n", + "\u001b[1m \u001b[0m\u001b[1;33mUsage: \u001b[0m\u001b[1moaeval run [OPTIONS] [config_path]\u001b[0m\u001b[1m \u001b[0m\u001b[1m \u001b[0m\r\n", + "\u001b[1m \u001b[0m\r\n", + " Run evaluation pipeline with the specified configuration. \r\n", + " \r\n", + " \u001b[2mArgs:\u001b[0m \r\n", + " \u001b[2m config_path (str | None): Path to the YAML configuration file. If not \u001b[0m \r\n", + " \u001b[2mprovided,\u001b[0m \r\n", + " \u001b[2m auto-discovery will search for standard config filenames in the \u001b[0m \r\n", + " \u001b[2mcurrent directory.\u001b[0m \r\n", + " \u001b[2m Defaults to None.\u001b[0m \r\n", + " \u001b[2m output (str): Override the output report format (terminal, markdown, html,\u001b[0m \r\n", + " \u001b[2mjson).\u001b[0m \r\n", + " \u001b[2m Defaults to None.\u001b[0m \r\n", + " \u001b[2m verbose (bool): Enable verbose output, showing the evaluation plan and \u001b[0m \r\n", + " \u001b[2mdetailed progress.\u001b[0m \r\n", + " \u001b[2m Defaults to False.\u001b[0m \r\n", + " \u001b[2m dry_run (bool): Validate the configuration and print the evaluation plan \u001b[0m \r\n", + " \u001b[2mwithout\u001b[0m \r\n", + " \u001b[2m executing the pipeline. Defaults to False.\u001b[0m \r\n", + " \u001b[2m metrics (str | None): A comma-separated list of metrics to execute, \u001b[0m \r\n", + " \u001b[2moverriding\u001b[0m \r\n", + " \u001b[2m the configuration settings. Defaults to None.\u001b[0m \r\n", + " \r\n", + " \u001b[2mReturns:\u001b[0m \r\n", + " \u001b[2m None. Generates evaluation reports and saves them to the configured \u001b[0m \r\n", + " \u001b[2mdirectory.\u001b[0m \r\n", + " \u001b[2m Raises typer.Exit(code=2) if configuration loading or validation fails.\u001b[0m \r\n", + " \r\n", + " \u001b[2mExample:\u001b[0m \r\n", + " \u001b[2m $ oaeval run config.yaml \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-output\u001b[0m\u001b[2m html \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-verbose\u001b[0m \r\n", + " \r\n", + "\u001b[2m╭─\u001b[0m\u001b[2m Arguments \u001b[0m\u001b[2m─────────────────────────────────────────────────────────────────\u001b[0m\u001b[2m─╮\u001b[0m\r\n", + "\u001b[2m│\u001b[0m config_path \u001b[1;2;33m<\u001b[0m\u001b[1;33mstr\u001b[0m\u001b[1;2;33m>\u001b[0m Path to configuration file. Auto-discovered if not \u001b[2m│\u001b[0m\r\n", + "\u001b[2m│\u001b[0m provided. \u001b[2m│\u001b[0m\r\n", + "\u001b[2m╰──────────────────────────────────────────────────────────────────────────────╯\u001b[0m\r\n", + "\u001b[2m╭─\u001b[0m\u001b[2m Options \u001b[0m\u001b[2m───────────────────────────────────────────────────────────────────\u001b[0m\u001b[2m─╮\u001b[0m\r\n", + "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-output\u001b[0m \u001b[1;32m-o\u001b[0m \u001b[1;2;33m<\u001b[0m\u001b[1;33mstr\u001b[0m\u001b[1;2;33m>\u001b[0m Override output format (terminal, markdown, html, \u001b[2m│\u001b[0m\r\n", + "\u001b[2m│\u001b[0m json). \u001b[2m│\u001b[0m\r\n", + "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-verbose\u001b[0m \u001b[1;32m-v\u001b[0m \u001b[1;33m \u001b[0m Enable verbose output. \u001b[2m│\u001b[0m\r\n", + "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-dry\u001b[0m\u001b[1;36m-run\u001b[0m \u001b[1;33m \u001b[0m Validate config and show evaluation plan without \u001b[2m│\u001b[0m\r\n", + "\u001b[2m│\u001b[0m running. \u001b[2m│\u001b[0m\r\n", + "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-metrics\u001b[0m \u001b[1;32m-m\u001b[0m \u001b[1;2;33m<\u001b[0m\u001b[1;33mstr\u001b[0m\u001b[1;2;33m>\u001b[0m Comma-separated list of metrics to run (overrides \u001b[2m│\u001b[0m\r\n", + "\u001b[2m│\u001b[0m config). \u001b[2m│\u001b[0m\r\n", + "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-help\u001b[0m \u001b[1;33m \u001b[0m Show this message and exit. \u001b[2m│\u001b[0m\r\n", + "\u001b[2m╰──────────────────────────────────────────────────────────────────────────────╯\u001b[0m\r\n", + "\r\n" + ] + } + ], + "source": [ + "!oaeval run --help" + ] + }, + { + "cell_type": "markdown", + "id": "4e6cd285", + "metadata": {}, + "source": [ + "\n", + "## 4. Initialize your first configuration\n", + "\n", + "`oaeval init` scaffolds a `config.yaml`. Its default is an **interactive wizard**, which would block\n", + "a *Run all*, so we pass `--no-interactive` to write sensible defaults instead. (When you work locally\n", + "in a real terminal, `oaeval init --interactive` gives you the friendly wizard.)" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "id": "ed28bb07", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-23T20:02:01.893064Z", + "iopub.status.busy": "2026-07-23T20:02:01.891148Z", + "iopub.status.idle": "2026-07-23T20:02:02.988712Z", + "shell.execute_reply": "2026-07-23T20:02:02.986016Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\r\n", + "\u001b[32mOK\u001b[0m Configuration created: config.yaml\r\n", + "\r\n", + "\u001b[33mNext steps:\u001b[0m\r\n", + " \u001b[1;36m1\u001b[0m. Review the configuration file\r\n", + " \u001b[1;36m2\u001b[0m. Run \u001b[1moaeval validate\u001b[0m to check it\r\n", + " \u001b[1;36m3\u001b[0m. Run \u001b[1moaeval run\u001b[0m to start evaluation\r\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "---\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "# OpenAgent Eval Configuration\r\n", + "# See documentation for options: https://github.com/OpenAgentHQ/openagent-eval\r\n", + "#\r\n", + "# For a fully offline dry-run (no API keys / vector store required), set:\r\n", + "# llm.provider: mock\r\n", + "# retriever.provider: mock\r\n", + "\r\n", + "dataset:\r\n", + " path: data/questions.json\r\n", + " # limit: 100\r\n", + "\r\n", + "llm:\r\n", + " provider: openai\r\n", + " model: gpt-4o-mini\r\n", + " temperature: 0.0\r\n", + "\r\n", + "retriever:\r\n", + " provider: chroma\r\n", + " settings:\r\n", + " collection_name: my_collection\r\n", + "\r\n", + "metrics:\r\n", + " retrieval:\r\n", + " - context_precision\r\n", + " - context_recall\r\n", + " - mrr\r\n", + " generation:\r\n", + " - faithfulness\r\n", + " - answer_relevancy\r\n", + " performance:\r\n", + " - latency\r\n", + " cost:\r\n", + " - token_count\r\n", + "\r\n", + "report:\r\n", + " output: terminal\r\n", + " output_dir: ./reports\r\n" + ] + } + ], + "source": [ + "!oaeval init --no-interactive --force\n", + "print(\"---\")\n", + "!cat config.yaml" + ] + }, + { + "cell_type": "markdown", + "id": "7b6b8685", + "metadata": {}, + "source": [ + "The generated file is a good tour of every knob. The key sections are:\n", + "\n", + "- **`dataset`** — where your test cases live and their format.\n", + "- **`llm`** — the model that generates answers (`provider`, `model`, `temperature`).\n", + "- **`retriever`** — the vector store / search backend.\n", + "- **`metrics`** — which retrieval, generation, performance, and cost metrics to compute.\n", + "- **`report`** — output format(s) and where reports are saved.\n", + "\n", + "Notice the header comment even tells you how to go **fully offline** — set `llm.provider: mock` and\n", + "`retriever.provider: mock`. That is exactly what we will do next, so this notebook needs **no API\n", + "keys**." + ] + }, + { + "cell_type": "markdown", + "id": "eb00a991", + "metadata": {}, + "source": [ + "\n", + "## 5. Prepare sample data\n", + "\n", + "A dataset is a JSON list of test cases. Each item can carry:\n", + "\n", + "- `question` — the user query\n", + "- `ground_truth` — the reference (expected) answer\n", + "- `context` — the reference context/passage the answer should be grounded in\n", + "- `metadata` — anything extra you want to keep\n", + "\n", + "Let's write a tiny dataset with `%%writefile`." + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "id": "3bdffbec", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-23T20:02:02.994262Z", + "iopub.status.busy": "2026-07-23T20:02:02.993505Z", + "iopub.status.idle": "2026-07-23T20:02:03.025642Z", + "shell.execute_reply": "2026-07-23T20:02:03.021021Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Writing data.json\n" + ] + } + ], + "source": [ + "%%writefile data.json\n", + "[\n", + " {\n", + " \"question\": \"What is Retrieval-Augmented Generation?\",\n", + " \"ground_truth\": \"RAG combines a retriever and a generator to produce grounded answers.\",\n", + " \"context\": \"Retrieval-Augmented Generation (RAG) pairs a retriever that fetches documents with a generator that writes an answer grounded in them.\",\n", + " \"metadata\": {\"id\": 1, \"topic\": \"rag\"}\n", + " },\n", + " {\n", + " \"question\": \"What is a vector database?\",\n", + " \"ground_truth\": \"A vector database stores embeddings for similarity search.\",\n", + " \"context\": \"Vector databases store high-dimensional embeddings and support fast nearest-neighbour similarity search.\",\n", + " \"metadata\": {\"id\": 2, \"topic\": \"infra\"}\n", + " },\n", + " {\n", + " \"question\": \"Why evaluate a RAG system?\",\n", + " \"ground_truth\": \"To measure retrieval and generation quality and catch regressions.\",\n", + " \"context\": \"Evaluating RAG quantifies retrieval accuracy and answer faithfulness so you can iterate and avoid regressions.\",\n", + " \"metadata\": {\"id\": 3, \"topic\": \"eval\"}\n", + " }\n", + "]" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "id": "a0e2944d", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-23T20:02:03.034016Z", + "iopub.status.busy": "2026-07-23T20:02:03.032093Z", + "iopub.status.idle": "2026-07-23T20:02:03.057508Z", + "shell.execute_reply": "2026-07-23T20:02:03.041145Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Loaded 3 test cases.\n", + "First question: What is Retrieval-Augmented Generation?\n" + ] + } + ], + "source": [ + "import json\n", + "\n", + "cases = json.load(open(\"data.json\"))\n", + "print(f\"Loaded {len(cases)} test cases.\")\n", + "print(\"First question:\", cases[0][\"question\"])" + ] + }, + { + "cell_type": "markdown", + "id": "133c792d", + "metadata": {}, + "source": [ + "\n", + "## 6. Run your first evaluation — *no API key needed* 🔑🚫\n", + "\n", + "This is the **\"No API Key?\"** path the docs promise. We write a config that uses the built-in\n", + "**`mock` LLM** and **`mock` retriever**. These are deterministic, offline stand-ins that let the\n", + "*entire* pipeline run in CI or in Colab with **zero secrets**." + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "id": "3af56eac", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-23T20:02:03.062281Z", + "iopub.status.busy": "2026-07-23T20:02:03.061736Z", + "iopub.status.idle": "2026-07-23T20:02:03.074779Z", + "shell.execute_reply": "2026-07-23T20:02:03.070770Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Overwriting config.yaml\n" + ] + } + ], + "source": [ + "%%writefile config.yaml\n", + "# Fully offline evaluation — no API keys required.\n", + "dataset:\n", + " path: data.json\n", + " format: json\n", + "\n", + "llm:\n", + " provider: mock # deterministic, offline — no network calls\n", + " model: mock-model\n", + " temperature: 0.0\n", + "\n", + "retriever:\n", + " provider: mock # deterministic, offline retriever\n", + " settings:\n", + " collection_name: demo_collection\n", + "\n", + "metrics:\n", + " retrieval:\n", + " - context_precision\n", + " - context_recall\n", + " - mrr\n", + " generation:\n", + " - faithfulness\n", + " - answer_relevancy\n", + " performance:\n", + " - latency\n", + " cost:\n", + " - token_count\n", + "\n", + "report:\n", + " output: terminal\n", + " output_dir: ./reports" + ] + }, + { + "cell_type": "markdown", + "id": "bd29cfd1", + "metadata": {}, + "source": [ + "Validate the config first — this catches typos and missing files before a run." + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "id": "7eff611d", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-23T20:02:03.079870Z", + "iopub.status.busy": "2026-07-23T20:02:03.079527Z", + "iopub.status.idle": "2026-07-23T20:02:04.275442Z", + "shell.execute_reply": "2026-07-23T20:02:04.274557Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\u001b[1;34mOpenAgent Eval\u001b[0m - Configuration Validator\r\n", + "\u001b[2mConfig: config.yaml\u001b[0m\r\n", + "\r\n", + "\u001b[1;36m1\u001b[0m\u001b[1m. Checking YAML syntax\u001b[0m\u001b[1;33m...\u001b[0m\r\n", + " \u001b[32mOK\u001b[0m YAML syntax valid\r\n", + "\r\n", + "\u001b[1;36m2\u001b[0m\u001b[1m. Validating configuration schema\u001b[0m\u001b[1;33m...\u001b[0m\r\n", + " \u001b[32mOK\u001b[0m Configuration schema valid\r\n", + "\r\n", + "\u001b[1;36m3\u001b[0m\u001b[1m. Checking API keys\u001b[0m\u001b[1;33m...\u001b[0m\r\n", + " \u001b[32mOK\u001b[0m All required API keys configured\r\n", + "\r\n", + "\u001b[1;36m4\u001b[0m\u001b[1m. Checking dataset\u001b[0m\u001b[1;33m...\u001b[0m\r\n", + " \u001b[32mOK\u001b[0m Dataset found: data.json\r\n", + " \u001b[2mSize: \u001b[0m\u001b[1;2;36m980\u001b[0m\u001b[2m B\u001b[0m\r\n", + "\r\n", + "\u001b[1;36m5\u001b[0m\u001b[1m. Checking output directory\u001b[0m\u001b[1;33m...\u001b[0m\r\n", + " \u001b[32mOK\u001b[0m Output directory exists: .\u001b[35m/\u001b[0m\u001b[95mreports\u001b[0m\r\n", + "\r\n", + "\u001b[1;36m6\u001b[0m\u001b[1m. Checking provider configuration\u001b[0m\u001b[1;33m...\u001b[0m\r\n", + " \u001b[2mLLM: mock \u001b[0m\u001b[1;2m(\u001b[0m\u001b[2mmock-model\u001b[0m\u001b[1;2m)\u001b[0m\r\n", + " \u001b[2mRetriever: mock\u001b[0m\r\n", + "\r\n", + "\u001b[1;36m7\u001b[0m\u001b[1m. Checking metrics\u001b[0m\u001b[1;33m...\u001b[0m\r\n", + " \u001b[2mConfigured: \u001b[0m\u001b[1;2;36m7\u001b[0m\u001b[2m metrics\u001b[0m\r\n", + " \u001b[2mRetrieval: context_precision, context_recall, mrr\u001b[0m\r\n", + " \u001b[2mGeneration: faithfulness, answer_relevancy\u001b[0m\r\n", + " \u001b[2mPerformance: latency\u001b[0m\r\n", + " \u001b[2mCost: token_count\u001b[0m\r\n", + "\r\n", + "\u001b[1mSummary:\u001b[0m\r\n", + "\u001b[32mPASSED\u001b[0m Configuration is valid\r\n", + "\u001b[33mWARNING\u001b[0m \u001b[1;36m1\u001b[0m \u001b[1;35mwarning\u001b[0m\u001b[1m(\u001b[0ms\u001b[1m)\u001b[0m\r\n", + " \u001b[33m- Created output directory: reports\u001b[0m\r\n", + "\r\n", + "\u001b[2mReady to run: oaeval run \u001b[0m\u001b[1;2m<\u001b[0m\u001b[1;2;95mconfig\u001b[0m\u001b[1;2m>\u001b[0m\r\n" + ] + } + ], + "source": [ + "!oaeval validate config.yaml" + ] + }, + { + "cell_type": "markdown", + "id": "541e4a7a", + "metadata": {}, + "source": [ + "Now run the evaluation. The mock providers make this finish instantly." + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "id": "b751b4b5", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-23T20:02:04.292961Z", + "iopub.status.busy": "2026-07-23T20:02:04.292453Z", + "iopub.status.idle": "2026-07-23T20:02:05.517328Z", + "shell.execute_reply": "2026-07-23T20:02:05.516470Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\u001b[1;34mOpenAgent Eval\u001b[0m v0.\u001b[1;36m4.8\u001b[0m\r\n", + "\u001b[2mConfiguration: config.yaml\u001b[0m\r\n", + "\r\n", + "\u001b[?25l\r", + "\u001b[2K\u001b[32m⠋\u001b[0m Loading configuration... \u001b[90m━\u001b[0m\u001b[90m━\u001b[0m\u001b[90m━\u001b[0m\u001b[90m━\u001b[0m\u001b[35m━\u001b[0m\u001b[91m━\u001b[0m\u001b[91m━\u001b[0m\u001b[91m━\u001b[0m\u001b[91m━\u001b[0m\u001b[91m━\u001b[0m\u001b[91m━\u001b[0m\u001b[91m━\u001b[0m\u001b[35m━\u001b[0m\u001b[90m━\u001b[0m\u001b[90m━\u001b[0m\u001b[90m━\u001b[0m\u001b[90m━\u001b[0m\u001b[90m━\u001b[0m\u001b[90m━\u001b[0m\u001b[90m━\u001b[0m\u001b[90m━\u001b[0m\u001b[90m━\u001b[0m\u001b[90m━\u001b[0m\u001b[90m━\u001b[0m\u001b[35m━\u001b[0m\u001b[91m━\u001b[0m\u001b[91m━\u001b[0m\u001b[91m━\u001b[0m\u001b[91m━\u001b[0m\u001b[91m━\u001b[0m\u001b[91m━\u001b[0m\u001b[91m━\u001b[0m\u001b[35m━\u001b[0m\u001b[90m━\u001b[0m\u001b[90m━\u001b[0m\u001b[90m━\u001b[0m\u001b[90m━\u001b[0m\u001b[90m━\u001b[0m\u001b[90m━\u001b[0m\u001b[90m━\u001b[0m \u001b[35m 0%\u001b[0m \u001b[33m0:00:00\u001b[0m" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\r", + "\u001b[2K Complete! \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[35m100%\u001b[0m \u001b[33m0:00:00\u001b[0m\r\n", + "\u001b[?25h\r\n", + "\u001b[32mOK\u001b[0m Evaluation complete!\r\n", + "\u001b[2mItems: \u001b[0m\u001b[1;2;36m3\u001b[0m\u001b[2m | Errors: \u001b[0m\u001b[1;2;36m0\u001b[0m\r\n", + "\u001b[2mReport saved to: reports/\u001b[0m\u001b[2;93ma08f8fc3-06e4-4de3-ac11-eeb0858e7103\u001b[0m\u001b[2m.json\u001b[0m\r\n" + ] + } + ], + "source": [ + "!oaeval run config.yaml" + ] + }, + { + "cell_type": "markdown", + "id": "90405f0f", + "metadata": {}, + "source": [ + "View the newest report with `oaeval report latest`." + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "id": "3d9cd65a", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-23T20:02:05.590597Z", + "iopub.status.busy": "2026-07-23T20:02:05.589851Z", + "iopub.status.idle": "2026-07-23T20:02:06.584778Z", + "shell.execute_reply": "2026-07-23T20:02:06.578781Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\u001b[1;34mOpenAgent Eval\u001b[0m - Report Viewer\r\n", + "\u001b[2mReport: latest\u001b[0m\r\n", + "\r\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\u001b[32m╭─\u001b[0m\u001b[32m───────────────────────────\u001b[0m\u001b[32m Evaluation Complete \u001b[0m\u001b[32m────────────────────────────\u001b[0m\u001b[32m─╮\u001b[0m\r\n", + "\u001b[32m│\u001b[0m \u001b[1mOpenAgent Eval Report\u001b[0m \u001b[32m│\u001b[0m\r\n", + "\u001b[32m╰──────────────────────────────────────────────────────────────────────────────╯\u001b[0m\r\n", + "\u001b[3m Summary \u001b[0m\r\n", + "\u001b[34m┌─────────────┬───┐\u001b[0m\r\n", + "\u001b[34m│\u001b[0m\u001b[36m \u001b[0m\u001b[36mTotal Items\u001b[0m\u001b[36m \u001b[0m\u001b[34m│\u001b[0m\u001b[37m \u001b[0m\u001b[37m3\u001b[0m\u001b[37m \u001b[0m\u001b[34m│\u001b[0m\r\n", + "\u001b[34m│\u001b[0m\u001b[36m \u001b[0m\u001b[36mSuccessful \u001b[0m\u001b[36m \u001b[0m\u001b[34m│\u001b[0m\u001b[37m \u001b[0m\u001b[37m3\u001b[0m\u001b[37m \u001b[0m\u001b[34m│\u001b[0m\r\n", + "\u001b[34m│\u001b[0m\u001b[36m \u001b[0m\u001b[36mFailed \u001b[0m\u001b[36m \u001b[0m\u001b[34m│\u001b[0m\u001b[37m \u001b[0m\u001b[37m0\u001b[0m\u001b[37m \u001b[0m\u001b[34m│\u001b[0m\r\n", + "\u001b[34m└─────────────┴───┘\u001b[0m\r\n", + "\u001b[3m Metrics \u001b[0m\r\n", + "\u001b[34m┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┓\u001b[0m\r\n", + "\u001b[34m┃\u001b[0m\u001b[1m \u001b[0m\u001b[1mMetric \u001b[0m\u001b[1m \u001b[0m\u001b[34m┃\u001b[0m\u001b[1m \u001b[0m\u001b[1m Score\u001b[0m\u001b[1m \u001b[0m\u001b[34m┃\u001b[0m\r\n", + "\u001b[34m┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━━┩\u001b[0m\r\n", + "\u001b[34m│\u001b[0m\u001b[36m \u001b[0m\u001b[36mcontext_precision\u001b[0m\u001b[36m \u001b[0m\u001b[34m│\u001b[0m\u001b[32m \u001b[0m\u001b[31m0.0000\u001b[0m\u001b[32m \u001b[0m\u001b[34m│\u001b[0m\r\n", + "\u001b[34m│\u001b[0m\u001b[36m \u001b[0m\u001b[36mcontext_recall \u001b[0m\u001b[36m \u001b[0m\u001b[34m│\u001b[0m\u001b[32m \u001b[0m\u001b[31m0.0000\u001b[0m\u001b[32m \u001b[0m\u001b[34m│\u001b[0m\r\n", + "\u001b[34m│\u001b[0m\u001b[36m \u001b[0m\u001b[36mmrr \u001b[0m\u001b[36m \u001b[0m\u001b[34m│\u001b[0m\u001b[32m \u001b[0m\u001b[31m0.0000\u001b[0m\u001b[32m \u001b[0m\u001b[34m│\u001b[0m\r\n", + "\u001b[34m│\u001b[0m\u001b[36m \u001b[0m\u001b[36mfaithfulness \u001b[0m\u001b[36m \u001b[0m\u001b[34m│\u001b[0m\u001b[32m \u001b[0m\u001b[31m0.1250\u001b[0m\u001b[32m \u001b[0m\u001b[34m│\u001b[0m\r\n", + "\u001b[34m│\u001b[0m\u001b[36m \u001b[0m\u001b[36manswer_relevancy \u001b[0m\u001b[36m \u001b[0m\u001b[34m│\u001b[0m\u001b[32m \u001b[0m\u001b[31m0.3333\u001b[0m\u001b[32m \u001b[0m\u001b[34m│\u001b[0m\r\n", + "\u001b[34m│\u001b[0m\u001b[36m \u001b[0m\u001b[36mlatency \u001b[0m\u001b[36m \u001b[0m\u001b[34m│\u001b[0m\u001b[32m \u001b[0m\u001b[32m1.0000\u001b[0m\u001b[32m \u001b[0m\u001b[34m│\u001b[0m\r\n", + "\u001b[34m│\u001b[0m\u001b[36m \u001b[0m\u001b[36mtoken_count \u001b[0m\u001b[36m \u001b[0m\u001b[34m│\u001b[0m\u001b[32m \u001b[0m\u001b[32m0.9976\u001b[0m\u001b[32m \u001b[0m\u001b[34m│\u001b[0m\r\n", + "\u001b[34m└───────────────────┴────────┘\u001b[0m\r\n", + "\u001b[3m Sample Results \u001b[0m\r\n", + "\u001b[34m┏━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\u001b[0m\r\n", + "\u001b[34m┃\u001b[0m\u001b[1m \u001b[0m\u001b[1m#\u001b[0m\u001b[1m \u001b[0m\u001b[34m┃\u001b[0m\u001b[1m \u001b[0m\u001b[1mQuestion \u001b[0m\u001b[1m \u001b[0m\u001b[34m┃\u001b[0m\u001b[1m \u001b[0m\u001b[1mMetrics \u001b[0m\u001b[1m \u001b[0m\u001b[34m┃\u001b[0m\r\n", + "\u001b[34m┡━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\u001b[0m\r\n", + "\u001b[34m│\u001b[0m\u001b[2m \u001b[0m\u001b[2m1\u001b[0m\u001b[2m \u001b[0m\u001b[34m│\u001b[0m What is Retrieval-Augmented Generation? \u001b[34m│\u001b[0m context_precision=0.00, contex \u001b[34m│\u001b[0m\r\n", + "\u001b[34m│\u001b[0m\u001b[2m \u001b[0m\u001b[2m2\u001b[0m\u001b[2m \u001b[0m\u001b[34m│\u001b[0m What is a vector database? \u001b[34m│\u001b[0m context_precision=0.00, contex \u001b[34m│\u001b[0m\r\n", + "\u001b[34m│\u001b[0m\u001b[2m \u001b[0m\u001b[2m3\u001b[0m\u001b[2m \u001b[0m\u001b[34m│\u001b[0m Why evaluate a RAG system? \u001b[34m│\u001b[0m context_precision=0.00, contex \u001b[34m│\u001b[0m\r\n", + "\u001b[34m└───┴─────────────────────────────────────────┴────────────────────────────────┘\u001b[0m\r\n", + "\u001b[2m╭─\u001b[0m\u001b[2m──────────────────────────────\u001b[0m\u001b[2m Configuration \u001b[0m\u001b[2m───────────────────────────────\u001b[0m\u001b[2m─╮\u001b[0m\r\n", + "\u001b[2m│\u001b[0m Dataset: \u001b[36mdata.json\u001b[0m \u001b[2m│\u001b[0m\r\n", + "\u001b[2m│\u001b[0m LLM: \u001b[36mmock/mock-model\u001b[0m \u001b[2m│\u001b[0m\r\n", + "\u001b[2m│\u001b[0m Output: \u001b[36mterminal\u001b[0m \u001b[2m│\u001b[0m\r\n", + "\u001b[2m╰──────────────────────────────────────────────────────────────────────────────╯\u001b[0m\r\n", + "\r\n", + "\u001b[2mReport ID: \u001b[0m\u001b[2;93ma08f8fc3-06e4-4de3-ac11-eeb0858e7103\u001b[0m\r\n" + ] + } + ], + "source": [ + "!oaeval report latest" + ] + }, + { + "cell_type": "markdown", + "id": "9233d68f", + "metadata": {}, + "source": [ + "### Reading the scores\n", + "\n", + "Each metric is normalised to **0.0–1.0** (higher is better):\n", + "\n", + "- **`context_precision` / `context_recall` / `mrr`** — *retrieval* quality: did we fetch the right\n", + " documents, and how highly were they ranked?\n", + "- **`faithfulness`** — is the generated answer supported by the retrieved context (no hallucination)?\n", + "- **`answer_relevancy`** — does the answer actually address the question?\n", + "- **`latency`** — a normalised speed score.\n", + "- **`token_count`** — a normalised cost signal.\n", + "\n", + "> **Note on the mock numbers:** because the `mock` retriever returns placeholder documents rather\n", + "> than a real vector search, the *retrieval* metrics here are illustrative (they show the pipeline\n", + "> working, not a real system's quality). Point OpenAgent Eval at a real retriever + LLM (see the\n", + "> [Optional section](#sec17)) to get meaningful scores. What matters right now: **every stage ran,\n", + "> offline, with no keys.**\n", + "\n", + "Every run is saved to `./reports/`. List them any time:" + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "id": "cd73f5b6", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-23T20:02:06.595326Z", + "iopub.status.busy": "2026-07-23T20:02:06.594971Z", + "iopub.status.idle": "2026-07-23T20:02:07.517223Z", + "shell.execute_reply": "2026-07-23T20:02:07.514978Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\u001b[1;34mOpenAgent Eval\u001b[0m - Evaluation History\r\n", + "\r\n", + "\u001b[3m Recent Evaluations \u001b[0m\r\n", + "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━┓\r\n", + "┃\u001b[1m \u001b[0m\u001b[1mID \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mDate \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mConfig \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mStatus\u001b[0m\u001b[1m \u001b[0m┃\r\n", + "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━┩\r\n", + "│\u001b[36m \u001b[0m\u001b[36ma08f8fc3-06e4-4de3-ac11-eeb0858e7103\u001b[0m\u001b[36m \u001b[0m│\u001b[32m \u001b[0m\u001b[32m2026-07-23\u001b[0m\u001b[32m \u001b[0m│\u001b[33m \u001b[0m\u001b[33mdata.json\u001b[0m\u001b[33m \u001b[0m│\u001b[1m \u001b[0m\u001b[1;32mOK\u001b[0m\u001b[1m \u001b[0m\u001b[1m \u001b[0m│\r\n", + "└──────────────────────────────────────┴────────────┴───────────┴────────┘\r\n", + "\r\n", + "\u001b[2mShowing \u001b[0m\u001b[1;2;36m1\u001b[0m\u001b[2m evaluations\u001b[0m\r\n" + ] + } + ], + "source": [ + "!oaeval list" + ] + }, + { + "cell_type": "markdown", + "id": "0f31a576", + "metadata": {}, + "source": [ + "\n", + "## 7. Corpus health audit\n", + "\n", + "Garbage in, garbage out: if your knowledge base has **contradictions, stale documents, duplicates,\n", + "or coverage gaps**, no amount of prompt tuning will save your RAG system. `oaeval audit` scans a\n", + "folder of documents *before* you connect it to RAG. It runs fully offline." + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "id": "5353e274", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-23T20:02:07.525295Z", + "iopub.status.busy": "2026-07-23T20:02:07.524921Z", + "iopub.status.idle": "2026-07-23T20:02:07.536892Z", + "shell.execute_reply": "2026-07-23T20:02:07.533233Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Wrote a 2-document corpus.\n" + ] + } + ], + "source": [ + "import os, textwrap\n", + "os.makedirs(\"corpus\", exist_ok=True)\n", + "\n", + "with open(\"corpus/rag.md\", \"w\") as f:\n", + " f.write(textwrap.dedent(\"\"\"\n", + " # Retrieval-Augmented Generation\n", + " RAG combines a retriever and a generator. The retriever fetches relevant\n", + " documents from a knowledge base, and the generator produces an answer\n", + " grounded in those documents.\n", + " \"\"\").strip())\n", + "\n", + "with open(\"corpus/vectors.md\", \"w\") as f:\n", + " f.write(textwrap.dedent(\"\"\"\n", + " # Vector Databases\n", + " A vector database stores embeddings and supports similarity search over\n", + " high-dimensional vectors. Examples include Chroma, Qdrant, and FAISS.\n", + " \"\"\").strip())\n", + "\n", + "print(\"Wrote a 2-document corpus.\")" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "id": "ae0144c5", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-23T20:02:07.544828Z", + "iopub.status.busy": "2026-07-23T20:02:07.544443Z", + "iopub.status.idle": "2026-07-23T20:02:08.729807Z", + "shell.execute_reply": "2026-07-23T20:02:08.714099Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\u001b[1;34mOpenAgent Eval\u001b[0m v0.\u001b[1;36m4.8\u001b[0m\r\n", + "\u001b[2mCorpus: .\u001b[0m\u001b[2;35m/corpus/\u001b[0m\r\n", + "\r\n", + "\u001b[?25l\r", + "\u001b[2K\u001b[32m⠋\u001b[0m Loading corpus... \u001b[35m━\u001b[0m\u001b[90m━\u001b[0m\u001b[90m━\u001b[0m\u001b[90m━\u001b[0m\u001b[90m━\u001b[0m\u001b[90m━\u001b[0m\u001b[90m━\u001b[0m\u001b[90m━\u001b[0m\u001b[90m━\u001b[0m\u001b[90m━\u001b[0m\u001b[90m━\u001b[0m\u001b[90m━\u001b[0m\u001b[35m━\u001b[0m\u001b[91m━\u001b[0m\u001b[91m━\u001b[0m\u001b[91m━\u001b[0m\u001b[91m━\u001b[0m\u001b[91m━\u001b[0m\u001b[91m━\u001b[0m\u001b[91m━\u001b[0m\u001b[35m━\u001b[0m\u001b[90m━\u001b[0m\u001b[90m━\u001b[0m\u001b[90m━\u001b[0m\u001b[90m━\u001b[0m\u001b[90m━\u001b[0m\u001b[90m━\u001b[0m\u001b[90m━\u001b[0m\u001b[90m━\u001b[0m\u001b[90m━\u001b[0m\u001b[90m━\u001b[0m\u001b[90m━\u001b[0m\u001b[35m━\u001b[0m\u001b[91m━\u001b[0m\u001b[91m━\u001b[0m\u001b[91m━\u001b[0m\u001b[91m━\u001b[0m\u001b[91m━\u001b[0m\u001b[91m━\u001b[0m\u001b[91m━\u001b[0m \u001b[35m 0%\u001b[0m \u001b[33m0:00:00\u001b[0m\r", + "\u001b[2K Audit complete! \u001b[91m━━━━━━━━\u001b[0m\u001b[90m╺\u001b[0m\u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[35m 20%\u001b[0m \u001b[33m0:00:00\u001b[0m\r\n", + "\u001b[?25h\r\n", + "\u001b[32m╭─\u001b[0m\u001b[32m───────────────────────────\u001b[0m\u001b[32m \u001b[0m\u001b[1;32mCorpus Health Score\u001b[0m\u001b[32m \u001b[0m\u001b[32m────────────────────────────\u001b[0m\u001b[32m─╮\u001b[0m\r\n", + "\u001b[32m│\u001b[0m \u001b[1;32m100.0%\u001b[0m — Healthy \u001b[32m│\u001b[0m\r\n", + "\u001b[32m╰──────────────────────────────────────────────────────────────────────────────╯\u001b[0m\r\n", + "\r\n", + "\u001b[1mSummary:\u001b[0m Corpus audit complete: \u001b[1;36m2\u001b[0m documents analyzed across \u001b[1;36m4\u001b[0m checks. No issues \r\n", + "found.\r\n", + "\r\n", + "\u001b[32mNo issues found!\u001b[0m\r\n" + ] + } + ], + "source": [ + "!oaeval audit ./corpus/" + ] + }, + { + "cell_type": "markdown", + "id": "b4117af3", + "metadata": {}, + "source": [ + "The **health score** summarises the corpus. The four checks are:\n", + "\n", + "- **Contradiction** — documents that assert conflicting facts.\n", + "- **Staleness** — documents older than a freshness threshold (`--staleness-days`).\n", + "- **Duplicate** — near-identical documents (`--similarity-threshold`).\n", + "- **Coverage** — thematic gaps across your corpus.\n", + "\n", + "Run a subset with `--checks`, e.g. `!oaeval audit ./corpus/ --checks contradiction,duplicate`." + ] + }, + { + "cell_type": "markdown", + "id": "b9b5098b", + "metadata": {}, + "source": [ + "\n", + "## 8. Failure diagnosis (blame attribution)\n", + "\n", + "When an evaluation scores poorly, *which component is at fault?* `oaeval diagnose` reads a saved\n", + "report and attributes each failure to one of three culprits:\n", + "\n", + "- **Retrieval** — the wrong documents were fetched.\n", + "- **Generation** — the LLM hallucinated or misread the context.\n", + "- **Chunking** — documents were split badly (too small, overlapping, etc.).\n", + "\n", + "Point it at the report file we just produced." + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "id": "734442ac", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-23T20:02:08.771262Z", + "iopub.status.busy": "2026-07-23T20:02:08.770934Z", + "iopub.status.idle": "2026-07-23T20:02:08.780103Z", + "shell.execute_reply": "2026-07-23T20:02:08.778853Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Diagnosing: reports/a08f8fc3-06e4-4de3-ac11-eeb0858e7103.json\n" + ] + } + ], + "source": [ + "import glob, os\n", + "\n", + "latest_report = max(glob.glob(\"reports/*.json\"), key=os.path.getmtime)\n", + "print(\"Diagnosing:\", latest_report)" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "id": "df913786", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-23T20:02:08.825482Z", + "iopub.status.busy": "2026-07-23T20:02:08.825201Z", + "iopub.status.idle": "2026-07-23T20:02:09.834441Z", + "shell.execute_reply": "2026-07-23T20:02:09.829715Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\u001b[34m╭─\u001b[0m\u001b[34m───────────────────────────\u001b[0m\u001b[34m Component Diagnosis \u001b[0m\u001b[34m────────────────────────────\u001b[0m\u001b[34m─╮\u001b[0m\r\n", + "\u001b[34m│\u001b[0m \u001b[1;34mDiagnosis Report\u001b[0m \u001b[34m│\u001b[0m\r\n", + "\u001b[34m│\u001b[0m Items analyzed: 3 \u001b[34m│\u001b[0m\r\n", + "\u001b[34m│\u001b[0m Overall health: 0.0% \u001b[34m│\u001b[0m\r\n", + "\u001b[34m╰──────────────────────────────────────────────────────────────────────────────╯\u001b[0m\r\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\r\n", + "\u001b[1;31mSystem Health: Unhealthy \u001b[0m\u001b[1;31m(\u001b[0m\u001b[1;31m0.0\u001b[0m\u001b[1;31m%\u001b[0m\u001b[1;31m)\u001b[0m\r\n", + "\r\n", + "\u001b[3m Blame Attribution \u001b[0m\r\n", + "┏━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━┓\r\n", + "┃\u001b[1m \u001b[0m\u001b[1mComponent \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mFailures\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mPercentage\u001b[0m\u001b[1m \u001b[0m┃\r\n", + "┡━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━┩\r\n", + "│\u001b[1m \u001b[0m\u001b[1;31mRetrieval\u001b[0m\u001b[1m \u001b[0m\u001b[1m \u001b[0m│ 6 │ 54.5% │\r\n", + "├────────────┼──────────┼────────────┤\r\n", + "│\u001b[1m \u001b[0m\u001b[1;33mGeneration\u001b[0m\u001b[1m \u001b[0m│ 5 │ 45.5% │\r\n", + "└────────────┴──────────┴────────────┘\r\n", + "\r\n", + "\u001b[3m Failure Modes \u001b[0m\r\n", + "┏━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┓\r\n", + "┃\u001b[1m \u001b[0m\u001b[1mFailure Mode \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mCount\u001b[0m\u001b[1m \u001b[0m┃\r\n", + "┡━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━┩\r\n", + "│\u001b[1m \u001b[0m\u001b[1mLow Context Relevance\u001b[0m\u001b[1m \u001b[0m│ 3 │\r\n", + "├───────────────────────┼───────┤\r\n", + "│\u001b[1m \u001b[0m\u001b[1mMissing Context \u001b[0m\u001b[1m \u001b[0m│ 3 │\r\n", + "├───────────────────────┼───────┤\r\n", + "│\u001b[1m \u001b[0m\u001b[1mHallucination \u001b[0m\u001b[1m \u001b[0m│ 3 │\r\n", + "├───────────────────────┼───────┤\r\n", + "│\u001b[1m \u001b[0m\u001b[1mOff-Topic Answer \u001b[0m\u001b[1m \u001b[0m│ 2 │\r\n", + "└───────────────────────┴───────┘\r\n", + "\r\n", + "\u001b[1mChunking Issues:\u001b[0m\r\n", + "\r\n", + " \u001b[36moverlapping_chunks\u001b[0m: Contexts \u001b[1;36m1\u001b[0m and \u001b[1;36m2\u001b[0m have high overlap \u001b[1m(\u001b[0m\u001b[33msimilarity\u001b[0m=\u001b[1;36m0\u001b[0m\u001b[1;36m.82\u001b[0m\u001b[1m)\u001b[0m, \r\n", + "suggesting duplicate chunking.\r\n", + "\r\n", + " \u001b[36moverlapping_chunks\u001b[0m: Contexts \u001b[1;36m1\u001b[0m and \u001b[1;36m3\u001b[0m have high overlap \u001b[1m(\u001b[0m\u001b[33msimilarity\u001b[0m=\u001b[1;36m0\u001b[0m\u001b[1;36m.82\u001b[0m\u001b[1m)\u001b[0m, \r\n", + "suggesting duplicate chunking.\r\n", + "\r\n", + " \u001b[36moverlapping_chunks\u001b[0m: Contexts \u001b[1;36m2\u001b[0m and \u001b[1;36m3\u001b[0m have high overlap \u001b[1m(\u001b[0m\u001b[33msimilarity\u001b[0m=\u001b[1;36m0\u001b[0m\u001b[1;36m.82\u001b[0m\u001b[1m)\u001b[0m, \r\n", + "suggesting duplicate chunking.\r\n", + "\r\n", + " \u001b[36moverlapping_chunks\u001b[0m: Contexts \u001b[1;36m1\u001b[0m and \u001b[1;36m2\u001b[0m have high overlap \u001b[1m(\u001b[0m\u001b[33msimilarity\u001b[0m=\u001b[1;36m0\u001b[0m\u001b[1;36m.82\u001b[0m\u001b[1m)\u001b[0m, \r\n", + "suggesting duplicate chunking.\r\n", + "\r\n", + " \u001b[36moverlapping_chunks\u001b[0m: Contexts \u001b[1;36m1\u001b[0m and \u001b[1;36m3\u001b[0m have high overlap \u001b[1m(\u001b[0m\u001b[33msimilarity\u001b[0m=\u001b[1;36m0\u001b[0m\u001b[1;36m.82\u001b[0m\u001b[1m)\u001b[0m, \r\n", + "suggesting duplicate chunking.\r\n", + "\r\n", + " \u001b[36moverlapping_chunks\u001b[0m: Contexts \u001b[1;36m2\u001b[0m and \u001b[1;36m3\u001b[0m have high overlap \u001b[1m(\u001b[0m\u001b[33msimilarity\u001b[0m=\u001b[1;36m0\u001b[0m\u001b[1;36m.82\u001b[0m\u001b[1m)\u001b[0m, \r\n", + "suggesting duplicate chunking.\r\n", + "\r\n", + "\u001b[1mRecommendations:\u001b[0m\r\n", + "\r\n", + " -> \u001b[1m[\u001b[0mRETRIEVAL\u001b[1m]\u001b[0m \u001b[1m(\u001b[0m\u001b[1;36m6\u001b[0m failures\u001b[1m)\u001b[0m Check if the retriever is using the correct \r\n", + "embedding model.\r\n", + " -> \u001b[1m[\u001b[0mRETRIEVAL\u001b[1m]\u001b[0m \u001b[1m(\u001b[0m\u001b[1;36m6\u001b[0m failures\u001b[1m)\u001b[0m Verify that the vector store contains the relevant\r\n", + "documents.\r\n", + " -> \u001b[1m[\u001b[0mRETRIEVAL\u001b[1m]\u001b[0m \u001b[1m(\u001b[0m\u001b[1;36m6\u001b[0m failures\u001b[1m)\u001b[0m Consider increasing the retrieval k value.\r\n", + " -> \u001b[1m[\u001b[0mRETRIEVAL\u001b[1m]\u001b[0m \u001b[1m(\u001b[0m\u001b[1;36m6\u001b[0m failures\u001b[1m)\u001b[0m Review chunking strategy to ensure documents are \r\n", + "split correctly.\r\n", + " -> \u001b[1m[\u001b[0mRETRIEVAL\u001b[1m]\u001b[0m \u001b[1m(\u001b[0m\u001b[1;36m6\u001b[0m failures\u001b[1m)\u001b[0m Check for index staleness — rebuild if documents \r\n", + "have changed.\r\n", + "\r\n" + ] + } + ], + "source": [ + "!oaeval diagnose {latest_report}" + ] + }, + { + "cell_type": "markdown", + "id": "cab3ae5b", + "metadata": {}, + "source": [ + "The output groups **failure signals** and then lists **recommendations** tagged by component\n", + "(`[RETRIEVAL]`, `[GENERATION]`, `[CHUNKING]`). Use the tag to decide where to spend your next hour:\n", + "re-chunking, swapping the embedding model, or tightening the generation prompt." + ] + }, + { + "cell_type": "markdown", + "id": "97175e19", + "metadata": {}, + "source": [ + "\n", + "## 9. Synthetic test-data generation\n", + "\n", + "Don't have a labelled test set yet? `oaeval synth` can **generate question/answer test cases from\n", + "your corpus** — including adversarial ones (unanswerable, misleading, multi-hop, …).\n", + "\n", + "Synthesis *requires a capable LLM* to write the Q&A pairs. The offline `mock` provider returns\n", + "placeholder text, so running it offline **completes successfully but produces 0 cases** — a useful\n", + "way to confirm the command works end-to-end before you plug in a real model." + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "id": "1806988a", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-23T20:02:09.839443Z", + "iopub.status.busy": "2026-07-23T20:02:09.837922Z", + "iopub.status.idle": "2026-07-23T20:02:10.837570Z", + "shell.execute_reply": "2026-07-23T20:02:10.834799Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Standard generation failed: Failed to parse LLM response (response_preview=[mock-answer] You are a test case generator for a RAG (Retrieval-Augmented Generation) evaluation system.\r\n", + "\r\n", + "Given the following document chunk, generate 3 diverse question-answer pairs.\r\n", + "Each question s)\r\n", + "\u001b[32mSaved \u001b[0m\u001b[1;32m0\u001b[0m\u001b[32m test cases to synth.json\u001b[0m\r\n" + ] + } + ], + "source": [ + "!oaeval synth --text \"Retrieval-Augmented Generation pairs a retriever with a generator to produce grounded answers.\" \\\n", + " --count 3 --llm-provider mock --llm-model mock-model --format json --output synth.json" + ] + }, + { + "cell_type": "markdown", + "id": "bb8fab24", + "metadata": {}, + "source": [ + "To get **real** synthetic test cases, run the same command with a real provider (e.g.\n", + "`--llm-provider openai --llm-model gpt-4o-mini`) after setting an API key — see the\n", + "[Optional section](#sec17). For example:\n", + "\n", + "```bash\n", + "!oaeval synth --corpus ./corpus/ --count 20 --adversarial \\\n", + " --llm-provider openai --llm-model gpt-4o-mini --output dataset.json\n", + "```" + ] + }, + { + "cell_type": "markdown", + "id": "a23c9b87", + "metadata": {}, + "source": [ + "\n", + "## 10. Comparing experiments\n", + "\n", + "Iterating on a RAG system means running many evaluations and asking *\"did that change help?\"*.\n", + "`oaeval compare` puts two runs side by side and computes per-metric deltas plus an overall winner.\n", + "\n", + "Let's create a second run (a copy of our config) so we have two experiments to compare." + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "id": "b3d1b41f", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-23T20:02:10.845202Z", + "iopub.status.busy": "2026-07-23T20:02:10.844435Z", + "iopub.status.idle": "2026-07-23T20:02:11.935156Z", + "shell.execute_reply": "2026-07-23T20:02:11.933765Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\u001b[1;34mOpenAgent Eval\u001b[0m v0.\u001b[1;36m4.8\u001b[0m\r\n", + "\u001b[2mConfiguration: config_v2.yaml\u001b[0m\r\n", + "\r\n", + "\u001b[?25l" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\r", + "\u001b[2K\u001b[32m⠋\u001b[0m Loading configuration... \u001b[90m━\u001b[0m\u001b[35m━\u001b[0m\u001b[91m━\u001b[0m\u001b[91m━\u001b[0m\u001b[91m━\u001b[0m\u001b[91m━\u001b[0m\u001b[91m━\u001b[0m\u001b[91m━\u001b[0m\u001b[91m━\u001b[0m\u001b[35m━\u001b[0m\u001b[90m━\u001b[0m\u001b[90m━\u001b[0m\u001b[90m━\u001b[0m\u001b[90m━\u001b[0m\u001b[90m━\u001b[0m\u001b[90m━\u001b[0m\u001b[90m━\u001b[0m\u001b[90m━\u001b[0m\u001b[90m━\u001b[0m\u001b[90m━\u001b[0m\u001b[90m━\u001b[0m\u001b[35m━\u001b[0m\u001b[91m━\u001b[0m\u001b[91m━\u001b[0m\u001b[91m━\u001b[0m\u001b[91m━\u001b[0m\u001b[91m━\u001b[0m\u001b[91m━\u001b[0m\u001b[91m━\u001b[0m\u001b[35m━\u001b[0m\u001b[90m━\u001b[0m\u001b[90m━\u001b[0m\u001b[90m━\u001b[0m\u001b[90m━\u001b[0m\u001b[90m━\u001b[0m\u001b[90m━\u001b[0m\u001b[90m━\u001b[0m\u001b[90m━\u001b[0m\u001b[90m━\u001b[0m\u001b[90m━\u001b[0m \u001b[35m 0%\u001b[0m \u001b[33m0:00:00\u001b[0m\r", + "\u001b[2K Complete! \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[35m100%\u001b[0m \u001b[33m0:00:00\u001b[0m\r\n", + "\u001b[?25h\r\n", + "\u001b[32mOK\u001b[0m Evaluation complete!\r\n", + "\u001b[2mItems: \u001b[0m\u001b[1;2;36m3\u001b[0m\u001b[2m | Errors: \u001b[0m\u001b[1;2;36m0\u001b[0m\r\n", + "\u001b[2mReport saved to: reports/\u001b[0m\u001b[2;93m319a2d80-ae9a-46e4-8c0f-5d8de9f5e32d\u001b[0m\u001b[2m.json\u001b[0m\r\n" + ] + } + ], + "source": [ + "!cp config.yaml config_v2.yaml\n", + "!oaeval run config_v2.yaml" + ] + }, + { + "cell_type": "code", + "execution_count": 25, + "id": "8406c54a", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-23T20:02:11.957078Z", + "iopub.status.busy": "2026-07-23T20:02:11.950715Z", + "iopub.status.idle": "2026-07-23T20:02:11.971043Z", + "shell.execute_reply": "2026-07-23T20:02:11.968944Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "baseline : a08f8fc3-06e4-4de3-ac11-eeb0858e7103\n", + "experiment: 319a2d80-ae9a-46e4-8c0f-5d8de9f5e32d\n" + ] + } + ], + "source": [ + "import glob, os\n", + "\n", + "# Grab the two most recent report IDs (filenames without the .json extension).\n", + "reports = sorted(glob.glob(\"reports/*.json\"), key=os.path.getmtime)\n", + "baseline_id = os.path.basename(reports[-2])[:-5]\n", + "experiment_id = os.path.basename(reports[-1])[:-5]\n", + "print(\"baseline :\", baseline_id)\n", + "print(\"experiment:\", experiment_id)" + ] + }, + { + "cell_type": "code", + "execution_count": 26, + "id": "18705382", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-23T20:02:11.975528Z", + "iopub.status.busy": "2026-07-23T20:02:11.974366Z", + "iopub.status.idle": "2026-07-23T20:02:12.827912Z", + "shell.execute_reply": "2026-07-23T20:02:12.824137Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\u001b[1;34mOpenAgent Eval\u001b[0m - Experiment Comparison\r\n", + "\u001b[2mComparing: \u001b[0m\u001b[2;93ma08f8fc3-06e4-4de3-ac11-eeb0858e7103\u001b[0m\u001b[2m vs \u001b[0m\r\n", + "\u001b[2;93m319a2d80-ae9a-46e4-8c0f-5d8de9f5e32d\u001b[0m\r\n", + "\r\n", + "============================================================\r\n", + " Experiment Comparison Report\r\n", + "============================================================\r\n", + "\r\n", + " Generated: \u001b[1;36m2026\u001b[0m-\u001b[1;36m07\u001b[0m-\u001b[1;36m23\u001b[0m \u001b[1;92m20:02:12\u001b[0m UTC\r\n", + " Baseline: \u001b[93ma08f8fc3-06e4-4de3-ac11-eeb0858e7103\u001b[0m\r\n", + " Experiment: \u001b[93m319a2d80-ae9a-46e4-8c0f-5d8de9f5e32d\u001b[0m\r\n", + "\r\n", + "METRIC COMPARISON\r\n", + "------------------------------------------------------------\r\n", + " Metric Baseline Experiment Delta\r\n", + " ----------------------------------------------------------\r\n", + " answer_relevancy \u001b[1;36m0.3333\u001b[0m \u001b[1;36m0.3333\u001b[0m = +\u001b[1;36m0.0000\u001b[0m\r\n", + " context_precision \u001b[1;36m0.0000\u001b[0m \u001b[1;36m0.0000\u001b[0m = +\u001b[1;36m0.0000\u001b[0m\r\n", + " context_recall \u001b[1;36m0.0000\u001b[0m \u001b[1;36m0.0000\u001b[0m = +\u001b[1;36m0.0000\u001b[0m\r\n", + " faithfulness \u001b[1;36m0.1250\u001b[0m \u001b[1;36m0.1250\u001b[0m = +\u001b[1;36m0.0000\u001b[0m\r\n", + " latency \u001b[1;36m1.0000\u001b[0m \u001b[1;36m1.0000\u001b[0m = \u001b[1;36m-0.0000\u001b[0m\r\n", + " mrr \u001b[1;36m0.0000\u001b[0m \u001b[1;36m0.0000\u001b[0m = +\u001b[1;36m0.0000\u001b[0m\r\n", + " token_count \u001b[1;36m0.9976\u001b[0m \u001b[1;36m0.9976\u001b[0m = +\u001b[1;36m0.0000\u001b[0m\r\n", + "\r\n", + "SUMMARY\r\n", + "------------------------------------------------------------\r\n", + " Metrics improved: \u001b[1;36m0\u001b[0m\r\n", + " Metrics regressed: \u001b[1;36m0\u001b[0m\r\n", + " Metrics unchanged: \u001b[1;36m7\u001b[0m\r\n", + "\r\n", + " Baseline overall: \u001b[1;36m0.3508\u001b[0m\r\n", + " Experiment overall: \u001b[1;36m0.3508\u001b[0m\r\n", + " Overall delta: \u001b[1;36m-0.0000\u001b[0m\r\n", + "\r\n", + " >> WINNER: \u001b[93ma08f8fc3-06e4-4de3-ac11-eeb0858e7103\u001b[0m\r\n", + "RESULT COUNTS\r\n", + "------------------------------------------------------------\r\n", + " Baseline results: \u001b[1;36m3\u001b[0m\r\n", + " Experiment results: \u001b[1;36m3\u001b[0m\r\n", + "\r\n", + "============================================================\r\n" + ] + } + ], + "source": [ + "!oaeval compare {baseline_id} {experiment_id}" + ] + }, + { + "cell_type": "markdown", + "id": "9d122de6", + "metadata": {}, + "source": [ + "Because both runs used identical (mock) settings, every delta is `+0.0000`. In real use you'd\n", + "change the model, retriever, chunk size, or prompt between runs and watch the deltas move — a\n", + "compact, quantitative answer to *\"is this version better?\"*." + ] + }, + { + "cell_type": "markdown", + "id": "b5667246", + "metadata": {}, + "source": [ + "\n", + "## 11. SDK usage (the Python API)\n", + "\n", + "Prefer code over the CLI? Everything is available programmatically. Use the SDK when you want to\n", + "embed evaluation inside a larger Python workflow, build a custom loop, or inspect results in memory.\n", + "\n", + "`Engine.run` is an **async** method, so we `await` it (top-level `await` works in Colab/Jupyter)." + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "id": "41de6530", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-23T20:02:12.834237Z", + "iopub.status.busy": "2026-07-23T20:02:12.833881Z", + "iopub.status.idle": "2026-07-23T20:02:13.244585Z", + "shell.execute_reply": "2026-07-23T20:02:13.243063Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Report type: EvaluationReport\n" + ] + } + ], + "source": [ + "from openagent_eval.core import Engine\n", + "from openagent_eval.config import load_config\n", + "\n", + "dataset = [\n", + " {\n", + " \"question\": \"What is RAG?\",\n", + " \"ground_truth\": \"Retrieval-Augmented Generation.\",\n", + " \"context\": \"RAG combines retrieval and generation to produce grounded answers.\",\n", + " },\n", + " {\n", + " \"question\": \"What is a vector database?\",\n", + " \"ground_truth\": \"A store for embeddings enabling similarity search.\",\n", + " \"context\": \"Vector databases store embeddings and support similarity search.\",\n", + " },\n", + "]\n", + "\n", + "config = load_config(\"config.yaml\") # our offline mock config\n", + "engine = Engine(config)\n", + "\n", + "report = await engine.run(dataset) # async → await\n", + "print(\"Report type:\", type(report).__name__)" + ] + }, + { + "cell_type": "code", + "execution_count": 28, + "id": "55c5154a", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-23T20:02:13.253929Z", + "iopub.status.busy": "2026-07-23T20:02:13.253348Z", + "iopub.status.idle": "2026-07-23T20:02:13.263994Z", + "shell.execute_reply": "2026-07-23T20:02:13.262872Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Items evaluated: 2\n", + "\n", + "Metric scores:\n", + " context_precision 0.0000\n", + " context_recall 0.0000\n", + " mrr 0.0000\n", + " faithfulness 0.1429\n", + " answer_relevancy 0.0000\n", + " latency 1.0000\n", + " token_count 0.9979\n" + ] + } + ], + "source": [ + "# report.summary is a plain dict — easy to log, assert on, or serialise.\n", + "summary = report.summary\n", + "print(\"Items evaluated:\", summary[\"total_items\"])\n", + "print()\n", + "print(\"Metric scores:\")\n", + "for name, score in summary[\"metrics_summary\"].items():\n", + " print(f\" {name:20s} {score:.4f}\")" + ] + }, + { + "cell_type": "markdown", + "id": "0342d4d3", + "metadata": {}, + "source": [ + "That `metrics_summary` dict is the programmatic entry point for dashboards, alerts, or the\n", + "threshold checks we build next." + ] + }, + { + "cell_type": "markdown", + "id": "8af3e410", + "metadata": {}, + "source": [ + "\n", + "## 12. CI/CD gating\n", + "\n", + "The point of evaluation is to **stop regressions from shipping**. OpenAgent Eval is designed to run\n", + "in CI: define **thresholds** (e.g. *faithfulness must be ≥ 0.8*) and fail the build if they are not\n", + "met.\n", + "\n", + "The CLI form is:\n", + "\n", + "```bash\n", + "oaeval test config.yaml -t faithfulness:gte:0.8 -t answer_relevancy:gte:0.7\n", + "```\n", + "\n", + "where each `-t` gate is `metric:operator:value` (operators: `gt`, `gte`, `lt`, `lte`, `eq`, `neq`).\n", + "The command exits **0** when every gate passes and **1** when any gate fails — exactly what a CI\n", + "runner needs.\n", + "\n", + "Below we demonstrate the **gating logic directly on the metrics we computed**, so you can see how a\n", + "pass/fail decision is made. This mirrors what a CI gate does with your report's scores." + ] + }, + { + "cell_type": "code", + "execution_count": 29, + "id": "95f8ae71", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-23T20:02:13.274763Z", + "iopub.status.busy": "2026-07-23T20:02:13.273455Z", + "iopub.status.idle": "2026-07-23T20:02:13.286186Z", + "shell.execute_reply": "2026-07-23T20:02:13.285272Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[FAIL] answer_relevancy 0.0000 >= 0.5\n", + "[PASS] latency 1.0000 >= 0.5\n", + "[FAIL] faithfulness 0.1429 >= 0.9\n", + "\n", + "CI would exit with code 1\n" + ] + } + ], + "source": [ + "scores = report.summary[\"metrics_summary\"]\n", + "\n", + "# Define the gates you'd enforce in CI: (metric, operator, threshold).\n", + "gates = [\n", + " (\"answer_relevancy\", \">=\", 0.5),\n", + " (\"latency\", \">=\", 0.5),\n", + " (\"faithfulness\", \">=\", 0.9), # deliberately strict to show a failure\n", + "]\n", + "\n", + "import operator as _op\n", + "OPS = {\">=\": _op.ge, \">\": _op.gt, \"<=\": _op.le, \"<\": _op.lt, \"==\": _op.eq}\n", + "\n", + "all_passed = True\n", + "for metric, sym, threshold in gates:\n", + " actual = scores.get(metric)\n", + " ok = actual is not None and OPS[sym](actual, threshold)\n", + " all_passed &= ok\n", + " status = \"PASS\" if ok else \"FAIL\"\n", + " print(f\"[{status}] {metric:18s} {actual:.4f} {sym} {threshold}\")\n", + "\n", + "print()\n", + "print(\"CI would exit with code\", 0 if all_passed else 1)" + ] + }, + { + "cell_type": "markdown", + "id": "2744e1c7", + "metadata": {}, + "source": [ + "### Wiring it into GitHub Actions\n", + "\n", + "A minimal workflow that gates every push/PR on evaluation quality:\n", + "\n", + "```yaml\n", + "name: Evaluation Gate\n", + "on: [push, pull_request]\n", + "jobs:\n", + " eval:\n", + " runs-on: ubuntu-latest\n", + " steps:\n", + " - uses: actions/checkout@v4\n", + " - uses: actions/setup-python@v5\n", + " with:\n", + " python-version: '3.11'\n", + " - run: pip install openagent-eval pytest\n", + " - run: oaeval test config.yaml -t faithfulness:gte:0.8 -t answer_relevancy:gte:0.7\n", + "```\n", + "\n", + "If any threshold fails, `oaeval test` returns a non-zero exit code and the job goes red." + ] + }, + { + "cell_type": "markdown", + "id": "b2e0d95c", + "metadata": {}, + "source": [ + "\n", + "## 13. Advanced — custom metrics\n", + "\n", + "Need a metric that isn't built in? Subclass `BaseMetric` and return a `MetricResult`. The interface\n", + "is intentionally tiny:\n", + "\n", + "- `name` / `description` — identify the metric.\n", + "- `evaluate(self, **kwargs) -> MetricResult` — receives the evaluation data as keyword arguments\n", + " (`answer`, `context`, `question`, `ground_truth`, …) and returns a score.\n", + "- `MetricResult(score, reason, metadata)` — the `score` **must be between 0.0 and 1.0**.\n", + "\n", + "Here's a custom metric that scores answers by normalised length." + ] + }, + { + "cell_type": "code", + "execution_count": 30, + "id": "aafd8295", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-23T20:02:13.303441Z", + "iopub.status.busy": "2026-07-23T20:02:13.302713Z", + "iopub.status.idle": "2026-07-23T20:02:13.316189Z", + "shell.execute_reply": "2026-07-23T20:02:13.315050Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "score : 0.08\n", + "reason : Answer has 8 words\n", + "metadata: {'word_count': 8}\n" + ] + } + ], + "source": [ + "from openagent_eval.metrics.base import BaseMetric, MetricResult\n", + "\n", + "\n", + "class AnswerWordCountMetric(BaseMetric):\n", + " name = \"answer_word_count\"\n", + " description = \"Normalised answer length (word count / 100, capped at 1.0).\"\n", + "\n", + " def evaluate(self, **kwargs) -> MetricResult:\n", + " answer = kwargs.get(\"answer\", \"\") or \"\"\n", + " word_count = len(answer.split())\n", + " return MetricResult(\n", + " score=min(word_count / 100, 1.0),\n", + " reason=f\"Answer has {word_count} words\",\n", + " metadata={\"word_count\": word_count},\n", + " )\n", + "\n", + "\n", + "metric = AnswerWordCountMetric()\n", + "result = metric.evaluate(\n", + " answer=\"Retrieval-Augmented Generation combines a retriever with a generator.\"\n", + ")\n", + "print(\"score :\", result.score)\n", + "print(\"reason :\", result.reason)\n", + "print(\"metadata:\", result.metadata)" + ] + }, + { + "cell_type": "markdown", + "id": "f9dc7bb2", + "metadata": {}, + "source": [ + "The 18+ built-in metrics all implement this same `BaseMetric` interface — you can browse them in\n", + "the `openagent_eval.metrics` package. The full built-in registry:" + ] + }, + { + "cell_type": "code", + "execution_count": 31, + "id": "f2b28f5c", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-23T20:02:13.320354Z", + "iopub.status.busy": "2026-07-23T20:02:13.320022Z", + "iopub.status.idle": "2026-07-23T20:02:13.326188Z", + "shell.execute_reply": "2026-07-23T20:02:13.325175Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Built-in metrics:\n", + " - answer_relevancy\n", + " - bertscore\n", + " - bleu\n", + " - context_precision\n", + " - context_recall\n", + " - exact_match\n", + " - f1_score\n", + " - faithfulness\n", + " - hallucination\n", + " - hit_rate\n", + " - latency\n", + " - mrr\n", + " - ndcg\n", + " - precision_at_k\n", + " - recall_at_k\n", + " - rouge\n", + " - semantic_similarity\n", + " - token_count\n" + ] + } + ], + "source": [ + "from openagent_eval.metrics import METRIC_REGISTRY\n", + "\n", + "print(\"Built-in metrics:\")\n", + "for name in sorted(METRIC_REGISTRY):\n", + " print(\" -\", name)" + ] + }, + { + "cell_type": "markdown", + "id": "5c9c339e", + "metadata": {}, + "source": [ + "\n", + "## 14. Tips, tricks & troubleshooting\n", + "\n", + "**Higher-quality faithfulness/relevancy scores.** These metrics use a small NLI (natural-language\n", + "inference) model when `transformers` is available, and gracefully fall back to a lexical-overlap\n", + "approximation when it isn't. For NLI-grade scoring, install the extra:\n", + "\n", + "```python\n", + "%pip install -q transformers torch\n", + "```\n", + "\n", + "**Use a real LLM locally & for free with Ollama.** On a machine with a GPU you can run a local model\n", + "and point the config at it (`llm.provider: ollama`). See the docs for the Ollama provider.\n", + "\n", + "**Persist files across Colab sessions — mount Google Drive:**\n", + "\n", + "```python\n", + "from google.colab import drive\n", + "drive.mount('/content/drive')\n", + "# now read/write under /content/drive/MyDrive/...\n", + "```\n", + "\n", + "**Common issues**\n", + "\n", + "| Symptom | Fix |\n", + "|---|---|\n", + "| `oaeval: command not found` | Re-run the install cell; restart the runtime if needed. |\n", + "| A command asks interactive questions and hangs | Use `oaeval init --no-interactive` (as we do here). |\n", + "| Retrieval metrics are all `0.0` | Expected with the `mock` retriever — plug in a real one. |\n", + "| `synth` produced 0 cases | The `mock` LLM can't write Q&A — use a real provider ([§17](#sec17)). |\n", + "| Provider errors about missing API keys | You're using a real provider — set the key ([§17](#sec17)) or switch back to `mock`. |\n", + "\n", + "**Share this notebook.** *File → Save a copy in Drive*, then *Share*, or push your copy to GitHub and\n", + "add the Colab badge from the top of this notebook." + ] + }, + { + "cell_type": "markdown", + "id": "a7382ef7", + "metadata": {}, + "source": [ + "\n", + "## 15. Next steps & resources\n", + "\n", + "- 📖 **Documentation:** \n", + "- 💻 **GitHub repository:** \n", + "- 🐛 **Issues:** \n", + "- 💬 **Discussions:** \n", + "- 📝 **Changelog:** \n", + "- 🤝 **Contributing:** \n", + "- 📚 **More example notebooks:** [`examples/`](https://github.com/OpenAgentHQ/openagent-eval/tree/main/examples)\n", + " — see `rag_evaluation_tutorial.ipynb` and `corpus_and_related_modules.ipynb` for deeper dives.\n", + "\n", + "**Related tools** in the RAG-evaluation space: RAGAS, DeepEval, TruLens. OpenAgent Eval's focus is a\n", + "**local-first, CLI-friendly** workflow with corpus auditing and blame attribution built in." + ] + }, + { + "cell_type": "markdown", + "id": "2d489777", + "metadata": {}, + "source": [ + "\n", + "## 16. Feedback & credit\n", + "\n", + "Was this notebook helpful? ⭐ **[Star OpenAgent Eval on GitHub](https://github.com/OpenAgentHQ/openagent-eval)**\n", + "— it genuinely helps the project.\n", + "\n", + "- 🐞 Found a rough edge? [Open an issue](https://github.com/OpenAgentHQ/openagent-eval/issues).\n", + "- 💡 Have an idea to improve this tutorial? PRs to [`examples/`](https://github.com/OpenAgentHQ/openagent-eval/tree/main/examples)\n", + " are very welcome.\n", + "\n", + "Thanks for learning OpenAgent Eval! 🎉" + ] + }, + { + "cell_type": "markdown", + "id": "ca99730a", + "metadata": {}, + "source": [ + "\n", + "## 17. Optional — using real API keys 🔐\n", + "\n", + "Everything above ran **offline** with the `mock` providers. To evaluate a *real* RAG system, point\n", + "OpenAgent Eval at a real LLM. **These cells are optional** — the notebook is complete without them,\n", + "and each is written to **skip gracefully when no key is present**, so *Run all* stays green.\n", + "\n", + "> 🔒 **Never hard-code a key.** Use `getpass` (prompts you, hidden input) or Colab's **Secrets\n", + "> Manager** (the 🔑 icon in the left sidebar), and read it from the environment." + ] + }, + { + "cell_type": "code", + "execution_count": 32, + "id": "3fb445c8", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-23T20:02:13.348233Z", + "iopub.status.busy": "2026-07-23T20:02:13.341525Z", + "iopub.status.idle": "2026-07-23T20:02:13.374230Z", + "shell.execute_reply": "2026-07-23T20:02:13.373007Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "(No interactive input available — staying offline.)\n", + "OpenAI key set: False\n" + ] + } + ], + "source": [ + "# OPTIONAL — set an API key for this session without hard-coding it.\n", + "# Skips silently if you don't enter one (just press Enter to skip).\n", + "import os, getpass\n", + "\n", + "# Preferred in Colab: read from the Secrets Manager (🔑 icon) if you've stored one there.\n", + "try:\n", + " from google.colab import userdata # type: ignore\n", + " _secret = userdata.get(\"OPENAI_API_KEY\")\n", + " if _secret:\n", + " os.environ[\"OPENAI_API_KEY\"] = _secret\n", + "except Exception:\n", + " pass\n", + "\n", + "# Fallback: prompt (hidden). Press Enter to skip and stay fully offline.\n", + "# Wrapped so a non-interactive \"Run all\" (no keyboard) skips instead of erroring.\n", + "if not os.environ.get(\"OPENAI_API_KEY\"):\n", + " try:\n", + " entered = getpass.getpass(\"OPENAI_API_KEY (press Enter to skip): \")\n", + " if entered.strip():\n", + " os.environ[\"OPENAI_API_KEY\"] = entered.strip()\n", + " except Exception:\n", + " print(\"(No interactive input available — staying offline.)\")\n", + "\n", + "print(\"OpenAI key set:\", bool(os.environ.get(\"OPENAI_API_KEY\")))" + ] + }, + { + "cell_type": "markdown", + "id": "44865ae6", + "metadata": {}, + "source": [ + "With a key set, swap `provider: mock` for a real provider in your config and run as before.\n", + "The next cell **only runs a real evaluation if a key is present** — otherwise it explains how to\n", + "enable it and does nothing (so it's safe in *Run all*)." + ] + }, + { + "cell_type": "code", + "execution_count": 33, + "id": "05ab7dcb", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-23T20:02:13.403755Z", + "iopub.status.busy": "2026-07-23T20:02:13.403409Z", + "iopub.status.idle": "2026-07-23T20:02:13.425182Z", + "shell.execute_reply": "2026-07-23T20:02:13.419897Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "No OPENAI_API_KEY set — skipping the real run.\n", + "Set a key in the cell above (or Colab Secrets) to try it, then re-run this cell.\n" + ] + } + ], + "source": [ + "# OPTIONAL — real-provider evaluation. Guarded: no key → no-op.\n", + "if os.environ.get(\"OPENAI_API_KEY\"):\n", + " real_config = \"\"\"\n", + "dataset:\n", + " path: data.json\n", + " format: json\n", + "llm:\n", + " provider: openai\n", + " model: gpt-4o-mini\n", + " api_key: ${OPENAI_API_KEY}\n", + "retriever:\n", + " provider: mock # swap for chroma/faiss/... when you have a real corpus\n", + " settings:\n", + " collection_name: demo\n", + "metrics:\n", + " generation: [faithfulness, answer_relevancy]\n", + " performance: [latency]\n", + " cost: [token_count]\n", + "report:\n", + " output: terminal\n", + " output_dir: ./reports\n", + "\"\"\"\n", + " with open(\"config_real.yaml\", \"w\") as f:\n", + " f.write(real_config)\n", + " print(\"Wrote config_real.yaml — running against OpenAI...\")\n", + " import subprocess\n", + " subprocess.run([\"oaeval\", \"run\", \"config_real.yaml\"], check=False)\n", + "else:\n", + " print(\"No OPENAI_API_KEY set — skipping the real run.\")\n", + " print(\"Set a key in the cell above (or Colab Secrets) to try it, then re-run this cell.\")" + ] + }, + { + "cell_type": "markdown", + "id": "21243e4c", + "metadata": {}, + "source": [ + "---\n", + "\n", + "*You made it! 🎉 You installed OpenAgent Eval, ran a full evaluation offline, audited a corpus,\n", + "diagnosed failures, compared experiments, used the SDK, built a custom metric, and saw how to gate\n", + "CI — all without a single API key. Happy evaluating!*" + ] + } + ], + "metadata": { + "colab": { + "provenance": [], + "toc_visible": true + }, + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.13.14" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/examples/rag_evaluation_tutorial.ipynb b/examples/rag_evaluation_tutorial.ipynb index a21dfa7..cca583d 100644 --- a/examples/rag_evaluation_tutorial.ipynb +++ b/examples/rag_evaluation_tutorial.ipynb @@ -181,7 +181,7 @@ }, { "cell_type": "code", - "execution_count": 11, + "execution_count": 1, "id": "cell-003", "metadata": {}, "outputs": [ @@ -190,7 +190,7 @@ "output_type": "stream", "text": [ "\n", - ":128: RuntimeWarning: 'openagent_eval.cli.main' found in sys.modules after import of package 'openagent_eval.cli', but prior to execution of 'openagent_eval.cli.main'; this may result in unpredictable behaviour\n", + ":130: RuntimeWarning: 'openagent_eval.cli.main' found in sys.modules after import of package 'openagent_eval.cli', but prior to execution of 'openagent_eval.cli.main'; this may result in unpredictable behaviour\n", "\n" ] } @@ -200,7 +200,7 @@ "import subprocess\n", "import sys\n", "\n", - "subprocess.run([sys.executable, \"-m\", \"pip\", \"install\", \"openagent-eval\"], check=True)\n", + "subprocess.run([sys.executable, \"-m\", \"pip\", \"install\", \"-q\", \"openagent-eval\", \"pytest\"], check=True)\n", "\n", "result = subprocess.run(\n", " [sys.executable, '-m', 'openagent_eval.cli.main', 'doctor'],\n", @@ -389,7 +389,7 @@ }, { "cell_type": "code", - "execution_count": 12, + "execution_count": 3, "id": "cell-008", "metadata": {}, "outputs": [ @@ -521,7 +521,7 @@ }, { "cell_type": "code", - "execution_count": 14, + "execution_count": 4, "id": "cell-010", "metadata": {}, "outputs": [ @@ -564,7 +564,7 @@ }, { "cell_type": "code", - "execution_count": 16, + "execution_count": 5, "id": "cell-012", "metadata": {}, "outputs": [ @@ -574,7 +574,7 @@ "text": [ "Answer: RAG combines retrieval with generation for grounded answers.\n", "Tokens: 11\n", - "Latency: 0.01ms\n" + "Latency: 0.00ms\n" ] } ], @@ -619,7 +619,7 @@ }, { "cell_type": "code", - "execution_count": 18, + "execution_count": 6, "id": "cell-014", "metadata": {}, "outputs": [ @@ -631,7 +631,7 @@ "Answer: [mock-answer] Context:\n", "[1] Retrieval-Augmented Generation (RAG) is a technique that enhances large language models by retrieving relevant documents fr...\n", "Contexts retrieved: 3\n", - "Total latency: 1.04ms\n", + "Total latency: 0.34ms\n", "Tokens: 247\n" ] } @@ -738,7 +738,7 @@ }, { "cell_type": "code", - "execution_count": 23, + "execution_count": 7, "id": "cell-017", "metadata": {}, "outputs": [ @@ -887,7 +887,7 @@ }, { "cell_type": "code", - "execution_count": 22, + "execution_count": 8, "id": "cell-020", "metadata": {}, "outputs": [ @@ -988,7 +988,7 @@ }, { "cell_type": "code", - "execution_count": 21, + "execution_count": 9, "id": "cell-022", "metadata": {}, "outputs": [ @@ -1053,7 +1053,7 @@ }, { "cell_type": "code", - "execution_count": 24, + "execution_count": 10, "id": "cell-024", "metadata": {}, "outputs": [ @@ -1143,7 +1143,7 @@ }, { "cell_type": "code", - "execution_count": 25, + "execution_count": 11, "id": "cell-026", "metadata": {}, "outputs": [ @@ -1151,258 +1151,176 @@ "name": "stderr", "output_type": "stream", "text": [ - "/usr/local/lib/python3.12/dist-packages/huggingface_hub/utils/_auth.py:138: UserWarning: \n", - "Error while fetching `HF_TOKEN` secret value from your vault: 'Requesting secret HF_TOKEN timed out. Secrets can only be fetched when running from the Colab UI.'.\n", - "You are not authenticated with the Hugging Face Hub in this notebook.\n", - "If the error persists, please let us know by opening an issue on GitHub (https://github.com/huggingface/huggingface_hub/issues/new).\n", - " warnings.warn(\n" - ] - }, - { - "data": { - "application/vnd.jupyter.widget-view+json": { - "model_id": "e8bad5fe238c4b41ad7edbb7c3b71759", - "version_major": 2, - "version_minor": 0 - }, - "text/plain": [ - "modules.json: 0%| | 0.00/349 [00:006} {'Recall':>6} {'MRR':>6} {'Hit@K':>6} \"\n", - " f\"{'Faith':>6} {'Rel':>6} {'Latency':>8} {'Tokens':>8}\"\n", - ")\n", - "print(header)\n", - "print(\"-\" * 70)\n", - "for r in batch_results:\n", - " print(\n", + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.086\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.generation.faithfulness\u001b[0m:\u001b[36mevaluate\u001b[0m:\u001b[36m70\u001b[0m - \u001b[34m\u001b[1mNLI scoring unavailable, using simple overlap: No module named 'transformers'\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.099\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.nli.judge\u001b[0m:\u001b[36mpipeline\u001b[0m:\u001b[36m88\u001b[0m - \u001b[34m\u001b[1mLoading NLI model: microsoft/deberta-v3-base-mnli\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.106\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.generation.relevancy\u001b[0m:\u001b[36mevaluate\u001b[0m:\u001b[36m71\u001b[0m - \u001b[34m\u001b[1mNLI scoring unavailable, using simple overlap: No module named 'transformers'\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.124\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.nli.judge\u001b[0m:\u001b[36mpipeline\u001b[0m:\u001b[36m88\u001b[0m - \u001b[34m\u001b[1mLoading NLI model: microsoft/deberta-v3-base-mnli\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.137\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.generation.faithfulness\u001b[0m:\u001b[36mevaluate\u001b[0m:\u001b[36m70\u001b[0m - \u001b[34m\u001b[1mNLI scoring unavailable, using simple overlap: No module named 'transformers'\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.146\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.nli.judge\u001b[0m:\u001b[36mpipeline\u001b[0m:\u001b[36m88\u001b[0m - \u001b[34m\u001b[1mLoading NLI model: microsoft/deberta-v3-base-mnli\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.158\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.generation.relevancy\u001b[0m:\u001b[36mevaluate\u001b[0m:\u001b[36m71\u001b[0m - \u001b[34m\u001b[1mNLI scoring unavailable, using simple overlap: No module named 'transformers'\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.159\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.nli.judge\u001b[0m:\u001b[36mpipeline\u001b[0m:\u001b[36m88\u001b[0m - \u001b[34m\u001b[1mLoading NLI model: microsoft/deberta-v3-base-mnli\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.163\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.generation.faithfulness\u001b[0m:\u001b[36mevaluate\u001b[0m:\u001b[36m70\u001b[0m - \u001b[34m\u001b[1mNLI scoring unavailable, using simple overlap: No module named 'transformers'\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.164\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.nli.judge\u001b[0m:\u001b[36mpipeline\u001b[0m:\u001b[36m88\u001b[0m - \u001b[34m\u001b[1mLoading NLI model: microsoft/deberta-v3-base-mnli\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.164\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.generation.relevancy\u001b[0m:\u001b[36mevaluate\u001b[0m:\u001b[36m71\u001b[0m - \u001b[34m\u001b[1mNLI scoring unavailable, using simple overlap: No module named 'transformers'\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.166\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.nli.judge\u001b[0m:\u001b[36mpipeline\u001b[0m:\u001b[36m88\u001b[0m - \u001b[34m\u001b[1mLoading NLI model: microsoft/deberta-v3-base-mnli\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.167\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.generation.faithfulness\u001b[0m:\u001b[36mevaluate\u001b[0m:\u001b[36m70\u001b[0m - \u001b[34m\u001b[1mNLI scoring unavailable, using simple overlap: No module named 'transformers'\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.168\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.nli.judge\u001b[0m:\u001b[36mpipeline\u001b[0m:\u001b[36m88\u001b[0m - \u001b[34m\u001b[1mLoading NLI model: microsoft/deberta-v3-base-mnli\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.173\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.generation.relevancy\u001b[0m:\u001b[36mevaluate\u001b[0m:\u001b[36m71\u001b[0m - \u001b[34m\u001b[1mNLI scoring unavailable, using simple overlap: No module named 'transformers'\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.174\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.nli.judge\u001b[0m:\u001b[36mpipeline\u001b[0m:\u001b[36m88\u001b[0m - \u001b[34m\u001b[1mLoading NLI model: microsoft/deberta-v3-base-mnli\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.175\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.generation.faithfulness\u001b[0m:\u001b[36mevaluate\u001b[0m:\u001b[36m70\u001b[0m - \u001b[34m\u001b[1mNLI scoring unavailable, using simple overlap: No module named 'transformers'\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.180\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.nli.judge\u001b[0m:\u001b[36mpipeline\u001b[0m:\u001b[36m88\u001b[0m - \u001b[34m\u001b[1mLoading NLI model: microsoft/deberta-v3-base-mnli\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.181\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.generation.relevancy\u001b[0m:\u001b[36mevaluate\u001b[0m:\u001b[36m71\u001b[0m - \u001b[34m\u001b[1mNLI scoring unavailable, using simple overlap: No module named 'transformers'\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.183\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.nli.judge\u001b[0m:\u001b[36mpipeline\u001b[0m:\u001b[36m88\u001b[0m - \u001b[34m\u001b[1mLoading NLI model: microsoft/deberta-v3-base-mnli\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.184\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.generation.faithfulness\u001b[0m:\u001b[36mevaluate\u001b[0m:\u001b[36m70\u001b[0m - \u001b[34m\u001b[1mNLI scoring unavailable, using simple overlap: No module named 'transformers'\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.184\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.nli.judge\u001b[0m:\u001b[36mpipeline\u001b[0m:\u001b[36m88\u001b[0m - \u001b[34m\u001b[1mLoading NLI model: microsoft/deberta-v3-base-mnli\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.185\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.generation.relevancy\u001b[0m:\u001b[36mevaluate\u001b[0m:\u001b[36m71\u001b[0m - \u001b[34m\u001b[1mNLI scoring unavailable, using simple overlap: No module named 'transformers'\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.186\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.nli.judge\u001b[0m:\u001b[36mpipeline\u001b[0m:\u001b[36m88\u001b[0m - \u001b[34m\u001b[1mLoading NLI model: microsoft/deberta-v3-base-mnli\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.187\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.generation.faithfulness\u001b[0m:\u001b[36mevaluate\u001b[0m:\u001b[36m70\u001b[0m - \u001b[34m\u001b[1mNLI scoring unavailable, using simple overlap: No module named 'transformers'\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.188\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.nli.judge\u001b[0m:\u001b[36mpipeline\u001b[0m:\u001b[36m88\u001b[0m - \u001b[34m\u001b[1mLoading NLI model: microsoft/deberta-v3-base-mnli\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.189\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.generation.relevancy\u001b[0m:\u001b[36mevaluate\u001b[0m:\u001b[36m71\u001b[0m - \u001b[34m\u001b[1mNLI scoring unavailable, using simple overlap: No module named 'transformers'\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.190\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.nli.judge\u001b[0m:\u001b[36mpipeline\u001b[0m:\u001b[36m88\u001b[0m - \u001b[34m\u001b[1mLoading NLI model: microsoft/deberta-v3-base-mnli\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.191\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.generation.faithfulness\u001b[0m:\u001b[36mevaluate\u001b[0m:\u001b[36m70\u001b[0m - \u001b[34m\u001b[1mNLI scoring unavailable, using simple overlap: No module named 'transformers'\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.192\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.nli.judge\u001b[0m:\u001b[36mpipeline\u001b[0m:\u001b[36m88\u001b[0m - \u001b[34m\u001b[1mLoading NLI model: microsoft/deberta-v3-base-mnli\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.193\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.generation.relevancy\u001b[0m:\u001b[36mevaluate\u001b[0m:\u001b[36m71\u001b[0m - \u001b[34m\u001b[1mNLI scoring unavailable, using simple overlap: No module named 'transformers'\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.194\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.nli.judge\u001b[0m:\u001b[36mpipeline\u001b[0m:\u001b[36m88\u001b[0m - \u001b[34m\u001b[1mLoading NLI model: microsoft/deberta-v3-base-mnli\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.195\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.generation.faithfulness\u001b[0m:\u001b[36mevaluate\u001b[0m:\u001b[36m70\u001b[0m - \u001b[34m\u001b[1mNLI scoring unavailable, using simple overlap: No module named 'transformers'\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.195\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.nli.judge\u001b[0m:\u001b[36mpipeline\u001b[0m:\u001b[36m88\u001b[0m - \u001b[34m\u001b[1mLoading NLI model: microsoft/deberta-v3-base-mnli\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.196\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.generation.relevancy\u001b[0m:\u001b[36mevaluate\u001b[0m:\u001b[36m71\u001b[0m - \u001b[34m\u001b[1mNLI scoring unavailable, using simple overlap: No module named 'transformers'\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.197\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.nli.judge\u001b[0m:\u001b[36mpipeline\u001b[0m:\u001b[36m88\u001b[0m - \u001b[34m\u001b[1mLoading NLI model: microsoft/deberta-v3-base-mnli\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.199\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.generation.faithfulness\u001b[0m:\u001b[36mevaluate\u001b[0m:\u001b[36m70\u001b[0m - \u001b[34m\u001b[1mNLI scoring unavailable, using simple overlap: No module named 'transformers'\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.200\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.nli.judge\u001b[0m:\u001b[36mpipeline\u001b[0m:\u001b[36m88\u001b[0m - \u001b[34m\u001b[1mLoading NLI model: microsoft/deberta-v3-base-mnli\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.200\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.generation.relevancy\u001b[0m:\u001b[36mevaluate\u001b[0m:\u001b[36m71\u001b[0m - \u001b[34m\u001b[1mNLI scoring unavailable, using simple overlap: No module named 'transformers'\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.201\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.nli.judge\u001b[0m:\u001b[36mpipeline\u001b[0m:\u001b[36m88\u001b[0m - \u001b[34m\u001b[1mLoading NLI model: microsoft/deberta-v3-base-mnli\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.203\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.generation.faithfulness\u001b[0m:\u001b[36mevaluate\u001b[0m:\u001b[36m70\u001b[0m - \u001b[34m\u001b[1mNLI scoring unavailable, using simple overlap: No module named 'transformers'\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.203\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.nli.judge\u001b[0m:\u001b[36mpipeline\u001b[0m:\u001b[36m88\u001b[0m - \u001b[34m\u001b[1mLoading NLI model: microsoft/deberta-v3-base-mnli\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.204\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.generation.relevancy\u001b[0m:\u001b[36mevaluate\u001b[0m:\u001b[36m71\u001b[0m - \u001b[34m\u001b[1mNLI scoring unavailable, using simple overlap: No module named 'transformers'\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.205\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.nli.judge\u001b[0m:\u001b[36mpipeline\u001b[0m:\u001b[36m88\u001b[0m - \u001b[34m\u001b[1mLoading NLI model: microsoft/deberta-v3-base-mnli\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.206\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.generation.faithfulness\u001b[0m:\u001b[36mevaluate\u001b[0m:\u001b[36m70\u001b[0m - \u001b[34m\u001b[1mNLI scoring unavailable, using simple overlap: No module named 'transformers'\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.206\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.nli.judge\u001b[0m:\u001b[36mpipeline\u001b[0m:\u001b[36m88\u001b[0m - \u001b[34m\u001b[1mLoading NLI model: microsoft/deberta-v3-base-mnli\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.207\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.generation.relevancy\u001b[0m:\u001b[36mevaluate\u001b[0m:\u001b[36m71\u001b[0m - \u001b[34m\u001b[1mNLI scoring unavailable, using simple overlap: No module named 'transformers'\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.207\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.nli.judge\u001b[0m:\u001b[36mpipeline\u001b[0m:\u001b[36m88\u001b[0m - \u001b[34m\u001b[1mLoading NLI model: microsoft/deberta-v3-base-mnli\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.208\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.generation.faithfulness\u001b[0m:\u001b[36mevaluate\u001b[0m:\u001b[36m70\u001b[0m - \u001b[34m\u001b[1mNLI scoring unavailable, using simple overlap: No module named 'transformers'\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.209\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.nli.judge\u001b[0m:\u001b[36mpipeline\u001b[0m:\u001b[36m88\u001b[0m - \u001b[34m\u001b[1mLoading NLI model: microsoft/deberta-v3-base-mnli\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.209\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.generation.relevancy\u001b[0m:\u001b[36mevaluate\u001b[0m:\u001b[36m71\u001b[0m - \u001b[34m\u001b[1mNLI scoring unavailable, using simple overlap: No module named 'transformers'\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.210\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.nli.judge\u001b[0m:\u001b[36mpipeline\u001b[0m:\u001b[36m88\u001b[0m - \u001b[34m\u001b[1mLoading NLI model: microsoft/deberta-v3-base-mnli\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.211\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.generation.faithfulness\u001b[0m:\u001b[36mevaluate\u001b[0m:\u001b[36m70\u001b[0m - \u001b[34m\u001b[1mNLI scoring unavailable, using simple overlap: No module named 'transformers'\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.212\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.nli.judge\u001b[0m:\u001b[36mpipeline\u001b[0m:\u001b[36m88\u001b[0m - \u001b[34m\u001b[1mLoading NLI model: microsoft/deberta-v3-base-mnli\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.212\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.generation.relevancy\u001b[0m:\u001b[36mevaluate\u001b[0m:\u001b[36m71\u001b[0m - \u001b[34m\u001b[1mNLI scoring unavailable, using simple overlap: No module named 'transformers'\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.213\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.nli.judge\u001b[0m:\u001b[36mpipeline\u001b[0m:\u001b[36m88\u001b[0m - \u001b[34m\u001b[1mLoading NLI model: microsoft/deberta-v3-base-mnli\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.214\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.generation.faithfulness\u001b[0m:\u001b[36mevaluate\u001b[0m:\u001b[36m70\u001b[0m - \u001b[34m\u001b[1mNLI scoring unavailable, using simple overlap: No module named 'transformers'\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.215\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.nli.judge\u001b[0m:\u001b[36mpipeline\u001b[0m:\u001b[36m88\u001b[0m - \u001b[34m\u001b[1mLoading NLI model: microsoft/deberta-v3-base-mnli\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.216\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.generation.relevancy\u001b[0m:\u001b[36mevaluate\u001b[0m:\u001b[36m71\u001b[0m - \u001b[34m\u001b[1mNLI scoring unavailable, using simple overlap: No module named 'transformers'\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.217\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.nli.judge\u001b[0m:\u001b[36mpipeline\u001b[0m:\u001b[36m88\u001b[0m - \u001b[34m\u001b[1mLoading NLI model: microsoft/deberta-v3-base-mnli\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.218\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.generation.faithfulness\u001b[0m:\u001b[36mevaluate\u001b[0m:\u001b[36m70\u001b[0m - \u001b[34m\u001b[1mNLI scoring unavailable, using simple overlap: No module named 'transformers'\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.219\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.nli.judge\u001b[0m:\u001b[36mpipeline\u001b[0m:\u001b[36m88\u001b[0m - \u001b[34m\u001b[1mLoading NLI model: microsoft/deberta-v3-base-mnli\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.219\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.generation.relevancy\u001b[0m:\u001b[36mevaluate\u001b[0m:\u001b[36m71\u001b[0m - \u001b[34m\u001b[1mNLI scoring unavailable, using simple overlap: No module named 'transformers'\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.220\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.nli.judge\u001b[0m:\u001b[36mpipeline\u001b[0m:\u001b[36m88\u001b[0m - \u001b[34m\u001b[1mLoading NLI model: microsoft/deberta-v3-base-mnli\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.221\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.generation.faithfulness\u001b[0m:\u001b[36mevaluate\u001b[0m:\u001b[36m70\u001b[0m - \u001b[34m\u001b[1mNLI scoring unavailable, using simple overlap: No module named 'transformers'\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.222\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.nli.judge\u001b[0m:\u001b[36mpipeline\u001b[0m:\u001b[36m88\u001b[0m - \u001b[34m\u001b[1mLoading NLI model: microsoft/deberta-v3-base-mnli\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.223\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.generation.relevancy\u001b[0m:\u001b[36mevaluate\u001b[0m:\u001b[36m71\u001b[0m - \u001b[34m\u001b[1mNLI scoring unavailable, using simple overlap: No module named 'transformers'\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.224\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.nli.judge\u001b[0m:\u001b[36mpipeline\u001b[0m:\u001b[36m88\u001b[0m - \u001b[34m\u001b[1mLoading NLI model: microsoft/deberta-v3-base-mnli\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.225\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.generation.faithfulness\u001b[0m:\u001b[36mevaluate\u001b[0m:\u001b[36m70\u001b[0m - \u001b[34m\u001b[1mNLI scoring unavailable, using simple overlap: No module named 'transformers'\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.225\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.nli.judge\u001b[0m:\u001b[36mpipeline\u001b[0m:\u001b[36m88\u001b[0m - \u001b[34m\u001b[1mLoading NLI model: microsoft/deberta-v3-base-mnli\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.226\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.generation.relevancy\u001b[0m:\u001b[36mevaluate\u001b[0m:\u001b[36m71\u001b[0m - \u001b[34m\u001b[1mNLI scoring unavailable, using simple overlap: No module named 'transformers'\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.226\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.nli.judge\u001b[0m:\u001b[36mpipeline\u001b[0m:\u001b[36m88\u001b[0m - \u001b[34m\u001b[1mLoading NLI model: microsoft/deberta-v3-base-mnli\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.228\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.generation.faithfulness\u001b[0m:\u001b[36mevaluate\u001b[0m:\u001b[36m70\u001b[0m - \u001b[34m\u001b[1mNLI scoring unavailable, using simple overlap: No module named 'transformers'\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.246\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.nli.judge\u001b[0m:\u001b[36mpipeline\u001b[0m:\u001b[36m88\u001b[0m - \u001b[34m\u001b[1mLoading NLI model: microsoft/deberta-v3-base-mnli\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.285\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.generation.relevancy\u001b[0m:\u001b[36mevaluate\u001b[0m:\u001b[36m71\u001b[0m - \u001b[34m\u001b[1mNLI scoring unavailable, using simple overlap: No module named 'transformers'\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.287\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.nli.judge\u001b[0m:\u001b[36mpipeline\u001b[0m:\u001b[36m88\u001b[0m - \u001b[34m\u001b[1mLoading NLI model: microsoft/deberta-v3-base-mnli\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.288\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.generation.faithfulness\u001b[0m:\u001b[36mevaluate\u001b[0m:\u001b[36m70\u001b[0m - \u001b[34m\u001b[1mNLI scoring unavailable, using simple overlap: No module named 'transformers'\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.289\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.nli.judge\u001b[0m:\u001b[36mpipeline\u001b[0m:\u001b[36m88\u001b[0m - \u001b[34m\u001b[1mLoading NLI model: microsoft/deberta-v3-base-mnli\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.291\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.generation.relevancy\u001b[0m:\u001b[36mevaluate\u001b[0m:\u001b[36m71\u001b[0m - \u001b[34m\u001b[1mNLI scoring unavailable, using simple overlap: No module named 'transformers'\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Config Prec Recall MRR Hit@K Faith Rel Latency Tokens\n", + "----------------------------------------------------------------------\n", + "k=1 1.000 0.333 1.000 1.000 0.373 0.446 0.0 287\n", + "k=3 1.000 1.000 1.000 1.000 0.505 0.446 0.0 496\n", + "k=5 1.000 1.000 1.000 1.000 0.505 0.446 0.0 496\n", + "k=10 1.000 1.000 1.000 1.000 0.505 0.446 0.0 496\n" + ] + } + ], + "source": [ + "# Batch evaluation: Compare multiple retriever configurations\n", + "from openagent_eval.config.models import (\n", + " Config, RetrieverConfig, LLMConfig, MetricsConfig, ReportConfig, OutputFormat,\n", + ")\n", + "from openagent_eval.core.engine import Engine\n", + "from openagent_eval.providers.retrievers.mock import MockRetriever\n", + "\n", + "\n", + "async def evaluate_with_config(name, retriever, k):\n", + " \"\"\"Run evaluation with a specific retriever configuration.\"\"\"\n", + " config = Config(\n", + " dataset=DatasetConfig(path=\"data/sample_questions.json\", format=\"json\"),\n", + " llm=LLMConfig(provider=\"mock\", model=\"mock-model\", temperature=0.0),\n", + " retriever=RetrieverConfig(provider=\"mock\", settings={\"k\": k}),\n", + " metrics=MetricsConfig(\n", + " retrieval=[\"context_precision\", \"context_recall\", \"mrr\", \"hit_rate\"],\n", + " generation=[\"faithfulness\", \"answer_relevancy\"],\n", + " performance=[\"latency\"],\n", + " cost=[\"token_count\"],\n", + " ),\n", + " report=ReportConfig(output=OutputFormat.JSON, output_dir=\"./reports\"),\n", + " parallel=True,\n", + " max_workers=4,\n", + " )\n", + "\n", + " engine = Engine(config=config, retriever=retriever, llm=mock_llm)\n", + " report = await engine.run(EVAL_DATASET)\n", + "\n", + " metrics = report.summary.get(\"metrics_summary\", {})\n", + " return {\n", + " \"name\": name,\n", + " \"k\": k,\n", + " \"context_precision\": metrics.get(\"context_precision\", 0),\n", + " \"context_recall\": metrics.get(\"context_recall\", 0),\n", + " \"mrr\": metrics.get(\"mrr\", 0),\n", + " \"hit_rate\": metrics.get(\"hit_rate\", 0),\n", + " \"faithfulness\": metrics.get(\"faithfulness\", 0),\n", + " \"answer_relevancy\": metrics.get(\"answer_relevancy\", 0),\n", + " \"avg_latency_ms\": report.summary.get(\"average_latency_ms\", 0),\n", + " \"total_tokens\": report.summary.get(\"total_tokens\", 0),\n", + " }\n", + "\n", + "\n", + "async def run_batch_comparison():\n", + " results = []\n", + " for k in [1, 3, 5, 10]:\n", + " # Mock retriever that returns ground-truth contexts for testing\n", + " mock_retriever = MockRetriever(collection_name=f\"test_k{k}\")\n", + " result = await evaluate_with_config(f\"k={k}\", mock_retriever, k)\n", + " results.append(result)\n", + " return results\n", + "\n", + "\n", + "batch_results = await run_batch_comparison()\n", + "\n", + "# Display comparison table\n", + "header = (\n", + " f\"{'Config':<10} {'Prec':>6} {'Recall':>6} {'MRR':>6} {'Hit@K':>6} \"\n", + " f\"{'Faith':>6} {'Rel':>6} {'Latency':>8} {'Tokens':>8}\"\n", + ")\n", + "print(header)\n", + "print(\"-\" * 70)\n", + "for r in batch_results:\n", + " print(\n", " f\"{r['name']:<10} {r['context_precision']:>6.3f} \"\n", " f\"{r['context_recall']:>6.3f} {r['mrr']:>6.3f} {r['hit_rate']:>6.3f} \"\n", " f\"{r['faithfulness']:>6.3f} {r['answer_relevancy']:>6.3f} \"\n", @@ -2685,10 +3163,150 @@ }, { "cell_type": "code", - "execution_count": 28, + "execution_count": 19, "id": "cell-045", "metadata": {}, "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.332\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.nli.judge\u001b[0m:\u001b[36mpipeline\u001b[0m:\u001b[36m88\u001b[0m - \u001b[34m\u001b[1mLoading NLI model: microsoft/deberta-v3-base-mnli\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.341\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.generation.faithfulness\u001b[0m:\u001b[36mevaluate\u001b[0m:\u001b[36m70\u001b[0m - \u001b[34m\u001b[1mNLI scoring unavailable, using simple overlap: No module named 'transformers'\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.344\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.nli.judge\u001b[0m:\u001b[36mpipeline\u001b[0m:\u001b[36m88\u001b[0m - \u001b[34m\u001b[1mLoading NLI model: microsoft/deberta-v3-base-mnli\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.345\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.generation.relevancy\u001b[0m:\u001b[36mevaluate\u001b[0m:\u001b[36m71\u001b[0m - \u001b[34m\u001b[1mNLI scoring unavailable, using simple overlap: No module named 'transformers'\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.347\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.nli.judge\u001b[0m:\u001b[36mpipeline\u001b[0m:\u001b[36m88\u001b[0m - \u001b[34m\u001b[1mLoading NLI model: microsoft/deberta-v3-base-mnli\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.348\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.generation.faithfulness\u001b[0m:\u001b[36mevaluate\u001b[0m:\u001b[36m70\u001b[0m - \u001b[34m\u001b[1mNLI scoring unavailable, using simple overlap: No module named 'transformers'\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.349\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.nli.judge\u001b[0m:\u001b[36mpipeline\u001b[0m:\u001b[36m88\u001b[0m - \u001b[34m\u001b[1mLoading NLI model: microsoft/deberta-v3-base-mnli\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.350\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.generation.relevancy\u001b[0m:\u001b[36mevaluate\u001b[0m:\u001b[36m71\u001b[0m - \u001b[34m\u001b[1mNLI scoring unavailable, using simple overlap: No module named 'transformers'\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.351\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.nli.judge\u001b[0m:\u001b[36mpipeline\u001b[0m:\u001b[36m88\u001b[0m - \u001b[34m\u001b[1mLoading NLI model: microsoft/deberta-v3-base-mnli\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.352\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.generation.faithfulness\u001b[0m:\u001b[36mevaluate\u001b[0m:\u001b[36m70\u001b[0m - \u001b[34m\u001b[1mNLI scoring unavailable, using simple overlap: No module named 'transformers'\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.353\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.nli.judge\u001b[0m:\u001b[36mpipeline\u001b[0m:\u001b[36m88\u001b[0m - \u001b[34m\u001b[1mLoading NLI model: microsoft/deberta-v3-base-mnli\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.353\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.generation.relevancy\u001b[0m:\u001b[36mevaluate\u001b[0m:\u001b[36m71\u001b[0m - \u001b[34m\u001b[1mNLI scoring unavailable, using simple overlap: No module named 'transformers'\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.354\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.nli.judge\u001b[0m:\u001b[36mpipeline\u001b[0m:\u001b[36m88\u001b[0m - \u001b[34m\u001b[1mLoading NLI model: microsoft/deberta-v3-base-mnli\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.355\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.generation.faithfulness\u001b[0m:\u001b[36mevaluate\u001b[0m:\u001b[36m70\u001b[0m - \u001b[34m\u001b[1mNLI scoring unavailable, using simple overlap: No module named 'transformers'\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.356\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.nli.judge\u001b[0m:\u001b[36mpipeline\u001b[0m:\u001b[36m88\u001b[0m - \u001b[34m\u001b[1mLoading NLI model: microsoft/deberta-v3-base-mnli\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.357\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.generation.relevancy\u001b[0m:\u001b[36mevaluate\u001b[0m:\u001b[36m71\u001b[0m - \u001b[34m\u001b[1mNLI scoring unavailable, using simple overlap: No module named 'transformers'\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.358\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.nli.judge\u001b[0m:\u001b[36mpipeline\u001b[0m:\u001b[36m88\u001b[0m - \u001b[34m\u001b[1mLoading NLI model: microsoft/deberta-v3-base-mnli\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.359\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.generation.faithfulness\u001b[0m:\u001b[36mevaluate\u001b[0m:\u001b[36m70\u001b[0m - \u001b[34m\u001b[1mNLI scoring unavailable, using simple overlap: No module named 'transformers'\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.360\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.nli.judge\u001b[0m:\u001b[36mpipeline\u001b[0m:\u001b[36m88\u001b[0m - \u001b[34m\u001b[1mLoading NLI model: microsoft/deberta-v3-base-mnli\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2026-07-23 22:45:27.361\u001b[0m | \u001b[34m\u001b[1mDEBUG \u001b[0m | \u001b[36mopenagent_eval.metrics.generation.relevancy\u001b[0m:\u001b[36mevaluate\u001b[0m:\u001b[36m71\u001b[0m - \u001b[34m\u001b[1mNLI scoring unavailable, using simple overlap: No module named 'transformers'\u001b[0m\n" + ] + }, { "name": "stdout", "output_type": "stream", @@ -2697,7 +3315,7 @@ " Experiment Comparison Report\n", "============================================================\n", "\n", - " Generated: 2026-07-10 10:40:03 UTC\n", + " Generated: 2026-07-23 20:45:27 UTC\n", " Baseline: Experiment 1 (k=3)\n", " Experiment: Experiment 2 (k=5)\n", "\n", @@ -2706,7 +3324,7 @@ " Metric Baseline Experiment Delta\n", " ----------------------------------------------------------\n", " answer_relevancy 0.4460 0.4460 = +0.0000\n", - " bertscore 0.6000 0.0000 -0.6000\n", + " bertscore 1.0000 0.0000 -1.0000\n", " bleu 1.0000 0.0000 -1.0000\n", " context_precision 0.0000 1.0000 + +1.0000\n", " context_recall 0.0000 1.0000 + +1.0000\n", @@ -2721,7 +3339,7 @@ " precision_at_k 0.0000 0.0000 = +0.0000\n", " recall_at_k 0.0000 0.0000 = +0.0000\n", " rouge 1.0000 0.0000 -1.0000\n", - " semantic_similarity 0.6000 0.0000 -0.6000\n", + " semantic_similarity 1.0000 0.0000 -1.0000\n", " token_count 0.9885 0.9950 + +0.0065\n", "\n", "SUMMARY\n", @@ -2730,9 +3348,9 @@ " Metrics regressed: 8\n", " Metrics unchanged: 6\n", "\n", - " Baseline overall: 0.4797\n", + " Baseline overall: 0.4459\n", " Experiment overall: 0.8495\n", - " Overall delta: +0.3698\n", + " Overall delta: +0.4035\n", "\n", " >> WINNER: Experiment 2 (k=5)\n", "RESULT COUNTS\n", @@ -2800,7 +3418,7 @@ }, { "cell_type": "code", - "execution_count": 35, + "execution_count": 20, "id": "cell-047", "metadata": {}, "outputs": [ diff --git a/mkdocs.yml b/mkdocs.yml index c95c403..05a81d1 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -183,6 +183,9 @@ nav: - Report Output Formats: reports-output-formats.md - Examples: - examples/index.md + - Colab Quickstart: examples/colab.md + - RAG Evaluation: examples/rag.md + - Corpus Health Auditor: examples/corpus.md - Community: - Contributing: contributing.md - Roadmap: roadmap.md