Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 37 additions & 0 deletions docs/examples/colab.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# Colab Quickstart Tutorial

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/OpenAgentHQ/openagent-eval/blob/main/examples/openagent_eval_colab_tutorial.ipynb)

A zero-setup, end-to-end tour of OpenAgent Eval that runs entirely in your browser.
It uses the built-in **mock** LLM and retriever providers, so it needs **no API keys**
and makes no network calls — click the badge above and run every cell.

## What you'll learn

- Installing OpenAgent Eval and verifying it with `oaeval doctor`
- Building a fully offline evaluation config with the `mock` providers
- Running an evaluation from both the CLI (`oaeval run`) and the Python SDK (`Engine`)
- Reading the terminal, Markdown, HTML and JSON reports
- Generating synthetic test cases with `oaeval synth`
- Optionally plugging in a real provider (OpenAI) through a guarded, key-optional cell

## Run it

- **In the browser:** [open in Google Colab](https://colab.research.google.com/github/OpenAgentHQ/openagent-eval/blob/main/examples/openagent_eval_colab_tutorial.ipynb)
and choose *Runtime → Run all*.
- **Locally:** download
[`openagent_eval_colab_tutorial.ipynb`](https://github.com/OpenAgentHQ/openagent-eval/blob/main/examples/openagent_eval_colab_tutorial.ipynb)
and open it with `jupyter notebook`.

## Prerequisites

None for the offline walkthrough — the notebook installs its own dependencies in the
first cell:

```bash
pip install -q openagent-eval pytest
```

The optional final section runs against a real provider when an `OPENAI_API_KEY` is
present (via Colab Secrets or an environment variable); without a key it skips itself
and the notebook stays fully offline.
39 changes: 39 additions & 0 deletions docs/examples/corpus.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
# Corpus Health Auditor Tutorial

A Jupyter notebook tour of OpenAgent Eval's corpus, diagnosis and synthesis modules:

- Auditing a document corpus for **staleness**, **duplicates** and **coverage** gaps
- Detecting **contradictions** with an LLM-as-judge
- Combining the checks into a single corpus health report
- Diagnosing evaluation failures with `DiagnosisAnalyzer`
- Generating synthetic and adversarial test cases

**Download:** [`corpus_and_related_modules.ipynb`](https://github.com/OpenAgentHQ/openagent-eval/blob/main/examples/corpus_and_related_modules.ipynb)

### What you'll learn

| Section | Topic |
|---------|-------|
| 1–3 | Installation and preparing a sample corpus |
| 4 | Corpus health: staleness, duplicate, coverage and contradiction detectors |
| 5 | A combined corpus audit report |
| 6 | Failure diagnosis with `DiagnosisAnalyzer` |
| 7 | Synthetic and adversarial test-case generation |

### Prerequisites

```bash
pip install openagent-eval jupyter
```

The staleness, duplicate, coverage and diagnosis sections run fully offline. The
contradiction detector and the synthesis sections use an LLM-as-judge — set a
`GROQ_API_KEY` (a free key from [Groq](https://console.groq.com)) to run them; without
a key those cells are skipped.

### Quick start

```bash
cd examples/
jupyter notebook corpus_and_related_modules.ipynb
```
53 changes: 19 additions & 34 deletions docs/examples/index.md
Original file line number Diff line number Diff line change
@@ -1,46 +1,31 @@
# Examples

Worked examples showing how to use OpenAgent Eval in practice.
Worked examples showing how to use OpenAgent Eval in practice. Each tutorial is a
runnable Jupyter notebook in the
[`examples/`](https://github.com/OpenAgentHQ/openagent-eval/tree/main/examples)
directory of the repository.

## RAG Evaluation Tutorial
| Tutorial | What it covers | Run it |
|----------|----------------|--------|
| [Colab Quickstart](colab.md) | Zero-setup, end-to-end evaluation that runs in the browser — no API keys required | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/OpenAgentHQ/openagent-eval/blob/main/examples/openagent_eval_colab_tutorial.ipynb) |
| [RAG Evaluation](rag.md) | A complete RAG pipeline scored with all 18 retrieval, generation, performance and cost metrics | Local Jupyter |
| [Corpus Health Auditor](corpus.md) | Corpus health checks (staleness, duplicates, coverage, contradiction), failure diagnosis and synthetic test-case generation | Local Jupyter |

A hands-on Jupyter notebook that walks through a complete RAG evaluation:

- Loading a dataset
- Configuring an LLM and retriever
- Running retrieval and generation metrics
- Interpreting the results

**Download:** [`rag_evaluation_tutorial.ipynb`](https://github.com/OpenAgentHQ/openagent-eval/blob/main/examples/rag_evaluation_tutorial.ipynb)

### What you'll learn

| Section | Topic |
|---------|-------|
| 1 | Setting up the environment and config |
| 2 | Loading and inspecting a dataset |
| 3 | Configuring LLM providers (OpenAI, Ollama, Mock) |
| 4 | Configuring retriever providers (Chroma, Memory, BM25) |
| 5 | Running the evaluation pipeline |
| 6 | Understanding retrieval metrics (precision, recall, MRR, NDCG) |
| 7 | Understanding generation metrics (faithfulness, relevancy, hallucination) |
| 8 | Running all 18 metrics together |
| 9 | Interpreting the report output |

### Prerequisites
## Running the notebooks locally

```bash
pip install openagent-eval jupyter
git clone https://github.com/OpenAgentHQ/openagent-eval
cd openagent-eval/examples
jupyter notebook
```

### Quick start

```bash
cd examples/
jupyter notebook rag_evaluation_tutorial.ipynb
```
The [Colab Quickstart](colab.md) needs no local setup at all — open it directly in
your browser.

## More examples

See the [scripts/](https://github.com/OpenAgentHQ/openagent-eval/tree/main/scripts) directory
in the repository for additional runnable examples.
For copy-paste snippets covering common workflows, see the
[Quickstart](../quickstart.md). The
[`scripts/`](https://github.com/OpenAgentHQ/openagent-eval/tree/main/scripts)
directory in the repository holds additional runnable examples.
41 changes: 41 additions & 0 deletions docs/examples/rag.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# RAG Evaluation Tutorial

A hands-on Jupyter notebook that walks through a complete RAG evaluation using
OpenAgent Eval's offline **mock** providers — no API keys required:

- Building a minimal RAG pipeline (chunking, a local keyword retriever, a mock LLM)
- Configuring an evaluation both from YAML and programmatically
- Running all 18 retrieval, generation, performance and cost metrics
- Diagnosing failures and interpreting the report output
- Writing a custom metric and comparing experiments

**Download:** [`rag_evaluation_tutorial.ipynb`](https://github.com/OpenAgentHQ/openagent-eval/blob/main/examples/rag_evaluation_tutorial.ipynb)

### What you'll learn

| Section | Topic |
|---------|-------|
| 1 | Introduction: what OpenAgent Eval is |
| 2 | Installing and verifying the package |
| 3 | Building a minimal RAG pipeline (retriever + mock LLM) |
| 4 | Evaluating the pipeline (YAML and programmatic config) |
| 5 | A deep dive on all 18 metrics |
| 6 | Interpreting results and diagnosing failures |
| 7 | Advanced usage: custom metrics, batch comparison, LLM judges |
| 8 | Best practices for production RAG evaluation |

### Prerequisites

```bash
pip install openagent-eval jupyter
```

### Quick start

```bash
cd examples/
jupyter notebook rag_evaluation_tutorial.ipynb
```

The notebook runs entirely offline with the `mock` providers, so you can work through
every cell without an API key.
Loading
Loading