docs: complete zero-setup Colab tutorial notebook

[Nitjsefnie]

Part of #188 — the notebook half you asked for on #219 (the companion video remains open for whoever takes it).

examples/openagent_eval_colab_tutorial.ipynb: 69 cells (33 code + 36 markdown), shipped with executed outputs like the existing examples. Executes end-to-end with ZERO API keys via the built-in mock provider stack (llm.provider: mock + retriever.provider: mock), covering the issue's full section list: install → terminal basics → init --no-interactive → sample data → run → corpus audit → diagnose → synth → compare → SDK → CI/CD gating → custom metrics → troubleshooting → next steps, plus an optional real-keys section (getpass / Colab Secrets, never hardcoded).

Verified the way a Colab user experiences it: fresh venv, pip install openagent-eval==0.4.6 from PyPI (not the repo checkout), jupyter nbconvert --execute → 33/33 code cells clean, re-executed with all provider keys scrubbed to match a clean Colab runtime.

Three honest notes baked into the notebook rather than papered over: the install cell also installs pytest (0.4.6's CLI import-crashes without it — filing that as an issue), the CI/CD section teaches gating via the working SDK threshold path because oaeval test is non-functional in 0.4.6 (also being filed), and synth under the mock provider honestly yields 0 cases (real synthesis needs a real key). Also: the issue body's transcribed API signatures have drifted from the code (BaseMetric.evaluate(**kwargs), positional audit/diagnose paths, async Engine.run) — the notebook uses the verified real surfaces.

Deliberately did NOT touch examples/README.md / root README badge (shared-file conflict risk while your merge queue is hot) — happy to add the index line + badge as a follow-up commit here if you want them in this PR.

Generated by Claude Fable 5 (brief, review), Claude Opus 4.8 (implementation)

[himanshu231204]

@Nitjsefnie update the notebook with the new release v.0.4.7

[Nitjsefnie]

Held the 0.4.7 notebook update — the 0.4.7 release artifacts are broken (wheel+sdist missing the whole openagent_eval.reports subpackage, every oaeval command dies at import; full evidence in the issue I just filed). Pinning the notebook to 0.4.7 would crash all ~15 CLI cells, so it stays on 0.4.6 (where everything executes) until a rebuilt release is up — then I'll re-pin, re-execute headless, and also re-test whether the #227 pytest workaround can come out.

[himanshu231204]

@Nitjsefnie update the notebook with the new release v.0.4.8

[Nitjsefnie]

Updated to 0.4.8 in 8666c9e and re-executed the whole notebook headless against the PyPI release — all 26 code cells clean.

Release re-test results, since they matter for your notes:

• release: the 0.4.7 wheel and sdist are missing the entire openagent_eval.reports subpackage — oaeval unusable #233 fixed — the 0.4.8 wheel restores openagent_eval.reports; artifacts verified good in a fresh venv.
• cli: oaeval crashes on a clean pip install — pytest imported at load but not a runtime dependency #227 still present — oaeval --help in a pytest-free venv still dies on import pytest (cicd/plugin.py load-time import), so the notebook's pytest line in the install cell stays; it's load-bearing, not cosmetic.
• cicd: oaeval test always fails — gate summary carries no metrics_summary, so every threshold reports 'metric not found' #228 still present — real run on 0.4.8: oaeval run produces faithfulness=0.125, then oaeval test config.yaml -t faithfulness:gte:0.0 (a gate everything should pass) exits 1 with "Metric 'faithfulness' not found in results" on every threshold. The CI/CD section keeps teaching the working SDK path; heads-up that the oaeval test line in any Actions workflow will fail on 0.4.8 regardless of scores until cicd: oaeval test always fails — gate summary carries no metrics_summary, so every threshold reports 'metric not found' #228 lands.

[himanshu231204]

@Nitjsefnie update all the previous example/notebook . and also add it to doc so that github pages is up to date

[Nitjsefnie]

All done in d72dd58:

• Both older notebooks refreshed to 0.4.8 and executed headless clean. corpus_and_related_modules: was pinned to 0.4.4 and its LLM cells crashed with NameError without a Groq key — now version-correct, with the four LLM-as-judge cells key-guarded like the Colab tutorial (the mock provider genuinely can't drive the synthesis parser, so guarding beats mocking there); non-LLM detectors run offline. rag_evaluation_tutorial: install cell now includes pytest (the cli: oaeval crashes on a clean pip install — pytest imported at load but not a runtime dependency #227 workaround — its doctor subprocess dies without it on a clean env) and outputs re-executed fresh; only that one source cell changed.
• Docs/Pages wired: new docs/examples/colab.md (open-in-Colab badge + raw download), rag.md, corpus.md, the examples index rewritten to cover all three, and the mkdocs nav updated. mkdocs build --strict exit 0.

Two heads-ups: docs/examples.md and docs/examples/index.md both resolve to /examples/ — a pre-existing URL collision I didn't touch (4 inbound links target the former); and the docs CI step runs with continue-on-error: true, so the local --strict pass is the real gate. Once this merges I'll start the #239 redesign on top.

[himanshu231204]

All done in d72dd58:

• Both older notebooks refreshed to 0.4.8 and executed headless clean. corpus_and_related_modules: was pinned to 0.4.4 and its LLM cells crashed with NameError without a Groq key — now version-correct, with the four LLM-as-judge cells key-guarded like the Colab tutorial (the mock provider genuinely can't drive the synthesis parser, so guarding beats mocking there); non-LLM detectors run offline. rag_evaluation_tutorial: install cell now includes pytest (the cli: oaeval crashes on a clean pip install — pytest imported at load but not a runtime dependency #227 workaround — its doctor subprocess dies without it on a clean env) and outputs re-executed fresh; only that one source cell changed.
• Docs/Pages wired: new docs/examples/colab.md (open-in-Colab badge + raw download), rag.md, corpus.md, the examples index rewritten to cover all three, and the mkdocs nav updated. mkdocs build --strict exit 0.

Two heads-ups: docs/examples.md and docs/examples/index.md both resolve to /examples/ — a pre-existing URL collision I didn't touch (4 inbound links target the former); and the docs CI step runs with continue-on-error: true, so the local --strict pass is the real gate. Once this merges I'll start the #239 redesign on top.

ok wait for a min to workflow completed

[github-actions]

🎉 Congratulations @Nitjsefnie!

Your pull request has been successfully merged into main. 🚀

Thank you for contributing to OpenAgentHQ and helping improve the project.

We truly appreciate your contribution and hope to see you back with more amazing PRs!

Happy Open Sourcing! ❤️

[himanshu231204]

good to go with #239