Create end-to-end tutorial with free APIs and local embeddings

[himanshu231204]

Problem

New users want to try OpenAgent Eval but don't have a clear, runnable example that works end-to-end with real (but free) API keys. The current examples in the repo are minimal and don't show a complete workflow from data preparation to evaluation results.

Proposed Solution

Create a complete, runnable Jupyter notebook or example project that demonstrates the full OpenAgent Eval workflow using free-tier APIs so users can follow along without spending money.

Requirements

1. Free API Keys (Recommended)

The example should use APIs that have generous free tiers:

Provider	Free Tier	Notes
Google Gemini	15 RPM, 1M tokens/day	Best free option for LLM
Groq	30 RPM, 14,400 req/day	Fast inference, generous limits
OpenAI	$5 credit for new accounts	Limited but works
Ollama	Unlimited (local)	Best for fully offline testing

Recommended: Use Google Gemini or Groq as primary LLM (both have excellent free tiers).

2. Free Embedding Options

Option	Cost	Notes
sentence-transformers (local)	Free	Runs on CPU, no API needed
Ollama embeddings	Free	Local, multiple models available
Google Gemini embeddings	Free tier	1,500 req/day free
HuggingFace embeddings	Free	Via sentence-transformers

Recommended: Use sentence-transformers with all-MiniLM-L6-v2 (fast, small, accurate).

3. Example Dataset

Create a sample dataset for RAG evaluation with 10-20 QA pairs. Include:

• Realistic questions about a specific domain (e.g., Python docs, company FAQ)
• Ground truth answers for evaluation
• Context/passages for retrieval

4. Complete Notebook Structure

1. Setup & Installation - Install openagent-eval, set up free API keys
2. Prepare Your Data - Load sample dataset, format for RAG evaluation
3. Configure Evaluation - Create config.yaml, set up providers
4. Run Evaluation - Initialize engine, run pipeline
5. Analyze Results - View reports, identify failure patterns
6. Improve Your RAG - Based on metrics, suggest improvements

5. Files to Create

File	Purpose
examples/end-to-end-tutorial/README.md	Overview and setup instructions
examples/end-to-end-tutorial/tutorial.ipynb	Jupyter notebook (main)
examples/end-to-end-tutorial/config.yaml	Evaluation configuration
examples/end-to-end-tutorial/data/sample_qa.json	Sample QA dataset
examples/end-to-end-tutorial/requirements.txt	Dependencies

6. Configuration Example

# config.yaml - Free-tier example
dataset:
  path: data/sample_qa.json
  format: qa

llm:
  provider: groq  # Free: 30 RPM
  model: llama-3.3-70b-versatile
  api_key: ${GROQ_API_KEY}

retriever:
  provider: memory  # In-memory for demo
  settings:
    embeddings:
      provider: sentence-transformers
      model: all-MiniLM-L6-v2

metrics:
  - faithfulness
  - answer_relevancy
  - context_precision
  - context_recall
  - exact_match
  - f1_score

reports:
  - terminal
  - markdown

Acceptance Criteria

• Notebook runs without errors on Google Colab or local Jupyter
• Uses only free APIs (no paid keys required)
• Includes clear setup instructions for getting free API keys
• Demonstrates full workflow: data -> config -> run -> analyze
• Shows how to interpret metrics and improve RAG
• Includes troubleshooting section for common issues
• Works on Windows, macOS, and Linux

Additional Notes

• Consider adding a Google Colab badge in the README for one-click launch
• Include screenshots of expected output
• Add a "Next Steps" section pointing to advanced features
• Could also create a YouTube walkthrough video (future enhancement)

[himanshu231204]

@Nitjsefnie take a look on issue#241

[Nitjsefnie]

Thanks @himanshu231204 — this is a great scope (Gemini/Groq free tiers + sentence-transformers local embeddings + a full data-prep → results walkthrough). Some of it overlaps with the Colab tutorial that just merged in #226 (the zero-setup free-API path), so part of the end-to-end story is already in place; #241 would extend it toward the local-embeddings + data-prep flow.

I've got it queued but won't get to a full build for a few days — I'll pick it up and open a PR then. If you'd rather have it sooner and someone else wants to take the local-embeddings example in the meantime, please don't hold it for me.

[himanshu231204]

issue#241 assign to you when you free open the pr .