GenAI Testing Tutorial - Complete Application

Overview

This repository is the working codebase and course material for a two-part GenAI testing curriculum.

Part 1 (Exercises 1-4) focuses on RAG testing fundamentals. Part 2 (Exercises 5-9) focuses on agentic testing for routing, state, reliability, security, and CI gating.

The application is intentionally configured with realistic failure modes so students can practice detection, diagnosis, and mitigation.

Course Flow

Student exercises: docs/exercises/Exercise-1.md to docs/exercises/Exercise-9.md
Instructor notes: docs/exercises/Exercise-1-Instructor-Notes.md to docs/exercises/Exercise-9-Instructor-Notes.md
AppSec demo materials: docs/appsec-demo/
Section transition deck text: docs/Section-Bridge-RAG-to-Agentic.md
Course pacing guide: docs/Course-Schedule.md

Recommended Course Timing

The revised labs are paced so the full course can fit inside roughly 13-14 hours of class time.

Exercise	Target Duration
Exercise 1	35-45 minutes
Exercise 2	35-45 minutes
Exercise 3	40-45 minutes
Exercise 4	45-60 minutes
Exercise 5	45-50 minutes
Exercise 6	45-55 minutes
Exercise 7	40-50 minutes
Exercise 8	40-50 minutes
Exercise 9	35-45 minutes

Exercise-only time is about 6.0 to 7.4 hours. The remaining course time is intended for lecture, demos, transitions, debriefs, and breaks. See docs/Course-Schedule.md for a full recommended agenda.

Architecture

Frontend (HTML/CSS/JS) → Flask Backend → RAG Pipeline → Ollama (Local SLM)
                                    ↓
                               ChromaDB Vector Store
                                    ↑
                            Knowledge Base Documents

Quick Start

Prerequisites

Python 3.8+
Ollama installed (https://ollama.com) or Codespaces devcontainer auto-setup enabled
Local SLM model pulled (default: llama3.2:1b)
Arize Phoenix available for trace and trajectory inspection in Exercises 4-6
2GB+ RAM for vector database
Windows PowerShell (for Windows users)

Installation

Clone and Navigate

cd "c:\Users\jpayne\Documents\Training\Notebooks for ML classes\TestingAITutorial"

Create Virtual Environment

python -m venv training-env
training-env\Scripts\activate  # Windows
# source training-env/bin/activate  # macOS/Linux

Install Dependencies
```
pip install -r requirements.txt
```
Configure Environment

copy .env.template .env

Optional: adjust OLLAMA_MODEL and AGENT_MODEL in .env


Pull a model locally if not already present:
```bash
ollama pull llama3.2:1b
# alternatives for better quality when resources allow: llama3.2:3b or llama3:8b

Agent-mode reliability controls for Exercises 5-6:

AGENT_MODEL=llama3.2:1b (fastest startup path for class demos)
AGENT_REQUEST_TIMEOUT_SECONDS=300 to avoid client-side timeouts on local CPU inference
AGENT_BOOTSTRAP_ON_ZERO_TOOLS=auto enables a transparent one-step bootstrap in student mode only
Set AGENT_BOOTSTRAP_ON_ZERO_TOOLS=false for pure-autonomy instructor demonstrations

Exercise Hub defaults to Student View only. Set EXERCISE_HUB_ENABLE_INSTRUCTOR=True in .env only for instructor-led sessions.

Student agentic access is available via ?agent=1. Exercise pages automatically pass exercise context back to chat for exercise-aware defaults.

Phoenix tracing is enabled by default for both Ask mode and Agent mode:

ENABLE_PHOENIX_ASK_TRACING=true
ENABLE_PHOENIX_AGENT_TRACING=true
PHOENIX_QUALITY_SIGNALS_ENABLED=false (optional quality-related span attributes; keep false to preserve baseline Exercise 4-6 behavior)
PHOENIX_PROJECT_NAME=strategiesfortestingai

Run Application
```
python run.py
```
If you see ModuleNotFoundError (for example flask_cors), dependencies were not installed in the interpreter you are using. Re-run:
```
python -m pip install -r requirements.txt
```
Start Phoenix (recommended for Exercises 4-6)
```
phoenix serve --host 0.0.0.0 --port 6006
```
Then open:
- Chat app: http://localhost:5000
- Phoenix UI: http://localhost:6006
Access Application
- Open http://localhost:5000
- Chat interface should load
- Try: "What are the key challenges in testing GenAI applications?"
- Use the bottom Ask / Agent toggle in chat to switch modes
- In Exercise 4, Ask mode is the intended path for Phoenix trace analysis
- In Exercise 5-9 flows, Exercise Hub sends exercise=<n> so trace/crew defaults can auto-adjust by exercise context

🚀 Quick Start (Alternative)

Use the quick launcher:

python launch.py

These provide menu-driven interfaces to:

Execute regression testing
Run evaluation framework checks
Run Ask-mode trace analysis support for Exercise 4
Start the Flask application
Run Section 7 and 9 automation suites
Access current exercise documentation

Project Structure

TestingAITutorial/
├── app/
│   ├── __init__.py          # Python package initialization
│   ├── agentic_testops.py   # Agentic backend used in Exercises 5-9
│   ├── main.py              # Flask application and API endpoints
│   ├── rag_pipeline.py      # RAG implementation with Ollama + local embeddings + ChromaDB
│   └── utils.py             # Utility functions and helpers
├── static/
│   ├── css/
│   │   └── style.css        # Professional styling with animations
│   └── js/
│       └── chat.js          # Interactive chat interface logic
├── templates/
│   ├── index.html           # Main chat interface template
│   └── exercise_hub.html    # In-app exercise content renderer
├── data/
│   ├── documents/           # Knowledge base documents (GenAI testing content)
│   │   ├── genai_testing_guide.md
│   │   ├── faq_genai_testing.md
│   │   ├── production_best_practices.md
│   │   └── evaluation_metrics.md
│   └── chroma_db/          # Vector database storage (auto-created)
├── tests/
│   └── evaluation_framework.py      # Advanced evaluation tools
├── experiments/             # Exercise support experiments
│   ├── __init__.py          # Package initialization
│   ├── retrieval_experiments.py    # Document retrieval optimization
├── regression_testing/      # Regression testing framework for Exercise 3
│   ├── __init__.py         # Package initialization
│   ├── regression_testing.py       # Core framework
│   └── config.json         # Configurable thresholds and settings
├── docs/                   # Course materials and demo docs
│   ├── exercises/
│   │   ├── Exercise-1.md ... Exercise-9.md
│   │   └── Exercise-1-Instructor-Notes.md ... Exercise-9-Instructor-Notes.md
│   ├── appsec-demo/
│   │   ├── appsec-agent-demo-runbook.md
│   │   ├── appsec-baseline-standard.md
│   │   └── appsec-speaker-cheat-sheet.md
│   ├── Section-Bridge-RAG-to-Agentic.md
├── section7_nfr_quickrun.py # Exercise 7 automation artifact generator
├── section9_agentic_test_suite.py # Exercise 9 CI-style artifact generator
├── temperature_demo.py      # Exercise 1 temperature variability demo
├── requirements.txt         # Python dependencies
├── .env.template           # Environment variable template  
├── run.py                  # Application entry point
├── launch.py               # Interactive launcher menu
└── README.md               # This file (main project overview)

Features

🤖 RAG Chatbot

Local SLM Generation: Uses Ollama-hosted SLMs (llama3.2:* or phi3.5:*)
Local Embeddings: Uses sentence-transformers for vector search
ChromaDB Vector Store: Local, persistent document storage
Custom Python RAG Pipeline: Purpose-built for classroom testing exercises
Ask-mode Phoenix tracing: Linear Chains -> Retriever -> LLM spans for Exercise 4
Source Attribution: Shows retrieved documents and similarity scores
Real-time Performance Metrics: Response times and statistics

🔭 Phoenix Observability

Ask mode tracing: Shows the deterministic RAG path for Exercise 4
Single-agent trajectories: Visualizes repeated tool loops for Exercise 5
Multi-agent handoff graph: Shows Triage -> Specialist -> Validator flow for Exercise 6
Shared tracing setup: Uses the same Phoenix project for Ask and Agent investigations

🎨 Professional Frontend

Modern UI: Clean, responsive design with animations
Real-time Chat: WebSocket-like experience with fetch API
Message History: Persistent chat sessions
Loading States: Typing indicators and progress feedback
Statistics Dashboard: System health and performance metrics
Mobile Responsive: Works on all device sizes

🧪 Comprehensive Testing Suite

Exercise 3 evaluation framework: tests/evaluation_framework.py
Exercise 3 regression framework: regression_testing/regression_testing.py
Exercise 4 optional retrieval experiment support: experiments/retrieval_experiments.py
Exercise 7 automation artifacts: section7_nfr_quickrun.py
Exercise 9 CI-style gate artifacts: section9_agentic_test_suite.py

📚 Rich Knowledge Base

GenAI Testing Guide: Comprehensive testing strategies
FAQ: Common questions about GenAI testing
Best Practices: Production deployment guidelines
Evaluation Metrics: Detailed metric explanations
Real-world Examples: Practical testing scenarios

Course Quick Reference

Exercise Progression

Exercises 1-3: RAG testing fundamentals (exploratory testing, goldens, evaluation)
Exercise 4: Ask-mode Phoenix trace analysis for deterministic RAG
Section bridge: docs/Section-Bridge-RAG-to-Agentic.md
Exercise 5: Single-agent trajectory hacking and span repetition
Exercise 6: Multi-agent handoff corruption and Phoenix graph analysis
Exercise 7: Reliability and overhead across Ask, single-agent, and crew modes
Exercise 8: Red teaming the current agentic system
Exercise 9: Ship / No-Ship decision from current automation evidence

Core Scripts

App: python run.py
Launcher: python launch.py
Section 7 quick-run: python section7_nfr_quickrun.py
Section 9 CI suite: python section9_agentic_test_suite.py

Essential Endpoints

GET / chat UI
POST /api/chat message processing
GET /api/health service health

Environment Setup

Copy .env.template to .env
Confirm OLLAMA_MODEL and OLLAMA_HOST in .env
Confirm ENABLE_PHOENIX_ASK_TRACING=true and ENABLE_PHOENIX_AGENT_TRACING=true
Ensure selected model is present locally: ollama pull <model>

Phoenix Setup

Start Phoenix locally: phoenix serve --host 0.0.0.0 --port 6006
Open Phoenix UI: http://localhost:6006
Use Phoenix in:
- Exercise 4 for Ask-mode traces
- Exercise 5 for single-agent trajectories
- Exercise 6 for multi-agent handoff graphs

Usage Examples

Basic Chat Testing

curl -X POST http://localhost:5000/api/chat \
     -H "Content-Type: application/json" \
     -d '{"message": "What is hallucination in GenAI?"}'

Running Exercise 3 Frameworks

# Run evaluation framework
python tests/evaluation_framework.py

# Run regression testing framework
python -m regression_testing.regression_testing

Running Exercise 3 in Offline Fixture Mode

If API credentials are unavailable in a classroom environment, run deterministic fixture mode:

python -m regression_testing.regression_testing --quick --offline
python tests/evaluation_framework.py --offline

Running Exercise 4 Optional Retrieval Experiment Support

python experiments/retrieval_experiments.py

Exercise 4 itself is now centered on Phoenix-based Ask-mode trace analysis. Use experiments/retrieval_experiments.py only as optional follow-up support if you want to investigate retrieval behavior more deeply after the trace review.

Running Exercise 7 and 9 Automation

python section7_nfr_quickrun.py
python section9_agentic_test_suite.py

Both scripts support a local in-process fallback transport if Flask dependencies are unavailable in the current interpreter. This keeps Exercise 7 and 9 artifact generation runnable in constrained environments.

Section 7 now compares Ask mode, single-agent mode, and crew mode using currently supported prompts.

Section 9 now gates the current system on:

routed factual answering
general-chat routing
harmful-content blocking
handoff-corruption observability
candidate style drift (pirate persona)

Preparing Precomputed Exercise Artifacts

To generate in-repo snapshots for shortened exercises:

python prepare_exercise_artifacts.py

Artifacts are written to artifacts/precomputed/ for instructor distribution.

Verifying Exercise Readiness (1-9)

Run structural checks for all exercises:

python verify_exercise_readiness.py

Run with smoke commands enabled:

python verify_exercise_readiness.py --smoke

Reports are written to artifacts/readiness/ as JSON and Markdown.

Resetting Agentic State Between Lab Runs

Use the API reset endpoint to clear in-memory session state safely:

curl -X POST http://localhost:5000/api/reset \
   -H "Content-Type: application/json" \
   -d '{"scope":"session","session_id":"exercise7-team1","reset_circuit_breaker":true}'

Reset all in-memory sessions:

curl -X POST http://localhost:5000/api/reset \
   -H "Content-Type: application/json" \
   -d '{"scope":"all","reset_circuit_breaker":true}'

Quality Assessment

from tests.evaluation_framework import EvaluationFramework
from app.rag_pipeline import RAGPipeline

pipeline = RAGPipeline()
evaluator = EvaluationFramework(pipeline)

# Run comprehensive evaluation
results = evaluator.evaluate_response_quality([
    {"query": "What is GenAI testing?", "expected_topics": ["testing", "genai"]}
])

print(f"Average Quality Score: {results['average_quality_score']}")

Regression Testing Integration

from regression_testing.regression_testing import RegressionTestFramework

# Create framework
framework = RegressionTestFramework()

# Run tests programmatically
results = framework.run_regression_tests(save_results=True)

# Check quality gate
gate_passed = (
    results['summary']['pass_rate'] >= 0.8 and 
    results['summary']['critical_failures'] == 0
)

print(f"Quality Gate: {'PASSED' if gate_passed else 'FAILED'}")

Learning Paths

🎓 Beginner Track

Complete student labs in order: Exercise 1 -> 4 (RAG section)
Deliver the section bridge before starting Exercise 5
Complete Exercise 5 -> 9 (agentic section)
Use Phoenix in Exercises 4-6 for trace and trajectory analysis
Use Section 7 and 9 automation scripts for standardized evidence

🔬 Intermediate Track

Map each exercise deliverable to one reusable rubric/checklist
Add 2-3 custom prompts per exercise to extend coverage
Compare baseline and post-change behavior using quick-run artifacts
Create a release recommendation from Exercise 9 evidence

🚀 Advanced Track

Extend section7_nfr_quickrun.py with additional NFR scenarios
Extend section9_agentic_test_suite.py with new showstopper gates
Integrate suite outputs into CI (artifact upload + gate decision)
Add production monitoring and drift checks aligned to Exercise 9

Troubleshooting

Common Setup Issues

"ModuleNotFoundError: No module named 'app' or 'regression_testing'"

Ensure commands are run from repository root
Use the launcher (python launch.py) or run python -m regression_testing.regression_testing

"KeyError: 'failed_tests' or 'avg_response_length'"

These issues have been FIXED in the test framework
Test data now includes all required keys for proper execution

"Ollama is not reachable"

Ensure Ollama is installed and running: ollama serve
Verify connectivity: curl http://127.0.0.1:11434/api/tags

"Phoenix UI is not reachable"

Start Phoenix manually: phoenix serve --host 0.0.0.0 --port 6006
Verify the port is open at http://localhost:6006
Keep ENABLE_PHOENIX_ASK_TRACING and ENABLE_PHOENIX_AGENT_TRACING enabled in .env

"Model not found in Ollama"

Pull the model referenced by OLLAMA_MODEL in .env
Example: ollama pull llama3.2:1b

"sentence-transformers import failed"

Ensure virtual environment is activated: training-env\Scripts\activate
Run pip install -r requirements.txt
Use python launch.py to run exercise workflows consistently

"sentence-transformers not available"

This package is optional but recommended for semantic similarity scoring in Exercise 3
Install with pip install sentence-transformers (or pip install -r requirements.txt)
If unavailable, the regression framework still runs with reduced semantic checks

"OLLAMA_MODEL or OLLAMA_HOST missing"

Copy .env.template to .env
Ensure OLLAMA_HOST and OLLAMA_MODEL are set

"ChromaDB initialization failed"

Ensure you have write permissions in the project directory
Delete data/chroma_db/ folder and restart if corrupted

"Flask app won't start"

Check that port 5000 is available
Set FLASK_PORT=5001 in .env to use a different port

Performance Issues

Slow first response

First query initializes the vector database (expected delay)
Subsequent queries should be faster

High memory usage

ChromaDB loads embeddings into memory
Reduce document collection size if needed

Quality Issues

Poor response quality

This may be intentional! Part of the learning exercise
Check if you're discovering the planted issues correctly

Off-topic responses

Test the system's domain boundaries
Document cases where it should vs. shouldn't know answers

Educational Value

This application provides hands-on experience with:

🎯 Core Concepts

Non-deterministic Testing: Dealing with probabilistic outputs
Quality vs. Performance Trade-offs: Balancing response quality and speed
Evaluation Metrics: Understanding different ways to measure success
Production Readiness: What it takes to deploy GenAI systems

🛡️ Safety and Robustness

Hallucination Detection: Identifying when AI generates false information
Bias Testing: Checking for unfair or inappropriate responses
Edge Case Handling: System behavior with unusual inputs
Adversarial Robustness: Resistance to malicious inputs

📈 Performance Engineering

Latency Optimization: Making responses faster
Scalability Testing: Handling multiple concurrent users
Resource Management: Efficient use of CPU, memory, and API calls
Monitoring and Alerting: Detecting issues in production

🔍 Quality Assurance

Multi-dimensional Evaluation: Beyond simple accuracy metrics
Consistency Testing: Ensuring reliable behavior
Regression Detection: Catching quality degradation
User Experience Focus: Testing from the user's perspective

Contributing

This is an educational project. If you find additional issues or have suggestions for improvements:

Document your findings clearly
Propose educational value of the change
Consider impact on learning objectives
Share with the instructor or class

License

This project is created for educational purposes. Use freely for learning and teaching GenAI testing concepts.

Happy Testing! 🧪🤖

Remember: The goal isn't just to build GenAI applications, but to build ones that are reliable, safe, and provide genuine value to users. Testing is how we ensure that promise is kept.

Uh oh!

FilesExpand file tree

README.md

Latest commit

History