Testing Guide

Testing Guide

Your starting point for everything testing-related. This page explains what testing docs exist, how they connect, and how to actually run tests.

---

TL;DR — We have 5,200+ automated tests across 334 test files. Run them with pytest. Coverage is strong (28 of 29 feature areas). The dedicated CM test suite now has 196 tests covering ALL conceptual model claims — 5 known gaps remain as xfail (cm_gap) markers (health version, webhook retry, trial days, circuit breaker cooldown, monthly token tracking). CI runs tests in 8 named categories with proper failure propagation (set -o pipefail). Below is a map of all testing docs and when to use each one.

---

How to Run Tests

# All tests
pytest

# With coverage
pytest --cov=src

# Specific test type
pytest tests/unit/
pytest tests/integration/
pytest tests/e2e/

# Conceptual Model tests (196 tests verifying code matches the spec)
pytest tests/conceptual_model/ -v

# Only CM tests that should pass (code matches spec)
pytest tests/conceptual_model/ -m cm_verified -v

# Only CM tests documenting gaps (code doesn't match spec yet)
pytest tests/conceptual_model/ -m cm_gap -v

# Specific feature area
pytest tests/routes/test_chat.py
pytest tests/services/test_circuit_breaker.py

Prerequisites: Python 3.10+, Redis running (for integration tests), Supabase running (for DB tests). See Troubleshooting if things break.

---

The Testing Docs — What They Are and When to Use Them

We have 5 testing documents. Here's how they connect:

Conceptual Model (the spec)
    │
    ├──► CM Unit Testing Plan ──► CM Unit Test Coverage Report
    │    "What SHOULD be tested"     "What IS vs ISN'T tested"
    │
    └──► Testing Plan ──────────► Test Coverage Audit
         "Manual API test cases"     "Automated coverage by feature"

1. Testing Plan — Manual API Test Cases

What: 250+ manual test cases organized by feature. Each is a curl/Postman command with expected result.

When to use: When you want to manually verify a feature works end-to-end. Great for QA sessions, pre-release verification, or when you're debugging something and want to isolate it.

Format: Tables with test #, method, endpoint, auth requirement, and expected response.

---

2. Conceptual-Model-Unit-Testing-Plan — What Should Be Tested

What: 196 unit tests derived from every testable claim in the Conceptual Model. Each has a test ID (CM-{area}.{number}), the unit under test, assertions, and mocks needed.

When to use: When you want to write a new test and need to know what the Conceptual Model demands be verified. Or when reviewing if a feature's tests are complete.

Format: Tables with test ID, test name, unit under test, assertions, mocks.

Where's the code? The actual test implementations live in tests/conceptual_model/ (18 files, 196 tests). See tests/conceptual_model/README.md for the full guide — file layout, markers, fixtures, and how to add new tests. The conceptual-model-tests.yml GitHub Actions workflow runs them on every PR and posts a per-section pass/fail summary as a sticky comment. 5 tests are marked cm_gap (xfail) documenting known spec-vs-code divergences.

---

3. CM Unit Test Coverage Report — What IS vs ISN'T Tested

What: Tracks coverage of all conceptual model claims by the dedicated CM test suite (196 tests in tests/conceptual_model/). All claims now have dedicated tests; 5 known gaps are marked xfail.

When to use: When you want to see which CM claims have known spec-vs-code divergences (the 5 xfail markers), or to review historical coverage progress.

Key numbers:

Status	Count	%
Covered (dedicated CM tests)	191	97.4%
Known gaps (xfail)	5	2.6%
Missing	0	0%

---

4. Test Coverage Audit — Automated Coverage by Feature

What: Feature-by-feature audit of automated test coverage. Maps the Testing Plan test cases to actual test files.

When to use: When you want to know "does feature X have tests?" Shows test files, test functions, and coverage status per feature area.

Key number: 28 of 29 feature areas have automated coverage (only Guardrails has none — because it's not implemented).

---

5. Test Mapping — Raw Test-to-Feature Mapping

What: The raw data — ~5,268 tests mapped to features. This is a large reference file, not meant for reading start to finish.

When to use: When you need to find a specific test for a specific feature, or when you're generating coverage reports.

---

Where to Start

I want to...	Go to...
Run the tests right now	pytest (commands above)
Manually verify a feature	Testing Plan
Write a new test	Conceptual-Model-Unit-Testing-Plan for what to test, CM Unit Test Coverage Report for what's missing
Check if a feature has tests	Test Coverage Audit
Find the test file for feature X	Test Mapping

---

Related

• Features Acceptance Criteria — What each feature must do (criteria the tests should verify)
• Delta Report — P0/P1/P2 issues that need test coverage
• Testing-Workflows-Locally — Running GitHub Actions CI locally with act