Skip to content

Sprint 4 PR 4.9 — make the Makefile reset the DB the tests actually use - #250

Merged
tomqwu merged 1 commit into
mainfrom
sprint-4-step-49-test-db-isolation
Aug 4, 2026
Merged

tomqwu merged 1 commit into
mainfrom
sprint-4-step-49-test-db-isolation

Conversation

@tomqwu

@tomqwu tomqwu commented Aug 4, 2026

Copy link
Copy Markdown
Owner

Summary

make test-all — the gate CLAUDE.md tells contributors to run before pushing a PR — has been red on main on any machine that had run the suite before. Measured at a09e422: 9 failed / 333 passed.

The cause is a path mismatch, not a flaky test.

The two halves that disagreed

  • tests/conftest.py:15 sets TESTING_FORCE_MEMORY=true unconditionally, which makes api/database.py:18-19 bind to /tmp/signupflow_test.db regardless of DATABASE_URL.
  • The Makefile defined TEST_DB_PATH := $(abspath test_roster.db) and rm'd that file between runs — a file the suite never reads. setup_test_data seeded it too, via the Makefile's exported DATABASE_URL.

So @echo "🔄 Rebuilding fresh SQLite test database..." was a no-op. Every run.

Why that turned into failures

The Makefile also exports SKIP_TEST_DB_FIXTURES=true, which makes conftest skip the per-test truncation fixture. With truncation skipped and the reset hitting the wrong file, nothing ever cleaned /tmp/signupflow_test.db. It accumulated across every run the repo had ever done — 518 organizations and 794 people by the time I diagnosed it. Tests that create fixed IDs (person_001, avail_test_org1, …) eventually hit rows left by earlier runs and got 409 CONFLICT. Failures grew run over run: a partially-dirty DB gave 2 failures, a dirtier one gave 9.

CI never noticed because it runs bare pytest on a fresh runner and never invokes make.

Fix

Point TEST_DB_PATH at /tmp/signupflow_test.db so both the reset and the seed target the DB the suite actually binds to, and derive every rm from that variable rather than hardcoding the stale filename.

-TEST_DB_PATH := $(abspath test_roster.db)
+TEST_DB_PATH := /tmp/signupflow_test.db
-	@rm -f test_roster.db test_roster.db-shm test_roster.db-wal
+	@rm -f $(TEST_DB_PATH) $(TEST_DB_PATH)-shm $(TEST_DB_PATH)-wal

Tests

New tests/unit/test_make_test_db_path.py (4 tests) pins both halves of the contract so they cannot drift apart again:

  1. api/database.py still redirects to the /tmp path under TESTING_FORCE_MEMORY
  2. conftest still sets that variable
  3. the Makefile's TEST_DB_PATH equals that path
  4. no Makefile recipe still references test_roster.db

Validation

Check Result
make test-all (from a dirty DB) unit 350, api 332, cli 16, integration 324 — zero failures (was 9 failed / 333 passed)
make test-all run twice back to back byte-identical counts — the target is now idempotent
pytest tests/unit tests/api tests/cli tests/contract tests/web 916 passed, 21 skipped
black / ruff clean
mypy api/utils api/core api/schemas clean, 61 files

The two-consecutive-runs check is the one that matters: before this change the second run was strictly worse than the first, which is the signature of the accumulation.

Follow-ups

  • The create-tests still use fixed IDs and are only safe because the DB is now reset between runs. Giving them unique per-test IDs would make them robust regardless of DB state — worth doing, but it's a broader change across several files and doesn't belong in this fix.

Generated by Claude Code

Summary:
`make test-all` — the gate CLAUDE.md tells contributors to run before
pushing — was red on main (9 failed / 333 passed at a09e422) on any
machine that had run the suite before. Root cause is a path mismatch:

- tests/conftest.py sets TESTING_FORCE_MEMORY=true, which makes
  api/database.py bind to /tmp/signupflow_test.db regardless of
  DATABASE_URL.
- The Makefile defined its test DB as ./test_roster.db and `rm`'d that
  file between runs — a file the suite never reads. `setup_test_data`
  seeded it too, via the Makefile's DATABASE_URL.

So every "rebuild fresh test database" step was a no-op. Combined with
the Makefile's SKIP_TEST_DB_FIXTURES=true (which skips the per-test
truncation fixture), /tmp/signupflow_test.db accumulated rows across
every run — 518 orgs and 794 people by the time this was diagnosed —
until fixed-ID create-tests collided and returned 409 CONFLICT. Failures
grew run over run.

CI was unaffected because it runs bare `pytest` on a fresh runner and
never invokes make, which is why this stayed hidden.

Fix: point TEST_DB_PATH at /tmp/signupflow_test.db so the reset and the
seed both target the database the suite binds to, and derive every `rm`
from that variable instead of hardcoding the stale filename.

Changed files:
- Makefile (TEST_DB_PATH + test-all/clean recipes)
- tests/unit/test_make_test_db_path.py (new, 4 tests)

The new tests pin both halves of the contract — that api/database.py
redirects under TESTING_FORCE_MEMORY, that conftest sets it, that
TEST_DB_PATH equals that path, and that no recipe references
test_roster.db — so the two cannot drift apart again.

Validation:
- `make test-all` from a dirty DB -> unit 350, api 332, cli 16,
  integration 324; zero failures (was 9 failed / 333 passed).
- Ran `make test-all` twice back to back: byte-identical counts, so the
  accumulation is gone and the target is now idempotent.
- pytest tests/unit tests/api tests/cli tests/contract tests/web
  -> 916 passed, 21 skipped.
- black + ruff clean; mypy strict clean (61 files).

Follow-ups:
- Those create-tests still rely on fixed IDs and are only safe because
  the DB is reset; switching them to unique per-test IDs would make them
  robust regardless of DB state.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013v7qGC4EZftRXXxdFcasAN
@tomqwu
tomqwu marked this pull request as ready for review August 4, 2026 19:54
@tomqwu
tomqwu merged commit 214e3f3 into main Aug 4, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants