Commit 095528e
PR-N1: remove verifier-protocol test doubles from Linux CI gate
ADR 0008 / no-test-doubles cleanup. PR-N1 retires the FakeVerifier
class hierarchy (FakeVerifier, _LyingVerifier, _RegressingVerifier,
_LyingFakeVerifier) and migrates its dispatch / state-mirror tests
to tests/integration/ where they run against the real Qwen3-0.6B
verifier. Linux CI gate now covers only verifier-independent code;
runtime correctness moves to the integration suite (Mac M4 / CUDA).
The architectural rule
----------------------
The Linux runner cannot load real model weights. Before PR-N1, the
100% Linux coverage gate forced a workaround: hand-code a verifier
mirror (FakeVerifier) that approximated the real verifier's state-
mutation contract and run dispatch tests against it. PR-E1c's
discussion surfaced this as a 'no test doubles' violation \u2014 a
hand-coded mirror is exactly what the principle excludes, regardless
of whether we call it a 'fake' or a 'mock'.
The fix:
- Linux gate covers verifier-INDEPENDENT modules:
inference_engine.server (HTTP shim, gRPC handler)
inference_engine.memory (slab pool)
inference_engine.scheduler (admission + queueing)
inference_engine.pipeline (cancellable producer/consumer)
inference_engine.session.store (data layer + INV-1 / INV-2)
sdks.python.kakeya (gRPC client)
training.repr_align (alignment training)
All 100% covered with NO test doubles for verifier protocol.
- Integration suite covers verifier-DEPENDENT modules:
inference_engine.session.coordinator
inference_engine.session.generator
against real Qwen3-0.6B in tests/integration/test_coordinator_real.py
and test_generator_real.py. Run via 'pytest -m integration'
on Mac M4 / CUDA hosts. PR-E2 (queued) ships the self-hosted
runner workflow; until then, scripts/review_pr_n1_on_mac.sh
drives the Mac M4 evidence run.
What was deleted
----------------
tests/inference_engine/session/test_coordinator.py
-450 lines. FakeVerifier (140 lines) + 19 dispatch tests +
INV-3 byte-exact tests + state-mirror tests.
tests/inference_engine/session/test_generator.py
-433 lines. 31 generator tests, all imported FakeVerifier
from test_coordinator.py.
tests/inference_engine/server/test_grpc_app.py
-617 / +X lines net. Stripped the FakeVerifier-using sections
(grpc_pair_with_appender + grpc_pair_with_generator fixtures
and their ~17 consumer tests). Kept all verifier-independent
tests (CreateSession, CloseSession, GetSessionInfo,
UNIMPLEMENTED defaults, factory tests). Re-added 4 error-mapping
tests that drive the Servicer with coordinator overrides
instead of FakeVerifier (raise the relevant exception type
from the override; verifier=None is safe because the override
never accesses self._verifier).
What was added
--------------
tests/integration/test_coordinator_real.py +361 lines, 25 tests
Coordinator dispatch + state-mirror + error paths against real
Qwen3-0.6B via the existing fresh_verifier_factory fixture.
tests/integration/test_generator_real.py +252 lines, 12 tests
Generator greedy / EOS / HistoryTruncated / INV / kv_live_bytes
sync against real Qwen3-0.6B.
tests/inference_engine/session/test_coordinator_validation.py
+84 lines, 5 tests
Pre-verifier validation paths (unknown session, empty append,
constructor) tested with verifier=None on Linux. No double.
tests/inference_engine/session/test_generator_validation.py
+165 lines, 12 tests
GenerationCoordinator's argument-validation paths (max_tokens,
sampling params, AppendTokens-must-precede-Generate, unknown
session, event dataclass frozenness) tested with verifier=None.
No double.
scripts/review_pr_n1_on_mac.sh +103 lines
Mac M4 reviewer aid that runs pytest -m integration and
produces pr-n1-mac-integration-tests-<unix>.json under
results/platform-tests/.
What was kept (out of PR-N1 scope)
----------------------------------
tests/inference_engine/scheduler/test_pooled_verifier.py
Uses _FakeVerifier / _RaisingVerifier. PR-D2 retires the
PooledVerifier module entirely (HTTP shim refactor onto
SessionStore), which makes this test file moot. Cleaning it
up now would be throwaway work; flagged in PR description.
tests/sdk/python/conftest.py
The FakeVerifier import is replaced by an inline
_MinimalVerifierStub class. The SDK tests are wire-layer
tests (encode/decode + status mapping); their truth is gRPC
transport correctness, not verifier numerics. The stub
satisfies VerifierProtocol shape but is documented as 'not a
verifier mirror'. End-to-end runtime correctness is covered
by tests/integration/.
Engine / tokenizer doubles (DeterministicEngine,
DeterministicTokenizer, _RaisingEngine, _ProxyEngine, etc.) in
tests/inference_engine/server/ and tests/inference_engine/scheduler/.
These are PR-N2 / PR-N3 scope (per the original 4-PR sequence).
CI workflow change
------------------
.github/workflows/ci.yaml: changed --cov=inference_engine.session
to --cov=inference_engine.session.store. The coordinator and
generator modules are no longer covered on Linux. They reach 100%
in the integration suite.
Linux verification
------------------
PYTHONPATH=.:sdks/python coverage run -m pytest <Linux gate paths>:
649 passed (was 682 in PR-D1 baseline; -33 net = removed ~50
FakeVerifier-driven tests, added ~17 verifier-independent
validation + gRPC error-mapping tests).
100% coverage on 1595 stmts (was 1660 in PR-D1; -65 net stmts
is the coordinator + generator now NOT in --cov= scope).
Mac M4 evidence (REQUIRED for merge)
------------------------------------
Per ADR 0008 \u00a79: this PR's runtime-correctness evidence lives in
the integration suite. Reviewer runs:
bash scripts/review_pr_n1_on_mac.sh
git add results/platform-tests/pr-n1-mac-*
git commit -m 'Mac M4 review evidence for PR-N1'
git push
Acceptance: all integration tests pass against real Qwen3-0.6B.
The INV-3 byte-exact GA gate (PR-E1) is included.
Stack
-----
PR-N1 is branched off main directly. References to _sync_slab_bytes
(introduced by PR-E1c, in flight as PR #52) are deferred to a
follow-up after PR-E1c merges; the helper itself is covered by
PR-E1c's own tests on its branch.
Next PRs
--------
PR-N2: remove DeterministicEngine + DeterministicTokenizer.
PR-N3: remove server-specific engine doubles.
PR-N4: post-N1/N2/N3 CI workflow consolidation.
Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>1 parent bec3d7b commit 095528e
10 files changed
Lines changed: 1255 additions & 1324 deletions
File tree
- .github/workflows
- scripts
- tests
- inference_engine
- server
- session
- integration
- sdk/python
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
72 | 72 | | |
73 | 73 | | |
74 | 74 | | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
75 | 83 | | |
76 | 84 | | |
77 | 85 | | |
| |||
85 | 93 | | |
86 | 94 | | |
87 | 95 | | |
88 | | - | |
| 96 | + | |
89 | 97 | | |
90 | 98 | | |
91 | 99 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
0 commit comments