Skip to content

Commit 095528e

Browse files
PR-N1: remove verifier-protocol test doubles from Linux CI gate
ADR 0008 / no-test-doubles cleanup. PR-N1 retires the FakeVerifier class hierarchy (FakeVerifier, _LyingVerifier, _RegressingVerifier, _LyingFakeVerifier) and migrates its dispatch / state-mirror tests to tests/integration/ where they run against the real Qwen3-0.6B verifier. Linux CI gate now covers only verifier-independent code; runtime correctness moves to the integration suite (Mac M4 / CUDA). The architectural rule ---------------------- The Linux runner cannot load real model weights. Before PR-N1, the 100% Linux coverage gate forced a workaround: hand-code a verifier mirror (FakeVerifier) that approximated the real verifier's state- mutation contract and run dispatch tests against it. PR-E1c's discussion surfaced this as a 'no test doubles' violation \u2014 a hand-coded mirror is exactly what the principle excludes, regardless of whether we call it a 'fake' or a 'mock'. The fix: - Linux gate covers verifier-INDEPENDENT modules: inference_engine.server (HTTP shim, gRPC handler) inference_engine.memory (slab pool) inference_engine.scheduler (admission + queueing) inference_engine.pipeline (cancellable producer/consumer) inference_engine.session.store (data layer + INV-1 / INV-2) sdks.python.kakeya (gRPC client) training.repr_align (alignment training) All 100% covered with NO test doubles for verifier protocol. - Integration suite covers verifier-DEPENDENT modules: inference_engine.session.coordinator inference_engine.session.generator against real Qwen3-0.6B in tests/integration/test_coordinator_real.py and test_generator_real.py. Run via 'pytest -m integration' on Mac M4 / CUDA hosts. PR-E2 (queued) ships the self-hosted runner workflow; until then, scripts/review_pr_n1_on_mac.sh drives the Mac M4 evidence run. What was deleted ---------------- tests/inference_engine/session/test_coordinator.py -450 lines. FakeVerifier (140 lines) + 19 dispatch tests + INV-3 byte-exact tests + state-mirror tests. tests/inference_engine/session/test_generator.py -433 lines. 31 generator tests, all imported FakeVerifier from test_coordinator.py. tests/inference_engine/server/test_grpc_app.py -617 / +X lines net. Stripped the FakeVerifier-using sections (grpc_pair_with_appender + grpc_pair_with_generator fixtures and their ~17 consumer tests). Kept all verifier-independent tests (CreateSession, CloseSession, GetSessionInfo, UNIMPLEMENTED defaults, factory tests). Re-added 4 error-mapping tests that drive the Servicer with coordinator overrides instead of FakeVerifier (raise the relevant exception type from the override; verifier=None is safe because the override never accesses self._verifier). What was added -------------- tests/integration/test_coordinator_real.py +361 lines, 25 tests Coordinator dispatch + state-mirror + error paths against real Qwen3-0.6B via the existing fresh_verifier_factory fixture. tests/integration/test_generator_real.py +252 lines, 12 tests Generator greedy / EOS / HistoryTruncated / INV / kv_live_bytes sync against real Qwen3-0.6B. tests/inference_engine/session/test_coordinator_validation.py +84 lines, 5 tests Pre-verifier validation paths (unknown session, empty append, constructor) tested with verifier=None on Linux. No double. tests/inference_engine/session/test_generator_validation.py +165 lines, 12 tests GenerationCoordinator's argument-validation paths (max_tokens, sampling params, AppendTokens-must-precede-Generate, unknown session, event dataclass frozenness) tested with verifier=None. No double. scripts/review_pr_n1_on_mac.sh +103 lines Mac M4 reviewer aid that runs pytest -m integration and produces pr-n1-mac-integration-tests-<unix>.json under results/platform-tests/. What was kept (out of PR-N1 scope) ---------------------------------- tests/inference_engine/scheduler/test_pooled_verifier.py Uses _FakeVerifier / _RaisingVerifier. PR-D2 retires the PooledVerifier module entirely (HTTP shim refactor onto SessionStore), which makes this test file moot. Cleaning it up now would be throwaway work; flagged in PR description. tests/sdk/python/conftest.py The FakeVerifier import is replaced by an inline _MinimalVerifierStub class. The SDK tests are wire-layer tests (encode/decode + status mapping); their truth is gRPC transport correctness, not verifier numerics. The stub satisfies VerifierProtocol shape but is documented as 'not a verifier mirror'. End-to-end runtime correctness is covered by tests/integration/. Engine / tokenizer doubles (DeterministicEngine, DeterministicTokenizer, _RaisingEngine, _ProxyEngine, etc.) in tests/inference_engine/server/ and tests/inference_engine/scheduler/. These are PR-N2 / PR-N3 scope (per the original 4-PR sequence). CI workflow change ------------------ .github/workflows/ci.yaml: changed --cov=inference_engine.session to --cov=inference_engine.session.store. The coordinator and generator modules are no longer covered on Linux. They reach 100% in the integration suite. Linux verification ------------------ PYTHONPATH=.:sdks/python coverage run -m pytest <Linux gate paths>: 649 passed (was 682 in PR-D1 baseline; -33 net = removed ~50 FakeVerifier-driven tests, added ~17 verifier-independent validation + gRPC error-mapping tests). 100% coverage on 1595 stmts (was 1660 in PR-D1; -65 net stmts is the coordinator + generator now NOT in --cov= scope). Mac M4 evidence (REQUIRED for merge) ------------------------------------ Per ADR 0008 \u00a79: this PR's runtime-correctness evidence lives in the integration suite. Reviewer runs: bash scripts/review_pr_n1_on_mac.sh git add results/platform-tests/pr-n1-mac-* git commit -m 'Mac M4 review evidence for PR-N1' git push Acceptance: all integration tests pass against real Qwen3-0.6B. The INV-3 byte-exact GA gate (PR-E1) is included. Stack ----- PR-N1 is branched off main directly. References to _sync_slab_bytes (introduced by PR-E1c, in flight as PR #52) are deferred to a follow-up after PR-E1c merges; the helper itself is covered by PR-E1c's own tests on its branch. Next PRs -------- PR-N2: remove DeterministicEngine + DeterministicTokenizer. PR-N3: remove server-specific engine doubles. PR-N4: post-N1/N2/N3 CI workflow consolidation. Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
1 parent bec3d7b commit 095528e

10 files changed

Lines changed: 1255 additions & 1324 deletions

File tree

.github/workflows/ci.yaml

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -72,6 +72,14 @@ jobs:
7272
# PYTHONPATH route avoids a setuptools build step in CI.
7373
PYTHONPATH: .:sdks/python
7474
run: |
75+
# PR-N1 (ADR 0008) split: this gate covers ONLY
76+
# verifier-independent code. The Linux runner cannot load
77+
# real Qwen3 weights, and PR-N1 retired the FakeVerifier
78+
# test double that previously stood in for them.
79+
# Verifier-dependent modules — currently
80+
# ``inference_engine.session.coordinator`` and
81+
# ``inference_engine.session.generator`` — move to the
82+
# tests/integration/ suite, gated on Mac M4 / CUDA hosts.
7583
pytest \
7684
tests/inference_engine/server/ \
7785
tests/inference_engine/memory/ \
@@ -85,7 +93,7 @@ jobs:
8593
--cov=inference_engine.memory \
8694
--cov=inference_engine.scheduler \
8795
--cov=inference_engine.pipeline \
88-
--cov=inference_engine.session \
96+
--cov=inference_engine.session.store \
8997
--cov=kakeya \
9098
--cov=training.repr_align \
9199
--cov-report=term \

scripts/review_pr_n1_on_mac.sh

Lines changed: 103 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,103 @@
1+
#!/usr/bin/env bash
2+
# Mac M4 review aid for PR-N1 (no-test-doubles cleanup, scope =
3+
# verifier-protocol mirror classes).
4+
#
5+
# PR-N1 retired FakeVerifier, _LyingVerifier, _RegressingVerifier,
6+
# and _LyingFakeVerifier from the Linux test tree. Their dispatch /
7+
# state-mirror tests moved to tests/integration/test_coordinator_real.py
8+
# and tests/integration/test_generator_real.py, where they run
9+
# against the real Qwen3-0.6B SinkWindowVerifier instead of a
10+
# hand-coded mirror. This script runs that integration suite on
11+
# Apple Silicon and produces the JSON evidence reviewers commit.
12+
#
13+
# Produces 1 artifact:
14+
#
15+
# results/platform-tests/pr-n1-mac-integration-tests-<unix>.json
16+
# pytest -m integration tests/integration/ — coordinator and
17+
# generator integration tests against real Qwen3 + the existing
18+
# INV-3 byte-exact GA gate. Acceptance: all tests pass.
19+
#
20+
# Usage (from repo root, on Mac M4 / arm64):
21+
#
22+
# bash scripts/review_pr_n1_on_mac.sh
23+
#
24+
# Then commit:
25+
#
26+
# git add results/platform-tests/pr-n1-mac-*
27+
# git commit -m "Mac M4 review evidence for PR-N1"
28+
# git push
29+
30+
set -euo pipefail
31+
32+
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
33+
cd "$ROOT"
34+
35+
stamp="$(date +%s)"
36+
out_dir="results/platform-tests"
37+
mkdir -p "$out_dir"
38+
39+
junit="$out_dir/pr-n1-mac-integration-tests-${stamp}.junit.xml"
40+
report="$out_dir/pr-n1-mac-integration-tests-${stamp}.json"
41+
42+
echo "==> integration suite (PR-N1 migrated tests + INV-3 GA gate)"
43+
PYTHONPATH=.:sdks/python python3 -m pytest \
44+
-m integration \
45+
tests/integration/ \
46+
--junitxml="$junit" \
47+
-v
48+
49+
PYTHONPATH=.:sdks/python python3 - "$junit" "$report" <<'PY'
50+
import json
51+
import platform
52+
import sys
53+
import xml.etree.ElementTree as ET
54+
55+
junit_path, out_path = sys.argv[1:3]
56+
jr = ET.parse(junit_path).getroot()
57+
58+
testsuites = list(jr.iter("testsuite"))
59+
total_tests = sum(int(ts.get("tests", "0")) for ts in testsuites)
60+
total_failures = sum(int(ts.get("failures", "0")) for ts in testsuites)
61+
total_errors = sum(int(ts.get("errors", "0")) for ts in testsuites)
62+
total_skipped = sum(int(ts.get("skipped", "0")) for ts in testsuites)
63+
64+
cases = []
65+
for tc in jr.iter("testcase"):
66+
cases.append({
67+
"classname": tc.get("classname"),
68+
"name": tc.get("name"),
69+
"time": float(tc.get("time", 0.0)),
70+
"outcome": (
71+
"failed" if tc.find("failure") is not None
72+
else "errored" if tc.find("error") is not None
73+
else "skipped" if tc.find("skipped") is not None
74+
else "passed"
75+
),
76+
})
77+
78+
report = {
79+
"schema_version": 1,
80+
"kind": "pr_n1_mac_integration_tests",
81+
"host": {
82+
"platform": platform.platform(),
83+
"machine": platform.machine(),
84+
"python": platform.python_version(),
85+
},
86+
"junit": {
87+
"tests": total_tests,
88+
"failures": total_failures,
89+
"errors": total_errors,
90+
"skipped": total_skipped,
91+
"cases": cases,
92+
},
93+
}
94+
with open(out_path, "w", encoding="utf-8") as fh:
95+
json.dump(report, fh, indent=2)
96+
print(f" -> {out_path}")
97+
PY
98+
99+
echo
100+
echo "==> Done. Commit:"
101+
echo " git add $out_dir/pr-n1-mac-*"
102+
echo " git commit -m 'Mac M4 review evidence for PR-N1'"
103+
echo " git push"

0 commit comments

Comments
 (0)