qa: fix writability check in fake harness + coverage target dir; close BUG-022 - #11
Open
Morgandri1 wants to merge 4 commits into
Open
qa: fix writability check in fake harness + coverage target dir; close BUG-022#11Morgandri1 wants to merge 4 commits into
Morgandri1 wants to merge 4 commits into
Conversation
|
Deployment failed for project wheel with the following error: Learn More: https://vercel.com/morgandri1s-projects?upgradeToPro=build-rate-limit |
Morgandri1
force-pushed
the
qa/sandbox-environment-fixes
branch
3 times, most recently
from
September 6, 2026 17:28
352390c to
0556ae6
Compare
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
Two fixes found running make check in a shared multi-tenant cloud sandbox (six agents, one /data volume) rather than the single-project docker container the tooling assumed: - qa/harness/fake-claude, fake-codex: the credential-routing dump defaulted to /data/wheel-fake-env.jsonl whenever os.path.isdir(dirname) was true. On a shared host /data exists and is listable but not writable by this uid (only /data/projects/<id> is ours) -- isdir was true, every spawn tried the write and died in env_dump(), and since env_dump() runs unconditionally on every spawn, one unwritable default failed 33 of 33 qa:harness-selftest cases that have nothing to do with credentials. Check writability (os.access(..., W_OK)), not just existence. - qa/tools/coverage_gate.py: always forced CARGO_TARGET_DIR to ROOT/target-cov, which lives on the same small shared volume as wheel.db/host.db. An instrumented workspace build there filled a 4.6G disk shared by every agent to 100% before it failed. WHEEL_COV_TARGET_DIR now overrides the location; unset, behaviour is unchanged. Also closes BUG-022: verified 1163cc9's write-proof (mode_holds via BEGIN IMMEDIATE; COMMIT) fixes the engine's non-exclusive open path, by reproducing the bug's exact scenario (a directory blocking -shm) at the crate level against wheel-sqlite directly -- this sandbox has no docker to rebuild wheel-engine:test and re-run the fixture itself. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…sion), and update the handoff BUG-025 (was 023): ENG-journal-override-cannot-disable-recovery is red on a live engine (PM, verified fresh on 9c793c3) but I could not reproduce it at the crate level even mimicking wheel-engine::db::open()'s exact double-configure sequence against wheel-sqlite directly -- TRUNCATE/WAL/DELETE all succeed with a real write. That rules out wheel-sqlite as the site and points at migrate()/ensure_node_tables/tables::query's second connection as the next place to look with real docker access, which this sandbox does not have. Cross-referenced against a concurrent QA session's 023 (free_port(0) returning literal 0, a harness bug that made the same test fail against a healthy engine) -- both were real, compounding: the harness bug hid the actual product-side timing issue (PR #12) underneath it. BUG-026 (was 024): a table node's sqlite table is only re-ensured at engine BOOT (W1, 0692797) -- a table dropped out of band while the engine keeps running stays missing until the next restart. Distinct failure mode, same underlying principle (a node's table must survive anything the board itself does not agree to). Renumbered 023/024 -> 025/026 on rebase: a concurrent QA session had already landed unrelated 023 (free_port(0)) and 024 (panic gates) on main first. docs/handoff/qa.md updated: BUG-022 closed (crate-level verification, docker fixture still owed), make check confirmed green on current main modulo the two sandbox-specific gates fixed in the prior commit, and a note that CI on main is churning too fast right now for a single red run to be evidence without naming the job and SHA. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
r.stderr[-2000:] or r.stdout[-2000:] showed only stderr whenever it was non-empty, sliced to its last 2000 chars from the END -- cargo lists every OTHER crate's test binaries after the one that actually failed, so the slice reliably kept that trailing noise and cut the failing test's name and panic, which sit earlier in the stream. Confirmed on PR #12's CI run: the printed excerpt jumped from "cargo llvm-cov failed" straight into a mid-word fragment of a later "Running tests/..." line, with the real failure nowhere in what got logged (PM, diagnosing a wheel-engine coverage failure with no way to see what failed). Now finds the last "failures:" block (or "panicked at" if the run aborted before cargo could print one) and prints from there, combining stdout AND stderr rather than only whichever one the previous version picked. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
PM pulled CI run 34047133420 (SDK's PR #12): WOW-table-survives-restart/columns fails for real -- "the table came back but would not accept its own configured columns (404) -- it was recreated from something other than the node config". Exactly the failure mode predicted when filing this: a self-heal that re-ensures existence without the node's real TableConfig columns passes the first half and fails the second. PM is holding #12 out of main until SDK rebuilds from the real columns rather than a bare CREATE TABLE. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Morgandri1
force-pushed
the
qa/sandbox-environment-fixes
branch
from
September 6, 2026 18:16
0556ae6 to
892ea66
Compare
Morgandri1
added a commit
that referenced
this pull request
Sep 7, 2026
Three of my messages to PM were beheaded in one evening (§3c #11) and the numbers were asked for five times. The contract already says git is the system of record and a message is only a notification; this applies that to a measurement instead of a bug. Records what the numbers are AND why cargo tree -d returning nothing is the wrong target: 11 of the 12 duplicates are upstream version skew no diff of ours can move, so a zero gate would be red forever for a reason nobody can act on — the permanent-red-X failure from f897af8. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F1nLndrG8rrdr5ar7P6WkN
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
qa/harness/fake-claude/fake-codex: the credential-routing env dump defaulted onisdir(dirname)alone. On a shared multi-tenant sandbox/dataexists but isn't writableby this uid, so every spawn died in
env_dump()(which runs unconditionally) — 33/33qa:harness-selftestcases failed for one root cause unrelated to credentials. Now checksos.access(..., W_OK).qa/tools/coverage_gate.py: always forcedCARGO_TARGET_DIRonto the repo's own volume.On a sandbox where the repo shares a small (4.6G) volume with
wheel.db/host.db, aninstrumented workspace build filled it to 100%.
WHEEL_COV_TARGET_DIRnow overrides thelocation; default behaviour unchanged.
qa/tools/coverage_gate.py: its own failure-reporting also truncated the actual failingtest out of the printed excerpt (
stderr[-2000:] or stdout[-2000:]kept only the LAST 2000chars, which on a real failure is trailing noise from other crates' test binaries listed
afterward, not the panic/FAILED line). Now finds the last
failures:/panicked atmarkerand prints from there. Found while diagnosing PR engine: BUG-022 timeout fix + live table self-heal + fix broken message tests #12's own red coverage gate — that CI log
was unreadable for exactly this reason.
1163cc9's write-proof fix for the engine's non-exclusive sqliteopen path by reproducing the bug's exact scenario (directory blocking
-shm) at the cratelevel against
wheel-sqlite— no docker available in this sandbox to re-run the fullqa:ENG-starts-without-shmimage fixture, noted honestly in the bug closure.unrelated 023/024 on rebase):
ENG-journal-override-cannot-disable-recovery(crate-levelcheck rules out
wheel-sqliteitself as the failure site) andWOW-table-survives-restart(boot-time table healing doesn't cover a live out-of-band drop).
Test plan
qa/harness/selftest.py— all cases pass with the writability fix (previously 33 FAILEDin this sandbox due to
/dataroot not being writable by this uid)<db>-shm) againstwheel-sqlite::open_configured(path, false)on current main — falls back totruncate, write succeedswheel-engine::db::open()'s exact double-configuresequence for
WHEEL_SQLITE_JOURNAL= TRUNCATE/WAL/DELETE — all succeed with a real writemake checkgreen on current main modulo the two fixed gates (verified with isolatedCARGO_HOME/TMPDIRto work around this sandbox's other artifacts — seedocs/handoff/qa.md)python3 -c "import ast; ast.parse(...)"oncoverage_gate.pyafter both edits🤖 Generated with Claude Code