Repository navigation
flaky: TestResolvedPathBackfill_WriteHoldUnderBudget_188 — wall-clock write-hold budget fails under CI load #267
Description
Activity
Plan — CS-pve-agent2 #267
Initial findings. These are measured on
0572e7f9with-cover, while twogo test ./...loops oncmd/serverrun alongside (load avg ~5–6 on 4 cores). I timed each phase of 200 batches of 500 rows:phase median p95 max read (outside tx) 1.9 ms 2.4 ms 6.7 ms resolve (outside tx) 1.1 ms 1.6 ms 3.8 ms writerMu wait + BEGIN 0.03 ms 0.05 ms 0.3 ms 500 UPDATEs + watermark 5.6 ms 7.9 ms 11.4 ms COMMIT 15.5 ms 30.3 ms 45.4 ms hold (total) 21.6 ms 36.3 ms 52.7 ms - COMMIT is about 75% of the hold. That is the WAL fsync (
synchronous=2, FULL, the driver default). - With
PRAGMA synchronous=OFFon the same load, the hold drops to a 3.1 ms median and a 12.8 ms max. - No auto-checkpoint fired in any batch.
- Resolution is already outside the transaction.
- Nothing in the transaction can be moved out without changing semantics: the UPDATEs, the watermark and the COMMIT must stay atomic.
- So the overshoot is disk-flush latency on a loaded runner, not code inside the transaction.
Plan:
- Leave the semantics of
resolvedPathBackfillBatchunchanged. Add one test seam for the resolver call, in the same style asreadProcSelfIOFn. - Rewrite
TestResolvedPathBackfill_WriteHoldUnderBudget_188with deterministic structural assertions. Test-only SQLite triggers call a Go probe function, so the probe sees every row UPDATE and every watermark write inside the real transaction. The test asserts:- rows per write transaction ≤ the batch size;
- exactly one write transaction (
WriterTx) and one watermark write per batch; - no resolution while
writerMuis held.
- Drop the absolute 250 ms wall-clock assertion. Log the hold, and add
BenchmarkResolvedPathBackfillBatch, which reports the hold per batch. The PR explains why no timing assertion stays. - Check two mutants: resolution moved into the
WriterTxclosure, and all rows in one transaction. Both must fail the new test. I will also show whether the old timing test catches them. - Run
-count=50 -coverunder parallel CPU load at least twice, then-raceon the whole package,go vet,gofmt, andtest-all.sh. Then open a draft PR.
Writes stay in the ingestor. No new
map[string]interface{}. The workflow files are not touched.- COMMIT is about 75% of the hold. That is the WAL fsync (
- added a commit that references this issue
on Oct 5, 2026
Problem
TestResolvedPathBackfill_WriteHoldUnderBudget_188incmd/ingestor/resolved_path_backfill_188_test.gofailed on master CI. The run was 37321760306 for0572e7f9, in "Build and test Go ingestor (with coverage)":The test measures the wall-clock write-transaction hold of the slowest of 10 batches (5000 rows, batch size 500) and asserts it is under a fixed 250 ms. The result depends on runner load and coverage instrumentation, not only on the code. The commit under test changed only two unrelated test files, and the same test passed on master earlier the same day.
Proposed approach
Keep the guarantee the test protects: one backfill batch must not hold the single write connection for long enough to stall live ingest. Make the assertion robust:
BEGIN).-coverand-race, and keep a separate benchmark (BenchmarkResolvedPathBackfillBatch) that reports the hold.Acceptance
go test -count=50 -run TestResolvedPathBackfill_WriteHoldUnderBudget_188 -cover ./...incmd/ingestorunder parallel CPU load.