P0: Investigate LSM Indexer performanceSmoke failure — 36.5s vs 10s debug limit on Validate run

[totalslacker]

Summary

IndexerTests.performanceSmoke — "indexing 5000 single-token 128-dim chunks completes within 5s" — took 36.5s vs the 10s debug-mode limit (3.6× overshoot) on the Validate run for PR #94. The test is at Tests/SwitchcraftTests/IndexerTests.swift:393-429 and is not currently disabled on CI. PR #94's Validate stage dismissed the failure as host load and verdicted "READY TO MERGE" anyway — a policy violation. This issue investigates the root cause and fixes it without relaxing the test through policy-violating means.

This is P0: it (and its sibling issues) blocks all merges until resolved.

Requirements

• Root cause of the 36.5s observation must be identified and documented. Exactly one of the following will be true:
  • (a) Real performance regression. A recent commit slowed the indexer hot path (Q4 dequant, residual encode, bucket flush, etc.) by 3–7×. Fix the regression; do not relax the test.
  • (b) Host-load / debug-budget mismatch. The debug path is legitimately 10–20× slower than release (as the test comment acknowledges), and the 10s budget leaves no headroom. Raise the debug budget only with measurement evidence (median of 11+ runs on a quiet machine) recorded in the test comment, and only if the new budget still distinguishes a 3–5× perf regression from normal variance.
  • (c) CI runner regression. macOS-15 runners have changed. Compare CI runtime against a quiet local workstation. If CI is the outlier, move the test to release-only configuration with a tighter budget — never via .disabled(if: CI).
• The test must pass ≥10 consecutive runs on CI in both debug and release after the fix.
• No skip mechanism may be added: no .disabled(if: CI), XCTSkipIf(...), XCTSkipUnless(...), #if !CI, @Test(.disabled(...)), or equivalent.
• No assertion may be deleted, weakened to a tautology, or made a no-op.
• Any budget change must be justified by measurement evidence (median of 11+ runs on a quiet machine) recorded in a comment next to the new value.
• No new failing tests may be introduced anywhere — not in debug, not in release, not intermittently. "Flaky" counts as failing under project policy.
• No previously-passing tests may have skip annotations added as a side effect of this fix.
• The full test suite after the fix must be at least as healthy as main was before this PR (i.e., one fewer failing test — the target — and no new failures elsewhere).

Scope

In scope:

• Investigate IndexerTests.performanceSmoke at Tests/SwitchcraftTests/IndexerTests.swift:393-429.
• Bisect git history (from the last green CI run on main) if a regression is suspected.
• Run timing measurements (≥11 runs, quiet machine, both debug and release) to characterize the actual runtime distribution.
• Fix the underlying cause — regression fix or evidence-backed budget adjustment.
• Verify the full suite is green (no net regression) after the fix.

Out of scope:

• Fixing other unrelated test failures discovered during investigation (file separate issues; treat them as P0 incidents per project policy).
• Skipping or disabling the performance test on CI for any reason.
• Raising the budget without measurement evidence.
• Deleting the test (only permissible if the Research stage produces an ADR demonstrating performance assertions of this kind cannot be measured reliably in this project's testing environment).

Prior Art / Context

• Tests/SwitchcraftTests/IndexerTests.swift:393-429 — the failing test. Budget: < 5.0s release, < 10.0s debug (doubled per test comment to account for debug-mode loop overhead).
• Test uses Indexer(storage: storage, config: .production) with InMemoryStorage, indexing 5000 random rows of 128-dim Q4-quantized embeddings.
• Observed: 36.5s on Validate stage local run (2026-05-07).
• PR Remove Layer 3b CPU fallback (0% recovery rate, dead code) #94 Validate report: incorrectly dismissed as host-load and verdicted READY TO MERGE — a policy violation under .claude/CLAUDE.md.
• Issue T5CoreMLEmbedder CPU fallback path never recovers — 1,038/1,038 fail in production, dead weight #93: prior discussion of why .disabled(if: CI) and "skip flaky on CI" are not acceptable resolutions on this project.
• Sibling WAL skip issues: resolved differently (those were pre-approved skips for known flaky concurrency microbench); this test has no such approval.

Risks / Dependencies

• If the root cause is a real regression, git bisect may reveal the offending commit was already merged to main — the fix may need a clean revert or targeted patch.
• If the root cause is debug-budget too tight, the evidence requirement (11+ runs, quiet machine) means measurement takes real wall-clock time before any code change can be made.
• If CI runner performance has degraded, the fix path (release-only budget or environment-specific budget) must be designed so a 3–5× real regression is still detectable.
• Any code change touching the indexer hot path requires regression tests per project policy (storage, index, codec files all have this requirement).

Acceptance Criteria

• Root cause identified and documented (real regression / host-load budget / runner change).
• If real regression: fix applied and test passes within the original budget.
• If budget adjustment: new budget set with measurement evidence (median of 11+ runs) recorded in the test comment, and the budget still distinguishes a 3–5× regression from normal variance.
• Test passes ≥10 consecutive runs on CI in both debug and release.
• No .disabled(if: CI), XCTSkip, #if !CI, or equivalent skip mechanism added anywhere.
• No previously-passing test newly fails or is newly skipped as a side effect.
• Full swift test (debug) and swift test -c release suite is at least as healthy as main was before this PR, with exactly one fewer failure (the target test).
• CI is green on both jobs (swift test (macOS) and swift test -c release (macOS)) on the PR.

Engineering Policy Reminder

Per .claude/CLAUDE.md: failing tests never merge. The Validate stage on PR #94 violated this by verdicting "READY TO MERGE" with this test red. Do not repeat that pattern on this issue's PR.

A fix that makes the target test pass by introducing fragility elsewhere is not a fix — it is moving the bug. Stop and escalate to a human comment instead of merging in that case.

[totalslacker]

Hard constraint for the agent picking up this issue

Do not introduce any new failing tests, anywhere, for any reason.

This includes:

• ❌ No new tests that fail locally on a clean checkout (debug or release).
• ❌ No new tests that fail intermittently — "flaky" tests count as failing tests under the project policy in .claude/CLAUDE.md.
• ❌ No new .disabled(if: ...), XCTSkipIf(...), XCTSkipUnless(...), #if !CI, @Test(.disabled(...)), or any other skip annotation on tests that previously ran.
• ❌ No "fixing" the existing failure by deleting the assertion, weakening it to a tautology, or making the test a no-op.
• ❌ No raising of a budget/threshold without measurement evidence (e.g. median of 11+ runs on a quiet machine) recorded in a comment next to the new value.

The Validate stage on this issue must verify with a fresh swift test (debug) and swift test -c release that:

1. The test this issue targets passes ≥10 consecutive runs.
2. No other test newly fails — i.e. the full suite is at least as healthy as main was before this PR, with one fewer failing/skipped test (the one you just fixed).
3. CI on the PR is green on both jobs (swift test (macOS) and swift test -c release (macOS)).

If you find a way to make this issue's target test pass only by introducing fragility somewhere else, that's not a fix — it's moving the bug. Stop and escalate to a human comment instead of merging.

(See also: issue #93 thread comment about why .disabled(if: CI) and "skip flaky on CI" are not acceptable resolutions on this project.)

[totalslacker]

🏭 Fabrik — stage: Specify
branch: fabrik/issue-97 | commit: d814646 | main: d814646 | 2026-05-07 21:15 UTC

The comment from @totalslacker adds a hard constraint — not answering an open question, but adding explicit requirements about test hygiene for the Implement and Validate stages. I'll incorporate this into a properly structured spec body.

[totalslacker]

🏭 Fabrik — stage: Research
branch: fabrik/issue-97 | commit: e83d644 | main: e83d644 | 2026-05-07 21:24 UTC

Now I have enough information to write comprehensive research findings. Let me produce the output.

---

Research Findings

Relevant Code

• Tests/SwitchcraftTests/IndexerTests.swift:393-429 — performanceSmoke test. Adds 5000 single-token 128-dim embeddings then calls flush() once; times only that pair; asserts < 10s (debug) / < 5s (release). Budget comment says "Swift loops in the residual encode path are 10–20× slower" — this attribution is incorrect (see Architecture Notes below).

• Sources/SwitchcraftCore/KMeans/KMeans.swift — Contains the two paths of interest: assignAll() (SGEMM via cblas_sgemm + plain Swift argmax loop) and cluster() (calls assignAll 6× for m=5000, k=1131, dims=128). The SGEMM is always Accelerate-compiled (fast in debug and release). The argmax loop is unoptimized Swift in debug mode.

• Sources/SwitchcraftCore/Indexer/Indexer.swift — performFlush() calls k-means and then writes k=1131 buckets via 1131 sequential await storage.insertBucket() calls. Rehydration init code (lines 115–172) runs only when storage has existing generations; the perf test starts with empty storage, so this is a no-op (early return at line 116).

• Sources/SwitchcraftCore/Indexer/IndexerConfig.swift — .production config: l0Capacity=1024, lsmFanout=16, giving L0 capacity 16384. 5000 < 16384, so all embeddings land in a single flush at level 0.

• Sources/SwitchcraftCore/Codec/Q4Codec.swift — encodeResiduals() iterates over residual floats. For 1131 buckets × avg 4.4 members × 128 dims ≈ 636K floats — this is not the dominant cost despite what the test comment claims.

• Sources/SwitchcraftCore/Indexer/LZ4.swift — Called once per bucket via IndicesCodec.encode. For a bucket with ~4 pairs, the input is 32 bytes. Trivially fast.

• Sources/SwitchcraftCore/Storage/InMemoryStorage.swift — In-memory actor. insertBucket copies the BucketRecord struct (~800 bytes of Data) into a dict. Trivial per call; 1131 actor hops total.

• adrs/012-performance-regression-thresholds.md — Relevant precedent: PerformanceTests was gated to release-only after debug mode on the macos-15 runner was measured at ~1.35× slower than M1 Ultra dev box (p50 326 ms vs 241 ms). The reasoning was that debug floors calibrated on a dev box fail on CI runners without meaningful regression signal.

• adrs/004-lsm-cascade-policy.md — Documents the k-formula: k = round(kCoefficient × sqrt(m)). With m=5000, k=round(16×70.71)=1131. Fixing this value or citing it before any change.

Architecture Notes

Computational profile for performanceSmoke:

With production config and 5000 single-token 128-dim embeddings, one flush does:

Step	Work	Debug speed
k-means SGEMM (cblas_sgemm)	m=5000, k=1131, dims=128; called 6×	Always fast (Accelerate, pre-compiled)
k-means argmax inner loop	6 × 5000 × 1131 = 33.93M iterations	Dominant debug cost — plain Swift loop
k-means sum accumulator	5 × 5000 × 128 = 3.2M float adds	Moderate debug cost
Q4 encode 636K floats	1131 buckets × ~4.4 × 128	Minor
1131 × insertBucket actor hops	Tiny dict insert each	Negligible

The test comment blames "Swift loops in the residual encode path" — this is inaccurate. The residual encode is ~636K operations. The k-means argmax is 33.93M iterations over a 22.6 MB scores matrix (allocated fresh on each of 6 calls). The argmax loop is the actual dominant cost in debug mode.

In release mode, LLVM vectorizes the argmax into SIMD code → ~0.1s. In debug mode, every array subscript has a bounds check and no vectorization → original measurement was 5.5s (commit 931d7bf message explicitly states this). That gives a true debug/release ratio of ~55×, not the 10–20× the test comment claims.

Hot path detail — assignAll:

1. Allocates [Float](repeating: 0, count: m * k) = 22.6 MB per call, 6 calls = 135.7 MB of allocation churn
2. cblas_sgemm fills the 22.6 MB scores matrix (fast)
3. Swift loop: for row in 0..<m { for c in 0..<k { if scores[off+c] > best { ... } } } — 5000 × 1131 iterations, bounds-checked in debug

Key gap: vDSP_maxvi from the already-imported Accelerate framework finds the maximum and its index in a float vector. It is always fast (pre-compiled, SIMD). A drop-in replacement for the inner argmax loop would process each row's k=1131 scores using vDSP_maxvi — this would make the argmax fast in debug mode too, eliminating the 55× debug/release ratio and giving orders-of-magnitude more headroom.

Relevant ADRs

• ADR 004 (004-lsm-cascade-policy.md): Defines k = round(kCoefficient × sqrt(m)). With production config and 5000 embeddings, k=1131. Any change to how argmax is computed in assignAll must not change the resulting assignments (same argmax semantics).

• ADR 012 (012-performance-regression-thresholds.md): Directly relevant. Documents that the macos-15 GitHub Actions runner is ~1.35× slower than M1 Ultra in debug mode for this kind of work, and that PerformanceTests was moved to release-only for exactly that reason. The same runner-speed gap is a risk factor for performanceSmoke's debug-mode budget. Also establishes the measurement methodology and tightening policy.

Constraints

1. No skip mechanism: The spec forbids .disabled(if: CI), XCTSkipIf, #if !CI, etc. The test must pass in both debug and release on CI.

2. Budget change requires evidence: Any change to the 10s debug or 5s release budget must be justified by median of ≥11 measurements on a quiet machine, recorded as a comment next to the new value.

3. Argmax semantics must be preserved: Any optimization to assignAll must produce identical assignments as the current logic (i.e., the maximum-dot-product centroid wins; ties may break differently if vDSP_maxvi picks a different tie-break than the current left-to-right scan, but ties between float values are astronomically rare in practice and Witchcraft has no tie-break contract).

4. Regression test requirement: If KMeans.swift is modified, the existing 8 KMeans unit tests must pass unchanged, and new regression tests are required per project policy.

5. performanceSmoke is not in PerformanceTests: It is in IndexerTests.swift with no .enabled(if: isReleaseBuild) gate. Unlike PerformanceTests, it currently runs in both debug and release. The spec prohibits adding a CI-skip annotation.

6. InMemoryStorage actor isolation: The 1131 insertBucket calls each involve an actor hop. With Swift's cooperative thread pool, each hop completes synchronously from the test's perspective but incurs scheduling overhead that is not controllable from the test side.

Technical Questions

• Measurement required before any budget change: The Implement agent must run ≥11 consecutive debug-mode runs of performanceSmoke on a quiet machine (no parallel builds, no Xcode, minimal background activity) and record the median and 95th-percentile timings. If the median is ≤8s and the 36.5s was a load outlier, the simplest fix is to either: (a) add a comment documenting the load sensitivity, or (b) implement the vDSP_maxvi argmax optimization (which eliminates the load sensitivity by making the argmax always fast). If the median is >10s, the vDSP_maxvi optimization is the required fix path.

• vDSP_maxvi tie-break parity: Does the current argmax loop pick the first index with the maximum score (left-to-right scan), and does vDSP_maxvi also return the first such index? If vDSP_maxvi returns a different index on a tie, the assignment is still correct (nearest-centroid semantics are satisfied) but bucket blobs would differ from the current implementation. The existing byte-identical test (sameInputSameBlobs) would fail. Either verify vDSP_maxvi's tie-break matches, or update that test with a comment explaining why blob identity may differ.

• vDSP_maxvi correctness when all scores are identical: Clusters with random 128-dim vectors are rarely tied, but the reseeded-empty-cluster case could produce identical centroids. Verify the fallback behavior.

Risks

1. Root cause may be load artifact, not code regression: All hot path code changes since 931d7bf are inconsequential for the perf test (rehydration skipped on empty storage; sampleDistinct shortcut removal costs <15μs). If measurements confirm the median is still ~5.5s on a quiet machine, the 36.5s was an extreme-load anomaly. A pure budget adjustment (with evidence) would be the minimal fix, but the 1.8× headroom is thin.

2. If vDSP_maxvi is used, sameInputSameBlobs test must be reviewed: The byte-identical test at lines 255–310 of IndexerTests.swift verifies that the same seed + same input produces byte-identical bucket blobs. If vDSP_maxvi breaks ties differently than the current left-to-right scan, the test would fail on tie inputs. This is unlikely for random 128-dim vectors but must be checked. If it fails, the expected hex values must be updated with a comment explaining the tie-break change.

3. macos-15 runner budget: Even after optimization, if the fix only runs measurements on an M3/M4 dev box, the CI runner may be slower. ADR 012 documented a 1.35× factor. Measurements on the local machine should include a CI-overhead multiplier when setting the new budget (i.e., new_budget ≥ median_local × 1.5 at minimum).

4. .timeLimit(.minutes(1)) trait: The test has a 1-minute Swift Testing time limit. This is the outer guard and not the actual assertion — the assertion at line 428 is the real performance gate. The .timeLimit just prevents the test from hanging the suite if something catastrophic happens. This is fine and should not be removed.

Documentation Impact

No user-facing documentation impact. This fix touches only internal test infrastructure (IndexerTests.swift) and potentially KMeans.swift with an Accelerate optimization. There are no new CLI commands, config options, workflow steps, or output formats introduced.

---

Used 32/50 turns, 2k input / 20k output tokens.