T5CoreMLEmbedder CPU fallback path never recovers — 1,038/1,038 fail in production, dead weight

[totalslacker]

Problem

The CPU fallback path added by #87 (and improved with diagnostics in #90) does not recover any inferences in practice. Every attempt to retry an ANE-failed inference on .cpuOnly also fails, and the embedder rethrows the original IOSurface error.

What's actually keeping bulk-index alive in production is the proactive model reload every reloadInterval calls (also from #87). The CPU fallback is dead weight: it costs an extra (failed) MLModel load + predict per failure, produces no successful results, and contributes nothing to the recovery story.

Production Data

A SafariUnfucker bulk-index run (2026-05-07) processed 5,570 successful inferences and 1,038 failures:

1038  cpu_fallback_failed  (every failure)
1038  Failed to allocate E5 buffer object. E5RT: Failed to allocate memory IOSurface object. (3)  (every failure's reason)
0     recovered_iosurface_exhaustion  (none)

category: "cpu_fallback_failed" fires when: ANE threw IOSurface, CPU fallback was attempted, and CPU also threw. Across 1,038 opportunities, CPU fallback recovered zero inferences.

The ANE pool DID recover during the run — three failure bursts each followed by recovery periods of 5–25 minutes. Recovery timing is consistent with the proactive reload (Layer 2) firing at a 500-call boundary, not with CPU fallback.

Summary

Remove the CPU fallback path (Layer 3b) from T5CoreMLEmbedder entirely. The path has a measured 0% recovery rate and adds per-failure overhead (one extra MLModel load + predict call) without benefit. The proactive reload (Layer 2) is the only mitigation that works; it should be the sole recovery mechanism. This simplifies predictWindow, eliminates cpuPredictorFactory from the actor's internal state, and removes tests for a dead code path.

Note on option (b) — log CPU-side error: The issue originally proposed logging the CPU-side error as a diagnostic step before removal. Based on source inspection, logCPUFallbackFailed (line 613 of T5CoreMLEmbedder.swift) already captures cpuErrorName, cpuErrorReason, and cpuCallStack in the JSONL row per ADR 021's addendum. Option (b) appears already complete. The Research stage should verify this against actual production JSONL rows; if the cpu error fields are absent, option (b) must be implemented before option (a) is shipped.

Requirements

• cpuPredictorFactory property is removed from T5CoreMLEmbedder (both the stored let and all assignments in all init paths).
• The Layer 3b CPU fallback branch in predictWindow (lines ~539–557) is removed. The reactive reload + ANE retry (Layer 3a, lines ~521–537) is retained.
• When an IOSurface exception is caught and the ANE retry also fails, the embedder rethrows the original error with the existing category: "error" JSONL row (the cpuPredictorFactory == nil path). The cpu_fallback_failed category is no longer produced.
• ADR 021 is amended to reflect Layer 3b's removal and the production evidence that motivated it.
• Tests for CPU fallback success (testIOSurfaceFallbackLogsRecovery) and CPU fallback failure (testCPUFactoryThrowsLogsDistinctError, testCPUPredictThrowsLogsDistinctError) are removed.
• All remaining tests continue to pass: proactive reload, autoreleasepool, IOSurface error logging, ANE retry.
• No change to the public init signatures (cpuPredictorFactory was never a public parameter).

Scope

In scope:

• Remove cpuPredictorFactory property and all wiring in production init paths.
• Remove cpuPredictorFactory parameter from the test-only internal init.
• Remove the Layer 3b branch from predictWindow.
• Remove the logCPUFallbackFailed private method and extractCPUErrorFields helper if they are unused after removal (verify — extractCPUErrorFields may have no other callers).
• Remove cpu_fallback_failed JSONL category (no longer produced).
• Update ADR 021 with a second addendum documenting this removal and the production evidence.
• Delete the three stress-test cases that cover CPU fallback behavior.

Out of scope:

• Changing reloadInterval default or adding adaptive byte-pressure reload threshold (tracked separately per ADR 021 addendum).
• Changing any other public API.
• Changes to MLPredictor, CoreMLFailureLogEntry, or the broader JSONL schema (only the cpu_fallback_failed category is retired).

Prior Art / Context

• ADR 021 documents the original three-layer design rationale and the 2026-05-06 addendum (issue ANE IOSurface mitigation (#87) is ineffective in production — CPU fallback fails silently, reload threshold above failure point #90 improvements). This issue adds a second addendum.
• Issue T5CoreMLEmbedder leaks ANE IOSurface buffers — bulk-index fails after ~1k inferences #87 — introduced CPU fallback (Layer 3b).
• Issue ANE IOSurface mitigation (#87) is ineffective in production — CPU fallback fails silently, reload threshold above failure point #90 — added cpu_fallback_failed category and cpu error fields to JSONL rows; also added Layer 3a (reactive reload + ANE retry).
• The production evidence (0/1,038 CPU recovery rate) is determinative. No further data gathering is needed before removing the path.

Risks / Dependencies

• Option (b) completeness: If the Research stage finds that cpu_fallback_failed JSONL rows do NOT include cpuErrorName/cpuErrorReason in the actual production files (contradicting the source code), then option (b) should be implemented first so future engineers have a record of why CPU fallback failed before the code is deleted.
• No public API surface change: cpuPredictorFactory was never exposed in a public init — no callers outside the package are affected.
• Test-only init signature change: The internal factory-based test init accepts cpuPredictorFactory as a parameter. Removing it changes internal-only API — no downstream impact.

[totalslacker]

🏭 Fabrik — stage: Specify
branch: fabrik/issue-93 | commit: d814646 | main: d814646 | 2026-05-07 20:26 UTC

Now I have the full picture. The logCPUFallbackFailed function at line 613 already captures and logs cpuErrorName, cpuErrorReason, and cpuCallStack — option (b) appears to already be implemented per the ADR and code. The issue author's claim that the CPU-side error "is still not surfaced" may be based on inspecting only the name/reason fields (ANE fields) in the JSONL rows, not the cpuError* fields.

The only remaining work is option (a): remove the CPU fallback path entirely. Let me write the spec.

---

Used 13/50 turns, 1k input / 12k output tokens.

[totalslacker]

🏭 Fabrik — stage: Research
branch: fabrik/issue-93 | commit: d814646 | main: d814646 | 2026-05-07 20:29 UTC

I now have a thorough understanding of everything. Let me compile the research findings.

---

Research Findings

Relevant Code

• Sources/SwitchcraftCoreML/T5CoreMLEmbedder.swift — The only production file that changes. Contains the cpuPredictorFactory property (line 111), all three init paths that wire or nil it (lines 209–216, 297, 320–341), the predictWindow method with Layer 3b (lines 498–558), and the four private methods to remove: logRecoveredIOSurface (592–611), logCPUFallbackFailed (613–639), extractCPUErrorFields (644–650), plus isIOSurfaceExhaustion and logNativeException (both survive).

• Tests/SwitchcraftTests/T5CoreMLEmbedderStressTests.swift — Contains all three tests to delete: testIOSurfaceFallbackLogsRecovery (lines 121–176), testCPUFactoryThrowsLogsDistinctError (lines 184–242), testCPUPredictThrowsLogsDistinctError (lines 250–298). Also contains two surviving tests: testStress2000IterationsVariedInputSizes and testReloadTriggeredAtInterval. The deleted tests all pass cpuPredictorFactory: to the factory-based internal init.

• adrs/021-ane-iosurface-pool-exhaustion-mitigation.md — Needs a second addendum (after the 2026-05-06 addendum at line 153) documenting the production evidence, Layer 3b removal, and the simplified two-state JSONL category scheme.

• Tests/SwitchcraftTests/T5CoreMLEmbedderFailureLogTests.swift — Untouched. Uses the stub init (no cpuPredictorFactory), tests category: "error" path — which is preserved.

• Tests/SwitchcraftTests/T5CoreMLEmbedderConcurrencyTests.swift — Untouched. Uses the stub init, tests re-entrancy guard only.

• Tests/SwitchcraftTests/T5CoreMLEmbedderTests.swift — Untouched. Asset-gated, uses the public init, no CPU fallback involvement.

• Tests/SwitchcraftTests/Support/CountingStubPredictor.swift — No change needed. The failInterval: feature still works for testIOSurfaceFallbackLogsRecovery's replacement in the surviving stress test (the stress test that doesn't use CPU fallback). CountingStubPredictor itself stays.

• Sources/SwitchcraftCoreML/MLPredictor.swift — Out of scope, unchanged.

Architecture Notes

T5CoreMLEmbedder is a Swift actor. predictWindow is the innermost predict helper called once per sliding-window iteration from encode. Currently it has four defense layers; Layer 3b is the one being removed.

Current predictWindow control flow (catch block):

catch nativeError as CoreMLNativeError:
  guard isIOSurfaceExhaustion && cpuPredictorFactory != nil else:
    → logNativeException ("error") + throw nativeError
  Layer 3a: force-reload predictor + ANE retry
    → succeeds: return retryResult
    → fails: log warning + fall through to Layer 3b
  Layer 3b: load cpuPredictor + predict on .cpuOnly
    → succeeds: logRecoveredIOSurface ("warning") + return
    → fails: logCPUFallbackFailed ("cpu_fallback_failed") + throw nativeError

After removing Layer 3b, the new control flow:

catch nativeError as CoreMLNativeError:
  guard isIOSurfaceExhaustion else:
    → logNativeException ("error") + throw nativeError
  Layer 3a: force-reload predictor + ANE retry
    → succeeds: return retryResult
    → fails: logNativeException ("error") + throw nativeError

The critical implementation detail: after Layer 3a's catch, instead of falling through to Layer 3b, the code must now call logNativeException and rethrow nativeError. The spec confirms this: "the embedder rethrows the original error with the existing category: "error" JSONL row (the cpuPredictorFactory == nil path)."

Guard condition change: guard isIOSurfaceExhaustion(nativeError), let cpuFactory = cpuPredictorFactory else → guard isIOSurfaceExhaustion(nativeError) else (removing the cpuPredictorFactory unwrap entirely).

appendJSONLRow / CoreMLFailureLogEntry stay unchanged. The optional cpuErrorName/cpuErrorReason/cpuCallStack fields on CoreMLFailureLogEntry will simply always be nil (omitted from JSON) after this change. The spec explicitly leaves these out of scope.

Relevant ADRs

• ADR 021 (021-ane-iosurface-pool-exhaustion-mitigation.md): Directly governing. Documents the original three-layer design and the 2026-05-06 addendum. Needs a second addendum for this issue. No conflict — the ADR documents why the decision was made, and the new addendum documents why it was revised.

• ADR 018 (018-objc-clang-target-for-exception-bridge.md): Referenced by ADR 021. catchingNSException bridge remains; Layer 3a still uses it. No change needed.

• ADR 010 (010-embedder-model-and-asset-distribution.md): ADR 021 cited it to sanction .cpuOnly override. Removing Layer 3b makes this citation in ADR 021 moot — the addendum should note that ADR 010 §(g) no longer applies to the Layer 3 path.

Option (b) Completeness — Verification

The spec's primary risk is: do cpu_fallback_failed JSONL rows in production actually include cpuErrorName/cpuErrorReason fields?

Source code analysis confirms option (b) is complete:

1. logCPUFallbackFailed (line 613) calls extractCPUErrorFields(cpuError) which always returns a non-nil (name, reason, callStack) triple — either from CoreMLNativeError.nativeException destructuring or from (nsErr.domain, nsErr.localizedDescription, []).
2. These non-nil values are passed as cpuErrorName: and cpuErrorReason: to appendJSONLRow.
3. CoreMLFailureLogEntry encodes them as optional fields; since they are non-nil in this path, JSONEncoder will include them in the JSON output.
4. The stress tests testCPUFactoryThrowsLogsDistinctError and testCPUPredictThrowsLogsDistinctError both assert that json["cpuErrorName"] as? String != nil and json["cpuErrorReason"] as? String != nil — these tests currently pass, confirming the fields appear in JSONL output.

Conclusion: option (b) is implemented and verified by passing tests. No blocking work before option (a).

Constraints

1. CoreMLFailureLogEntry must not change (out of scope). The optional cpu* fields stay in the struct even though they'll be nil after this change.

2. No public API change. cpuPredictorFactory was never in a public init. The removal only affects internal init signatures — no downstream impact outside the package.

3. Layer 3a control flow requires explicit log+rethrow after ANE retry failure. Currently the failed-retry catch block just logs a warning and falls through to Layer 3b. After removal, it must explicitly log and rethrow — otherwise IOSurface failures that survive Layer 3a would be silently lost (no JSONL row, no thrown error).

4. No ANE retry test exists in isolation. The spec says "ANE retry" tests should continue to pass, but there is no standalone test for Layer 3a. testIOSurfaceFallbackLogsRecovery exercised Layer 3a+3b together. After deletion, the only Layer 3a coverage is the asset-gated stressRealAsset10kIterations. The planner should consider adding a stub-based ANE retry test.

5. logRecoveredIOSurface is only called from Layer 3b (line 546). After removal, it has no callers and must be deleted. Same for logCPUFallbackFailed (only called at line 554) and extractCPUErrorFields (only called from logCPUFallbackFailed at line 620).

6. The suite doc-comment on T5CoreMLEmbedderStressTests (line 10–16) mentions "reactive CPU fallback" — it will need updating to reflect the removed tests.

Technical Questions

All technical questions are resolved. No open items.

Risks

• Missing standalone ANE retry test. After deleting the three CPU fallback tests, Layer 3a (reactive reload + ANE retry) has no stub-based unit test. The asset-gated 10k stress test covers the overall path but won't run on CI without a local .mlpackage. A planner may want to add a lightweight stub test for Layer 3a to fill this coverage gap. This is not a blocker for the removal but is a test-coverage risk worth surfacing.

• Log+rethrow path correctness. The current Layer 3a catch block only warns and falls through; after removal it must log+rethrow. If this is omitted, IOSurface errors that survive Layer 3a's reload+retry would be swallowed silently. The planner should call this out explicitly in the task checklist.

• Doc-comment drift. The class-level doc-comment, predictWindow's doc-comment, and encode's doc-comment all mention CPU fallback and cpu_fallback_failed. These need updating. Low risk of breakage but important for correctness of the public documentation.

Documentation Impact

No user-facing documentation impact. cpuPredictorFactory was never a public parameter. No public API changes. No CLI commands, config options, or workflow steps are affected. ADR 021 is amended (developer-facing, not user-facing).

---

Used 22/50 turns, 0k input / 9k output tokens.