Embedder must enforce input-size guard internally — oversized inputs poison ANE pool for all subsequent inferences

[totalslacker]

Problem

T5CoreMLEmbedder (and any other Embedder implementation the library provides) has no internal upper bound on the total number of tokens it will pass toward a CoreML prediction. Callers can hand it inputs the underlying CoreML model literally cannot allocate output for, and the failure mode is catastrophic and silent for the pool, not just the call:

• One oversized input throws an MLE5OutputPortBinder bindAndReturnError IOSurface allocation failure during output binding.
• The ANE pool is left in a degraded state.
• Every subsequent inference — including small inputs — then fails with the same allocation error, until the host process restarts.

This was observed during a SafariUnfucker bulk-index run: the first failure had inputLength=593,285 tokens (~600k). At typical T5 dims (768) × 4 bytes per float, the output tensor for a sequence that long is roughly 1.8 GB of contiguous IOSurface — well outside what the runtime can allocate. After that single page, every subsequent input failed (sizes from <1k to 100k+ tokens, 6,157 failures total in the run, all the same error).

The ~600k-token page that triggered the poisoning was a real page from the user's browsing history. Real-world content includes huge docs, paginated GitHub PRs, archived RSS dumps, etc. This is not a synthetic edge case.

Why this is the embedder's job, not the caller's

• The embedder is the only component that knows the model's capacity, the dim, the dtype, and the realistic IOSurface ceiling on the platform it's running on. Asking every caller (CLI consumers, SafariUnfucker, MCP servers, batch jobs) to know this and pre-truncate is bug-prone — each caller has to learn this lesson the hard way.
• The contract must be: encode(_:) either returns embeddings or throws a typed, recoverable error. It must never put the embedder in a state where the next call fails too.
• Even if a caller "should know better," it's a footgun: Switchcraft is the library that exists specifically because consumers don't want to think about CoreML internals.

Summary

Add a configurable overflow policy to T5CoreMLEmbedder (and expose it via a maxInputTokens property on the Embedder protocol or the concrete type) so that no input exceeding a safe maximum ever reaches MLPredictor.predict. On overflow the embedder either silently truncates the token sequence to the safe maximum (default) or throws a typed EmbedderError.inputTooLarge(actual:max:) error, depending on the configured policy. The maximum is documented and queryable by consumers.

Requirements

• R1: Overflow guard — After tokenization, if the token count exceeds the embedder's maxInputTokens limit, the configured overflow policy is applied before any call to MLPredictor.predict (or equivalent). No oversized token sequence is ever passed to the CoreML backend.
• R2: Two policies — The embedder supports two policies, selectable at init:
  • .truncate (default) — silently truncate the token sequence to maxInputTokens and embed the prefix, returning successfully.
  • .reject — throw EmbedderError.inputTooLarge(actual: Int, max: Int) so the caller can decide whether to skip, summarize, or split the input.
• R3: Typed error — EmbedderError.inputTooLarge(actual: Int, max: Int) is a new public error type (or new case on an existing public error enum) in SwitchcraftCore or SwitchcraftCoreML. It must be Sendable and Equatable.
• R4: Exposed maximum — T5CoreMLEmbedder exposes a nonisolated let maxInputTokens: Int property. Consumers can read this at any time without entering the actor.
• R5: No pool poisoning — A stress test of 1,000 sequential encode calls, interleaved with one or more deliberate inputs well above maxInputTokens, must complete without any IOSurface allocation failures or ANE pool degradation. Every normal-sized input must succeed.
• R6: Default is truncate — The default overflow policy (when none is specified at init) is .truncate. This matches the most common consumer need (search, classification, retrieval) where the prefix is informative.
• R7: Tests — New tests must cover: (a) truncation policy encodes without error and returns non-empty embeddings; (b) reject policy throws the expected typed error; (c) the stress test described in R5; (d) maxInputTokens property returns the expected value.

Scope

In scope:

• Adding the overflow policy and maxInputTokens to T5CoreMLEmbedder
• Adding the EmbedderError.inputTooLarge error type
• Exposing maxInputTokens on T5CoreMLEmbedder (and optionally adding it to the Embedder protocol — see Open Questions)
• Tests for all new behavior per R7

Out of scope:

• Post-hoc ANE pool recovery (that is issue T5CoreMLEmbedder leaks ANE IOSurface buffers — bulk-index fails after ~1k inferences #87's concern; this issue is structural prevention)
• Chunking/splitting strategies — callers who want to split rather than truncate can do so using the typed error from the .reject policy
• Text-length pre-checks before tokenization (a fast but imprecise pre-filter may be added as an optimization, but the authoritative check is on the token count after tokenization)
• Changes to SlidingWindow window-level limits (the per-window cap of windowSize = 512 already exists; the missing guard is on total token count)

Open Questions

(None currently blocking. Questions below are design decisions for the Research/Plan stages.)

• Q1: Should maxInputTokens be added to the Embedder protocol? Adding it to the protocol makes it queryable on any embedder but requires all conformers to implement it. An alternative is to keep it only on T5CoreMLEmbedder and let callers downcast when needed. Since the safety contract is embedder-internal, keeping it on the concrete type may be sufficient for now.
• Q2: What is the concrete safe value for maxInputTokens on the XTR-base-en model? The issue notes it should be "derived from the model's actual capacity." For fixed-window models this is windowSize (512). For models with dynamic input shapes, the limit may be derived from a practical IOSurface budget (e.g., 8,192 or 16,384 total tokens). Research should determine the right value and whether it should be hardcoded, computed at model load time, or passed by the caller.
• Q3: What is the mechanism of the 1.8 GB failure — single dynamic-shape prediction or batch pre-allocation? This matters for where exactly the guard must be inserted (pre-tokenization total count vs. per-window count) but does not change the observable contract. Research should clarify.

Prior Art / Context

• T5's published max_position_embeddings is 512. The Switchcraft implementation already uses a windowSize: Int = 512 parameter and SlidingWindow to handle inputs longer than 512 tokens by splitting into overlapping windows. The new guard applies to total token count across all windows, not the per-window limit.
• Hugging Face Transformers enforces this silently in its AutoTokenizer via truncation=True, max_length=<model.config.max_position_embeddings> — a well-established convention for the truncation-default pattern.
• ANE pool poisoning via IOSurface allocation failure is a documented footgun in CoreML usage; the fix is always pre-validation, not recovery (recovery of a poisoned pool requires process restart and is unreliable).
• Issue T5CoreMLEmbedder leaks ANE IOSurface buffers — bulk-index fails after ~1k inferences #87 adds post-hoc recovery as a safety net but is explicitly not a substitute for this structural prevention.

Risks / Dependencies

• Relationship to T5CoreMLEmbedder leaks ANE IOSurface buffers — bulk-index fails after ~1k inferences #87: This issue is independent of and takes priority over T5CoreMLEmbedder leaks ANE IOSurface buffers — bulk-index fails after ~1k inferences #87. Even if T5CoreMLEmbedder leaks ANE IOSurface buffers — bulk-index fails after ~1k inferences #87 is never merged, this fix prevents the failure mode entirely. If both land, T5CoreMLEmbedder leaks ANE IOSurface buffers — bulk-index fails after ~1k inferences #87's recovery becomes a true last-resort net rather than the primary defense.
• Protocol change risk: If maxInputTokens is added to Embedder, all existing conformers (including test mocks) will require updates. The Research stage should enumerate all conformers.
• Truncation and recall: Truncating to the first N tokens is semantically lossy for documents where relevant content appears late. This is a known trade-off and acceptable for the library's primary use case (retrieval). The .reject policy exists for callers that cannot tolerate silent data loss.

[totalslacker]

🏭 Fabrik — stage: Specify
branch: fabrik/issue-89 | commit: e37ff5f | main: e37ff5f | 2026-05-06 15:06 UTC

Now I have everything I need. The spec is clear enough to complete without blocking — the fabrik:yolo label plus the well-defined acceptance criteria mean I should write a complete spec and advance.

---

Used 10/50 turns, 0k input / 14k output tokens.

[totalslacker]

🏭 Fabrik — stage: Research
branch: fabrik/issue-89 | commit: 39344b5 | main: 39344b5 | 2026-05-06 15:11 UTC

I now have enough information to write comprehensive research findings.

Research Findings

Relevant Code

• Sources/SwitchcraftCoreML/T5CoreMLEmbedder.swift — Primary change target. The public actor implementing Embedder via CoreML. encode(_:) at line 321 is where the overflow guard inserts: after tokenizer.encode(text, addSpecialTokens: true) (line 342), before SlidingWindow.plan(...) (line 364). Also hosts T5CoreMLEmbedderError, which may gain the new inputTooLarge case (or a new EmbedderError type will be introduced in SwitchcraftCore).

• Sources/SwitchcraftCore/Embedding/Embedder.swift — The Embedder protocol. Currently declares only dims, modelIdentifier, and encode(_:). Relevant to Q1: whether maxInputTokens belongs here or on the concrete type only.

• Sources/SwitchcraftCoreML/SlidingWindow.swift — Pure-Swift sliding-window planner/merger. No changes needed here; the overflow guard sits above this, preventing too many windows from being planned.

• Sources/SwitchcraftCoreML/MLPredictor.swift — MLPredictor seam used by tests. Not changed, but relevant for understanding the stub injection points for overflow tests.

• Sources/SwitchcraftCore/SwitchcraftCore.swift — Module umbrella for SwitchcraftCore. A new EmbedderError type (per spec R3) would live here or near Embedder.swift.

• Sources/SwitchcraftMetal/T5MetalEmbedder.swift — Second Embedder conformer. Uses SlidingWindow identically to T5CoreMLEmbedder. Its workspace buffers are pre-allocated at init for exactly windowSize tokens; many Metal command buffer submissions could cause memory pressure but there is no IOSurface failure mode. If maxInputTokens is added to the protocol, this file needs a new property implementation.

• Tests/SwitchcraftTests/Support/MockEmbedder.swift — Third Embedder conformer (test-only). Trivial to add maxInputTokens: Int { Int.max } if protocol gains the property.

• Tests/SwitchcraftTests/T5CoreMLEmbedderStressTests.swift — Existing stress/reload tests using CountingStubPredictor. New overflow tests should follow this file's pattern.

• Tests/SwitchcraftTests/Support/CountingStubPredictor.swift — Fast stub that can simulate IOSurface failures. Overflow tests will use the same stub — no new predictor needed.

---

Architecture Notes

Data flow through T5CoreMLEmbedder.encode(_:):

1. Whitespace trim → early return for empty input
2. tokenizer.encode(text, addSpecialTokens: true) → [Int32] tokens ← overflow guard inserts HERE
3. Proactive reload check (callCount % reloadInterval == 0)
4. SlidingWindow.plan(tokenCount: tokens.count, windowSize: windowSize, stride: stride) → window start offsets
5. For each start: build [1, windowSize] Int32 MLMultiArray, call predictWindow(provider:inputLength:)
6. predictWindow wraps each call in autoreleasepool { catchingNSException { predictor.predict(input:) } } with IOSurface CPU fallback (ADR 021 Layer 3)
7. SlidingWindow.merge(...) → final [Float]

The overflow guard is a single insertion between steps 2 and 3 — it either truncates tokens to maxInputTokens elements or throws before any CoreML work begins.

Mechanism of the 1.8 GB failure (Q3 resolved):

The CoreML model's input shape is fixed at [1, 512] (ADR 010(c)). It is physically impossible to pass 593,285 tokens in one predictor.predict call. The actual failure path is:

593,285 chars ÷ ~4 chars/token ≈ 148k tokens → SlidingWindow.plan generates ≈ (148,000 − 512) / 256 + 1 ≈ 577 windows

577 consecutive IOSurface-backed CoreML predictions within a single encode() call exhaust the ANE pool before it can drain, even with per-window autoreleasepool. ADR 021 Layer 1 validates pool drainage between individual window calls, but a 577-window burst overwhelms the mechanism. The "1.8 GB" figure in the issue description assumes a hypothetical variable-length model — it is not the literal failure mode, but the conclusion (pool exhaustion after one oversized input) is accurate.

The proactive reload (ADR 021 Layer 2) increments callCount once per encode call, not once per window. A 577-window call counts as 1 against the reload interval, so no reload fires mid-encode for the long input. This is the gap the overflow guard closes.

Relevant ADRs

• ADR 011 (011-sliding-window-long-input-strategy.md): Governs the sliding-window strategy. Section (f) notes windowSize, stride, and minNorm are configurable at init with well-defined defaults. The new maxInputTokens parameter follows the same pattern: a configurable, nonisolated let property with a safe default.

• ADR 021 (021-ane-iosurface-pool-exhaustion-mitigation.md): The three-layer defence added by issue T5CoreMLEmbedder leaks ANE IOSurface buffers — bulk-index fails after ~1k inferences #87. Issue Embedder must enforce input-size guard internally — oversized inputs poison ANE pool for all subsequent inferences #89 is the structural prevention layer that sits above Layer 1/2/3. ADR 021 explicitly acknowledges this is independent and complementary: "Issue Embedder must enforce input-size guard internally — oversized inputs poison ANE pool for all subsequent inferences #89 is independent of and takes priority over T5CoreMLEmbedder leaks ANE IOSurface buffers — bulk-index fails after ~1k inferences #87." No conflict — the overflow guard prevents oversized inputs from ever entering the window loop, so ADR 021's layers never see the problematic burst.

• ADR 009 (009-public-api-shape.md): Locks the Embedder protocol signature. Adding maxInputTokens to the protocol is an additive extension but must be cited as a protocol-level change. Section (b) describes the current protocol; the overflow guard's nonisolated let maxInputTokens: Int property follows the same pattern as dims and modelIdentifier.

• ADR 010 (010-embedder-model-and-asset-distribution.md): Section (c) confirms the fixed [1, 512] input shape. The maxInputTokens guard operates at a higher level than the per-window 512-token limit and is consistent with ADR 010's model-shape contract.

---

Constraints

1. Fixed CoreML input shape. The XTR-base-en model has a static [1, 512] input (ADR 010(c)). The overflow guard targets total token count across all windows, NOT the per-window limit. These are two distinct, independent limits.

2. nonisolated requirement. The spec requires maxInputTokens to be readable without entering the actor. This follows the existing pattern (dims, modelIdentifier, minNorm, windowSize, stride are all nonisolated let). A let stored property set at init satisfies this automatically.

3. Actor isolation for the policy check. The overflow guard runs inside the actor (in encode(_:), which is actor-isolated), so no concurrency issues with reading maxInputTokens or overflowPolicy.

4. Error type must be Sendable and Equatable. The spec requires this. If the error is a new case on T5CoreMLEmbedderError, that enum already is both. If it's a new EmbedderError enum in SwitchcraftCore, those conformances are trivial to add.

5. Default is .truncate, not .reject. This is explicitly specified (R6). Truncation should preserve the first maxInputTokens tokens (prefix-truncation), which is the established convention (HuggingFace truncation=True, max_length=...).

6. Module boundaries. EmbedderError in SwitchcraftCore can be thrown by both T5CoreMLEmbedder and T5MetalEmbedder. T5CoreMLEmbedderError in SwitchcraftCoreML is accessible only to CoreML callers. The spec names it EmbedderError.inputTooLarge, suggesting a new enum in SwitchcraftCore is preferred.

7. SwitchcraftCoreML target gating. T5CoreMLEmbedder is inside #if canImport(CoreML). The new overflow policy and error type do NOT need CoreML; they are pure Swift. If they live in SwitchcraftCore, they're always available. If they live in SwitchcraftCoreML, they remain gated.

---

Technical Questions

• TQ1 (Q1 from spec): Add maxInputTokens to Embedder protocol or keep it on T5CoreMLEmbedder only? Adding it to the protocol requires T5MetalEmbedder and MockEmbedder to implement it. T5MetalEmbedder has no IOSurface failure mode but many Metal submissions from a huge input do incur memory/scheduling cost. The simplest correct approach: add maxInputTokens: Int { get } to the protocol (making the safety contract queryable by any consumer) and have MockEmbedder return Int.max and T5MetalEmbedder return its own sensible default (or Int.max if no guard is added to it). This avoids callers needing to downcast.

• TQ2 (Q2 from spec): What concrete value should maxInputTokens default to? The CoreML model's per-window input is fixed at 512 tokens. The question is how many total tokens (i.e., how many windows) is safe to process in a single encode call without risking ANE pool exhaustion. The existing ADR 021 stress tests drive 10k short inputs (≤600 tokens → 1-2 windows each). 8,192 total tokens → ~31 windows; 16,384 → ~63 windows. Both seem safe based on ADR 021's autoreleasepool-per-window evidence. The spec suggests expressing this as a multiplier of windowSize or as a flat constant; the Plan stage should choose.

• TQ3: New EmbedderError type in SwitchcraftCore or new case on T5CoreMLEmbedderError in SwitchcraftCoreML? The spec uses the name EmbedderError.inputTooLarge, implying a new type. A new EmbedderError enum in SwitchcraftCore (next to Embedder.swift) is more forward-compatible — T5MetalEmbedder could throw it too. A new case on the existing T5CoreMLEmbedderError is simpler but CoreML-scoped. The Plan stage should choose; the research recommendation is SwitchcraftCore for the reason above.

• TQ4: Where should the OverflowPolicy type be declared? Options: (a) a nested type T5CoreMLEmbedder.OverflowPolicy (scoped to the concrete type), (b) a free public enum EmbedderOverflowPolicy in SwitchcraftCoreML, (c) a free public enum EmbedderOverflowPolicy in SwitchcraftCore. Options (a) and (b) are scoped to the CoreML module; option (c) keeps policy types alongside the protocol. If Q1 resolves to keep overflow on the concrete type only, option (a) or (b) is fine. If the protocol gains overflow support, option (c) is cleaner.

• TQ5: Should T5MetalEmbedder also gain the overflow guard? The Metal embedder uses SlidingWindow identically. While it lacks IOSurface exhaustion, generating 577 Metal command buffers in sequence is both slow and memory-intensive. If maxInputTokens goes on the protocol (TQ1), a matching guard in T5MetalEmbedder.encode is natural. If it stays on the concrete type, the Metal embedder is untouched. This is a Plan-stage decision.

---

Risks

1. maxInputTokens chosen too low. A default of 512 tokens (the window size) would truncate most multi-sentence inputs, breaking the sliding-window value proposition. The value must be substantially above windowSize.

2. Protocol change requiring all conformers to update. If maxInputTokens is added to Embedder, MockEmbedder and T5MetalEmbedder must both add the property. MockEmbedder is trivial; T5MetalEmbedder is a larger file but a nonisolated let addition is straightforward.

3. Test coverage of the stress test (R5). The stub-based 1,000-call test with oversized inputs can be implemented without a real CoreML model (the guard fires before any predictor call). However, the test should verify that normal-sized subsequent calls succeed after an oversized input is rejected or truncated — this requires counting embedder calls after the oversized one.

4. Tokenizer dependency in overflow tests. Tests that produce inputs of exactly maxInputTokens + N tokens need the real tokenizer fixture (xtr-base-en.tokenizer.json). The fixture is present in Tests/Fixtures and is used by existing stress tests; this is not a blocker.

5. Re-entrancy guard interaction. The overflow guard fires inside the re-entrancy guard's inFlight window. If the overflow policy is .reject, the defer block still runs correctly (inFlight → false, waiters resumed). No change to the re-entrancy mechanism is needed.

---

Documentation Impact

No CLI commands, config files, or YAML fields are added. The changes are library API: a new maxInputTokens: Int property, a new overflowPolicy init parameter, and a new error type. These are additive, backward-compatible (the init parameters have defaults), and affect only Swift callers who use T5CoreMLEmbedder directly. Consumers who rely only on the Embedder protocol and do not downcast are unaffected.

The CHANGELOG and README should be updated to document the new parameters and the EmbedderError.inputTooLarge error, but that is part of the PR/Validate stage, not implementation. No ADR update is required for this change (it implements the structural prevention described in ADR 021's context section).

---

Used 34/50 turns, 0k input / 14k output tokens.