Process hard-crashes after exactly 2^14 = 16384 output NDArray allocations — floor(16384/num_outputs) inferences (all compute units; not fixed by releasing outputs or reloading the model)

[xocialize]

Summary

Sustained inference exhausts IOSurface-backed NDArray output storage after roughly 8,000–9,000 calls and the process dies with an uncatchable Swift precondition (EXC_BREAKPOINT / SIGTRAP). It reproduces on every compute unit (ANE, GPU, CPU-preferred), on two unrelated models, and releasing the outputs and reloading the AIModel do not reclaim.

This is a hard blocker for any long-running inference process: a service, a batch job, or a video pipeline will die after a few thousand frames regardless of what the caller does.

macOS 27.0 (26A5421a), Mac17,7 (M5 Max, 128 GB), CoreAIRuntime 3600.83.2.14.1, ANEServices 10.19, coreai-core==1.0.0b2, coreai-torch==0.4.1, Python 3.12.13.

Reproduction

import asyncio, numpy as np
from coreai.runtime import AIModel, NDArray, SpecializationOptions, ComputeUnitKind

async def main():
    opts = SpecializationOptions.from_preferred_compute_unit_kind(ComputeUnitKind.neural_engine())
    model = await AIModel.load("model.aimodel", specialization_options=opts)
    fn = model.load_function(next(iter(model.function_names)))
    x = NDArray(np.random.rand(1, 3, 224, 224).astype(np.float16))
    for i in range(1, 20001):
        await fn({"pixel_values": x})          # result discarded immediately
        if i % 1000 == 0:
            print(i, flush=True)

asyncio.run(main())

Dies between 8,000 and 9,000 every time.

Observed

Two assertion sites, depending on the model's output shape:

CoreAIRuntime/NDArray+SharedStorage.swift:108: Fatal error: Failed to allocate storage for
NDArray with requirements: NDArrayDescriptor(scalarType: float16, shape: [1, 20, 56, 56],
alignments: [1, 1, 1, 32, 1], interleave: [1, 1, 1, 1], ordering: [0, 1, 2, 3],
storageKind: ioSurface)

CoreAIRuntime/NDArray+Pool.swift:77: Fatal error: Failed to allocate storage for NDArray
with byteCount: 1572864, sk: ioSurface, st: float16

Crash thread:

0  libswiftCore.dylib  _assertionFailure(_:_:file:line:flags:) + 216
1  CoreAIRuntime       0x… + 494168
…
12 libswift_Concurrency.dylib  completeTaskWithClosure(...)
Exception Type: EXC_BREAKPOINT (SIGTRAP)

What was ruled out

Hypothesis	Result
ANE-specific	No. Reproduces on the GPU lane at the same count.
Caller retaining results	No. Results are discarded (del / never bound); gc.collect() every 100 iterations changes nothing.
Per-AIModel accumulation	No. Reloading the model and re-fetching the function every 2,000 calls still dies at the same point.
Memory pressure	No. VM report at death shows Writable regions Total ≈ 308 MB on a 128 GB machine.
Model-specific	No. Two unrelated models (a 1.2 M-param SR convnet with one [1,3,512,512] output; a small ViT segmentation model with two outputs) both die.
System-wide accumulation	No. Each fresh process gets its own ~8,000, so it is per-process and resets on exit.

That combination points at IOSurface handles rather than bytes — a per-process IOSurface limit being reached because output-backing surfaces are not returned to the pool.

Expected

Either the output storage is recycled so a steady-state inference loop runs indefinitely, or — at minimum — the failure surfaces as a catchable error rather than a Swift precondition. As it stands the process cannot defend itself: there is nothing to catch, and no API to drain or bound the pool.

Impact

Any process doing sustained inference will terminate after a few thousand calls. At the ~1 ms/inference this model achieves, that is under 10 seconds of continuous work. The only workaround we have found is to shard the workload across subprocesses and restart before ~8,000 calls, which is not viable for a latency-sensitive or stateful service.

Possibly related

#11 (runtime clobbers an unrelated live tensor) also involves runtime storage lifetime, though the symptom there is corruption rather than exhaustion.

[xocialize]

Additional data from the crash reports, which sharpen the "reloading does not help" row.

Reloading the AIModel leaks ANEServicesThreads and Neural Engine reserved regions.

A run that reloaded the model every 2,000 calls (5 reloads before dying) produced a crash report with:

Thread 3:: ANEServicesThread
Thread 4:: ANEServicesThread
Thread 5:: ANEServicesThread
Thread 6:: ANEServicesThread
Thread 7:: ANEServicesThread

— five live ANEServicesThreads plus the original, one per reload, none reaped. Its VM summary shows:

Neural Engine (reserved)   35.6M   5    reserved VM address space (unallocated)

against 7296K / 1 region in the single-load runs. So each AIModel.load adds an ANEServices thread and a Neural Engine reserved region that persist for the life of the process.

This means the obvious mitigation — periodically rebuild the model to drain the pool — makes the situation strictly worse: it still hits the same IOSurface ceiling at the same point, and adds a thread and an NE region each time.

All four crash reports carry the same faulting stack (libswiftCore _assertionFailure → CoreAIRuntime ×6 → _coreai_runtime_os → completeTaskWithClosure) and the same EXC_BREAKPOINT, whether the preferred unit was the ANE or the GPU.

[xocialize]

Correction from the reporter: the cap is on output allocations, not inferences

I filed this as "~8000 calls". That number is an artifact of my repro, and the wrong unit
sends you looking for the wrong thing. The limit is a hard, exact:

2^14 = 16384 output NDArray allocations per process.

max_inferences = floor(16384 / num_outputs)

My original repro had two outputs, so it died at ~8192 and I recorded that as an inference
count. There is no leak proportional to call count — the count is constant, and the only reason
it looked model-dependent is that models have different output arities.

Measured

Synthetic models differing only in output count (identical conv, identical shapes):

num_outputs	last inference completed	× num_outputs
1	16383	16383
2	8191	16382
4	4095	16380

Real models, all single-output timm convnets at fp16 224²: resnet18, resnet50,
mobilenetv4_conv_small, efficientnet_b0, convnext_tiny — all die at 16381
(0–3 short of the synthetic figure; presumably a small fixed setup cost).

Two further properties:

• Warmup calls count, and the budget is per-process from start. Varying warmup on
  resnet18 over 0 / 3 / 25 gives last-completed 16381 / 16378 / 16356 —
  warmup + counted = 16381 in every case.
• Compute unit is irrelevant. neural_engine and gpu lanes die at the identical count.

Repro

Run with 1, 2 and 4 to see the divisor. Dies by uncatchable SIGTRAP at floor(16384/N).

import asyncio, sys, torch, torch.nn as nn
from pathlib import Path
from coreai_torch import TorchConverter, get_decomp_table
from coreai.runtime import AIModel, NDArray, SpecializationOptions, ComputeUnitKind

N = int(sys.argv[1]) if len(sys.argv) > 1 else 1
LANE = sys.argv[2] if len(sys.argv) > 2 else "neural_engine"
PREDICTED = 16384 // N

class NOut(nn.Module):
    def __init__(self, n):
        super().__init__()
        self.n, self.c = n, nn.Conv2d(3, 8, 3, padding=1)
    def forward(self, x):
        y = self.c(x)
        return tuple(y * (i + 1) for i in range(self.n))

async def main():
    m = NOut(N).eval().half()
    x = torch.randn(1, 3, 32, 32).half()
    ep = torch.export.export(m, args=(x,)).run_decompositions(get_decomp_table())
    prog = (TorchConverter()
            .add_exported_program(ep, input_names=["x"],
                                  output_names=[f"o{i}" for i in range(N)])
            .to_coreai())
    prog.optimize()
    p = Path(f"/tmp/repro75_n{N}.aimodel"); prog.save_asset(p)

    opts = SpecializationOptions.from_preferred_compute_unit_kind(
        getattr(ComputeUnitKind, LANE)())
    mdl = await AIModel.load(str(p), specialization_options=opts)
    fn = mdl.load_function(next(iter(mdl.function_names)))
    nd = NDArray(x.numpy())

    print(f"outputs={N} lane={LANE} predicted death at {PREDICTED}", flush=True)
    i = 0
    while i < PREDICTED + 5000:
        await fn({"x": nd}); i += 1
        if i % 1000 == 0 or i >= PREDICTED - 5:
            print(f"  completed {i}  (allocations {i * N})", flush=True)
    print("SURVIVED - bug not present")

asyncio.run(main())

Observed: N=1 → dies after 16383. N=2 → dies after 8191. Both exit 133 (SIGTRAP).

Because the precondition is uncatchable, the count has to be flush()ed and fsync()ed
before the call that dies, or you lose it.

A third assertion site, carrying the constant

On the small synthetic models the trap lands on a different CoreAIRuntime site than the two
NDArray+SharedStorage.swift:108 / NDArray+Pool.swift:77 frames in the original report — and
register x22 = 0x4000 (16384) is live at the trap, next to
type metadata for DefaultStringInterpolation:

x7:  0xffffffffb00007ff   x22: 0x0000000000004000
x19: 0x00000001ef06a510   x21: 0x00000001ef063828   -> DefaultStringInterpolation metadata

0   libswiftCore   _assertionFailure(_:_:file:line:flags:) + 216
1   CoreAIRuntime  + 473972
2   CoreAIRuntime  + 543172
3   CoreAIRuntime  + 322924
4   CoreAIRuntime  + 390700
5   CoreAIRuntime  + 222760
6   CoreAIRuntime  + 277712

That reads as a capacity check building an interpolated message around its own constant, which
is the most direct evidence I can offer that 16384 is a fixed table size rather than an emergent
resource limit.

Why the distinction matters downstream

• The budget is computable, therefore schedulable. "~8000, cause unknown" forces a
  conservative guess; 16384 / num_outputs lets a supervisor recycle a worker at a known margin.
  That is the difference between ruling the runtime out for long-lived services and budgeting
  for it.
• Output arity becomes an architecture decision with a runtime price. Splitting a head into
  four outputs quarters process lifetime versus one concatenated output sliced host-side. That is
  a surprising thing for model authors to have to know.

Everything else in the original report still reproduces: not fixed by releasing outputs, not
fixed by reloading the model (which additionally leaks an ANEServicesThread and a Neural Engine
reserved region per reload, per the comment above), not memory pressure, per-process not
system-wide.

I've updated the issue title to match; the original title said "~8000 calls".

Environment: macOS 27.0 (26A5421a), M5 Max 128 GB, CoreAIRuntime 3600.83.2.14.1,
ANEServices 10.19, coreai-core==1.0.0b2, coreai-torch==0.4.1, Python 3.12.13.