debugging comparator cannot execute the source ExportedProgram — TypeError: 'int' object is not callable in _guards_fn (0.4.1 and 0.4.2)

[xocialize]

Summary

coreai_torch.debugging.comparator.create_comparator_for_programs cannot execute the source ExportedProgram, so the op-by-op comparator is unusable. It fails with TypeError: 'int' object is not callable while running the exported module's _guards_fn.

Reproduces on both shipping combinations: coreai-torch 0.4.1 / torch 2.11.0 and coreai-torch 0.4.2 / torch 2.13.0. macOS 27.0 (26A5421a), CoreAIRuntime 3600.83.2.14.1, coreai-core 1.0.0b2.

This matters more than a typical tooling bug because of what it blocks — see Why this is load-bearing below.

Reproduction

import asyncio, torch, torch.nn as nn
from coreai_torch import TorchConverter, get_decomp_table
from coreai_torch.debugging.comparator import create_comparator_for_programs, Comparator

class M(nn.Module):
    def __init__(self):
        super().__init__(); self.c = nn.Conv2d(3, 8, 3, 1, 1); self.r = nn.ReLU()
    def forward(self, x):
        return self.r(self.c(x))

async def main():
    torch.manual_seed(0)
    m = M().eval(); x = torch.rand(1, 3, 16, 16)
    ep = torch.export.export(m, args=(x,)).run_decompositions(dict(get_decomp_table()))
    prog = TorchConverter().add_exported_program(
        ep, input_names=["x"], output_names=["out"]).to_coreai()
    cmp = await create_comparator_for_programs(ep, prog, "main")
    await cmp.compare(lambda *_: Comparator.Status.PASS, {"x": x.numpy()})

asyncio.run(main())

Observed

TypeError: 'int' object is not callable

While executing %_guards_fn : [num_users=0] = call_module[target=_guards_fn](args = (%x,), kwargs = {})
Original traceback:
None

The failure is in executing the source program (torch.fx side), not in anything Core AI computes.

Why this is load-bearing

The comparator is not just a convenience here — for ANE numerics questions it is the only faithful instrument, and I would like to explain why, since it also argues for prioritising this.

The obvious alternative is to instrument the model: re-export with intermediate tensors added as extra outputs, then compare layer by layer. That does not work on this stack. Adding outputs changes fusion, and in my case it changed ANE eligibility outright — a graph that compiled and ran ANE-resident stopped compiling once probe outputs were added:

_ANECompiler : ANECCompile() FAILED
Compiler internal error: It has to be valid custom strides
- From PEFUSED_GOC Layer: … TERNARY_DYNAMIC_GOC: Pre-Scale: 1, ScaleBiasNegate: Y,
  ScaleBiasBroadcast: [ W:192 ]

The ANE lane then silently fell back, and the instrumented run reported ANE numbers identical to the GPU numbers to the last decimal. Truncation and prefix models have the same problem: any re-export is a different artifact.

So to investigate a discrepancy that appears only on one compute unit, one needs to read intermediates out of the same compiled asset — which is exactly what CoreAIInspector / create_comparator_for_programs are for, and create_comparator_for_programs even accepts specialization_options so it can target a lane. With it broken, there is no supported way to localize a lane-specific numerical difference.

Concretely, this is blocking me on a real case: a small ViT segmentation model measures fp32 134 dB, fp16 GPU 66.62 dB, and fp16 ANE 43.58 dB against the same reference, from the identical .aimodel. Same bytes, same weights, 23 dB between two lanes. I cannot localize it.

Minor issues noticed alongside

Not bugs as such, but they cost time and may be worth a docs pass:

• Nothing is exported at the coreai_torch.debugging package level — create_comparator_for_programs must be imported from .comparator, and the docs' create_validator_for_* names are likewise only in .validator.
• create_comparator_for_programs and Comparator.compare are both coroutines; the documentation examples do not show await.
• Comparator.Status has only PASS / FAIL / UNKNOWN. A SKIP would be useful for nodes whose outputs are not comparable (differing shapes, non-tensor results) — currently they have to be reported as PASS or UNKNOWN, neither of which is accurate.