[Bug] qwen3.5-0.8b-cuda-gpu:3 fails to load: published vision.onnx ships ORT's own Memcpy nodes, so the memcpy pass collides on reload

[emepetres]

Describe the issue

qwen3.5-0.8b-cuda-gpu:3 cannot be loaded. It fails with:

genai_model_instance.cc:59 fl::GenAIModelInstance::GenAIModelInstance failed to load model
qwen3.5-0.8b-cuda-gpu:3: This is an invalid model.
Error: Duplicate definition of name (pad_CUDAExecutionProvider).

qwen3.5-0.8b-generic-cpu:3 runs the same workload fine, and :3 is already the latest
version of the CUDA variant, so there is nothing to upgrade to.

This looks like the same root cause as #1039 (qwen3.5-9b-generic-gpu:3, macOS, failing on
image_grid_thw_CUDAExecutionProvider), so please close this as a duplicate if you would
rather keep it in one place. I am filing separately because it is a different artifact on a
different platform and acceleration backend, and because I have an analysis of the cause
that I could not find recorded anywhere.

The cause is in the published artifact, not in the caller

vision.onnx inside the published download already contains ONNX Runtime's own
Memcpy nodes and their <tensor>_<EPName> outputs. The graph appears to have been
exported after an ORT CUDA placement and memcpy-insertion pass, rather than before it.

Parsing the shipped vision.onnx of qwen3.5-0.8b-cuda-gpu:3:

Memcpy nodes baked into the graph: 55
  Memcpy_token_218  MemcpyFromHost  ['pad']  -> ['pad_CUDAExecutionProvider']
  Memcpy_token_219  MemcpyFromHost  ['prod'] -> ['prod_CUDAExecutionProvider']
  Memcpy_token_257  MemcpyToHost    ['val_1217_CUDAExecutionProvider'] -> ['val_1217']
  ...

So pad_CUDAExecutionProvider is not a duplicated name in the file — the file is a valid
graph, with no duplicate node names, node outputs or initializers. It is defined exactly
once, as the output of Memcpy_token_218. When ORT loads the model it runs its memcpy
transformer again over the same graph, regenerates the name pad_CUDAExecutionProvider for
the same tensor, and collides with the copy that was already baked in.

The contrast with a model that loads makes this concrete. qwen3-vl-2b-instruct-cuda-gpu:2
also ships baked Memcpy nodes (32 of them), but not one for pad:

	qwen3.5-0.8b-cuda-gpu:3 (fails)	qwen3-vl-2b-instruct-cuda-gpu:2 (loads)
baked Memcpy nodes	55	32
producer of pad	node_pad (Pad)	node_pad (Pad)
consumer of pad	Memcpy_token_218 (MemcpyFromHost)	n8 (Size)
pad_CUDAExecutionProvider	present, output of the baked memcpy	absent

That also explains why #1039 sees a _CUDAExecutionProvider name while loading a
generic-gpu (WebGPU) variant: the CUDA-suffixed names are baked into the exported
artifact, so they travel with the model regardless of which EP is later used to run it.

To reproduce the analysis on any affected model (no GPU needed, no external data loaded):

import onnx
g = onnx.load("vision.onnx", load_external_data=False).graph
print([f"{n.name} {n.op_type} {list(n.input)} -> {list(n.output)}"
       for n in g.node if n.op_type.startswith("Memcpy")])

If this reading is right, the fix is in the model publishing pipeline — export the graph
before ORT's placement/memcpy pass — rather than in ORT or in Foundry Local. A defensive
alternative would be for the memcpy transformer to make generated names unique rather than
failing, but that would leave a redundant round trip in the graph.

To reproduce

# pip install foundry-local-sdk==2.0.1
from foundry_local_sdk import ChatSession, Configuration, FoundryLocalManager

manager = FoundryLocalManager(Configuration(app_name="repro"))
manager.download_and_register_eps()

model = manager.catalog.get_model("qwen3.5-0.8b")   # resolves to qwen3.5-0.8b-cuda-gpu:3
print(model.id, model.info.runtime)
model.download()
model.load()                                        # raises here
ChatSession(model)

Ruled out while investigating, in case it saves someone the time:

• Not EP over-registration. Registering only CUDAExecutionProvider
  (download_and_register_eps(names=["CUDAExecutionProvider"])) fails identically.
  Registering nothing fails earlier, with requires CUDAExecutionProvider which is not registered, so the registration call is both necessary and not the trigger.
• Not a corrupt local cache. A clean download into a fresh app_name fails the same way.
• Not an outdated variant. get_model_versions("qwen3.5-0.8b") offers -cuda-gpu:3 and
  :2; get_latest_version returns :3, the one that fails.
• Not model-family-wide. qwen3.5-0.8b-generic-cpu:3 and qwen3-vl-2b-instruct-cuda-gpu:2
  both load and run on the same machine.

Urgency

Not urgent. -generic-cpu:3 is a working fallback on the same machine (3.5 s to load,
2.9 s to describe a 640x360 frame).

Platform and architecture

Windows X64

OS Version

Windows 11 Enterprise 26200

Installation type

Released package/binary

Foundry Local version

foundry-local-sdk 2.0.1 (Python, in-process); CLI 0.8.119

API or surface area

Python SDK

Acceleration backend

WebGPU/CUDA

[hanbitmyths]

Javi Carnero (@emepetres), Foundry Local team published v4 version of Qwen3.5 models eliminating memcpy nodes. Please, download and try again. Let us know if an issue is consistent.