[Bug] Foundry Local: QNN NPU EP shows heavy CPU usage during multi-round tool-calling, GPU EP does not

[aclinick]

Describe the issue

During multi-round tool-calling (agentic loop: call model → get tool_calls → feed tool results back → call model again, repeated up to N rounds), the QNN NPU execution provider shows sustained high CPU utilization (~74-84% across all logical cores) alongside only partial NPU compute utilization (~43-50%, sawtooth). The same model, same prompts, same multi-round loop code path run against the GPU (DirectML) execution provider instead shows the expected "healthy offload" pattern: GPU saturated near 99%, CPU at only ~16-20%.

This suggests a QNN NPU EP-specific issue in how repeated in-session calls are handled (e.g. per-round re-quantization, dispatch overhead, or CPU-side fallback ops that compound across rounds) — not a problem with the model weights or the general orchestration code, since GPU with the literal same code shows no spike, and a single non-looping NPU call also shows no spike.

on the :2 version of qwen2.5-7b this behavior did not happen

Environment

• Foundry Local: 0.10.3 (also reproduced pre/post the 2026-09
  foundry-local-sdk/Microsoft.AI.Foundry.Local.WinML upgrade — this is
  not new/introduced by that upgrade, present both before and after)
• Model: qwen2.5-7b-instruct-qnn-npu:3 vs qwen2.5-7b-instruct-generic-gpu
  (DirectML) / qwen2.5-7b-instruct-openvino-gpu
• SDKs affected: both foundry-local-sdk (Node.js) and
  Microsoft.AI.Foundry.Local.WinML (.NET) — not SDK-specific
• Devices reproduced on:
  • Snapdragon X2 Elite (Surface Laptop for Business 13.8in 8th Ed),
    Windows 11 ARM64, NPU driver 30.0.225.10000 (5/21/2026)
  • A second, untouched Snapdragon X (original) Surface device, using the
    real shipped Zava Dental production app (native .NET/WinUI, same
    Microsoft.AI.Foundry.Local.WinML SDK family)
  • Intel Core Ultra 5 device: GPU path shows the healthy pattern
    (CPU ~8-12%, GPU ~95%) for comparison — Intel does not have a QNN NPU
    to directly compare, included here as supporting cross-platform GPU
    evidence that the "high CPU during multi-round loop" pattern is not a
    generic property of the multi-round tool-calling loop itself.

To reproduce

Load a model's QNN NPU variant (e.g. qwen2.5-7b-instruct-qnn-npu:3)
via the SDK (FoundryLocalManager / catalog / model.load() /
createChatClient()).
2. Run a multi-round tool-calling loop: send a prompt with 3-4 tool
schemas offered, receive a tool call, feed a canned tool result back as
a new message, call the model again — repeat for several rounds
(reproduced with up to 5 rounds).
3. Watch Task Manager's CPU and NPU graphs during the run.
4. Observed: CPU pinned at ~74-84% across all logical cores; NPU
compute utilization only ~43-50%, sawtoothing.
5. Repeat steps 1-4 but load the GPU (DirectML) variant instead of the
NPU variant, same loop, same prompts.
6. Observed: GPU compute saturates near 99%; CPU drops to ~16-20%.
7. Repeat steps 1-2 but with a single, non-looping completeChat()
call (no round loop) on the NPU variant, using the same system prompt
and same 4 tool schemas.
8. Observed: CPU only reaches ~15-37% here (proportional to the NPU's
prefill/generation bumps) — the healthy pattern, same shape as GPU.

Expected behavior

CPU utilization during NPU-backed multi-round tool-calling should look
similar to the GPU-backed case: NPU compute saturated, CPU largely idle
(aside from normal host-side orchestration overhead). The entire purpose
of an NPU execution provider is to offload compute away from the CPU.

Actual behavior

CPU utilization is high (~74-84%, all cores) specifically when: (a) the
QNN NPU EP is used, and (b) the call is part of a multi-round loop
(≥2 sequential completeChat() calls in the same session/context, as
happens naturally in tool-calling). Neither condition alone reproduces
the issue:

• NPU + single-shot call → healthy CPU (~15-37%)
• GPU + multi-round loop → healthy CPU (~16-20%)
• NPU + multi-round loop → high CPU (~74-84%)

Urgency

This negates any of the NPU value since the CPU is pegged at ~90% during the LLM run along with the NPU from 40-90%

Platform and architecture

Windows ARM64

OS Version

WIndows 11 26H1, Windows 11 preview, Windows 11 25H@

Installation type

Released package/binary

Foundry Local version

CLI 0.10.3 and latest SDK

API or surface area

JavaScript SDK

Hardware acceleration/backend

WinML EPs

Backend/runtime version

This applies to all backends and languages (.net and JS) and CLI

[samuel100]

If the previous version (2) of the model did not have the CPU issue then it may be indicative of an erroneous setting in the model configuration. Doing a diff on the configuration files indicated two value changes from version 2 to 3. To validate if this is the root cause:

1. Locate and open the following folder - ~/.{APP_NAME}/cache/models/Microsoft/{MODEL_NAME}/genai_config.json
2. Search for the following parameter (there should be 2 matches): htp_graph_finalization_optimization_mode. Update the value from 0 to 3 for both matches.