SDK not behaving like CLI

[hovrawl]

I have the Foundry Local CLI installed which behaves as expected, nice and easy, handles NPU for my Intel OpenVino boosted device.

The issue I run into is the SDK does not list the same models in the catalog, even when I attempt to use the WebUrl and point it to the local endpoint for my Foundry, which prevents me from using NPU models through the c# SDK in my application.

I am using the Foundry Local Windows SDK for C#

The list of models from foundry cli:

foundry --version                                                                                                                 
0.8.113+70167233c5

foundry model list                                                                                                                
Alias                          Device     Task           File Size    License      Model ID                                         
-----------------------------------------------------------------------------------------------
phi-4                          GPU        chat           8.83 GB      MIT          phi-4-openvino-gpu:1
                               GPU        chat           8.37 GB      MIT          Phi-4-generic-gpu:1
                               CPU        chat           10.16 GB     MIT          Phi-4-generic-cpu:1
----------------------------------------------------------------------------------------------------------
phi-3.5-mini                   GPU        chat           1.95 GB      MIT          Phi-3.5-mini-instruct-openvino-gpu:1
                               GPU        chat           2.16 GB      MIT          Phi-3.5-mini-instruct-generic-gpu:1
                               CPU        chat           2.53 GB      MIT          Phi-3.5-mini-instruct-generic-cpu:1
--------------------------------------------------------------------------------------------------------------------------
phi-3-mini-128k                GPU        chat           2.27 GB      MIT          Phi-3-mini-128k-instruct-openvino-gpu:1
                               GPU        chat           2.13 GB      MIT          Phi-3-mini-128k-instruct-generic-gpu:1
                               CPU        chat           2.54 GB      MIT          Phi-3-mini-128k-instruct-generic-cpu:2
-----------------------------------------------------------------------------------------------------------------------------       
phi-3-mini-4k                  NPU        chat           2.13 GB      MIT          Phi-3-mini-4k-instruct-openvino-npu:1
                               GPU        chat           2.01 GB      MIT          Phi-3-mini-4k-instruct-openvino-gpu:1
                               GPU        chat           2.13 GB      MIT          Phi-3-mini-4k-instruct-generic-gpu:1
                               CPU        chat           2.53 GB      MIT          Phi-3-mini-4k-instruct-generic-cpu:2
---------------------------------------------------------------------------------------------------------------------------
mistral-7b-v0.2                NPU        chat           4.07 GB      apache-2.0   Mistral-7B-Instruct-v0-2-openvino-npu:1
                               GPU        chat           4.27 GB      apache-2.0   Mistral-7B-Instruct-v0-2-openvino-gpu:1
                               GPU        chat           4.07 GB      apache-2.0   mistralai-Mistral-7B-Instruct-v0-2-generic-gpu:1 
                               CPU        chat           4.07 GB      apache-2.0   mistralai-Mistral-7B-Instruct-v0-2-generic-cpu:2 
---------------------------------------------------------------------------------------------------------------------------------------
deepseek-r1-14b                GPU        chat           7.87 GB      MIT          DeepSeek-R1-Distill-Qwen-14B-openvino-gpu:1      
                               GPU        chat           10.27 GB     MIT          deepseek-r1-distill-qwen-14b-generic-gpu:3       
                               CPU        chat           11.51 GB     MIT          deepseek-r1-distill-qwen-14b-generic-cpu:3       
---------------------------------------------------------------------------------------------------------------------------------   
deepseek-r1-7b                 NPU        chat           5.58 GB      MIT          DeepSeek-R1-Distill-Qwen-7B-openvino-npu:1       
                               GPU        chat           4.19 GB      MIT          DeepSeek-R1-Distill-Qwen-7B-openvino-gpu:1       
                               GPU        chat           5.58 GB      MIT          deepseek-r1-distill-qwen-7b-generic-gpu:3        
                               CPU        chat           6.43 GB      MIT          deepseek-r1-distill-qwen-7b-generic-cpu:3        
--------------------------------------------------------------------------------------------------------------------------------    
qwen2.5-coder-0.5b             NPU        chat, tools    0.52 GB      apache-2.0   qwen2.5-coder-0.5b-instruct-openvino-npu:3       
                               GPU        chat, tools    0.36 GB      apache-2.0   qwen2.5-coder-0.5b-instruct-openvino-gpu:2       
                               GPU        chat, tools    0.52 GB      apache-2.0   qwen2.5-coder-0.5b-instruct-generic-gpu:4        
                               CPU        chat, tools    0.80 GB      apache-2.0   qwen2.5-coder-0.5b-instruct-generic-cpu:4        
--------------------------------------------------------------------------------------------------------------------------------    
phi-4-mini-reasoning           NPU        chat           2.78 GB      MIT          Phi-4-mini-reasoning-openvino-npu:2
                               GPU        chat           2.47 GB      MIT          Phi-4-mini-reasoning-openvino-gpu:2
                               GPU        chat           3.15 GB      MIT          Phi-4-mini-reasoning-generic-gpu:3
                               CPU        chat           4.52 GB      MIT          Phi-4-mini-reasoning-generic-cpu:3
-------------------------------------------------------------------------------------------------------------------------
qwen2.5-0.5b                   NPU        chat, tools    0.52 GB      apache-2.0   qwen2.5-0.5b-instruct-openvino-npu:3
                               GPU        chat, tools    0.36 GB      apache-2.0   qwen2.5-0.5b-instruct-openvino-gpu:2
                               GPU        chat, tools    0.68 GB      apache-2.0   qwen2.5-0.5b-instruct-generic-gpu:4
                               CPU        chat, tools    0.80 GB      apache-2.0   qwen2.5-0.5b-instruct-generic-cpu:4
--------------------------------------------------------------------------------------------------------------------------
qwen2.5-1.5b                   NPU        chat, tools    1.51 GB      apache-2.0   qwen2.5-1.5b-instruct-openvino-npu:3
                               GPU        chat, tools    1.00 GB      apache-2.0   qwen2.5-1.5b-instruct-openvino-gpu:2
                               GPU        chat, tools    1.51 GB      apache-2.0   qwen2.5-1.5b-instruct-generic-gpu:4
                               CPU        chat, tools    1.78 GB      apache-2.0   qwen2.5-1.5b-instruct-generic-cpu:4
--------------------------------------------------------------------------------------------------------------------------
qwen2.5-coder-1.5b             NPU        chat, tools    0.52 GB      apache-2.0   qwen2.5-coder-1.5b-instruct-openvino-npu:3       
                               GPU        chat, tools    0.99 GB      apache-2.0   qwen2.5-coder-1.5b-instruct-openvino-gpu:2       
                               GPU        chat, tools    1.25 GB      apache-2.0   qwen2.5-coder-1.5b-instruct-generic-gpu:4        
                               CPU        chat, tools    1.78 GB      apache-2.0   qwen2.5-coder-1.5b-instruct-generic-cpu:4        
--------------------------------------------------------------------------------------------------------------------------------    
phi-4-mini                     NPU        chat, tools    3.60 GB      MIT          phi-4-mini-instruct-openvino-npu:2
                               GPU        chat, tools    2.15 GB      MIT          phi-4-mini-instruct-openvino-gpu:2
                               GPU        chat, tools    3.72 GB      MIT          Phi-4-mini-instruct-generic-gpu:5
                               CPU        chat, tools    4.80 GB      MIT          Phi-4-mini-instruct-generic-cpu:5
------------------------------------------------------------------------------------------------------------------------
qwen2.5-14b                    GPU        chat, tools    4.79 GB      apache-2.0   qwen2.5-14b-instruct-openvino-gpu:2
                               GPU        chat, tools    9.30 GB      apache-2.0   qwen2.5-14b-instruct-generic-gpu:4
                               CPU        chat, tools    11.06 GB     apache-2.0   qwen2.5-14b-instruct-generic-cpu:4
-------------------------------------------------------------------------------------------------------------------------
qwen2.5-coder-14b              GPU        chat, tools    9.08 GB      apache-2.0   qwen2.5-coder-14b-instruct-openvino-gpu:2        
                               GPU        chat, tools    8.79 GB      apache-2.0   qwen2.5-coder-14b-instruct-generic-gpu:4
                               CPU        chat, tools    11.06 GB     apache-2.0   qwen2.5-coder-14b-instruct-generic-cpu:4
-------------------------------------------------------------------------------------------------------------------------------     
qwen2.5-coder-7b               NPU        chat, tools    4.73 GB      apache-2.0   qwen2.5-coder-7b-instruct-openvino-npu:2
                               GPU        chat, tools    4.80 GB      apache-2.0   qwen2.5-coder-7b-instruct-openvino-gpu:2
                               GPU        chat, tools    4.73 GB      apache-2.0   qwen2.5-coder-7b-instruct-generic-gpu:4
                               CPU        chat, tools    6.16 GB      apache-2.0   qwen2.5-coder-7b-instruct-generic-cpu:4
------------------------------------------------------------------------------------------------------------------------------      
qwen2.5-7b                     NPU        chat, tools    5.20 GB      apache-2.0   qwen2.5-7b-instruct-openvino-npu:2
                               GPU        chat, tools    4.79 GB      apache-2.0   qwen2.5-7b-instruct-openvino-gpu:2
                               GPU        chat, tools    5.20 GB      apache-2.0   qwen2.5-7b-instruct-generic-gpu:4
                               CPU        chat, tools    6.16 GB      apache-2.0   qwen2.5-7b-instruct-generic-cpu:4
------------------------------------------------------------------------------------------------------------------------
gpt-oss-20b                    CPU        chat           12.26 GB     MIT          gpt-oss-20b-generic-cpu:1

VS using the demo code and its available models in the catalog:

Available models for your hardware:
  - Alias: deepseek-r1-14b (Id: deepseek-r1-distill-qwen-14b-generic-gpu:3)
  - Alias: deepseek-r1-7b (Id: deepseek-r1-distill-qwen-7b-generic-gpu:3)
  - Alias: gpt-oss-20b (Id: gpt-oss-20b-generic-cpu:1)
  - Alias: mistral-7b-v0.2 (Id: mistralai-Mistral-7B-Instruct-v0-2-generic-gpu:1)
  - Alias: phi-3-mini-128k (Id: Phi-3-mini-128k-instruct-generic-gpu:1)
  - Alias: phi-3-mini-4k (Id: Phi-3-mini-4k-instruct-generic-gpu:1)
  - Alias: phi-3.5-mini (Id: Phi-3.5-mini-instruct-generic-gpu:1)
  - Alias: phi-4 (Id: Phi-4-generic-gpu:1)
  - Alias: phi-4-mini (Id: Phi-4-mini-instruct-generic-gpu:5)
  - Alias: phi-4-mini-reasoning (Id: Phi-4-mini-reasoning-generic-gpu:3)
  - Alias: qwen2.5-0.5b (Id: qwen2.5-0.5b-instruct-generic-gpu:4)
  - Alias: qwen2.5-1.5b (Id: qwen2.5-1.5b-instruct-generic-gpu:4)
  - Alias: qwen2.5-14b (Id: qwen2.5-14b-instruct-generic-gpu:4)
  - Alias: qwen2.5-7b (Id: qwen2.5-7b-instruct-generic-gpu:4)
  - Alias: qwen2.5-coder-0.5b (Id: qwen2.5-coder-0.5b-instruct-generic-gpu:4)
  - Alias: qwen2.5-coder-1.5b (Id: qwen2.5-coder-1.5b-instruct-generic-gpu:4)
  - Alias: qwen2.5-coder-14b (Id: qwen2.5-coder-14b-instruct-generic-gpu:4)
  - Alias: qwen2.5-coder-7b (Id: qwen2.5-coder-7b-instruct-generic-gpu:4)
  - Alias: whisper-base (Id: openai-whisper-base-generic-cpu:1)
  - Alias: whisper-large-v3-turbo (Id: openai-whisper-large-v3-turbo-generic-cpu:2)
  - Alias: whisper-medium (Id: openai-whisper-medium-generic-cpu:1)
  - Alias: whisper-small (Id: openai-whisper-small-generic-cpu:1)
  - Alias: whisper-tiny (Id: openai-whisper-tiny-generic-cpu:2)

Even when passing in the ID of an NPU model I want to use, it does not work.

I am unsure if I am doing something wrong or there is a discrepancy in the API and SDK

[samuel100]

Thanks for raising an issue @hovrawl! The SDK should pick up the OV model just like the CLI. Would you be able to share:

1. the contents of your .csproj file - I just want to double check the settings for WinML.
2. The log file of the app (should be in a folder ~/.foundry_local_samples/logs)

My suspicion is that WinML is not registering the OV EP due to a Windows update blocking the registration. Would you be able to check whether Windows has any updates pending - if not, re-run and see if you still have the same issue?

[lebakken]

I did not report the original problem, but I am also seeing differences between the Foundry Local CLI and the Foundry Local SDK. When I run the CLI on my hardware, it selects the CUDA Execution Provider:

C:\Users\Public>foundry --version
0.8.113+70167233c5

C:\Users\Public>foundry model run qwen2.5-0.5b
Downloading qwen2.5-0.5b-instruct-cuda-gpu:4...
[####################################] 100.00 % [Time remaining: about 0s]        11.5 MB/s/s

Interactive Chat. Enter /? or /help for help.

When I run the "HelloFoundryLocalSdk" sample, it selects the OpenVINO Execution Provider:

info: FoundryLocalCore[0]
      Service configuration complete.
Registering execution providers... ◷    Done.

Downloading model: 100.00%
Downloading model: 100.00%
Loading model qwen2.5-0.5b-instruct-openvino-gpu:2...done.

The CUDA Execution Provider loads and runs much faster on my hardware. How can I make the Foundry Local SDK select the same Execution Provider as the Foundry Local CLI? The HelloFoundryLocalSdk.csproj is unmodified from the published sample, and here's the log file:

2025-12-13 15:57:35.324 -06:00 [INF] Configuring services.
2025-12-13 15:57:35.460 -06:00 [INF] Finished initializing EPDetector
2025-12-13 15:57:35.460 -06:00 [INF] Attempt 1: Autoregistration of certified execution providers in progress.
2025-12-13 15:57:35.460 -06:00 [INF] Started autoregistering certified EPs
2025-12-13 15:57:35.461 -06:00 [INF] Created AzureModelCatalog. URI:https://ai.azure.com/api/eastus/ux/v1.0 Cache directory:C:\Users\craig/.foundry_local_samples/cache/models
2025-12-13 15:57:35.462 -06:00 [INF] Getting latest info from the Azure catalog and for locally cached models.
2025-12-13 15:57:35.462 -06:00 [INF] Getting info for locally cached models.
2025-12-13 15:57:35.462 -06:00 [INF] Found 0 locally cached models.
2025-12-13 15:57:35.462 -06:00 [INF] EPs are not finalized. Ensuring finalization before fetching catalog.
2025-12-13 15:57:35.462 -06:00 [INF] [Telemetry] AppName:foundry_local_samples UserAgent: Command:CoreInitialize Status:Success Direct:True Time:0ms
2025-12-13 15:57:35.494 -06:00 [INF] EnsureEpsDownloadedAsync called - starting EP download
2025-12-13 15:57:35.783 -06:00 [INF] Provider 1/3: NvTensorRTRTXExecutionProvider is in NotReady state. Downloading and ensuring provider is ready.
2025-12-13 15:57:35.869 -06:00 [INF] Provider NvTensorRTRTXExecutionProvider download and ensuring ready attempt: Success
2025-12-13 15:57:35.904 -06:00 [INF] Provider 1/3: NvTensorRTRTXExecutionProvider is now in Ready state. Now registering provider.
2025-12-13 15:57:35.965 -06:00 [INF] Provider 1/3: NvTensorRTRTXExecutionProvider is successfully autoregistered and has ended in Ready state. It is installed at C:\Program Files\WindowsApps\MicrosoftCorporationII.WinML.NVIDIA.TRT-RTX.EP.1.8_1.8.14.0_x64__8wekyb3d8bbwe\ExecutionProvider\onnxruntime_providers_nv_tensorrt_rtx.dll. Download validation status: Certified
2025-12-13 15:57:36.040 -06:00 [INF] Provider 2/3: OpenVINOExecutionProvider is in NotReady state. Downloading and ensuring provider is ready.
2025-12-13 15:57:36.109 -06:00 [INF] Provider OpenVINOExecutionProvider download and ensuring ready attempt: Success
2025-12-13 15:57:36.142 -06:00 [INF] Provider 2/3: OpenVINOExecutionProvider is now in Ready state. Now registering provider.
2025-12-13 15:57:36.478 -06:00 [INF] Provider 2/3: OpenVINOExecutionProvider is successfully autoregistered and has ended in Ready state. It is installed at C:\Program Files\WindowsApps\MicrosoftCorporationII.WinML.Intel.OpenVINO.EP.1.8_1.8.26.0_x64__8wekyb3d8bbwe\ExecutionProvider\onnxruntime_providers_openvino_plugin.dll. Download validation status: Certified
2025-12-13 15:57:36.478 -06:00 [INF] Ensuring CUDA EP is ready.
2025-12-13 15:57:36.478 -06:00 [INF] CUDAExecutionProvider: Checking for CUDA EP MSIX package...
2025-12-13 15:57:36.593 -06:00 [INF] CUDAExecutionProvider: Found installed CUDA EP package at: C:\Program Files\WindowsApps\Microsoft.FoundryLocal.CUDA.EP_1.0.0.0_x64__8wekyb3d8bbwe
2025-12-13 15:57:36.593 -06:00 [INF] CUDAExecutionProvider is already installed.
2025-12-13 15:57:36.593 -06:00 [INF] Registering CUDAExecutionProvider ...
2025-12-13 15:57:36.727 -06:00 [INF] CUDAExecutionProvider is successfully installed and ready for use from: C:\Program Files\WindowsApps\Microsoft.FoundryLocal.CUDA.EP_1.0.0.0_x64__8wekyb3d8bbwe
2025-12-13 15:57:36.727 -06:00 [INF] Finished attempt to autoregister certified EPs; finished in 1266ms
2025-12-13 15:57:36.727 -06:00 [INF] Successfully downloaded and registered the following EPs: NvTensorRTRTXExecutionProvider, OpenVINOExecutionProvider, CUDAExecutionProvider.
Valid EPs: CPUExecutionProvider, WebGpuExecutionProvider, OpenVINOExecutionProvider, CUDAExecutionProvider
2025-12-13 15:57:36.727 -06:00 [INF] [Telemetry] AppName:foundry_local_samples UserAgent: Command:AutoRegisterCertifiedEps Status:Success Direct:False Time:1264ms
2025-12-13 15:57:36.727 -06:00 [INF] Device: CPU,EPs: CPUExecutionProvider
2025-12-13 15:57:36.727 -06:00 [INF] EP download completed successfully. Status: Successfully downloaded and registered the following EPs: NvTensorRTRTXExecutionProvider, OpenVINOExecutionProvider, CUDAExecutionProvider.
Valid EPs: CPUExecutionProvider, WebGpuExecutionProvider, OpenVINOExecutionProvider, CUDAExecutionProvider
2025-12-13 15:57:36.727 -06:00 [INF] Device: GPU,EPs: WebGpuExecutionProvider,WebGpuExecutionProvider,OpenVINOExecutionProvider,OpenVINOExecutionProvider.AUTO,CUDAExecutionProvider
2025-12-13 15:57:36.727 -06:00 [INF] [Telemetry] AppName:foundry_local_samples UserAgent: Command:AutoRegisterCertifiedEps Status:Success Direct:True Time:1232ms
2025-12-13 15:57:36.727 -06:00 [INF] Fetching latest models from Azure Foundry catalog
2025-12-13 15:57:36.727 -06:00 [INF] [Telemetry] AppName:foundry_local_samples UserAgent: Command:AutoRegisterCertifiedEps Status:Success Direct:True Time:1232ms
2025-12-13 15:57:37.321 -06:00 [INF] Model Phi-4-reasoning-generic-cpu does not have a valid prompt template.
2025-12-13 15:57:37.654 -06:00 [INF] Model Phi-4-reasoning-cuda-gpu does not have a valid prompt template.
2025-12-13 15:57:37.654 -06:00 [INF] Model Phi-4-reasoning-generic-gpu does not have a valid prompt template.
2025-12-13 15:57:37.782 -06:00 [INF] Got 80 models from Azure Foundry catalog.
2025-12-13 15:57:37.783 -06:00 [INF] Populated model info for 0 models.
2025-12-13 15:57:37.783 -06:00 [INF] Getting latest info from the Azure catalog and for locally cached models.
2025-12-13 15:57:37.783 -06:00 [INF] Getting info for locally cached models.
2025-12-13 15:57:37.783 -06:00 [INF] Found 0 locally cached models.
2025-12-13 15:57:37.783 -06:00 [INF] Fetching latest models from Azure Foundry catalog
2025-12-13 15:57:37.931 -06:00 [INF] Model Phi-4-reasoning-generic-cpu does not have a valid prompt template.
2025-12-13 15:57:38.320 -06:00 [INF] Model Phi-4-reasoning-cuda-gpu does not have a valid prompt template.
2025-12-13 15:57:38.320 -06:00 [INF] Model Phi-4-reasoning-generic-gpu does not have a valid prompt template.
2025-12-13 15:57:38.440 -06:00 [INF] Got 80 models from Azure Foundry catalog.
2025-12-13 15:57:38.440 -06:00 [INF] Populated model info for 0 models.
2025-12-13 15:57:38.441 -06:00 [INF] [Telemetry] AppName:foundry_local_samples UserAgent: Command:ModelList Status:Success Direct:True Time:1704ms
2025-12-13 15:57:38.508 -06:00 [INF] ExecuteCommandWithCallbackManaged: download_model Data:{"Params":{"Model":"qwen2.5-0.5b-instruct-openvino-gpu:2"}}
2025-12-13 15:57:38.508 -06:00 [INF] Azure Download model Uri:azureml://registries/azureml/models/qwen2.5-0.5b-instruct-openvino-gpu/versions/2 path: revision:null outputDirectory:C:\Users\craig/.foundry_local_samples/cache/models\Microsoft\qwen2.5-0.5b-instruct-openvino-gpu-2
2025-12-13 15:57:39.163 -06:00 [INF] v2/added_tokens.json|C:\Users\craig/.foundry_local_samples/cache/models\Microsoft\qwen2.5-0.5b-instruct-openvino-gpu-2\v2/added_tokens.json
2025-12-13 15:57:39.163 -06:00 [INF] v2/config.json|C:\Users\craig/.foundry_local_samples/cache/models\Microsoft\qwen2.5-0.5b-instruct-openvino-gpu-2\v2/config.json
2025-12-13 15:57:39.163 -06:00 [INF] v2/genai_config.json|C:\Users\craig/.foundry_local_samples/cache/models\Microsoft\qwen2.5-0.5b-instruct-openvino-gpu-2\v2/genai_config.json
2025-12-13 15:57:39.163 -06:00 [INF] v2/generation_config.json|C:\Users\craig/.foundry_local_samples/cache/models\Microsoft\qwen2.5-0.5b-instruct-openvino-gpu-2\v2/generation_config.json
2025-12-13 15:57:39.163 -06:00 [INF] v2/merges.txt|C:\Users\craig/.foundry_local_samples/cache/models\Microsoft\qwen2.5-0.5b-instruct-openvino-gpu-2\v2/merges.txt
2025-12-13 15:57:39.163 -06:00 [INF] v2/openvino_detokenizer/openvino_detokenizer.bin|C:\Users\craig/.foundry_local_samples/cache/models\Microsoft\qwen2.5-0.5b-instruct-openvino-gpu-2\v2/openvino_detokenizer/openvino_detokenizer.bin
2025-12-13 15:57:39.163 -06:00 [INF] v2/openvino_detokenizer/openvino_detokenizer.xml|C:\Users\craig/.foundry_local_samples/cache/models\Microsoft\qwen2.5-0.5b-instruct-openvino-gpu-2\v2/openvino_detokenizer/openvino_detokenizer.xml
2025-12-13 15:57:39.163 -06:00 [INF] v2/openvino_model_dy.bin|C:\Users\craig/.foundry_local_samples/cache/models\Microsoft\qwen2.5-0.5b-instruct-openvino-gpu-2\v2/openvino_model_dy.bin
2025-12-13 15:57:39.163 -06:00 [INF] v2/openvino_model_dy.onnx|C:\Users\craig/.foundry_local_samples/cache/models\Microsoft\qwen2.5-0.5b-instruct-openvino-gpu-2\v2/openvino_model_dy.onnx
2025-12-13 15:57:39.163 -06:00 [INF] v2/openvino_model_dy.xml|C:\Users\craig/.foundry_local_samples/cache/models\Microsoft\qwen2.5-0.5b-instruct-openvino-gpu-2\v2/openvino_model_dy.xml
2025-12-13 15:57:39.163 -06:00 [INF] v2/openvino_tokenizer/openvino_tokenizer.bin|C:\Users\craig/.foundry_local_samples/cache/models\Microsoft\qwen2.5-0.5b-instruct-openvino-gpu-2\v2/openvino_tokenizer/openvino_tokenizer.bin
2025-12-13 15:57:39.163 -06:00 [INF] v2/openvino_tokenizer/openvino_tokenizer.xml|C:\Users\craig/.foundry_local_samples/cache/models\Microsoft\qwen2.5-0.5b-instruct-openvino-gpu-2\v2/openvino_tokenizer/openvino_tokenizer.xml
2025-12-13 15:57:39.163 -06:00 [INF] v2/special_tokens_map.json|C:\Users\craig/.foundry_local_samples/cache/models\Microsoft\qwen2.5-0.5b-instruct-openvino-gpu-2\v2/special_tokens_map.json
2025-12-13 15:57:39.163 -06:00 [INF] v2/tokenizer.json|C:\Users\craig/.foundry_local_samples/cache/models\Microsoft\qwen2.5-0.5b-instruct-openvino-gpu-2\v2/tokenizer.json
2025-12-13 15:57:39.163 -06:00 [INF] v2/tokenizer_config.json|C:\Users\craig/.foundry_local_samples/cache/models\Microsoft\qwen2.5-0.5b-instruct-openvino-gpu-2\v2/tokenizer_config.json
2025-12-13 15:57:39.163 -06:00 [INF] v2/vocab.json|C:\Users\craig/.foundry_local_samples/cache/models\Microsoft\qwen2.5-0.5b-instruct-openvino-gpu-2\v2/vocab.json
2025-12-13 15:57:39.163 -06:00 [INF] Azure Download totalSizeToDownload:384257030
2025-12-13 15:57:39.163 -06:00 [INF] Azure Downloading blob name:v2/added_tokens.json contentLength:629
2025-12-13 15:57:39.164 -06:00 [INF] Azure Downloading blob name:v2/config.json contentLength:801
2025-12-13 15:57:39.165 -06:00 [INF] Azure Downloading blob name:v2/genai_config.json contentLength:1688
2025-12-13 15:57:39.166 -06:00 [INF] Azure Downloading blob name:v2/generation_config.json contentLength:256
2025-12-13 15:57:39.167 -06:00 [INF] Azure Downloading blob name:v2/merges.txt contentLength:1671853
2025-12-13 15:57:39.168 -06:00 [INF] Azure Downloading blob name:v2/openvino_detokenizer/openvino_detokenizer.bin contentLength:2189639
2025-12-13 15:57:39.170 -06:00 [INF] Azure Downloading blob name:v2/openvino_detokenizer/openvino_detokenizer.xml contentLength:9792
2025-12-13 15:57:39.170 -06:00 [INF] Azure Downloading blob name:v2/openvino_model_dy.bin contentLength:358224308
2025-12-13 15:57:39.172 -06:00 [INF] Azure Downloading blob name:v2/openvino_model_dy.onnx contentLength:749
2025-12-13 15:57:39.172 -06:00 [INF] Azure Downloading blob name:v2/openvino_model_dy.xml contentLength:2335820
2025-12-13 15:57:39.173 -06:00 [INF] Azure Downloading blob name:v2/openvino_tokenizer/openvino_tokenizer.bin contentLength:5588641
2025-12-13 15:57:39.174 -06:00 [INF] Azure Downloading blob name:v2/openvino_tokenizer/openvino_tokenizer.xml contentLength:25941
2025-12-13 15:57:39.174 -06:00 [INF] Azure Downloading blob name:v2/special_tokens_map.json contentLength:644
2025-12-13 15:57:39.175 -06:00 [INF] Azure Downloading blob name:v2/tokenizer.json contentLength:11421894
2025-12-13 15:57:39.176 -06:00 [INF] Azure Downloading blob name:v2/tokenizer_config.json contentLength:7542
2025-12-13 15:57:39.176 -06:00 [INF] Azure Downloading blob name:v2/vocab.json contentLength:2776833
2025-12-13 15:57:39.227 -06:00 [INF] Azure Download Completed name:v2/added_tokens.json
2025-12-13 15:57:39.227 -06:00 [INF] Azure Download Completed name:v2/added_tokens.json
2025-12-13 15:57:39.271 -06:00 [INF] Azure Download Completed name:v2/config.json
2025-12-13 15:57:39.271 -06:00 [INF] Azure Download Completed name:v2/config.json
2025-12-13 15:57:39.318 -06:00 [INF] Azure Download Completed name:v2/genai_config.json
2025-12-13 15:57:39.318 -06:00 [INF] Azure Download Completed name:v2/genai_config.json
2025-12-13 15:57:39.329 -06:00 [INF] Azure Download Completed name:v2/generation_config.json
2025-12-13 15:57:39.329 -06:00 [INF] Azure Download Completed name:v2/generation_config.json
2025-12-13 15:57:39.343 -06:00 [INF] Azure Download Completed name:v2/openvino_detokenizer/openvino_detokenizer.xml
2025-12-13 15:57:39.375 -06:00 [INF] Azure Download Completed name:v2/openvino_model_dy.onnx
2025-12-13 15:57:39.417 -06:00 [INF] Azure Download Completed name:v2/tokenizer_config.json
2025-12-13 15:57:39.487 -06:00 [INF] Azure Download Completed name:v2/special_tokens_map.json
2025-12-13 15:57:39.918 -06:00 [INF] Azure Download Completed name:v2/openvino_tokenizer/openvino_tokenizer.xml
2025-12-13 15:57:51.040 -06:00 [INF] Azure Download Completed name:v2/merges.txt
2025-12-13 15:57:51.040 -06:00 [INF] Azure Download Completed name:v2/merges.txt
2025-12-13 15:57:51.839 -06:00 [INF] Azure Download Completed name:v2/vocab.json
2025-12-13 15:57:52.780 -06:00 [INF] Azure Download Completed name:v2/openvino_detokenizer/openvino_detokenizer.bin
2025-12-13 15:57:52.780 -06:00 [INF] Azure Download Completed name:v2/openvino_detokenizer/openvino_detokenizer.bin
2025-12-13 15:57:52.780 -06:00 [INF] Azure Download Completed name:v2/openvino_detokenizer/openvino_detokenizer.xml
2025-12-13 15:57:57.743 -06:00 [INF] Azure Download Completed name:v2/openvino_model_dy.xml
2025-12-13 15:57:58.947 -06:00 [INF] Azure Download Completed name:v2/tokenizer.json
2025-12-13 15:58:01.661 -06:00 [INF] Azure Download Completed name:v2/openvino_tokenizer/openvino_tokenizer.bin
2025-12-13 15:58:11.562 -06:00 [INF] Azure Download Completed name:v2/openvino_model_dy.bin
2025-12-13 15:58:11.562 -06:00 [INF] Azure Download Completed name:v2/openvino_model_dy.bin
2025-12-13 15:58:11.562 -06:00 [INF] Azure Download Completed name:v2/openvino_model_dy.onnx
2025-12-13 15:58:11.562 -06:00 [INF] Azure Download Completed name:v2/openvino_model_dy.xml
2025-12-13 15:58:11.562 -06:00 [INF] Azure Download Completed name:v2/openvino_tokenizer/openvino_tokenizer.bin
2025-12-13 15:58:11.562 -06:00 [INF] Azure Download Completed name:v2/openvino_tokenizer/openvino_tokenizer.xml
2025-12-13 15:58:11.562 -06:00 [INF] Azure Download Completed name:v2/special_tokens_map.json
2025-12-13 15:58:11.562 -06:00 [INF] Azure Download Completed name:v2/tokenizer.json
2025-12-13 15:58:11.562 -06:00 [INF] Azure Download Completed name:v2/tokenizer_config.json
2025-12-13 15:58:11.562 -06:00 [INF] Azure Download Completed name:v2/vocab.json
2025-12-13 15:58:11.563 -06:00 [INF] Created inference_model.json at C:\Users\craig/.foundry_local_samples/cache/models\Microsoft\qwen2.5-0.5b-instruct-openvino-gpu-2\v2\inference_model.json
2025-12-13 15:58:11.563 -06:00 [INF] [Telemetry] AppName:foundry_local_samples UserAgent: Command:ModelDownload Status:Success Direct:True Time:33055ms
2025-12-13 15:58:11.563 -06:00 [INF] [Telemetry] AppName:foundry_local_samples UserAgent: Command:ModelDownload Status:Success Direct:True Time:33055ms
2025-12-13 15:58:38.936 -06:00 [INF] Model loaded successfully: qwen2.5-0.5b-instruct-openvino-gpu:2
2025-12-13 15:58:38.937 -06:00 [INF] [Telemetry] AppName:foundry_local_samples UserAgent: Command:ModelLoad Status:Success Direct:True Time:27364ms
2025-12-13 15:58:38.948 -06:00 [INF] [Telemetry] AppName:foundry_local_samples UserAgent: Command:ListLoadedModels Status:Success Direct:True Time:0ms
2025-12-13 15:58:39.020 -06:00 [INF] ExecuteCommandWithCallbackManaged: chat_completions Data:{"Params":{"OpenAICreateRequest":"{\u0022messages\u0022:[{\u0022role\u0022:\u0022user\u0022,\u0022content\u0022:\u0022Why is the sky blue?\u0022}],\u0022stream\u0022:true,\u0022model\u0022:\u0022qwen2.5-0.5b-instruct-openvino-gpu:2\u0022}"}
2025-12-13 15:58:42.283 -06:00 [INF] TTFT: 1 ms, TTST: 74
2025-12-13 15:58:42.285 -06:00 [INF] [Telemetry] AppName:foundry_local_samples UserAgent: Command:CoreChatCompletion Status:Success Direct:True Time:3264ms
2025-12-13 15:58:42.289 -06:00 [INF] Unloading model:qwen2.5-0.5b-instruct-openvino-gpu:2
2025-12-13 15:58:42.443 -06:00 [INF] [Telemetry] AppName:foundry_local_samples UserAgent: Command:ModelUnload Status:Success Direct:True Time:154ms

[hovrawl]

G'day @samuel100

I pulled up my test program and its a pretty simple .csproj, I recognize there are other properties that were set in the documentation of the MS learn site.

<Project Sdk="Microsoft.NET.Sdk">

    <PropertyGroup>
        <OutputType>Exe</OutputType>
        <TargetFramework>net9.0-windows10.0.26100</TargetFramework>
        <ImplicitUsings>enable</ImplicitUsings>
        <Nullable>enable</Nullable>
        <Configurations>Release;Debug</Configurations>
        <Platforms>x64</Platforms>
    </PropertyGroup>

    <ItemGroup>
        <PackageReference Include="Intel.ML.OnnxRuntime.OpenVino" Version="1.23.0" />
        <PackageReference Include="Microsoft.AI.Foundry.Local" Version="0.8.2.1" />
        <PackageReference Include="Microsoft.Extensions.Logging" Version="9.0.10" />
        <PackageReference Include="OpenAI" Version="2.5.0" />
    </ItemGroup>
</Project>

However, even after changing my csprof to match the example the results are the same:

<Project Sdk="Microsoft.NET.Sdk">

    <PropertyGroup>
        <OutputType>Exe</OutputType>
        <TargetFramework>net9.0-windows10.0.26100</TargetFramework>
        <Nullable>enable</Nullable>
        <Configurations>Release;Debug</Configurations>
        <RootNamespace>app-name</RootNamespace>
        <Platforms>x64</Platforms>
        <ImplicitUsings>enable</ImplicitUsings>
        <WindowsAppSDKSelfContained>false</WindowsAppSDKSelfContained>
        <WindowsPackageType>None</WindowsPackageType>
        <EnableCoreMrtTooling>false</EnableCoreMrtTooling>
    </PropertyGroup>

    <ItemGroup>
        <PackageReference Include="Intel.ML.OnnxRuntime.OpenVino" Version="1.23.0" />
        <PackageReference Include="Microsoft.AI.Foundry.Local" Version="0.8.2.1" />
        <PackageReference Include="Microsoft.Extensions.Logging" Version="9.0.10" />
        <PackageReference Include="OpenAI" Version="2.5.0" />
    </ItemGroup>
</Project>

The results are identical to what I reported earlier.

Further, I do not have logs at ~/.foundry_local_samples/logs but I do have them for the CLI at ~/.foundry/logs which did not generate any logs for today when running the program, only running the CLI would generate logs:

2025-12-15 08:26:06.372 +11:00 [INF] Starting Foundry Local CLI with ''
2025-12-15 08:26:17.200 +11:00 [INF] Starting Foundry Local CLI with 'model list'
2025-12-15 08:26:17.213 +11:00 [INF] Fetching model list from Azure Foundry catalog...
2025-12-15 08:26:17.473 +11:00 [INF] Timeout connecting to service
System.TimeoutException: The operation has timed out.
   at System.IO.Pipes.NamedPipeClientStream.ConnectInternal(Int32, CancellationToken, Int32) + 0xd5
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread, ExecutionContext, ContextCallback, Object) + 0x3d
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread, ExecutionContext, ContextCallback, Object) + 0x70
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task&, Thread) + 0x6a
--- End of stack trace from previous location ---
   at Microsoft.Neutron.Rpc.Client.RpcSessionExtensions.<CreatePipeRpcSessionAsync>d__2`1.MoveNext() + 0x6f
--- End of stack trace from previous location ---
   at Microsoft.AI.Foundry.Local.Common.ServiceManagement.<ConnectClientAsync>d__8.MoveNext() + 0x6f
--- End of stack trace from previous location ---
   at Microsoft.AI.Foundry.Local.Common.ServiceManagement.<CheckIsRunning>d__9.MoveNext() + 0x5f
2025-12-15 08:26:17.474 +11:00 [INF] Starting service <C:\Program Files\WindowsApps\Microsoft.FoundryLocal_0.8.113.28694_x64__8wekyb3d8bbwe\Inference.Service.Agent.exe --urls="http://127.0.0.1:0/" --OpenAIServiceSettings:ModelDirPath="C:\Users\Nathan\.foundry\cache\models" --JsonRpcServer:Run=true --JsonRpcServer:PipeName="inference_agent" --Logging:LogLevel:Default="Information">
2025-12-15 08:26:17.982 +11:00 [INF] Found service endpoints: http://127.0.0.1:58493
2025-12-15 08:26:17.982 +11:00 [INF] Service is started on http://127.0.0.1:58493/, PID 11744!
2025-12-15 08:26:17.983 +11:00 [INF] Command:ModelInit Status:Success Direct:True  Time:769ms
2025-12-15 08:26:17.983 +11:00 [INF] Command:ServiceStart Status:Failure Direct:False  Time:770ms
2025-12-15 08:26:17.983 +11:00 [INF] Checking EP autoregistration status...
2025-12-15 08:26:18.003 +11:00 [INF] Provider 1/1: OpenVINOExecutionProvider is in NotReady state. Downloading and ensuring provider is ready.
2025-12-15 08:26:18.020 +11:00 [INF] UserAgent:foundry-local-CLI/0.8.113+70167233c5 Command:ServiceStatus Status:Success Direct:True Time:0ms
2025-12-15 08:26:18.021 +11:00 [INF] Processing EP autoregistration status...
2025-12-15 08:26:18.021 +11:00 [INF] Command:ServiceStatus Status:Success Direct:False  Time:38ms
2025-12-15 08:26:18.023 +11:00 [INF] UserAgent:foundry-local-CLI/0.8.113+70167233c5 Command:AutoRegisterCertifiedEps Status:Success Direct:True Time:0ms
2025-12-15 08:26:18.023 +11:00 [INF] Reporting progress of running EP download
2025-12-15 08:26:18.039 +11:00 [INF] Provider OpenVINOExecutionProvider download and ensuring ready attempt: Success
2025-12-15 08:26:18.051 +11:00 [INF] Provider 1/1: OpenVINOExecutionProvider is now in Ready state. Now registering provider.
2025-12-15 08:26:18.741 +11:00 [INF] Provider 1/1: OpenVINOExecutionProvider is successfully autoregistered and has ended in Ready state. It is installed at C:\Program Files\WindowsApps\MicrosoftCorporationII.WinML.Intel.OpenVINO.EP.1.8_1.8.26.0_x64__8wekyb3d8bbwe\ExecutionProvider\onnxruntime_providers_openvino_plugin.dll. Download validation status: Certified
2025-12-15 08:26:18.745 +11:00 [INF] Finished attempt to autoregister certified EPs; finished in 811ms
2025-12-15 08:26:18.746 +11:00 [INF] Successfully downloaded and registered the following EPs: OpenVINOExecutionProvider.
Valid EPs: CPUExecutionProvider, WebGpuExecutionProvider, OpenVINOExecutionProvider
2025-12-15 08:26:18.770 +11:00 [INF] Command:ServiceAutoRegister Status:Success Direct:False  Time:749ms
2025-12-15 08:26:18.772 +11:00 [INF] Device: CPU,EPs: CPUExecutionProvider
2025-12-15 08:26:18.772 +11:00 [INF] Device: GPU,EPs: WebGpuExecutionProvider,OpenVINOExecutionProvider,OpenVINOExecutionProvider.AUTO
2025-12-15 08:26:18.773 +11:00 [INF] Device: NPU,EPs: OpenVINOExecutionProvider,OpenVINOExecutionProvider.AUTO
2025-12-15 08:26:20.305 +11:00 [INF] Model Phi-4-reasoning-generic-cpu:1 does not have a valid prompt template.
2025-12-15 08:26:20.828 +11:00 [INF] Model Phi-4-reasoning-generic-gpu:1 does not have a valid prompt template.
2025-12-15 08:26:21.315 +11:00 [INF] Total models fetched across all pages: 68
2025-12-15 08:26:21.316 +11:00 [INF] UserAgent:foundry-local-CLI/0.8.113+70167233c5 Command:ModelList Status:Success Direct:True Time:2544ms
2025-12-15 08:26:21.328 +11:00 [INF] Command:ModelList Status:Success Direct:True  Time:4115ms
2025-12-15 08:26:21.330 +11:00 [INF] Stream disconnected

I spent time last week after posting this cloning the current state of this repo and testing the Foundry Local SDK that was in here, it is not the WinML one, I did not find the source code for that, but when compiling and pointing my program to this locally compiled version, while it seems it is a decently older version of the SDK, it did at least pick up on the NPU models.

Happy to help with any further diagnosis.

[samuel100]

CUDAExecutionProvider

@lebakken thanks for reporting! It looks like you are getting the correct EPs registered. We sort the model variants in Foundry Local SDK using the following logic:

NPU > Vendor-Specific-GPU (e.g. TRTRTX, Vitis, OV) > CUDA > Generic-GPU > Vendor-Specific-CPU (e.g. OV, Vitis) > Generic-CPU

The default variant is the first in the list that is compatible with your available hardware. So, in your case you are seeing OV-GPU as the default because that is ahead of CUDA.

You can see the order, using:

var model = await catalog.GetModelAsync("qwen2.5-0.5b") ?? throw new Exception("Model not found");
foreach (var variant in model.Variants)
{
    Console.WriteLine($"Found variant: {variant.Id}");
}

In the CLI, the order of preference is subtly different;

NPU > Vendor-Specific-GPU (e.g. CUDA, TRTRTX, Vitis, OV) > Generic-GPU > Vendor-Specific-CPU > Generic-CPU

i.e. CUDA is given the same priority as OV-GPU. There is no secondary sort that puts CUDA ahead of OV-GPU - it is just returned from the catalog in that order. Like the SDK, the default variant is the first in the list that is compatible with your hardware.

I actually believe this is a bug in the CLI as it does not distinguish between CUDA from other vendor specific GPUs. The fact it selects CUDA could be due to alphabetical sort. We need to investigate this a bit more.

In the SDK, you can override the default variant using the following:

model.SelectVariant("qwen2.5-0.5b-instruct-cuda-gpu:4");

Interestingly, in the SDK you have both OV-GPU and TRTRTX-GPU, which given the logic are assigned the same priority. But OV-GPU is ordered before TRTRTX-GPU (again, I think this might be an alphabetical sort coming back from the Foundry Catalog).

I'll do some more digging with the team to see how variants with the same priority get ordered.

[samuel100]

@hovrawl -- If you're not using the WinML version then it will not detect the OpenVino EP, which would means it will not allow you to run EP specific models. Therefore you should run the following sample project:

https://github.com/microsoft/Foundry-Local/tree/main/samples/cs/GettingStarted/windows/HelloFoundryLocalSdk

Clone this repo

git clone https://github.com/microsoft/Foundry-Local.git

and then open the following solution:

Foundry-Local/samples/cs/GettingStarted/windows/FoundrySamplesWinML.sln

Run the HelloFoundryLocalSdk project.

You're see in the csproj it is referencing the WinML version:

<ItemGroup>
    <PackageReference Include="Microsoft.AI.Foundry.Local.WinML" />
</ItemGroup>

[hovrawl]

G'day mate apologies for the late response, holiday season and will try get around to the computer with time to test when I can, will keep the thread updated if I screwed up my references and all is good after changing.

[hovrawl]

It is definitely embarrass to realise I was not using the WinML package the entire time, I definitely intended to. After changing my reference it definitely picks up the NPU models much like the CLI tool, it could download and load a model too per the example, though I am having issues with the Chat completions through code; the stream returns immediately without any chunks.

Updated project.cs

<Project Sdk="Microsoft.NET.Sdk">

    <PropertyGroup>
        <OutputType>Exe</OutputType>
        <TargetFramework>net9.0-windows10.0.26100</TargetFramework>
        <Nullable>enable</Nullable>
        <Configurations>Release;Debug</Configurations>
        <RootNamespace>app-name</RootNamespace>
        <Platforms>x64</Platforms>
        <ImplicitUsings>enable</ImplicitUsings>
        <WindowsAppSDKSelfContained>false</WindowsAppSDKSelfContained>
        <WindowsPackageType>None</WindowsPackageType>
        <EnableCoreMrtTooling>false</EnableCoreMrtTooling>
    </PropertyGroup>

    <ItemGroup>
        <PackageReference Include="Intel.ML.OnnxRuntime.OpenVino" Version="1.23.0" />
        <PackageReference Include="Microsoft.AI.Foundry.Local.WinML" Version="0.8.2.1" />
        <PackageReference Include="Microsoft.Extensions.Logging" Version="9.0.10" />
        <PackageReference Include="OpenAI" Version="2.5.0" />
    </ItemGroup>
</Project>

This is how I am getting the chat client after loading a model (qwen2.5-coder-0.5b:npu)

// Get a chat client
var chatClient = await model.GetChatClientAsync();

// Create a chat message
List<ChatMessage> messages = new()
{
    new ChatMessage { Role = "user", Content = "What coding languages are you proficient in?" }
};


var streamingResponse = chatClient.CompleteChatStreamingAsync(messages, ct);
await foreach (var chunk in streamingResponse)
{
    Console.Write(chunk.Choices[0].Message.Content);
    Console.Out.Flush();
}
Console.WriteLine();

I have tried the config with both the url for my local foundry server running and not and with and without the WebService config part:

var config = new Configuration
{
    AppName = "app-name",
    LogLevel = Microsoft.AI.Foundry.Local.LogLevel.Information,
    Web = new Configuration.WebService
    {
        Urls = "http://127.0.0.1:55588"
    }
};

I have loaded the same model in the CLI tool and it loads and runs the chats fine. I am unsure what I am missing or doing wrong.
I appreciate that it has been some time since the previous response & it is the holiday season so we will all be taking breaks.