[sandlbn]

What this changes

Fixes #20

Adds two skills and their Harbor tasks.

ai-inference-deploy is a top-level router for a vague request like "deploy this model
on my Intel GPU". It reads what kind of GPU is in the machine, picks the runtime, and hands
off to the skill that does the work. Before this, a user had to choose among about 20
*-xpu-* skills on their own, and the skills it hands off to assume a Battlemage card, which
gives wrong answers on Intel laptop GPUs.

openvino-gpu-run serves models with OpenVINO Model Server (OVMS). It's the default for
an endpoint on a laptop GPU, and the only runtime in the catalog that runs natively on Windows.

How routing works

The GPU is queried. scripts/query_gpus.py (read-only, standard library) merges whichever of
these answer and prints JSON:

Source	Gives
torch.xpu.get_device_properties	name, total memory (works on every Intel GPU)
xpu-smi discovery -j	name, memory, PCI address, health state (Arc Pro B cards)
OpenCL via ctypes	whether the GPU shares system memory (integrated GPU), and total GPU memory as the last fallback (Windows, or a host with neither torch nor xpu-smi)

Free RAM, which caps an integrated GPU's budget, comes from GlobalMemoryStatusEx on Windows
and /proc/meminfo on Linux. If memory is still null, the skill asks the user and never
fills in a default.

Request	Discrete GPU	Integrated GPU
GGUF weights	llamacpp-xpu-run	llamacpp-xpu-run
Endpoint (default)	vllm-xpu-run	openvino-gpu-run
In-process	torch-xpu-run	torch-xpu-run
Prefix caching / grammar	sglang-xpu-run	not supported → openvino-gpu-run

On a host with both, the discrete GPU is used and the user is told the integrated one is left
out. Windows goes to openvino-gpu-run; every other path is Linux-only, and the skill says so
rather than adapting the Linux steps.

The router also stops two upstream defaults from applying silently. It always passes the
queried memory and GPU count to model-can-it-fit and xpu-deploy-plan, including for "will
it fit" and benchmark requests, because xpu-deploy-plan otherwise assumes an Arc Pro B70
with 32 GiB. And it doesn't treat a missing xpu-smi result as a failure on integrated GPUs,
since xpu-smi doesn't list them.

Files

• skills/ai-inference-deploy/: SKILL.md, scripts/query_gpus.py,
  references/{runtime-choice,gpu-targets}.md, evals/evals.json (10 cases)
• skills/openvino-gpu-run/: SKILL.md, evals/evals.json (4 cases)
• evaluation/harbor/tasks/ai-inference-deploy-routing/: 6 requests. The gpus.json files
  come from Arc Pro B70 / B50 hosts, a Windows 11 Arc B390 run reported in review (memory
  null), and a Linux Panther Lake laptop; the sixth is a synthetic NVIDIA host that the router
  has to refuse. Provenance is noted in task.toml.
• evaluation/harbor/tasks/openvino-gpu-run-launch/: the agent writes a launch command,
  which the verifier parses (-v or --mount)
• skills.yaml: two entries; skills.sh.json: both added to the GPU group, and the group's
  description now covers integrated GPUs as well as discrete ones

Tested on hardware

Following each skill's documented commands:

GPU (as queried)	Test	Result
Arc Pro B70, 31.89 GiB, discrete	vLLM (podman): query → fit → launch → chat	Pass: reply "Paris"; xpu-smi ps shows vLLM on device 0
Arc Pro B70	OVMS (podman)	Pass: reply "Paris."; server process holds 0.93 GiB of GPU memory
Arc Pro B50, 15.92 GiB, discrete	vLLM (podman), same chain	Pass: reply "Paris"
8 × Arc Pro B70	query	Pass: 8 devices, matches lspci
Panther Lake integrated, Linux, 55.1 GiB available	OVMS (Docker, latest-gpu sha256:a928903f…), OpenVINO/Qwen3-0.6B-int4-ov	Pass: reply "The capital of France is Paris."; server holds 499 MiB drm-resident-gtt (17 MiB before the model loaded)
Panther Lake integrated, SR-IOV virtual GPU passed to a Windows 11 VM, driver 32.0.101.8860	query_gpus.py	Pass: 8.09 GiB GPU memory (OpenCL) and 14.09 GB free RAM. The previous version printed null for both, reproducing @benjaminum's report
Same Windows VM	OVMS 2026.4.1 native ovms.exe, same model	Pass: reply "Paris"; Get-Counter shows ovms holding 0.48 GiB of shared GPU memory

The memory figures are the query's output; xpu-deploy-plan would have used 32 GiB for both
discrete cards. Logs for the B70/B50 runs are attached below.

Checklist

• I checked description against requests a user would really type — see
  CONTRIBUTING.md. ("deploy qwen", "serve a model on my Arc", "run this on my laptop's GPU",
  "vllm or sglang?", a Windows request)
• python3 tools/validate_skills.py passes locally.

Also run locally: run_evals.py --validate (44 cases), lint_task_leakage.py --fail-on-leak 5
(both new tasks score 0), sync_external.py --check, both Harbor verifiers against their
oracle answers (15/15 and 5/5; both fail on a wrong answer), and harbor run --agent oracle
on both tasks (2/2 trials at reward 1.0, 0 errors).

[benjaminum]

I tried the skills on my Windows system and it successfully deployed a model with ovms 🚀

The gpu query script was not able to get information about the memory on my system

• Win11 25H2
• Gfx driver: 32.0.101.8991

[benjaminum]

On Windows there is no information about the available mem

{
  "devices": [
    {
      "index": 0,
      "name": "Intel(R) Arc(TM) B390 GPU",
      "bdf": null,
      "memory_bytes": null,
      "unified_memory": true,
      "state": null,
      "source": [
        "opencl"
      ],
      "class": "igpu"
    }
  ],
  "mem_available_bytes": null
}

[xaleryb]

Two things need to change before this merges; the rest below can follow.

1. Windows memory. On Win11 with no torch and no xpu-smi, query_gpus.py prints memory_bytes: null and mem_available_bytes: null (same as benjaminum's report). from_opencl() never reads CL_DEVICE_GLOBAL_MEM_SIZE (0x101F); the same OpenCL.dll returns 16.24 GiB on this machine. Free memory comes only from /proc/meminfo. Please read 0x101F as the fallback, use GlobalMemoryStatusEx for free RAM on Windows, and have SKILL.md say to ask the user when memory is null. Stage 6 otherwise has no number to pass to model-can-it-fit / xpu-deploy-plan.

2. Hardware table. The last row marks the integrated-GPU OVMS test as passed, while the text under the table says the integrated-GPU path has no hardware result yet. Please relabel the row as untested and attach the results once they exist, along with logs for the B70/B50 runs.

Not blocking:

• openvino-gpu-run: a Windows install path (how to get ovms.exe) and a Windows-native way to confirm the GPU is in use. The fdinfo check is Linux/Docker only.
• A tie-break for hosts that report both an iGPU and a dGPU.
• Routing task: add a supported=false case (or drop scope-boundary from covers), check that skill names exist in the catalog, and replace windows-request/gpus.json, which is byte-identical to the Linux iGPU capture.

[napetrov]

Nice work. The stage tables read well, the "say GPU, not XPU" guidance is a real usability win, every skill name the router refers to exists in skills/, and both verifiers grade the decision rather than the wording.

Inline comments, roughly in order of importance:

1. query_gpus.py: identical cards all get card 0's xpu-smi entry, so a degraded second card is reported as normal (reproduced).
2. query_gpus.py: if xpu-smi prints anything that isn't JSON, the whole script crashes (reproduced with a stub).
3. query_gpus.py: Windows has no memory source (same root cause as @benjaminum's finding); suggested fix inline.
4. Harbor gguf-request contradicts SKILL.md Stage 5, so a correct agent can fail.
5. The integrated-GPU/Windows gpus.json fixtures look hand-written, but the description says they were captured from hardware.
6. The plan verb is handed off before the query runs, so the B70/32 GiB default still applies.
7. openvino-gpu-run: no way to confirm the GPU on Windows; depends on a script from another skill; latest-py + GPU question.

Description nits: one sentence is cut off ("standard library, about merges whichever"), "Upstream issues:" has no links, the Lunar/Panther Lake "Pass" row contradicts "no hardware result yet", and it says 7 evals where evals.json has 9.

[napetrov]

Several cards with the same name all get card 0's xpu-smi data. next(s for s in smi_devs if s["name"] == dev["name"]) always returns the first match, so on a machine with identical cards every entry gets device 0's bdf, state and memory_bytes.

I reproduced this by stubbing in two B70s, with card 1 degraded:

[('0000:03:00.0', 'normal', 1), ('0000:03:00.0', 'normal', 1)]

The router stops on state != normal, so a degraded second card goes unnoticed. The 8 × B70 run would have looked fine, because it only checked the device count.

Suggestion: line devices up by index (torch and xpu-smi both list them in Level Zero order unless ZE_AFFINITY_MASK is set), or match on BDF, and never reuse an xpu-smi entry once it has been matched.

[napetrov]

json.loads has no error handling here. If xpu-smi discovery -j prints anything that isn't JSON (an error message, a warning before the JSON, a version that has no -j), the JSONDecodeError crashes main(), and the torch and OpenCL results are lost too. I confirmed this by stubbing subprocess.run to return Error: no device found.

I haven't checked what xpu-smi actually prints on an integrated-GPU-only host. But that is the host class this script exists for, so it seems worth guarding: wrap both calls in try/except (ValueError, KeyError) and return []. Same for dev["device_id"].

[napetrov]

+1 to @benjaminum's finding. Here is where it comes from: on Windows, torch and xpu-smi are usually missing, from_opencl returns only the unified-memory flag, and this function reads /proc/meminfo, which doesn't exist there. So both memory_bytes and mem_available_bytes come back null.

Two cheap fixes, both standard library:

• In from_opencl, also read CL_DEVICE_GLOBAL_MEM_SIZE (0x101F, ctypes.c_uint64), and use it as the last fallback for memory_bytes. For an iGPU this is the driver's cap on shared memory, which is the right upper bound.
• On win32, get available memory from kernel32.GlobalMemoryStatusEx (ullAvailPhys).

It would also help to add benjaminum's real output as a routing case, so the router has to handle memory_bytes: null rather than only seeing fixtures where it is filled in.

[napetrov]

Minor, and I haven't verified it: I believe xpu-smi's device_type is "GPU" for every device, not "integrated"/"discrete". If so, this always gives None, and the class comes only from OpenCL. Can you confirm from a real discovery -j output? If OpenCL is the only real source, this branch could go.

[napetrov]

The plan verb finishes here, before Stage 2's query and before Stage 6's "pass memory explicitly; never rely on their defaults". So "will Qwen-32B fit on my GPU?" reaches model-can-it-fit / xpu-deploy-plan with no queried memory, and xpu-deploy-plan falls back to "Default target: Arc Pro B70 (32 GiB)". The PR description says preventing that silent default is part of the point.

Suggestion: run query_gpus.py before handing off plan (and bench), or make the plan row continue to Stage 6 for the inputs.

[napetrov]

This check only works on Linux (/proc/<pid>/fdinfo), and the skill offers ovms.exe on Windows. On Windows, "Done when" can never be met. Could you add a Windows check, e.g. Get-Counter '\GPU Process Memory(*)\*' filtered to the ovms PID, or Task Manager's per-process GPU memory column?

[napetrov]

query_gpus.py ships under ai-inference-deploy/scripts/. With npx skills add a user can install openvino-gpu-run on its own, and then this step has nothing to run. Either vendor the script, or state the dependency (and how to manage without it).

[napetrov]

Question: does latest-py include the GPU plugin and compute runtime? It's the only row that doesn't use the -gpu image, and latest-gpu-py doesn't exist on Docker Hub (404). If -py is CPU-only, this path ends up on the CPU, which is exactly what the skill warns about. Then it would need to convert with -py and serve the exported model with -gpu.

[napetrov]

Nit: this only accepts a host:container -v spec, so --mount type=bind,src=$HOME/models,dst=/models, an equally correct launch, fails. Matching dst=/target= as well would avoid rejecting correct work.

[napetrov]

Nit: "Four requests"; there are five cases.

[sandlbn]

@napetrov @benjaminum Thanks for review, will address issues and fix them Today EOD.

[sandlbn]

All comments should be addressed