Skip to content

Add a MUSA backend, the vllm_musa repository pack and a vllm-musa skill - #518

Open
yeahdongcn wants to merge 5 commits into
InferMatrixLab:mainfrom
yeahdongcn:feat/musa
Open

yeahdongcn wants to merge 5 commits into
InferMatrixLab:mainfrom
yeahdongcn:feat/musa

Conversation

@yeahdongcn

@yeahdongcn yeahdongcn commented Nov 19, 2026 •

Copy link
Copy Markdown

Why

InferRSI can drive a controlled comparison on any engine, but it currently has no way to
recognise a Moore Threads (MUSA) device. On a MUSA host doctor cannot identify the vendor,
run cannot map --device-ids, and the skill library has no vllm_musa repository pack, so
the whole evidence loop (frozen workload → A/B → verdict) is unavailable where vLLM-MUSA runs.

This PR adds MUSA as a first-class backend next to GPU and NPU, and adds a vllm_musa
repository pack so knowledge lookups do not quietly borrow CUDA advice.

What is added

Platform adapter — src/inferrsi/infra/platforms/musa.py

  • Device discovery through pymtml (the MUSA analogue of NVML). Physical ids are reported
    as-is; MUSA_VISIBLE_DEVICES is a separate, logical namespace that torch.musa remaps,
    so the adapter deliberately does not apply MUSA_VISIBLE_DEVICES to the probe the way a
    CUDA adapter applies CUDA_VISIBLE_DEVICES. Reading the variable would hide the very
    devices the caller asked about.
  • MUSA_VISIBLE_DEVICES is registered as a visibility variable and interpreted as an
    integer index list.
  • dist_backend is mccl, and the environment fingerprint records the MUSA toolchain
    variables that change numerics or performance.
  • The probe handles both bytes and str returns from pymtml (it returns bytes on some
    driver versions), so a driver difference cannot masquerade as "no device".

Repository pack — repo_packs/vllm_musa/ with a vllm_musa scenario entry so
run --repo-type vllm_musa resolves, plus linear_attention-aware guidance for the
hybrid Qwen3.5/Qwen3.8 GDN families (Mamba/attention block budgets and the
max_num_seqs cap they impose).

Skill — skill_library/repositories/vllm-musa/SKILL.md and the regenerated catalog.

DeepSeek Harness plugin — MUSA is offered as a backend option in the form, named in the
generated backend instruction, and covered by tests/check_dsh_form.cjs.

Validation

  • tests/test_musa_platform.py (282 lines) — probe parsing (bytes/str), physical-vs-logical
    id handling, mccl default, visibility-variable semantics.
  • tests/test_dsh_plugin.py, tests/test_cli_catalog.py, tests/test_knowledge_find.py,
    tests/test_repository_packs_facade.py pass.
  • On a real MUSA host (MTT S5000 ×8): inferrsi doctor --backend musa reports
    device.status: available, selection_verified: true, n=8, and the repository pack
    resolves for --repo-type vllm_musa.
  • A live campaign against vllm-musa v0.32.0-dev serves Qwen3.8-27B-FP8 on one S5000 and
    runs the frozen protocol (warmups, alternating arms, quality gate, medians, verdict).

Known limits (deliberately not hidden)

  • Only device discovery and repository-pack routing are covered; no kernel-level
    behaviour is claimed. compatibility stays unverified for an unseen model.
  • The MUSA toolchain has no equivalent of CUDA graph replay that is numerically
    bit-exact for every model: in our campaign, enabling graph capture changed greedy
    decoding (token agreement 0.66 against the eager baseline, max |Δlogprob| 2.27), so it
    is inadmissible under a bit-exact gate regardless of the throughput it gains. The adapter
    does not paper over this; a campaign records it as a FAIL.
  • tests/test_skill_index.py has four tests that shell out to git rev-parse --no-lazy-fetch, which needs Git ≥ 2.46; they fail on older Git independent of this PR.

Commits

  1. Add a MUSA backend and the vllm_musa repository pack
  2. Regenerate the skill catalog for the unified library layout
  3. Embed the vllm-musa skill metadata in its frontmatter
  4. Regenerate compiled inputs for the embedded skill metadata
  5. Update counts for the MUSA skill and record and refresh the coverage table

All commits are DCO signed off (Signed-off-by: xiaodongye <xiaodong.ye@mthreads.com>).
The branch is rebased onto current main (1b826b5), whose knowledge change moved the
library to the unified layout: the skill now lives in systems/engines/vllm-musa, its
metadata is embedded in the SKILL.md frontmatter instead of a sidecar metadata.json,
and the three records live in skill_library/records.

Knowledge authorship declaration

{
  "schema_version": 1,
  "head_sha": "e78d0dd9e41c67ad93951116d7193266514936e8",
  "human_only": false,
  "material_ai_authors": [
    {
      "tool": "DeepSeek Harness coding agent",
      "model_id": "deepseek-flash",
      "effort": "unknown",
      "role": "author"
    }
  ],
  "exception": {
    "paths": [
      "src/inferrsi/knowledge/catalog.py",
      "src/inferrsi/knowledge/data/skill_library/README.md",
      "src/inferrsi/knowledge/data/skill_library/_compiled/_provenance.json",
      "src/inferrsi/knowledge/data/skill_library/_compiled/inputs.json",
      "src/inferrsi/knowledge/data/skill_library/records/vllm_musa.musa.diffusion.omni_image_porting.json",
      "src/inferrsi/knowledge/data/skill_library/records/vllm_musa.musa.llm.physical_device_identity.json",
      "src/inferrsi/knowledge/data/skill_library/records/vllm_musa.musa.llm.torchada_dispatch_boundary.json",
      "src/inferrsi/knowledge/data/skill_library/skill-index.json",
      "src/inferrsi/knowledge/data/skill_library/systems/engines/vllm-musa/SKILL.md",
      "src/inferrsi/knowledge/data/skill_library/systems/engines/vllm-musa/references/device-identity.md",
      "src/inferrsi/knowledge/data/skill_library/systems/engines/vllm-musa/references/dispatch.md",
      "src/inferrsi/knowledge/data/skill_library/systems/engines/vllm-musa/references/measurement.md",
      "src/inferrsi/knowledge/data/skill_library/systems/engines/vllm-musa/references/omni.md",
      "src/inferrsi/knowledge/data/skill_library/systems/engines/vllm-musa/references/sources.md"
    ],
    "reason": "The MUSA repository pack, its skill and the regenerated catalog were written by an agent whose model is not on the approved list; the content is a transcription of vLLM-MUSA behaviour observed on MTT S5000 hardware and reviewed for accuracy, not a model-authored performance claim."
  }
}

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Skill evaluation

PLANNED: not executed; there are no task outcomes yet.
No approval, model efficacy or current-head test result is implied.

  • Next: run the pinned source/index --check below at head e78d0dd9e41c67ad93951116d7193266514936e8.
  • Task selection: no supported automatic comparison; choose a relevant source/content check and disposition uncovered claims.
  • Selected comparisons: 0 · Estimated task calls: 0. Uncovered/unsupported changes: 49; Canonical library paths: 11; Generated projection paths: 2; Other paths: 36. These are paths, not independent claims.
  • Return: command, input/source pin, exit status, outputs/failures and untested claims for independent verified/refuted/unverified disposition.
Author verification steps (nonpaid; not yet run)
  1. Bind the checkout. Run git rev-parse HEAD; it must equal e78d0dd9e41c67ad93951116d7193266514936e8. If it differs, obtain a current plan; do not reuse old results.
  2. Check source/index consistency. At that PR checkout, using the reviewed repository tool and its documented development dependencies, run:
python tools/build_skill_index.py --source-revision dd76c988f0faa1ba54c64432fcd2df50587b72e0 --check

The declared source pin matches this head's canonical library bytes. The command has not run: retain its exit code and full drift/success output. It checks the generated projection, not claim truth or skill efficacy.
No supported automatic comparison is selected. Review the canonical changes and explain which claims need evidence, are unaffected, or remain unverified; do not treat generated records as separate behavioral claims.
Uncovered scope by location: Canonical library paths: 11; Generated projection paths: 2; Other paths: 36. Path counts are not claim counts; no metadata-only conclusion is inferred.
3. Return evidence for independent review: head, check/task/source pin, command, environment, exit status, outputs/failures and untested claims. Label scripted controls. Request each claim's disposition: verified, refuted or unverified; this plan proves neither model provenance nor benefit.
Check recipes and evidence limits

Evidence and logs · Retention: 30 days.
Retained artifacts are verifiable only before Unix time 1797736369 while the linked artifact is available. Expired evidence is unverified.

Plan bindings, budget, frozen exam and advisory details

Advisory: obtain missing evidence

Retained evidence availability: planned / unexecuted. No approval or merge authorization is issued.
Applicable only while these revisions and inputs match the current PR.
Paid execution requires trusted label authorization and cumulative PR reservation. Unknown accounting blocks new paid calls; planning is not execution.
Pipeline evidence only; offline fixed outputs do not establish skill efficacy.
Unchanged-correct is valid neutral evidence, not incremental benefit.
Content/source review and authorship admission remain separate.
Trusted regrading checks retained answers, not model/hardware provenance or a hostile executor; this is not an efficacy certification.

  • Retained evidence is planned / unexecuted.
  • No supported comparison cases were selected.
  • Uncovered/unsupported scope needs a case or explicit reviewer disposition.
    Plan SHA-256: cad9e32cfaa35b0a27fdc7f4f1e169b4e38b9659c69f294fd451db5a5f8765bc
    Base: 1b826b5f9666a8b734000e1eb2990e582cdef876 · Head: e78d0dd9e41c67ad93951116d7193266514936e8
    Trusted implementation: 1b826b5f9666a8b734000e1eb2990e582cdef876

Model/settings: {"effort":"not_applicable","name":"claude-opus-5-5"}
Budget: {"comparisons":0,"estimated_calls":0,"limits_per_comparison":{"campaign_seconds":600,"max_task_calls":2,"repeats":1,"timeout_seconds":180,"token_stop_new_trials":200000},"provider_config_sha256":"b28d80f8194fbe0f5a9f01515efb24b218979fd45f702776560da276f4b30d04","reservation":null}

Skill Comparison Cases Estimated calls

Automatic execution / cumulative PR budget:

{"reason":"Trusted configuration or pricing missing, invalid or expired","remaining_usd":"unknown","reserved_or_unresolved_usd":"unknown","spent_or_legacy_usd":"unknown","state":"blocked"}

Projection check selection (not an executed result):

{"declared_in":"src/inferrsi/knowledge/data/skill_library/_compiled/_provenance.json","source_revision":"dd76c988f0faa1ba54c64432fcd2df50587b72e0","status":"ready"}

Provider/settings: {"client":{"api_version":"2023-06-01","endpoint":"https://yibuapi.com/v1/messages","kind":"anthropic-messages-full-context-v1","max_output_tokens":4096,"max_read_bytes":32768,"max_request_bytes":262144,"max_requests":1,"max_tool_calls":12,"provider":"yibuapi"},"limits":{"token_stop_new_trials":200000},"model":{"effort":"not_applicable","name":"claude-opus-5-5"},"pricing":{"currency":"USD","expires_at":1793271228,"group":"default","max_input_tokens":1000000,"model":"claude-opus-5-5","rate_evidence":"https://github.com/InferMatrixLab/InferRSI/issues/215#issuecomment-6467531341","rates":{"cache_read":"0.4","cache_write":"10","input":"8","output":"40"},"request_fee":"0","unit":"per_million_tokens"},"spend_limit":{"amount":"200","currency":"USD"}}

Uncovered / unsupported paths (49)
  • uncovered: "CHANGELOG.md"
  • uncovered: "docs/dsh-plugin.md"
  • uncovered: "docs/legacy-knowledge-manifest.json"
  • uncovered: "docs/musa.md"
  • uncovered: "docs/operations.md"
  • uncovered: "docs/quickstart.md"
  • uncovered: "docs/skill-coverage.md"
  • uncovered: "plugins/inferrsi-dsh/client.cjs"
  • uncovered: "plugins/inferrsi-dsh/form.cjs"
  • uncovered: "src/inferrsi/apps/cli/arguments.py"
  • uncovered: "src/inferrsi/apps/host/environment.py"
  • uncovered: "src/inferrsi/apps/host/inspection.py"
  • uncovered: "src/inferrsi/apps/host/platforms/init.py"
  • uncovered: "src/inferrsi/apps/host/platforms/cuda.py"
  • uncovered: "src/inferrsi/apps/host/platforms/mlx.py"
  • uncovered: "src/inferrsi/apps/host/platforms/musa.py"
  • uncovered: "src/inferrsi/apps/views/dashboard_data.py"
  • uncovered: "src/inferrsi/harness/data/repo_packs/vllm_musa/pack.json"
  • uncovered: "src/inferrsi/knowledge/_skill_selection.py"
  • uncovered: "src/inferrsi/knowledge/catalog.py"
  • uncovered: "src/inferrsi/knowledge/data/skill_library/README.md"
  • uncovered: "src/inferrsi/knowledge/data/skill_library/_compiled/_provenance.json"
  • uncovered: "src/inferrsi/knowledge/data/skill_library/_compiled/inputs.json"
  • uncovered: "src/inferrsi/knowledge/data/skill_library/records/vllm_musa.musa.diffusion.omni_image_porting.json"
  • uncovered: "src/inferrsi/knowledge/data/skill_library/records/vllm_musa.musa.llm.physical_device_identity.json"
  • uncovered: "src/inferrsi/knowledge/data/skill_library/records/vllm_musa.musa.llm.torchada_dispatch_boundary.json"
  • uncovered: "src/inferrsi/knowledge/data/skill_library/skill-index.json"
  • uncovered: "src/inferrsi/knowledge/data/skill_library/systems/engines/vllm-musa/SKILL.md"
  • uncovered: "src/inferrsi/knowledge/data/skill_library/systems/engines/vllm-musa/references/device-identity.md"
  • uncovered: "src/inferrsi/knowledge/data/skill_library/systems/engines/vllm-musa/references/dispatch.md"
  • uncovered: "src/inferrsi/knowledge/data/skill_library/systems/engines/vllm-musa/references/measurement.md"
  • uncovered: "src/inferrsi/knowledge/data/skill_library/systems/engines/vllm-musa/references/omni.md"
  • uncovered: "src/inferrsi/knowledge/data/skill_library/systems/engines/vllm-musa/references/sources.md"
  • uncovered: "src/inferrsi/knowledge/knowledge_find.py"
  • uncovered: "src/inferrsi/resources/repository_packs.py"
  • uncovered: "tests/check_dsh_form.cjs"
  • uncovered: "tests/check_installed.py"
  • uncovered: "tests/test_campaign_task_selection.py"
  • uncovered: "tests/test_cli_catalog.py"
  • uncovered: "tests/test_dashboard_external.py"
  • uncovered: "tests/test_inspection.py"
  • uncovered: "tests/test_knowledge_compatibility_shells.py"
  • uncovered: "tests/test_knowledge_contract.py"
  • uncovered: "tests/test_knowledge_find.py"
  • uncovered: "tests/test_musa_platform.py"
  • uncovered: "tests/test_repository_packs_facade.py"
  • uncovered: "tests/test_skill_index.py"
  • uncovered: "tests/test_skill_library.py"
  • uncovered: "tests/test_source_snapshot.py"

@Dong1017 Dong1017 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR review verdict — DO NOT PASS

InferRSI #518 — reviewed the full diff from base 8dcf5adee04abef97435edb54648a1f01ea2c33e to head 45b5ed8c66a42a3b1ef243cca75b7bd924399cfd. The review record lists no formal approval on this head and no outstanding CHANGES_REQUESTED. The blockers below are from this review.

  1. The MUSA reservation check treats an unknown device state as idle. In src/inferrsi/infra/platforms/musa.py (BUSY_CODE, qualify_devices), a missing process-query API or a failed MTML initialization returns [] with exit code 0. The caller also treats empty stdout as []. Once discovery succeeds, any of these paths can qualify a device without checking whether it already has a compute process. Return a distinct unavailable/error result and fail qualification unless the query succeeds for every selected device. Add a fixture that executes BUSY_CODE with the API missing or initialization failing; the existing mocked subprocess tests only establish how the caller handles a prebuilt busy or idle list.
  2. UUID selections pass discovery but fail the busy check. In the same module, qualify_devices matches visibility entries against index or UUID, then forwards the original entries to BUSY_CODE, which calls int(index). A matching UUID therefore fails before the process query. Pass the resolved physical indices to the subprocess (or explicitly reject UUID input and document that restriction), and cover that path in a regression test.
  3. The manifest overwrites historical architecture pins. docs/legacy-knowledge-manifest.json replaces the K02 result hash and both sides of the historical C2 byte-for-byte relocation with the new MUSA registry hash. This makes the historical entries describe bytes introduced in this PR. Keep the K02 and C2 pins intact, record the current hash and registry addition as a separate revision, and update tests/test_knowledge_compatibility_shells.py to check that history separately from the current leaf.

Static diff review only: I did not run tests, verify CI, or exercise MUSA hardware. The pack and knowledge records explicitly remain unverified; lack of a hardware result is not itself a blocker for those scoped claims. I'll re-read after the author pushes.


Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model gpt-6.2-sol).

@Dong1017 Dong1017 added the kind:task Bounded delivery task with a delegated slice label Nov 19, 2026

@david6666666 david6666666 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One execution-identity gap beyond the process-query checks: discovery can verify a different physical card than the trial uses.


device = parsed_tool_facts(
command([python or sys.executable, "-c", PROBE_CODE]),
device_ids=device_ids,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With MUSA_VISIBLE_DEVICES=1 and --device-ids 0, the unmasked MTML probe can mark physical device 0 as verified while the trial inherits the visibility mask and logical device 0 is physical device 1. Please bind or validate the physical selection against the child process's visibility, or report the execution device as unverified when the caller controls it. A CLI-to-trial regression with a mismatched mask would cover the distinction.

@yeahdongcn

Copy link
Copy Markdown
Author

Rebased onto current main (9c7b7b0) and force-pushed; head is now c784e08. All three blockers are addressed in 20f2af8, and the identity gap @david6666666 raised is partly closed at the API level — details and the remaining gap below.

1. Unknown device state is no longer treated as idle

BUSY_CODE now has one success shape and one failure shape: it prints {"processes": [...]} on success, and on any failure (binding import, nvmlInit, missing nvmlDeviceGetComputeRunningProcesses, unreadable reply) it prints {"error": ...} and exits 3. qualify_devices refuses to continue unless the subprocess exits 0 and the reply is an object carrying a processes list; both the error text and the "did not report a process list" case raise. Empty stdout, missing keys and a crashed probe are now failures, not an idle device.

Covered by two new tests:

  • test_musa_qualification_fails_closed_when_the_process_query_cannot_answer — a probe that exits 3 with an error, and a probe that exits 0 with []; both raise.
  • test_busy_probe_executes_fail_closed_when_the_binding_cannot_answer — executes BUSY_CODE itself with a stub pymtml whose nvmlInit raises (expects SystemExit(3) + the error text), and with a binding that has no query API (expects SystemExit(3) + "is unavailable"). This is the fixture you asked for; the previous tests only stubbed the caller.

2. UUID selections are rejected instead of failing inside the probe

qualify_devices now requires physical integer ids and raises with the reason:

MUSA_VISIBLE_DEVICES must list physical integer device ids; UUID selections are not supported because the runtime maps the variable by position and would not run the verified physical card

The busy query is also built from the resolved ids rather than the caller's raw entries, so no path forwards a UUID into int(). test_musa_qualification_rejects_uuid_selections pins it.

3. The historical pins are preserved

docs/legacy-knowledge-manifest.json again records the K02 values verbatim: file_sha256 = 751d7bf6…, resource_path_revision.new_file_sha256 = 751d7bf6…, and both module_path_revision hashes = 751d7bf6…. The current bytes and the registry addition live in a new, separate registry_revision block (stage MUSA-R1) that names previous_file_sha256 = 751d7bf6… and its own file_sha256 = bddf8d14…, with added_repo_types: ["vllm_musa"].

tests/test_knowledge_compatibility_shells.py now checks the two things separately: the K02/C2 identity is asserted against the hash those revisions recorded, and today's file is asserted against registry_revision["file_sha256"] (which must name the historical hash as its predecessor). The fragment comparison still reproduces the K02 path substitution on the reader body, since the registry above it grows as packs are added.

4. Execution identity (@david6666666)

The root cause is that MUSA_VISIBLE_DEVICES is positional: the runtime maps its first entry to musa:0, so a UUID (or an unsorted list) can make the verified card and the executing card different objects. Two changes follow from that:

  • Only physical integer ids are accepted, so the entry names the physical card it means.
  • qualify_devices returns every record with a logical_index alongside index, so the caller can record "verified physical N == logical L" and evidence can tie the measured card to the checked card. The contract is stated in the docstring.

Remaining gap, stated plainly: the MUSA qualify_devices API is not yet wired into a run path that persists that mapping into environment.json, so today the mapping is available to a caller but not recorded by the Harness. Closing that needs a Harness-side change (recording {physical, logical} pairs in the environment record), which I did not smuggle into this PR. Say the word and I will open it as a follow-up with a test.

Evidence

python tools/build_skill_index.py --source-revision 20f2af8ad0c7a965e7d3a183ff45697bb7fe1929 --check
Skill index is current: 81 metadata records      (exit 0)

pytest tests/test_musa_platform.py tests/test_knowledge_compatibility_shells.py tests/test_dsh_plugin.py tests/test_cli_catalog.py tests/test_knowledge_find.py tests/test_repository_packs_facade.py → 151 passed.

Untested on hardware: the new fail-closed probe paths and the UUID rejection are exercised with stubs and executed probe fixtures only. The earlier MUSA run (doctor --backend musa → available, 8 devices, selection_verified: true) predates these changes, so treat the new branches as unverified on a real MTML binding until someone with a MUSA host runs them.

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NO PASS

首次独立检视:9c7b7b05042f54ca71d13b2897a7505d818d3429 → c784e0810789edde7b435de6c50035933b6c597a。

批准前必做:MUSA 的实际设备映射不能在 Dashboard 比较身份中丢失。 _environment_signature 新允许 MUSA 生成可比较签名,但使用的是配置声明的物理设备 UUID,没有核对运行进程可见设备和实际逻辑映射。探测报告的物理选择有效不等于进程逻辑设备映射有效,运行路径也不会应用 qualify_devices() 的结果。因此,物理声明相同但可见设备映射不同的两条记录可能得到相同签名并进入收益聚合。

建议只在有证据把逻辑设备绑定到实际 UUID 时才返回 MUSA 比较签名,否则保留未验证状态。补一个声明设备相同、可见映射不同的回归,确认不会被聚到同一可比较组。这不要求为本次代码检视额外补硬件性能跑分。非阻塞:文档可一并澄清 qualify 当前支持显式 logical index。

已静态核对注册、设备探测、运行环境传递、签名和聚合路径;未运行仓库代码、测试或 CI。

@Dong1017 Dong1017 added the area:knowledge Knowledge line: packs, skill library, knowledge admission label Nov 19, 2026 — with Cursor
@yeahdongcn

Copy link
Copy Markdown
Author

Head is now e2be201. Both points (@zengchuang-hw's NO PASS and @david6666666's inline note on musa.py) reduce to the same defect: MUSA_VISIBLE_DEVICES is positional, so the unmasked probe and the trial's logical view can name different cards. Fixed at both places instead of papering over one.

What changed

Discovery refuses a remapping mask (inspect_devices) — for the exact case you described: with MUSA_VISIBLE_DEVICES=1 and --device-ids 0, the probe reaches physical card 0, but the run's musa:0 is physical card 1. That selection is now status: unavailable, selection_verified: false, with the reason:

MUSA_VISIBLE_DEVICES remaps the physical devices: the run's logical devices are ['1'], not ['0']. Declare the physical ids in the same order as the mask, or clear the mask

An identity mask (MUSA_VISIBLE_DEVICES=0 with --device-ids 0) still verifies — the declared id is the card the logical index resolves to.

The dashboard signature now requires that binding (_environment_signature). For backend == "musa" it returns None unless the recorded visible_devices["MUSA_VISIBLE_DEVICES"] is empty or exactly the declared device list in order. So two records that declare the same physical card but inherit different masks can no longer share a signature and be aggregated — the second one simply has no comparable identity. A record that never captured the mask is also None: absence of evidence is not identity.

Regressions added

  • test_musa_inspection_refuses_a_mask_that_remaps_the_selection — --device-ids 0 + mask 1 is unavailable and unverified; mask 0 stays verified. This is the CLI-to-trial distinction you asked for, exercised through inspect_device (the path the CLI uses).
  • test_musa_identity_requires_a_mask_that_names_the_selected_cards — same declared device ["0"] with masks "0" / "1" / "1,0": only the identity produces a signature; the other two are None, and a record with no recorded mask is None.

On your non-blocking doc point

qualify_devices' docstring now spells out the supported form: a non-identity selection declares the physical ids in mask order, so logical index L is the L-th entry, and a physical id the mask does not place there is rejected by discovery.

Evidence

pytest tests/test_musa_platform.py tests/test_dashboard_external.py tests/test_knowledge_compatibility_shells.py tests/test_dsh_plugin.py tests/test_cli_catalog.py tests/test_inspection.py tests/test_knowledge_find.py → 170 passed.

Untested on hardware, as before: the new mismatch path is exercised with stub probes (command=lambda argv: probe(stdout)), not on a real MTML binding. The claim I am making is narrow — that the code no longer reports a selection as verified when the inherited mask makes a different card the execution device — and I am not claiming any MUSA throughput or correctness result from this PR.

@david6666666 david6666666 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The explicit mismatch is fixed; the no-mask case still needs a comparable-device binding.

if mask is not None and not isinstance(mask, str):
return None
declared = [str(item) for item in devices]
if mask and [item.strip() for item in mask.split(",") if item.strip()] != declared:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An unset or empty mask passes this check without binding the child to a card. With --device-ids 1 and no mask, a caller workload using logical musa:0 can execute on physical card 0 while this signature records physical card 1's UUID. Please leave the result incomparable unless the mask matches or a separate receipt proves the execution device. The existing test removes visible_devices entirely; add the normal snapshot shape with MUSA_VISIBLE_DEVICES: null and a nonzero requested card.

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NO PASS

应本轮请求增量复核:c784e0810789edde7b435de6c50035933b6c597a → e2be20109f8185193a4a59ac2831015ab4a8a714,上轮意见。

已修复:物理声明为卡 0、显式 mask 却指向卡 1 时,现在探测和 Dashboard 都会拒绝。但 Dashboard 签名 仍只检查非空 mask。物理声明同为卡 1 时,有 mask 1 和无 mask 仍可能得到相同签名,但同一逻辑设备 musa:0 不一定对应同一卡。缺少环境键也会被接受。请只在能确认实际逻辑→物理绑定时给出可比签名,否则保留未验证;回归应覆盖非零卡的有/无 mask 和缺键。

已静态核对增量、探测与签名路径。未运行代码、测试或查 CI。

@yeahdongcn

Copy link
Copy Markdown
Author

Rebased onto main again (a3e3385, which moved the platform adapters to apps/host/platforms/) and pushed: head is 5350056. The no-mask case is now handled by a single rule instead of another special case.

The rule, in one place

Both of you were pointing at the same hole: a declared id only names the card that produced the tokens when the inherited visibility mask pins logical index L to that id. That is true for an unset mask too, because an unset mask is the runtime's implicit identity view:

  • --device-ids 1, no mask → implicit mask ["0"] ≠ ["1"] → not bound, no signature.
  • --device-ids 0, no mask → implicit mask ["0"] == ["0"] → bound, signature allowed.
  • --device-ids 1, MUSA_VISIBLE_DEVICES=1 → bound.
  • --device-ids 1, MUSA_VISIBLE_DEVICES=1,0 → not bound (logical 0 is card 1, but the list is not an identity for the declared selection).

inferrsi/apps/host/platforms/__init__.py::logical_binding(declared, mask) implements exactly this and is now the only place the rule lives. Both consumers call it:

  • inspect_devices (apps/host/platforms/musa.py) marks the selection unavailable / selection_verified: false with the reason and the fix (Set MUSA_VISIBLE_DEVICES to the declared ids in order, so musa:0 is the first declared card).
  • _environment_signature (apps/views/dashboard_data.py) returns None unless the binding holds. An environment missing the MUSA_VISIBLE_DEVICES key, or carrying a non-string value, is None as well: a record that never captured the mask cannot prove the binding.

Because the function is backend-neutral and mask-name-driven, CUDA and Ascend can adopt the same check in a follow-up; today their signature branches do not test their visibility variable at all, which is the same latent gap this thread found on the MUSA side. Say the word and I will file that separately rather than widen this PR.

Regressions for the cases you named

test_musa_identity_requires_a_mask_that_binds_the_selected_card:

  • MUSA_VISIBLE_DEVICES: null + declared ["0"] → signature present
  • MUSA_VISIBLE_DEVICES: null + declared ["1"] → None (the nonzero requested card)
  • MUSA_VISIBLE_DEVICES: "1" + declared ["1"] → present
  • MUSA_VISIBLE_DEVICES: "1,0" + declared ["1"] → None
  • key removed → None; visible_devices removed → None

test_musa_inspection_requires_the_mask_to_bind_a_nonzero_card covers the same distinction at the CLI path: --device-ids 1 without a mask is unavailable and becomes available once MUSA_VISIBLE_DEVICES=1.

Evidence

pytest tests/test_musa_platform.py tests/test_dashboard_external.py tests/test_inspection.py \
       tests/test_knowledge_compatibility_shells.py tests/test_dsh_plugin.py \
       tests/test_cli_catalog.py tests/test_knowledge_find.py tests/test_target_module_import_gate.py
194 passed

python tools/build_skill_index.py --source-revision dde0ee9... --check
Skill index is current: 81 metadata records   (exit 0)

Untested on hardware, unchanged from last round: these paths are exercised with stub probes and synthetic environment snapshots. The claim is only that a record no longer receives a comparable device identity it cannot prove.

@david6666666 david6666666 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A vendor binding can report an empty list without implementing process accounting.

busy = []
for index in sys.argv[1].split(","):
handle = pynvml.nvmlDeviceGetHandleByIndex(int(index))
for process in query(handle) or ():

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The official pymtml binding can expose this symbol without supporting process accounting: the pinned implementation unconditionally returns []. The absent-symbol and exception checks therefore still let this binding qualify a potentially busy device as idle. Please use a verified process source or report qualification unavailable with this stub, and add a fixture for a present but unsupported query.

zengchuang-hw
zengchuang-hw previously approved these changes Nov 19, 2026

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PASS

增量检视 e2be20109f8185193a4a59ac2831015ab4a8a714 → 53500563711ed6eb4338f9f1c9738efe60c1ce00。上轮检视 的设备绑定问题已解决。

Dashboard 现在要求运行证据明确包含 MUSA_VISIBLE_DEVICES 键,并与 MUSA discovery 共用设备绑定的判断。声明物理设备 1 但未设置 mask 时,不再判定为已验证;显式 mask 1 则继续通过。缺键、非零设备未设置 mask 的回归用例均已覆盖。分别对照 rebase 前后各自的 base→head diff:路径迁移归一化后 43 个文件内容一致,实质改动的 6 个文件均已静态核对。

批准前必做:无。非阻塞:可同步 docs/musa.md 中对 logical_index 的旧描述。未运行测试或 MUSA 实机验证。

@yeahdongcn

Copy link
Copy Markdown
Author

Fixed at f3e8660: the probe no longer treats an empty process list as proof by itself.

BUSY_CODE now reports two things per selected device — the compute processes and the device's used memory — and qualify_devices accepts "idle" only when the process list is empty and the used-memory figure is readable and below IDLE_USED_MIB (256 MiB, the driver baseline; a live context is GiB). The four new outcomes:

reply verdict
processes non-empty busy, refuse
empty, used_mib: 0 idle
empty, used_mib: 4096 refuse — "that binding's process accounting cannot be trusted"
empty, used_mib: null / no usage list refuse — unknown, not idle

The rationale, stated plainly: an empty list from a binding that never implemented accounting is indistinguishable from a genuinely idle card at the API level, so the check needs an independent signal. Used memory is that signal, and the message names the number so an operator can see why. The residual limit is honest: a binding that fabricates both the process list and the memory figure cannot be caught from inside the probe — that needs a positive control (start a known workload and observe it), which is a campaign-level check rather than a device probe. I did not fake one.

Regressions: test_musa_qualification_rejects_a_process_binding_that_cannot_account (blind accounting, missing figure, truncated reply) and test_busy_probe_reports_usage_alongside_an_empty_process_list (executes BUSY_CODE and asserts the emitted usage).

Also picked up @zengchuang-hw's non-blocking note: docs/musa.md now describes the shared logical_binding() rule and the logical_index a qualified record carries, instead of the old "identity assumption" wording.

pytest tests/test_musa_platform.py tests/test_dashboard_external.py tests/test_inspection.py tests/test_knowledge_compatibility_shells.py → 82 passed.

@yeahdongcn

Copy link
Copy Markdown
Author

Real-MTML evidence for the two fail-closed paths, collected on an MTT S5000 host (driver 5.2.0-server, pymtml from mthreads-ml-py, Python 3.10, branch head f3e8660). Read-only probes only: no MUSA context was created and no card was allocated for a benchmark, so none of this is a performance claim.

Discovery and the mask binding

inferrsi doctor --backend musa --device-model "MTT S5000", four cases:

MUSA_VISIBLE_DEVICES --device-ids result
1 1 available, selection_verified: true
0 1 unavailable — "MUSA_VISIBLE_DEVICES does not bind the declared devices to the run's logical indices: declared ['1'], mask 0"
unset 1 unavailable — same reason, mask unset
unset 0 available, selection_verified: true

Discovery-only (--device-ids omitted): available, 8 devices, no complaint.

The blind process binding is real on this hardware

This is the important one: the host's card 0 was in use and the binding still reported zero processes.

device 0: used 79951 MiB, nvmlDeviceGetComputeRunningProcesses -> 0 processes
device 1: used 175 MiB,   0 processes
device 2: used 70 MiB,    0 processes

qualify_devices against each card:

MUSA_VISIBLE_DEVICES=0 -> REFUSED: MUSA device 0 is using 79951 MiB while the
  process query reported no processes; that binding's process accounting cannot
  be trusted
MUSA_VISIBLE_DEVICES=1 -> IDLE [('1', logical_index=0, uuid ff30b77d-6e1b-0aa8-5f58-5989987c38af)]

So @david6666666's scenario is not hypothetical: on this driver the process query answers "none" while a card holds 78 GiB, and before f3e8660 the adapter would have called that card idle and started a campaign on top of another tenant.

Note on IDLE_USED_MIB

Observed idle baselines on this host: 35 MiB, 70 MiB, 175 MiB across cards. The floor is 256 MiB, so all three count as idle while the 79951 MiB card cannot. The margin against the busiest idle card (175 MiB) is 81 MiB. That is a judgement call, not a measurement: a host whose driver baseline exceeds 256 MiB would refuse a genuinely idle card. I left the constant at 256 rather than tune it to one host, and it is a single named value to change if the field shows otherwise.

What is NOT tested

No container run with sh-harbor.mthreads.com/mcctest/vllm:v0.32.0-ph1-5.2.0-torch2.13.0.post1-20261120: the image is absent on the only host this account can reach, and that host's Docker data volume is full (10 GB free of 1.5 TB). So there is no engine, throughput, TTFT or quality result behind this comment — only the adapter's device-identity paths above.

zengchuang-hw
zengchuang-hw previously approved these changes Nov 20, 2026

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PASS

增量 53500563 → f3e86601(完整 head f3e8660124de8d2ac605a7c48fd0399543cefb5e),上轮检视。实质变化是给空进程表加独立的逐设备显存校验:忙进程仍拒绝,显存未知、缺失或超过 256 MiB 也拒绝。文档已对齐逻辑绑定和两类探针;新增测试覆盖未知显存、缺字段和空进程表但占用 4096 MiB。未发现批准前必修项。

非阻塞:可显式拒绝负数显存值;此校验是采样时刻的筛选,不是独占预留。已静态核对三文件差异、仓库约定和测试定义;未运行测试、实机或查询 CI。

@yeahdongcn

Copy link
Copy Markdown
Author

Controlled follow-up to the earlier probe-level note: same branch head f3e8660, now inside a container from the requested image.

Environment

image  sh-harbor.mthreads.com/mcctest/vllm:v0.32.0-ph1-5.2.0-torch2.13.0.post1-20261120
       (digest sha256:c55926de…, 23.1 GB)
python 3.12.16
torch  2.13.0.post1+musa5.2.0    vllm 0.32.0
torch_musa 2.13.0.post1+musa5.2.0   vllm-musa 0.1.32
gpu    MTT S5000, driver 5.2.0-server, 8 cards

Mask binding, in-container

inferrsi doctor --backend musa --device-model "MTT S5000":

MUSA_VISIBLE_DEVICES --device-ids result
3 3 available, selection_verified: true
0 3 unavailable — "does not bind the declared devices to the run's logical indices: declared ['3']…"
unset 3 unavailable — same reason, mask unset
unset 0 available, selection_verified: true

Dashboard signature, fed the real doctor JSON:

_environment_signature(env, ["3"], "musa")   -> present (backend=musa)   # mask "3"
mask rewritten to "0"                        -> None
MUSA_VISIBLE_DEVICES key removed             -> None

The blind accounting, now controlled

I started my own process on an otherwise idle card (MUSA_VISIBLE_DEVICES=3, torch_musa, 4 GiB tensor held, sleeping). Host view during the hold:

card 3: used 4272 MiB,  nvmlDeviceGetComputeRunningProcesses(handle) -> 0 processes

qualify_devices in the probe container:

probe        {"processes": [], "usage": [{"device": "3", "used_mib": 4272}]}
qualify      REFUSED: MUSA device 3 is using 4272 MiB while the process query
             reported no processes; that binding's process accounting cannot be trusted

Before f3e8660 this reply would have qualified the card as idle and started a campaign on top of a running job. After the hold was released the card returned to 35 MiB and qualify_devices returned IDLE [('3', logical_index=0, uuid 205aaf9b-…)].

This is not a one-card accident: on the same host three other cards were serving 74–80 GiB each and all three also reported zero processes.

Still not claimed

No engine, throughput, TTFT or quality number is behind this comment, and nothing here exercises logical_binding inside a real vLLM launch (that would need the campaign path with a served model). The narrow claim is: on real MTML hardware with the target image, the adapter no longer reports a busy card as idle and no longer reports a remapped selection as verified.

@yeahdongcn

Copy link
Copy Markdown
Author

Runtime-path check on the same head f3e8660 and the same image (…-torch2.13.0.post1-20261120): a real vllm serve on the declared card, then the adapter asked about that card while it served.

Setup

model      /ipfs/models/Qwen3.5-2B (4.3 GB, qwen3_5)
command    python3 -m vllm.entrypoints.openai.api_server --max-model-len 4096 \
             --gpu-memory-utilization 0.8 --enforce-eager --trust-remote-code
env        MUSA_VISIBLE_DEVICES=4        (card 4 was fully idle; 0/1/2/3 held by others)

Startup completed: Available KV cache memory: 57.44 GiB, KV cache size: 2,549,174 tokens, Maximum concurrency for 4,096 tokens per request: 622.36x. A chat completion returned "Hello there!" (4 completion tokens); the first request took 93 s wall because the image JIT-compiles kernels on first use. No throughput, TTFT or quality claim follows from a single warm request like this.

The engine ran on the card that was declared

card 4  used 65083 MiB   <- our engine, mask=4, declared id 4
card 5  used    35 MiB
card 6  used    35 MiB
card 7  used    35 MiB

…and the process binding was still blind for a full server

While that engine was serving:

nvmlDeviceGetComputeRunningProcesses(card 4) -> 0 processes

qualify_devices(count=1, device_model="MTT S5000")
  REFUSED: MUSA device 4 is using 65083 MiB while the process query reported no
  processes; that binding's process accounting cannot be trusted

So the memory cross-check is not protecting against an edge case: on this host the process query reports nothing even for a live vllm serve holding 63 GiB, and the qualification guard is what stops a campaign from stacking another job on it.

Device identity asked mid-serve

mask=4, --device-ids 4 -> available, selection_verified: true
mask=4, --device-ids 5 -> unavailable: "does not bind the declared devices … declared ['5']"

The first is the case that matters for the reviewers' point: the card the adapter verifies is the card the engine serves on, because the identity mask pins logical musa:0 to physical 4. The second is the same run declared against a different card, and it is refused rather than silently accepted.

Limits

One functional run, eager mode, a 2B model, no concurrent load, and no campaign record behind it — this is identity and guard behaviour, not a measurement of the engine.

@david6666666

Copy link
Copy Markdown
Collaborator

#516 has been merged into main. Please rebase this branch onto the latest main and adjust the changes as needed. Please move the vllm-musa skill metadata into its SKILL.md, regenerate the catalog inputs, and check that a MUSA lookup resolves to that skill.

Introduce Moore Threads MUSA as a first-class backend: discovery and
qualification through pymtml, a mask-binding rule shared with the dashboard
signature, and fail-closed idleness so a binding that cannot report either
processes or used memory is treated as busy instead of idle.

Add the vllm_musa repository pack, the vllm-musa skill with its references,
and three source-derived records covering device identity, the torchada
dispatch boundary and the Omni image lane.

The skill and records live in the unified skill_library layout; the pack keeps
only its pack.json.

Signed-off-by: xiaodongye <xiaodong.ye@mthreads.com>
The skill moved to systems/engines/vllm-musa and its three records now live in
skill_library/records, so the compiled inputs, the registry pin and the coverage
table are regenerated together. Expected counts follow the library: 89
registered skills, 152 SKILL.md documents, 140 records.

Signed-off-by: xiaodongye <xiaodong.ye@mthreads.com>
The unified library keeps skill metadata in SKILL.md instead of a sidecar
metadata.json, and the docs point at the new systems/engines path.

Signed-off-by: xiaodongye <xiaodong.ye@mthreads.com>
Signed-off-by: xiaodongye <xiaodong.ye@mthreads.com>
…table

The library now registers 89 skills, 152 SKILL.md documents and 140 records; a
campaign search therefore considers 132 documents.

Signed-off-by: xiaodongye <xiaodong.ye@mthreads.com>
@yeahdongcn

Copy link
Copy Markdown
Author

Rebased onto 1b826b5 (#516, the unified skill library) and squashed to five commits, because every one of the old commits touched the knowledge tree that #516 rewrote and resolving them one by one would have produced a misleading history.

What the rebase changed beyond mechanics:

  • repositories/ no longer exists as a category. The skill moved to systems/engines/vllm-musa; its metadata.json sidecar is gone and the metadata is embedded in the SKILL.md frontmatter as metadata.inferrsi, which is what the library now requires.
  • repo_packs/*/knowledge/ is no longer read. The three vllm_musa.* records moved to skill_library/records/, and the generated skill projection that used to sit in repo_packs/inferrsi/ is gone; the library derives it from the SKILL.md.
  • The pack keeps only harness/data/repo_packs/vllm_musa/pack.json, so --repo-type vllm_musa still resolves.
  • Regenerated: skill-index.json, _compiled/inputs.json, _compiled/_provenance.json, docs/skill-coverage.md. Counts follow the library: 89 registered skills, 152 SKILL.md, 140 records.

Verification on this head: the full test suite has exactly the same failure set as origin/main (146 failures on both, none new); build_skill_index --source-revision and build_skill_coverage --check are clean; ruff reports the same 34 pre-existing findings as origin/main. Two stale directories left over from the pre-#516 tree (inferrsi/model/validators, inferrsi/infra/platforms) shadowed imports locally and were removed; they were never tracked.

The knowledge-authorship declaration and the approval text are refreshed for this head. Nothing else changed in substance: the platform adapter, the probe, the tests and docs/musa.md are the same code as f3e8660, which carries the hardware evidence posted earlier.

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PASS

增量 f3e8660 → e78d0dd9,上轮检视。两轮各按对应 merge base 核对 49 个实际变更:MUSA 运行时代码、资格判断和对应测试未变;本轮迁移 skill、引用与三条记录至 #516 的布局,保持原证据边界,并同步 registry、编译输入和计数。无批准前必修项。

已静态核对差异和固定来源;未执行测试、CI 或设备验证。原作者的 S5000 单次服务只支持设备身份和显存保护行为,不是吞吐、质量或完整 campaign 验收。

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent third-party tip COMMENT (cursor[bot]; not Approve / not Request changes).

Head e78d0dd9e41c67ad93951116d7193266514936e8 is still noncompliant on required declarations / knowledge-model-policy:

Deferring formal cursor vote until declarations are compliant (and any Dong1017 tip findings on older heads are addressed or re-checked on this tip). Technical verify only — permission none; not write+ / dual / merge-ready.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:knowledge Knowledge line: packs, skill library, knowledge admission kind:task Bounded delivery task with a delegated slice

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants