fix[notask]: decouple mobile shard validation from desktop integration tests

[iancris]

🎯 Problem

packages/vla-ggml/scripts/generate-mobile-integration-tests.js wrote integration.auto.cjs and then asserted that every runner was assigned to a Device Farm shard on every platform. That script is chained into npm run test:integration:

test:integration → test:integration:generate → test:mobile:generate → generate-mobile-integration-tests.js

So a mobile scheduling assertion sat directly in the desktop test path, and anything it threw took desktop integration tests down on all seven platforms.

That is what happened when #4006 removed the pi05 group to stop scheduling deferred pi05 mobile tests. pi05.test.js stayed in test/integration/, so runPi05Test became "unassigned" and bare aborted:

Generated .../test/mobile/integration.auto.cjs with 4 integration runners.
Uncaught Error: [ios] Tests not assigned to any group in test-groups.json:
  runPi05Test
##[error]Process completed with exit code 134.

Two things make this worse than a one-off:

• Generation had already succeeded. The file was written correctly with all four runners; the abort was purely the policy check. The coupling bought nothing.
• QVAC-23984 fix: stop scheduling deferred pi05 mobile tests #4006's own CI could not catch it. It touched only test-groups.json and a workflow comment — no native changes — so integration-tests was skipped on its PR. The only job that would have failed was the one the gate skipped. main has been red for every VLA PR that actually runs integration tests since.

📝 How

Split generation from validation, and make the deliberate omission explicit rather than invisible.

The generator only generates. validateGroups is removed. Nothing in generate-mobile-integration-tests.js can throw on scheduling state any more.

The rules move to scripts/lib/validate-test-groups.js — pure, dependency-free, no fs and no process.exit, so the same rules run from the CLI and from unit tests. scripts/validate-mobile-tests.js calls it; that script already existed but was wired into nothing.

The lib also owns generatedRunnerNames(content), the async function run<Name> extractor. It takes file contents rather than a path so the module stays fs-free, and it lives beside the rules that consume its output so validate-mobile-tests.js and the unit suite share one implementation. A test asserting on its own copy of that pattern would prove nothing about the extractor that actually runs.

test-groups.json gains a top-level deferred key:

{
  "ios":      { "smolvla": [...], "groot": [...] },
  "android":  { "smolvla": [...], "groot": [...] },
  "deferred": ["runPi05Test"]
}

This preserves #4006's outcome — pi05 is still not scheduled — while keeping "deliberately not run" distinguishable from "forgotten". Deleting the runner from the file, as before, made those two states identical.

deferred must stay top-level. The CI composites consume only .<platform> and ignore sibling keys — upload-to-devicefarm/action.yml says so explicitly ("Other top-level keys (e.g. OCR's perf_report_filter) are ignored"), and OCR already relies on it. Nested under ios/android it would be scheduled as a real shard, so that is reserved and reported, not merely documented: to the coverage rules a nested deferred is just another array of runner names, so the file would otherwise validate clean while upload-to-devicefarm billed a shard literally named "deferred".

deferred also accepts a { "<platform>": [runner] } map, for a runner scheduled on one platform and deferred on another. A flat array cannot express that — the scheduled-and-deferred rule fires on the platform that does run it. VLA uses the flat form; the map form is what the llm/ocr ports need, where the platform sets differ. This is why platformNames excludes deferred by name rather than by shape: an object-form deferred is a non-array object, so shape-based inference would read it as a platform demanding full coverage.

Platform detection can be pinned. validateTestGroups(groups, runners, { platforms }) overrides shape inference. Inference is right for VLA, whose only top-level maps are platforms, and it means a newly added platform key is covered automatically — but llm-llamacpp ships top-level iosWeekly/androidWeekly maps that are schedules, not platforms, so a caller there pins the list instead. A pinned platform missing from the file is reported rather than silently passing.

The check moves to a job that actually runs. test:unit now ends with test:mobile:groups && test:mobile:validate, and test:unit runs in ts-checks, which needs only authorize + fork-approval. A PR like #4006 that touches nothing native now gets the check.

New unit suite at scripts/__tests__/mobile-test-groups.test.js (16 tests), matching the mobile-test-groups.test.js pattern asr-ggml and tts-ggml already have and this package lacked. It covers the real committed data plus unassigned runners, group typos, stale deferred entries (flat and per-platform), scheduled-and-deferred contradictions, a deferred nested inside a platform, per-platform deferral, a deferred map keyed by a non-platform, pinned-vs-inferred platform lists, and non-platform metadata keys.

File	Change
scripts/generate-mobile-integration-tests.js	−38: validateGroups removed; generation only
scripts/lib/validate-test-groups.js	new: the rules + the shared runner extractor, pure and testable
scripts/validate-mobile-tests.js	+group coverage; dead mtime heuristic removed
scripts/__tests__/mobile-test-groups.test.js	new: 16 tests
test/mobile/test-groups.json	deferred: ["runPi05Test"]
test/mobile/README.md	documents the key, why it is top-level, and how to run a deferred runner on device
package.json	wires both into test:unit

One incidental removal: validate-mobile-tests.js compared test-file mtimes against integration.auto.cjs. buildFileContents derives that file from the sorted filenames under test/integration/ and never opens a test file, so editing a test's body cannot make it stale — the check could only produce false positives, and chaining test:mobile:validate into test:unit would have promoted each one into a hard failure telling the author to regenerate a byte-identical file. It is removed rather than CI-gated.

The content-based reference diff that replaces it covers the staleness that matters day to day: a test file added, renamed or removed. It does not cover a change to the generator's own template (the __shouldRunTest guard, the header comments), since it compares only the runIntegrationModule paths — that still needs a manual npm run test:mobile:generate. Nothing regressed there: the mtime check compared test-file timestamps, not the generator's, so it never caught that either. Closing the gap properly means calling buildFileContents from the validator, which is not possible while the generator requires bare-fs.

🧪 Tested

CI runs:
https://github.com/tetherto/qvac/actions/runs/32758510083
https://github.com/tetherto/qvac/actions/runs/32758735695
https://github.com/tetherto/qvac/actions/runs/32756119505

Verified against real bare v1.31.0 and node --test.

The A/B that matters — the generator run against the exact test-groups.json from main (no deferred key, pi05.test.js present), the same data that produced exit 134 in CI:

Command	On main	On this branch
bare ./scripts/generate-mobile-integration-tests.js	exit 134 (SIGABRT)	exit 0 — "Generated … with 4 integration runners."
node scripts/validate-mobile-tests.js	check did not exist here	exit 1, names runPi05Test on both platforms
node --test scripts/__tests__/mobile-test-groups.test.js	did not exist	2 of 16 fail, correctly

The same mistake is still caught — twice — but in a two-minute ungated job instead of by aborting seven platforms' integration suites.

With the fix in place:

• bare ./scripts/generate-mobile-integration-tests.js → exit 0.
• node scripts/validate-mobile-tests.js → ✅ 4 runners, group coverage OK.
• node --test across both script suites → 24/24 pass, including the pre-existing generate-prestage-block tests (no regression).
• prettier --check with prettier-config-holepunch → clean. node --check on every changed JS file → clean. Both changed JSON files parse.

Two claims verified rather than assumed:

• deferred is inert to CI. Ran the composites' own jq expressions against the new file. upload-to-devicefarm still resolves [{smolvla, runAddonTest|runEsmNamedExportsTest}, {groot, runGrootTest}]; validate-devices still returns 3 runner names and spec-count 2. Identical to before — runPi05Test never reaches Device Farm.
• integration.auto.cjs regenerates byte-identical. git status shows it unmodified after running the generator, confirming only the assertion was removed, not a byte of generated output.

Also checked: no *.test.js under test/integration/ is gitignored, so the validator cannot fail on a clean checkout; and generate-prestage-block.js derives its runner list from MODEL_SHARDS, not from test-groups.json, so it is unaffected.

⚠️ Breaking changes

None. No addon, runtime or native code is touched — only scripts, their tests, the shard map and its README. Device Farm scheduling is byte-identical, and the published package is unaffected (scripts/ is not in package.json files).

Two follow-ups worth noting, neither introduced by this PR:

• llm-llamacpp and ocr-ggml carry the same landmine. Both chain test:mobile:generate into test:integration:generate and have a throwing validateGroups, and both ship a test-groups.json. Editing either the way QVAC-23984 fix: stop scheduling deferred pi05 mobile tests #4006 edited VLA's takes down their desktop integration too. They are left out because release-<package>-<x.y.z> is per package, so a three-addon PR cannot be back-ported or reverted per release line — and the three fixes are not mechanical duplicates (llm has iosWeekly/androidWeekly family logic plus a hardcoded isOverrideOnly list, ocr has perf_report_filter). Each needs its own PR; the lib here is already shaped for both (pinned platform list, per-platform deferred). Tickets still to be filed — noted in review, not yet done.
• No CHANGELOG entry, matching QVAC-23984 fix: stop scheduling deferred pi05 mobile tests #4006, which shipped none. sanity-checks does not require one for a scripts-only fix.

🔍 Review round 2

Five findings from @yingying0906, all fixed in 0afebd906:

1. The unit suite carried a byte-copy of the runner-name regex, so it asserted on its own copy rather than the extractor that runs → shared generatedRunnerNames in the lib, plus a test pinning it against the real committed integration.auto.cjs so a template rename cannot yield zero runners and make every coverage rule vacuously pass.
2. deferred was global, so per-platform deferral was unrepresentable and the llm/ocr ports were blocked → the map form, and platformNames now excluding deferred by name rather than by shape.
3. isPlatformEntry read every top-level object as a platform, which breaks on llm's iosWeekly/androidWeekly → options.platforms.
4. The reference diff does not catch generator-template changes → wording corrected to say so (nothing regressed; the mtime check missed it too).
5. validate-devices builds its allowlist from .<platform> too, so -f tests=runPi05Test is rejected → README now documents both halves of the gate, including that _skipMobilePi05 skips both pi05 cases on mobile regardless of scheduling, so un-deferring alone runs zero tests and passes green.

🤖 Generated with Claude Code

[github-actions]

Review Status

Current Status: ✅ APPROVED
Approvals so far: Team Lead: 1, Member: 1

[github-actions]

License compliance — clean

No new dependency license findings in this PR.

Warn-only (shadow) mode — this check does not block merges yet.

Updated automatically by the canonical license compliance workflow.

NOTICE presence (advisory)

Missing NOTICE (advisory, does not block):

• ./.github/actions/release-merge-guard
• ./docs/website
• ./packages/ggml-coload-smoke
• ./packages/fabric/test/integration
• ./packages/inference-addon-cpp/mobile
• ./packages/sdk/e2e
• ./packages/llm-llamacpp/benchmarks/performance
• ./packages/llm-llamacpp/benchmarks/server
• ./packages/vla-ggml/sim/server
• ./packages/embed-llamacpp/benchmarks/performance
• ./packages/embed-llamacpp/benchmarks/server
• ./packages/asr-ggml/benchmarks/server

[yingying0906]

Ran the branch locally. The diagnosis is right and the split is the right shape, nits inline.

What I checked myself:

1. ts-checks in on-pr-vla.yml only needs authorize and fork-approval and runs npm run test:unit, and #4006 touched packages/vla-ggml/** so it matches the paths filter. The check would have caught it.
2. deferred really is inert to CI. upload-to-devicefarm reads .[$p] only and validate-devices scopes to PLAT_KEY.
3. prettier --check with the real prettier-config-holepunch is clean on all changed scripts, and node scripts/validate-mobile-tests.js exits 0 with and without CI=true.
4. Dropping the mtime check is fine. buildFileContents only reads filenames and never opens a test file, so the timestamp could never mean anything.

Two things that don't fit on a line:

The PR body is behind the head commit. It says the mtime heuristic is skipped when CI is set and that the new suite has 8 tests, but 7ef057275 removed mtime completely and the suite has 9. The body ends up as the squash message so worth fixing.

llm-llamacpp and ocr-ggml still have the same landmine and main can go red the same way from either. Splitting per release line makes sense, just file the tickets instead of leaving them in the PR body.

No CHANGELOG is fine, sanity-checks is green.

[iancris]

/review