[VHS] Make lab campaigns work-conserving and stall-aware

[mattmre]

Problem

The EGV/Qwen campaign eventually produced a valid negative result, but orchestration was too serialized around long waits and repeated gate checks. Available accelerators and review capacity were idle while independent work could have progressed. This is a process defect because it increases time-to-evidence and delays detection of zero-signal campaigns.

Required behavior

• Treat model generation, deterministic evaluation, artifact sealing, independent review, and publication checks as separate dependency-aware lanes.
• Dispatch ready work to any available worker; no accelerator waits for unrelated review or publication work.
• Persist a machine-readable lane state with owner, input digest, start time, last-progress time, resource assignment, output digest, and terminal reason.
• Poll long-running jobs asynchronously; no controller may block the whole campaign on one wait.
• Emit a progress checkpoint at least every 5 minutes while work is active.
• Detect zero semantic signal early: after the preregistered pilot checkpoint, stop or pivot before the full matrix rather than spending the full budget blindly.
• Detect a stalled lane after 10 minutes without new evidence and automatically retry, reassign, or fail closed with a recorded reason.
• Record accelerator utilization and explain any idle interval longer than 5 minutes while runnable work exists.
• Keep credentials, host addresses, private paths, prompts, hidden evaluator material, and topology out of public artifacts.

Acceptance criteria

1. A deterministic scheduler simulation proves that two independent generation lanes run concurrently and that evaluation begins as soon as its first dependency is ready.
2. A hung-job fixture proves stall detection, bounded cancellation, retry/reassignment, and durable recovery without duplicate promotion.
3. A zero-pass fixture proves the semantic-signal checkpoint stops the campaign at the frozen pilot boundary.
4. A restart fixture proves the scheduler resumes from the durable DAG without repeating completed work.
5. A machine-readable utilization report identifies busy, legitimately blocked, and avoidably idle time per resource.
6. An adversarial review attempts to create deadlock, duplicate work, stale promotion, false progress, and secret/topology leakage.

Claim boundary

Closing this issue proves work-conserving orchestration and stall handling under the tested fixtures. It does not prove model quality, research utility, distributed training, or cross-node model transport.