Run five fresh-install trials on unfamiliar machines

[vivekchand]

We have no evidence about what a first install feels like to someone who did not build it.

Five people, machines we do not control, no help from us. Cover macOS, Linux and Windows, and include Claude Code and Codex users — not only OpenClaw.

This is a usability gate, not a statistic: five is enough to find the blocking bug, and not enough to estimate a rate.

Done when

• At least four of five reach a real session with no rescue.
• Every failure is written down with an owner and either a fix or a documented workaround.
• Anything that blocked two or more people is fixed before the trials are considered passed.

[vivekchand]

Split this into the half a machine can do and the half it cannot, so the human trials are a 20-minute job rather than a project.

The machine half is now automated (#5718): scripts/verify_published_wheel.py, run on ubuntu/macos/windows by .github/workflows/verify-published-wheel.yml, installs the published wheel from PyPI and checks version resolution, that it serves, the no-runtime state, a fully blocked network, and a clean uninstall. Green there rules out most mechanical failure before anyone's time is spent.

The human half is docs/INSTALL_TRIAL.md — carried on the #5726 branch. It has the tester mix (the axes that have actually broken: Windows, Claude Code/Codex rather than only OpenClaw, a machine with no agent history, someone who has never heard of the project), the exact message to send, a per-trial recording block, and the one rule that makes it worth running:

Do not help. If you answer a question, that trial is over — record it as a failure with the question they asked, because a reader on Hacker News will hit the same wall with nobody to ask.

The gate stays 4 of 5 unaided, recorded as passed/failed plus the blocker list, never as a percentage. Five people find the blocking bug; five people cannot estimate a rate.

This issue still needs five real people. Nothing above changes that — it only removes the setup cost. Anything that blocks two or more of them gets fixed before the gate counts as passed.

[vivekchand]

Machine half re-run against the CURRENT wheel, so the human trials start from a known-good artifact

Not the trials themselves. This issue asks for five people on machines we do not control with no help from us, and I am none of those. Running it myself would produce a number that looks like the gate and is not it.

What I could usefully do is make sure the five testers do not spend their time on a mechanical failure we could have found first.

The gate had gone stale. verify-published-wheel.yml last ran at 15:41 on 2026-09-09. Six releases have shipped since (0.12.848 through 0.12.853), including a schema migration (15 → 16), a new module, and store changes. A tester today installs 0.12.853, and no wheel since 0.12.847 had been verified on any platform.

Dispatched it for 0.12.853. All four legs green (run 34428750905):

success  ubuntu-latest  / py3.9
success  ubuntu-latest  / py3.11
success  macos-latest   / py3.11
success  windows-latest / py3.11

Each leg exercising the six things a first-time user does:

PASS  install      pip resolved 0.12.853
PASS  neutral cwd  --version reports 0.12.853
PASS  serves       / returned 200, /api/overview returned JSON
PASS  no-runtime   first run names the missing runtime state
PASS  offline      starts and serves with all egress black-holed
PASS  uninstall    console script and package both gone

Also ran the same script locally on darwin before dispatching, with the same result.

So Windows and Linux are clear on the current wheel, which the prior comment names as the axes that have actually broken. The remaining risk in the trials is real usability, which is what the trials are for.

One structural note, not a change I made

The workflow is workflow_dispatch by design — its own header says "a release gate rather than a per-commit check: pin one version, run it, record the result." That is defensible. What is observably not happening is the dispatching: none of tonight's six releases triggered it, and the gate sat 11 hours behind the artifact users were installing.

Wiring it to fire after release-on-merge succeeds would close the gap without making it a per-commit cost. That is a change to the release process rather than a bug, so I have not made it.

Still needs humans

• five testers, machines we do not control, no rescue
• at least four of five reach a real session
• every failure written down with an owner
• anything blocking two or more fixed before the gate passes

docs/INSTALL_TRIAL.md is on main and carries the tester mix, the message to send, and the recording block.

---

Generated by Claude Code