Skip to content

Commit 51ff901

Browse files
cursoragentFluffyAIcode
cursoragent
and
FluffyAIcode
committed
docs(skill): add reusable 'pin self-hosted runner Python env' skill + prompt
Captures the diagnosis+fix for the post-reboot ModuleNotFoundError (mlx_lm) on the kakeya-mac-m4 runner: lightweight env-probe diagnosis, 3-layer fix (pin venv on the runner agent PATH via .path/.env|launchd|systemd; resolve a pinned interpreter in the workflow/executor instead of bare python3; fail-fast import gate), reboot-inclusive verification, and the Cloud-VM-vs-runner distinction (Mac-only deps belong on the runner, not the Linux Cloud Agent env). Includes a ready-to-paste setup-agent prompt; generalized for any Claude/Codex agent. Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
1 parent f13594d commit 51ff901

2 files changed

Lines changed: 194 additions & 0 deletions

File tree

‎docs/kakeyainferenceenginebuildskill.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -351,5 +351,6 @@ If any answer is "no", write the weaker, true claim.
351351
- v0.5-cuda scorecard (+ honest §5): `docs/reports/kakeya-inference-engine-v0.5-cuda.md`
352352
- Engine vs vLLM long-context journey: `docs/reports/kakeya-engine-vs-vllm-h200.md`, `docs/reports/kakeya-vs-vllm-longcontext-h200.md`
353353
- MLX port lessons: `docs/mlx-port-lessons.md`
354+
- Self-hosted runner Python pinning (reboot-proof mlx_lm/torch/transformers): `docs/skills/pin-selfhosted-runner-python-env-skill.md`
354355
- f_θ training pipeline: `docs/design/k3-f-theta-training-pipeline.md`
355356
- Session capacity / cross-host: `docs/adr/0014-agent-connection-capacity-and-cross-host-topology-tests.md`

‎docs/skills/pin-selfhosted-runner-python-env-skill.md‎

Lines changed: 193 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,193 @@
1+
# Skill: Pin a self-hosted runner's Python env (survive reboots, reproducible heavy ML deps)
2+
3+
**Reusable across agents (Claude / Codex / Cursor).** Copy this file or paste the
4+
prompt in the appendix. It is written to be repo-agnostic; the concrete examples
5+
use a GitHub Actions self-hosted Mac runner driving MLX (`mlx_lm`/`torch`/
6+
`transformers`), but the pattern applies to any self-hosted runner (Mac or Linux)
7+
that runs heavy ML/native deps from a virtualenv.
8+
9+
---
10+
11+
## 1. When to use this skill
12+
13+
Trigger it when **a self-hosted runner job fails on a missing module that "used to
14+
work"**, especially after a host **reboot / OS or Python upgrade / runner
15+
re-register**. Classic signatures:
16+
17+
- `ModuleNotFoundError: No module named 'mlx_lm'` (or `torch`, `transformers`, …)
18+
in a job that previously passed.
19+
- The failure is **fast** (seconds) — it dies at `import`, before any real work.
20+
- A **lightweight probe** (one that only needs stdlib + a base package) still
21+
passes, proving the runner is *online* but pointing at the **wrong interpreter**.
22+
- The interpreter version changed (e.g. `python=3.14.3` where it used to be
23+
`3.13.x`), or `pkg=None` for a package that should be installed.
24+
25+
Root cause is almost always: the workflow invokes a **bare `python3`**, and after
26+
the reboot the default `python3` on `PATH` is no longer the venv that has the
27+
deps. The venv still exists; nothing points at it.
28+
29+
---
30+
31+
## 2. Diagnose first (don't guess)
32+
33+
Run the **cheapest possible probe** through the same runner path to read the
34+
interpreter + module state, instead of assuming. Example (adapt the import list):
35+
36+
```bash
37+
python3 - <<'PY'
38+
import sys
39+
def v(m):
40+
try:
41+
mod = __import__(m); return getattr(mod, "__version__", "ok")
42+
except Exception as e:
43+
return f"MISSING ({e.__class__.__name__})"
44+
print("python =", sys.version.split()[0], "| exe =", sys.executable)
45+
for m in ("mlx", "mlx_lm", "torch", "transformers"):
46+
print(f"{m} = {v(m)}")
47+
PY
48+
```
49+
50+
Decision rule:
51+
- **Runner online + probe shows wrong `python`/`exe` or `MISSING` deps** → this skill (interpreter pinning).
52+
- **Probe itself never starts (job stuck `queued`/`pending`)** → the runner *agent*
53+
is down; restart the agent first (different problem).
54+
55+
> In CI-driven runners, route the probe through the same executor the real jobs
56+
> use (so `PATH`/env match). A one-liner like the above, committed as a tiny
57+
> "env-probe" job/preset, is worth keeping permanently.
58+
59+
---
60+
61+
## 3. Fix — three layers (do all three; they are defense-in-depth)
62+
63+
### Layer A — Pin the interpreter the runner *agent* sees (host side, durable)
64+
65+
Make the venv's `bin` the first thing on the **runner agent's** `PATH`, so a bare
66+
`python3` resolves to the venv even across reboots. Pick the mechanism for how the
67+
agent is launched:
68+
69+
- **GitHub Actions runner as a service (recommended).** The runner reads a
70+
`.env` and a `.path` file in its install dir at start:
71+
```bash
72+
cd ~/actions-runner
73+
echo "$HOME/kakeya-venv/bin" > .path # prepended to PATH
74+
echo "VIRTUAL_ENV=$HOME/kakeya-venv" >> .env
75+
./svc.sh stop && ./svc.sh start # reload
76+
```
77+
(`.path` is concatenated ahead of the system PATH for every job; `.env` injects
78+
process env. Both persist across reboots because the service re-reads them.)
79+
- **launchd plist (macOS), if not using `svc.sh`.** In the runner's
80+
`~/Library/LaunchAgents/<runner>.plist`, set:
81+
```xml
82+
<key>EnvironmentVariables</key>
83+
<dict>
84+
<key>PATH</key><string>/Users/&lt;you&gt;/kakeya-venv/bin:/usr/bin:/bin:/usr/sbin:/sbin</string>
85+
</dict>
86+
```
87+
then `launchctl unload/load` the plist.
88+
- **systemd (Linux self-hosted).** In the runner unit:
89+
`Environment="PATH=/opt/kakeya-venv/bin:%h/.local/bin:/usr/bin:/bin"`, then
90+
`systemctl daemon-reload && systemctl restart <runner>`.
91+
92+
Verify: `python3 -c "import mlx_lm, torch, transformers; print('ok')"` from a job.
93+
94+
### Layer B — Make the workflow/executor resolve a *pinned* interpreter (repo side, robust)
95+
96+
Never call a bare `python3` for the heavy job. Resolve an explicit interpreter so
97+
the repo is robust even if Layer A drifts:
98+
99+
1. Add a repo/runner variable, e.g. `KAKEYA_MAC_PYTHON`, pointing at the venv
100+
python (`/Users/<you>/kakeya-venv/bin/python`). Default-discover if unset:
101+
```bash
102+
PYBIN="${KAKEYA_MAC_PYTHON:-}"
103+
for c in "$PYBIN" "$HOME/kakeya-venv/bin/python" "$(command -v python3.13)" "$(command -v python3)"; do
104+
[ -n "$c" ] && [ -x "$c" ] && "$c" -c 'import mlx_lm' 2>/dev/null && { PYBIN="$c"; break; }
105+
done
106+
```
107+
2. Use `$PYBIN` (or substitute a `${PYTHON}` token in your command templates)
108+
instead of `python3` for the actual workload. If your executor spawns argv
109+
lists (no shell), resolve the token to `$PYBIN` before `subprocess.run`.
110+
111+
### Layer C — Fail fast with a clear message (repo side, observability)
112+
113+
Before the expensive step, assert the deps and **print a fix hint** so the next
114+
failure is self-explanatory instead of a deep `ModuleNotFoundError`:
115+
116+
```bash
117+
"$PYBIN" - <<'PY' || { echo "::error::runner python missing ML deps — see pin-selfhosted-runner-python-env-skill.md (Layer A)"; exit 90; }
118+
import mlx_lm, torch, transformers # noqa
119+
PY
120+
```
121+
122+
---
123+
124+
## 4. Verify the fix
125+
126+
1. Re-run the lightweight env-probe → correct `python`/`exe`, all deps present.
127+
2. Re-run one **real** (heavy) job → no `ModuleNotFoundError`, completes.
128+
3. **Reboot the host and re-run** (the actual regression you are fixing) → still
129+
green. This step is the whole point; do not skip it.
130+
131+
---
132+
133+
## 5. Generalizing to a *Cloud Agent* VM env setup (different machine!)
134+
135+
Do **not** confuse the self-hosted runner with the Cloud Agent VM:
136+
- The **Cloud Agent VM** is typically Linux; it runs the *client* that dispatches
137+
jobs and the unit-test gate. **Mac-only deps (MLX) do not belong there.** Put
138+
only what the client/tests need into the Cloud Agent env setup (base image +
139+
startup script), and pin versions.
140+
- The **self-hosted runner** is where the heavy/native/Mac deps live. Pin them
141+
there (Layers A–C above), not in the Cloud VM env setup.
142+
143+
For the Cloud Agent VM specifically: bake stable deps into the **base image**, do
144+
slow-changing installs in the **startup script**, and pin versions so a new VM is
145+
reproducible. (In Cursor, this is the "env setup agent" config.)
146+
147+
---
148+
149+
## 6. Anti-patterns
150+
151+
- ❌ `pip install` the missing dep into whatever `python3` happens to be active
152+
(often a too-new system Python with no wheels for `torch`/`mlx_lm`). Pin to the
153+
known-good venv instead.
154+
- ❌ Hardcoding an absolute interpreter path in many places. Resolve once
155+
(variable + discovery) and reuse.
156+
- ❌ "It works now" without a reboot test — the regression is reboot-triggered.
157+
- ❌ Relying on an interactive shell's `source venv/bin/activate`; CI jobs and
158+
services don't run your `.zshrc`.
159+
160+
---
161+
162+
## Appendix — ready-to-paste prompt for a setup agent
163+
164+
> **Task: make our self-hosted CI runner's Python environment reboot-proof.**
165+
>
166+
> Symptom: jobs on our self-hosted runner fail fast with
167+
> `ModuleNotFoundError: No module named 'mlx_lm'` after the host rebooted; a
168+
> lightweight env-probe shows the runner's default `python3` switched to a newer
169+
> interpreter that lacks our ML stack (`mlx_lm`/`torch`/`transformers`), while the
170+
> known-good venv still exists but is no longer on `PATH`.
171+
>
172+
> Do all of the following, smallest-diff first, and verify each:
173+
> 1. **Diagnose:** run a tiny probe that prints `sys.version`, `sys.executable`,
174+
> and import status of `mlx_lm, torch, transformers` through the same path the
175+
> real jobs use. Confirm the wrong interpreter / missing modules.
176+
> 2. **Host (runner agent):** pin the venv's `bin` ahead of system `PATH` for the
177+
> runner service so a bare `python3` resolves to the venv across reboots — via
178+
> the runner's `.path`/`.env` files (GitHub Actions `svc.sh`), or the
179+
> launchd/systemd unit's `PATH` env. Reload the service.
180+
> 3. **Repo (workflow/executor):** stop calling bare `python3` for the heavy job.
181+
> Resolve a pinned interpreter from a `*_PYTHON` repo/runner variable, with a
182+
> discovery fallback that picks the first candidate where `import mlx_lm`
183+
> succeeds; use it for the workload commands.
184+
> 4. **Repo (fail-fast):** before the expensive step, assert
185+
> `import mlx_lm, torch, transformers` and emit a clear `::error::` with a link
186+
> to this skill if missing (exit non-zero).
187+
> 5. **Verify, including a reboot:** env-probe green, one real heavy job green,
188+
> then reboot the host and re-run the same job — must still be green.
189+
> 6. **Pin versions** in the venv (freeze a lockfile) and document the venv path +
190+
> rebuild steps so the environment is reproducible, not just patched.
191+
>
192+
> Keep the heavy/native deps on the self-hosted runner only; do NOT add Mac-only
193+
> deps to the Cloud Agent (Linux) VM env setup.

0 commit comments

Comments
 (0)