Skip to content

Commit 8af1015

Browse files
mmckyclaude
andcommitted
CI: instrument the cold-cache execution failure, pin the mystmd engine
Diagnostics for the jax_intro.md kernel death on cold-cache runs (kernel dies with no Python traceback ~66s into execution; numpy.md is collateral). Instruments only -- execution itself is unchanged (default parallelism, no XLA env) so the failure reproduces faithfully: * Pin the QuantEcon/mystmd clone to the exact main SHA the failing runs used (main has not moved since 2026-06-12). A branch name here makes the engine a moving target between runs of the same commit. * Detached 5s sampler logging host RAM, top-RSS processes and GPU memory for the life of the job. * always() step reading the sampler log back plus dmesg, to confirm or rule out the host OOM killer -- a SIGKILL with no traceback is consistent with it but unproven. The cache key only hashes lectures/**/*.md, so this ci.yml-only commit reuses the failing runs' key; no saved cache exists under it (the failed jobs saved none), guaranteeing the cold execution path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent f8a3e85 commit 8af1015

1 file changed

Lines changed: 47 additions & 5 deletions

File tree

.github/workflows/ci.yml

Lines changed: 47 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -42,11 +42,19 @@ jobs:
4242
pip install -U "jax[cuda13]"
4343
python scripts/test-jax-install.py
4444
45+
# Pinned to the exact SHA of the fork's main (unmoved since 2026-06-12,
46+
# so this is the same engine the 2026-08-03 cold-run failures used).
47+
# A branch name here makes the engine a moving target between runs of
48+
# the same lecture commit, which makes failures impossible to bisect.
4549
- name: Install mystmd (QuantEcon fork)
4650
run: |
47-
git clone --branch main --depth 1 \
48-
https://github.com/QuantEcon/mystmd.git /tmp/qe-mystmd
51+
mkdir -p /tmp/qe-mystmd
4952
cd /tmp/qe-mystmd
53+
git init -q
54+
git remote add origin https://github.com/QuantEcon/mystmd.git
55+
git fetch -q --depth 1 origin 96ee78f550ff2e0410730f328352e98d3e29d338
56+
git checkout -q FETCH_HEAD
57+
git rev-parse HEAD
5058
bun install
5159
bun run build
5260
npm install -g /tmp/qe-mystmd/packages/mystmd
@@ -57,15 +65,49 @@ jobs:
5765
- name: Verify mystmd version
5866
run: myst --version
5967

68+
# DIAGNOSIS (PR #363 cold-cache jax_intro kernel death): sample host
69+
# RAM, top-RSS processes and GPU memory every 5s for the rest of the
70+
# job. The nohup'd loop survives step boundaries; the runner reaps it
71+
# at job end. Read back in the always() step after the build.
72+
- name: Start memory monitor
73+
run: |
74+
nohup bash -c 'while true; do
75+
echo "=== $(date -u +%H:%M:%S)"
76+
free -m | sed -n "2,3p"
77+
ps -eo pid,rss,etime,comm --sort=-rss | head -8
78+
nvidia-smi --query-gpu=memory.used,memory.total --format=csv,noheader 2>/dev/null
79+
sleep 5
80+
done' >/tmp/memmon.log 2>&1 &
81+
echo "monitor started"
82+
6083
# Built with the fork's `myst` CLI rather than `jupyter book build`.
6184
# jupyter-book 2.x is a Python shim that execs a mystmd bundle vendored
62-
# inside its own wheel (jupyter_book/dist/jupyter-book.cjs) and offers no
63-
# way to point it at another CLI — so `jupyter book build` silently
64-
# ignored the fork we install above and built with upstream mystmd.
85+
# inside its own wheel (jupyter_book/dist/jupyter-book.cjs) and offers
86+
# no supported way to point it at another CLI — so `jupyter book build`
87+
# silently ignored the fork we install above and built with upstream
88+
# mystmd (visible as 23 `unknown export output extension: *.ipynb`
89+
# errors, since ipynb export is a fork-only capability).
6590
- name: Build HTML
6691
working-directory: ./lectures
6792
run: myst build --html --execute
6893

94+
# A kernel death with no Python traceback is consistent with a SIGKILL
95+
# from the host OOM killer, but that has never been confirmed. dmesg
96+
# covers the whole job because runs-on boots a fresh instance per run.
97+
- name: Execution diagnostics
98+
if: always()
99+
run: |
100+
echo "::group::memory timeline (5s samples)"
101+
cat /tmp/memmon.log || true
102+
echo "::endgroup::"
103+
echo "::group::kernel log (OOM killer / segfaults)"
104+
(sudo dmesg -T || dmesg -T) 2>/dev/null | grep -iE "out of memory|oom|killed process|segfault" || echo "no OOM/segfault records"
105+
echo "::endgroup::"
106+
echo "::group::final state"
107+
free -m
108+
nvidia-smi 2>/dev/null || true
109+
echo "::endgroup::"
110+
69111
- name: Upload build output
70112
uses: actions/upload-artifact@v7
71113
with:

0 commit comments

Comments
 (0)