Skip to content

Commit 94a7e52

Browse files
mmckyclaude
andcommitted
CI: disable JAX GPU preallocation for concurrent notebook execution
Fixes the cold-cache kernel death diagnosed in the instrumented run (30806359596). The three JAX lectures execute concurrently on the g4dn runner's single T4; whichever kernel touches the GPU first preallocates JAX's default 75% (11.3GiB of 15.3GiB). The diagnostics showed jax_intro's kernel losing that race, clamping to the ~3.4GiB remainder, and aborting ~40s later when a cell outgrew the pool -- an XLA fatal with no Python traceback, no OOM-killer or segfault records in dmesg, and GPU usage pinned at 3427MiB until the death. The jupyter-book-era green cold run won the same race by seconds -- the engine swap changed execution phasing, not semantics -- and the JB1 production build on main executes sequentially, which is why neither ever hit this. XLA_PYTHON_CLIENT_PREALLOCATE=false switches JAX to on-demand allocation so the concurrent kernels share the GPU. The memory monitor and dmesg diagnostics stay in place; this run doubles as the fix's validation. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent 8af1015 commit 94a7e52

1 file changed

Lines changed: 10 additions & 0 deletions

File tree

.github/workflows/ci.yml

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -89,6 +89,16 @@ jobs:
8989
# errors, since ipynb export is a fork-only capability).
9090
- name: Build HTML
9191
working-directory: ./lectures
92+
env:
93+
# Three JAX lectures (jax_intro, autodiff, numpy_vs_numba_vs_jax)
94+
# execute concurrently on the g4dn's single T4. By default the first
95+
# JAX process to touch the GPU preallocates 75% of it (11.3GiB of
96+
# 15.3GiB); the 2026-08-03 diagnostics showed jax_intro's kernel then
97+
# clamped to the ~3.4GiB remainder and aborted with no Python
98+
# traceback once a later cell outgrew that pool (dmesg had no
99+
# OOM-killer or segfault records — an XLA fatal, not a host kill).
100+
# On-demand allocation lets the concurrent kernels share the GPU.
101+
XLA_PYTHON_CLIENT_PREALLOCATE: 'false'
92102
run: myst build --html --execute
93103

94104
# A kernel death with no Python traceback is consistent with a SIGKILL

0 commit comments

Comments
 (0)