Commit 94a7e52
CI: disable JAX GPU preallocation for concurrent notebook execution
Fixes the cold-cache kernel death diagnosed in the instrumented run
(30806359596). The three JAX lectures execute concurrently on the
g4dn runner's single T4; whichever kernel touches the GPU first
preallocates JAX's default 75% (11.3GiB of 15.3GiB). The diagnostics
showed jax_intro's kernel losing that race, clamping to the ~3.4GiB
remainder, and aborting ~40s later when a cell outgrew the pool --
an XLA fatal with no Python traceback, no OOM-killer or segfault
records in dmesg, and GPU usage pinned at 3427MiB until the death.
The jupyter-book-era green cold run won the same race by seconds --
the engine swap changed execution phasing, not semantics -- and the
JB1 production build on main executes sequentially, which is why
neither ever hit this.
XLA_PYTHON_CLIENT_PREALLOCATE=false switches JAX to on-demand
allocation so the concurrent kernels share the GPU. The memory
monitor and dmesg diagnostics stay in place; this run doubles as
the fix's validation.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>1 parent 8af1015 commit 94a7e52
1 file changed
Lines changed: 10 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
89 | 89 | | |
90 | 90 | | |
91 | 91 | | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
92 | 102 | | |
93 | 103 | | |
94 | 104 | | |
| |||
0 commit comments