Shards precompiling into one shared depot can race, and sharding the whole fleet made it likely

[sotashimozono]

A shard died on the default branch with

ERROR: LoadError: The following 1 direct dependency failed to precompile:
SystemError: opening file "…/.julia/compiled/v1.12/Markdown/AREjX_vfzSw.ji": No such file or directory

One shard read a cache file while another was replacing it. On a self-hosted pool the depot is persistent and shared per box, so N shards on the same box precompile into the same ~/.julia/compiled.

Why this is worth writing down rather than re-running past

The probability is a function of how many jobs precompile concurrently, and that number just changed by an order of magnitude. Before today, one repository ran one test job. Today thirty-odd repositories each run four to sixteen shards, on the same pool, and they will keep doing that on every push. The migration did not introduce the hazard; it moved it from "essentially never" to "expect it".

The failure is at least loud, and a re-run clears it. But a fleet where any run can fail for a reason unrelated to the code is a fleet where red checks stop being read — which is the thing several of this package's other guards exist to prevent.

What might fix it

• julia-actions/cache already handles the hosted case; the self-hosted case is the one with a shared writable depot.
• A per-shard depot — JULIA_DEPOT_PATH under the runner's workspace, with the shared one appended read-only. Costs precompilation time per shard, which is exactly what the shared depot was buying.
• prebuild already exists and sidesteps it by precompiling once — but it is documented hosted-only, because Julia keys a package's compiled cache on its absolute source path and that path carries the runner's name on a self-hosted pool. Worth re-checking whether that reasoning still holds for DEPENDENCIES, which is where the race is: the failing file was Markdown, a stdlib, not the package under test.

That last point may be the cheap fix: dependencies do not change per commit and their caches could be built once and shared read-only, leaving only the package under test to be compiled per shard.

Status

Two repositories hit it on the same afternoon; both re-runs are pending as I write this. Filing before knowing whether they clear, because "the re-run was green" is not evidence the race is gone — it is evidence that a race is a race.

[sotashimozono]

Both re-runs are green, which is what a race looks like and not a reason to close this.

repo	evidence
ITensorcMPS.jl	error read directly — compiled/v1.12/Markdown/….ji: No such file or directory. Green on re-run. Cause known, transience confirmed.
SingleSiteDMRG.jl	green on re-run, but its log blob had already expired, so I never saw the error. Transient, cause unverified — I am not claiming it was this.

So: one confirmed instance, one that merely resembles it.

The reason to keep this open is not the two failures. It is that both landed within an hour of each other, on the afternoon the pool went from running one test job per repository to running four-to-sixteen shards per repository across thirty of them. That ratio is now permanent. Two in one afternoon at the very start of the new steady state is the number worth acting on, not the fact that re-running cleared them.

The narrow observation from the confirmed case still looks like the cheap fix: the file that lost the race was Markdown, a stdlib — a dependency, not the package under test. Dependencies do not change per commit, so their caches could be built once and shared read-only, leaving only the package under test compiled per shard. That is prebuild's idea applied to the half of the problem where the absolute-source-path objection does not bite.