Build once and distribute, instead of every shard building for itself

[sotashimozono]

Every shard currently runs julia-buildpkg for itself, so an N-shard run instantiates and precompiles the same project N times. The obvious fix is to build once and hand the result to the children.

The distinction that decides whether this is worth it

Redundant precompilation across shards is concurrent. Eight shards each spending 50 s precompiling costs 400 runner-seconds but only 50 s of wall clock. Building once and distributing serialises that work:

today:      precompile ∥ precompile ∥ …  →  test     =  precompile + max_bin
build-once: build → distribute → test                 =  build + download + max_bin

So on wall clock this is neutral at best and usually slightly worse. On runner-seconds it is a large win — Nx precompile becomes 1x.

Writing this down because it is the kind of claim that otherwise gets argued from intuition each time it comes up. If the goal is a faster PR, this is not the lever. If the goal is cheaper CI, it is a good one.

Where it does buy wall clock

When shards start staggered — see #8 — a late starter that can restore a prebuilt depot is cheap instead of paying full startup. Build-once and work-stealing compose; neither substitutes for the other.

Options

approach	effect	cost
Prebuild job uploading the depot as an artifact, shards restore it	Nx precompile becomes 1x	serialises the build; a large artifact transferred N times
julia-actions/cache with an explicit depot input	same cache, one location	still restores per job
Sysimage via PackageCompiler	biggest cut to per-shard startup	slow to build, and must match the --check-bounds setting or Julia will not use it (see #10)
Persistent depot on a self-hosted pool	already does this	why the private repos' fixed cost differs from the hosted ones

Prerequisite

Do not start here. #10 has to establish where the fixed cost actually goes first — there is a live hypothesis there that --check-bounds=yes makes the test step recompile a stack julia-buildpkg already precompiled under different flags. If that holds, it is a larger lever than distributing a build, and cheaper to act on.

TODO

• Settle Confirm the shared depot cache actually lowered the per-shard fixed cost #10 first: measure a warm steady-state run and find where the per-shard startup actually goes.
• Only if precompilation is still a significant share: prototype a prebuild job with a depot artifact, behind an input.
• Report both wall clock and runner-seconds in the comparison. Reporting one without the other is how this decision gets made wrongly.

[sotashimozono]

Reversing my earlier recommendation on this issue

Earlier in #10 I recommended closing this as measured-not-worth-building. That measurement was taken on this repository — one dependency, julia-buildpkg at 3–4s — and I said at the time it needed re-checking against a heavy consumer. Here is that check, and it goes the other way decisively.

QAtlas.jl, 16 shards, measured on three consecutive main runs

Inside the julia-runtest step of a single shard:

Precompiling for configuration --code-coverage=@/home/runner/work/QAtlas.jl/QAtlas.jl
    4566.5 ms  ✓ ForwardDiff
     691.2 ms  ✓ ForwardDiff → ForwardDiffStaticArraysExt
     819.6 ms  ✓ AbstractQAtlas → AbstractQAtlasForwardDiffExt
  218671.0 ms  ✓ QAtlas
     788.7 ms  ✓ QAtlas → QAtlasForwardDiffExt
   5 dependencies successfully precompiled in 220 seconds. 60 already precompiled.

QAtlas itself: 219 s. Every run. Every shard. Confirmed across runs: 220 s, 211 s, 220 s — all on a warm cache hit (Cache restored from key: shard-s01;...), with 60–61 dependencies correctly served from it.

precompile, per shard	~215 s
shards	16
runner-seconds per run, spent precompiling the same package	~3,440
on the critical path of every shard	~215 s
whole run, all 16 shards	9,911 s

So roughly 35% of QAtlas's CI is one package being precompiled sixteen times.

Why the cache does not help

The dependencies are cached and restored (60 of 65). What is not is the package under test, precompiled for the coverage configuration — a different configuration from the one julia-buildpkg produced, and rebuilt from the workspace checkout each run. The 219 s is @compile_workload; see the note in the QAtlas memory that 98% of its precompile time is the workload block.

That is exactly the shape this issue proposes to fix: build the artifact once, distribute it to the shards, rather than having each shard rebuild it.

Where that leaves the gate

The gate on this issue was "measure it on a repository whose buildpkg is large". Measured:

	precompile per shard	shards	worth building?
TestShards.jl	3–4 s	8	no
QAtlas.jl	~215 s	16	yes — ~3,440 runner-seconds a run

Reopening the direction. Related: #29 (the org's runner budget is shared, so 3,440 s of avoidable work in one repo is taken from every other repo's queue) and #19 (coverage cost per shard), which is the same configuration boundary seen from the other side.

One caveat worth stating: whether a coverage-instrumented precompile artifact can be moved between jobs at all is not established here. This measurement says the prize is large; it does not say it is collectable.

[sotashimozono]

The gate is passed: precompiled output DOES transfer between jobs

That was the one thing that could have killed this direction, and it was genuinely open — in #19 two shards in the same run rejected a byte-identical cached artefact from each other, cause still unknown. If .ji files did not survive a move between machines, no amount of arithmetic about the prize would have mattered.

Measured on this repository, same branch, same shard count, prebuild the only variable:

	shard s1 log
prebuild: false	Precompiling for configuration --code-coverage=@… → 1 dependency successfully precompiled in 1 seconds. 31 already precompiled.
prebuild: true	no precompile line at all

The shards adopted a cache built in a different job, on a different machine, and skipped precompilation entirely.

The cost of adopting it is negligible

Per shard, averaged over four:

step
download-artifact	1.8 s
untar into the depot	0.0 s
artifact size (this package)	4.4 MB

On this suite it is a net loss, exactly as predicted

	shard runner-sec	total runner-sec
prebuild: false	340	581
prebuild: true	352	614

The prebuild job costs ~24 s plus its own job overhead, to save ~1 s of precompilation on each of four shards. Predicted before running: a loss here, a win at scale. It is a loss here.

The wall clock figures from these two runs are not usable — 522 s against 223 s, while the shard durations were 85 s and 88 s. That is the queue spread, not the strategy, and reading it as a result would repeat the mistake this repository has already documented twice.

Extrapolation, and what remains unmeasured

At QAtlas.jl's numbers (~219 s of precompilation inside every one of sixteen shards):

saved      16 × 219   = 3,504 runner-seconds
cost       ~250 s prebuild job + 16 × ~2 s adoption
net        ≈ 3,220 runner-seconds a run

Two things are not established and should be before this is turned on there:

1. Artifact size. 4.4 MB here is the whole of ~/.julia/compiled for a package with one dependency. QAtlas has sixty-odd, and the transfer is paid once per shard. If it is 400 MB the adoption cost stops being 1.8 s. Shipping only the delta over the depot cache is possible and not implemented.
2. Wall clock. The build is serialised in front of the shards, so it buys runner time and costs some wall clock. Which side of that trade is wanted is a fact about the account — see Shard counts are per repo; the runner budget is per org #29, where 3,220 runner-seconds is capacity taken from every other repository in the org — and not about the suite. Hence an input, off by default.

Shipped in #36 as prebuild, off by default.

[sotashimozono]

(a) is answered, and the premise it rested on was wrong

This issue and the handoff both assumed QAtlas has "sixty-odd dependencies", which is what made
artifact size look like a risk. It has 4 direct deps, no [sources], and the prebuild step
instantiates --project=. — not the test env — so the eight [extras] never enter. Measured by
reproducing the prebuild step verbatim (same command, same flags) in an isolated
JULIA_DEPOT_PATH against a fresh resolve, since Manifest.toml is gitignored there:

	TestShards	QAtlas
closure packages	1 dep	23
compiled.tgz	4.4 MB	25.2 MB (25,239,850 B)
compiled/ unpacked	—	98 MB
precompile wall	—	630 s, of which QAtlas itself is 521.6 s (99%)

~25 MB × 16 shards ≈ 400 MB a run. Not the transfer that was feared. Two limits worth stating:
CI's artifact will exceed this, because the job runs julia-actions/cache@v3 and then tars all
of ~/.julia/compiled, so restored default-configuration entries ride along — which raises the
separate question of whether the step should tar only what it built. And per-shard adoption cost
at this size is not measured; scaling 1.8 s on 4.4 MB by size gives ~10 s, but that is arithmetic.

What actually needs establishing: the bridge, not the trade

The interesting thing about prebuild is not whether it pays on a given suite — that is the
caller's configuration to choose and keep. It is that it carries a build across environments,
and what makes that valid is not yet written down.

Two properties the mechanism depends on, neither currently checked at runtime:

The flags have to match. The prebuild job precompiles under --check-bounds=yes --code-coverage, deliberately, because that is a different precompile configuration from
julia-buildpkg's and produces different cache files. A consumer whose shards run under any other
configuration gets an artifact that is transferred and then ignored. Worth noting how invisible
this is: the expensive precompilation lives inside the test step, not in buildpkg — QAtlas's
buildpkg step is 6 s while ~219 s of its 680 s test step is compiling — so a maintainer reading
step durations would not see the cost move, or fail to move.

The paths have to match. Dependencies live under $HOME/.julia/packages, which is the same
location in every job. The package under test is compiled from $GITHUB_WORKSPACE, which is
identical on hosted runners (/home/runner/work/<repo>/<repo>) but carries the runner's own name
on a self-hosted pool (.../actions-runners/<runner>/_work/<repo>/<repo>). The gate in #36 was
passed on ubuntu-latest — this repository's CI passes no runner input — while the consumer
example in the workflow header is runner: '["self-hosted","rosina"]'. If Julia rejects a cache
whose source path moved, then on a self-hosted pool the artifact would transfer and the package's
own .ji would be rebuilt anyway — and for a suite like QAtlas that .ji is 99% of the cost.

I am measuring the second one rather than asserting it; I will post the result either way. The
first is checkable by construction.

And the adopt step cannot tell. It is tar -C "$HOME/.julia" -xzf compiled.tgz followed by
echo "Adopted the shared compiled cache." — it reports success for having unpacked a file, not
for the cache being used. Adoption was confirmed for #36 by reading a shard's log for the absence
of a precompile line. That is the property worth making the mechanism state about itself, since a
silently-rejected cache costs the transfer and returns nothing.

(An earlier revision of this comment proposed decomposing Diagnosis.fixed so the package could
report whether prebuild pays. Withdrawn — choosing the configuration is the caller's, and the
package should not abstract over which option is optimal.)

[sotashimozono]

Measured: the bridge carries the dependencies and drops the package

Promised above, so here it is either way — it went the way the hypothesis feared.

Three arms against one prebuilt depot (QAtlas, --check-bounds=yes --code-coverage, isolated
JULIA_DEPOT_PATH). Trees are byte-identical and copied with cp -a, so timestamps are
preserved and the only variable is location:

arm	project path	coverage root	result
control	as built	as built	0.39 s, no precompile
moved	different	moved with it	534 s, ✓ QAtlas rebuilt
third	different	held at the original	494 s, ✓ QAtlas rebuilt

The third arm is the disambiguation: with the --code-coverage=@dir argument held at the value
the cache was built under, a moved project path still forces a rebuild. The invalidator is the
absolute source path, not the coverage argument.

Every arm reported 17 already precompiled. So the split is clean, and it is structural rather
than incidental:

• Dependencies bridge. They are compiled from $HOME/.julia/packages, the same location in
  every job on every machine.
• The package under test does not. It is compiled from $GITHUB_WORKSPACE.

$GITHUB_WORKSPACE is /home/runner/work/<repo>/<repo> on hosted runners — identical on every
machine, which is why the #36 gate passed: this repository's CI passes no runner input and runs
on ubuntu-latest. On a self-hosted pool it is .../actions-runners/<runner>/_work/<repo>/<repo>
and carries the runner's own name, so a prebuild job on one runner and a shard on another do
not agree. The workflow header advertises exactly that configuration:
runner: '["self-hosted","rosina"]'.

The consequence is the bad one. For a suite like QAtlas the package's own .ji is 99% of the
precompilation (521.6 s of 527 s). So on a self-hosted pool the artifact would be built,
uploaded, downloaded once per shard — and then the expensive half rebuilt anyway.

There is a second-order twist on the pool this workflow points at: the nine panza-* runners run
as one user with no JULIA_DEPOT_PATH override, so they already share /home/souta/.julia
(17 GB of compiled). The half of the artifact that can cross is a half that machine already
had; the half that would pay for itself is the half that cannot.

What this says about the mechanism

Not that prebuild should be on or off anywhere — that is the caller's configuration to pick and
keep. It is that the bridge has a precondition it does not state and cannot check: the
workspace path must agree between the job that builds and the job that consumes. Hosted runners
satisfy it by construction, which is precisely why it was invisible.

And the adopt step cannot notice when it fails. It is:

tar -C "$HOME/.julia" -xzf compiled.tgz
echo "Adopted the shared compiled cache."

which reports success for having unpacked a file. A rejected cache costs the full transfer and
returns nothing, silently. Given this package's rule — say what the constraint is, do not resolve
it — the mechanism should be able to state whether the bridge held, rather than leaving it to be
confirmed by reading a shard's log for the absence of a precompile line.

Environment: Julia 1.12 locally, matching the julia-version: '1' the sharded workflow defaults
to. Not re-derived on CI.