The sharded workflow needs no artifacts at all — and one 0.9 KB unguarded upload stops every shard when the account hits its quota

[sotashimozono]

The sharded workflow uploads six kinds of artifact per run. None of them needs to be an artifact, and because one of them is not guarded, an account that reaches its Actions artifact quota stops running tests entirely — on every repository, on every branch, including ones whose diff cannot be at fault.

What happened

From 2026-08-18 15:57 onward, every CI run in a six-repository fleet failed. ci / test / Shard labels went red and took All shards passed, Collect coverage and records and Coverage (upload) with it, because that job builds the shard matrix and nothing downstream runs without it. Zero test shards executed. main and CompatHelper's own branches failed identically, which is what ruled out any diff as the cause.

The visible tail of the log is an Azure BlobNotFound, which reads like a transient blip. It is not. Four reruns across four different runners (rosina-org-13, rosina-org-15, panza-4, panza-6) were spent on that reading before the real error was found further up the same step:

##[error]Failed to CreateArtifact: Artifact storage quota has been hit.
Unable to upload any new artifacts. Usage is recalculated every 6-12 hours.

The quota is enforced on CreateArtifact, before any bytes are sent. Shrinking an artifact does not help; a 0-byte one fails the same way.

And the artifact that stopped the fleet is the smallest one:

artifact	count	mean size	total
testshards-coverage-<sid>	2077	104.2 KB	211.41 MB
testshards-records	320	16.5 KB	5.17 MB
testshards-out-<sid>	2656	4.9 KB	12.74 MB
testshards-lcov	344	5.2 KB	1.74 MB
testshards-timings	357	0.9 KB	0.31 MB

-timings is uploaded at line 331 with no guard and if-no-files-found: error. It is a shard-balancing hint: without it the shards are balanced worse and still run correctly. A 0.9 KB advisory file is a hard gate on the entire test suite.

The migration: none of the six needs to be an artifact

Traced every upload to its consumer:

artifact	consumer	what it really is	where it belongs
-timings	line 481, a later run	regenerable cross-run hint	actions/cache
-prebuild	line 509, same run	a precompiled depot	actions/cache (or julia-actions/cache)
-records	—	records	actions/cache or step summary
-coverage-<sid>	line 671, the merge job	intra-run job-to-job transfer	upload each shard to Codecov directly with a flag
-lcov	nothing downloads it	the merged report	falls away with the merge job
-out-<sid>	line 882, diagnostics	logs	already in the job log; $GITHUB_STEP_SUMMARY for the digest

actions/cache is the right home for the first three for three independent reasons: a separate quota, LRU eviction (so 357 generations never accumulate), and a miss is harmless rather than fatal.

The coverage path is where the size is, and it is a re-implementation. Today: eight shards write artifacts → a merge job downloads all of them → merges with LCOV → sends one upload to Codecov. But Codecov already merges multiple uploads per commit, keyed by flag. Sending from each shard directly deletes the eight artifacts and the merge job, and gives per-shard flags — strictly more information than the single merged number available now.

The only case that currently justifies an artifact is "no CODECOV_TOKEN, so publish -lcov instead". The percentage is already written to the step summary, so the report body is rarely what anyone wants; if it is kept, make it conditional on the token being absent rather than produced on every run.

Why this is worth doing rather than working around

A prune of expired artifacts helps and is being done (30457 queued; the account went 2.72 GB → 1.47 GB), but it is after-the-fact: GitHub does not delete expired artifacts promptly — measured, one created 2026-07-14 with a one-day retention was still present on 2026-08-19, five weeks later, still counting against the quota. retention-days controls when an artifact becomes useless, not when it stops being charged.

So the storage will fill again. What this issue proposes is that when it does, it should not stop the tests.

Suggested order

1. Unblock first, minimally. Guard the -timings upload (inputs.record-timings already exists and does not gate it) or make its failure non-fatal. One or two lines; restores test execution during any future quota event.
2. Move -timings / -prebuild / -records to actions/cache. Different quota, self-evicting.
3. Send coverage per shard to Codecov with flags; delete the merge job, the eight coverage artifacts and -lcov.
4. Drop -out; the content is in the log.

After (1)–(4) the workflow uploads no artifacts at all and loses nothing — it gains per-shard coverage flags and stops being able to fail this way.

Measured 2026-08-19 across ParameterizedITensor.jl, ParaLinearAlgebra.jl, FunctionMeasures.jl, ComplexAnalysis.jl, ITensorcMPS.jl, ParameterizedITensorMPS.jl.

---

Two existing issues this joins up with — and one caveat against my own proposal

Found after filing; both are the same defect seen from another side.

#74 — "Per-shard coverage artifacts ship 8 copies of the source tree; 85.7 % of the payload is text collect already has checked out." That is the same 211 MB measured above, diagnosed from the payload side. Sending each shard to Codecov directly does not shrink that payload — it removes the artifact entirely, so #74 is resolved rather than mitigated. Worth doing them as one change rather than two.

#72 — "The depot cache has a 0 % hit rate: N shards save the SAME key, each uploading the whole depot, and the repo quota evicts everything."

This is a caveat against step (2) above, and it should be read before acting on it. My proposal moves -timings / -prebuild / -records to actions/cache on the argument that cache has a separate quota and evicts by LRU. #72 says the cache is already being thrashed by the shards to a 0 % hit rate, with eviction wiping everything.

So "move it to cache" is not automatically safer — it moves a small, well-behaved payload into a store that is currently being overwhelmed by a large, badly-keyed one. The three files here are 0.9 KB, ~16 KB and a depot, and only the depot is large; putting the two small ones in cache is fine on its own terms, but the depot belongs in #72's fix, not in this one.

Revised ordering, then:

1. Guard the -timings upload — unblocks the fleet during any quota event, one or two lines, no dependency on anything else.
2. Coverage per shard straight to Codecov; deletes the merge job, the eight coverage artifacts and -lcov. Resolves Per-shard coverage artifacts ship 8 copies of the source tree — 85.7% of the payload is text collect already has checked out #74 by removing the payload rather than trimming it. This is where 97 % of the bytes are.
3. -out → the job log / step summary.
4. -timings and -records (0.9 KB and 16.5 KB) → actions/cache.
5. -prebuild → hold until The depot cache has a 0% hit rate: N shards save the SAME key, each uploading the whole depot, and the repo quota evicts everything #72, since it is the depot and The depot cache has a 0% hit rate: N shards save the SAME key, each uploading the whole depot, and the repo quota evicts everything #72 says the depot cache is the thing that is broken.

Steps 1–3 need no cache at all and remove the large majority of the problem, which is the argument for doing them first.

[sotashimozono]

Correction to this issue's central measurement. I wrote here that retention does not free the quota:

measured, one created 2026-07-14 with a one-day retention was still present on 2026-08-19, five weeks later, still counting against the quota. retention-days controls when an artifact becomes useless, not when it stops being charged.

That is wrong, and the error is in how the artifacts API was read. An expired artifact keeps its metadata row, including size_in_bytes, after GitHub has deleted the bytes. Measured on lab-sotashimozono/FunctionMeasures.jl, all 6666 records paged:

expired records: 4894    with size_in_bytes > 0: 4894    total 21.3 MB
live    records: 1772                                    total 15.6 MB

Every record I read as "still present, still charged" carries expired: true. Filtering on expired == false across all 25 private repos of the account gives 11,411 live artifacts, 90.7 MB, against a documented 500 MB free-plan pool. Retention is working.

Two consequences for this issue:

1. The share attributed to coverage counters is a share of cumulative-created bytes, not stored bytes. By live bytes -coverage-* is 0 MB — it has a 1-day retention. The families that actually hold the footprint are -out-* (58.7 MB, 7 days) and -records (16.9 MB, 30 days), which together are 84 %.
2. Deletion and retention were the right levers after all. The reason the prune appeared not to work is the 6–12 hour usage recalculation, not a failure to free anything — which is also why the outage outlasted the fix by most of a day. Uploads recovered with no workflow change (run 32232523231, 2026-08-19T08:25Z).

The parts of this issue that stand: the quota is enforced at CreateArtifact before any bytes are sent, so shrinking payloads cannot rescue an in-flight run; pruning cannot restore service inside the recalculation window; and -out is the completeness check's input rather than duplicated log output, as corrected earlier in this thread.

#80 was built on the erroneous half and has been closed with its own refutation. #81 carries the version that follows from the corrected numbers.

[sotashimozono]

Closing: the headline — "the sharded workflow needs no artifacts at all" — does not survive, and the table that motivated it was measured wrong.

The table mixes cumulative-created bytes with stored bytes. An expired artifact keeps its metadata row, size_in_bytes included, after GitHub deletes the bytes; filtering expired == false puts -coverage-* at 0 MB live (1-day retention), not the 211 MB shown here. The families that actually hold the footprint are -out-* (58.7 MB, 7 days) and -records (16.9 MB, 30 days) — 84 % between them. Corrected numbers and the small fix that follows: #81.

The retention claim in this thread was also wrong and is retracted in the comment above: retention works, and the reason the prune appeared not to help is the 6–12 hour usage recalculation.

What survives, and is worth keeping in history:

• the quota is enforced at CreateArtifact before any bytes are sent, so shrinking a payload cannot rescue an in-flight run
• the visible BlobNotFound tail reads like a transient blip and is not; four reruns across four runners were spent on that reading before the real error was found further up the same step
• -out is the completeness check's input, not duplicated log output

What is already fixed: the unguarded -timings upload that gated the matrix — #76 and #77.

The design this issue asked for became #80, which is closed with its own refutation.