feat: the pieces for carrying completeness evidence in the shards' job logs - #84
Closed
sotashimozono wants to merge 1 commit into
Closed
sotashimozono wants to merge 1 commit into
sotashimozono wants to merge 1 commit into
Conversation
Contributor
|
📚 Docs preview: https://codes.sota-shimozono.com/TestShards.jl/previews/PR84/ (updates on each push to this PR) |
…b logs
Adds the transport and leaves it UNREFERENCED. The workflow wiring is a follow-up,
because a `uses: QAtlasHub/TestShards.jl/actions/read-evidence@main` cannot resolve until
the action is on `main` — measured, on the first attempt at this:
Can't find 'action.yml' ... for action
'QAtlasHub/TestShards.jl/actions/read-evidence@main'
`uses:` accepts no expression, so `@${{ github.sha }}` is not a way out, and a relative
path is the trap `actions/read-history` documents (a `run:`/relative `uses:` in a
REUSABLE workflow resolves in the CALLER's checkout). So a new action and its first
reference cannot land together. Splitting also means nothing in this PR can change any
consumer's behaviour: none of it is called yet.
WHY THE TRANSPORT AT ALL. The completeness gate is the one job still failing across the
fleet under a full artifact quota, and it fails CORRECTLY — `units observed = 0`, "no
shard reported how many units it observed — cannot verify the run" — because the
evidence never arrives. Artifact storage is metered in GigabyteHours against a
per-period allowance; once crossed, `CreateArtifact` refuses BEFORE any bytes are sent,
so a 0-byte artifact fails exactly as a large one does, and deleting artifacts does not
un-accrue the period.
`actions/cache` was measured as the replacement and does not work: save succeeds and the
entry lists as active on the right ref, restore misses every key — including a
default-branch entry written by this repo's own CI, and with a 45 s gap to rule out
propagation. Four runs, three controls.
What does work, verified end to end in a live run: the shard PRINTS its evidence and the
collector reads that shard's job log back through the Actions API. No storage, no
residue, and only `actions: read` — which a fork pull request also has.
THE PARSING IS THE WHOLE RISK, SO IT IS IN THE PACKAGE. The runner echoes a step's
script into the log before running it, so the marker appears TWICE: once as source with
`$(...)` unexpanded, once as output. Taking the first match decodes the literal
`$(base64 -w0 ev.tsv)`; that surfaced as `base64: invalid input` by luck rather than as
a wrong answer. `evidence_from_log` discriminates by CONSTRUCTION — base64's alphabet
cannot spell an unexpanded substitution — so no ordering convention is relied on. Three
mutations confirm the tests can go red: widening the payload class, accepting a
duplicate shard, and treating "no evidence anywhere" as success.
Two more corrections the measurement forced, each of which would have shipped broken:
* `gh api .../actions/jobs/{id}/logs` does not work. The endpoint answers 302 to a
blob URL and gh exits 1 with no useful message; `curl -sSL` follows it.
* The API names a reusable workflow's job with the CALLER's prefix — `ci / test /
shard s5`, not `shard s5` — so a name filter would depend on what the caller called
its job, and one that matches nothing reads zero logs and reports a
complete-looking empty run. There is no name filter: the marker discriminates.
`evidence_cli` rebuilds the same `all-shards.tsv` / `all-ran.tsv` the artifact path
produced, against the real formats read off `src/json.jl`, and the test feeds the result
to an UNCHANGED `completeness_cli`. The check itself is not touched by any of this.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
sotashimozono
force-pushed
the
feat/log-evidence-transport
branch
from
August 20, 2026 04:28
040e093 to
262ff7a
Compare
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
Member
Author
|
Closing unmerged — keeping the current behaviour. The measurements are the part worth keeping, and they are recorded in #82:
Nothing here was referenced by the workflow, so closing changes no behaviour. |
sotashimozono
added a commit
that referenced
this pull request
Aug 24, 2026
…ct quota cannot close (#74) (#85) * feat: the completeness evidence gets a second road that a full artifact quota cannot close (#74) `Every unit ran, exactly once` is a CORRECTNESS claim — this file's own comment says only such a claim may fail a suite — and it is the one thing here that cannot be skipped when its input goes missing. A full artifact storage quota took the artifact road out and the gate with it, and a fleet of green suites read red for six days. Cache is the transport that clears every constraint this workflow has: job outputs matrix legs OVERWRITE each other Checks/Actions API need `checks: read` / `actions: read`, and a reusable workflow may not request more than its CALLER granted, so asking makes every consumer fail to START a new action a new action and its first reference cannot land together (#84) cache none of the above, and a quota of its own MEASURED FIRST, on a probe branch, 8 legs and a collector, with a negative control that asks for a key nobody saved: never-saved key hit='' s1..s8 hit='true' files=3 distinct legs recovered: 8/8, 480 bytes THE PATH MUST MATCH THE SAVE'S EXACTLY, and that is not obvious: a cache entry is identified by its key AND by a version derived from the `path` inputs. The probe's first run saved `evidence` and restored into `parts/sN` and read 'Cache saved with key: X' in the leg against 'Cache not found for input keys: X' in the collector, same run, same key. So the collector restores to the shard's own path and moves the files afterwards. The cache road feeds ONLY the gate. The records merge keeps reading `parts` alone: two roads delivering the same `records-*.jsonl` would double every record, while the gate's three lists are `sort -u`'d and are therefore idempotent under duplication. A LADDER rather than a loop, because `uses:` cannot be looped and the shard count is an input. Sixteen covers twice the default; a run with more shards keeps today's behaviour. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: scope the evidence key by CALLER — one run can invoke this workflow twice This repo's own CI does exactly that: `ci` with 8 shards and `solo` with 1. A key of `run_id + sid` alone made solo's collector restore ci's shard s1, and solo's gate reported '3 unit(s) ran twice: 1 on local and s1' — its own evidence is labelled `local`, what it picked up was ci's. The probe could not have found this: it had one caller. The self-test did, because it has two. `artifact-prefix` is the discriminator this workflow already uses for the same reason on the artifact road. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore: version bump for the evidence cache road Additive: the artifact road is unchanged and every new step is `continue-on-error: true`, so the second road cannot fail a suite by existing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #82. Adds the transport and leaves it unreferenced — the workflow wiring is a follow-up. Pairs with #83, which fixes the other failing job.
Why it is split
The first attempt wired it in the same PR and CI answered:
uses:accepts no expression, so@${{ github.sha }}is not a way out, and a relative path is exactly the trapactions/read-historywas extracted to document — a relative reference in a reusable workflow resolves in the caller's checkout, which works in this repo's CI and exits 127 on every consumer. A new action and its first reference cannot land together.The split has a second benefit: nothing here is called yet, so no consumer's behaviour can change.
The problem this exists for
The only failing step in
collect(measured,lab-sotashimozono/ParameterizedITensor.jlrun32238414796), and it fails correctly: thepattern:download succeeds with zero results, the merge succeeds on nothing, and the gate refuses rather than passing blind.Artifact storage is metered in GigabyteHours against a per-period allowance. Once crossed,
CreateArtifactrefuses before any bytes are sent, and deleting artifacts does not un-accrue the period. The transport has to stop being storage.actions/cachewas measured and does not workCache saved with key: …, listed active on the right refFour runs, no error, no diagnostic.
What works
Verified end to end in a live run:
recovered 3 of 3.Actions storageactions: readrun_attempt/attempts/{n}/jobsWhy the parsing is in the package
The repo's own rule — "a merge written as a step script is unreachable from
Pkg.test()" — and it bites hardest here, because every mistake made while measuring this was a parsing mistake:$(...)unexpanded, once as output. First-match decodes the literal$(base64 -w0 ev.tsv). It surfaced asbase64: invalid inputby luck, not as a wrong answer. Fixed by construction — base64's alphabet cannot spell an unexpanded substitution — so no "prefer the last match" convention is relied on.gh api .../jobs/{id}/logsdoes not work. 302 to a blob URL; gh exits 1 with no useful message.curl -sSLfollows it.ci / test / shard s5. A name filter would depend on the caller's job name, and one matching nothing reads zero logs and reports a complete-looking empty run. No name filter; the marker discriminates./runs/{id}/attempts/{n}/jobs, not/runs/{id}/jobs— the latter merges attempts, so a re-run double-counts while looking clean.evidence_scanrefuses a duplicate shard id outright.Verification
test/core/test_evidence.jl— round trip through tabs/newlines/5 KB, the double-marker log verbatim, absent vs corrupt, duplicate shards, andevidence_clireproducing the two files byte for byte against the real formats read offsrc/json.jl, then fed to an unchangedcompleteness_cli.actions/read-evidence/read-evidence.test.sh—TS_GH/TS_CURLare indirected so all six branches run with no network and no token: all readable; one unreadable (warned, not fatal — which shard is missing is the gate's judgement, not the fetcher's); none readable (fatal); empty job list (fatal); failing job-list call (fatal).actions/read-history/read-history.test.shstill passes.One finding outside this PR
format-checkhere resolved JuliaFormatter 2.12.6 fromversion="2". That floating spec is the defectlab-sotashimozono/.githubalready fixed and documented — it pinsJF_VERSION: '2.10.1'exactly, with a comment recording that a range "silently resolved to 2.3.2 while a laptop resolved 2.10.1".QAtlasHub/.github's copy never got that fix, so formatting locally against the sibling org's pin produces a red check here. Worth a separate issue onQAtlasHub/.github.🤖 Generated with Claude Code