Skip to content

feat: the pieces for carrying completeness evidence in the shards' job logs - #84

Closed
sotashimozono wants to merge 1 commit into
mainfrom
feat/log-evidence-transport
Closed

sotashimozono wants to merge 1 commit into
mainfrom
feat/log-evidence-transport

Conversation

@sotashimozono

@sotashimozono sotashimozono commented Aug 20, 2026

Copy link
Copy Markdown
Member

Part of #82. Adds the transport and leaves it unreferenced — the workflow wiring is a follow-up. Pairs with #83, which fixes the other failing job.

Why it is split

The first attempt wired it in the same PR and CI answered:

Can't find 'action.yml' ... for action 'QAtlasHub/TestShards.jl/actions/read-evidence@main'

uses: accepts no expression, so @${{ github.sha }} is not a way out, and a relative path is exactly the trap actions/read-history was extracted to document — a relative reference in a reusable workflow resolves in the caller's checkout, which works in this repo's CI and exits 127 on every consumer. A new action and its first reference cannot land together.

The split has a second benefit: nothing here is called yet, so no consumer's behaviour can change.

The problem this exists for

Every unit ran, exactly once   FAILURE
| units observed | 0 |
TestShards: no shard reported how many units it observed — cannot verify the run.

The only failing step in collect (measured, lab-sotashimozono/ParameterizedITensor.jl run 32238414796), and it fails correctly: the pattern: download succeeds with zero results, the merge succeeds on nothing, and the gate refuses rather than passing blind.

Artifact storage is metered in GigabyteHours against a per-period allowance. Once crossed, CreateArtifact refuses before any bytes are sent, and deleting artifacts does not un-accrue the period. The transport has to stop being storage.

actions/cache was measured and does not work

control result
save in a matrix leg succeedsCache saved with key: …, listed active on the right ref
restore by exact key, same run miss
…with a 45 s gap miss — not propagation
…a key from a previous run, confirmed present via the API miss — not a same-run race
…a default-branch key written by this repo's own CI miss — not a ref-scope rule
a key never written miss (correct)

Four runs, no error, no diagnostic.

What works

Verified end to end in a live run: recovered 3 of 3.

artifacts cache job logs
Actions storage metered none none
works here yes no yes
permission runtime token runtime token actions: read
fork PR yes untested yes
residue retention LRU none
re-run isolation run-scoped (mixes attempts) key needs run_attempt /attempts/{n}/jobs

Why the parsing is in the package

The repo's own rule — "a merge written as a step script is unreachable from Pkg.test()" — and it bites hardest here, because every mistake made while measuring this was a parsing mistake:

  1. The runner echoes the step script into the log, so the marker appears twice: once as source with $(...) unexpanded, once as output. First-match decodes the literal $(base64 -w0 ev.tsv). It surfaced as base64: invalid input by luck, not as a wrong answer. Fixed by construction — base64's alphabet cannot spell an unexpanded substitution — so no "prefer the last match" convention is relied on.
  2. gh api .../jobs/{id}/logs does not work. 302 to a blob URL; gh exits 1 with no useful message. curl -sSL follows it.
  3. The API names a reusable workflow's job with the caller's prefixci / test / shard s5. A name filter would depend on the caller's job name, and one matching nothing reads zero logs and reports a complete-looking empty run. No name filter; the marker discriminates.
  4. /runs/{id}/attempts/{n}/jobs, not /runs/{id}/jobs — the latter merges attempts, so a re-run double-counts while looking clean. evidence_scan refuses a duplicate shard id outright.

Verification

  • test/core/test_evidence.jl — round trip through tabs/newlines/5 KB, the double-marker log verbatim, absent vs corrupt, duplicate shards, and evidence_cli reproducing the two files byte for byte against the real formats read off src/json.jl, then fed to an unchanged completeness_cli.
  • Mutation-tested, because a green test proves nothing until it can go red: widening the payload character class (the bug actually shipped in the probe), accepting a duplicate shard, treating "no evidence anywhere" as success. All three go red; restoring goes green.
  • actions/read-evidence/read-evidence.test.shTS_GH/TS_CURL are indirected so all six branches run with no network and no token: all readable; one unreadable (warned, not fatal — which shard is missing is the gate's judgement, not the fetcher's); none readable (fatal); empty job list (fatal); failing job-list call (fatal).
  • actions/read-history/read-history.test.sh still passes.

One finding outside this PR

format-check here resolved JuliaFormatter 2.12.6 from version="2". That floating spec is the defect lab-sotashimozono/.github already fixed and documented — it pins JF_VERSION: '2.10.1' exactly, with a comment recording that a range "silently resolved to 2.3.2 while a laptop resolved 2.10.1". QAtlasHub/.github's copy never got that fix, so formatting locally against the sibling org's pin produces a red check here. Worth a separate issue on QAtlasHub/.github.

🤖 Generated with Claude Code

@github-actions github-actions Bot added the enhancement New feature or request label Aug 20, 2026
@github-actions

Copy link
Copy Markdown
Contributor

📚 Docs preview: https://codes.sota-shimozono.com/TestShards.jl/previews/PR84/

(updates on each push to this PR)

…b logs

Adds the transport and leaves it UNREFERENCED. The workflow wiring is a follow-up,
because a `uses: QAtlasHub/TestShards.jl/actions/read-evidence@main` cannot resolve until
the action is on `main` — measured, on the first attempt at this:

    Can't find 'action.yml' ... for action
    'QAtlasHub/TestShards.jl/actions/read-evidence@main'

`uses:` accepts no expression, so `@${{ github.sha }}` is not a way out, and a relative
path is the trap `actions/read-history` documents (a `run:`/relative `uses:` in a
REUSABLE workflow resolves in the CALLER's checkout). So a new action and its first
reference cannot land together. Splitting also means nothing in this PR can change any
consumer's behaviour: none of it is called yet.

WHY THE TRANSPORT AT ALL. The completeness gate is the one job still failing across the
fleet under a full artifact quota, and it fails CORRECTLY — `units observed = 0`, "no
shard reported how many units it observed — cannot verify the run" — because the
evidence never arrives. Artifact storage is metered in GigabyteHours against a
per-period allowance; once crossed, `CreateArtifact` refuses BEFORE any bytes are sent,
so a 0-byte artifact fails exactly as a large one does, and deleting artifacts does not
un-accrue the period.

`actions/cache` was measured as the replacement and does not work: save succeeds and the
entry lists as active on the right ref, restore misses every key — including a
default-branch entry written by this repo's own CI, and with a 45 s gap to rule out
propagation. Four runs, three controls.

What does work, verified end to end in a live run: the shard PRINTS its evidence and the
collector reads that shard's job log back through the Actions API. No storage, no
residue, and only `actions: read` — which a fork pull request also has.

THE PARSING IS THE WHOLE RISK, SO IT IS IN THE PACKAGE. The runner echoes a step's
script into the log before running it, so the marker appears TWICE: once as source with
`$(...)` unexpanded, once as output. Taking the first match decodes the literal
`$(base64 -w0 ev.tsv)`; that surfaced as `base64: invalid input` by luck rather than as
a wrong answer. `evidence_from_log` discriminates by CONSTRUCTION — base64's alphabet
cannot spell an unexpanded substitution — so no ordering convention is relied on. Three
mutations confirm the tests can go red: widening the payload class, accepting a
duplicate shard, and treating "no evidence anywhere" as success.

Two more corrections the measurement forced, each of which would have shipped broken:

  * `gh api .../actions/jobs/{id}/logs` does not work. The endpoint answers 302 to a
    blob URL and gh exits 1 with no useful message; `curl -sSL` follows it.
  * The API names a reusable workflow's job with the CALLER's prefix — `ci / test /
    shard s5`, not `shard s5` — so a name filter would depend on what the caller called
    its job, and one that matches nothing reads zero logs and reports a
    complete-looking empty run. There is no name filter: the marker discriminates.

`evidence_cli` rebuilds the same `all-shards.tsv` / `all-ran.tsv` the artifact path
produced, against the real formats read off `src/json.jl`, and the test feeds the result
to an UNCHANGED `completeness_cli`. The check itself is not touched by any of this.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@sotashimozono
sotashimozono force-pushed the feat/log-evidence-transport branch from 040e093 to 262ff7a Compare August 20, 2026 04:28
@sotashimozono sotashimozono changed the title feat: carry the completeness evidence in the shards' job logs, not in artifacts feat: the pieces for carrying completeness evidence in the shards' job logs Aug 20, 2026
@codecov

codecov Bot commented Aug 20, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 0% with 94 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/evidence.jl 0.00% 94 Missing ⚠️

📢 Thoughts on this report? Let us know!

@sotashimozono

Copy link
Copy Markdown
Member Author

Closing unmerged — keeping the current behaviour.

The measurements are the part worth keeping, and they are recorded in #82:

  • actions/cache does not work here. Save succeeds and the entry lists as active on the right ref; restore misses every key — including a default-branch entry written by this repo's own CI, with a 45 s gap to rule out propagation. Four runs, three controls.
  • The log channel does work, verified end to end in a live run: recovered 3 of 3, no storage, no residue, actions: read only.
  • A new action and its first reference cannot land in one PR. uses: …@main cannot resolve before the action is on main, uses: takes no expression, and a relative reference in a reusable workflow resolves in the caller's checkout.
  • Three parsing traps, each of which would have shipped broken: the runner echoes the step script so the marker appears twice; gh api …/jobs/{id}/logs answers 302 and gh does not follow it; the API names a reusable workflow's job with the caller's prefix (ci / test / shard s5), so any name filter depends on what the caller called its job.

Nothing here was referenced by the workflow, so closing changes no behaviour.

@sotashimozono
sotashimozono deleted the feat/log-evidence-transport branch August 20, 2026 04:41
sotashimozono added a commit that referenced this pull request Aug 24, 2026
…ct quota cannot close (#74) (#85)

* feat: the completeness evidence gets a second road that a full artifact quota cannot close (#74)

`Every unit ran, exactly once` is a CORRECTNESS claim — this file's own comment
says only such a claim may fail a suite — and it is the one thing here that
cannot be skipped when its input goes missing. A full artifact storage quota
took the artifact road out and the gate with it, and a fleet of green suites
read red for six days.

Cache is the transport that clears every constraint this workflow has:

  job outputs        matrix legs OVERWRITE each other
  Checks/Actions API need `checks: read` / `actions: read`, and a reusable
                     workflow may not request more than its CALLER granted, so
                     asking makes every consumer fail to START
  a new action       a new action and its first reference cannot land together (#84)
  cache              none of the above, and a quota of its own

MEASURED FIRST, on a probe branch, 8 legs and a collector, with a negative
control that asks for a key nobody saved:

  never-saved key   hit=''
  s1..s8            hit='true'  files=3
  distinct legs recovered: 8/8, 480 bytes

THE PATH MUST MATCH THE SAVE'S EXACTLY, and that is not obvious: a cache entry
is identified by its key AND by a version derived from the `path` inputs. The
probe's first run saved `evidence` and restored into `parts/sN` and read
'Cache saved with key: X' in the leg against 'Cache not found for input keys: X'
in the collector, same run, same key. So the collector restores to the shard's
own path and moves the files afterwards.

The cache road feeds ONLY the gate. The records merge keeps reading `parts`
alone: two roads delivering the same `records-*.jsonl` would double every
record, while the gate's three lists are `sort -u`'d and are therefore
idempotent under duplication.

A LADDER rather than a loop, because `uses:` cannot be looped and the shard
count is an input. Sixteen covers twice the default; a run with more shards
keeps today's behaviour.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: scope the evidence key by CALLER — one run can invoke this workflow twice

This repo's own CI does exactly that: `ci` with 8 shards and `solo` with 1. A
key of `run_id + sid` alone made solo's collector restore ci's shard s1, and
solo's gate reported '3 unit(s) ran twice: 1 on local and s1' — its own evidence
is labelled `local`, what it picked up was ci's.

The probe could not have found this: it had one caller. The self-test did,
because it has two.

`artifact-prefix` is the discriminator this workflow already uses for the same
reason on the artifact road.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* chore: version bump for the evidence cache road

Additive: the artifact road is unchanged and every new step is
`continue-on-error: true`, so the second road cannot fail a suite by existing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant