Compose with Pinax: a unit-testset provider, so a sharded suite reports margins instead of counts

[sotashimozono]

A sharded suite currently reports counts. Pinax's test bridge reports margins, code and convergence figures — and the two cannot be used together at all. This issue is the design for the extension that composes them, plus the two measured reasons an extension is required rather than convenient.

1. Measured: they are mutually blind today

Pinax.test installs a capturing root testset (PinaxTestSet <: AbstractTestSet), and a suite's nested @testsets are captured because Test builds a nested set from the parent's type. TestShards does not go through @testset: _run constructs Test.DefaultTestSet(key) by hand (deliberately — a root @testset throws before returning its tree, so a failed unit could not be read back).

Consequence, measured on a two-line suite (@shard begin include("unit_a.jl") end, one passing @test isapprox(e, 0.8; rtol=0.01)):

TestShards: 1/1 units ran — 1 pass, 0 fail, 0 error
Pinax test report: 0/0 passed          ← empty, and GREEN

An empty-but-green report is the worst possible failure: it is indistinguishable from a suite with no tests. Two independent causes, both verified:

1. Type. Inside a unit the ambient testset is a DefaultTestSet, so Pinax's one integration seam (Pinax._current_container()) returns :inert — no checks recorded, and @figure / @code / @desc written in a test silently no-op.
2. Nesting. _run does push_testset / pop_testset without Test.finish, and Pinax nests a child into its parent in finish. So even with the right type the unit would not appear:

root = PinaxTestSet("root"); Test.push_testset(root)
u = PinaxTestSet("unit_a.jl"); Test.push_testset(u)
Test.record(Test.get_testset(), <a Pass>)
Test.pop_testset();  length(root.children)   # → 0     ← vanished
Test.finish(u);      length(root.children)   # → 1     ← nested, 1 check, got = 0.8

The reverse direction fails too: _section reads counts by walking ts.results and matching r isa Test.DefaultTestSet. A foreign testset type has neither, so its counts would be silently dropped from UnitRecord — and the balancing history and the completeness verdict are built on those counts.

So this is not "wire two packages together". Either package alone, in the other's presence, loses data quietly.

2. Why not translate the records instead

The cheap alternative is a one-way translation: UnitRecord/Section → Pinax.TestNode (Pinax's closure-free dump DTO), then Pinax.render_test_report. It needs no extension point at all.

It is the wrong design, because Section carries aggregate counts (npass/nfail/nerror/nbroken) and Pinax's entire value is per-assertion: Check(got, want, delta, tol, pass). Aggregates cannot produce a margin, a convergence figure, or the code region behind an assertion. Translating records would yield a report that shows verdicts and calls itself a Pinax report — strictly worse than what Pinax already produces unsharded, and it would look right.

Whatever we build, Pinax's own capture has to see the assertions. That means the unit testset must be Pinax's type, which means TestShards needs an extension point.

3. The extension point (three operations, one hook)

Everything the composition needs is at the boundary of _run. Introduce a unit-testset provider:

# core — the default is exactly today's behaviour
unit_testset(key)      = Test.DefaultTestSet(key)   # open
unit_close(ts)         = nothing                    # after pop_testset
unit_fold(ctx, ts)     = _section(ctx, ts)          # read a Section tree back out

• unit_close and unit_fold dispatch on the testset type, so an extension adds methods and never overwrites one.
• unit_testset is a choice, not a dispatch — there is nothing to dispatch on. Register it the way Pinax registers its own seam: a Ref assigned in the extension's __init__ (a Ref assignment, not a method override, so no precompile-time method overwriting). Registering a second provider is a loud error naming both — two tools cannot both own the type of one testset, and silently letting the last one win is the kind of thing that shows up as a mysteriously empty report.

_run becomes:

ts = unit_testset(key)
Test.push_testset(ts)
try  body()  catch err  Test.record(ts, Test.Error(...))  finally  Test.pop_testset()  end
unit_close(ts)                     # Pinax: Test.finish(ts) → nests into the capturing root
sec = unit_fold(ctx, ts)           # Pinax: counts from its own folded tree

Four lines changed, and with no extension loaded the bytes executed are the same as today.

ext/TestShardsPinaxExt.jl

[weakdeps]   Pinax = "e782a80a-…"
[extensions] TestShardsPinaxExt = "Pinax"

Test is already a hard dependency of TestShards, so loading Pinax alongside it also loads PinaxTestExt — PinaxTestSet is always available inside this extension.

• unit_testset: return PinaxTestSet(key) only when a Pinax capture is ambient —
  Test.get_testset_depth() > 0 && Test.get_testset() isa PinaxTestSet. No environment sniffing, and this is the crucial guard: a package that merely depends on Pinax must not have its testset type changed. Under a bare Pkg.test() the check is false and the default stands (Pinax's own invariant V: the report changes nothing about how the suite runs).
• unit_close(ts::PinaxTestSet): Test.finish(ts). Safe at depth > 0 — Pinax's finish only throws at the root, and TestShards' own end-of-block failure re-signal is unchanged.
• unit_fold(ctx, ts::PinaxTestSet): build the Section tree from Pinax's tree. Pinax exposes _ntests / _nfail / _nerror / _nbroken duck-typed over it, and PinaxTestSet.elapsed carries the duration, so the balancing history and the completeness verdict are unaffected — the same numbers by a different route. That is the property to test first.

evidence! needs no change: it keys a Dict on the testset object, and a PinaxTestSet is mutable, so it works with any type.

4. What the composition buys

A shard's Pinax root dumps instead of rendering (PINAX_TEST_DUMP), each shard uploads its dump, and one merge job renders all shards as one document: one page per unit, sections mirroring the @testset nesting, per-check margins, the code behind each check, and @testset for sweeps folded into convergence figures. The shard boundary does not appear in the output.

This needs Pinax ≥ 0.39.16 (QAtlasHub/Pinax.jl#90): before it, a dump carried checks and codes only, so a sharded report silently lost every @desc / @table / @raw and re-ordered what survived. Sharded and unsharded are now the same document except for a captured @figure, whose generator is a closure and cannot be serialized (that case warns loudly).

CI shape — it fits the existing reusable workflow, which already has small merge jobs for timings, completeness and coverage:

shard s1..sN   TESTSHARDS_ID / PINAX_TEST_DUMP=pinax-dumps/s1.toml   → artifact
merge          Pinax.render_test_report(readdir("pinax-dumps"); out="test-report")

A new pinax: true workflow input, off by default.

5. The claim worth making

Neither package can state this alone: a sharded test report that can prove it is complete. TestShards observes the whole unit sequence in every shard, so it knows exactly which units should exist (that is what completeness already checks); Pinax holds the merged document. Together the report can carry the verdict "N units, all accounted for" as part of the artifact rather than as a line in an expired CI log — a sharded report that is missing a shard currently looks like a smaller suite.

That needs a small Pinax API (a panel injected into the overview page — render_test_report(…; overview_panels)), so it is phase 3, not phase 1.

6. Phases

1. The extension point + TestShardsPinaxExt (§3). Test first that UnitRecord counts and durations are identical through both folds, on the same suite — that is the regression that would silently corrupt balancing. Then: a sharded run of a real suite renders one document with margins.
2. evidence! → Pinax content. evidence!(; tolerance, achieved, oracle) is already "what this test established"; in a Pinax report it should appear, as a table beside the check. Content, not structure — no test file changes.
3. Completeness as a first-class panel (§5), gated on the small Pinax API.

Not in scope: rendering from records (§2), and parallelism inside a job (ParallelTestRunner.jl / ReTestItems.jl — they compose with this and are unaffected).

[sotashimozono]

Phase 1 is implemented in #26. Two refinements to §3 came out of writing it, both worth recording:

The three operations are one registered provider, not three dispatched functions. §3 proposed unit_close(ts) / unit_fold(ctx, ts) as methods dispatching on the testset type, with only unit_testset behind a Ref. But fold has to return a Section, and a provider living in another package would then have to name this package's types. The provider is instead a single registration with open / close / fold, where fold returns plain data — (; name, duration, npass, nfail, nerror, nbroken, sections), with omitted fields defaulting to zero — and this package converts. The consumer names nothing of ours but the registration call.

open returning nothing is the guard, not a separate predicate. §3 put the "is a Pinax capture ambient?" check inside the extension's unit_testset. Making declining part of the protocol is better: the provider decides per unit, so the same mechanism covers "the tool is installed but not capturing" for every future consumer, and the core needs no notion of what "active" means.

Both are in the docs (docs/src/composing.md).

The measured facts in §1 are unchanged and are now regression tests. One more, found while implementing: PinaxTestSet lives in Pinax's own Test-triggered extension, and two extensions have no defined load order between them, so its type has to be resolved on first use rather than at __init__.

Phases 2 and 3 stay open here. Phase 3's Pinax-side API (a panel injected into the overview page) is still unwritten.

[sotashimozono]

#26 is green, including the pinax job. Three packaging constraints came out of it that §3 did not anticipate, and they decide where the consumer can live — worth having written down before phases 2 and 3:

1. [extras] Pinax breaks Julia 1.10. Pkg.test fails outright with expected package Pinax [e782a80a] to be registered. [sources], the mechanism for locating an unregistered package, arrived in 1.11; this package supports 1.10, which is the current LTS. Dropping 1.10 for an optional extension is the wrong trade.
2. A [sources] entry must name a package in deps/extras. So "keep the source, drop the extra" is not a way out — Pkg.develop on this package then fails from any other environment with Sources for Pinax not listed in deps or extras.
3. A bare weakdep costs nothing. Verified on 1.10 (Pkg.resolve + using TestShards clean, the extension simply absent) and from a consumer environment without Pinax (full suite green).

Hence: Pinax is declared only as a weakdep, and the composition is exercised by a dedicated CI job that builds its own environment. That job is the only place the shipped extension runs.

It also settles a question §3 left implicit — which package should host the extension. Whoever hosts it needs the other to be registered, so it is symmetric today and neither can register while it hosts. TestShards hosting is the better end: AutoRegister.yml here is deliberately disabled anyway, whereas Pinax is actively registering, so a weakdep on an unregistered package would break it now rather than later.

One more, from CI rather than design: an assertion of the form !occursin("shard", md) for "the split is invisible" is a false positive in this repository — its own name contains the word, and the report's provenance table carries the repository name. It passed locally, where provenance has no repo row, and failed in CI. The assertion is now on the ids (one page per unit) and on the merged overview counting every check from both shards, which is the actual claim.

[sotashimozono]

Phase 3's Pinax half is done: QAtlasHub/Pinax.jl#93 (green, independent of the provider PRs).

render_test_report(…; overview) takes a zero-argument function and runs it with the overview page open, so a caller writes @desc / @table / @raw and the content lands there — in every backend, so a verdict is data in agent.json rather than only pixels. A throwing hook is a diagnostic and the report still renders.

That is deliberately not a "completeness panel": it is the general seam, and completeness is one caller. §5 here asked for a panel injected into the overview page; a panel-list keyword would have been a parallel description of content Pinax already has macros for, and raw HTML alone would vanish from the LaTeX and agent backends.

What remains on this side is the consumer:

Pinax.render_test_report(dumps; out = "test-report", overview = () -> begin
    @desc md"**Every unit ran exactly once** — 24 observed, 24 run."
    @table (; metric = ["units observed", "units run"], value = [24, 24]) caption = "Completeness"
end)

completeness already computes exactly those numbers and that verdict, so the work is wiring, not logic. The open question is where the wiring lives: a merge step in sharded-tests.yml behind a pinax: true input would put the call in this repository's workflow. That is allowed under the constraint that decided the extension's home — a workflow input is not a package dependency, so it does not touch Project.toml or the registration chain — but it does mean this repository's workflow knows about Pinax. The alternative is documenting the snippet and leaving the merge step to the consumer's own CI.

I would rather not decide that while three PRs are in flight (#26 here, Pinax #91 and #93). Order: #26 → #91 (re-run its TestShards provider job) → #93, then the wiring.

[sotashimozono]

Phases 2 and 3 are done.

phase 2 — evidence! as content	#34 here (TestShards.evidence(ts)) + QAtlasHub/Pinax.jl#94
phase 3 — the completeness verdict in the artifact	Pinax#93 (the hook) + Pinax#95 (completeness_overview)

evidence! now shows up as a table on the section that recorded it, beside the check and its code — native rows in agent.json, which is the binding for reconciling a claim against what was actually compared. It needed one addition here: the dictionary was only reachable through CURRENT[].evidence, so evidence(ts) is the reader, and neither side names the other's internals.

Completeness rides in the report:

c = TestShards.completeness(windows, ran)
Pinax.render_test_report(dumps; out = "test-report", overview = Pinax.completeness_overview(c))

A whole run says so; a holed one says **Completeness FAILED.** 1 unit never ran: position 2. and carries the numbers as native rows. §5's claim — a sharded test report that can prove it is complete — is now something a run demonstrates rather than something the design argues.

What is deliberately still open: where the merge step lives. Nothing calls the two lines above yet. The options are unchanged — a merge job in sharded-tests.yml behind a pinax: true input, or documenting the snippet and leaving it to the consumer's CI. I did not take it, for two reasons worth recording:

1. Enabling it in the reusable workflow is not one job. Each shard would have to run the suite under a capture with PINAX_TEST_DUMP set, upload dumps, and then a merge job renders — which means changing how the shard step invokes the suite, and that step is where the coverage work also lives (Convert coverage once, instead of once per shard #33 is in flight there). Colliding with that for a feature nobody has asked for yet is the wrong trade.
2. The packaged form makes the wiring two lines wherever it ends up, so nothing is blocked by leaving it undecided — which was not true before #95.

My inclination is the workflow input, once #33 has settled, because #13 is about the fleet adopting this and re-deriving the YAML per repository is exactly the friction a reusable workflow exists to remove.

[sotashimozono]

The workflow-input option costs coverage, and that decides it

Picking this back up now that #33 has settled, which was the stated precondition. The conclusion
goes the other way from the inclination recorded above, for a reason that is checkable rather
than a preference.

Pinax.test() is not a replacement for Pkg.test — it delegates to an unmodified one, injecting
a -L preamble that installs a capturing root, and drives the rest through PINAX_TEST_OUT /
PINAX_TEST_DUMP / PINAX_TEST_TITLE. That is much lighter than "changing how the shard step
invokes the suite" suggested. So far so good.

The problem is the seam. julia-actions/julia-runtest has no input that injects -L. Its
inputs are check_bounds, coverage, depwarn, force_latest_compatible_version, inline,
prefix, project, annotate, compiled_modules, allow_reresolve, test_args, optimize —
and prefix goes in front of the julia command (it is for xvfb-run), not into its flags.

So a pinax: true input would have to replace julia-runtest with a hand-rolled Pinax.test()
call. And Pinax.test() calls Pkg.test(; julia_args=-L $preamble) — no coverage
argument. The shard would therefore stop producing .cov counters, which is the entire input
to the collect job #33 just built, plus it would drop the check_bounds / inline / depwarn
settings the action supplies. Trading a shipped feature for a new one is the wrong trade, and
re-implementing an ecosystem action to get there is the wrong shape besides.

Worth noting the same trap sits in our own docs already: composing.md shows the sharded flow as

PINAX_TEST_DUMP=pinax-dumps/s1.toml julia -e 'using Pinax, Test; Pinax.test("test/runtests.jl")'

which is the test(runtests::AbstractString) form — documented as "in the current process (no
sandbox)". A consumer copying it gets no Pkg.test sandbox, no test-env resolution and no
coverage. That is a doc bug independently of what we decide here.

There is a third option, and it dissolves the conflict

The -L preamble is not load-bearing. Pinax._install_test_capture!() reads its whole
configuration from the environment, pushes a root PinaxTestSet, and registers an atexit hook
that dumps or renders and sets the exit code. Nothing in it requires being reached through -L;
that route exists only so Pinax.test() needs no edit to runtests.jl.

A TestShards consumer already edits runtests.jl — wrapping the includes in
TestShards.@shard is the whole adoption story. So the capture can be installed there instead:

using MyPackage, TestShards, Pinax
Pinax.install_test_capture!()          # no-op unless PINAX_TEST_DUMP / _OUT is set
TestShards.@shard begin
    include("core/a.jl")
end

Then everything composes and nothing is given up:

• the shard keeps running julia-actions/julia-runtest, so coverage, sandbox, check_bounds
  and the rest are untouched;
• per-shard dumping is one environment variable, PINAX_TEST_DUMP, set the way TESTSHARDS_OUT
  already is — absolute, into the workspace, so it survives sandbox teardown;
• the reusable workflow's share shrinks to what a reusable workflow is actually good at: upload
  the dumps, and one job that renders them.

This needs one thing on the Pinax side: a public name for the install, since
_install_test_capture! is private and documented advice should not reach into it.

What I would do

Not the pinax: true input as scoped. Instead: make the install public in Pinax, correct
composing.md to the sandboxed form, and then decide whether the upload+render pair is worth an
input — it is a much smaller question once the invocation is no longer at stake.

Not implemented; this is the design note. Happy to be overruled on the third option — it does put
one more line in the consumer's runtests.jl, which is exactly what Pinax.test() was built to
avoid.