From 3329b641c2302dfce726c933abbd3ca938bd171e Mon Sep 17 00:00:00 2001 From: sotashimozono Date: Thu, 10 Sep 2026 12:47:49 +0000 Subject: [PATCH 1/5] Julia 1.13 turned the testset stack into a ScopedValue MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Every repo calling `sharded-tests.yml@main` is red on the `ci` leg since the runner's "latest Julia 1" moved 1.12.7 → 1.13.0: ERROR: LoadError: UndefVarError: `push_testset` not defined in `Test` Not a rename. Through 1.12 the current testset was a STACK in task-local storage, entered with `Test.push_testset` and left with `Test.pop_testset`. In 1.13 it is `Test.CURRENT_TESTSET`, a `ScopedValue` — which cannot be pushed or popped at all, so both functions are gone. `Test.@testset` itself switched to `@with` for the same reason. The `ci-lts` leg stayed green because it pins 1.10. `_run` now enters through `_with_testset(f, ts)`, one implementation per side of that change. Nothing else moves. Measured rather than assumed, three things: - The reason `_run` hand-rolls this at all — a top-level `@testset` throws instead of returning, so a FAILED unit's tree cannot be read back — is still true on 1.13. So `@testset` is not an escape from the internals here. - `Base.catch_stack()` is not part of the breakage: on 1.13 it returns the same `Base.ExceptionStack` as `current_exceptions()`, `==` and all. Left alone. - Negative control: unfixed `main` on 1.13 fails with exactly the CI error; fixed, the suite passes on BOTH 1.10 and 1.13, so the new branch did not buy 1.13 by breaking the old one. Stated in the comment rather than hidden: this swaps one Test internal for another. Neither `push_testset` nor `CURRENT_TESTSET` is public API, so a later reshuffle breaks this again — and the reason we cannot use the public route is written next to it. Co-Authored-By: Claude Opus 5 (1M context) --- Project.toml | 2 +- src/run.jl | 66 ++++++++++++++++++++++++++++++++++++++-------------- 2 files changed, 50 insertions(+), 18 deletions(-) diff --git a/Project.toml b/Project.toml index cc10b2e..0a1f4d9 100644 --- a/Project.toml +++ b/Project.toml @@ -1,6 +1,6 @@ name = "TestShards" uuid = "acceef1d-f5e0-4fe4-a546-818dc56ce7b2" -version = "0.3.40" +version = "0.3.41" authors = ["sota shimozono "] [deps] diff --git a/src/run.jl b/src/run.jl index 3c51cee..db38155 100644 --- a/src/run.jl +++ b/src/run.jl @@ -2,6 +2,37 @@ # Running a unit # ───────────────────────────────────────────────────────────────────────────────────── +# Run `f` with `ts` as the current testset, and leave the previous one current afterwards. +# +# Two implementations because Julia 1.13 changed what "current testset" IS. Through 1.12 it was +# a STACK in task-local storage, entered with `Test.push_testset` and left with +# `Test.pop_testset`. In 1.13 it became a `ScopedValue` (`Test.CURRENT_TESTSET`), which cannot +# be pushed and popped at all — a scoped value is entered, and `push_testset`/`pop_testset` no +# longer exist. `Test.@testset` itself switched to `@with` for the same reason. +# +# Both spellings reach into Test's internals, which is what breaks: neither `push_testset` nor +# `CURRENT_TESTSET` is public API. The alternative — using `@testset` and reading its return — +# is not available, because a top-level `@testset` throws instead of returning when the unit +# fails, which is the whole reason this function exists. +@static if VERSION >= v"1.13" + function _with_testset(f, ts) + return Base.ScopedValues.with( + f, + Test.CURRENT_TESTSET => ts, + Test.TESTSET_DEPTH => Test.get_testset_depth() + 1, + ) + end +else + function _with_testset(f, ts) + Test.push_testset(ts) + try + return f() + finally + Test.pop_testset() + end + end +end + function _key(ctx::ShardContext, path::AbstractString) return replace(relpath(abspath(path), ctx.root), '\\' => '/') end @@ -12,9 +43,10 @@ end Observe a unit, and run `body` if this shard owns it. The observation counter advances either way — that is what keeps `index`, and the round-robin fallback, identical across shards. -The testset is pushed and popped by hand rather than via `@testset` so that the tree can be -read back even when the unit failed: a top-level `@testset` throws before returning its result. -Failure is re-signalled once, at the end of the whole block. +The testset is entered by hand rather than via `@testset` so that the tree can be read back +even when the unit failed: a top-level `@testset` throws before returning its result (still +true on 1.13 — measured, not assumed). Failure is re-signalled once, at the end of the whole +block. """ function _run(ctx::ShardContext, key::AbstractString, body) ctx.seen += 1 @@ -23,21 +55,21 @@ function _run(ctx::ShardContext, key::AbstractString, body) push!(ctx.ran, (index, String(key))) ts = _unit_testset(key) - Test.push_testset(ts) t0 = time() - try - body() - catch err - # An error escaping the unit (a load error, say) is recorded as the unit's error rather - # than aborting the shard, so the remaining units still run and still get recorded. - Test.record( - ts, - Test.Error( - :nontest_error, Expr(:tuple), err, Base.catch_stack(), LineNumberNode(0) - ), - ) - finally - Test.pop_testset() + _with_testset(ts) do + try + body() + catch err + # An error escaping the unit (a load error, say) is recorded as the unit's error + # rather than aborting the shard, so the remaining units still run and still get + # recorded. + Test.record( + ts, + Test.Error( + :nontest_error, Expr(:tuple), err, Base.catch_stack(), LineNumberNode(0) + ), + ) + end end # A tool that attaches a finished testset to its parent can only do it now that the parent is # current again — which is the whole reason a hand-popped testset needs this second step. From 1c946484919ed8b487cc69bca165e1b928a4f909 Mon Sep 17 00:00:00 2001 From: sotashimozono Date: Thu, 10 Sep 2026 13:29:48 +0000 Subject: [PATCH 2/5] Review fixes: pin both branches in CI, and say what the depth line and the scope actually do MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The depth line was the thin part. `TESTSET_DEPTH => get_testset_depth() + 1` is new hand-maintained state with no analogue on the pre-1.13 branch, and two separate things read it, both failing quietly at 0: - `Test.finish` decides top-level-ness from it. A failing nested `@testset` inside a unit would then throw instead of recording, `_run` would catch that as a `:nontest_error`, and a FAIL would be reported as an ERROR. - stdlib's `@testset` infers an untyped nested set's type as `get_testset_depth() == 0 ? DefaultTestSet : typeof(get_testset())`, so a 0 drops a registered provider's type — the case `provider.jl`'s own header describes as recording nothing at all, silently. Neither was covered. The passing nested case already is (`test_records.jl` and `test_diagnose.jl` assert "outer"/"inner" appear), the failing one was not, and the only "a unit fails" test uses a bare `@test false`, which never reaches the depth logic. Added it, and mutation-checked it rather than trusting it: dropping the depth binding turns the suite into "0 failed and 3 errored across 23 units" — the predicted signature, not just some red. CI could not have kept either branch honest. Only one side of the `@static if` compiles per run; the sharded jobs take whatever `'1'` resolves to, which moves with Julia's release calendar, and `compat` pinned only the 1.10 floor. It now pins 1.13 as well, so both arms stay deliberately under test after `'1'` moves on. The scope also propagates differently, and this is the one real behaviour change: a `ScopedValue` is inherited by tasks spawned inside it where task-local storage was not. Measured both ways. For a unit that joins what it spawns, 1.13 is strictly better — results that used to vanish into the fallback testset now land correctly, even when the old code `wait`ed. For a unit that does not join, a deterministic drop becomes a race against `unit_fold`, which for a tool whose job is counting results is the worse of the two. `@shard`'s docstring now states the requirement. Also: `run.jl` described the same mechanism two ways three lines apart — the docstring said "entered by hand" and the comment below still said "hand-popped". `provider.jl` said "after the testset is popped" twice. Timing was right in all three; the word was left over from the stack. `Base.catch_stack` gets an `XXX` rather than a change: measured on 1.13 it returns the same `Base.ExceptionStack` as `current_exceptions()`, and upstream's own `@testset` has moved to the public name. Not in this commit, deliberately: `Test.record` is unguarded against a provider whose testset type has no `Error` method (pre-existing path, this commit only moved it); `provider.jl` reads `ts.results` without the `results_lock` 1.13 added to `DefaultTestSet`; and a late failure from an unjoined task can segfault Julia 1.13's JIT — reproduced with bare `Test` and `ScopedValues`, no TestShards involved, so it is an upstream report, not a fix here. Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/CI.yml | 6 +++++- src/provider.jl | 8 ++++---- src/run.jl | 28 +++++++++++++++++++++++++++- test/core/test_failure.jl | 28 ++++++++++++++++++++++++++++ 4 files changed, 64 insertions(+), 6 deletions(-) diff --git a/.github/workflows/CI.yml b/.github/workflows/CI.yml index 8bf49e5..b885ebd 100644 --- a/.github/workflows/CI.yml +++ b/.github/workflows/CI.yml @@ -130,7 +130,11 @@ jobs: strategy: fail-fast: false matrix: - julia: ['1.10'] + # The floor, and the version at which `Test`'s testset stack became a `ScopedValue`. + # `_with_testset` has one implementation per side of that boundary and only ONE is + # compiled per run, so pinning both is the only way each stays under test — the sharded + # jobs above take whatever `'1'` resolves to, which moves with Julia's release calendar. + julia: ['1.10', '1.13'] steps: - uses: actions/checkout@v7 - uses: julia-actions/setup-julia@v3 diff --git a/src/provider.jl b/src/provider.jl index 71f5e4a..2e4e109 100644 --- a/src/provider.jl +++ b/src/provider.jl @@ -13,8 +13,8 @@ # So the type is a registered choice. A provider supplies three operations: # # open(key) -> an AbstractTestSet, or `nothing` to decline this unit (use the default) -# close(ts) -> nothing; run after the testset is popped, for a tool that attaches a finished -# set to its parent there (`Test.finish` does exactly that) +# close(ts) -> nothing; run after the testset stops being current, for a tool that attaches +# a finished set to its parent there (`Test.finish` does exactly that) # fold(ts) -> the counts + structure, as PLAIN DATA (see `unit_fold`) # # `open` returning `nothing` is what keeps this inert: a provider decides per unit whether its @@ -36,8 +36,8 @@ rather than a method override, so nothing is overwritten at precompile time. - `open(key::String)` returns the `AbstractTestSet` for a unit, or **`nothing`** to decline it and leave the default in place. Decline unless the tool's capture is actually running: a suite that merely depends on the tool must not have its testset type changed underneath it. - - `close(ts)` runs after the testset is popped. A tool that attaches a finished testset to its - parent does it here (`Test.finish`), which is the only moment at which it can. + - `close(ts)` runs after the testset stops being current. A tool that attaches a finished + testset to its parent does it here (`Test.finish`), which is the only moment at which it can. - `fold(ts)` returns the counts and structure as plain data — see [`unit_fold`](@ref). This is what keeps the balancing history and the completeness verdict correct when the testset is not ours, and it is the first thing to test: the same suite must yield the same numbers whichever diff --git a/src/run.jl b/src/run.jl index db38155..553b885 100644 --- a/src/run.jl +++ b/src/run.jl @@ -14,6 +14,20 @@ # `CURRENT_TESTSET` is public API. The alternative — using `@testset` and reading its return — # is not available, because a top-level `@testset` throws instead of returning when the unit # fails, which is the whole reason this function exists. +# +# `TESTSET_DEPTH` moves with `CURRENT_TESTSET` because two things read it, and both fail +# QUIETLY if it says 0. `Test.finish` uses it to decide whether a testset is top-level — a +# top-level one THROWS instead of recording, so a failing nested `@testset` inside a unit would +# be caught by `_run` below and filed as an `:nontest_error`, turning a FAIL into an ERROR. And +# stdlib's `@testset` infers an untyped nested set's type as +# `get_testset_depth() == 0 ? DefaultTestSet : typeof(get_testset())`, so a 0 there drops a +# registered provider's type (see `provider.jl`) — which is the case that reports nothing at all +# and reports it silently. +# +# The scope is inherited by tasks SPAWNED inside it, which task-local storage was not. For a +# unit that joins what it spawns that is strictly better — results that used to vanish into the +# fallback testset now land correctly. For one that does not join, it converts a deterministic +# drop into a race against `unit_fold`; `@shard`'s docstring states the requirement. @static if VERSION >= v"1.13" function _with_testset(f, ts) return Base.ScopedValues.with( @@ -65,6 +79,10 @@ function _run(ctx::ShardContext, key::AbstractString, body) # recorded. Test.record( ts, + # XXX: `Base.catch_stack` is an internal spelling of what is now + # `Base.current_exceptions()`; measured on 1.13 the two return the same + # `Base.ExceptionStack` (`==`), and upstream's own `@testset` has moved to the + # public name. Left as-is so this commit changes one thing. Test.Error( :nontest_error, Expr(:tuple), err, Base.catch_stack(), LineNumberNode(0) ), @@ -72,7 +90,7 @@ function _run(ctx::ShardContext, key::AbstractString, body) end end # A tool that attaches a finished testset to its parent can only do it now that the parent is - # current again — which is the whole reason a hand-popped testset needs this second step. + # current again — which is the whole reason a hand-entered testset needs this second step. _unit_close(ts) dt = time() - t0 sec = unit_fold(ctx, ts) @@ -275,6 +293,14 @@ docstring for the whole picture. The test root — what unit keys are relative to — is the directory of the file this macro is written in, so keys are stable no matter where CI runs from. + +A unit must `wait`/`fetch`/`@sync` every task it spawns before its own top-level code returns. +A unit is folded and reported the moment that code returns, so a result arriving later is +recorded into a testset nobody reads again. On Julia ≥ 1.13 this is worse than it sounds: a +spawned task inherits the unit's testset for its whole life, so a LATE FAILURE lands on the +right object after the unit has already been reported green. Before 1.13 the same result was +dropped on the floor instead — deterministically, which is the only thing that made it +survivable. """ macro shard(body) root = abspath(dirname(String(__source__.file))) diff --git a/test/core/test_failure.jl b/test/core/test_failure.jl index 34a96a0..6bf14c5 100644 --- a/test/core/test_failure.jl +++ b/test/core/test_failure.jl @@ -12,3 +12,31 @@ using .TSHelpers # re-signalled once at the end instead of thrown where it happened. @test "bad.jl" in unit_keys(out) end + +@testset "a failing NESTED testset is a fail, not an error" begin + # The depth the unit's testset is entered at is what tells an inner `@testset` it is not + # top-level. Get it wrong and the inner one's `finish` throws instead of recording, `_run` + # catches that as a `:nontest_error`, and a real FAIL is reported as an ERROR — the same + # count, in the wrong column, with nothing red to say so. + # + # The bare `@test false` above cannot see this: it records straight onto the current testset + # and never reaches the depth logic at all. + d = make_suite(; extra="include(\"nested.jl\")") + write( + joinpath(d, "nested.jl"), + """ + using Test + @testset "outer" begin + @test true + @testset "inner" begin + @test false + end + end + """, + ) + ok, log, out = run_suite(d) + @test !ok + @test occursin("[FAIL] nested.jl", log) + @test occursin("(1 pass, 1 fail, 0 error)", log) + @test !occursin("(0 pass, 0 fail, 1 error)", log) +end From 8c6b6b5a5d57718f317582287c3c35df150ec509 Mon Sep 17 00:00:00 2001 From: sotashimozono Date: Thu, 10 Sep 2026 14:00:28 +0000 Subject: [PATCH 3/5] Move the 1.13 explanation to docs, widen the compat matrix, branch on the name MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Thirty-one lines of comment above fourteen lines of code. The whole account of what Julia 1.13 changed, why the branch tests what it tests, what `TESTSET_DEPTH` is for, and how task propagation differs now lives in `docs/src/testset-internals.md`, registered in `make.jl`'s pages. What stays in `run.jl` is what a reader at that line needs: there are two implementations, both reach into `Test` internals, the branch tests the NAME, and the depth travels with the testset. Six lines. The `XXX` on `catch_stack` keeps its register and a pointer, not its argument. `@shard`'s docstring keeps the "join what you spawn" requirement — that is a requirement on the caller, not an internal, so the API doc is where it belongs. The branch tests `isdefined(Test, :CURRENT_TESTSET)` rather than `VERSION >= v"1.13"`. They agree on every released version — measured on 1.10, 1.11, 1.12, 1.13 and 1.14-DEV — and disagree exactly where a version test is wrong: `v"1.13.0-rc1" >= v"1.13"` is false, so an rc, which does have `CURRENT_TESTSET`, would take the branch that calls `push_testset` and die on the bug this split exists to avoid. `compat` now runs 1.10, 1.11, 1.12 and 1.13 instead of just the two ends. Suite passes on all four locally, plus 1.14-DEV. Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/CI.yml | 6 +-- docs/make.jl | 1 + docs/src/testset-internals.md | 86 +++++++++++++++++++++++++++++++++++ src/run.jl | 36 +++------------ 4 files changed, 95 insertions(+), 34 deletions(-) create mode 100644 docs/src/testset-internals.md diff --git a/.github/workflows/CI.yml b/.github/workflows/CI.yml index b885ebd..9f493d7 100644 --- a/.github/workflows/CI.yml +++ b/.github/workflows/CI.yml @@ -130,11 +130,7 @@ jobs: strategy: fail-fast: false matrix: - # The floor, and the version at which `Test`'s testset stack became a `ScopedValue`. - # `_with_testset` has one implementation per side of that boundary and only ONE is - # compiled per run, so pinning both is the only way each stays under test — the sharded - # jobs above take whatever `'1'` resolves to, which moves with Julia's release calendar. - julia: ['1.10', '1.13'] + julia: ['1.10', '1.11', '1.12', '1.13'] steps: - uses: actions/checkout@v7 - uses: julia-actions/setup-julia@v3 diff --git a/docs/make.jl b/docs/make.jl index eee9d66..9ade2e0 100644 --- a/docs/make.jl +++ b/docs/make.jl @@ -20,6 +20,7 @@ makedocs(; "Records" => "records.md", "Guarantees" => "guarantees.md", "Composing" => "composing.md", + "Testset internals" => "testset-internals.md", "API" => "api.md", "References" => "references.md", ], diff --git a/docs/src/testset-internals.md b/docs/src/testset-internals.md new file mode 100644 index 0000000..8b7752f --- /dev/null +++ b/docs/src/testset-internals.md @@ -0,0 +1,86 @@ +# Testset internals + +A unit runs inside a testset this package enters by hand, not through `@testset`. That one +decision is why `Test`'s internals appear in `src/run.jl` at all, and why the file carries two +implementations of the same six lines. + +## Why not `@testset` + +A top-level `@testset` **throws instead of returning** when something inside it fails. TestShards +needs the opposite: a failed unit's tree has to be readable, because the counts in +[`unit_fold`](@ref) and the per-unit record are built from it, and because the remaining units +still have to run. So `_run` enters the testset itself, runs the body, leaves, and re-signals +failure once at the end of the whole shard. + +Measured on 1.10, 1.12 and 1.13 alike — this is not a version-specific quirk that a newer Julia +removed the need for. + +## What Julia 1.13 changed + +Through 1.12, "the current testset" was a **stack in task-local storage**: + +```julia +function push_testset(ts::AbstractTestSet) # julia ≤ 1.12 + testsets = get(task_local_storage(), :__BASETESTNEXT__, AbstractTestSet[]) + push!(testsets, ts) + setindex!(task_local_storage(), testsets, :__BASETESTNEXT__) +end +``` + +In 1.13 it is a scoped value: + +```julia +const CURRENT_TESTSET = ScopedValue{AbstractTestSet}(FallbackTestSet()) # julia ≥ 1.13 +const TESTSET_DEPTH = ScopedValue{Int}(0) +``` + +A scoped value cannot be pushed and popped — it is *entered* — so `push_testset` and +`pop_testset` are gone rather than renamed. This was deliberate and announced: Julia's own +`NEWS.md` for 1.13 carries "The testset stack was changed to use `ScopedValue` rather than task +local storage", and `Test.@testset` itself now expands to `@with(CURRENT_TESTSET => ts, +TESTSET_DEPTH => get_testset_depth() + 1, expr)`. + +`_with_testset` is that difference and nothing else. Everything downstream — `_unit_close`, +`unit_fold`, the records, the printed line — sees the same thing either way. + +## Why the branch tests the name, not the version + +`@static if isdefined(Test, :CURRENT_TESTSET)`, not `VERSION >= v"1.13"`. + +The two agree on every released version — measured on 1.10, 1.11, 1.12, 1.13 and 1.14-DEV — and +disagree in exactly the place a version test is wrong: `v"1.13.0-rc1" >= v"1.13"` is `false`, +because a prerelease sorts before its own release. An rc **has** `CURRENT_TESTSET`, so a version +test would send it down the branch that calls `push_testset` and it would die on the very bug +this split exists to avoid. + +## Why `TESTSET_DEPTH` moves with `CURRENT_TESTSET` + +Two things read the depth, and both fail quietly when it reads `0`: + +- `Test.finish` decides from it whether a testset is top-level, and a top-level one **throws + instead of recording**. A failing nested `@testset` inside a unit would then be caught by + `_run`'s own `catch` and filed as a `:nontest_error` — a **fail reported as an error**. +- stdlib's `@testset` infers an untyped nested set's type as + `get_testset_depth() == 0 ? DefaultTestSet : typeof(get_testset())`. A `0` there drops a + registered provider's testset type (see [Composing](composing.md)), which is the case that + records nothing at all and reports that silently. + +`test/core/test_failure.jl` pins the first of these directly. + +## Tasks spawned inside a unit + +A scoped value is inherited by tasks spawned inside its scope; task-local storage was not. For a +unit that `wait`s or `fetch`es everything it spawns, 1.13 is strictly better — results that used +to vanish into the fallback testset now land on the right testset. For a unit that does **not** +join, the same inheritance means a late result targets a testset the shard has already folded +and reported, turning a deterministic drop into a race. + +So: **a unit must join every task it spawns before its own top-level code returns.** This is +stated on `@shard` as well, because it is a requirement on the caller, not an internal detail. + +## What is still not public + +Neither `push_testset` nor `CURRENT_TESTSET` is exported or marked `public`. This package +therefore depends on an internal on both sides of the branch, and a later reshuffle upstream +will break it again in the same way. The public route — `@testset` and its return value — is +unavailable for the reason at the top of this page. diff --git a/src/run.jl b/src/run.jl index 553b885..562caf5 100644 --- a/src/run.jl +++ b/src/run.jl @@ -4,31 +4,11 @@ # Run `f` with `ts` as the current testset, and leave the previous one current afterwards. # -# Two implementations because Julia 1.13 changed what "current testset" IS. Through 1.12 it was -# a STACK in task-local storage, entered with `Test.push_testset` and left with -# `Test.pop_testset`. In 1.13 it became a `ScopedValue` (`Test.CURRENT_TESTSET`), which cannot -# be pushed and popped at all — a scoped value is entered, and `push_testset`/`pop_testset` no -# longer exist. `Test.@testset` itself switched to `@with` for the same reason. -# -# Both spellings reach into Test's internals, which is what breaks: neither `push_testset` nor -# `CURRENT_TESTSET` is public API. The alternative — using `@testset` and reading its return — -# is not available, because a top-level `@testset` throws instead of returning when the unit -# fails, which is the whole reason this function exists. -# -# `TESTSET_DEPTH` moves with `CURRENT_TESTSET` because two things read it, and both fail -# QUIETLY if it says 0. `Test.finish` uses it to decide whether a testset is top-level — a -# top-level one THROWS instead of recording, so a failing nested `@testset` inside a unit would -# be caught by `_run` below and filed as an `:nontest_error`, turning a FAIL into an ERROR. And -# stdlib's `@testset` infers an untyped nested set's type as -# `get_testset_depth() == 0 ? DefaultTestSet : typeof(get_testset())`, so a 0 there drops a -# registered provider's type (see `provider.jl`) — which is the case that reports nothing at all -# and reports it silently. -# -# The scope is inherited by tasks SPAWNED inside it, which task-local storage was not. For a -# unit that joins what it spawns that is strictly better — results that used to vanish into the -# fallback testset now land correctly. For one that does not join, it converts a deterministic -# drop into a race against `unit_fold`; `@shard`'s docstring states the requirement. -@static if VERSION >= v"1.13" +# Two implementations because Julia 1.13 replaced the testset stack with a `ScopedValue`. Both +# reach into `Test` internals; the branch tests for the NAME rather than the version, and +# `TESTSET_DEPTH` has to travel with `CURRENT_TESTSET`. Why, for all three, is in +# `docs/src/testset-internals.md`. +@static if isdefined(Test, :CURRENT_TESTSET) function _with_testset(f, ts) return Base.ScopedValues.with( f, @@ -79,10 +59,8 @@ function _run(ctx::ShardContext, key::AbstractString, body) # recorded. Test.record( ts, - # XXX: `Base.catch_stack` is an internal spelling of what is now - # `Base.current_exceptions()`; measured on 1.13 the two return the same - # `Base.ExceptionStack` (`==`), and upstream's own `@testset` has moved to the - # public name. Left as-is so this commit changes one thing. + # XXX: `Base.catch_stack` is internal; `Base.current_exceptions()` is + # the public spelling — see `docs/src/testset-internals.md`. Test.Error( :nontest_error, Expr(:tuple), err, Base.catch_stack(), LineNumberNode(0) ), From fed4f1bd6623d1829fcf248d83cd18d57ea111b9 Mon Sep 17 00:00:00 2001 From: sotashimozono Date: Thu, 10 Sep 2026 14:09:55 +0000 Subject: [PATCH 4/5] Make the docs page's checkable claims run MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two of the four code blocks were assertions a reader has to take on trust; `julia` blocks are inert, so they rot silently. Both are now `jldoctest` — the repo's first, and `doctest` is on by default, so they run in the existing docs build. - the contract both `_with_testset` implementations must meet (`get_testset() === ts`, one level deeper, and back afterwards); - `v"1.13.0-rc1" >= v"1.13"` is `false`, which is the whole reason the branch tests the name rather than the version. The first one caught its own mistake on the way in. Written as an absolute depth it read `(true, 1)` in a bare session and `(true, 2)` under Documenter, which runs doctests inside a testset of its own — the assertion was coupled to the ambient environment. The property `_with_testset` actually has is the INCREMENT: one level deeper, then back. Asserted that way it holds anywhere, and asserting the "back" half covers the `finally` this change removed. Verified it can fail: expecting `(true, 2)` turns the build red. The other two blocks stay `julia` because they are quotations of `stdlib/Test/src/Test.jl` and cannot run — the 1.12 one does not exist on 1.13, nor the 1.13 one before it. The prose now says so, rather than leaving them looking like examples. Co-Authored-By: Claude Opus 5 (1M context) --- docs/src/testset-internals.md | 42 ++++++++++++++++++++++++++++++----- 1 file changed, 36 insertions(+), 6 deletions(-) diff --git a/docs/src/testset-internals.md b/docs/src/testset-internals.md index 8b7752f..1699f83 100644 --- a/docs/src/testset-internals.md +++ b/docs/src/testset-internals.md @@ -17,7 +17,9 @@ removed the need for. ## What Julia 1.13 changed -Through 1.12, "the current testset" was a **stack in task-local storage**: +Through 1.12, "the current testset" was a **stack in task-local storage** (quoting +`stdlib/Test/src/Test.jl`, so these two blocks are not runnable here — the first no longer +exists on 1.13, the second not before it): ```julia function push_testset(ts::AbstractTestSet) # julia ≤ 1.12 @@ -41,17 +43,45 @@ local storage", and `Test.@testset` itself now expands to `@with(CURRENT_TESTSET TESTSET_DEPTH => get_testset_depth() + 1, expr)`. `_with_testset` is that difference and nothing else. Everything downstream — `_unit_close`, -`unit_fold`, the records, the printed line — sees the same thing either way. +`unit_fold`, the records, the printed line — sees the same thing either way, which is the +contract both implementations have to meet: + +```jldoctest +julia> using Test, TestShards + +julia> ts = Test.DefaultTestSet("a unit"); + +julia> outer = Test.get_testset_depth(); + +julia> TestShards._with_testset(ts) do + Test.get_testset() === ts, Test.get_testset_depth() - outer + end +(true, 1) + +julia> Test.get_testset_depth() == outer +true +``` + +The depth is asserted as an INCREMENT, not as `1`: it counts from whatever is already open, so +the absolute value depends on the caller — inside Documenter's own testset this block reads `2` +where a bare session reads `1`. One level deeper, and back where it started, is the property +`_with_testset` actually has. ## Why the branch tests the name, not the version `@static if isdefined(Test, :CURRENT_TESTSET)`, not `VERSION >= v"1.13"`. The two agree on every released version — measured on 1.10, 1.11, 1.12, 1.13 and 1.14-DEV — and -disagree in exactly the place a version test is wrong: `v"1.13.0-rc1" >= v"1.13"` is `false`, -because a prerelease sorts before its own release. An rc **has** `CURRENT_TESTSET`, so a version -test would send it down the branch that calls `push_testset` and it would die on the very bug -this split exists to avoid. +disagree in exactly the place a version test is wrong, because a prerelease sorts before its own +release: + +```jldoctest +julia> v"1.13.0-rc1" >= v"1.13" +false +``` + +An rc **has** `CURRENT_TESTSET`, so a version test would send it down the branch that calls +`push_testset` and it would die on the very bug this split exists to avoid. ## Why `TESTSET_DEPTH` moves with `CURRENT_TESTSET` From 738881cebac5783087b4e7e7c0ceb9b06f08b770 Mon Sep 17 00:00:00 2001 From: sotashimozono Date: Thu, 10 Sep 2026 14:14:22 +0000 Subject: [PATCH 5/5] Drop an @ref that only resolves inside a docstring MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The docs preview failed on `Cannot resolve @ref for [`unit_fold`](@ref)`. The identical link works at provider.jl:41 because a docstring resolves @ref in its own module context; a hand-written page does not get that. The link added nothing a code span does not, so it is a code span now. The doctests themselves ran and passed in that build — the failure was in the cross-reference pass after them. Co-Authored-By: Claude Opus 5 (1M context) --- docs/src/testset-internals.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/src/testset-internals.md b/docs/src/testset-internals.md index 1699f83..2995ac6 100644 --- a/docs/src/testset-internals.md +++ b/docs/src/testset-internals.md @@ -8,7 +8,7 @@ implementations of the same six lines. A top-level `@testset` **throws instead of returning** when something inside it fails. TestShards needs the opposite: a failed unit's tree has to be readable, because the counts in -[`unit_fold`](@ref) and the per-unit record are built from it, and because the remaining units +`unit_fold` and the per-unit record are built from it, and because the remaining units still have to run. So `_run` enters the testset itself, runs the body, leaves, and re-signals failure once at the end of the whole shard.