CLAUDE.md declares agent.json a contract. Three id schemes in the test-report path are order- or line-dependent, so the contract identifiers are not stable across runs.
Check ids are a global emission counter. src/testset.jl:1283-1284:
counter[] += 1
chk.id = Symbol("t", counter[])Assigned in document-walk order across the whole report. Add one @test to the first test file and every subsequent check in every subsequent file is renumbered. _benchmark_verdict (src/document.jl:236) emits failed:["t47"] into agent.json. Under shards the order additionally depends on the merge order at :1494 (reduce(vcat, r.children for r in roots)), i.e. on the order readdir returned the dump files.
Code-block ids are keyed on line numbers. src/testset.jl:366-368:
Symbol("code_", slug(basename(file)), "_", lo, "_", hi)Inserting a blank line at the top of a test file changes every code artifact id and anchor in that file, changes every Check.code_anchor referencing them, and invalidates the render cache for the whole file. Gallery deep links break on any commit that shifts lines.
Page-id disambiguation is first-seen order. _unique_id! (src/testset.jl:1149) resolves a collision as base-2, base-3. _slug is flat (:167), so a/util.jl and b/util.jl collide; which one becomes util_jl and which util_jl-2 depends on shard and merge order, so the generated HTML filenames swap between CI runs.
Fix
Derive check ids from something stable -- the file, the enclosing testset path and an index within it -- rather than a global counter; key code blocks on content or a declared name rather than a line span; and make page-id disambiguation deterministic in the source path, not in arrival order.
CLAUDE.md declares
agent.jsona contract. Three id schemes in the test-report path are order- or line-dependent, so the contract identifiers are not stable across runs.Check ids are a global emission counter.
src/testset.jl:1283-1284:Assigned in document-walk order across the whole report. Add one
@testto the first test file and every subsequent check in every subsequent file is renumbered._benchmark_verdict(src/document.jl:236) emitsfailed:["t47"]intoagent.json. Under shards the order additionally depends on the merge order at:1494(reduce(vcat, r.children for r in roots)), i.e. on the orderreaddirreturned the dump files.Code-block ids are keyed on line numbers.
src/testset.jl:366-368:Inserting a blank line at the top of a test file changes every code artifact id and anchor in that file, changes every
Check.code_anchorreferencing them, and invalidates the render cache for the whole file. Gallery deep links break on any commit that shifts lines.Page-id disambiguation is first-seen order.
_unique_id!(src/testset.jl:1149) resolves a collision asbase-2,base-3._slugis flat (:167), soa/util.jlandb/util.jlcollide; which one becomesutil_jland whichutil_jl-2depends on shard and merge order, so the generated HTML filenames swap between CI runs.Fix
Derive check ids from something stable -- the file, the enclosing testset path and an index within it -- rather than a global counter; key code blocks on content or a declared name rather than a line span; and make page-id disambiguation deterministic in the source path, not in arrival order.