Randomized sweep points + an accumulating CI evidence database (→ analyzer figures → docs)

[sotashimozono]

Design note / vision — not scheduled. Recorded so the migration work keeps it in mind.

The idea

1. Constraint-edge sweep points (@identity, @bound) become randomized rather than a fixed list.
2. Each CI run's results accumulate into a database (idea: a separate repo, e.g. QAtlasRegistry).
3. A separate QAtlasAnalyzer turns that database into figures.
4. QAtlasDocumenter publishes those figures into the docs.

What already exists (measured 2026-07-21, not estimated)

Most of the pipeline is already running:

• Emission — QATLAS_EMIT=1 on push:main writes verification cards; each card already carries subject (the fetched value), independent (the oracle value), at (the parameter point), abserr, atol, convergence_rate, reliability, plus provenance commit / env (julia version, runner, run id) / date.
• Transport — per-shard evidence-*.jsonl artifacts → the record-evidence job.
• Store — the ci-evidence orphan branch, currently 427 rows. (ci-timings does the same for shard timings.)
• Analysis + docs — docs/atlas/generate.jl already computes the R1 metrics (univ / edp / coh / cited / risk, feas, rate_struct, rate_inrepo) and emits 385 per-hub pages, the facet pages, the graph export and the sitemap. This is the documenter.

So the remaining work is much smaller than "build three repos".

The blocker (one line of jq)

record-evidence merges with

jq -s 'reduce .[] as $c ({}; .[$c.hub + "|" + $c.route] = $c) | ...'

i.e. it deduplicates by hub|route, latest wins. The 427 rows are a snapshot of current coverage, not a history. Randomizing the points changes nothing until this key changes — every run would simply overwrite the last. Keying on hub|route|at (or +commit) makes it accumulate; that then needs a retention policy, since it grows without bound.

Two hazards to design around

1. Randomness on PRs would make CI flaky. A PR failing on an unrelated random draw trains everyone to ignore red. Mitigation: the workflow already splits PR from main via QATLAS_TEST_PROFILE (fast/full/nightly) and QATLAS_EMIT — put randomness on push:main / nightly only and keep PRs deterministic. No new machinery needed.
2. Reproducibility. A failure at a random point is not re-runnable unless the draw is recoverable. Use a seeded PRNG and record the seed in the evidence card next to at.

With those two, this is just property-based testing with a persistent corpus — and because each card stores subject and independent at a known point, an accumulating corpus yields error-vs-parameter curves for free, which is exactly what the figures would show.

Notes

• _sweep_points expands with for v in vals, so it already accepts any iterable — randomized points need no change to the expansion. The friction is id stability (_point_id embeds the values, and check ids are the sharding/reporting key) and the merge key, not the generator.
• QAtlasHub/QAtlasRegistry is currently completely empty (no Registry.toml). Its name collides with the Julia package registry concept — worth settling what that repo is for before using it as a data store.

Suggested slicing (cheapest first)

1. Change the merge key to accumulate + add a retention policy. Existing pipeline starts building history immediately, before any randomness exists.
2. Seeded random sweeps, push:main / nightly only, seed recorded in the card.
3. Analyzer figures, once there is a distribution to plot.