Skip to content

Run the data build from main only, sync from the lock, publish one release per build - #98

Merged
baz8080 merged 10 commits into
mainfrom
claude/trusting-dirac-17nbln-ci
Sep 24, 2026
Merged

baz8080 merged 10 commits into
mainfrom
claude/trusting-dirac-17nbln-ci

Conversation

@baz8080

@baz8080 baz8080 commented Sep 24, 2026 •

Copy link
Copy Markdown
Owner

PR G of the fixes from the retroactive review: CI safeguards, a publish that can't leave the latest release without a DB, and the removal of the closed_at replay input.

Changes

  • Main only. A Build DB dispatch from anywhere other than main is refused by its own job, which fails visibly. The build-db concurrency group sits on the build job, so a refused dispatch can't displace a queued scheduled run. pages.yml also requires workflow_run.head_branch == 'main'.
  • uv sync --locked in all three workflows, with uv pinned to 0.12.18 next to the setup-uv commit SHA.
    • 0.12.18 passes uv lock --check on the current lock and installs CPython 3.14.7.
    • An earlier pin, 0.8.17, would only have offered 3.14.0rc2.
  • One release per build (owner decision).
    • Every data build publishes its own release, tagged YYYY-MM-DD-HHMM (UTC), with gh release create --latest.
    • gh creates the release as a draft, uploads and then publishes it, so the latest release always holds a complete uisce.db. That is about two releases a day instead of one.
    • This replaces gh release upload --clobber, which deleted the old asset before uploading, so a failure in between left the day's release with no DB.
    • scripts/publish-db.sh and scripts/fetch-db.sh are now the only way to publish or download. Both workflows, the README, pipeline-dependencies.md and build.py's refresh hint use them.
  • astral-sh/setup-uv pinned to its commit (c771a70, tag v9.0.0). It runs in the job with contents: write.
  • The replay_closed_at dispatch input and its step are removed (owner decision).
    • A dry run over all 75 releases found 0 values to stamp or change.
    • The script stays, run by hand. It now reads the date from the tag's first 10 characters, sorts snapshots by tag, and its documented loop excludes drafts.

Review

First round, 10 findings. The first design replaced the asset inside a per-day release by uploading under a temporary name and renaming. The review found four failure paths in that swap, and those fixes led to the swap being moved into scripts.

Second round, 10 findings. It found three more states the swap could strand:

  • the recovery view's errors were swallowed;
  • no recovery ran before the download step;
  • both copies could be present at once, and the stale one was then used.

All three came from juggling two names inside one release, so the owner chose one release per build. That removes the swap and the fallback, and with them 7 of the 10 findings.

The other three:

  • the uv pin landed on a Python release candidate;
  • the asset name depended on the file name (the # suffix in gh release create is only a display label, per gh's source);
  • a download recipe in notes/pipeline-dependencies.md had been missed.

Also found while testing: snapshot file names sort 2026-09-24-1845.db before 2026-09-24.db, so the replay now sorts by tag, and a test covers it.

Verification

  • shellcheck and actionlint pass.
  • uvx uv@0.12.18 lock --check passes.
  • A fake gh exercised: a publish (the previous release untouched), a second publish in the same minute (refused), a DB file under another name (still published as uisce.db), and a fetch.
  • uv run ruff check and uv run pytest pass: 687 tests, merged with current main.
  • The first real run is the next scheduled Build DB after merge. I'll check its release and the Pages deploy.

Notes

  • data-quality.md:
    • "The replay has nothing left to recover (2026-09-24)";
    • "One release per build (2026-09-24)", including the rejected swap.
  • CLAUDE.md: rows for the replay input and for one release per build.

🤖 Generated with Claude Code

https://claude.ai/code/session_0168hLW2X3mkV3Jhm26LJSwQ

Build DB could be dispatched from any branch, and a branch run publishes
its DB as the latest release and pushes to main. Its first step now fails
the job with an error unless the ref is refs/heads/main: a failure shows
up in the Actions list, a skipped job would not. Build site's
workflow_run trigger also requires head_branch == main, so a branch run
of Build DB can never deploy the site.

uv sync takes --locked in all three workflows. statusui is a git
dependency with no rev in pyproject.toml, so uv.lock is its only pin; a
pyproject/uv.lock mismatch now fails the job instead of re-resolving
statusui to whatever its default branch holds. uv lock --check passes on
this commit.

astral-sh/setup-uv is pinned to the commit of v9.0.0 (a lightweight tag,
resolved with git ls-remote). It is the only third-party action; the
actions/* ones are left on their tags.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0168hLW2X3mkV3Jhm26LJSwQ
The second build of a day re-publishes to the release the first one
created, with gh release upload --clobber. --clobber deletes the old
asset and then uploads, so an upload that fails (a 34 MB transfer is the
step most likely to) left the latest release with no uisce.db, and every
later Build DB and Build site run then failed at its download.

The upload now goes to uisce.db.next, which is never read while
uisce.db exists. Only once it has landed is the old uisce.db deleted and
uisce.db.next renamed to uisce.db (PATCH on the release asset). A failed
upload therefore touches nothing a reader uses. What is left is the gap
between the DELETE and the PATCH, two small API calls: GitHub has no
atomic replace, and two assets cannot share a name, so some gap is
unavoidable while the name stays uisce.db. It is covered twice:

- Both downloads fall back to uisce.db.next when the latest release has
  no uisce.db. In that state uisce.db.next is always the complete new
  DB, because the old one is only deleted after the upload succeeds.
- The next Publish DB finishes an interrupted swap (renames a lone
  uisce.db.next to uisce.db) before it stages its own upload, so its
  --clobber on the staging name can never delete the only copy.

A first build of the day still uses gh release create, which uploads
into a draft and publishes it only once the asset is there, so it never
exposed an empty release.

Rejected: a new tag per build (the replay writes the tag into closed_at
as a date, and the release count would double) and readers walking back
to an older release (it would build on a DB that lacks the earlier
build's observations, and archive a stale DB as the day's).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0168hLW2X3mkV3Jhm26LJSwQ
Dry run of uisce-replay-closed-at over all 75 release snapshots
(2026-06-30 to 2026-09-23) against a copy of the 2026-09-23 release:
6,961 transitions found, every one already stamped with the same date,
0 rows to stamp and 0 that would change. The replay_closed_at input is
left in place for the owner to decide on; its --limit 100 runs out
around 2026-10-19.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0168hLW2X3mkV3Jhm26LJSwQ
The dry run over all 75 releases found nothing left to recover, so the
input and its step go (owner decision). The script stays for a DB
restored from an older release, run by hand. The roadmap entry is
closed and the outcome recorded in data-quality.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0168hLW2X3mkV3Jhm26LJSwQ
From a code review of this PR:

- The swap only repaired today's release, so a run dying mid-swap on
  its day's last build stranded that release with only uisce.db.next.
  publish-db.sh repairs the latest published release first.
- releases/tags/<tag> 404s on a draft that `gh release view` finds, which
  wedged the day after a killed create. The script reads assets through
  `gh release view` (apiUrl) and replaces a leftover draft.
- It deleted uisce.db without checking the staged upload was listed; it
  now stops first. A 5xx on the view was read as "no release" and then
  failed on create; only "release not found" means absent now.
- fetch-db.sh is the one download, with the .next fallback, used by both
  workflows, the README, build.py's hint and the replay docstring (whose
  loop now lists up to 1000 releases).
- The main-only refusal is its own job and the build-db concurrency group
  moved onto the build job, so a refused dispatch cannot displace a
  queued scheduled run.
- uv is pinned (0.8.17) beside the setup-uv SHA, so --locked cannot fail
  on a new uv release. CLAUDE.md gains the replay row.

Both scripts were run against a fake gh through: no release, second
build, stranded yesterday, stranded today, leftover draft, a 502 on the
view, and a lost upload; actionlint and shellcheck pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0168hLW2X3mkV3Jhm26LJSwQ
A second code review of the swap found three more ways it could strand a
release or quietly lose a build's DB, all from juggling two asset names
inside one release. Owner decision: every build publishes its own
release, tagged YYYY-MM-DD-HHMM (UTC). gh creates it as a draft, uploads
and publishes, so the latest release always holds a complete uisce.db.

- publish-db.sh is a single `gh release create --latest`, staging the
  file as uisce.db whatever it is called (the `#` suffix is only a
  display label). fetch-db.sh is a plain download with no fallback, so
  its errors are its own.
- replay_closed_at stamps the tag's date part and sorts by tag, since
  "2026-09-24-1845.db" sorts before "2026-09-24.db" by file name; the
  documented loop excludes drafts.
- uv is pinned at 0.12.18, not 0.8.17, which only offers CPython
  3.14.0rc2; 0.12.18 passes `uv lock --check` and installs 3.14.7.
- pipeline-dependencies.md's recipe uses fetch-db.sh; data-quality.md
  records the decision and the rejected swap; CLAUDE.md gains a row.

Checked: shellcheck, actionlint, a fake gh (publish, same-minute refusal,
a non-uisce file name, fetch), `uvx uv@0.12.18 lock --check`, ruff, pytest.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0168hLW2X3mkV3Jhm26LJSwQ
@baz8080 baz8080 changed the title Run the data build from main only, sync from the lock, never leave a release without its DB Run the data build from main only, sync from the lock, publish one release per build Sep 24, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0168hLW2X3mkV3Jhm26LJSwQ
@baz8080
baz8080 merged commit 03af24d into main Sep 24, 2026
2 checks passed
@baz8080
baz8080 deleted the claude/trusting-dirac-17nbln-ci branch September 24, 2026 11:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants