Skip to content

fix: rebuild :latest on every merge to main (kill the OAuth recurrence forever)#75

Merged
fatherlinux merged 1 commit into
mainfrom
fix/auto-rebuild-on-main
May 20, 2026
Merged

fix: rebuild :latest on every merge to main (kill the OAuth recurrence forever)#75
fatherlinux merged 1 commit into
mainfrom
fix/auto-rebuild-on-main

Conversation

@fatherlinux
Copy link
Copy Markdown
Member

Why

The OAuth Missing code verifier error on acquacotta.crunchtools.com has recurred six times since early 2026. Each recurrence triggered a different code-level "fix" (tmpfs → persistent volume → signed cookies → signed state). The signed-state fix in #72 (commit ac182fb, 2026-05-02) was actually correct.

The fixes kept getting overwritten by the archived sibling repo crunchtools/acquacotta-old, which had a weekly GHA cron (30 4 * * 1, Mondays 04:30 UTC) rebuilding pre-fix code and pushing to the same quay.io/crunchtools/acquacotta:latest tag. Every fix shipped from this repo got clobbered the following Monday morning.

acquacotta-old was archived on 2026-05-19 (disables all workflows). Production was force-rebuilt and pulled on lotor the same day; OAuth now works.

This PR closes the remaining gap so the OAuth fix — and every future fix — actually ships on merge.

Changes

  • .github/workflows/container-build.yml — add push: branches: [main] trigger. Every merge to main now rebuilds and pushes :latest. Existing v* tag and workflow_dispatch triggers retained.
  • .specify/memory/constitution.md (v1.2.0 → v1.3.0) — Release & Versioning section now mandates:
    • Builds trigger on BOTH push:main and push:tags:v*.
    • Image tags are unique to one repository (no zombie siblings).
    • Deployed systemd units carry io.containers.autoupdate=registry + PODMAN_SYSTEMD_UNIT labels so lotor's nightly podman-auto-update.timer pulls automatically.

Verification

After merge:

  • GHA "Build and Push Container" runs on the merge commit (proves the new trigger fires).
  • New image lands at quay.io/crunchtools/acquacotta:latest with the merge SHA in labels.
  • ssh -p 22422 root@lotor.dc3.crunchtools.com 'podman auto-update --dry-run' lists acquacotta.crunchtools.com.service.
  • OAuth login at https://acquacotta.crunchtools.com succeeds.

No app.py changes; OAuth code already correct.

…rule

The OAuth 'Missing code verifier' bug recurred six times because the build
workflow only rebuilt :latest on `v*` tags, and the archived sibling repo
crunchtools/acquacotta-old had a weekly cron pushing pre-fix code to the
same quay.io/crunchtools/acquacotta:latest tag — clobbering every fix
within one Monday morning.

The sibling repo is now archived. Adding `push: branches: [main]` here so
main and :latest stay in sync without manual tagging, and amending the
constitution so no future contributor (or zombie repo) can reintroduce
this class of drift.

Constitution v1.3.0:
- Container builds MUST trigger on both push:main and push:tags:v*.
- Image tags MUST be unique to one repository (the 2026-05-19 incident).
- Deployed systemd units MUST set podman autoupdate + PODMAN_SYSTEMD_UNIT
  labels so lotor's nightly auto-update pulls :latest automatically.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@fatherlinux fatherlinux merged commit ccc5377 into main May 20, 2026
2 of 3 checks passed
@fatherlinux fatherlinux deleted the fix/auto-rebuild-on-main branch May 20, 2026 01:02
Copy link
Copy Markdown

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the project's constitution to formalize container build triggers for both main branch pushes and version tags, mandates unique image tags to prevent overwriting production images, and defines requirements for automated updates on the 'lotor' host. Feedback was provided regarding the Podman labels for systemd units, noting that 'io.containers.autoupdate.systemd_unit' is the standard label for auto-updates rather than 'PODMAN_SYSTEMD_UNIT'.


Container image tags MUST be unique to this repository. No other repository — including archived siblings or forks — may publish to the same `quay.io/crunchtools/<image>` tag. A zombie repo sharing a tag will silently overwrite production. (See 2026-05-19 incident: `acquacotta-old`'s weekly cron clobbered the OAuth fix six times in a row.)

Deployed systemd units on lotor MUST include `--label io.containers.autoupdate=registry` and `--label PODMAN_SYSTEMD_UNIT=<unit>.service` so the nightly `podman-auto-update.timer` pulls new `:latest` images automatically.
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The standard Podman label for specifying the systemd unit for auto-updates is io.containers.autoupdate.systemd_unit. While PODMAN_SYSTEMD_UNIT is a standard environment variable used by Podman when running under systemd, using it as a label (via --label) is non-standard for the auto-update mechanism. Additionally, please note that the acquacotta.crunchtools.com.service file currently in the repository does not yet include these labels, which contradicts the mandate being established here.

Suggested change
Deployed systemd units on lotor MUST include `--label io.containers.autoupdate=registry` and `--label PODMAN_SYSTEMD_UNIT=<unit>.service` so the nightly `podman-auto-update.timer` pulls new `:latest` images automatically.
Deployed systemd units on lotor MUST include `--label io.containers.autoupdate=registry` and `--label io.containers.autoupdate.systemd_unit=<unit>.service` so the nightly `podman-auto-update.timer` pulls new `:latest` images automatically.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant