From 2abf59800c03148d7be12dea8cca88fc375f4662 Mon Sep 17 00:00:00 2001 From: Ryan Barlow <7389646+ryanbarlow97@users.noreply.github.com> Date: Sat, 17 Oct 2026 12:33:32 +0000 Subject: [PATCH 1/4] docs(provincesystem): retire the SQLite rollback and rename entitlement tables Both sites run on PostgreSQL. Deploys always run prepare and studio, refuse revisions from before the move and keep only PostgreSQL dumps. Preview clones list Dev's tables from its schema. rpc_player_meta is now web_entitlements and character_player_meta is now character_account_meta. Login codes drop their plaintext once spent, revoked or expired. Companion to TF-Minecraft/ProvinceSystem#267. Co-Authored-By: Claude Opus 5.5 --- projects/ProvinceSystem/DEPLOY.md | 93 ++++------ projects/ProvinceSystem/STAGING.md | 12 +- projects/ProvinceSystem/docs/README.md | 2 +- projects/ProvinceSystem/docs/architecture.md | 8 +- .../ProvinceSystem/docs/characters/creator.md | 4 +- .../ProvinceSystem/docs/cosmetics/drinks.md | 3 +- .../ProvinceSystem/docs/cosmetics/skins.md | 4 +- .../docs/identity/auth-security.md | 7 +- .../ProvinceSystem/docs/identity/tfmcweb.md | 4 +- .../ProvinceSystem/docs/map/title-editor.md | 2 +- .../ProvinceSystem/docs/ops/postgresql.md | 167 +++++++----------- projects/RPCharacters/docs/realm-wipe.md | 2 +- 12 files changed, 125 insertions(+), 183 deletions(-) diff --git a/projects/ProvinceSystem/DEPLOY.md b/projects/ProvinceSystem/DEPLOY.md index 73dad79..39b8479 100644 --- a/projects/ProvinceSystem/DEPLOY.md +++ b/projects/ProvinceSystem/DEPLOY.md @@ -11,22 +11,16 @@ ProvinceSystem changes reach the website in three steps, all driven by GitHub Ac Paths below are on the TF server. -All site records use native PostgreSQL after the site's verified cutover; CoreProtect's external SQLite -databases are unchanged. Main (`www`) remains a separate, explicitly requested manual deployment and cutover. -See [Native PostgreSQL operations](docs/ops/postgresql.md) for the reviewed copy, verification and rollback -procedure, including retention of the one-off SQLite import/rollback schema for at least 30 days. +Both sites keep their records in PostgreSQL; CoreProtect's SQLite databases are read separately. See +[PostgreSQL storage](docs/ops/postgresql.md) for configuration, migrations, backups and restores. ## Branch previews -Native preview deployment requires the repository variable `NATIVE_POSTGRES_PREVIEWS_READY=true`. Keep it -disabled until the host scripts, database isolation and Dev migration are ready. Builds can still run while -deployment is held. The host also requires Dev's verified authority marker at +Previews deploy while the repository variable `NATIVE_POSTGRES_PREVIEWS_READY` is `true`. Set it to `false` +to hold preview deployment; builds still run. The host also requires Dev's authority marker at `/home/tfmc/provincesystem-dev-data/data/.postgres-authority.json`, a backend image labelled `net.tfminecraft.storage=postgres`, and both images' `org.opencontainers.image.revision` labels matching the -requested commit. A branch without the native backend is refused. After Dev and database isolation pass, -enable readiness for a controlled first Preview workflow run on a reviewed native branch. Verify creation, -redeployment and teardown, including database/role cleanup. Leave readiness enabled only after that lifecycle -passes; disable it again if validation fails. +requested commit. A branch from before the move to PostgreSQL is refused. Pushing a branch builds its backend and frontend images on GitHub, in parallel, and pushes them to `ghcr.io/tf-minecraft/provincesystem-preview-{backend,frontend}:` (public packages). Pushes to `main` build @@ -51,9 +45,10 @@ the `ps-preview-router` Caddy container (`127.0.0.1:8090`, behind the `*.tfminec database and assets. - **Clone:** the trusted live Dev helper ([`backend/src/database/preview.py`](https://github.com/TF-Minecraft/ProvinceSystem/blob/main/backend/src/database/preview.py)) - creates the 61-table site schema using reviewed migrations under the preview owner. A site-data-only custom - dump/restore copies Dev's site records and identity counters; it copies no Precedent or patch note data, - ownership or ACLs. Schema creation and privileged restore use trusted Dev tooling, not branch code. + creates the site schema using reviewed migrations under the preview owner. It lists Dev's public tables, + leaving out Precedent (`precedent_*`), patch notes (`patchnote_*`) and `schema_migrations`, and a data-only + custom dump/restore copies those tables' rows and identity counters. No ownership or ACLs are copied. Schema + creation and privileged restore use trusted Dev tooling; branch code takes no part. - **Scrub:** before startup, the helper clears sessions, user sessions, Discord OAuth/link state, preview sign-in tickets, Microsoft link state, and Patreon tokens, OAuth and pending-link state, including creator credentials. It revokes redemption and war declaration codes and clears redemption-code plaintext, retaining rows needed by @@ -78,8 +73,7 @@ the `ps-preview-router` Caddy container (`127.0.0.1:8090`, behind the `*.tfminec - **Publication:** `up` reports success and sets the expiry only after the public UI and API pass health checks. Host preview and Dev deployment locks prevent cloning while Dev is being deployed or migrated. -Before enabling native previews, a Dev administrator must explicitly run the isolation setup from the trusted -Dev backend: +Preview isolation needs a one-time administrator setup, run from the trusted Dev backend: ```bash sudo -n -u tfmc /usr/local/sbin/ps-compose dev exec -T backend python3 -m src.database.preview setup @@ -87,8 +81,8 @@ sudo -n -u tfmc /usr/local/sbin/ps-compose dev exec -T backend python3 -m src.da This revokes `PUBLIC CONNECT` on connectable databases in Dev's PostgreSQL cluster, preserving database-owner access and existing direct grants. Provisioning grants each runtime preview login access only to its own -database and refuses creation if either login can connect elsewhere. Review this cluster-wide change in the -[native operations procedure](docs/ops/postgresql.md) before enabling the workflow variable. +database and refuses creation if either login can connect elsewhere. See +[preview provisioning](docs/ops/postgresql.md#preview-provisioning). Inspect or stop previews by hand: @@ -118,56 +112,45 @@ the owner of both checkouts. Deploys to the same site run one at a time. `site-deploy`: -1. Refuses commits not on `origin/main` (exit 4) and, on www, a checkout with uncommitted changes to tracked files - (exit 1). On dev, `git checkout --detach` keeps the expected local edits and refuses if they would be overwritten. +1. Refuses commits not on `origin/main` or from before the move to PostgreSQL (no + `backend/src/database/operations.py`), both with exit 4, and, on www, a checkout with uncommitted changes to + tracked files (exit 1). On dev, `git checkout --detach` keeps the expected local edits and refuses if they would be overwritten. 2. Selects private `/home/tfmc/.site-images-.json` image tags for the exact requested revision and builds both services using Docker's cache before stopping the backend for a forward deployment. The overlay is applied after the site's private Compose override. Both running container image IDs must match those revision tags before the deployment can pass its health checks, including an already-deployed shortcut. 3. Starts/checks the `db` service and writes a private custom-format PostgreSQL backup as - `postgres.before-deploy--.dump`, publishing it by an atomic rename after `pg_dump` succeeds. - Only these PostgreSQL deploy dumps are pruned to the newest 10. While SQLite is still authoritative, it also - takes a final consistent `province.db.before-deploy--` backup with SQLite's backup API after - stopping the backend. -4. For native target code, runs - [`src.database.operations`](https://github.com/TF-Minecraft/ProvinceSystem/blob/main/backend/src/database/operations.py) - from the reviewed target image while the backend is stopped. The first cutover archives the original SQLite - database and legacy schema/columns under the data mount's `migration-rollback/`, copies and verifies site - records, then atomically publishes `.postgres-authority.json`. An interrupted committed copy must verify - exactly before adoption; an existing populated target is not erased. Later deploys check the marked database - identity and migration history and apply native migrations without recopying SQLite. Studio provisioning - requires a separate restricted `DATABASE_STUDIO_URL` login for the same database. + `postgres.before-deploy--.dump`, publishing it by an atomic rename after `pg_dump` succeeds, + and keeps the newest 10. +4. Runs [`src.database.operations`](https://github.com/TF-Minecraft/ProvinceSystem/blob/main/backend/src/database/operations.py) + `prepare` and then `studio` from the target image while the backend is stopped. `prepare` checks that the + configured database is the one named by the data mount's `.postgres-authority.json` marker, then applies + pending migrations. `studio` provisions the Database Studio login, which must be a separate restricted + `DATABASE_STUDIO_URL` login for the same database ([deploy steps](docs/ops/postgresql.md#deploy-steps)). 5. Starts both services without rebuilding, verifies their image IDs, then waits up to 3 minutes for both health checks. - On failure it stops the backend, restores the previous code, rebuilds and checks again, then exits 5. When - returning to legacy code after a cutover, it first reverse-exports and verifies PostgreSQL into SQLite, - including writes accepted during the health-check window. A requested native-to-legacy downgrade likewise - exports with the current native image before checking out and starting old code. If reverse export fails, - keep the backend stopped for operator inspection. If a legacy target accepts SQLite writes and then fails, - the tracked offline transition verifies the unchanged PostgreSQL baseline before importing those accepted - writes back in one transaction. Independently changed PostgreSQL targets are refused. Native code rollback - retains PostgreSQL data and uses retained, checksum-verified definitions of already-applied migrations; - no deploy automatically restores a dump. + On failure it stops the backend, restores the previous code, rebuilds, runs `prepare` and `studio` again and + checks health, then exits 5. If any of those steps fails, the backend stays stopped. Returning to an older + revision keeps PostgreSQL data and uses retained, checksum-verified definitions of migrations already applied; + no deploy restores a dump. 6. After a healthy www deploy that changes `backend/src/patchnotes/`, runs `sudo tfmc-patchnotes-update ` so the [patch note jobs](docs/ops/patch-notes-jobs.md) use the same code. A failure there is logged and does not fail the deploy. Each run appends a line to `/home/tfmc/site-deploy.log`. The private `/home/tfmc/.site-deployed-` record tracks the last healthy revision separately from Git HEAD, -so a retry resumes an interrupted build/copy instead of reporting an undeployed checkout as successful. +so a retry resumes an interrupted build or migration instead of reporting an undeployed checkout as successful. Revision-tagged images carry site/revision labels. Weekly image pruning waits for both deployment locks and removes unused site images only after 30 days; images referenced by containers remain protected. -Keep original SQLite copies, consistent cutover backups, legacy archives and the one-off import/rollback schema -for the full rollback window of **at least 30 days**. These are outside the newest-10 PostgreSQL dump pruning; -the deployment script does not automatically delete them. Follow the -[copy and rollback procedure](docs/ops/postgresql.md) before changing authority or removing retained material. +`province.db` files and the `migration-rollback/` folder left from before the move to PostgreSQL can stay on the +host as plain backups. No deploy or tool reads them. ### Rolling back www -Run **Deploy** with site `www` and an older commit on `main` only when explicitly requested. A native code rollback -keeps current PostgreSQL data. A downgrade to legacy code uses the stopped-backend reverse export above so -post-cutover writes survive. A full PostgreSQL restore is a separate manual operation and also affects Precedent -and patch notes; use the [native rollback procedure](docs/ops/postgresql.md) to choose the appropriate recovery. +Run **Deploy** with site `www` and an older commit on `main` only when explicitly requested. The commit must be from +after the move to PostgreSQL. The older code runs against the current data and schema, so check that the migrations +added since that commit leave a schema it can use. A full PostgreSQL restore is a separate manual operation that +also affects Precedent and patch notes; see [backups and restores](docs/ops/postgresql.md#backups-and-restores). ### Deploying by hand @@ -210,9 +193,9 @@ and waits until it is healthy. Each site's password is `POSTGRES_PASSWORD` in it `DATABASE_URL=postgresql://provincesystem:@db:5432/provincesystem`. Dev's compose file builds `DATABASE_URL` from the same password, and its override repeats it because its `environment: !override` replaces the list. `DATABASE_STUDIO_URL` must address that same database using a separate restricted login; it must not reuse the -owner credentials. Initial restriction of `PUBLIC` large-object constructor privileges is an explicit, -reviewed administrator step in [Native PostgreSQL operations](docs/ops/postgresql.md); preserve owner access -and review existing direct grants. Deploys refresh the Studio grants after migrations and refuse unsafe roles. +owner credentials. The deploy's `studio` step restricts `PUBLIC` large-object constructor privileges while +keeping owner access and direct grants, refreshes the Studio grants after migrations and refuses unsafe roles +([deploy steps](docs/ops/postgresql.md#deploy-steps)). PostgreSQL reads `POSTGRES_PASSWORD` only when it creates the volume; to change the password later, run `ALTER USER provincesystem PASSWORD '…'` in `psql` and update both files. @@ -220,8 +203,8 @@ PostgreSQL reads `POSTGRES_PASSWORD` only when it creates the volume; to change sudo -u tfmc ps-compose main exec db psql -U provincesystem provincesystem # a shell ``` -Do not treat copying the asset mounts as a database import. Use the -[native operations guide](docs/ops/postgresql.md) for copies, restores, validation and rollback. +Copying the asset mounts does not copy site records. See [PostgreSQL storage](docs/ops/postgresql.md) for +backups and restores. ## Keys and secrets diff --git a/projects/ProvinceSystem/STAGING.md b/projects/ProvinceSystem/STAGING.md index e96fb87..92ca513 100644 --- a/projects/ProvinceSystem/STAGING.md +++ b/projects/ProvinceSystem/STAGING.md @@ -8,7 +8,7 @@ All site records live in the stack's PostgreSQL `db` service (`pgvector/pgvector `tfmc-staging_db` named volume. The backend waits for the database health check and uses `DATABASE_URL`. The clone's `backend/src/data`, `input`, `defines` and `output` directories are mounted for assets and generated files; copying these directories does not copy site records. CoreProtect remains an external SQLite source. -Use [Native PostgreSQL operations](docs/ops/postgresql.md) for reviewed data copies, migrations and rollback. +See [PostgreSQL storage](docs/ops/postgresql.md) for migrations, backups and restores. **Product docs:** [docs/README.md](docs/README.md) · **Flows:** [docs/flows/journeys.md](docs/flows/journeys.md) · **Dev flags:** [docs/ops/dev-config.md](docs/ops/dev-config.md) @@ -26,8 +26,7 @@ do not run `git reset --hard` there. If a manual deployment is necessary, use `sudo -u tfmc site-deploy deploy dev ` on the TF server. Do not run the local staging scripts on the TF server: their Compose project name, `tfmc-staging`, is also used -by managed Dev. Main (`www`) deploys only through an explicitly requested **Actions → Deploy → Run workflow**; -local staging does not cut it over. +by managed Dev. Main (`www`) deploys only through an explicitly requested **Actions → Deploy → Run workflow**. ## Start a separate local stack @@ -64,8 +63,9 @@ Stop later: ``` Stopping removes containers and the stack network, but retains the database volume and mounted assets. -Removing the volume deletes the local site records. For a copy of another site's data, follow the -[native copy procedure](docs/ops/postgresql.md), which handles database records separately from assets. +Removing the volume deletes the local site records. To load another site's records, restore one of its +PostgreSQL deploy dumps into this stack's database with `pg_restore` +([backups and restores](docs/ops/postgresql.md#backups-and-restores)); copy asset directories separately. ## Same-host integrations @@ -110,7 +110,7 @@ configuration for a shared or longer-lived stack. `POSTGRES_PASSWORD` comes from it when initializing the volume, so changing the variable alone does not rotate an existing database password. Do not copy host Dev administrator credentials into local or branch-preview containers. If testing Database Studio, provision a separate restricted login and set `DATABASE_STUDIO_URL` for the same local database as -described in the [native guide](docs/ops/postgresql.md). +described in the [PostgreSQL guide](docs/ops/postgresql.md). ## Auth hardening diff --git a/projects/ProvinceSystem/docs/README.md b/projects/ProvinceSystem/docs/README.md index 547848e..5d60266 100644 --- a/projects/ProvinceSystem/docs/README.md +++ b/projects/ProvinceSystem/docs/README.md @@ -57,5 +57,5 @@ Schema assets: [assets/map-export-schema.json](assets/map-export-schema.json) (S - Deployment and QA checklists: [STAGING.md](../STAGING.md) - Preview and deployment workflow: [DEPLOY.md](../DEPLOY.md) -- Database configuration, migrations and rollback: [PostgreSQL](ops/postgresql.md) +- Database configuration, migrations, backups and restores: [PostgreSQL](ops/postgresql.md) - Update reference: [UPDATE.md](../UPDATE.md) diff --git a/projects/ProvinceSystem/docs/architecture.md b/projects/ProvinceSystem/docs/architecture.md index 5125abe..b4aed95 100644 --- a/projects/ProvinceSystem/docs/architecture.md +++ b/projects/ProvinceSystem/docs/architecture.md @@ -100,7 +100,7 @@ App Router `loading.tsx` boundaries cover route navigation, with outlines suited | Map | `map_routes`, `data_routes`, `file_routes`, `regen_routes` | | Skins | codes, redeem, upload, staff review, plugin approved | | Drinks | redeem, submit, catalog, staff review | -| Characters | session, create, roster, wardrobe, `rpc_player_meta` | +| Characters | session, create, roster, wardrobe, web entitlements (`PUT /characters/plugin/rpc-player-meta`) | | Profile | dashboard, skin/drink start, code sign-in (`POST /profile/redeem`), code wardrobe (`GET /profile/wardrobe`) | | Identity | Discord link, guild grace | | Diagnostic | Server ping, resource pack copy, speed test, player reports | @@ -154,7 +154,7 @@ Map and cosmetics share Docker and process space but not business logic. `DATABASE_URL` points to the site's Compose `db` service and persistent volume. Startup applies checked, ordered migrations from `backend/src/database/migrations/`. -Configuration, backups and rollback: [PostgreSQL storage](ops/postgresql.md). +Configuration, migrations, backups and restores: [PostgreSQL storage](ops/postgresql.md). | Table family | Purpose | |--------------|---------| @@ -162,6 +162,8 @@ Configuration, backups and rollback: [PostgreSQL storage](ops/postgresql.md). | `drink_submissions`, `drink_textures`, `drink_catalog` | Drinks workflow | | `discord_links`, `discord_link_codes` | Identity | | Character sessions, roster, wardrobe, wills and catalog | Characters API | +| `web_entitlements` | Per-realm rank entitlements from TFMCWeb: character limit, name colours, wardrobe slots, skin kinds, drink perks, permission flags | +| `character_account_meta` | Per-account data from RPCharacters' roster push: 18+ answer, account creation time, kit cooldowns | | `users`, `user_sessions`, OAuth states | Website accounts and sign-in | | Patreon and LuckPerms tables | Entitlement state and plugin outboxes | | Chronicle and Ledger index tables | Map history indexes | @@ -174,7 +176,7 @@ backend/src/output/{map}/ # generated map assets ``` Map assets remain under `backend/src/output/{map}/…`. PNGs stay on disk for serving and pack export. -CoreProtect is a separate read-only SQLite source; retained `province.db` copies are offline rollback resources. +CoreProtect is a separate read-only SQLite source. ## Auth model diff --git a/projects/ProvinceSystem/docs/characters/creator.md b/projects/ProvinceSystem/docs/characters/creator.md index 50ffda8..ec1125f 100644 --- a/projects/ProvinceSystem/docs/characters/creator.md +++ b/projects/ProvinceSystem/docs/characters/creator.md @@ -81,7 +81,9 @@ and RPCharacters trims name colours to the rank's limit at login. ProvinceSystem runs the same name, race age, trait and clue checks on `POST /characters` ([creation_rules.py](https://github.com/TF-Minecraft/ProvinceSystem/blob/main/backend/src/characters/creation_rules.py)), so the wizard shows the problem instead of the create being rejected later. Pending creates count -towards the slot limit, and the create page says so when every slot is taken. In the wizard, a trait +towards the slot limit, and the create page says so when every slot is taken. The website takes the limit from +the player's TFMCWeb entitlements (`web_entitlements`); a player TFMCWeb has never synced gets the catalogue's +default limit. In the wizard, a trait whose requirement is missing cannot be picked, and picks are dropped when their stage closes or their requirement goes, for example evil archetypes once the personality that unlocks them is removed. diff --git a/projects/ProvinceSystem/docs/cosmetics/drinks.md b/projects/ProvinceSystem/docs/cosmetics/drinks.md index 4b39334..0a79207 100644 --- a/projects/ProvinceSystem/docs/cosmetics/drinks.md +++ b/projects/ProvinceSystem/docs/cosmetics/drinks.md @@ -66,10 +66,9 @@ sequenceDiagram - `drink_submissions` - recipe JSON, status, player UUID, optional texture - `drink_textures` - owner UUID, CMD, IA id, refcount, png path - `drink_catalog` - ingredients allowlist + category labels + effects blacklist -- `drink_player_meta` - legacy `allow_drink_texture` + `name_colour_stops` (fallback when `rpc_player_meta` has no row) - `drink_notifications` - staff/bot outbox -**API (`/drinks`):** redeem · submit · catalog · staff pending/approve/deny · plugin catalog. Drink entitlements come from TFMCWeb via `PUT /characters/plugin/rpc-player-meta`; `PUT /drinks/plugin/player-meta` is deprecated. +**API (`/drinks`):** redeem · submit · catalog · staff pending/approve/deny · plugin catalog. Drink entitlements (`allow_drink_texture`, `allow_drink_message`, name colours) come from TFMCWeb via `PUT /characters/plugin/rpc-player-meta` and are stored in `web_entitlements`. ## Plugin surface (DrinkBuilder) diff --git a/projects/ProvinceSystem/docs/cosmetics/skins.md b/projects/ProvinceSystem/docs/cosmetics/skins.md index 968d479..252584a 100644 --- a/projects/ProvinceSystem/docs/cosmetics/skins.md +++ b/projects/ProvinceSystem/docs/cosmetics/skins.md @@ -81,7 +81,7 @@ Upload requires a prior **Discord link** for that UUID ([identity/tfmcweb.md](.. ## Skin-upload entitlements (TFMCWeb) -Configured in TFMCWeb `config.yml` `player-meta.skins`; pushed on join/reload via `PUT /characters/plugin/rpc-player-meta`. `PUT /skins/plugin/player-meta` is deprecated. +Configured in TFMCWeb `config.yml` `player-meta.skins`; pushed on join/reload via `PUT /characters/plugin/rpc-player-meta` and stored in `web_entitlements`. | Rank | Cooldown | Kinds (additive inherit) | Armor 3D helmet | |------|----------|--------------------------|-----------------| @@ -205,7 +205,7 @@ the `/token create` rules: | Check | Refusal | |-------|---------| | Discord link in good standing | `discord` | -| Rank perks synced from TFMCWeb (`rpc_player_meta`) | `join_server` | +| Rank perks synced from TFMCWeb (`web_entitlements`) | `join_server` | | `skin_token_cooldown_days` is not -1 | `rank` | | Last skin or drink code is older than the cooldown | `cooldown`, with `next_at` | diff --git a/projects/ProvinceSystem/docs/identity/auth-security.md b/projects/ProvinceSystem/docs/identity/auth-security.md index 6d6227b..b2703c0 100644 --- a/projects/ProvinceSystem/docs/identity/auth-security.md +++ b/projects/ProvinceSystem/docs/identity/auth-security.md @@ -201,7 +201,10 @@ their views and their Profile sessions; the role change or revocation is the aud | Profile / map staff | `profile` scope session from `/profile` redeem or a linked Discord account ([Profile without a code](#profile-without-a-code)); carries `player_uuid` and `realm_id` for permission checks | | No website passwords | Codes are UUID-bound and not shareable by design | -Codes are **hashed at rest** (SHA-256). Plaintext shown once in-game at mint. +Codes are **hashed at rest** (SHA-256). Plaintext shown once in-game at mint. The `codes` row keeps the +plaintext only while the code is open, for the plugin list of active codes (`GET /skins/plugin/codes/active`). +Redeeming a profile code, using a skin or drink code for a submission, and revoking a code clear it in the same +update. The backend's hourly housekeeping loop clears it once a code expires unused. ### Scope enforcement @@ -222,7 +225,7 @@ Invalid or expired tokens return **401**. Wrong scope returns **403**. ### Staff-only maps - Map entries in `maps.yml` declare `public: false` and optional per-map `staff_permission`. -- `ensure_map_access()` returns **403** without a valid profile Bearer session and matching permission from `rpc_player_meta` / LuckPerms sync. +- `ensure_map_access()` returns **403** without a valid profile Bearer session and matching permission from `web_entitlements` (TFMCWeb's LuckPerms sync). ### Map title editor write diff --git a/projects/ProvinceSystem/docs/identity/tfmcweb.md b/projects/ProvinceSystem/docs/identity/tfmcweb.md index 200e437..28583f8 100644 --- a/projects/ProvinceSystem/docs/identity/tfmcweb.md +++ b/projects/ProvinceSystem/docs/identity/tfmcweb.md @@ -124,7 +124,7 @@ Rules: - Codes consumed on submit (skin/drink) or redeem (profile). - **Shared mint cooldown (`skin` + `drink`):** owned by **TFMCWeb** only. Profile is not in the shared family. - **Command gate:** LP `tfmcweb.token.create`; staff perm `tfmcweb.token.create.staff`. -- Skin and drink **upload** entitlements: TFMCWeb `player-meta.*` → `PUT /characters/plugin/rpc-player-meta`. +- Skin and drink **upload** entitlements: TFMCWeb `player-meta.*` → `PUT /characters/plugin/rpc-player-meta`, stored per realm in `web_entitlements`. That row is the website's only source for a player's character limit, name colours and wardrobe slots. A player TFMCWeb has never synced gets the creation catalogue's default character limit. - TFMCWeb sends player meta on join, on config reload and with `/web syncmeta [player]`. About a second after LuckPerms recalculates an online player's permissions, or a group's (which checks every online player), it sends again, so a rank granted while they play (Patreon, staff panel or `/lp`) reaches the website without a rejoin. These LuckPerms-triggered sends skip a player whose meta has not changed since the last send, because LuckPerms recalculates every online player and group on each `sync-minutes` storage sync. Sends for one player are attempted one at a time, newest meta last. A timed-out send can still be applied by the website after a newer one, so after a failed send TFMCWeb resends that player's meta at every recalculation until one succeeds at least 10 minutes after the last failure. This repair is best effort and relies on recalculations happening, which the periodic sync provides. After restoring the website database, run `/web syncmeta` on each server: TFMCWeb cannot see the restore and would otherwise skip online players whose meta has not changed until they rejoin. ## Discord gate + RPCharacters freeze @@ -186,7 +186,7 @@ Set `apply-ranks: true` on exactly one server because LuckPerms storage is share ## Realm gateway -Per-realm HTTP gateway, `rpc_player_meta` sync, realm-scoped token policy, and scoped data queues are **shipped**. Map staff permission `tfmc.map.staff` arrives via player meta for staff map and editor gates. +Per-realm HTTP gateway, `web_entitlements` sync, realm-scoped token policy, and scoped data queues are **shipped**. Map staff permission `tfmc.map.staff` arrives via player meta for staff map and editor gates. ### Site-wide snapshots are owned by the primary server diff --git a/projects/ProvinceSystem/docs/map/title-editor.md b/projects/ProvinceSystem/docs/map/title-editor.md index 95e41fc..237489d 100644 --- a/projects/ProvinceSystem/docs/map/title-editor.md +++ b/projects/ProvinceSystem/docs/map/title-editor.md @@ -85,7 +85,7 @@ flowchart TD Same as staff map viewer: 1. `/profile` redeem → Bearer session. -2. `permission_flags["tfmc.map.staff"]` on `rpc_player_meta` for map `realm_id`. +2. `permission_flags["tfmc.map.staff"]` on `web_entitlements` for map `realm_id`. 3. Editor province data and regen-trigger endpoints return **403** without permission. SimpleFactions `POST /data/upload/{mode}` (including title tiers) is not staff Bearer; it uses the same internal IP check as plugin regen. The editor UI does not write title tiers through the API; staff changes leave as a ZIP (see [Offline export](#offline-export)). `POST /{map}/editor/regen/{type}` starts a staff regeneration. diff --git a/projects/ProvinceSystem/docs/ops/postgresql.md b/projects/ProvinceSystem/docs/ops/postgresql.md index 573aca7..3cbf146 100644 --- a/projects/ProvinceSystem/docs/ops/postgresql.md +++ b/projects/ProvinceSystem/docs/ops/postgresql.md @@ -5,14 +5,13 @@ Characters, Patreon, LuckPerms snapshots, Chronicle, Ledger and staff tools. `DA startup. CoreProtect remains an external SQLite database opened read-only. PNGs, uploaded assets and map history files remain on disk. -Source: [database package](https://github.com/TF-Minecraft/ProvinceSystem/tree/main/backend/src/database), -[copy tool](https://github.com/TF-Minecraft/ProvinceSystem/blob/main/backend/src/scripts/tools/copy_province_db.py). +Source: [database package](https://github.com/TF-Minecraft/ProvinceSystem/tree/main/backend/src/database). See [previews and deploys](../../DEPLOY.md) for host paths and Actions, and [Database Studio](database-studio.md) for its separate restricted login. ## Runtime and configuration -Each site's existing `pgvector/pgvector:pg17` Compose `db` service stores site records alongside Precedent and +Each site's `pgvector/pgvector:pg17` Compose `db` service stores site records alongside Precedent and patch notes. The named PostgreSQL volume and asset mounts are separate persistence boundaries; copying `backend/src/data/` does not copy site records. Main and Dev use separate databases and volumes. @@ -28,119 +27,74 @@ Keep connection strings in protected environment files. Do not print them in log command arguments. The host's Dev `environment: !override` requires its private override to include the Studio connection; adding it only to the base Compose file is insufficient. -The native connection context borrows one pooled connection for a transaction, committing on success and +The connection context borrows one pooled connection for a transaction, committing on success and rolling back on error. Rows are dictionaries. UTC timestamps, JSONB, booleans and identity columns use their PostgreSQL types; timestamp serialization and public JSON formats stay compatible with callers. Session settings bound queries and locks. Domain advisory locks coordinate checks that span multiple rows or the absence of a row; file guards coordinate wardrobe and map assets across workers. +An hourly housekeeping loop in the backend, which also runs at startup, clears data that has outlived its use. +It currently clears the plaintext of login codes that expired unused +([opaque Bearer sessions](../identity/auth-security.md#opaque-bearer-sessions)). + ## Schema migrations Ordered SQL files in `backend/src/database/migrations/` are recorded with checksums in `schema_migrations`. Startup applies pending migrations under a database advisory lock and then seeds required singleton rows. -Edited or reordered applied migrations are refused. Native deployment and startup retain verbatim SQL files -in the private data mount's `postgres-migrations/`. When restoring older compatible native code, the migrator -uses checksum-verified retained definitions only for newer versions already applied to that database; it +Edited or reordered applied migrations are refused. Deployment and startup retain verbatim SQL files +in the private data mount's `postgres-migrations/`. When restoring older code, the migrator uses +checksum-verified retained definitions only for newer versions already applied to that database; it never applies a newer retained file absent from the selected code. Missing or changed definitions still -refuse startup. Add a new numbered migration for a schema change; -do not edit an applied file or restore the removed SQLite startup upgrader. During the rollback window, -a schema change must also preserve the reviewed import/export contract and include compatible tool updates -and round-trip proof. The one-off copier refuses unexpected schema drift rather than exporting incomplete -rows; defer a schema change if its rollback representation has not been verified. - -The one-off copier migrates **without seeds**, because its target site tables must be empty. Precedent and -patch-note migrations remain separate and retain their existing data. Their active PostgreSQL driver -dependencies remain required. - -## Per-site cutover - -Use **Actions → Deploy** and the reviewed host tooling. A merge to `main` deploys Dev after Build passes. -Main/www changes only through an explicit manual `www` deployment; a Dev cutover does not migrate Main. - -Before a site's first native deployment: - -1. Install the merged `site-deploy`, `ps-compose` and `ps-preview` host scripts. Preserve private Compose - overrides and back them up before editing. -2. Keep the existing PostgreSQL database/volume and owner URL. Configure a generated separate - `DATABASE_STUDIO_URL` for that same database. Do not pass the owner connection to Studio or preview images. -3. Verify the SQLite source and exercise a copy and reverse export on an isolated PostgreSQL target. - Preserve the report privately; report counts and checksums rather than player rows or tokens. -4. Deploy through Actions. The target backend image is built before the forward downtime. The host stops - the backend, takes a full custom PostgreSQL dump and final consistent SQLite backup, and runs - `src.database.operations prepare` from the reviewed target image before normal seeded startup. -5. Provision Studio using the administrator connection. The reviewed deployment operation explicitly - restricts PUBLIC large-object constructor access while preserving owner/direct grants, then creates or - refreshes the restricted login. Start the backend only after copy and grant verification succeed. -6. Check API/UI health, database identity, row/hash verification, identities, Studio restrictions and the - integrations appropriate to that site. - -The copy validates all 61 site tables before target row writes, imports and verifies counts/canonical -hashes in a transaction, and preserves identity high-water marks, including deleted SQLite IDs. It archives -retired columns/tables with their DDL. Unrelated Precedent and patch-note tables are preserved. - -The private `data/.postgres-authority.json` records the verified target's name and migration history, -source snapshot hash and table checks. Normal deployments validate that identity and migrate without -recopying stale SQLite. If copy committed before marker publication failed, a retry adopts the target only -after exact source verification. The first copy attempt is recorded before import, so if legacy code resumes -and accepts more SQLite writes, retry can restore those writes against its recorded PostgreSQL baseline. -Inconsistent untracked nonempty targets are refused, never cleared. +refuse startup. Add a new numbered migration for a schema change; do not edit an applied file. + +Precedent and patch-note tables have their own migrations and keep their data. + +## Deploy steps + +Deploys run through **Actions → Deploy** and the host's `site-deploy` +([site deploys](../../DEPLOY.md#site-deploys)). A merge to `main` deploys Dev after Build passes; www changes +only through an explicit manual `www` deployment. With the backend stopped, `site-deploy` runs two steps from +the target image: + +```bash +python3 -m src.database.operations prepare +python3 -m src.database.operations studio +``` + +`prepare` reads the private `data/.postgres-authority.json` marker, written when the site moved to PostgreSQL. +It refuses a site without the marker, and a database whose name differs from the marker's or that no longer +holds every migration the marker records with the same checksum and time. It then applies pending migrations +without seeding. + +`studio` provisions the restricted Studio login using the administrator connection. It restricts `PUBLIC` +large-object constructor access while keeping owner and direct grants, then creates or refreshes the login. +It refuses a `DATABASE_STUDIO_URL` that points at another database or reuses the owner login. Either step +prints only its name and the exception type on failure, so URLs and row values stay out of logs, and the +backend stays stopped. `/home/tfmc/.site-deployed-` separately records the last healthy revision. Git HEAD can already point -at a target whose build or cutover was interrupted; it is not sufficient evidence of deployment success. +at a target whose build or migration was interrupted; it is not sufficient evidence of deployment success. A private `.site-images-.json` Compose overlay selects immutable tags for both services. Cached builds and running-image verification bind successful deployment to that exact revision, even after interruption. -## Standalone copy and verification +## Backups and restores -These commands are for an isolated rehearsal or an explicitly stopped site. Run from `backend/` with its -reviewed dependencies and `DATABASE_URL` supplied privately. Stop all writers before either direction. +Deployment backups are private custom PostgreSQL dumps in the site's `deploy-backups/`, published from a +`.part` file after `pg_dump` succeeds; the newest 10 are kept. These full dumps include Precedent and patch +notes. Restoring one is a separate stopped-site database restore, and no deploy restores a dump +automatically. After a restore, run `/web syncmeta` on each server so TFMCWeb resends player entitlements +([TFMCWeb](../identity/tfmcweb.md)). -```bash -python -m src.scripts.tools.copy_province_db to-postgres \ - --sqlite /private/province.snapshot.db --archive-legacy /private/legacy-archive -python -m src.scripts.tools.copy_province_db to-postgres \ - --sqlite /private/province.snapshot.db --verify-only -python -m src.scripts.tools.copy_province_db to-sqlite --sqlite /private/province.rollback.db -``` +`province.db` files and the `data/migration-rollback/` folder left from before the move to PostgreSQL can stay +on the host as plain backups. No deploy or tool reads them. -The source SQLite file is opened read-only with integrity and foreign-key checks. Import requires empty -site tables. `--verify-only` changes no rows; compare with the frozen source before opening the site to -writes. After PostgreSQL accepts new writes, that old SQLite snapshot is no longer an equality baseline. -Reverse export reads a consistent PostgreSQL snapshot and publishes a new mode-0600 SQLite file only after -verification. It refuses an existing destination. - -## Backups and rollback window - -Routine native deployment backups are private custom PostgreSQL dumps in the site's `deploy-backups/`, -published from a `.part` file after `pg_dump` succeeds; the newest 10 are retained. These full dumps include -Precedent and patch notes. Restoring one is a separate stopped-site database restore, not a code-only rollback. - -Keep the original `province.db`, consistent cutover backups, `data/migration-rollback/` snapshots and legacy -archives, and the one-off copier plus frozen `legacy_sqlite.sql` for **at least 30 days** after cutover. They -are outside newest-ten PostgreSQL pruning. Remove them only after the rollback window closes and retention -is explicitly reviewed. The frozen SQLite schema is an offline rollback resource, not a runtime migration. - -When returning to legacy code, `site-deploy` stops the backend and uses the retained native tool to export -and verify current PostgreSQL rows before atomically replacing the serving SQLite file. This includes -writes accepted since cutover and during a failed target's health-check window. Original snapshots and each -reverse export remain retained. If export fails, the backend stays stopped; do not start legacy code against -the old SQLite copy. - -The private `data/.postgres-rollback-transition.json` records SQLite authority and the immutable, verified -export baseline before legacy code starts. If that legacy release accepts writes and then fails health, -recovery stops it, snapshots SQLite and imports its accepted writes back only after PostgreSQL exactly -matches a recorded baseline, including all identity counters. All 61 site tables are locked and replaced -in one transaction; unrelated tables remain untouched and outside foreign keys refuse replacement. -Snapshots and import attempts remain recorded so a retry after commit but before marker publication can -verify and continue. Unexpected PostgreSQL writes are refused instead of overwritten. - -A rollback between native revisions keeps PostgreSQL data and uses the retained, checksum-verified SQL -definitions described above. These must be compatible with the previous application release; retaining a -file does not make a destructive schema change compatible. No rollback restores a full dump automatically. +Deploying an older revision keeps PostgreSQL data and uses the retained, checksum-verified SQL +definitions described above. The older code must work with the current schema; retaining a file does not +make a destructive schema change compatible. ## Preview provisioning -After Dev's verified cutover, explicitly run the trusted Dev administrator setup: +Preview isolation needs a one-time administrator setup on Dev's cluster: ```bash sudo -n -u tfmc /usr/local/sbin/ps-compose dev exec -T backend \ @@ -148,17 +102,16 @@ sudo -n -u tfmc /usr/local/sbin/ps-compose dev exec -T backend \ ``` This removes PUBLIC database CONNECT while retaining database-owner access and existing direct grants. -Each preview owner and Studio login then receives CONNECT only to its own database. Run this on Dev's -cluster; Main's separate cluster is unchanged. After Dev and this isolation setup pass, set the repository -variable `NATIVE_POSTGRES_PREVIEWS_READY=true` for a controlled Preview workflow run on a reviewed native -branch. Verify creation, redeployment and teardown, including removal of its database and roles. Keep the -variable enabled only if that lifecycle passes; disable it again on any failure. - -The trusted controller creates the schema under the preview owner, clones only site-table data, scrubs -credentials and validates Studio before starting branch images. Its credentials stay inside trusted Dev -tooling. Failed creation, redeploy, expiry, branch deletion and orphan cleanup retain their metadata until -the marked database and roles have been removed. Detailed environment, networking and lifecycle behavior -is in [DEPLOY.md](../../DEPLOY.md#branch-previews). +Each preview owner and Studio login then receives CONNECT only to its own database. Main's separate +cluster is unchanged. The repository variable `NATIVE_POSTGRES_PREVIEWS_READY=true` lets previews deploy; +set it to `false` if preview creation, redeployment or teardown stops working. + +The trusted controller creates the schema under the preview owner, clones site-table data, scrubs +credentials and validates Studio before starting branch images. The site tables are Dev's public tables +except Precedent (`precedent_*`), patch notes (`patchnote_*`) and `schema_migrations`. Its credentials stay +inside trusted Dev tooling. Failed creation, redeploy, expiry, branch deletion and orphan cleanup retain +their metadata until the marked database and roles have been removed. Detailed environment, networking and +lifecycle behaviour is in [DEPLOY.md](../../DEPLOY.md#branch-previews). ## Tests and API smokes @@ -173,7 +126,7 @@ python -m pytest -q python scripts/character_session_smoke.py python scripts/character_ingest_smoke.py python scripts/creation_catalog_smoke.py -python scripts/rpc_player_meta_smoke.py +python scripts/web_entitlements_smoke.py python scripts/skins_e2e_smoke.py python scripts/lore_items_smoke.py ``` diff --git a/projects/RPCharacters/docs/realm-wipe.md b/projects/RPCharacters/docs/realm-wipe.md index 0706ba1..93f76c7 100644 --- a/projects/RPCharacters/docs/realm-wipe.md +++ b/projects/RPCharacters/docs/realm-wipe.md @@ -17,7 +17,7 @@ Each realm has its own website: Dev (realm `dev`) uses the dev site, and Main (r | Realm | Read from the TFMCWeb gateway (`realm.id`), same as pending ingest and catalog sync. Deletes site tables for **this** `realm_id` only. Aborts if TFMCWeb cannot supply a realm id. | | Confirm | `/rpcharacter wipe website` then `/rpcharacter wipe website confirm`. 30s TTL, same sender. No bare `/rpcharacter confirm`. | | Auth | Command: `rpcharacters.admin`, console allowed. HTTP: `X-Plugin-Key` via `ProvinceSystemClient`. | -| Player meta | Does **not** clear `rpc_player_meta` / `character_player_meta` (ranks, 18+, slots). | +| Player meta | Does **not** clear `web_entitlements` (rank entitlements from TFMCWeb) or `character_account_meta` (18+ answer, account age, kit cooldowns). | | Local data | Does **not** delete `plugins/RPCharacters/data/**`. In-game characters reappear on the site at their owner's next roster push. | Only one server should ingest a realm; two ingesting servers would race on pending ack. From 87b36ae304b1fb1ac8d6b3198545ef7956041dfa Mon Sep 17 00:00:00 2001 From: Ryan Barlow <7389646+ryanbarlow97@users.noreply.github.com> Date: Sat, 17 Oct 2026 12:53:13 +0000 Subject: [PATCH 2/4] docs(provincesystem): migrations only go forward prepare refuses a revision that lacks a migration the database already has, so an older-commit deploy is refused after a newer migration and a failed deploy that migrated keeps the backend stopped. Previews of branches cut before a new Dev migration need a rebase. Companion to TF-Minecraft/ProvinceSystem#267 (d839fc24). Co-Authored-By: Claude Opus 5.5 --- projects/ProvinceSystem/DEPLOY.md | 22 ++++++++-------- .../ProvinceSystem/docs/ops/postgresql.md | 25 +++++++++---------- 2 files changed, 24 insertions(+), 23 deletions(-) diff --git a/projects/ProvinceSystem/DEPLOY.md b/projects/ProvinceSystem/DEPLOY.md index 39b8479..8f64424 100644 --- a/projects/ProvinceSystem/DEPLOY.md +++ b/projects/ProvinceSystem/DEPLOY.md @@ -48,7 +48,9 @@ the `ps-preview-router` Caddy container (`127.0.0.1:8090`, behind the `*.tfminec creates the site schema using reviewed migrations under the preview owner. It lists Dev's public tables, leaving out Precedent (`precedent_*`), patch notes (`patchnote_*`) and `schema_migrations`, and a data-only custom dump/restore copies those tables' rows and identity counters. No ownership or ACLs are copied. Schema - creation and privileged restore use trusted Dev tooling; branch code takes no part. + creation and privileged restore use trusted Dev tooling; branch code takes no part. The schema follows Dev's + migrations, so once Dev has a new migration, a preview of a branch cut before it fails its migration check + until the branch is rebased onto `main`. - **Scrub:** before startup, the helper clears sessions, user sessions, Discord OAuth/link state, preview sign-in tickets, Microsoft link state, and Patreon tokens, OAuth and pending-link state, including creator credentials. It revokes redemption and war declaration codes and clears redemption-code plaintext, retaining rows needed by @@ -108,7 +110,7 @@ the owner of both checkouts. Deploys to the same site run one at a time. | Compose target | `ps-compose main` | `ps-compose dev` | | Health checks | `127.0.0.1:3000/` and `127.0.0.1:8000/maps/accessible` | `127.0.0.1:13001/` and `127.0.0.1:18001/maps/accessible` | | Backups | `/home/tfmc/provincesystem-data/deploy-backups/` | `/home/tfmc/provincesystem-dev-data/deploy-backups/` | -| Older commit than deployed | deployed (this is a rollback) | skipped as `already deployed` | +| Older commit than deployed | deployed if it has every migration the database has ([Rolling back www](#rolling-back-www)) | skipped as `already deployed` | `site-deploy`: @@ -124,14 +126,14 @@ the owner of both checkouts. Deploys to the same site run one at a time. and keeps the newest 10. 4. Runs [`src.database.operations`](https://github.com/TF-Minecraft/ProvinceSystem/blob/main/backend/src/database/operations.py) `prepare` and then `studio` from the target image while the backend is stopped. `prepare` checks that the - configured database is the one named by the data mount's `.postgres-authority.json` marker, then applies - pending migrations. `studio` provisions the Database Studio login, which must be a separate restricted + configured database is the one named by the data mount's `.postgres-authority.json` marker and refuses a + revision that lacks a migration the database already has, then applies pending migrations. `studio` provisions the Database Studio login, which must be a separate restricted `DATABASE_STUDIO_URL` login for the same database ([deploy steps](docs/ops/postgresql.md#deploy-steps)). 5. Starts both services without rebuilding, verifies their image IDs, then waits up to 3 minutes for both health checks. On failure it stops the backend, restores the previous code, rebuilds, runs `prepare` and `studio` again and - checks health, then exits 5. If any of those steps fails, the backend stays stopped. Returning to an older - revision keeps PostgreSQL data and uses retained, checksum-verified definitions of migrations already applied; - no deploy restores a dump. + checks health, then exits 5. If any of those steps fails, the backend stays stopped. When the failed revision + had already applied a new migration, `prepare` refuses the previous revision and the log records + `failed: rollback database; backend kept stopped`; deploy a fix forward. No deploy restores a dump. 6. After a healthy www deploy that changes `backend/src/patchnotes/`, runs `sudo tfmc-patchnotes-update ` so the [patch note jobs](docs/ops/patch-notes-jobs.md) use the same code. A failure there is logged and does not fail the deploy. @@ -147,9 +149,9 @@ host as plain backups. No deploy or tool reads them. ### Rolling back www -Run **Deploy** with site `www` and an older commit on `main` only when explicitly requested. The commit must be from -after the move to PostgreSQL. The older code runs against the current data and schema, so check that the migrations -added since that commit leave a schema it can use. A full PostgreSQL restore is a separate manual operation that +Run **Deploy** with site `www` and an older commit on `main` only when explicitly requested. Migrations only go +forward: `prepare` refuses a commit that lacks a migration the database already has, and the deploy log names the +version. Once www has a newer migration, fix forward with a new commit on `main` instead. A full PostgreSQL restore is a separate manual operation that also affects Precedent and patch notes; see [backups and restores](docs/ops/postgresql.md#backups-and-restores). ### Deploying by hand diff --git a/projects/ProvinceSystem/docs/ops/postgresql.md b/projects/ProvinceSystem/docs/ops/postgresql.md index 3cbf146..4919048 100644 --- a/projects/ProvinceSystem/docs/ops/postgresql.md +++ b/projects/ProvinceSystem/docs/ops/postgresql.md @@ -41,11 +41,10 @@ It currently clears the plaintext of login codes that expired unused Ordered SQL files in `backend/src/database/migrations/` are recorded with checksums in `schema_migrations`. Startup applies pending migrations under a database advisory lock and then seeds required singleton rows. -Edited or reordered applied migrations are refused. Deployment and startup retain verbatim SQL files -in the private data mount's `postgres-migrations/`. When restoring older code, the migrator uses -checksum-verified retained definitions only for newer versions already applied to that database; it -never applies a newer retained file absent from the selected code. Missing or changed definitions still -refuse startup. Add a new numbered migration for a schema change; do not edit an applied file. +Edited or reordered applied migrations are refused, and deployment and startup keep verbatim copies of the +applied files in the private data mount's `postgres-migrations/`. Migrations only go forward: a deploy refuses a +revision that lacks a migration the database already has, because its code expects the older schema. Add a new +numbered migration for a schema change; do not edit an applied file. To undo a change, add another migration. Precedent and patch-note tables have their own migrations and keep their data. @@ -63,14 +62,15 @@ python3 -m src.database.operations studio `prepare` reads the private `data/.postgres-authority.json` marker, written when the site moved to PostgreSQL. It refuses a site without the marker, and a database whose name differs from the marker's or that no longer -holds every migration the marker records with the same checksum and time. It then applies pending migrations -without seeding. +holds every migration the marker records with the same checksum and time. It also refuses a revision that lacks +a migration the database already has; the deploy log names the version and says to deploy a fix forward. It then +applies pending migrations without seeding. `studio` provisions the restricted Studio login using the administrator connection. It restricts `PUBLIC` large-object constructor access while keeping owner and direct grants, then creates or refreshes the login. -It refuses a `DATABASE_STUDIO_URL` that points at another database or reuses the owner login. Either step -prints only its name and the exception type on failure, so URLs and row values stay out of logs, and the -backend stays stopped. +It refuses a `DATABASE_STUDIO_URL` that points at another database or reuses the owner login. On failure, +either step prints its name and, for a refusal, the reason; other errors print only the exception type, so URLs +and row values stay out of logs. The backend stays stopped. `/home/tfmc/.site-deployed-` separately records the last healthy revision. Git HEAD can already point at a target whose build or migration was interrupted; it is not sufficient evidence of deployment success. @@ -88,9 +88,8 @@ automatically. After a restore, run `/web syncmeta` on each server so TFMCWeb re `province.db` files and the `data/migration-rollback/` folder left from before the move to PostgreSQL can stay on the host as plain backups. No deploy or tool reads them. -Deploying an older revision keeps PostgreSQL data and uses the retained, checksum-verified SQL -definitions described above. The older code must work with the current schema; retaining a file does not -make a destructive schema change compatible. +Deploying an older revision keeps PostgreSQL data, and works only while that revision has every migration +the database has. After a newer migration, fix forward or restore a dump taken before it. ## Preview provisioning From a370cf2297670676df1eb76a473a526429e3f590 Mon Sep 17 00:00:00 2001 From: Ryan Barlow <7389646+ryanbarlow97@users.noreply.github.com> Date: Sat, 17 Oct 2026 12:57:01 +0000 Subject: [PATCH 3/4] docs(provincesystem): site-deploy checks migrations before deploying site-deploy compares schema_migrations with the target revision's migration files and refuses a revision missing one (exit 4). After a failed deploy it checks the last healthy revision the same way and keeps the backend stopped if the database is newer. Companion to TF-Minecraft/ProvinceSystem#267 (aa54b7b3). Co-Authored-By: Claude Opus 5.5 --- projects/ProvinceSystem/DEPLOY.md | 26 +++++++++++-------- .../ProvinceSystem/docs/ops/postgresql.md | 21 ++++++++++----- 2 files changed, 30 insertions(+), 17 deletions(-) diff --git a/projects/ProvinceSystem/DEPLOY.md b/projects/ProvinceSystem/DEPLOY.md index 8f64424..759a481 100644 --- a/projects/ProvinceSystem/DEPLOY.md +++ b/projects/ProvinceSystem/DEPLOY.md @@ -116,7 +116,10 @@ the owner of both checkouts. Deploys to the same site run one at a time. 1. Refuses commits not on `origin/main` or from before the move to PostgreSQL (no `backend/src/database/operations.py`), both with exit 4, and, on www, a checkout with uncommitted changes to - tracked files (exit 1). On dev, `git checkout --detach` keeps the expected local edits and refuses if they would be overwritten. + tracked files (exit 1). On dev, `git checkout --detach` keeps the expected local edits and refuses if they would + be overwritten. It then reads `schema_migrations` from the site's `db` and compares it with the target + revision's files under `backend/src/database/migrations/`. A revision missing an applied migration is refused + before anything changes (exit 4, log ` predates database migration NNNN; fix forward`). 2. Selects private `/home/tfmc/.site-images-.json` image tags for the exact requested revision and builds both services using Docker's cache before stopping the backend for a forward deployment. The overlay is applied after the site's private Compose override. Both running container image IDs must match those @@ -126,14 +129,15 @@ the owner of both checkouts. Deploys to the same site run one at a time. and keeps the newest 10. 4. Runs [`src.database.operations`](https://github.com/TF-Minecraft/ProvinceSystem/blob/main/backend/src/database/operations.py) `prepare` and then `studio` from the target image while the backend is stopped. `prepare` checks that the - configured database is the one named by the data mount's `.postgres-authority.json` marker and refuses a - revision that lacks a migration the database already has, then applies pending migrations. `studio` provisions the Database Studio login, which must be a separate restricted - `DATABASE_STUDIO_URL` login for the same database ([deploy steps](docs/ops/postgresql.md#deploy-steps)). + configured database is the one named by the data mount's `.postgres-authority.json` marker, repeats the + migration check, then applies pending migrations. `studio` provisions the Database Studio login, which must be + a separate restricted `DATABASE_STUDIO_URL` login for the same database ([deploy steps](docs/ops/postgresql.md#deploy-steps)). 5. Starts both services without rebuilding, verifies their image IDs, then waits up to 3 minutes for both health checks. - On failure it stops the backend, restores the previous code, rebuilds, runs `prepare` and `studio` again and - checks health, then exits 5. If any of those steps fails, the backend stays stopped. When the failed revision - had already applied a new migration, `prepare` refuses the previous revision and the log records - `failed: rollback database; backend kept stopped`; deploy a fix forward. No deploy restores a dump. + On failure it stops the backend and checks the last healthy revision against `schema_migrations` the same + way. If the failed deploy applied a migration that revision predates, the backend stays stopped and the log + records `failed: database is newer than the last healthy revision; backend kept stopped`; deploy a fix + forward. Otherwise it restores the previous code, rebuilds, runs `prepare` and `studio` again and checks + health, then exits 5. If any of those steps fails, the backend stays stopped. No deploy restores a dump. 6. After a healthy www deploy that changes `backend/src/patchnotes/`, runs `sudo tfmc-patchnotes-update ` so the [patch note jobs](docs/ops/patch-notes-jobs.md) use the same code. A failure there is logged and does not fail the deploy. @@ -150,9 +154,9 @@ host as plain backups. No deploy or tool reads them. ### Rolling back www Run **Deploy** with site `www` and an older commit on `main` only when explicitly requested. Migrations only go -forward: `prepare` refuses a commit that lacks a migration the database already has, and the deploy log names the -version. Once www has a newer migration, fix forward with a new commit on `main` instead. A full PostgreSQL restore is a separate manual operation that -also affects Precedent and patch notes; see [backups and restores](docs/ops/postgresql.md#backups-and-restores). +forward: `site-deploy` refuses a commit that lacks a migration the database already has (exit 4), and the log +names the version. Once www has a newer migration, fix forward with a new commit on `main`. A full PostgreSQL +restore is a separate manual operation that also affects Precedent and patch notes; see [backups and restores](docs/ops/postgresql.md#backups-and-restores). ### Deploying by hand diff --git a/projects/ProvinceSystem/docs/ops/postgresql.md b/projects/ProvinceSystem/docs/ops/postgresql.md index 4919048..64d1857 100644 --- a/projects/ProvinceSystem/docs/ops/postgresql.md +++ b/projects/ProvinceSystem/docs/ops/postgresql.md @@ -43,7 +43,8 @@ Ordered SQL files in `backend/src/database/migrations/` are recorded with checks Startup applies pending migrations under a database advisory lock and then seeds required singleton rows. Edited or reordered applied migrations are refused, and deployment and startup keep verbatim copies of the applied files in the private data mount's `postgres-migrations/`. Migrations only go forward: a deploy refuses a -revision that lacks a migration the database already has, because its code expects the older schema. Add a new +revision that lacks a migration the database already has, because its code expects the older schema +([deploy steps](#deploy-steps)). Add a new numbered migration for a schema change; do not edit an applied file. To undo a change, add another migration. Precedent and patch-note tables have their own migrations and keep their data. @@ -52,8 +53,17 @@ Precedent and patch-note tables have their own migrations and keep their data. Deploys run through **Actions → Deploy** and the host's `site-deploy` ([site deploys](../../DEPLOY.md#site-deploys)). A merge to `main` deploys Dev after Build passes; www changes -only through an explicit manual `www` deployment. With the backend stopped, `site-deploy` runs two steps from -the target image: +only through an explicit manual `www` deployment. + +Before changing anything, `site-deploy` reads `schema_migrations` from the site's `db` and compares it with +the target revision's files under `backend/src/database/migrations/`. A revision missing an applied migration +is refused with exit 4 and the log line ` predates database migration NNNN; fix forward`. After a failed +deploy it checks the last healthy revision the same way before rolling back; if the failed deploy applied a +migration that revision predates, the backend stays stopped and the log records +`failed: database is newer than the last healthy revision; backend kept stopped`. The check lives in +`site-deploy` because revisions from before it have no such check in their own code. + +With the backend stopped, `site-deploy` then runs two steps from the target image: ```bash python3 -m src.database.operations prepare @@ -62,9 +72,8 @@ python3 -m src.database.operations studio `prepare` reads the private `data/.postgres-authority.json` marker, written when the site moved to PostgreSQL. It refuses a site without the marker, and a database whose name differs from the marker's or that no longer -holds every migration the marker records with the same checksum and time. It also refuses a revision that lacks -a migration the database already has; the deploy log names the version and says to deploy a fix forward. It then -applies pending migrations without seeding. +holds every migration the marker records with the same checksum and time. It also repeats the migration check +above, naming the version in the deploy log. It then applies pending migrations without seeding. `studio` provisions the restricted Studio login using the administrator connection. It restricts `PUBLIC` large-object constructor access while keeping owner and direct grants, then creates or refreshes the login. From ac52f7eb99e0a3b28f9cd312867ddfc87c3ff39a Mon Sep 17 00:00:00 2001 From: Ryan Barlow <7389646+ryanbarlow97@users.noreply.github.com> Date: Sat, 17 Oct 2026 13:02:42 +0000 Subject: [PATCH 4/4] docs(provincesystem): match the rollback record wording Co-Authored-By: Claude Opus 5.5 --- projects/ProvinceSystem/DEPLOY.md | 2 +- projects/ProvinceSystem/docs/ops/postgresql.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/projects/ProvinceSystem/DEPLOY.md b/projects/ProvinceSystem/DEPLOY.md index 759a481..45e7000 100644 --- a/projects/ProvinceSystem/DEPLOY.md +++ b/projects/ProvinceSystem/DEPLOY.md @@ -135,7 +135,7 @@ the owner of both checkouts. Deploys to the same site run one at a time. 5. Starts both services without rebuilding, verifies their image IDs, then waits up to 3 minutes for both health checks. On failure it stops the backend and checks the last healthy revision against `schema_migrations` the same way. If the failed deploy applied a migration that revision predates, the backend stays stopped and the log - records `failed: database is newer than the last healthy revision; backend kept stopped`; deploy a fix + records `failed: last healthy revision does not fit the database (see log); backend kept stopped`; deploy a fix forward. Otherwise it restores the previous code, rebuilds, runs `prepare` and `studio` again and checks health, then exits 5. If any of those steps fails, the backend stays stopped. No deploy restores a dump. 6. After a healthy www deploy that changes `backend/src/patchnotes/`, runs `sudo tfmc-patchnotes-update ` so the diff --git a/projects/ProvinceSystem/docs/ops/postgresql.md b/projects/ProvinceSystem/docs/ops/postgresql.md index 64d1857..5669b50 100644 --- a/projects/ProvinceSystem/docs/ops/postgresql.md +++ b/projects/ProvinceSystem/docs/ops/postgresql.md @@ -60,7 +60,7 @@ the target revision's files under `backend/src/database/migrations/`. A revision is refused with exit 4 and the log line ` predates database migration NNNN; fix forward`. After a failed deploy it checks the last healthy revision the same way before rolling back; if the failed deploy applied a migration that revision predates, the backend stays stopped and the log records -`failed: database is newer than the last healthy revision; backend kept stopped`. The check lives in +`failed: last healthy revision does not fit the database (see log); backend kept stopped`. The check lives in `site-deploy` because revisions from before it have no such check in their own code. With the backend stopped, `site-deploy` then runs two steps from the target image: