feat: media library, production DB tier, and CI gaps - #110
Merged
Conversation
- nginx: hardened static `location /wp-content/` serving /mnt/cephfs/media (GET/HEAD only, 403 on script/exe extensions, nosniff + strict CSP, immutable cache) plus a dotfile deny. No PHP handler, so nothing executes. - compose: backend anchor gains MEDIA_ROOT, MEDIA_BASE_URL, and an rw bind-mount of the CephFS media tree so the upload endpoint can write. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
innodb_redo_log_capacity is a MySQL 8.0.30+ variable, not a MariaDB one. Both primary.cnf and replica.cnf set it, with a comment claiming it replaces innodb_log_file_size on MariaDB 11.x. It does not: mariadbd exits with "unknown variable" and the container crash-loops. Caught when the dry-run database inherited the setting; the production primary and replica would have failed identically on first boot. All three configs now use innodb_log_file_size. Adds a throwaway dry-run database so the CMS can be brought up before the dedicated DB hardware exists. compose.mariadb-dev.yml is an overlay on compose.cms.yml, so the node joins triangle_net and is reachable as DB_HOST=mariadb-dev. dev.cnf sizes InnoDB for Delta (~3.8 GB RAM total, shared with both CMS slots) rather than the dedicated 8 GB the production configs assume. Also: - nginx: drop `expires 30d` from the media block. It emitted a second, weaker Cache-Control alongside the explicit add_header, and CDNs disagree about which duplicate wins. - cms.env.example: add MEDIA_HOST_PATH/MEDIA_ROOT/MEDIA_BASE_URL, which compose.cms.yml already consumed but the template never listed. - README: document installing the nginx site, media serving and its failure modes, and image pruning on Delta's small root filesystem. cms.env.dryrun.example records the values verified working on Delta. CMS_REBUILD_TAXONOMY_COUNTS_ON_STARTUP must stay false against the init_schema.sql seed: the rebuild reads an articles.categories column the seed lacks, and the error is fatal. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Bringing the overlay up cold raced the backend against MariaDB and logged five "connection refused" errors before the retry loop won. Production cannot express this dependency -- the database lives on another host and the backend simply retries -- but in the dry-run overlay the database is a sibling service, so wait for its health check. Startup is now clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The tracked seed carried an older schema than wordpress-etl now emits. Most consequentially it lacked articles.categories, which RebuildTaxonomyArticleCounts selects unconditionally, so running with CMS_REBUILD_TAXONOMY_COUNTS_ON_STARTUP=true against this seed failed with "Unknown column 'categories'" and crash-looped the backend. Adds the seven columns the ETL emits that the seed lacked (creation_date, author_ids, authors, featured_img_id, categories, metadata, excerpt) and widens types to match ArticleFormatter.CMS_SCHEMA (BIGINT ids, LONGTEXT text columns). Retains focus_keyword, meta_description and seo_title, which the CMS uses and the ETL does not emit. authors and articles_authors likewise widened to BIGINT to match AuthorFormatter and ArtAuthFormatter. articles_authors keeps AUTO_INCREMENT even though the ETL declares a bare BIGINT PRIMARY KEY: the seed's INSERTs do not supply id, so the ETL's exact DDL would reject every row. Verified by loading the full seed into a scratch database and running the backend against it with the taxonomy rebuild enabled: no errors, no restarts. Note the rebuild is a no-op on this data -- the seed populates tags for 8381 articles but categories for none -- so real category counts still require an ETL-produced dataset. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…eal hosts Adds the primary/replica compose overlays, tuned .cnf files, the replication and MaxScale user init scripts, and the MaxScale readwritesplit config. Also carries the observability overlay, which was already in the tree. Tuning is sized for the hosts that were actually provisioned: 4 vCPU / 4 GB unprivileged LXC containers, not the 8-16 GB VMs originally requested. - innodb_buffer_pool_size 5G -> 1G. The entire dataset is ~92 MB, and a 5G pool on a 4 GB host prevents startup outright. - innodb_log_file_size 512M -> 256M. Ample for this write volume and keeps crash recovery fast. - Drop innodb_buffer_pool_instances, removed in MariaDB 10.6. Verified that 11.x accepts and ignores it silently rather than refusing to start, so this is a cleanup rather than a fix. - Pin bind-address to the primary's internal NIC. Ubuntu's stock 50-server.cnf binds 127.0.0.1, which leaves the primary unreachable from MaxScale, so the drop-in must also sort after it (installed as 70-triangle-primary.cnf). - Set query_classifier_cache_size explicitly. MaxScale defaults to ~15% of system memory, but inside an LXC whose cgroup limit reads "max" it sees the Proxmox host's RAM instead: it sized the cache at 9.38 GiB on a 4 GB container. Now pinned to 64M (observed 61 MiB). Note the deployed hosts run these packages natively via apt, not through the compose files, because Docker inside unprivileged LXC needs Proxmox-side nesting. MariaDB is 11.8 LTS rather than the 11.7 pinned here: deb.mariadb.org carries only LTS lines, and 11.7 is an EOL short-term release. Also documents the media library reindex, which is required at cutover because serving the migrated files is independent of listing them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Replaces the three /v1/media placeholders (which returned 501 via the
misnamed handlers.Users) with a DB-backed media library.
Server:
- New `media` table via EnsureMediaTable, following the existing runtime-schema
pattern. `path` (wp-content-relative) is the row's stable identity; `url` is
rendered through MEDIA_BASE_URL per response rather than stored, so it cannot
go stale.
- GET /v1/media (paginated, server-side search, MIME-family filter, sort),
GET /v1/media/gallery (trimmed shape for pickers), GET /v1/media/{id},
PATCH /v1/media/{id} (alt text, caption), DELETE /v1/media/{id}.
- POST /v1/media/index walks MEDIA_ROOT and adds anything missing, which is how
the migrated CephFS corpus enters the library. It skips WordPress's generated
-WxH derivatives, which would otherwise bury the library in near-duplicates,
and re-running preserves alt text already curated in the CMS.
- POST /v1/media (upload) is ported from patch/media-url-canonicalization and
now also records the library row.
- DELETE refuses with 409 if any article still references the asset, so a delete
here cannot blank out a published page's image.
Frontend:
- mediaView.tsx: multi-file upload, debounced server-side search, paging, a
detail panel for alt text/caption, and a reindex action. Upload, delete and
reindex are admin-gated. Previously the Upload and Delete buttons were inert.
- TrixEditor: drop the stale "backend not wired up" warnings, and send uploads
to apiBaseUrl() with credentials. It posted to a bare /v1/media with no
cookies, which 401s now that the endpoint is auth-gated and the API commonly
runs on a different origin.
Tests cover the storage guarantees without a database (no-clobber naming,
filename sanitising, path containment) plus DSN-gated integration tests for the
DB and HTTP layers, following the existing CMS_TEST_DSN pattern. The two
integration suites serialise on a MySQL advisory lock because they share the
`media` table and `go test ./...` runs packages in parallel.
This commit also carries in-progress polls/comments work that was already
modified in the working tree; those changes share files with the media work
(routes.go, main.go, types.go, api_responses.go) and could not be split cleanly.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two gaps in the pipeline. deploy_scripts_test.sh existed but nothing ran it, so the least reversible code in the repo — slot selection, the transactional Nginx switch and its restore-on-failure paths, the preflight checks — had a test suite that never executed on a PR. It now runs as its own CI job. The suite stubs docker, curl and nginx on PATH, so it needs no daemon, database, or privileges. rollback.sh was only invocable by SSH-ing into Delta from inside the VPN, during an incident, which is when that is hardest. rollback.yml exposes it as a manual workflow on the same self-hosted runner and trust boundary as deploy.yml (deployment code from the protected default branch). It shares the delta-production-deploy concurrency group so a rollback can never interleave with a deployment, and holds no packages permission because switching slots pulls no images. The slot input is passed through the environment rather than interpolated into the script body. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Media library (API + page), the production database tier, and two CI gaps closed.
Merge before the first self-hosted-runner deploy.
deploy.ymlchecks outdeployment code from the protected default branch, so until this lands,
maincarries neither the media library nor the hardened
deployment_preflightincommon.sh.(An earlier version of this note claimed a deploy from
mainwould overwrite theNginx site and kill media serving. That is not the case: the deploy scripts
only rewrite
active-upstreams.conf— foursetdirectives naming the activeslot's ports — and never touch
sites-enabled. The site config carrying/wp-content/is installed manually during bootstrap and is already live onDelta. Merging matters for the code, not to protect media serving.)
Media library
GET /v1/media,/v1/media/{id}, and/v1/media/gallerypreviously returned 501via a misnamed
handlers.Usersplaceholder, and the Media page's Upload and Deletebuttons were inert.
mediatable viaEnsureMediaTable, following the existing runtime-schemapattern.
path(wp-content-relative) is the row's stable identity;urlisrendered through
MEDIA_BASE_URLper response rather than stored, so it can't gostale if the media host changes.
pickers, get,
PATCHfor alt text/caption, andDELETE.POST /v1/media/indexwalksMEDIA_ROOTand adds anything missing — this is howthe migrated corpus enters the library. It skips WordPress's generated
-WxHderivatives, which would otherwise bury the library in near-duplicates, and
re-running preserves alt text already curated in the CMS.
POST /v1/media(upload) is ported frompatch/media-url-canonicalizationandnow also records the library row.
DELETErefuses with 409 if any article still references the asset, so adelete here can't blank out a published page's image.
for alt text/caption, and a reindex action. Upload/delete/reindex are admin-gated.
TrixEditorposted to a bare/v1/mediawith no credentials. That was harmlesswhile the endpoint returned 501 to everyone, but 401s now that upload is
auth-gated and the API commonly runs on a different origin — it now uses
apiBaseUrl()with credentials, matchinguseApiFetch.Database tier
Config for the hosts that were actually provisioned: 4 vCPU / 4 GB unprivileged
LXC containers, not the 8–16 GB VMs originally requested.
innodb_buffer_pool_size5G → 1G. The entire dataset is ~92 MB, and a 5G poolon a 4 GB host prevents startup outright.
innodb_log_file_size512M → 256M.innodb_buffer_pool_instances, removed in MariaDB 10.6. Verified 11.xaccepts and ignores it silently rather than refusing to start, so this is a
cleanup, not a fix.
bind-addresspinned to the primary's NIC — Ubuntu's stock50-server.cnfbinds127.0.0.1, which leaves the primary unreachable from MaxScale.query_classifier_cache_sizeset explicitly. MaxScale defaults to ~15% of systemmemory, but inside an LXC whose cgroup limit reads
maxit sees the Proxmoxhost's RAM: it sized the cache at 9.38 GiB on a 4 GB container. Now 64M.
Note the deployed hosts run these natively via apt, not through the compose files
(Docker in unprivileged LXC needs Proxmox-side nesting), and MariaDB is 11.8 LTS
rather than the 11.7 pinned here —
deb.mariadb.orgcarries only LTS lines and 11.7is an EOL short-term release.
CI
deploy_scripts_test.shnow runs in CI. It existed but nothing invoked it, sothe least reversible code in the repo — slot selection, the transactional Nginx
switch and its restore-on-failure paths, the preflight checks — had a suite that
never ran on a PR. It stubs
docker/curl/nginxonPATH, so it needs nodaemon or privileges.
rollback.yml.rollback.shwas only invocable by SSH-ing into Delta frominside the VPN during an incident. Now a manual workflow on the same runner and
trust boundary as
deploy.yml, sharing thedelta-production-deployconcurrencygroup so a rollback can never interleave with a deployment. No
packagespermission, since switching slots pulls no images.
Testing
go build,go vet,go test ./...all clean.tsc -bandeslintclean, production build succeeds.sanitising, path containment), plus
CMS_TEST_DSN-gated integration tests for theDB and HTTP layers, following the existing pattern. Run against a real MariaDB
11.8, not left skipping: upload → list → patch → delete, the 409-in-use path,
reindex idempotency, and derivative-skipping all verified. The two integration
suites serialise on a MySQL advisory lock because they share the
mediatable andgo test ./...runs packages in parallel.After merge
invisible to the CMS until then: Media → Reindex, or
POST /v1/media/index. It's a full walk of the tree, so allow time on the realcorpus; idempotent and safe to re-run.
GET /v1/articlesreturning 401 without a session is expected on this build; theread un-gating is on the separate
patch/media-url-canonicalizationbranch.Note on scope
The media commit also carries in-progress polls/comments work that was already
modified in the working tree. Those changes share files with the media work
(
routes.go,main.go,types.go,api_responses.go) and couldn't be splitwithout interactive hunk staging.
Unrelated, but flagged by the push:
server/internal/database/init_schema.sqlis50.06 MB, past GitHub's recommended maximum and heading for the 100 MB hard
limit that blocks pushes. It's a generated seed, so it probably wants git-ignoring
or LFS.