Skip to content

feat: media library, production DB tier, and CI gaps - #110

Merged
ssavutu merged 7 commits into
mainfrom
patch/delta-media-serving
Jul 30, 2026
Merged

feat: media library, production DB tier, and CI gaps#110
ssavutu merged 7 commits into
mainfrom
patch/delta-media-serving

Conversation

@ssavutu

@ssavutu ssavutu commented Jul 30, 2026

Copy link
Copy Markdown
Member

Media library (API + page), the production database tier, and two CI gaps closed.

Merge before the first self-hosted-runner deploy. deploy.yml checks out
deployment code from the protected default branch, so until this lands, main
carries neither the media library nor the hardened deployment_preflight in
common.sh.

(An earlier version of this note claimed a deploy from main would overwrite the
Nginx site and kill media serving. That is not the case: the deploy scripts
only rewrite active-upstreams.conf — four set directives naming the active
slot's ports — and never touch sites-enabled. The site config carrying
/wp-content/ is installed manually during bootstrap and is already live on
Delta. Merging matters for the code, not to protect media serving.)

Media library

GET /v1/media, /v1/media/{id}, and /v1/media/gallery previously returned 501
via a misnamed handlers.Users placeholder, and the Media page's Upload and Delete
buttons were inert.

  • New media table via EnsureMediaTable, following the existing runtime-schema
    pattern. path (wp-content-relative) is the row's stable identity; url is
    rendered through MEDIA_BASE_URL per response rather than stored, so it can't go
    stale if the media host changes.
  • List (paginated, server-side search, MIME-family filter, sort), gallery shape for
    pickers, get, PATCH for alt text/caption, and DELETE.
  • POST /v1/media/index walks MEDIA_ROOT and adds anything missing — this is how
    the migrated corpus enters the library. It skips WordPress's generated -WxH
    derivatives, which would otherwise bury the library in near-duplicates, and
    re-running preserves alt text already curated in the CMS.
  • POST /v1/media (upload) is ported from patch/media-url-canonicalization and
    now also records the library row.
  • DELETE refuses with 409 if any article still references the asset, so a
    delete here can't blank out a published page's image.
  • Frontend: multi-file upload, debounced server-side search, paging, a detail panel
    for alt text/caption, and a reindex action. Upload/delete/reindex are admin-gated.
  • TrixEditor posted to a bare /v1/media with no credentials. That was harmless
    while the endpoint returned 501 to everyone, but 401s now that upload is
    auth-gated and the API commonly runs on a different origin — it now uses
    apiBaseUrl() with credentials, matching useApiFetch.

Database tier

Config for the hosts that were actually provisioned: 4 vCPU / 4 GB unprivileged
LXC containers, not the 8–16 GB VMs originally requested.

  • innodb_buffer_pool_size 5G → 1G. The entire dataset is ~92 MB, and a 5G pool
    on a 4 GB host prevents startup outright.
  • innodb_log_file_size 512M → 256M.
  • Dropped innodb_buffer_pool_instances, removed in MariaDB 10.6. Verified 11.x
    accepts and ignores it silently rather than refusing to start, so this is a
    cleanup, not a fix.
  • bind-address pinned to the primary's NIC — Ubuntu's stock 50-server.cnf binds
    127.0.0.1, which leaves the primary unreachable from MaxScale.
  • query_classifier_cache_size set explicitly. MaxScale defaults to ~15% of system
    memory, but inside an LXC whose cgroup limit reads max it sees the Proxmox
    host's
    RAM: it sized the cache at 9.38 GiB on a 4 GB container. Now 64M.

Note the deployed hosts run these natively via apt, not through the compose files
(Docker in unprivileged LXC needs Proxmox-side nesting), and MariaDB is 11.8 LTS
rather than the 11.7 pinned here — deb.mariadb.org carries only LTS lines and 11.7
is an EOL short-term release.

CI

  • deploy_scripts_test.sh now runs in CI. It existed but nothing invoked it, so
    the least reversible code in the repo — slot selection, the transactional Nginx
    switch and its restore-on-failure paths, the preflight checks — had a suite that
    never ran on a PR. It stubs docker/curl/nginx on PATH, so it needs no
    daemon or privileges.
  • New rollback.yml. rollback.sh was only invocable by SSH-ing into Delta from
    inside the VPN during an incident. Now a manual workflow on the same runner and
    trust boundary as deploy.yml, sharing the delta-production-deploy concurrency
    group so a rollback can never interleave with a deployment. No packages
    permission, since switching slots pulls no images.

Testing

  • Server: go build, go vet, go test ./... all clean.
  • Frontend: tsc -b and eslint clean, production build succeeds.
  • Media storage guarantees covered without a database (no-clobber naming, filename
    sanitising, path containment), plus CMS_TEST_DSN-gated integration tests for the
    DB and HTTP layers, following the existing pattern. Run against a real MariaDB
    11.8
    , not left skipping: upload → list → patch → delete, the 409-in-use path,
    reindex idempotency, and derivative-skipping all verified. The two integration
    suites serialise on a MySQL advisory lock because they share the media table and
    go test ./... runs packages in parallel.
  • Swagger docs regenerated.

After merge

  1. Run the media index once, after the media rsync — the corpus is on disk but
    invisible to the CMS until then: Media → Reindex, or
    POST /v1/media/index. It's a full walk of the tree, so allow time on the real
    corpus; idempotent and safe to re-run.
  2. GET /v1/articles returning 401 without a session is expected on this build; the
    read un-gating is on the separate patch/media-url-canonicalization branch.

Note on scope

The media commit also carries in-progress polls/comments work that was already
modified in the working tree. Those changes share files with the media work
(routes.go, main.go, types.go, api_responses.go) and couldn't be split
without interactive hunk staging.

Unrelated, but flagged by the push: server/internal/database/init_schema.sql is
50.06 MB, past GitHub's recommended maximum and heading for the 100 MB hard
limit that blocks pushes. It's a generated seed, so it probably wants git-ignoring
or LFS.

ssavutu and others added 7 commits July 26, 2026 23:09
- nginx: hardened static `location /wp-content/` serving /mnt/cephfs/media
  (GET/HEAD only, 403 on script/exe extensions, nosniff + strict CSP,
  immutable cache) plus a dotfile deny. No PHP handler, so nothing executes.
- compose: backend anchor gains MEDIA_ROOT, MEDIA_BASE_URL, and an rw
  bind-mount of the CephFS media tree so the upload endpoint can write.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
innodb_redo_log_capacity is a MySQL 8.0.30+ variable, not a MariaDB one.
Both primary.cnf and replica.cnf set it, with a comment claiming it
replaces innodb_log_file_size on MariaDB 11.x. It does not: mariadbd
exits with "unknown variable" and the container crash-loops. Caught when
the dry-run database inherited the setting; the production primary and
replica would have failed identically on first boot. All three configs
now use innodb_log_file_size.

Adds a throwaway dry-run database so the CMS can be brought up before
the dedicated DB hardware exists. compose.mariadb-dev.yml is an overlay
on compose.cms.yml, so the node joins triangle_net and is reachable as
DB_HOST=mariadb-dev. dev.cnf sizes InnoDB for Delta (~3.8 GB RAM total,
shared with both CMS slots) rather than the dedicated 8 GB the
production configs assume.

Also:
- nginx: drop `expires 30d` from the media block. It emitted a second,
  weaker Cache-Control alongside the explicit add_header, and CDNs
  disagree about which duplicate wins.
- cms.env.example: add MEDIA_HOST_PATH/MEDIA_ROOT/MEDIA_BASE_URL, which
  compose.cms.yml already consumed but the template never listed.
- README: document installing the nginx site, media serving and its
  failure modes, and image pruning on Delta's small root filesystem.

cms.env.dryrun.example records the values verified working on Delta.
CMS_REBUILD_TAXONOMY_COUNTS_ON_STARTUP must stay false against the
init_schema.sql seed: the rebuild reads an articles.categories column
the seed lacks, and the error is fatal.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Bringing the overlay up cold raced the backend against MariaDB and
logged five "connection refused" errors before the retry loop won.
Production cannot express this dependency -- the database lives on
another host and the backend simply retries -- but in the dry-run
overlay the database is a sibling service, so wait for its health check.
Startup is now clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The tracked seed carried an older schema than wordpress-etl now emits.
Most consequentially it lacked articles.categories, which
RebuildTaxonomyArticleCounts selects unconditionally, so running with
CMS_REBUILD_TAXONOMY_COUNTS_ON_STARTUP=true against this seed failed with
"Unknown column 'categories'" and crash-looped the backend.

Adds the seven columns the ETL emits that the seed lacked
(creation_date, author_ids, authors, featured_img_id, categories,
metadata, excerpt) and widens types to match ArticleFormatter.CMS_SCHEMA
(BIGINT ids, LONGTEXT text columns). Retains focus_keyword,
meta_description and seo_title, which the CMS uses and the ETL does not
emit. authors and articles_authors likewise widened to BIGINT to match
AuthorFormatter and ArtAuthFormatter.

articles_authors keeps AUTO_INCREMENT even though the ETL declares a
bare BIGINT PRIMARY KEY: the seed's INSERTs do not supply id, so the
ETL's exact DDL would reject every row.

Verified by loading the full seed into a scratch database and running the
backend against it with the taxonomy rebuild enabled: no errors, no
restarts. Note the rebuild is a no-op on this data -- the seed populates
tags for 8381 articles but categories for none -- so real category
counts still require an ETL-produced dataset.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…eal hosts

Adds the primary/replica compose overlays, tuned .cnf files, the replication
and MaxScale user init scripts, and the MaxScale readwritesplit config. Also
carries the observability overlay, which was already in the tree.

Tuning is sized for the hosts that were actually provisioned: 4 vCPU / 4 GB
unprivileged LXC containers, not the 8-16 GB VMs originally requested.

- innodb_buffer_pool_size 5G -> 1G. The entire dataset is ~92 MB, and a 5G
  pool on a 4 GB host prevents startup outright.
- innodb_log_file_size 512M -> 256M. Ample for this write volume and keeps
  crash recovery fast.
- Drop innodb_buffer_pool_instances, removed in MariaDB 10.6. Verified that
  11.x accepts and ignores it silently rather than refusing to start, so this
  is a cleanup rather than a fix.
- Pin bind-address to the primary's internal NIC. Ubuntu's stock 50-server.cnf
  binds 127.0.0.1, which leaves the primary unreachable from MaxScale, so the
  drop-in must also sort after it (installed as 70-triangle-primary.cnf).
- Set query_classifier_cache_size explicitly. MaxScale defaults to ~15% of
  system memory, but inside an LXC whose cgroup limit reads "max" it sees the
  Proxmox host's RAM instead: it sized the cache at 9.38 GiB on a 4 GB
  container. Now pinned to 64M (observed 61 MiB).

Note the deployed hosts run these packages natively via apt, not through the
compose files, because Docker inside unprivileged LXC needs Proxmox-side
nesting. MariaDB is 11.8 LTS rather than the 11.7 pinned here: deb.mariadb.org
carries only LTS lines, and 11.7 is an EOL short-term release.

Also documents the media library reindex, which is required at cutover because
serving the migrated files is independent of listing them.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Replaces the three /v1/media placeholders (which returned 501 via the
misnamed handlers.Users) with a DB-backed media library.

Server:
- New `media` table via EnsureMediaTable, following the existing runtime-schema
  pattern. `path` (wp-content-relative) is the row's stable identity; `url` is
  rendered through MEDIA_BASE_URL per response rather than stored, so it cannot
  go stale.
- GET /v1/media (paginated, server-side search, MIME-family filter, sort),
  GET /v1/media/gallery (trimmed shape for pickers), GET /v1/media/{id},
  PATCH /v1/media/{id} (alt text, caption), DELETE /v1/media/{id}.
- POST /v1/media/index walks MEDIA_ROOT and adds anything missing, which is how
  the migrated CephFS corpus enters the library. It skips WordPress's generated
  -WxH derivatives, which would otherwise bury the library in near-duplicates,
  and re-running preserves alt text already curated in the CMS.
- POST /v1/media (upload) is ported from patch/media-url-canonicalization and
  now also records the library row.
- DELETE refuses with 409 if any article still references the asset, so a delete
  here cannot blank out a published page's image.

Frontend:
- mediaView.tsx: multi-file upload, debounced server-side search, paging, a
  detail panel for alt text/caption, and a reindex action. Upload, delete and
  reindex are admin-gated. Previously the Upload and Delete buttons were inert.
- TrixEditor: drop the stale "backend not wired up" warnings, and send uploads
  to apiBaseUrl() with credentials. It posted to a bare /v1/media with no
  cookies, which 401s now that the endpoint is auth-gated and the API commonly
  runs on a different origin.

Tests cover the storage guarantees without a database (no-clobber naming,
filename sanitising, path containment) plus DSN-gated integration tests for the
DB and HTTP layers, following the existing CMS_TEST_DSN pattern. The two
integration suites serialise on a MySQL advisory lock because they share the
`media` table and `go test ./...` runs packages in parallel.

This commit also carries in-progress polls/comments work that was already
modified in the working tree; those changes share files with the media work
(routes.go, main.go, types.go, api_responses.go) and could not be split cleanly.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two gaps in the pipeline.

deploy_scripts_test.sh existed but nothing ran it, so the least reversible code
in the repo — slot selection, the transactional Nginx switch and its
restore-on-failure paths, the preflight checks — had a test suite that never
executed on a PR. It now runs as its own CI job. The suite stubs docker, curl
and nginx on PATH, so it needs no daemon, database, or privileges.

rollback.sh was only invocable by SSH-ing into Delta from inside the VPN, during
an incident, which is when that is hardest. rollback.yml exposes it as a manual
workflow on the same self-hosted runner and trust boundary as deploy.yml
(deployment code from the protected default branch). It shares the
delta-production-deploy concurrency group so a rollback can never interleave with
a deployment, and holds no packages permission because switching slots pulls no
images. The slot input is passed through the environment rather than interpolated
into the script body.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@ssavutu
ssavutu merged commit e62c7d1 into main Jul 30, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant