Speed up the public read paths - #212
Merged
Merged
Conversation
Article JSON is highly redundant -- repeated keys, HTML markup -- and the CMS shipped all of it raw, both to the editor SPA and through Cloudflare to the public site. A twenty-item listing measured 16,433 bytes on the wire and 5,178 gzipped; an article detail 8,551 and 3,172. Content-Type gates it, so media-library JPEGs and PNGs pass through untouched rather than spending CPU to grow slightly. Bodies under 512 bytes are left alone, because the gzip header and trailer alone are 18 of them and a short error payload comes out larger compressed. Placement is load-bearing and the middleware says so: this must wrap Recovery, not sit inside it. Inside, a panic would close the gzip stream during unwinding and Recovery's plain-text 500 would be appended to a finished stream, giving the client a body it cannot decode. The consequence is that Logging and Metrics keep counting uncompressed bytes -- the same number they reported before, so the dashboards stay comparable. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Nginx served the SPA bundle uncompressed and with no freshness headers at all, so every asset was re-downloaded or at best revalidated on each visit. The main JS chunk goes 247,941 -> 77,811 bytes with gzip on. Vite fingerprints asset filenames with a content hash, so a given URL can never change what it serves -- a new build produces new names. That makes /assets/ safe to cache permanently, which turns a repeat visit into no requests at all. index.html is the one file that must not be cached: it carries the references to those hashed names, and a stale copy points the browser at assets the last deploy removed. /assets/ also stops falling back to index.html, so a missing asset is a 404 rather than a page of HTML served with a .js name. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every one of the fifteen routes was imported eagerly, so opening the dashboard downloaded the article editor, Trix, the media library and the settings screens first. Initial load was 961.8 kB raw / 260.3 kB gzipped; it is now 485.0 kB / 147.5 kB. The three screens a session can start on -- dashboard, login, auth callback -- stay eager. Splitting a landing route only moves its download from the bundle into a second round trip the user waits through on a blank page. React and the UI kit go in their own chunks because they change only when a dependency is upgraded, while app code changes every deploy. A routine deploy now invalidates the app chunk and leaves 194 kB of vendor code in the browser cache, which is what the immutable caching added alongside is there to exploit. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every listing selected `text`, the full article body, and threw it away: ScanArticle loaded it into Content and articleListItems never read it. The excerpt comes from `excerpt`, falling back to `description`, and is derived from the body only at write time. One default page of twenty articles was fetching 97,270 bytes of body to use 12,905 bytes of excerpt, and the homepage paid that six times over building its section blocks. The four hand-maintained column lists this required are gone with it. They were coupled to one positional Scan, so adding a column in the wrong place silently shifted every value after it into the wrong field -- the comment above them warned as much, and dropping `text` meant adding a fifth. The SELECT lists and the Scan targets now derive from one ordered articleColumnSet, so they cannot disagree. ArticleColumns is the full set for the article detail endpoint, the one read that renders the body; ArticleSummaryColumns is everything else. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`slug` is how every article is addressed -- the detail endpoint, the comment thread, the permalink check, the featured-article write -- and nothing indexed it, so each of those scanned the whole corpus to return one row. `articles_authors` had only a primary key on its own id, so resolving a page of bylines scanned the entire join table, once per listing and once per homepage block. Both are prefix/secondary ADD INDEX, deliberately. The obvious fix for slug is to narrow the LONGTEXT to VARCHAR first, and that is a trap: retyping a column rewrites the table, and rewriting `articles` re-tokenizes 44MB of bodies into the two FULLTEXT indexes. Measured at 8m53s for one ALTER on a corpus this size, during which a second connection running the CMS's own startup migration sat in "Waiting for table metadata lock" -- in production that is the newsroom's writes queued behind a container that is not yet serving traffic. The prefix index takes 0.31s on the same shape and yields the same plan. 191 characters is above the longest slug in the corpus (154) and stays under the 767-byte limit older row formats impose. A prefix index only narrows candidates -- InnoDB rechecks the full value -- so it can cost an extra row read but never return a wrong article. Not UNIQUE: uniqueness is a property of the data, and the corpus belongs to the ETL. Production already carries one duplicated slug from an archived pair, which a UNIQUE index would turn into a CMS that will not start. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Section filtering ran `LOWER(categories) LIKE '%"news"%'` against a LONGTEXT column. A leading wildcard rules out every index, so each section page read the whole articles table -- twice, since the listing pages with a COUNT(*) alongside -- and so did each of the homepage's six blocks and every taxonomy recount. article_categories is a derived index of that column: one row per (article, category title), so the question becomes an equality join. The optimizer flips it, driving from the index and doing eq_ref primary lookups instead of applying REPLACE/LOWER to a LONGTEXT for 6,038 rows. A section page goes from 76.2ms to 40.2ms, search from 111.6ms to 66.9ms. It is strictly derived, never authoritative. `articles`.`categories` remains the source of truth, article writes keep the index in step, and it is rebuilt at every startup -- deliberately unconditional, and deliberately not behind CMS_REBUILD_TAXONOMY_COUNTS_ON_STARTUP. The corpus is periodically reloaded wholesale by the ETL, which renumbers article ids: stale contents would not merely be old, they would point at the wrong articles. A rebuild an operator has to remember is one that gets forgotten on the reseed that needs it most, and it costs one scan of a ten-thousand-row table. Creating it is fatal on failure, unlike the index migrations, because the queries now depend on it: a CMS that booted without it would serve empty sections rather than slow ones. EXISTS rather than IN because callers negate the fragment (see ReportOrphanedArticles), and NOT IN over a subquery that can yield NULL evaluates to UNKNOWN and silently matches nothing. The LIKE mechanism is deleted rather than left dead. Its rules -- the anchoring that keeps Women's Basketball out of mens-basketball, the alias map, the restored possessives -- move intact to CategoryMatchValues, and their tests move with them: matchesCategories now composes the two real halves, what the rebuild would index and what the query would ask for. Verified against the full corpus: for all 76 categories the old and new predicates select identical article sets, and every one of the 132 unindexed articles genuinely has no categories. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Six changes to the paths the public site and the editor actually hit. No
behaviour changes: the API returns the same bytes it did before.
Measured
Against the real corpus (9,371 articles locally, 10,113 in production),
median of 12 requests over a kept-alive connection:
/v1/sections/news/articles/v1/search?q=drexel/v1/homepage/v1/articles?limit=20Wire size, same corpus:
/v1/articles?limit=20/v1/articles/{slug}Article lookup by slug goes from
type: ALL, 5,150 rows totype: ref,1 row.
Verification
The old binary and the new one were run against the same live database
and their JSON diffed. Eight endpoints — listing, paged listing, three
section pages, a subsection, homepage, search — byte-identical, zero
differences.
Separately, for all 76 categories in the corpus the old
LIKEpredicateand the new indexed predicate select the same article sets, and all 132
articles the index omits genuinely have no categories.
go vetclean; full Go suite including MariaDB integration tests, plusfrontend lint and vitest, all pass.
Production notes
Nothing here needs a reseed. Every migration is an in-place
ADD INDEXor a new table built by scanning what is already there.The one thing worth knowing before merge: a schema change that rewrites
articlesis not viable on this database. It re-tokenizes 44MB of bodiesinto the two FULLTEXT indexes — 8m53s for a single ALTER at this corpus
size, during which the CMS's own startup migration was observed sitting in
Waiting for table metadata lock. Both index migrations here avoid thatdeliberately; the reasoning is in the code so it does not get undone.
Production currently carries one duplicated slug (an archived pair), which
is why the slug index is not UNIQUE.
Not done, deliberately
The paging
COUNT(*)is now the largest remaining cost on/v1/articles.Two rewrites were built and measured, then rejected — a join-table
EXISTS(nets under 4ms and moves the definition of "filed" onto indexesthat match the columns by data rather than by construction) and an indexed
generated column (needs the table rewrite above). The analysis is recorded
in
article_categories.goso the next person does not re-derive it.🤖 Generated with Claude Code