fix(nginx): stop caching 404s for 30 days on legacy media - #141
Merged
Conversation
`add_header ... always` applies the header to error responses too, so every 404 from the /wp-content/ block went out with `Cache-Control: public, max-age=2592000, immutable`. Cloudflare took that at face value and pinned "this file does not exist" at the edge for a month. The consequence is that any file added to the corpus AFTER something first requested it stays invisible for 30 days -- on disk, served correctly by this nginx, still 404 to every visitor. It reads as a failed copy rather than a cache hit, which is what makes it expensive to diagnose. Hit while migrating wp-content/uploads/newsletter/ to CephFS: the files landed, the origin served them, and the edge kept replaying a 404 cached minutes earlier. Confirmed by cf-cache-status: HIT with a stale age, while the same path with a cache-busting query string returned 200. Drop `always` from the Cache-Control add_header only. Without it the header applies to 2xx/3xx, and misses fall back to Cloudflare's short default 404 TTL, so a later-added file self-heals. X-Content-Type-Options and Content-Security-Policy keep `always` on purpose -- those must apply to error responses. This does not purge anything already cached; existing poisoned entries age out on their own. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The
/wp-content/block sets its cache header withadd_header ... always. That flag applies the header to error responses too, so every 404 from the media tree went out with:Cloudflare took that at face value and pinned "this file does not exist" at the edge for 30 days.
The consequence: any file added to the corpus after something first requested it stays invisible for a month — on disk, served correctly by nginx, still 404 to every visitor. It reads as a failed copy rather than a cache hit, which is what makes it expensive to diagnose.
How it surfaced
Found while migrating
wp-content/uploads/newsletter/to CephFS. The files landed and the origin served them, but the edge kept replaying a 404 cached minutes before the copy finished:…/nl-header-1-1200x0.png404,cf-cache-status: HIT,age: 35385…/nl-header-1-1200x0.png?v=1200 image/png, 90,191 bytes,cf-cache-status: MISSSame path, same origin — the only difference is a cache key Cloudflare hadn't poisoned. Origin
curlon Delta returned200throughout.Fix
Drop
alwaysfrom theCache-Controladd_headeronly. Without the flag it applies to 2xx/3xx, so misses fall back to Cloudflare's short default 404 TTL and a later-added file self-heals.X-Content-Type-OptionsandContent-Security-Policykeepalwaysdeliberately — those must apply to error responses.Immutable caching for real images is unchanged.
Scope
sudo cp+nginx -t+systemctl reload nginxon Delta.CMS-Testing, 400337c): the/proxy/route now retries once past an edge-cached 404. That rescues already-poisoned paths; this PR fixes the mechanism.Verification after deploy
curl -sI http://localhost/wp-content/uploads/newsletter/does-not-exist-xyz.png | grep -i cache-controlExpected: no
Cache-Controlline on the 404, while a real image still returnspublic, max-age=2592000, immutable.🤖 Generated with Claude Code