From 2c03171b8b79d7aa8cba325ef8f183ed9cf6e414 Mon Sep 17 00:00:00 2001 From: ssavutu Date: Tue, 1 Sep 2026 16:32:39 -0400 Subject: [PATCH] Stop the media rsync clamping the upload ACL mask The documented media sync used --chmod=D755, which is what caused the 2026-09-01 outage where POST /v1/media returned 500 on every upload. A chmod on an ACL'd directory sets the ACL mask from the mode's group bits, and the mask caps every named entry. D755 therefore clamped the backend's user:10001:rwx grant to #effective:r-x on every directory the sync touched. getfacl still prints user:10001:rwx and ls shows only a +, so it reads as correct, and uploads keep working until the 1st of the next month -- the first time the handler has to create a new YYYY/MM directory. D775 keeps the mask at rwx and matches the 0775 the CMS itself creates month directories with. Verified on CephFS against a fixture reproducing the live ACL: D755 reproduces the clamp exactly, D775 leaves the mask rwx, the new month directory inherits the grant, uid 10001 can create the next month, and the synced files stay world-readable for nginx. deploy/README.md claimed the default ACL entry meant this "does not need repeating every month". That is true until something chmods the tree, and believing it is what let the bug hide for a month; it now documents the mask. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Tx3ywgRQLzzrZWNcSdtTpr --- deploy/README.md | 20 ++++++++++++++++++++ docs/ETL-REBUILD.md | 11 +++++++++++ docs/HANDOVER.md | 26 +++++++++++++++++++++++++- 3 files changed, 56 insertions(+), 1 deletion(-) diff --git a/deploy/README.md b/deploy/README.md index d4042f3..5651b20 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -252,6 +252,26 @@ sudo find /mnt/cephfs/media/wp-content/uploads -type d \ The `d:` (default) entry is what makes each new `YYYY/MM` directory inherit the grant, so this does not need repeating every month. +**But a later chmod silently disables it.** A chmod on an ACL'd directory sets +the **ACL mask** from the mode's group bits, and the mask caps every named +entry: `chmod 755` leaves `user:10001:rwx` printed but `#effective:r-x`. An +rsync of the corpus run with `--chmod=D755` does this to every directory it +touches (see `docs/HANDOVER.md` — that command now uses `D775`). + +Nothing notices at the time. Uploads keep working until the 1st of the next +month, when the handler first has to create a new `YYYY/MM`, and then every +upload 500s. Check the mask, not the entry: + +```bash +getfacl -pc /mnt/cephfs/media/wp-content/uploads/2026 | grep -E '10001|mask::' +# want: user:10001:rwx (no "#effective:") and mask::rwx +setfacl -m m::rwx /mnt/cephfs/media/wp-content/uploads{,/2026} # repair +``` + +Only those two levels matter: `uploads/` creates the year, `uploads/YYYY/` +creates the month. `triangle-infrastructure`'s `delta_cms_host` role repairs +both on every playbook run. + Without the `acl` package, setgid does the same job in plain POSIX, at the cost of changing group ownership rather than adding a grant beside it: diff --git a/docs/ETL-REBUILD.md b/docs/ETL-REBUILD.md index d9690b7..66d6db7 100644 --- a/docs/ETL-REBUILD.md +++ b/docs/ETL-REBUILD.md @@ -229,6 +229,17 @@ times on .../2026/08" is one directory's mtime, not a data failure — ignore it Newly synced files can 404 publicly for up to 4 hours while the pre-sync edge 404 ages out. `?v=` returns 200 immediately; that confirms the file is fine. +**Upload ACL mask.** The sync can leave media *readable* and uploads broken. A +chmod on an ACL'd directory rewrites the ACL mask, capping the backend's +`user:10001:rwx` grant to `#effective:r-x`; the symptom is delayed until the 1st +of the next month, when `POST /v1/media` starts returning 500 on every upload. +Check the mask, not the entry: + +```bash +getfacl -pc /mnt/cephfs/media/wp-content/uploads/2026 | grep -E '10001|mask::' +# want: user:10001:rwx (no "#effective:") and mask::rwx +``` + **Orphaned author links** — byline present in `articles.authors` but the join returns nothing, so the API emits `authors: null`: diff --git a/docs/HANDOVER.md b/docs/HANDOVER.md index 10dd8b3..ad1f2d4 100644 --- a/docs/HANDOVER.md +++ b/docs/HANDOVER.md @@ -225,12 +225,36 @@ added to WP after the last sync will 404. The rsync must be **pushed from** `10.248.40.141` — Delta holds no key to pull: ```bash -ssh tadmin@10.248.40.141 'rsync -a --no-owner --no-group --chmod=D755,F644 \ +ssh tadmin@10.248.40.141 'rsync -a --no-owner --no-group --chmod=D775,F664 \ --exclude="*.php" --exclude="*.exe" --exclude="*.sh" \ /var/www/html/thetriangle.org/wp-content/uploads/2026/ \ tadmin@10.248.40.168:/mnt/cephfs/media/wp-content/uploads/2026/' ``` +**`D775`, not `D755` — the group bit is load-bearing.** The CMS backend runs as +uid 10001 and owns none of this tree, so its write access comes from a POSIX ACL +(`user:10001:rwx` plus the `default:` entries new directories inherit). A chmod +on an ACL'd directory sets the **ACL mask** from the mode's group bits, so +`D755` silently clamps that entry to `#effective:r-x`. `getfacl` still prints +`user:10001:rwx` and `ls` shows only a `+`, so it reads as correct. + +Uploads then keep working until the 1st of the next month, when the handler +first has to create a new `YYYY/MM` directory and can't — `POST /v1/media` +returns 500 on **every** upload with `mkdir .../uploads/YYYY/MM: permission +denied` in the backend log. That is exactly the 2026-09-01 outage. + +`D775` keeps the mask at `rwx` and matches the `0775` the CMS itself creates +month directories with. Verify after any sync — the mask, not the entry, is +what decides: + +```bash +getfacl -pc /mnt/cephfs/media/wp-content/uploads/2026 | grep -E '10001|mask::' +# want: user:10001:rwx (no "#effective:") and mask::rwx +``` + +`ansible-playbook playbooks/delta-host.yml` in `triangle-infrastructure` repairs +a clamped mask, but only on the next run — the fix belongs in the command above. + It exits **23** with "failed to set times" on one directory. That is a directory mtime owned by another uid, not a data failure — ignore it.