diff --git a/deploy/README.md b/deploy/README.md index d4042f3..5651b20 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -252,6 +252,26 @@ sudo find /mnt/cephfs/media/wp-content/uploads -type d \ The `d:` (default) entry is what makes each new `YYYY/MM` directory inherit the grant, so this does not need repeating every month. +**But a later chmod silently disables it.** A chmod on an ACL'd directory sets +the **ACL mask** from the mode's group bits, and the mask caps every named +entry: `chmod 755` leaves `user:10001:rwx` printed but `#effective:r-x`. An +rsync of the corpus run with `--chmod=D755` does this to every directory it +touches (see `docs/HANDOVER.md` — that command now uses `D775`). + +Nothing notices at the time. Uploads keep working until the 1st of the next +month, when the handler first has to create a new `YYYY/MM`, and then every +upload 500s. Check the mask, not the entry: + +```bash +getfacl -pc /mnt/cephfs/media/wp-content/uploads/2026 | grep -E '10001|mask::' +# want: user:10001:rwx (no "#effective:") and mask::rwx +setfacl -m m::rwx /mnt/cephfs/media/wp-content/uploads{,/2026} # repair +``` + +Only those two levels matter: `uploads/` creates the year, `uploads/YYYY/` +creates the month. `triangle-infrastructure`'s `delta_cms_host` role repairs +both on every playbook run. + Without the `acl` package, setgid does the same job in plain POSIX, at the cost of changing group ownership rather than adding a grant beside it: diff --git a/docs/ETL-REBUILD.md b/docs/ETL-REBUILD.md index d9690b7..66d6db7 100644 --- a/docs/ETL-REBUILD.md +++ b/docs/ETL-REBUILD.md @@ -229,6 +229,17 @@ times on .../2026/08" is one directory's mtime, not a data failure — ignore it Newly synced files can 404 publicly for up to 4 hours while the pre-sync edge 404 ages out. `?v=` returns 200 immediately; that confirms the file is fine. +**Upload ACL mask.** The sync can leave media *readable* and uploads broken. A +chmod on an ACL'd directory rewrites the ACL mask, capping the backend's +`user:10001:rwx` grant to `#effective:r-x`; the symptom is delayed until the 1st +of the next month, when `POST /v1/media` starts returning 500 on every upload. +Check the mask, not the entry: + +```bash +getfacl -pc /mnt/cephfs/media/wp-content/uploads/2026 | grep -E '10001|mask::' +# want: user:10001:rwx (no "#effective:") and mask::rwx +``` + **Orphaned author links** — byline present in `articles.authors` but the join returns nothing, so the API emits `authors: null`: diff --git a/docs/HANDOVER.md b/docs/HANDOVER.md index 10dd8b3..ad1f2d4 100644 --- a/docs/HANDOVER.md +++ b/docs/HANDOVER.md @@ -225,12 +225,36 @@ added to WP after the last sync will 404. The rsync must be **pushed from** `10.248.40.141` — Delta holds no key to pull: ```bash -ssh tadmin@10.248.40.141 'rsync -a --no-owner --no-group --chmod=D755,F644 \ +ssh tadmin@10.248.40.141 'rsync -a --no-owner --no-group --chmod=D775,F664 \ --exclude="*.php" --exclude="*.exe" --exclude="*.sh" \ /var/www/html/thetriangle.org/wp-content/uploads/2026/ \ tadmin@10.248.40.168:/mnt/cephfs/media/wp-content/uploads/2026/' ``` +**`D775`, not `D755` — the group bit is load-bearing.** The CMS backend runs as +uid 10001 and owns none of this tree, so its write access comes from a POSIX ACL +(`user:10001:rwx` plus the `default:` entries new directories inherit). A chmod +on an ACL'd directory sets the **ACL mask** from the mode's group bits, so +`D755` silently clamps that entry to `#effective:r-x`. `getfacl` still prints +`user:10001:rwx` and `ls` shows only a `+`, so it reads as correct. + +Uploads then keep working until the 1st of the next month, when the handler +first has to create a new `YYYY/MM` directory and can't — `POST /v1/media` +returns 500 on **every** upload with `mkdir .../uploads/YYYY/MM: permission +denied` in the backend log. That is exactly the 2026-09-01 outage. + +`D775` keeps the mask at `rwx` and matches the `0775` the CMS itself creates +month directories with. Verify after any sync — the mask, not the entry, is +what decides: + +```bash +getfacl -pc /mnt/cephfs/media/wp-content/uploads/2026 | grep -E '10001|mask::' +# want: user:10001:rwx (no "#effective:") and mask::rwx +``` + +`ansible-playbook playbooks/delta-host.yml` in `triangle-infrastructure` repairs +a clamped mask, but only on the next run — the fix belongs in the command above. + It exits **23** with "failed to set times" on one directory. That is a directory mtime owned by another uid, not a data failure — ignore it.