Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions deploy/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -252,6 +252,26 @@ sudo find /mnt/cephfs/media/wp-content/uploads -type d \
The `d:` (default) entry is what makes each new `YYYY/MM` directory inherit the
grant, so this does not need repeating every month.

**But a later chmod silently disables it.** A chmod on an ACL'd directory sets
the **ACL mask** from the mode's group bits, and the mask caps every named
entry: `chmod 755` leaves `user:10001:rwx` printed but `#effective:r-x`. An
rsync of the corpus run with `--chmod=D755` does this to every directory it
touches (see `docs/HANDOVER.md` — that command now uses `D775`).

Nothing notices at the time. Uploads keep working until the 1st of the next
month, when the handler first has to create a new `YYYY/MM`, and then every
upload 500s. Check the mask, not the entry:

```bash
getfacl -pc /mnt/cephfs/media/wp-content/uploads/2026 | grep -E '10001|mask::'
# want: user:10001:rwx (no "#effective:") and mask::rwx
setfacl -m m::rwx /mnt/cephfs/media/wp-content/uploads{,/2026} # repair
```

Only those two levels matter: `uploads/` creates the year, `uploads/YYYY/`
creates the month. `triangle-infrastructure`'s `delta_cms_host` role repairs
both on every playbook run.

Without the `acl` package, setgid does the same job in plain POSIX, at the cost
of changing group ownership rather than adding a grant beside it:

Expand Down
11 changes: 11 additions & 0 deletions docs/ETL-REBUILD.md
Original file line number Diff line number Diff line change
Expand Up @@ -229,6 +229,17 @@ times on .../2026/08" is one directory's mtime, not a data failure — ignore it
Newly synced files can 404 publicly for up to 4 hours while the pre-sync edge
404 ages out. `?v=<ts>` returns 200 immediately; that confirms the file is fine.

**Upload ACL mask.** The sync can leave media *readable* and uploads broken. A
chmod on an ACL'd directory rewrites the ACL mask, capping the backend's
`user:10001:rwx` grant to `#effective:r-x`; the symptom is delayed until the 1st
of the next month, when `POST /v1/media` starts returning 500 on every upload.
Check the mask, not the entry:

```bash
getfacl -pc /mnt/cephfs/media/wp-content/uploads/2026 | grep -E '10001|mask::'
# want: user:10001:rwx (no "#effective:") and mask::rwx
```

**Orphaned author links** — byline present in `articles.authors` but the join
returns nothing, so the API emits `authors: null`:

Expand Down
26 changes: 25 additions & 1 deletion docs/HANDOVER.md
Original file line number Diff line number Diff line change
Expand Up @@ -225,12 +225,36 @@ added to WP after the last sync will 404. The rsync must be **pushed from**
`10.248.40.141` — Delta holds no key to pull:

```bash
ssh tadmin@10.248.40.141 'rsync -a --no-owner --no-group --chmod=D755,F644 \
ssh tadmin@10.248.40.141 'rsync -a --no-owner --no-group --chmod=D775,F664 \
--exclude="*.php" --exclude="*.exe" --exclude="*.sh" \
/var/www/html/thetriangle.org/wp-content/uploads/2026/ \
tadmin@10.248.40.168:/mnt/cephfs/media/wp-content/uploads/2026/'
```

**`D775`, not `D755` — the group bit is load-bearing.** The CMS backend runs as
uid 10001 and owns none of this tree, so its write access comes from a POSIX ACL
(`user:10001:rwx` plus the `default:` entries new directories inherit). A chmod
on an ACL'd directory sets the **ACL mask** from the mode's group bits, so
`D755` silently clamps that entry to `#effective:r-x`. `getfacl` still prints
`user:10001:rwx` and `ls` shows only a `+`, so it reads as correct.

Uploads then keep working until the 1st of the next month, when the handler
first has to create a new `YYYY/MM` directory and can't — `POST /v1/media`
returns 500 on **every** upload with `mkdir .../uploads/YYYY/MM: permission
denied` in the backend log. That is exactly the 2026-09-01 outage.

`D775` keeps the mask at `rwx` and matches the `0775` the CMS itself creates
month directories with. Verify after any sync — the mask, not the entry, is
what decides:

```bash
getfacl -pc /mnt/cephfs/media/wp-content/uploads/2026 | grep -E '10001|mask::'
# want: user:10001:rwx (no "#effective:") and mask::rwx
```

`ansible-playbook playbooks/delta-host.yml` in `triangle-infrastructure` repairs
a clamped mask, but only on the next run — the fix belongs in the command above.

It exits **23** with "failed to set times" on one directory. That is a
directory mtime owned by another uid, not a data failure — ignore it.

Expand Down
Loading