gap: prod and dev compose files are not version controlled, reviewed, or visible to CI #207

Prod's and dev's Docker Compose files are load-bearing deployment configuration that is not in version control, never reviewed, and invisible to CI. A defect in them cannot be caught by anything we have.

This is filed because it already cost us, twice in one week.

The evidence

[verified: 2026-10-16/17]

The repo carries docker-compose.yml, docker-compose.example.yml, docker-compose.no-mosquitto.yml, docker-compose.staging.yml and docker-compose.staging.no-mosquitto.yml — none of which describes how prod actually runs.

What it cost, concretely

#204 (this week). The region-layers bind mount reused ${DATA_DIR}, so /app/public/region-layers was served from the data directory. The region definitions page returned the SPA's index.html instead of layers.json. Shipped by hand at 17:08 UTC, went live with the v0.3.3 deploy at 20:37, broken until ~23:45. Nothing could have caught it: there is no review step, no schema check, and no test that inspects the resolved mounts.

#205 (the same release). #187 added the region scoping feature and touched no compose file. The mount its feature requires exists only on one box, created by hand, three hours before the tag. The feature is not reproducible from the repo — a fresh deploy would ship the page with no data behind it.

#183 (earlier). Prod ran without a healthThresholds block for weeks while dev had one, because the two boxes' configs drift independently with nothing comparing them.

This is also precisely the hazard written into docs/UPSTREAM.md checklist item 1: every CI artifact derives from the code, so code and fixtures move together and stay self-consistent; the only disagreeing artifact is the config deployed on the boxes, which CI has never seen. The compose file is the same class of artifact as config.json and has the same blind spot.

What is actually missing

1. No version control. The only history is .bak files. There is no diff, no author, no reason-for-change, and no way to answer "when did this line appear and why" except by comparing backups — which is how chore(prod): region-layers mount reuses ${DATA_DIR}, so the region defs page serves the data dir #204's timeline had to be reconstructed.
2. No review. A one-character error in a path (regions-layers vs region-layers was suggested in Discord and would have silently mounted an empty directory) reaches prod with nobody else looking.
3. No validation. docker compose config exists and would have caught nothing here, because the file was syntactically valid — the defect was semantic. A check that the two bind sources differ, or that each source exists, would have caught it.
4. No drift detection. Nothing compares prod's compose to dev's, or either to anything in the repo.

Options, and this needs a decision rather than a default

Listing these without picking one, because the right answer depends on how much ceremony the team wants around box config.

A. Track the compose files in the repo and deploy them. Most thorough: PR, review, CI validation, and the deploy copies the file. Cost is that deploy-prod.yml grows a step that overwrites box state, which is a meaningful change to a working deploy path and needs care around the corescope:prod local-tag prereq.

B. Track them in the repo but do not deploy them. Keeps the file authoritative and reviewable; applying stays manual. Cheap to adopt, and drift is detectable by diff. Weaker than A because nothing enforces the box matches.

C. Ansible. ITblacksheep's own instinct — "I need to switch our docker compose to ansible" — and it generalises past compose to the whole host config (sudoers, sshd AllowUsers, users, docker). Biggest investment, best end state, and it subsumes the provisioning gotchas found in infra#48.

D. A CI check that validates the deployed file. Narrower: something that reads the box's compose and asserts invariants (distinct bind sources, every source exists, image is the expected local tag). Catches #204's class without changing where the file lives.

B and D together are the cheap combination that would have caught #204. C is where this probably ends up.

Scope note

This covers prod and dev compose. The same argument applies to /opt/corescope/.env, /etc/sudoers.d/*, and /etc/ssh/sshd_config.d/10-offband.conf — the last of which has a hand-maintained AllowUsers allowlist that silently refuses any user not added to it (infra#48). Whether to widen this issue to all host config or keep it to compose is part of the decision above.

Done when

A defect in prod's compose can be caught before it reaches prod, and "when did this line change and why" is answerable without diffing backup files.

Refs #204, #205, #183, #187, OKI-Mesh/infra#48