Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 23 additions & 9 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -205,7 +205,7 @@ jobs:
- uses: actions/checkout@v4
with:
submodules: recursive
- run: docker build -t tower-finder:ci .
- run: docker build -t retina-server:ci .

env-parity:
name: Staging/production parity
Expand Down Expand Up @@ -500,13 +500,20 @@ jobs:
# cache reused); the disk pre-flight now catches that root cause, so a
# full wipe is redundant and only forced slow from-scratch rebuilds.
docker builder prune -f --keep-storage 10GB
# Deploy the app FIRST, on its own — building only `tower-finder`
# Deploy the app FIRST, on its own — building only `server`
# means a broken Dockerfile.fleet can't abort the staging app deploy
# (which would block the whole pipeline). Fleet is a separate step below.
docker compose up -d --build tower-finder
# --remove-orphans because this deploy renamed the compose
# service. container_name is pinned, so the container kept by the
# old service key is an orphan holding the exact name the new one
# needs, and compose leaves orphans alone by default: the create
# then fails with a name conflict and the stack does not come up.
# Verified on retina-test before this landed. It stays afterwards
# as the general guard for a removed or renamed service.
docker compose up -d --build --remove-orphans server
# Wait for server to become healthy (up to 90s)
for i in $(seq 1 18); do
if docker compose exec -T tower-finder \
if docker compose exec -T server \
python3 -c "import urllib.request; urllib.request.urlopen('http://localhost:8000/api/health')" 2>/dev/null; then
echo "Staging server healthy after ~$((i*5))s"
break
Expand Down Expand Up @@ -787,17 +794,24 @@ jobs:
# cause, so a full wipe is redundant and only forced slow from-scratch
# rebuilds. 10GB leaves ample headroom on both droplets (>60GB free).
docker builder prune -f --keep-storage 10GB
# Deploy the app FIRST, on its own. Building only `tower-finder`
# Deploy the app FIRST, on its own. Building only `server`
# here means a broken Dockerfile.fleet can't abort the production
# app deploy. Production runs no fleet simulator (see the `sim`
# profile in docker-compose.prod.yml), so `tower-finder` is the
# profile in docker-compose.prod.yml), so `server` is the
# whole of the production deploy; staging still builds both.
docker compose up -d --build tower-finder
# --remove-orphans because this deploy renamed the compose
# service. container_name is pinned, so the container kept by the
# old service key is an orphan holding the exact name the new one
# needs, and compose leaves orphans alone by default: the create
# then fails with a name conflict and the stack does not come up.
# Verified on retina-test before this landed. It stays afterwards
# as the general guard for a removed or renamed service.
docker compose up -d --build --remove-orphans server

# Wait for backend to become healthy (up to 90s — matches staging
# and the healthcheck start_period; a cold boot can need >60s).
for i in $(seq 1 18); do
if docker compose exec -T tower-finder python3 -c "import urllib.request; urllib.request.urlopen('http://localhost:8000/api/health')" 2>/dev/null; then
if docker compose exec -T server python3 -c "import urllib.request; urllib.request.urlopen('http://localhost:8000/api/health')" 2>/dev/null; then
echo "Deploy successful (healthy after ~$((i*5))s)"
break
fi
Expand Down Expand Up @@ -874,7 +888,7 @@ jobs:
# pre-flights with the running stack untouched.
#
# Rolling back in that case would be actively harmful rather than
# merely pointless: tower-finder:rollback and the newest deploy-*
# merely pointless: retina-server:rollback and the newest deploy-*
# tag still describe the PREVIOUS deploy's snapshot, so rollback.sh
# would drag production back past a deploy that succeeded in order
# to recover from one that never started.
Expand Down
11 changes: 9 additions & 2 deletions .github/workflows/deploy-test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -174,7 +174,14 @@ jobs:
pip3 install -e libs/retina-custody/ --break-system-packages --ignore-installed -q
pip3 install -e libs/retina-simulation/ --break-system-packages -q
docker builder prune -f --keep-storage 10GB
docker compose up -d --build tower-finder
# --remove-orphans because this deploy renamed the compose
# service. container_name is pinned, so the container kept by the
# old service key is an orphan holding the exact name the new one
# needs, and compose leaves orphans alone by default: the create
# then fails with a name conflict and the stack does not come up.
# Verified on retina-test before this landed. It stays afterwards
# as the general guard for a removed or renamed service.
docker compose up -d --build --remove-orphans server

# ── Injection point 3 ─────────────────────────────────────────────
# The containers have been replaced. This is the case auto-rollback
Expand All @@ -192,7 +199,7 @@ jobs:
fi

for i in $(seq 1 18); do
if docker compose exec -T tower-finder python3 -c "import urllib.request; urllib.request.urlopen('http://localhost:8000/api/health')" 2>/dev/null; then
if docker compose exec -T server python3 -c "import urllib.request; urllib.request.urlopen('http://localhost:8000/api/health')" 2>/dev/null; then
echo "Deploy successful (healthy after ~$((i*5))s)"
break
fi
Expand Down
3 changes: 0 additions & 3 deletions ONBOARDING.md
Original file line number Diff line number Diff line change
Expand Up @@ -237,9 +237,6 @@ branch, open a PR, get it green, then merge.
branch got no tests, no lint and no build while the lone green automated-review
tick made the page read as passing. Stacked PRs opened since run the full matrix,
but an older branch keeps the old workflow until it is rebased.
- **The compose service is `tower-finder`, not `server`.** `docker compose logs
server` returns nothing and reads as an all-clear when the app is down. Check
`docker compose ps --services` first.
- **A new per-environment key needs an `env-parity` entry** or CI fails.

## Where to go next
Expand Down
2 changes: 1 addition & 1 deletion backend/tests/test_golden_path.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@

Rollback note
─────────────
`deploy/rollback.sh` — rolls back to the saved Docker image (tower-finder:rollback)
`deploy/rollback.sh` — rolls back to the saved Docker image (retina-server:rollback)
or to a specific git ref: `deploy/rollback.sh <tag|commit>`
`deploy/pre-deploy.sh` — saves the current image + creates a git tag.
Called by CI before every deploy; also safe to run manually.
Expand Down
16 changes: 8 additions & 8 deletions deploy/check-env-parity.py
Original file line number Diff line number Diff line change
Expand Up @@ -70,16 +70,16 @@
# Different droplet sizes.
r"^services\.[^.]+\.deploy\.resources\.limits\.(cpus|memory)$",
# Which environment this is, and the hostnames that follow from it.
r"^services\.tower-finder\.environment\.RETINA_ENV$",
r"^services\.server\.environment\.RETINA_ENV$",
# The name a droplet gives itself in an alert, which is a different question
# from RETINA_ENV above and so a different variable. RETINA_ENV picks which
# backend guards apply, and staging and test both answer `test` to that
# during the build-out (ClickUp 86cb1emcx), so it cannot tell an alert's
# origin apart. See services/alerting.py.
r"^services\.tower-finder\.environment\.ALERT_ENVIRONMENT$",
r"^services\.tower-finder\.environment\.CORS_ORIGINS$",
r"^services\.tower-finder\.environment\.CSP_CONNECT_SRC$",
r"^services\.tower-finder\.environment\.HOST_[A-Z_]+$",
r"^services\.server\.environment\.ALERT_ENVIRONMENT$",
r"^services\.server\.environment\.CORS_ORIGINS$",
r"^services\.server\.environment\.CSP_CONNECT_SRC$",
r"^services\.server\.environment\.HOST_[A-Z_]+$",
# AUTH_ALLOW_ANONYMOUS_ADMIN and SYNTHETIC_FLEET_ENABLED are deliberately
# absent from this list: each is set to the same value in every environment,
# so a difference is drift rather than a decision, and CI should fail if one
Expand All @@ -89,7 +89,7 @@
# has none and closes it, so the two legitimately differ here. Recorded rather
# than silently allowed: if staging ever needs node ingest, it should be
# opened deliberately and this entry revisited.
r"^services\.tower-finder\.ports(\..*)?$",
r"^services\.server\.ports(\..*)?$",
# The whole fleet service, not just its FLEET_* scale knobs. Production runs
# no simulator at all (docker-compose.prod.yml puts it behind an unenabled
# `sim` profile), so it drops out of the merged prod config entirely and
Expand All @@ -101,7 +101,7 @@
r"^services\.fleet(\..*)?$",
# Production alone joins the external edge network that fronts
# tower-finder-service; staging has no such stack.
r"^services\.tower-finder\.networks(\..*)?$",
r"^services\.server\.networks(\..*)?$",
r"^networks(\..*)?$",
# Compose records the file list it was assembled from.
r"^name$",
Expand Down Expand Up @@ -215,7 +215,7 @@ def render(env_values: dict[str, str], out_path: Path) -> str:
def check_nginx(tmp: Path) -> list[str]:
rendered = {}
for env, overlay in OVERLAYS.items():
service_env = compose_config(overlay)["services"]["tower-finder"]["environment"]
service_env = compose_config(overlay)["services"]["server"]["environment"]
values = {k: service_env[k] for k in HOST_VARS}
# Pass TLS_ENABLED through when the overlay sets it, so the render below
# reflects what the environment would actually serve. Without this the
Expand Down
2 changes: 1 addition & 1 deletion deploy/classify-migration-gap.py
Original file line number Diff line number Diff line change
Expand Up @@ -144,7 +144,7 @@ def classify(target_ref: str) -> tuple[str, bool]:
verdict = (
" The restored code will query a schema that no longer matches. Downgrade\n"
" before trusting this rollback:\n"
" docker compose exec tower-finder \\\n"
" docker compose exec server \\\n"
f' sh -c "cd /app/backend && python3 -m alembic downgrade {_revision_of(restored[-1])}"'
)

Expand Down
8 changes: 4 additions & 4 deletions deploy/pre-deploy.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2,17 +2,17 @@
# ── Pre-Deploy Snapshot ──────────────────────────────────────────────────────
# Run BEFORE each deploy to save rollback points.
#
# 1. Tags the current running Docker image as `tower-finder:rollback`
# 1. Tags the current running Docker image as `retina-server:rollback`
# 2. Creates a git tag `deploy-<YYYYMMDD-HHMMSS>` on the current commit
#
# Usage: deploy/pre-deploy.sh
# ─────────────────────────────────────────────────────────────────────────────
set -euo pipefail

APP_DIR="${APP_DIR:-/opt/retina-server}"
IMAGE_NAME="tower-finder"
COMPOSE_SERVICE="tower-finder"
FLEET_IMAGE_NAME="tower-finder-fleet"
IMAGE_NAME="retina-server"
COMPOSE_SERVICE="server"
FLEET_IMAGE_NAME="retina-server-fleet"
FLEET_SERVICE="fleet"

cd "$APP_DIR"
Expand Down
10 changes: 5 additions & 5 deletions deploy/rollback.sh
Original file line number Diff line number Diff line change
Expand Up @@ -9,15 +9,15 @@
# How it works:
# Before each deploy, the CI pipeline (or manual deploy) should call
# `deploy/pre-deploy.sh` which tags the current image as
# `tower-finder:rollback` and creates a git tag `deploy-<timestamp>`.
# `retina-server:rollback` and creates a git tag `deploy-<timestamp>`.
#
# This script restores service from that saved image or a given git ref.
# ─────────────────────────────────────────────────────────────────────────────
set -euo pipefail

APP_DIR="${APP_DIR:-/opt/retina-server}"
IMAGE_NAME="tower-finder"
FLEET_IMAGE_NAME="tower-finder-fleet"
IMAGE_NAME="retina-server"
FLEET_IMAGE_NAME="retina-server-fleet"
# No `-f` flags below: the host's ./.env sets COMPOSE_FILE to the shared base
# plus that host's overlay (deploy/env.*.example), so `docker compose` here
# resolves exactly what the deploy resolved. Passing docker-compose.yml alone
Expand Down Expand Up @@ -172,7 +172,7 @@ else
# the retag is a harmless no-op — as is the "no rollback image" branch,
# since pre-deploy.sh finds no fleet image to save there either.
# Guarded because a box predating this change (or one where pre-deploy.sh
# found no running fleet) never captured tower-finder-fleet:rollback — in
# found no running fleet) never captured retina-server-fleet:rollback — in
# that case roll the app back cleanly and leave the fleet image as-is.
if docker image inspect "${FLEET_IMAGE_NAME}:rollback" >/dev/null 2>&1; then
docker tag "${FLEET_IMAGE_NAME}:rollback" "${FLEET_IMAGE_NAME}:latest"
Expand All @@ -196,7 +196,7 @@ fi
# ── Wait for health ──────────────────────────────────────────────────────────
echo "Waiting for server to become healthy..."
for i in $(seq 1 12); do
if docker compose exec -T tower-finder \
if docker compose exec -T server \
python3 -c "import urllib.request; urllib.request.urlopen('http://localhost:8000/api/health')" 2>/dev/null; then
if [ "$DB_NEEDS_DOWNGRADE" = 1 ]; then
echo "Service back after ~$((i*5))s, but a database downgrade is outstanding."
Expand Down
4 changes: 2 additions & 2 deletions docker-compose.local.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,9 +24,9 @@
# Requires Docker Compose >= 2.24 for the !reset / !override merge tags.

services:
tower-finder:
server:
container_name: retina-local-server
# The base pins tower-finder:latest for the rollback tooling on the droplets.
# The base pins retina-server:latest for the rollback tooling on the droplets.
# Reset it so a laptop build cannot collide with a tag the deploy scripts key
# on, and gets Compose's implicit per-project name instead.
image: !reset null
Expand Down
4 changes: 2 additions & 2 deletions docker-compose.prod.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
# the same commit, so the divergence is a recorded decision rather than drift.

services:
tower-finder:
server:
container_name: retina-prod-server
# Consumed by services/alerting.py, which otherwise sees the container id or
# the service name (varies by Compose version), neither of which says which
Expand Down Expand Up @@ -101,7 +101,7 @@ services:
# The sizing below is kept, not deleted, so the fleet can be brought back for
# an afternoon without a PR:
# docker compose --profile sim up -d --build --no-deps fleet
# docker compose restart tower-finder # re-registers the synthetic nodes
# docker compose restart server # re-registers the synthetic nodes
# Staging and test are unaffected and still run theirs by default.
profiles: ["sim"]
environment:
Expand Down
4 changes: 2 additions & 2 deletions docker-compose.staging.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,14 +16,14 @@
# the same commit, so the divergence is a recorded decision rather than drift.

services:
tower-finder:
server:
container_name: retina-staging-server
# Consumed by services/alerting.py, which otherwise sees the container id or
# the service name (varies by Compose version), neither of which says which
# droplet raised an alert.
hostname: retina-staging
# No TCP ingest on staging. The base publishes 3012 for real receiver nodes;
# staging has none, and its own fleet reaches the server as tower-finder:3012
# staging has none, and its own fleet reaches the server as server:3012
# over the compose network, never via the host port. Verified with `ss`: zero
# established connections on it. Publishing it was therefore pure exposure —
# and worse than it looks, because RETINA_ENV=test silences the missing
Expand Down
4 changes: 2 additions & 2 deletions docker-compose.test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -19,14 +19,14 @@
# that matters.

services:
tower-finder:
server:
container_name: retina-test-server
# Consumed by services/alerting.py, which otherwise sees the container id or
# the service name (varies by Compose version), neither of which says which
# droplet raised an alert.
hostname: retina-test
# No TCP ingest, for the same reason staging closes it: there are no real
# receiver nodes here, and the fleet reaches the server as tower-finder:3012
# receiver nodes here, and the fleet reaches the server as server:3012
# over the compose network rather than via the host port. Publishing it would
# be pure exposure — and RETINA_ENV=test silences the missing-RADAR_NODE_TOKEN
# warning, so anything on the internet could register itself as a node.
Expand Down
18 changes: 9 additions & 9 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,15 +17,15 @@
# overlay. deploy/check-env-parity.sh enforces that in CI.

services:
tower-finder:
server:
build: .
# Pin an explicit image name so it is stable and independent of the Compose
# project (directory) name. deploy/pre-deploy.sh + deploy/rollback.sh key the
# rollback on `tower-finder:latest`; without this, Compose builds an
# auto-named `<dir>-tower-finder` image and the rollback tag is never the one
# rollback on `retina-server:latest`; without this, Compose builds an
# auto-named `<dir>-server` image and the rollback tag is never the one
# Compose actually runs, so `rollback.sh` silently restarts the bad build.
# In the base so staging is rollback-capable on the same terms as prod.
image: tower-finder:latest
image: retina-server:latest
ports:
- "80:80"
- "443:443"
Expand Down Expand Up @@ -109,13 +109,13 @@ services:
build:
context: .
dockerfile: Dockerfile.fleet
# Pin an explicit image name (mirrors the app's `image: tower-finder:latest`)
# Pin an explicit image name (mirrors the app's `image: retina-server:latest`)
# so pre-deploy/rollback can save + retag a stable fleet image. Without this
# Compose auto-names it `<dir>-fleet` and a rollback would leave the fleet on
# the bad deploy's build while only the app reverted.
image: tower-finder-fleet:latest
image: retina-server-fleet:latest
depends_on:
tower-finder:
server:
condition: service_healthy
# RADAR_API_KEY comes from backend/.env (the same file the app loads). The
# backend enforces X-API-Key on its ingest whenever it has the key set
Expand All @@ -128,13 +128,13 @@ services:
env_file:
- backend/.env
environment:
- FLEET_HOST=tower-finder
- FLEET_HOST=server
- FLEET_PORT=3012
- FLEET_VALIDATE=false
# HTTP base URL for live ADS-B / ground-truth pushes (the moving "truth"
# tracks). Must point at the server over the compose network, not the
# fleet's own localhost. Reaches uvicorn directly (UVICORN_HOST=0.0.0.0).
- FLEET_VALIDATION_URL=http://tower-finder:8000
- FLEET_VALIDATION_URL=http://server:8000
- FLEET_SEED=42
restart: unless-stopped
logging:
Expand Down
2 changes: 1 addition & 1 deletion docs/alerting.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ Three layers, in order of what they catch:
`host: <value>` line, then one `key: value` line per `meta` entry. Both
shapes carry `environment` (from `ALERT_ENVIRONMENT`, or the literal
`unknown` when unset or empty) and `host` (from `socket.gethostname()`, set
by the `hostname:` each droplet overlay gives its `tower-finder` service, or
by the `hostname:` each droplet overlay gives its `server` service, or
`unknown` if that call fails or returns empty), because each droplet's
`ALERT_WEBHOOK_URL` points at its own channel: channel routing is
configuration, and a misrouted URL would otherwise put an alert in the
Expand Down
Loading
Loading