Skip to content

stage safe bootstrap - #366

Merged
Sancus merged 2 commits into
stagefrom
feat/stage-deploy-safety
Mar 5, 2026
Merged

stage safe bootstrap#366
Sancus merged 2 commits into
stagefrom
feat/stage-deploy-safety

Conversation

@e9e4e5f0faef

Copy link
Copy Markdown
Collaborator

Description

Safety layers for the first stage ECS deployment. Ensures zero tasks run and zero database writes are possible on pulumi up, even if individual layers fail.

Safety is achieved even if CI pushes a new image: desired counts remain 0, autoscaling suspended, schedules disabled.

Safety layers

Layer What How
1. Compute ECS services start with 0 tasks desired_count: 0 on all 3 services
2. Autoscaling Autoscaler cannot bring tasks up suspend: true, min_capacity: 0 on all 3 services
3. Scheduler Cron jobs cannot fire All 16 EventBridge schedules created as DISABLED
4. Database App uses read-only DB credentials if tasks somehow start BOOTSTRAP_SAFE=true uses RO DB secret (atn/stage/mysql_ro); missing secret fails fast rather than falling back to RW

Files changed (4)

File Change
infra/pulumi/config.stage.yaml desired_count: 0, suspend: true, min_capacity: 0, BOOTSTRAP_SAFE=true env var on all services, force_delete: false on ECR
infra/pulumi/__main__.py Schedules default to DISABLED, cron task definition uses settings_local_stage, BOOTSTRAP_SAFE=true
settings_local_stage.py BOOTSTRAP_SAFE toggle: picks atn/stage/mysql_ro when true, sets Django READ_ONLY
.gitignore Broadened preview-*.txt pattern

Pulumi state imports (not in code -- cloud side state only)

The following AWS resources were imported into the Pulumi stack so the ECR publishing path (OIDC role, policy, and repo) is managed declaratively:

  • ECR repository atn-stage-addons-server (imported under Pulumi resource thunderbird-addons-stage-addons-server)
  • IAM role thunderbird-addons-stage-gha-ecr-publish (imported under Pulumi resource thunderbird-addons-stage-gha-ecr-publish)
  • IAM policy addons-stage-ci-ecr-push (imported under Pulumi resource thunderbird-addons-stage-gha-ecr-push-policy)
  • IAM role-policy attachment (imported under Pulumi resource thunderbird-addons-stage-gha-ecr-policy-attachment)

On next pulumi up, Pulumi will normalise these resources (ECR IMMUTABLE -> MUTABLE, trust policy hardening with job_workflow_ref, tag standardisation) -- confirmed in pulumi preview output.

Scale-up sequence (after deploy)

  1. Run RO healthcheck as one-off ECS task (validates app boots with all backends)
  2. Unsuspend versioncheck autoscaling and set min_capacity: 1 (read-heavy, safest first)
  3. Observe, then unsuspend web
  4. Workers last (coordinate with EC2 worker shutdown to avoid double-processing)
  5. Enable EventBridge schedules one by one

Only after prerequisites are met and we explicitly flip desired counts/schedules.

Prerequisites before scaling services above 0

  • Create RO MySQL user (atn_stage_ro@10.100.% with SELECT only) -- request made
  • Create atn/stage/mysql_ro secret in Secrets Manager with RO credentials

Checklist

  • Add a description of the changes introduced in this PR
  • The change has been successfully run locally (pulumi preview passes with all safety layers verified)
  • Screenshots -- N/A, no UI changes

@e9e4e5f0faef
e9e4e5f0faef requested a review from Sancus February 28, 2026 11:50
@e9e4e5f0faef e9e4e5f0faef self-assigned this Feb 28, 2026
@Sancus
Sancus merged commit 4c9a6d6 into stage Mar 5, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants