feat(workflows): email admins on AWS SES tenant reputation changes

[meikelmosby]

Problem

Stacked on #75237 (which stacks on #74762). With the Standard reputation policy, AWS can pause a project's email sending — but until now nobody would tell the customer. This is the top-priority follow-up from the AWS-tenant switch: when AWS changes a tenant's sending status or raises reputation findings, the project's admins must hear about it.

Changes

Design principle: events are change signals, not state. The webhook never trusts an event payload — it extracts the team-<id> tenant name and enqueues a sync that reads the authoritative state from the SES API (SESProvider.get_tenant_reputation, from #75237). Duplicate, reordered, or partially-shaped events are therefore harmless, and the reconciliation sweep shares the exact same code path.

State + transitions – TeamWorkflowsConfig gains ses_tenant_sending_status, ses_tenant_reputation_impact, ses_tenant_state_synced_at (migration 0017; db_default so raw non-Django INSERTs keep working). apply_ses_tenant_state persists the new state and emails project admins on meaningful transitions only:

• sending paused (→ DISABLED): reuses the existing suspension email with a provider-pause reason
• sending re-enabled (DISABLED → ENABLED/REINSTATED): reuses the re-enabled email
• findings escalating (none → LOW, LOW/none → HIGH): new email_sending_reputation_finding template — LOW is a fix-this warning, HIGH is "[Action required] … at risk of being paused"

The stored state is the dedupe mechanism (unchanged state sends nothing), the first-ever sync sets a baseline without notifying (so rollout doesn't mass-email), and de-escalations stay silent. Recipient selection reuses _get_project_admins_to_notify_of_email_sending_suspension — admin+, not gated by notification settings.

Webhook – POST /webhooks/workflows/ses-events, following the repo's external-webhook pattern (@csrf_exempt function view, ack fast, Celery for the work). Auth is two-layer: full SNS signature verification (new sns_verification.py, ported from the Node SES webhook — cert-URL host validation, cached cert fetch, canonical string-to-sign, SignatureVersion 1/2) plus a topic-ARN allowlist (WORKFLOWS_SES_EVENTS_SNS_TOPIC_ARNS, empty by default so the endpoint ships inert). Handles SubscriptionConfirmation with SubscribeURL host validation. Only SNS SignatureVersion 2 (SHA256) is accepted — version 1 signs with SHA1, so instead of verifying a weak hash we require the topic be configured with SignatureVersion=2 (one-time aws sns set-topic-attributes --attribute-name SignatureVersion --attribute-value 2, added to the wiring runbook below).

Reconciliation sweep – daily Celery beat task sweeping every team with an SES email integration through the same sync, since EventBridge delivery is best-effort. Kept daily (3 SES API calls per team) to stay inside SES API rate limits; the webhook is the real-time path.

Note

AWS-side wiring is a runbook step, not code: EventBridge rule on source = aws.ses for the tenant detail-types ("Sending Status Disabled/Enabled", "Advisor Recommendation Status Open/Resolved") → SNS topic with SignatureVersion=2 → HTTPS subscription to this endpoint, then set WORKFLOWS_SES_EVENTS_SNS_TOPIC_ARNS to the topic ARN. In-app notifications (create_notification) are a possible follow-up; this PR ships the email path.

How did you test this code?

• 12 transition-service tests, mostly one parameterized table: baseline-on-first-sync, unchanged-state no-op, pause/unpause/reinstate, first-finding/escalation/de-escalation/resolution, plus payload assertions (provider reason, impact).
• 8 webhook tests: enqueue from detail.tenantName, tenant extraction from resource ARNs, non-aws.ses sources acked but ignored, unknown topic → 403, bad signature → 403, empty allowlist → inert 500, subscription confirmation fetches the SubscribeURL, off-AWS SubscribeURL rejected.
• 12 SNS-verification tests with a real RSA signature roundtrip (self-signed cert): valid signature accepted; tampered message/topic, garbage signature, non-SNS or HTTP cert URLs, unknown types all rejected; URL-validation table including the sns.us-east-1.amazonaws.com.evil.com spoof.
• makemigrations --check clean; repo-wide mypy clean.
• I (Claude) haven't exercised a live EventBridge → SNS delivery; the SNS envelope handling follows the AWS signature spec and mirrors the battle-tested Node webhook.

Automatic notifications

• Publish to changelog?
• Alert Sales and Marketing teams?

Docs update

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Built with PostHog Code (Claude). Skills followed: django-migrations (additive fields with db_default, checked external table writers), writing-tests (parameterized transition table), improving-drf-endpoints n/a (no DRF surface — webhook is a plain view per the repo's webhook convention). Key decision: fetch-on-event instead of trusting event payloads, so the webhook and reconciliation sweep converge on one code path and AWS's evolving event schema can't break state handling.

---

Created with PostHog Code

[posthog]

🦔 ReviewHog reviewed this pull request

Found 0 must fix, 3 should fix, 5 consider.

Published 8 findings (view the review).

[github-actions]

🤖 CI report

⚠️ Django migration SQL — 1 new migration to review

We've detected new migrations on this PR. Review the SQL output for each migration:

products/workflows/backend/migrations/0017_teamworkflowsconfig_ses_tenant_state.py

BEGIN;
--
-- Add field ses_tenant_reputation_impact to teamworkflowsconfig
--
ALTER TABLE "workflows_teamworkflowsconfig" ADD COLUMN "ses_tenant_reputation_impact" varchar(8) DEFAULT '' NOT NULL;
--
-- Add field ses_tenant_sending_status to teamworkflowsconfig
--
ALTER TABLE "workflows_teamworkflowsconfig" ADD COLUMN "ses_tenant_sending_status" varchar(16) DEFAULT '' NOT NULL;
--
-- Add field ses_tenant_state_synced_at to teamworkflowsconfig
--
ALTER TABLE "workflows_teamworkflowsconfig" ADD COLUMN "ses_tenant_state_synced_at" timestamp with time zone NULL;
COMMIT;

Last updated: 2026-07-30 14:56 UTC (c128f3d)

✅ Django migration risk — migration analysis complete

We've analyzed your migrations for potential risks.

Summary: 2 Safe | 1 Needs Review | 0 Blocked

⚠️ Needs Review

May have performance impact

workflows.0016_drop_emailreputationsnapshot_fk
  └─ #1 ✅ SeparateDatabaseAndState
     Wrapper operation - see nested operations for risk: RunSQL
     database_operations: RunSQL
     └─ #2 ⚠️ RunSQL: DROP CONSTRAINT is fast but needs deployment safety review

✅ Safe

Brief or no lock, backwards compatible

workflows.0015_delete_emailreputationsnapshot
  └─ #1 ✅ SeparateDatabaseAndState: Only state operations (no database changes)

workflows.0017_teamworkflowsconfig_ses_tenant_state
  └─ #1 ✅ AddField
     Adding NOT NULL field with constant default (safe in PG11+)
     model: teamworkflowsconfig, field: ses_tenant_reputation_impact
  └─ #2 ✅ AddField
     Adding NOT NULL field with constant default (safe in PG11+)
     model: teamworkflowsconfig, field: ses_tenant_sending_status
  └─ #3 ✅ AddField
     Adding nullable field requires brief lock
     model: teamworkflowsconfig, field: ses_tenant_state_synced_at

📚 How to Deploy These Changes Safely

AddField:

This operation acquires a brief lock but doesn't rewrite the table.

Deployment uses lock timeouts with automatic retries, so lock contention will cause retries rather than connection pile-up.

RunSQL:

⚠️ Deployment Safety: While DROP CONSTRAINT is instant (no table lock), dropping constraints can break running code during rolling deployments.

Safe pattern:

1. Ensure no running code relies on the constraint (uniqueness checks, foreign key validation, etc.)
2. If replacing with a new constraint, deploy the new one first
3. Wait at least one full deployment cycle before dropping the old constraint
4. Consider keeping unused constraints if removal risk outweighs benefits

Common scenarios:

• Dropping UNIQUE constraints: Ensure code handles potential duplicates
• Dropping FOREIGN KEY constraints: Ensure code doesn't assume referential integrity
• Replacing constraints: Add new → deploy → wait → drop old

See the migration safety guide

Last updated: 2026-07-30 14:56 UTC (c128f3d)

[posthog]

ReviewHog Alpha 🦔 If you find any issues helpful - please reply "valid", "invalid", etc., for evaluation purposes 🙏

[posthog]

ReviewHog Report

Business logic

Issues: 5 issues

Files (7)

• products/workflows/backend/migrations/0017_teamworkflowsconfig_ses_tenant_state.py
• products/workflows/backend/models/team_workflows_config.py
• products/workflows/backend/services/ses_tenant_state.py
• products/workflows/backend/tasks/ses_tenant_state.py
• posthog/tasks/email.py
• posthog/templates/email/email_sending_reputation_finding.html
• posthog/tasks/scheduled.py

What were the main changes

• Migration 0017 adds ses_tenant_sending_status, ses_tenant_reputation_impact, ses_tenant_state_synced_at to TeamWorkflowsConfig with db_default so raw non-Django INSERTs keep working
• apply_ses_tenant_state/sync_ses_tenant_state persist authoritative SES tenant state and email project admins only on meaningful transitions (pause, unpause/reinstate, reputation escalation); unchanged state and de-escalations are silent, and the first-ever sync sets a baseline without notifying
• New Celery tasks: sync_ses_tenant_state_task (webhook-triggered) and reconcile_ses_tenant_states (daily sweep over teams with an SES email integration), the latter registered in scheduled.py at 6:30 AM UTC
• New send_email_sending_reputation_finding email task plus email_sending_reputation_finding.html template distinguishing LOW (warning) vs HIGH (action required) impact

Api

Issues: 3 issues

Files (4)

• products/workflows/backend/api/ses_events_webhook.py
• products/workflows/backend/services/sns_verification.py
• posthog/urls.py
• posthog/settings/ses.py

What were the main changes

• New POST /webhooks/workflows/ses-events endpoint that extracts the team- tenant from EventBridge SES events and enqueues sync_ses_tenant_state_task rather than trusting event payload state
• Handles SNS SubscriptionConfirmation with SubscribeURL host validation, and Notification messages by parsing the embedded EventBridge event and filtering to source aws.ses
• New sns_verification.py implementing full SNS signature verification: cert-URL host validation, cached certificate fetch, canonical string-to-sign construction, and SignatureVersion 1/2 RSA verification
• Two-layer auth: SNS signature proves the message is from AWS, new WORKFLOWS_SES_EVENTS_SNS_TOPIC_ARNS allowlist (empty by default, ships inert) proves it's from the project's own topic
• URL routing wires the new webhook view into posthog/urls.py