Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions .ilana/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -345,7 +345,7 @@ Redis queue polling (200ms..2s), not blocking primitives (multi-condition wake-u
## Broadcasts / bulk sending (v0.36)

- Orchestration ONLY over the EXISTING send pipeline (MIME/DKIM/outbox/queue/worker/SMTP unchanged): a broadcast never becomes one giant SMTP transaction — each recipient becomes its own normal `messages` row via the SAME `database.InsertMessage` POST /v1/emails uses.
- State machine (`broadcasts.status`): accepted -> expanding -> completed | failed. "completed" means every recipient was either suppressed or handed to the existing pipeline — NEVER delivered/inbox placement. "failed" is used only for a whole-broadcast condition (From domain no longer verified), never for one bad recipient.
- State machine (`broadcasts.status`): accepted -> expanding -> completed | failed, with accepted/expanding -> cancelled. "completed" means every recipient was either suppressed or handed to the existing pipeline — NEVER delivered/inbox placement. "failed" is used for a whole-broadcast failure; migration 000035 rollback also maps cancelled broadcasts to failed while preserving the cancellation reason.
- Two durable phases per (broadcast, contact): SNAPSHOT (`broadcast_recipients` row: contact_id/email/name/attributes copied once) then MATERIALIZE (render -> MIME -> DKIM -> `InsertMessage`, status pending->suppressed|materialized). `internal/broadcast.Expander` (mirrors `internal/dispatch`'s outbox-polling pattern) advances both phases in bounded per-tick batches (200 audience rows / 25 recipients), never the whole audience in memory or one transaction.
- Audience snapshot semantics: `broadcasts.audience_snapshot_at` is stamped by PostgreSQL's `now()` at acceptance (not Go's clock — avoids Go/DB clock skew, found and fixed during testing). Expansion only ever reads `audience_members` rows with `created_at <= audience_snapshot_at`, so a contact added afterward is NEVER included, however long expansion takes (tested). A member removed from the audience before its batch is scanned is a documented narrow race (RSK-038), not a correctness bug.
- Template snapshot: `broadcasts.subject_template/text_template/html_template` copy the Template's CURRENT source at acceptance; the live `templates` table is NEVER re-read during expansion (tested: edited-then-deleted template still produces the original frozen content for every recipient, mutation-checked).
Expand All @@ -355,8 +355,8 @@ Redis queue polling (200ms..2s), not blocking primitives (multi-condition wake-u
- Suppression: reused authority (`suppression.Normalize` + `SuppressedForTenant`), checked LATE — immediately before materialization, not just at acceptance — so a suppression created while a broadcast is still expanding still stops any not-yet-materialized recipient (tested). Suppressed recipients consume zero quota and zero SMTP attempts.
- Abuse controls: materialization charges the SAME tenant recipient bucket as `/v1/emails` (`ratelimit.TenantRecipientKey`) and respects the same `CountTenantQueued`/`CountPendingOutbox` backpressure before each batch; a refusal leaves recipients `pending` for a later tick (this IS the broadcast's backpressure/fairness mechanism — it never bypasses v0.31, it proceeds more slowly). API keys cannot multiply broadcast throughput (same tenant-scoped accounting).
- Tenant isolation: every DB query tenant-scoped; audience/template ownership validated authoritatively in `handleCreate`, not merely because auth passed; cross-tenant references are 404, never disclosing which resource was missing/foreign (tested both directions).
- Cancellation: DEFERRED (not implemented) — explicitly documented rather than faked; a partial/fake cancel that still lets workers send would be worse than none.
- API: `POST/GET /v1/broadcasts`, `GET /v1/broadcasts/{id}/recipients` (keyset). No PATCH, no DELETE (deliberately omitted — see limitations). Scopes `broadcasts:read`/`broadcasts:write`.
- Cancellation: accepted/expanding broadcasts can stop further snapshots and message handoffs (DEC-250). Repeating cancellation returns the cancelled broadcast. Already queued messages continue through the ordinary delivery pipeline; cancellation does not recall them.
- API: `POST/GET /v1/broadcasts`, `GET /v1/broadcasts/{id}/recipients` (keyset), `POST /v1/broadcasts/{id}/cancel` and `POST /v1/broadcasts/{id}/reschedule`. Both control endpoints are tenant-scoped and require `broadcasts:write`; reads require `broadcasts:read`. No PATCH or DELETE.
- Feedback/events: no `broadcast.*` events or bounce truth store added — materialized recipients are normal messages, so v0.32 feedback and existing `email.*` events already apply via the message's own id; `BroadcastRecipient.message_id` is the correlation handle.
- Metrics: `mailx_broadcast_expansion_batches_total{phase,outcome}` (bounded: 3x5 series). No broadcast/tenant/contact id ever labeled.
- Retention: `broadcasts`/`broadcast_recipients` are durable historical/operational records (like `delivery_attempts`), not auto-expired; deleting a broadcast is not implemented, so no cascade-into-history risk exists yet.
Expand All @@ -376,7 +376,7 @@ Redis queue polling (200ms..2s), not blocking primitives (multi-condition wake-u
- Restart/downtime: no additional recovery path needed — a scheduled row is ordinary durable Postgres state (`outbox`/`broadcasts`), found by the SAME polling queries dispatch/expansion already run every tick; a crashed or restarted process simply resumes polling and any now-due row becomes eligible, exactly like the pre-existing v0.15/v0.36 crash-recovery model. No new crash window was introduced.
- Concurrency: unaffected — outbox dispatch was already duplicate-safe (JobID + idempotent `MarkOutboxDispatched`), and `ClaimActiveBroadcasts`'s `FOR UPDATE SKIP LOCKED` (unchanged) still prevents two expander instances claiming the same broadcast.
- Idempotency: `send_at`/`scheduled_at` are ordinary fields in the fingerprinted request struct (`idempotency.Fingerprint` hashes the whole validated request) — a retry with the same Idempotency-Key but a different scheduled time is a 409 conflict, not a silent reschedule (tested both paths).
- Reschedule/cancel: NOT implemented (DEC-130). `send_at` is immutable after acceptance, same posture as v0.36's broadcast-configuration immutability (DEC-123). Deferred rather than adding unsafe/racy cancel semantics (RSK-039 unchanged).
- Broadcast controls (DEC-250): `POST /v1/broadcasts/{id}/reschedule` accepts a future RFC3339 `send_at` only while status is accepted. `POST /v1/broadcasts/{id}/cancel` changes accepted/expanding to cancelled and repeats idempotently. PostgreSQL row locks fence stale activation, snapshots, claims and message handoffs; recipient progress commits with the message and outbox. Cancelled pending recipients are projected at read time; existing messages remain materialized, and analytics counts only the durable snapshot. Migration `000035_broadcast_control.up.sql` adds cancelled to the status constraint. Its down migration changes cancelled broadcasts to failed with `failure_reason = 'cancelled'` before restoring the previous constraint, so rollback does not reactivate them. Upgrade every expander and message-handoff process before enabling controls.
- Fairness/backpressure: unaffected — a large overdue backlog is still bounded by the SAME per-tick batch sizes (outbox's fair round-robin batch, expander's 200/25 batches); scheduling introduces no new unbounded-selection path.
- Indexes: `idx_broadcasts_active_due (send_at, created_at) WHERE status IN ('accepted','expanding')`. EXPLAIN at 2003 active broadcasts (2000 future-scheduled, 3 due): bitmap index scan, 8 buffer hits, 0.018ms — no sequential scan (`TestClaimActiveBroadcastsDueScanAtScale`, MET-072).
- API: no new scopes — reuses `emails:send`/`broadcasts:write`. OpenAPI documents `scheduled_at`/`send_at` on both request and response schemas, including the "eligibility boundary, not exact delivery time" semantics.
Expand Down
2 changes: 2 additions & 0 deletions .ilana/decisions.md
Original file line number Diff line number Diff line change
Expand Up @@ -254,3 +254,5 @@ Process decisions above (DEC-001..DEC-007) belong to the v0.23 FLEET run and sta
- DEC-247 [v0.47 dashboard API keys] (renumbered from DEC-243): (a) create/rotate pass a per-human `apiKeyMintLimitMiddleware` (`Policy.APIKeyMintRate`/`APIKeyMintBurst`, default 1/30 rps burst 10, env `MAILX_LIMIT_APIKEY_MINT_RPS`/`_BURST`), same shape as `orgInviteLimitMiddleware`; list/revoke are unlimited beyond humanAuth. (b) HTTP rotate uses grace 0 (old key invalid immediately); rotating a revoked/expired key -> 409 `api_key_not_active`. (c) Revoke matches the CLI exactly: an already-revoked key yields 404 `api_key_not_found` and no change (NOT idempotent-204 like invite revoke) - deliberate consistency with `database.RevokeAPIKey`.
- DEC-248 [org API keys, merge review]: independently verified the API-key-management agent's work before pushing (full test suite + `-race`, manual review of `orgAPIKeyHandler`/`orgKey`'s cross-tenant isolation and scope validation, OpenAPI, Docker boot smoke) and additionally live-tested the exact end-to-end claim that matters most: signed up a real account via `POST /v1/auth/signup`, created an org, minted an API key via `POST /v1/orgs/{id}/api-keys`, and used the raw key to successfully call the real `GET /v1/domains` (a `requireScope`-protected, API-key-only route) against the running server - confirming a dashboard-signed-up user can now reach the rest of the tenant-scoped API without any CLI/operator involvement, closing the gap this milestone exists to close. Resolved a DEC-number collision with the concurrently-built email-verification work (both independently claimed DEC-241..243 from the same base commit): renumbered this work's entries to DEC-245..247 in decisions.md and the two source files (`internal/api/apikeys_handler.go`, `internal/ratelimit/policy.go`) that reference them.
- DEC-249 [PR #28, CodeRabbit review pass]: fixed 3 findings. (1) **Rotate error mapping** (functional correctness): `auth.Service.Rotate` returned a bare `fmt.Errorf` string for a revoked/expired key, not the `auth.ErrRevoked`/`ErrExpired` sentinels used elsewhere - so the HTTP rotate handler's `errors.Is` check could never match the race where a key is revoked/expired concurrently between the handler's own unlocked pre-check and the actual `Rotate` call, falling through to a generic 500 instead of the correct 409 `api_key_not_active`. Fixed by wrapping both returns with their sentinel (`%w`) and adding both to the handler's `errors.Is` check; strengthened the two existing tests (`TestServiceRotateRejectsExpiredKey`/`RevokedKey`, previously only checking `err != nil`) to assert the specific sentinel. (2) **Unbounded verification-email send** (stability): `SignUp` called `sendVerificationEmail` synchronously with no timeout on its own request context, so a slow `SubmissionAcceptor.Accept` (synchronous file+DB work) could stall account creation indefinitely; a naive fix of detaching it into a goroutine tied to a context that gets canceled after the response is written would risk losing the durable outbound record mid-write. Fixed by wrapping the call in a 10-second `context.WithTimeout`, keeping it synchronous (so success/failure still means the record was actually durably written or cleanly gave up) while capping how long a slow mailer can hold up signup. (3) **Stale architecture doc**: `architecture.md`'s email-verification section still described the pre-DEC-244 invalidate-before-send design; updated to describe the current create-send-then-supersede flow. Verified: gofmt/go vet/go build clean, full `go test ./...` and `go test -race ./...` clean, Docker rebuild+boot smoke clean.

- DEC-250 [broadcast lifecycle controls, PR #30]: tenant-scoped `POST /v1/broadcasts/{id}/cancel` and `POST /v1/broadcasts/{id}/reschedule` require `broadcasts:write`. Cancel accepted/expanding broadcasts without recalling queued messages; reschedule only accepted broadcasts to a future RFC3339 `send_at`. Lock the broadcast before recipient rows, and commit message/outbox handoff with recipient progress. Recheck due time during activation. Project cancelled pending recipients at read time instead of updating the whole audience. Migration 000035 adds cancelled to the status constraint; rollback maps cancelled to failed with `failure_reason = 'cancelled'` before restoring the old constraint, preventing reactivation. Upgrade all participating processes before using controls.
3 changes: 3 additions & 0 deletions .ilana/ledger.md
Original file line number Diff line number Diff line change
Expand Up @@ -359,3 +359,6 @@ Merged the API-key-management agent's work (built concurrently with email verifi

## 2026-09-26 | PR #28: CodeRabbit review pass | GATE PASS
Checked CodeRabbit on PR #28 as asked and got 3 real, all-minor findings, fixed all three. The most consequential: auth.Service.Rotate returned a bare error string instead of a matchable sentinel for a revoked/expired key, so the exact race the HTTP handler was written to handle (key revoked concurrently between its own pre-check and the Rotate call) fell through to a generic 500 instead of the intended 409 - the handler's error-mapping code was correct in shape but had nothing real to match against. Fixed by wrapping the returns with auth.ErrRevoked/ErrExpired and strengthening two existing tests that had been asserting only "an error happened" to assert the actual sentinel. Also bounded a previously-unbounded email send in SignUp with a 10s timeout (stalls signup less under a slow mailer, without risking the durable-record-loss a naive goroutine-detach would introduce) and fixed a stale architecture.md paragraph describing the pre-DEC-244 email-verification design. Verified: gofmt/go vet/go build clean, full go test ./... and go test -race ./... clean, Docker rebuild+boot smoke clean. Ilana updated: decisions.md (DEC-249), state.json (DEC counter).

## 2026-10-17 | PR #30 broadcast-control documentation | VERIFIED
Updated architecture, DEC-250, milestones, RSK-039 and state to match the cancel/reschedule endpoints, cancelled status and migration 000035. Documented rollback to failed with `failure_reason = 'cancelled'`, preserving the stop across rollback/reapply. Verified against route registration, handlers, OpenAPI and both migration files. Focused existing endpoint, OpenAPI and PostgreSQL rollback/reapply tests passed without skips. No production code changes.
5 changes: 5 additions & 0 deletions .ilana/milestones.md
Original file line number Diff line number Diff line change
Expand Up @@ -132,3 +132,8 @@ explicitly deferred to a later phase — v0.47 as a whole is NOT complete.
## v0.47 dashboard — Org API-key management over HTTP — COMPLETE (DEC-245..247)
- Owner-managed create/list/rotate/revoke under `/v1/orgs/{id}/api-keys` (human JWT), reusing the CLI's `auth.Service`. Unblocker for the rest of the dashboard: domains/emails/templates/webhooks/logs routes stay API-key + `requireScope`, and a web-signed-up org can now mint that key itself.
- Tests: `internal/api/apikeys_handler_test.go` (non-member 404 / non-owner 403 table; HTTP-created key authenticates on real `GET /v1/domains`; scope enforced (403 without domains:read); list leaks no key material; rotate kills old key and new key works; revoke kills key; unknown/duplicate/empty scope 422). Rate limiter itself not exercised by a test (mirrors the tested invite limiter).

## Broadcast lifecycle controls — PR #30
- Implemented: tenant-scoped `POST /v1/broadcasts/{id}/cancel` and `POST /v1/broadcasts/{id}/reschedule` (`broadcasts:write`). Cancellation changes accepted/expanding to cancelled; rescheduling accepts a future RFC3339 `send_at` only while accepted. Already queued messages continue.
- Migration 000035 adds cancelled to the broadcast status constraint. Rollback changes cancelled broadcasts to failed with `failure_reason = 'cancelled'` before restoring the previous constraint; rollback/reapply does not restart them.
- Verified: endpoint workflow, authorization/isolation, input validation, expanding-state conflicts, OpenAPI/runtime route agreement and migration rollback/reapply against real PostgreSQL. Upgrade all participating processes before enabling controls. Pending PR review; no new numbered milestone declared.
Loading
Loading