Skip to content

[SES-PHASE-1-3] - PLU-744: add ses event parser#1629

Open
m0nggh wants to merge 2 commits into
feat/ses/trunkfrom
feat/ses/add-parser-and-worker-processor
Open

[SES-PHASE-1-3] - PLU-744: add ses event parser#1629
m0nggh wants to merge 2 commits into
feat/ses/trunkfrom
feat/ses/add-parser-and-worker-processor

Conversation

@m0nggh
Copy link
Copy Markdown
Contributor

@m0nggh m0nggh commented May 19, 2026

TL;DR

Adds a SES event processor that suppresses or whitelists email addresses based on bounce and complaint notifications received via SNS/SQS.

TODOs:

  • The postman SES configuration set needs to be configured to only send bounce and complaint event types to the postman SNS topic (now it sends all)

What changed?

  • Added ses-event-parser.ts helper that parses SQS messages containing SNS-wrapped SES events, exposing typed interfaces for bounce and complaint events.
  • Added process-ses-event.ts with a processSesEvent function that:
    • Suppresses email addresses on permanent bounces (storing bounce subtype and SES message ID).
    • Logs and ignores transient/undetermined bounces without suppressing.
    • Suppresses email addresses on complaint events (abuse, fraud, virus, other, or null feedback types).
    • Auto-whitelists previously suppressed emails when a not-spam complaint is received.
    • Logs a warning for any unrecognised event types.
  • Added SNS/SES fixture JSON files for permanent bounce, transient bounce, abuse complaint, and not-spam complaint events to support testing.
  • Added unit tests for parseSqsMessage covering valid events, invalid JSON, missing Message field, and missing eventType.
  • Added integration tests for processSesEvent covering suppression on permanent bounce, no suppression on transient bounce, suppression on abuse complaint, auto-whitelisting on not-spam complaint, graceful handling of not-spam for non-suppressed emails, and idempotency.

How to test?

Run the unit and integration tests:

vitest run src/helpers/__tests__/ses-event-parser.test.ts
vitest run src/helpers/__tests__/process-ses-event.itest.ts

The fixture files under ses-test-events/ can also be used to manually invoke the processor against a local SQS queue to verify end-to-end behaviour.

Why make this change?

SES delivers bounce and complaint notifications via SNS → SQS. Without handling these events, suppressed or complaining recipients would continue to receive emails, risking SES account reputation and deliverability. This processor ensures that permanently bounced and complained-about addresses are suppressed from future sends, while also providing a recovery path when recipients mark emails as not-spam.

@m0nggh m0nggh force-pushed the feat/ses/add-parser-and-worker-processor branch from 66e7e1a to 7fa6e51 Compare May 19, 2026 09:22
@m0nggh m0nggh force-pushed the feat/ses/add-email-suppression-migration-and-model branch 2 times, most recently from 3075d8d to f6d6920 Compare May 19, 2026 09:50
@m0nggh m0nggh force-pushed the feat/ses/add-parser-and-worker-processor branch from 7fa6e51 to 901af9b Compare May 19, 2026 09:50
@m0nggh m0nggh changed the title add ses event parser and worker processor [SES-PHASE-1-3] - PLU-744: add ses event parser and worker processor May 19, 2026
@linear
Copy link
Copy Markdown

linear Bot commented May 19, 2026

PLU-744

Comment thread packages/backend/src/helpers/process-ses-event.ts
@m0nggh m0nggh changed the title [SES-PHASE-1-3] - PLU-744: add ses event parser and worker processor [SES-PHASE-1-3] - PLU-744: add ses event parser May 22, 2026
Comment thread packages/backend/src/helpers/ses-event-parser.ts
Comment thread packages/backend/src/helpers/process-ses-event.ts
Comment thread packages/backend/src/helpers/ses-event-parser.ts
Comment thread packages/backend/src/helpers/process-ses-event.ts
Copy link
Copy Markdown
Contributor

@ogp-weeloong ogp-weeloong left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lmk what you think about the zod stuff. Otherwise i dont think there's any big blockers tbh (no trapdoor decisions)

Copy link
Copy Markdown
Contributor

@pregnantboy pregnantboy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks mostly good.

Comment on lines +85 to +86
const emails = complainedRecipients.map((r) => r.emailAddress)
const whitelisted = await EmailSuppressionEntry.whitelistEmails(emails)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is interesting. i guess a user who can emit a not-spam complain has a valid email

Comment thread packages/backend/src/helpers/process-ses-event.ts
@m0nggh m0nggh force-pushed the feat/ses/add-parser-and-worker-processor branch from b0fd3f3 to 3fddfc4 Compare June 8, 2026 09:03
@m0nggh m0nggh force-pushed the feat/ses/add-email-suppression-migration-and-model branch from f6d6920 to 61afd7d Compare June 8, 2026 09:03
@datadog-opengovsg

This comment has been minimized.

@m0nggh m0nggh force-pushed the feat/ses/add-parser-and-worker-processor branch from 3fddfc4 to 59cbb20 Compare June 8, 2026 11:01
@m0nggh m0nggh force-pushed the feat/ses/add-email-suppression-migration-and-model branch from 61afd7d to 8af238d Compare June 8, 2026 11:01
@m0nggh m0nggh force-pushed the feat/ses/add-parser-and-worker-processor branch from 59cbb20 to 8c50986 Compare June 8, 2026 12:02
@m0nggh m0nggh force-pushed the feat/ses/add-email-suppression-migration-and-model branch 2 times, most recently from 47578d4 to c635cfa Compare June 8, 2026 12:04
@m0nggh m0nggh force-pushed the feat/ses/add-parser-and-worker-processor branch from 8c50986 to 95b7b1c Compare June 8, 2026 12:04
@m0nggh m0nggh requested a review from Copilot June 8, 2026 12:05
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds SES bounce/complaint event handling support to the backend, including parsing of SNS-wrapped SES events delivered via SQS and a processor that updates the email suppression/whitelist state based on event semantics.

Changes:

  • Introduces parseSqsMessage + typed SES event shapes for Bounce/Complaint events.
  • Adds processSesEvent to suppress on permanent bounces/complaints and auto-whitelist on not-spam complaints.
  • Adds unit/integration tests and fixture JSON payloads for representative SES events.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
packages/backend/src/helpers/ses-event-parser.ts New helper to parse SQS → SNS envelope → SES event JSON into a typed shape.
packages/backend/src/helpers/process-ses-event.ts New processor to apply suppression/whitelist logic based on parsed SES events.
packages/backend/src/helpers/tests/ses-event-parser.test.ts Unit tests for parser success/error paths.
packages/backend/src/helpers/tests/process-ses-event.itest.ts Integration tests validating suppression/whitelisting behavior and idempotency.
packages/backend/ses-test-events/ses-bounce-permanent.json Fixture for a permanent bounce event.
packages/backend/ses-test-events/ses-bounce-transient.json Fixture for a transient bounce event.
packages/backend/ses-test-events/ses-complaint-abuse.json Fixture for an abuse complaint event.
packages/backend/ses-test-events/ses-complaint-not-spam.json Fixture for a not-spam complaint event (auto-whitelist).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread packages/backend/src/helpers/process-ses-event.ts
Comment thread packages/backend/src/helpers/process-ses-event.ts
Comment on lines +73 to +77
logger.error('Complaint event missing complaint payload', {
event: 'ses-malformed-event',
sqsMessageId,
messageId: mail?.messageId,
})
Comment on lines +58 to +60
// Transient / Undetermined — log only, do not suppress
logger.info('Transient bounce received — no suppression', {
event: 'ses-transient-bounce',
Comment on lines +25 to +31
interface SesComplaint {
complainedRecipients: ComplainedRecipient[]
timestamp: string
feedbackId: string
complaintFeedbackType?: string
complaintSubType?: string
}
@m0nggh m0nggh force-pushed the feat/ses/add-parser-and-worker-processor branch from 95b7b1c to cdaa0e1 Compare June 8, 2026 15:02
Base automatically changed from feat/ses/add-email-suppression-migration-and-model to feat/ses/trunk June 8, 2026 15:02
@m0nggh m0nggh force-pushed the feat/ses/add-parser-and-worker-processor branch from cdaa0e1 to 79a7684 Compare June 8, 2026 15:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants