-
Notifications
You must be signed in to change notification settings - Fork 12
[SES-PHASE-1-3] - PLU-744: add ses event parser #1629
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
m0nggh
merged 3 commits into
feat/ses/trunk
from
feat/ses/add-parser-and-worker-processor
Jun 8, 2026
+415
−0
Merged
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
12 changes: 12 additions & 0 deletions
12
packages/backend/ses-test-events/ses-bounce-permanent.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| { | ||
| "Type": "Notification", | ||
| "MessageId": "fixture-msg-001", | ||
| "TopicArn": "arn:aws:sns:ap-southeast-2:123456789012:ses-events", | ||
| "Subject": null, | ||
| "Message": "{\"eventType\":\"Bounce\",\"bounce\":{\"bounceType\":\"Permanent\",\"bounceSubType\":\"NoEmail\",\"bouncedRecipients\":[{\"emailAddress\":\"bounce@example.com\",\"action\":\"failed\",\"status\":\"5.1.1\",\"diagnosticCode\":\"smtp; 550 5.1.1 user unknown\"}],\"timestamp\":\"2026-05-11T10:00:00.000Z\",\"feedbackId\":\"feedback-001\"},\"mail\":{\"timestamp\":\"2026-05-11T09:59:00.000Z\",\"source\":\"noreply@plumber.gov.sg\",\"sourceArn\":\"arn:aws:ses:ap-southeast-2:123456789012:identity/plumber.gov.sg\",\"sendingAccountId\":\"123456789012\",\"messageId\":\"ses-msg-001\",\"destination\":[\"bounce@example.com\"]}}", | ||
| "Timestamp": "2026-05-11T10:00:01.000Z", | ||
| "SignatureVersion": "1", | ||
| "Signature": "EXAMPLE", | ||
| "SigningCertURL": "https://sns.ap-southeast-2.amazonaws.com/SimpleNotificationService-example.pem", | ||
| "UnsubscribeURL": "https://sns.ap-southeast-2.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:ap-southeast-2:123456789012:ses-events:example" | ||
| } |
12 changes: 12 additions & 0 deletions
12
packages/backend/ses-test-events/ses-bounce-transient.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| { | ||
| "Type": "Notification", | ||
| "MessageId": "fixture-msg-002", | ||
| "TopicArn": "arn:aws:sns:ap-southeast-2:123456789012:ses-events", | ||
| "Subject": null, | ||
| "Message": "{\"eventType\":\"Bounce\",\"bounce\":{\"bounceType\":\"Transient\",\"bounceSubType\":\"MailboxFull\",\"bouncedRecipients\":[{\"emailAddress\":\"full@example.com\",\"action\":\"failed\",\"status\":\"4.2.2\",\"diagnosticCode\":\"smtp; 452 4.2.2 Mailbox full\"}],\"timestamp\":\"2026-05-11T10:00:00.000Z\",\"feedbackId\":\"feedback-002\"},\"mail\":{\"timestamp\":\"2026-05-11T09:59:00.000Z\",\"source\":\"noreply@plumber.gov.sg\",\"sendingAccountId\":\"123456789012\",\"messageId\":\"ses-msg-002\",\"destination\":[\"full@example.com\"]}}", | ||
| "Timestamp": "2026-05-11T10:00:01.000Z", | ||
| "SignatureVersion": "1", | ||
| "Signature": "EXAMPLE", | ||
| "SigningCertURL": "https://sns.ap-southeast-2.amazonaws.com/SimpleNotificationService-example.pem", | ||
| "UnsubscribeURL": "https://sns.ap-southeast-2.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:ap-southeast-2:123456789012:ses-events:example" | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| { | ||
| "Type": "Notification", | ||
| "MessageId": "fixture-msg-003", | ||
| "TopicArn": "arn:aws:sns:ap-southeast-2:123456789012:ses-events", | ||
| "Subject": null, | ||
| "Message": "{\"eventType\":\"Complaint\",\"complaint\":{\"complainedRecipients\":[{\"emailAddress\":\"complainer@example.com\"}],\"timestamp\":\"2026-05-11T10:00:00.000Z\",\"feedbackId\":\"feedback-003\",\"complaintSubType\":null,\"complaintFeedbackType\":\"abuse\"},\"mail\":{\"timestamp\":\"2026-05-11T09:59:00.000Z\",\"source\":\"noreply@plumber.gov.sg\",\"sendingAccountId\":\"123456789012\",\"messageId\":\"ses-msg-003\",\"destination\":[\"complainer@example.com\"]}}", | ||
| "Timestamp": "2026-05-11T10:00:01.000Z", | ||
| "SignatureVersion": "1", | ||
| "Signature": "EXAMPLE", | ||
| "SigningCertURL": "https://sns.ap-southeast-2.amazonaws.com/SimpleNotificationService-example.pem", | ||
| "UnsubscribeURL": "https://sns.ap-southeast-2.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:ap-southeast-2:123456789012:ses-events:example" | ||
| } |
12 changes: 12 additions & 0 deletions
12
packages/backend/ses-test-events/ses-complaint-not-spam.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| { | ||
| "Type": "Notification", | ||
| "MessageId": "fixture-msg-004", | ||
| "TopicArn": "arn:aws:sns:ap-southeast-2:123456789012:ses-events", | ||
| "Subject": null, | ||
| "Message": "{\"eventType\":\"Complaint\",\"complaint\":{\"complainedRecipients\":[{\"emailAddress\":\"notspam@example.com\"}],\"timestamp\":\"2026-05-11T10:00:00.000Z\",\"feedbackId\":\"feedback-004\",\"complaintSubType\":null,\"complaintFeedbackType\":\"not-spam\"},\"mail\":{\"timestamp\":\"2026-05-11T09:59:00.000Z\",\"source\":\"noreply@plumber.gov.sg\",\"sendingAccountId\":\"123456789012\",\"messageId\":\"ses-msg-004\",\"destination\":[\"notspam@example.com\"]}}", | ||
| "Timestamp": "2026-05-11T10:00:01.000Z", | ||
| "SignatureVersion": "1", | ||
| "Signature": "EXAMPLE", | ||
| "SigningCertURL": "https://sns.ap-southeast-2.amazonaws.com/SimpleNotificationService-example.pem", | ||
| "UnsubscribeURL": "https://sns.ap-southeast-2.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:ap-southeast-2:123456789012:ses-events:example" | ||
| } |
105 changes: 105 additions & 0 deletions
105
packages/backend/src/helpers/__tests__/process-ses-event.itest.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,105 @@ | ||
| import { readFileSync } from 'fs' | ||
| import { resolve } from 'path' | ||
| import { describe, expect, it } from 'vitest' | ||
|
|
||
| import { parseSqsMessage } from '@/helpers/ses-event-parser' | ||
| import EmailSuppressionEntry from '@/models/email-suppression-entry' | ||
|
|
||
| import { processSesEvent } from '../process-ses-event' | ||
|
|
||
| function loadFixture(name: string): string { | ||
| return readFileSync( | ||
| resolve(__dirname, '../../../ses-test-events', name), | ||
| 'utf-8', | ||
| ) | ||
| } | ||
|
|
||
| function makeJobData(fixtureName: string) { | ||
| return { | ||
| sesEvent: parseSqsMessage(loadFixture(fixtureName)), | ||
| sqsMessageId: `test-sqs-${fixtureName}`, | ||
| } | ||
| } | ||
|
|
||
| describe('processSesEvent', () => { | ||
| it('should suppress email on permanent bounce', async () => { | ||
| await processSesEvent(makeJobData('ses-bounce-permanent.json')) | ||
|
|
||
| const suppressed = await EmailSuppressionEntry.getSuppressedEmails([ | ||
| 'bounce@example.com', | ||
| ]) | ||
| expect(suppressed).toEqual(['bounce@example.com']) | ||
|
|
||
| const row = await EmailSuppressionEntry.query().findOne({ | ||
| email: 'bounce@example.com', | ||
| }) | ||
| expect(row.reason).toBe('BOUNCE') | ||
| expect(row.reasonDetail).toBe('NoEmail') | ||
| expect(row.sesMessageId).toBe('ses-msg-001') | ||
| }) | ||
|
|
||
| it('should NOT suppress email on transient bounce', async () => { | ||
| await processSesEvent(makeJobData('ses-bounce-transient.json')) | ||
|
|
||
| const suppressed = await EmailSuppressionEntry.getSuppressedEmails([ | ||
| 'full@example.com', | ||
| ]) | ||
| expect(suppressed).toEqual([]) | ||
| }) | ||
|
|
||
| it('should suppress email on abuse complaint', async () => { | ||
| await processSesEvent(makeJobData('ses-complaint-abuse.json')) | ||
|
|
||
| const suppressed = await EmailSuppressionEntry.getSuppressedEmails([ | ||
| 'complainer@example.com', | ||
| ]) | ||
| expect(suppressed).toEqual(['complainer@example.com']) | ||
|
|
||
| const row = await EmailSuppressionEntry.query().findOne({ | ||
| email: 'complainer@example.com', | ||
| }) | ||
| expect(row.reason).toBe('COMPLAINT') | ||
| expect(row.reasonDetail).toBe('abuse') | ||
| }) | ||
|
|
||
| it('should auto-whitelist on not-spam complaint', async () => { | ||
| // First suppress the email | ||
| await EmailSuppressionEntry.upsertSuppression({ | ||
| email: 'notspam@example.com', | ||
| reason: 'COMPLAINT', | ||
| reasonDetail: 'abuse', | ||
| }) | ||
|
|
||
| // Then process not-spam complaint | ||
| await processSesEvent(makeJobData('ses-complaint-not-spam.json')) | ||
|
|
||
| const suppressed = await EmailSuppressionEntry.getSuppressedEmails([ | ||
| 'notspam@example.com', | ||
| ]) | ||
| expect(suppressed).toEqual([]) | ||
|
|
||
| const row = await EmailSuppressionEntry.query().findOne({ | ||
| email: 'notspam@example.com', | ||
| }) | ||
| expect(row.lastWhitelistedAt).not.toBeNull() | ||
| }) | ||
|
|
||
| it('should handle not-spam complaint for non-suppressed email gracefully', async () => { | ||
| await processSesEvent(makeJobData('ses-complaint-not-spam.json')) | ||
|
|
||
| const rows = await EmailSuppressionEntry.query().where({ | ||
| email: 'notspam@example.com', | ||
| }) | ||
| expect(rows).toHaveLength(0) | ||
| }) | ||
|
|
||
| it('should be idempotent — processing same bounce twice does not duplicate', async () => { | ||
| await processSesEvent(makeJobData('ses-bounce-permanent.json')) | ||
| await processSesEvent(makeJobData('ses-bounce-permanent.json')) | ||
|
|
||
| const rows = await EmailSuppressionEntry.query().where({ | ||
| email: 'bounce@example.com', | ||
| }) | ||
| expect(rows).toHaveLength(1) | ||
| }) | ||
| }) |
77 changes: 77 additions & 0 deletions
77
packages/backend/src/helpers/__tests__/ses-event-parser.test.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,77 @@ | ||
| import { readFileSync } from 'fs' | ||
| import { resolve } from 'path' | ||
| import { assert, describe, expect, it } from 'vitest' | ||
|
|
||
| import { parseSqsMessage, SesEventType } from '@/helpers/ses-event-parser' | ||
|
|
||
| function loadFixture(name: string): string { | ||
| return readFileSync( | ||
| resolve(__dirname, '../../../ses-test-events', name), | ||
| 'utf-8', | ||
| ) | ||
| } | ||
|
|
||
| describe('ses-event-parser', () => { | ||
| describe('parseSqsMessage', () => { | ||
| it('should parse a permanent bounce event', () => { | ||
| const result = parseSqsMessage(loadFixture('ses-bounce-permanent.json')) | ||
| assert(result.eventType === SesEventType.Bounce) // narrows the union | ||
| expect(result.bounce.bounceType).toBe('Permanent') | ||
| expect(result.bounce.bounceSubType).toBe('NoEmail') | ||
| expect(result.bounce.bouncedRecipients).toHaveLength(1) | ||
| expect(result.bounce.bouncedRecipients[0].emailAddress).toBe( | ||
| 'bounce@example.com', | ||
| ) | ||
| expect(result.mail.messageId).toBe('ses-msg-001') | ||
| }) | ||
|
|
||
| it('should parse a transient bounce event', () => { | ||
| const result = parseSqsMessage(loadFixture('ses-bounce-transient.json')) | ||
| assert(result.eventType === SesEventType.Bounce) // narrows the union | ||
| expect(result.bounce.bounceType).toBe('Transient') | ||
| expect(result.bounce.bounceSubType).toBe('MailboxFull') | ||
| }) | ||
|
|
||
| it('should parse a complaint event', () => { | ||
| const result = parseSqsMessage(loadFixture('ses-complaint-abuse.json')) | ||
| assert(result.eventType === SesEventType.Complaint) // narrows the union | ||
| expect(result.complaint.complaintFeedbackType).toBe('abuse') | ||
| expect(result.complaint.complainedRecipients).toHaveLength(1) | ||
| expect(result.complaint.complainedRecipients[0].emailAddress).toBe( | ||
| 'complainer@example.com', | ||
| ) | ||
| }) | ||
|
|
||
| it('should parse a not-spam complaint event', () => { | ||
| const result = parseSqsMessage(loadFixture('ses-complaint-not-spam.json')) | ||
| assert(result.eventType === SesEventType.Complaint) // narrows the union | ||
| expect(result.complaint.complaintFeedbackType).toBe('not-spam') | ||
| }) | ||
|
|
||
| it('should throw on invalid JSON in SQS body', () => { | ||
| expect(() => parseSqsMessage('not json')).toThrow() | ||
| }) | ||
|
|
||
| it('should throw on missing Message field in SNS envelope', () => { | ||
| expect(() => | ||
| parseSqsMessage(JSON.stringify({ Type: 'Notification' })), | ||
| ).toThrow(/Message/) | ||
| }) | ||
|
|
||
| it('should throw on missing eventType in SES event', () => { | ||
| const snsEnvelope = JSON.stringify({ | ||
| Type: 'Notification', | ||
| Message: JSON.stringify({ mail: {} }), | ||
| }) | ||
| expect(() => parseSqsMessage(snsEnvelope)).toThrow(/eventType/) | ||
| }) | ||
|
|
||
| it('should throw on an unhandled event type', () => { | ||
| const snsEnvelope = JSON.stringify({ | ||
| Type: 'Notification', | ||
| Message: JSON.stringify({ eventType: 'Delivery', mail: {} }), | ||
| }) | ||
| expect(() => parseSqsMessage(snsEnvelope)).toThrow(/eventType/) | ||
| }) | ||
| }) | ||
| }) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,99 @@ | ||
| import logger from '@/helpers/logger' | ||
| import { SesEvent, SesEventType } from '@/helpers/ses-event-parser' | ||
| import EmailSuppressionEntry from '@/models/email-suppression-entry' | ||
|
|
||
| export interface SesEventInput { | ||
| sesEvent: SesEvent | ||
| sqsMessageId: string | ||
| } | ||
|
|
||
| /** | ||
| * Process a parsed SES event (called by the SQS consumer's handleMessage). | ||
| * | ||
| * NOTE on recipient arrays: | ||
| * `bouncedRecipients` and `complainedRecipients` are arrays per the SES | ||
| * spec, but in practice each event we receive will only contain ONE | ||
| * recipient. This is because our SES sender (sendViaSes) calls | ||
| * SendEmailCommand with a single `ToAddresses` per call. The loops below | ||
| * still iterate to remain faithful to the spec and to be safe if Phase 2 | ||
| * ever introduces multi-recipient sends via SendBulkEmailCommand. | ||
| */ | ||
| export async function processSesEvent(data: SesEventInput): Promise<void> { | ||
| const { sesEvent, sqsMessageId } = data | ||
| const { mail } = sesEvent | ||
|
|
||
| // sesEvent is a discriminated union on eventType (Bounce | Complaint), so the | ||
| // payload for each branch is guaranteed present — no defensive null checks | ||
| // needed, and the union is exhaustive. | ||
| if (sesEvent.eventType === SesEventType.Bounce) { | ||
| const { bounceType, bounceSubType, bouncedRecipients } = sesEvent.bounce | ||
|
|
||
| if (bounceType === 'Permanent') { | ||
| // TODO: add micro-optimisation for upsertSuppression to blacklist multiple recipient emails in phase 2 | ||
| for (const recipient of bouncedRecipients) { | ||
| await EmailSuppressionEntry.upsertSuppression({ | ||
| email: recipient.emailAddress, | ||
| reason: 'BOUNCE', | ||
| reasonDetail: bounceSubType, | ||
| sesMessageId: mail.messageId, | ||
| }) | ||
|
|
||
| logger.info('Email suppressed due to permanent bounce', { | ||
| event: 'ses-email-suppressed', | ||
| email: recipient.emailAddress, | ||
| bounceType, | ||
| bounceSubType, | ||
| sesMessageId: mail.messageId, | ||
| sqsMessageId, | ||
| }) | ||
| } | ||
| } else { | ||
| // Transient / Undetermined — log only, do not suppress | ||
| logger.info('Transient bounce received — no suppression', { | ||
| event: 'ses-transient-bounce', | ||
|
m0nggh marked this conversation as resolved.
|
||
| bounceType, | ||
| bounceSubType, | ||
| recipients: bouncedRecipients.map((r) => r.emailAddress), | ||
| sesMessageId: mail.messageId, | ||
| sqsMessageId, | ||
| }) | ||
| } | ||
| return | ||
| } | ||
|
|
||
| if (sesEvent.eventType === SesEventType.Complaint) { | ||
| const { complainedRecipients, complaintFeedbackType } = sesEvent.complaint | ||
|
|
||
| if (complaintFeedbackType === 'not-spam') { | ||
| // Auto-whitelist: recipient marked the email as not-spam | ||
| const emails = complainedRecipients.map((r) => r.emailAddress) | ||
| const whitelisted = await EmailSuppressionEntry.whitelistEmails(emails) | ||
|
Comment on lines
+69
to
+70
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. this is interesting. i guess a user who can emit a not-spam complain has a valid email |
||
|
|
||
| logger.info('Auto-whitelisted emails due to not-spam complaint', { | ||
| event: 'ses-auto-whitelist', | ||
| whitelisted, | ||
| sesMessageId: mail.messageId, | ||
| sqsMessageId, | ||
| }) | ||
| } else { | ||
| // abuse, fraud, virus, other, null — suppress | ||
| for (const recipient of complainedRecipients) { | ||
| await EmailSuppressionEntry.upsertSuppression({ | ||
| email: recipient.emailAddress, | ||
| reason: 'COMPLAINT', | ||
| reasonDetail: complaintFeedbackType ?? 'other', | ||
| sesMessageId: mail.messageId, | ||
| }) | ||
|
|
||
| logger.info('Email suppressed due to complaint', { | ||
| event: 'ses-email-suppressed', | ||
| email: recipient.emailAddress, | ||
| complaintFeedbackType, | ||
| sesMessageId: mail.messageId, | ||
| sqsMessageId, | ||
| }) | ||
| } | ||
| } | ||
| return | ||
| } | ||
| } | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.