Fix Deterministic Email

[SagiEv]

Fix Deterministic Email → Application Matching and Status Detection

Problem

The current email processing pipeline can incorrectly identify the company, application, role, or status associated with an incoming email.

These bugs are currently happening without any AI analysis.

This issue is focused exclusively on fixing the existing deterministic email-processing pipeline.

Incorrect matching or event detection can cause the wrong application to be updated and can corrupt application history.

The deterministic pipeline must reliably answer these questions in order:

1. Which company does this email belong to?
2. Which specific application does this email belong to?
3. What does this email represent?
4. Is the detected event valid?
5. Is the resulting status transition allowed?

Only after all of these checks pass should an application status be modified.

---

Known Bugs

1. ScaleOps incorrectly identified as POSIT

Actual email

From:

no-reply@eu.greenhouse-mail.io

Subject:

ScaleOps Application Update for Sagi Evroni

Incorrect result

The system produced:

Application Rejected: POSIT

The email was actually related to ScaleOps, but the deterministic matching logic associated it with the POSIT application.

This caused the POSIT application status to be changed incorrectly.

Expected behavior

The email should be associated with the ScaleOps application.

The POSIT application must remain unchanged.

The presence of multiple applications in the system must not cause an email containing an explicit company name to be associated with an unrelated company.

---

2. Microsoft application acknowledgement incorrectly identified as Interview

Actual email

From:

Microsoft Careers <donotreply@email.careers.microsoft.com>

Subject:

Thank you for your application!

Incorrect result

The system classified the email as:

Interview Scheduled: Microsoft

and changed the application status to:

Interview

Expected behavior

This is a generic application acknowledgement.

It should be classified as an application-received/application-confirmation type event.

It must not be classified as an interview.

Generic acknowledgement language such as:

• Thank you for your application
• We received your application
• Your application has been received
• Application submitted
• We appreciate your interest
• Thank you for applying

must not trigger an Interview status.

An Interview event requires stronger evidence.

---

3. Microsoft Software Engineer incorrectly matched to Software Engineering Intern

The system associated an email/application with:

Microsoft — Software Engineer — 200019052

when the actual application was:

Microsoft — Software Engineering Intern

The system must not merge or confuse different applications simply because their company and job titles are similar.

These are separate applications:

• Microsoft — Software Engineer
• Microsoft — Software Engineering Intern

Company matching alone is not sufficient when multiple applications exist for the same company.

Important job-title differences must be preserved, including:

• Intern
• Junior
• Senior
• Staff
• Principal
• Lead
• Manager
• Director
• Contract
• Part-time
• Full-time

Job-title normalization may be used for similarity scoring, but it must not remove material differences.

---

4. GitHub/Vercel notification incorrectly identified as Follow-up

Actual email

From:

vercel[bot] <notifications@github.com>

Subject:

Re: [SagiEv/JobPilot] Status Update Flow Improvements (PR #69)

Incorrect result

The email was identified as:

Follow-up Received

The application status was then changed to:

follow_up

and subsequently moved to:

Applied

Problem

This is a GitHub repository / pull-request notification.

It is not a recruiting email and should not affect any job application.

The deterministic event detection appears to be relying too heavily on generic words such as:

• status
• update
• reply
• follow-up

These words are not sufficient evidence that an email is related to a recruiting process.

Expected behavior

The email should be identified as unrelated to recruiting.

For example:

UNRELATED

or another equivalent non-recruiting event.

No application status should be changed.

---

Root Problem

The current deterministic processing logic does not sufficiently separate:

1. Email source/context
2. Company matching
3. Application matching
4. Event detection
5. Status transition

A mistake early in the pipeline can propagate all the way to an incorrect application status update.

For example:

Email → wrong company → wrong application → rejection → status mutation

or:

Email → generic "status update" keyword → follow-up → application status mutation

The system must fail safely instead.

---

Goal

Build a reliable deterministic email-processing pipeline that:

• Correctly identifies the company.
• Correctly identifies the specific application.
• Keeps similar applications separate.
• Correctly classifies recruiting events.
• Detects unrelated emails.
• Prevents generic acknowledgement emails from becoming interviews.
• Prevents unrelated emails from becoming follow-ups.
• Prevents ambiguous matches from changing application status.
• Validates status transitions before mutation.
• Preserves application history.
• Provides enough information to debug why an email was matched and processed.
• Works entirely without AI.

---

Proposed Deterministic Processing Pipeline

The pipeline should follow this general flow:

Incoming Email
↓
Extract Email Metadata
↓
Identify Email Source / Context
↓
Identify Company
↓
Find Candidate Applications
↓
Match Specific Application
↓
Determine Event
↓
Validate Event
↓
Validate Status Transition
↓
Update Application

Each stage should produce an explicit result that can be inspected and tested.

---

1. Extract Email Metadata

Extract and preserve deterministic information from the incoming email.

Possible fields include:

• Message ID
• Thread ID
• Sender name
• Sender email
• Sender domain
• Reply-To
• Subject
• Email body
• Email timestamp
• URLs contained in the email
• ATS information
• Requisition IDs
• Application IDs
• Job IDs
• Other explicit identifiers

The original email information must remain available for debugging and auditing.

The system should never discard the original subject, sender, or body after processing.

---

2. Identify Email Source / Context

Before interpreting an email as a recruiting event, determine whether the email is likely related to recruiting.

Some senders/domains clearly represent non-recruiting activity.

Examples include:

• GitHub notifications
• Vercel bots
• CI/CD systems
• Pull request notifications
• Repository notifications
• Development tooling
• Marketing emails
• Newsletters
• Automated system notifications

For example:

From:

vercel[bot] <notifications@github.com>

Subject:

Re: [SagiEv/JobPilot] Status Update Flow Improvements (PR #69)

should strongly indicate a GitHub/development context.

The word status in the subject must not be enough to classify this as a recruiting status update.

---

3. Company Matching

Company matching must use deterministic evidence.

Suggested priority:

1. Explicit company name in the subject
2. Explicit company name in the body
3. Job URL
4. Application URL
5. Requisition ID
6. Application ID
7. ATS metadata
8. Known sender/domain mapping
9. Existing email/application relationship
10. Other application context

Strong explicit evidence must not be overridden by weaker evidence.

Example

Subject:

ScaleOps Application Update for Sagi Evroni

The deterministic matcher should identify:

company = ScaleOps

It must not select POSIT simply because a POSIT application exists in the database.

Explicit company information should have significantly higher priority than a generic application candidate.

---

4. Application Matching

After identifying the company, the system must determine which specific application the email belongs to.

Company matching alone is never sufficient when multiple applications exist for the same company.

The matcher should consider available signals such as:

• Company
• Exact job title
• Normalized job title
• Requisition ID
• Application ID
• ATS job ID
• Job URL
• Candidate/application identifier
• Application date
• Email references
• Existing application metadata
• Thread context
• Sender/domain
• Explicit role information

The system should combine multiple signals instead of relying on a single broad match.

---

5. Exact Identifiers Must Have Highest Priority

When the email contains an exact identifier, it should be used as the strongest application-matching signal.

Examples:

• Requisition ID
• Application ID
• ATS job ID
• Job URL
• Application URL

For example, if an email contains:

Requisition ID: 200019052

the matcher should prefer the application with that exact requisition ID.

An approximate title match must never override an exact application identifier.

---

6. Job Title Matching

Job-title normalization can be useful for finding candidates, but normalization must not erase important differences.

For example:

Software Engineer

and:

Software Engineering Intern

must remain distinguishable.

Similarly:

Senior Software Engineer

must not automatically match:

Software Engineer

when both applications exist.

Important qualifiers must be preserved:

• Intern
• Internship
• Junior
• Senior
• Staff
• Principal
• Lead
• Manager
• Director
• Contract
• Part-time
• Full-time

The matching algorithm may calculate title similarity, but similarity must not automatically mean identity.

---

7. Multiple Applications at the Same Company

The system must explicitly support multiple applications at the same company.

Example:

• Microsoft — Software Engineer
• Microsoft — Software Engineering Intern
• Microsoft — Senior Software Engineer

If an email only identifies:

Microsoft

then the system does not have enough information to safely select one of these applications.

The result should therefore be considered ambiguous.

It must not arbitrarily select the first or highest-ranked application.

---

8. Ambiguous Application Matching

Introduce an explicit matching result.

Possible values:

• HIGH_CONFIDENCE
• MEDIUM_CONFIDENCE
• LOW_CONFIDENCE
• AMBIGUOUS
• NO_MATCH

Only a high-confidence application match should be allowed to automatically modify application state.

For:

• Medium-confidence matches
• Low-confidence matches
• Ambiguous matches
• No-match cases

the system must not automatically update an application's status.

Instead:

• Preserve the email.
• Preserve the extracted information.
• Record the matching result.
• Record why the match was uncertain.
• Make the email available for later review or processing.

Failing safely is preferable to updating the wrong application.

---

9. Deterministic Event Detection

Once the application has been identified, determine what the email represents.

Possible events include:

• APPLICATION_SUBMITTED
• APPLICATION_RECEIVED
• APPLICATION_CONFIRMATION
• RECRUITER_CONTACT
• FOLLOW_UP
• HR_SCREEN
• INTERVIEW_INVITATION
• INTERVIEW_SCHEDULED
• INTERVIEW_RESCHEDULED
• INTERVIEW_COMPLETED
• REJECTION
• OFFER
• WITHDRAWAL
• UNRELATED
• UNKNOWN

Event detection should be based on explicit deterministic rules.

The rules should be:

• Explicit
• Testable
• Explainable
• Easy to debug
• Independent from status mutation

---

10. Generic Application Acknowledgements Must Not Become Interviews

Generic application acknowledgements must be explicitly handled.

Examples:

Thank you for your application

We received your application

Your application has been received

Application submitted

We appreciate your interest

Thank you for applying

These should map to an acknowledgement/received event.

They must not map to:

INTERVIEW

unless additional explicit interview evidence exists.

Interview Evidence

An Interview event should require stronger evidence such as:

• Explicit interview invitation
• Explicit scheduling language
• Interview date
• Interview time
• Interview scheduling link
• Recruiter asking to schedule an interview
• Clear interview-stage language
• Meeting invitation related to an interview

For example:

Subject:

Thank you for your application!

must not result in:

status = INTERVIEW

---

11. Follow-Up Detection

Follow-up detection must also require recruiting context.

Generic words such as:

• status
• update
• reply
• follow-up
• checking in

are not sufficient on their own.

For example:

From:

vercel[bot] <notifications@github.com>

Subject:

Re: [SagiEv/JobPilot] Status Update Flow Improvements (PR #69)

must not become:

FOLLOW_UP

The system should first determine that the email is a GitHub/development notification and therefore unrelated to recruiting.

---

12. Unrelated Email Detection

The deterministic pipeline should explicitly support an UNRELATED event.

Examples include:

• GitHub notifications
• Vercel notifications
• Pull request notifications
• Repository notifications
• CI/CD notifications
• Development tooling
• Marketing emails
• Newsletters
• Automated system notifications

An unrelated email must not create an application event and must not modify application status.

---

13. Validate Event Before Status Mutation

Event detection and status mutation must be separate operations.

The flow should be:

Email
↓
Matched Application
↓
Detected Event
↓
Event Validation
↓
Status Transition Validation
↓
Status Update

For example:

APPLICATION_RECEIVED → INTERVIEW

must be rejected unless the email contains sufficient explicit interview evidence.

Likewise:

UNRELATED → FOLLOW_UP → APPLIED

must never happen.

---

14. Status Transition Rules

The system should define explicit allowed status transitions.

For every detected event, determine whether that event is allowed to change the current application status.

Example:

Application acknowledgement:

APPLICATION_RECEIVED

should not automatically transition to:

INTERVIEW

unless interview evidence exists.

An unrelated email:

UNRELATED

should never transition an application to:

• FOLLOW_UP
• APPLIED
• INTERVIEW
• REJECTED
• OFFER
• Any other recruiting status

Invalid transitions should be rejected before application mutation.

---

15. Status Changes Must Require a Valid Application Match

Before changing application status, the system must verify:

1. The email belongs to a recruiting context.
2. The company match is valid.
3. The application match is sufficiently confident.
4. The event classification is valid.
5. The event is supported by the email.
6. The status transition is allowed.

Only then should the application be modified.

---

16. Preserve Application History

Incorrect email processing must never silently overwrite application history.

Application events should be treated as historical records.

If an event is determined to be incorrect later, the system should preserve the original audit information and record the correction rather than silently rewriting history.

This is important for debugging and understanding how an incorrect status was created.

---

17. Idempotent Email Processing

Processing the same email more than once must not create repeated or unintended status changes.

The system should detect:

• Previously processed message IDs
• Duplicate message IDs
• Duplicate thread events
• Reprocessed emails
• Previously applied status transitions

The same email should produce the same deterministic result when processed repeatedly.

---

18. Audit Information

Every automated application status change should record enough information to explain exactly why it happened.

Recommended fields:

• Email ID
• Thread ID
• Sender
• Sender domain
• Subject
• Matched application ID
• Matched company
• Matched role
• Detected event
• Previous status
• New status
• Matching confidence
• Matching evidence
• Event detection rule
• Status transition rule
• Reason for transition

Example:

{
  "emailId": "...",
  "applicationId": "...",
  "company": "ScaleOps",
  "event": "REJECTION",
  "previousStatus": "APPLIED",
  "newStatus": "REJECTED",
  "matchingConfidence": "HIGH",
  "matchingEvidence": [
    "Company name explicitly present in subject",
    "Known Greenhouse sender"
  ],
  "reason": "Explicit ScaleOps rejection email matched to ScaleOps application"
}
# Regression Tests

## Test 1 — ScaleOps vs POSIT

### Given

Email subject:

`ScaleOps Application Update for Sagi Evroni`

Existing applications:

- ScaleOps
- POSIT

### Expected

The deterministic matcher returns:

`matched_company = ScaleOps`

The ScaleOps application may be considered for processing.

The POSIT application must not be modified.

---

## Test 2 — Microsoft Application Acknowledgement

### Given

From:

`donotreply@email.careers.microsoft.com`

Subject:

`Thank you for your application!`

### Expected

The event must not be:

`INTERVIEW`

The application must not automatically transition to:

`INTERVIEW`

The email should instead be classified as an application acknowledgement/received event.

---

## Test 3 — Microsoft Engineer vs Intern

### Given

Two applications exist:

- Microsoft — Software Engineer
- Microsoft — Software Engineering Intern

### Expected

The matcher must keep the applications separate.

The applications must never be merged solely because:

`company = Microsoft`

The correct application must be selected using available identifiers and role information.

If there is insufficient information, the result should be ambiguous rather than selecting the wrong application.

---

## Test 4 — GitHub/Vercel Notification

### Given

From:

`vercel[bot] <notifications@github.com>`

Subject:

`Re: [SagiEv/JobPilot] Status Update Flow Improvements (PR #69)`

### Expected

The email must be classified as:

`UNRELATED`

or another equivalent non-recruiting event.

It must not be classified as:

`FOLLOW_UP`

It must not modify an application's status.

It must not cause the application to become:

`follow_up`

It must not cause the application to become:

`Applied`

---

# Additional Regression Tests

Add tests covering:

- Multiple applications at the same company
- Multiple applications with similar job titles
- Multiple Greenhouse applications
- Missing job titles
- Missing company names
- Emails containing multiple company names
- Generic ATS emails
- Interview emails without explicit dates
- Interview emails with explicit dates
- Interview emails with scheduling links
- Rejection emails
- Offer emails
- Follow-up emails
- Ambiguous emails
- GitHub notifications
- Vercel notifications
- CI/CD notifications
- Emails containing the word "status" but unrelated to recruiting
- Emails containing the word "update" but unrelated to recruiting
- Emails containing the word "reply" but unrelated to recruiting
- Duplicate email processing
- Previously processed emails
- Multiple possible application matches
- Missing requisition IDs
- Conflicting company signals
- Conflicting role signals
- Invalid status transitions
- Historical application events

---

# Acceptance Criteria

- [ ] ScaleOps emails cannot incorrectly modify POSIT applications.
- [ ] Generic Microsoft application acknowledgements cannot create Interview events.
- [ ] Microsoft Software Engineer and Software Engineering Intern applications remain separate.
- [ ] GitHub/Vercel notifications cannot create Follow-up events.
- [ ] Unrelated emails cannot modify application status.
- [ ] Application matching uses more than company name.
- [ ] Exact identifiers have higher priority than fuzzy title matching.
- [ ] Important job-title qualifiers are preserved.
- [ ] Similar roles cannot automatically be treated as the same application.
- [ ] Multiple applications at the same company are handled safely.
- [ ] Ambiguous matches do not modify application status.
- [ ] Generic acknowledgement emails cannot create Interview events.
- [ ] Generic "status/update/reply" language cannot create Follow-up events without recruiting context.
- [ ] Event detection is separated from status mutation.
- [ ] Status transitions are validated before mutation.
- [ ] Unrelated emails are explicitly supported.
- [ ] Processing works entirely without AI.
- [ ] Duplicate email processing is idempotent.
- [ ] Every automated status change contains an auditable reason.
- [ ] Existing valid interview processing continues to work.
- [ ] Existing valid rejection processing continues to work.
- [ ] Existing application history is not silently overwritten.
- [ ] All four known bugs have regression tests.
- [ ] Backend tests pass.
- [ ] Integration tests pass.
- [ ] Critical email → application → event → status E2E flow passes.

---

# Definition of Done

- [ ] Deterministic email metadata extraction reviewed
- [ ] Email source/context detection implemented
- [ ] Deterministic company matching fixed
- [ ] Deterministic application matching fixed
- [ ] Exact identifier matching implemented/prioritized
- [ ] Job-title matching improved
- [ ] Important job-title qualifiers preserved
- [ ] Multiple-application matching handled
- [ ] Ambiguous matching handled safely
- [ ] Unrelated email detection implemented
- [ ] Interview detection rules fixed
- [ ] Follow-up detection rules fixed
- [ ] Status-transition validation implemented
- [ ] Application history protection implemented
- [ ] Idempotent email processing verified
- [ ] Audit information implemented
- [ ] ScaleOps/POSIT regression test added
- [ ] Microsoft acknowledgement regression test added
- [ ] Microsoft Engineer/Intern regression test added
- [ ] GitHub/Vercel regression test added
- [ ] Additional edge-case tests added
- [ ] Backend tests pass
- [ ] Integration tests pass
- [ ] E2E tests pass