Add Optional AI Email Analysis to the Email Processing Pipeline

# Add Optional AI Email Analysis to the Email Processing Pipeline

## Problem

The deterministic email-processing pipeline is responsible for identifying the company, application, event, and status transition.

However, email content often contains useful information that is difficult to extract reliably using deterministic rules alone.

Examples include:

- Interview details
- Recruiter information
- Application stage
- Job title
- Requisition ID
- Application ID
- Interview date
- Interview time
- Interview timezone
- Interview links
- Scheduling links
- Requested actions
- Natural-language rejection messages
- Natural-language interview invitations
- Additional application context

We want to add an **optional AI email analysis layer** that can extract this information and provide more precise structured data.

AI should act as an **enrichment layer** on top of the deterministic pipeline.

It should not replace the deterministic processing logic and must never directly mutate application state.

---

# Goal

Add optional AI-powered email analysis that can extract structured information from an email and provide it to the existing processing pipeline.

The desired architecture is:

Incoming Email
↓
Deterministic Processing
↓
Company / Application Matching
↓
Optional AI Analysis
↓
Structured AI Data
↓
Merge With Deterministic Data
↓
Validate / Resolve Conflicts
↓
Determine Final Event
↓
Validate Status Transition
↓
Update Application

The deterministic pipeline remains the source of truth for application matching and status mutation.

---

# Non-Goal

This issue is **not** about replacing deterministic processing with AI.

AI must not become:

Email
↓
AI
↓
Status Change

Instead, AI should work as:

Email
↓
Deterministic Processing
↓
Optional AI Enrichment
↓
Deterministic Validation
↓
Status Transition Validation
↓
Status Change

AI output must always be validated before it can affect application state.

---

# Why AI Is Useful

Deterministic rules work well when information is explicit.

For example:

Subject:

`Interview invitation - Software Engineer`

or:

`Requisition ID: 200019052`

These values can be extracted deterministically.

However, recruiters often use natural language.

For example:

`We really enjoyed speaking with you and would love to continue the conversation with our engineering team. Would you be available next Tuesday afternoon?`

A deterministic matcher may not reliably determine that this represents an interview-related event.

AI can extract structured information such as:

- Event type
- Job title
- Interview date
- Interview time
- Recruiter
- Requested action
- Application stage

The deterministic validation layer can then decide whether this information is safe to use.

---

# AI Responsibilities

AI may be used for:

- Extracting structured information
- Understanding natural-language email content
- Identifying company
- Identifying job title
- Identifying requisition ID
- Identifying application ID
- Identifying ATS/job identifiers
- Identifying recruiter/contact
- Identifying application stage
- Identifying event type
- Extracting interview date
- Extracting interview time
- Extracting timezone
- Extracting interview links
- Extracting scheduling links
- Extracting requested actions
- Identifying explicit status language
- Identifying whether an email is likely unrelated
- Providing confidence scores
- Providing supporting evidence

---

# Suggested AI Output

The AI should return structured data rather than instructions to mutate application state.

Example:

```json
{
  "company": {
    "value": "Microsoft",
    "confidence": 0.98
  },
  "jobTitle": {
    "value": "Software Engineering Intern",
    "confidence": 0.91
  },
  "requisitionId": {
    "value": "200019052",
    "confidence": 0.99
  },
  "applicationId": {
    "value": null,
    "confidence": 0
  },
  "event": {
    "value": "INTERVIEW_INVITATION",
    "confidence": 0.94
  },
  "interview": {
    "date": "2026-09-03",
    "time": "14:00",
    "timezone": "Asia/Jerusalem",
    "confidence": 0.96
  },
  "recruiter": {
    "name": "John Smith",
    "email": "john@example.com",
    "confidence": 0.88
  },
  "requestedAction": {
    "value": "Confirm availability",
    "confidence": 0.93
  },
  "evidence": [
    "Explicit interview invitation",
    "Interview date and time provided",
    "Recipient asked to confirm availability"
  ]
}
```
The AI output is data only.

It must not directly perform:

application.status = ...

AI should only return structured information that is consumed by the existing processing and validation pipeline.

---

# AI as a Fallback for Missing Information

AI is especially useful when deterministic extraction cannot identify enough information.

Example:

Subject:

`Next steps`

Body:

`We enjoyed speaking with you and would like to invite you to meet with the engineering team next Tuesday at 14:00.`

Deterministic extraction may produce:

`event = UNKNOWN`

AI may produce:

`event = INTERVIEW_INVITATION`

and:

`interviewTime = 14:00`

The deterministic validation layer should then determine whether this information is sufficient to create or update the corresponding application event.

---

# AI-Assisted Application Matching

AI may provide additional information that helps the deterministic matcher identify the correct application.

Example:

Existing applications:

- Microsoft — Software Engineer
- Microsoft — Software Engineering Intern

Email:

`We're contacting you regarding your application for our Software Engineering Internship...`

AI extracts:

`jobTitle = Software Engineering Intern`

The deterministic matcher can use this as an additional signal.

The final application selection must still be performed and validated by the application-matching logic.

AI should not simply declare that an email belongs to a specific application and mutate that application.

---

# Preserve Role Differences

AI must not normalize different roles into an overly generic title.

For example:

- Software Engineer
- Software Engineering Intern
- Senior Software Engineer
- Staff Software Engineer

must remain distinguishable.

The AI output should preserve important qualifiers such as:

- Intern
- Internship
- Junior
- Senior
- Staff
- Principal
- Lead
- Manager
- Director
- Contract
- Part-time
- Full-time

The goal is to extract the actual role mentioned in the email, not to collapse it into a generic category.

---

# AI Event Classification

AI can help classify natural-language events.

## Interview

Example:

`We'd like to move forward and schedule a conversation with the engineering team.`

Possible AI result:

`event = INTERVIEW_INVITATION`

`confidence = 0.91`

## Rejection

Example:

`After careful consideration, we've decided not to continue with your application at this time.`

Possible AI result:

`event = REJECTION`

`confidence = 0.96`

## Follow-up

Example:

`Just checking in regarding the application we discussed last week.`

Possible AI result:

`event = FOLLOW_UP`

`confidence = 0.93`

These results must still be validated before changing application state.

---

# Generic Application Acknowledgements

AI must not turn generic application acknowledgements into interview events.

Example:

Subject:

`Thank you for your application!`

Expected:

`event = APPLICATION_RECEIVED`

or:

`event = APPLICATION_CONFIRMATION`

Not:

`event = INTERVIEW`

There must be actual interview evidence.

The AI should understand the difference between:

- Application acknowledgement
- Recruiter outreach
- Interview invitation
- Interview scheduling
- Interview confirmation

---

# Unrelated Emails

AI can help identify emails that are unrelated to recruiting.

Example:

From:

`vercel[bot] <notifications@github.com>`

Subject:

`Re: [SagiEv/JobPilot] Status Update Flow Improvements (PR #69)`

Possible AI result:

`event = UNRELATED`

`category = GITHUB_NOTIFICATION`

`confidence = 0.99`

This information can help the deterministic pipeline avoid treating the email as a recruiting event.

The final application status must remain unchanged.

---

# AI Must Not Create False Follow-Ups

AI must not classify an email as a recruiting follow-up solely because it contains words such as:

- status
- update
- reply
- follow up
- checking in

For example:

From:

`vercel[bot] <notifications@github.com>`

Subject:

`Re: [SagiEv/JobPilot] Status Update Flow Improvements (PR #69)`

must not become:

`FOLLOW_UP`

and must not cause:

`Applied`

or any other application status change.

The AI should consider the complete email context rather than isolated keywords.

---

# AI Analysis Is Optional

The core email-processing system must continue to work when AI is unavailable.

AI processing should be optional.

The system must continue operating when:

- AI is disabled
- AI service is unavailable
- AI request fails
- AI request times out
- AI returns invalid data
- AI returns incomplete data
- AI returns low-confidence data

AI failure must never break the core email pipeline.

Expected behavior:

AI unavailable
↓
Skip AI analysis
↓
Continue deterministic processing

---

# Suggested Integration

The preferred architecture is:

Incoming Email
↓
Deterministic Fact Extraction
↓
Company / Application Matching
↓
Deterministic Event Detection
↓
Optional AI Analysis
↓
Merge / Resolve Information
↓
Deterministic Validation
↓
Status Transition Validation
↓
Application Update

The deterministic pipeline remains the source of truth for:

- Application identity
- Company identity when deterministic evidence exists
- Exact identifiers
- Status transitions
- Final application mutation

AI provides additional information that can improve extraction and classification.

---

# Conflict Resolution

If deterministic and AI results disagree, resolve them according to evidence strength.

## Strong deterministic evidence

Use the deterministic value.

Example:

Deterministic:

`company = ScaleOps`

AI:

`company = POSIT`

Final:

`company = ScaleOps`

AI must not override explicit deterministic evidence.

## Missing deterministic information

AI may provide a candidate value.

Example:

Deterministic:

`jobTitle = null`

AI:

`jobTitle = Software Engineering Intern`

Final:

`jobTitle = Software Engineering Intern`

`source = AI`

The source of each value should be identifiable.

---

# Field Source Tracking

For extracted fields, store:

- Value
- Source
- Confidence

Example:

`company = ScaleOps`

`source = subject`

`confidence = 1.0`

Example:

`jobTitle = Software Engineering Intern`

`source = AI`

`confidence = 0.91`

Possible sources include:

- subject
- body
- sender
- domain
- URL
- ATS
- application
- AI

This makes it possible to understand where each piece of information came from.

---

# AI Result Storage

AI analysis should be stored separately from canonical application state.

Suggested structure:

email
├── deterministicData
├── aiAnalysis
│   ├── extractedCompany
│   ├── extractedJobTitle
│   ├── requisitionId
│   ├── applicationId
│   ├── event
│   ├── interview
│   ├── recruiter
│   ├── requestedAction
│   ├── confidence
│   ├── evidence
│   ├── model
│   ├── promptVersion
│   └── analyzedAt
└── processingResult

This allows AI analysis to be:

- Re-run
- Compared
- Debugged
- Audited
- Improved
- Disabled without affecting deterministic processing

---

# AI Versioning

Store enough information to identify how an AI analysis was generated.

Recommended fields:

- Model
- Prompt version
- Analysis version
- Analyzed timestamp

Example:

`model = ...`

`promptVersion = ...`

`analysisVersion = ...`

`analyzedAt = ...`

This is important because AI output can change when the model or prompt changes.

---

# AI Output Validation

All AI output must be validated before being consumed by the application-processing pipeline.

Validate:

- Schema
- Enum values
- Dates
- Times
- Timezones
- Email addresses
- URLs
- Confidence values
- Requisition IDs
- Application IDs
- Job titles

Invalid AI output should be rejected or marked as failed.

The system must not trust arbitrary AI-generated values.

---

# Status Mutation Rules

AI must never directly mutate application state.

The final process should be:

AI Analysis
↓
Structured Information
↓
Deterministic Validation
↓
Status Transition Rules
↓
Application Mutation

For example:

AI:

`event = INTERVIEW_INVITATION`

`confidence = 0.92`

does not automatically mean:

`status = INTERVIEW`

The system must still validate:

- Correct company
- Correct application
- Supporting email evidence
- Event validity
- Allowed status transition
- Existing application state

---

# AI Failure Handling

## AI unavailable

Expected:

AI analysis skipped.

Deterministic processing continues.

## AI timeout

Expected:

AI analysis timed out.

Deterministic processing continues.

## Invalid AI response

Expected:

AI result rejected.

Deterministic data preserved.

## Low-confidence AI result

Expected:

AI result stored as low confidence.

Deterministic result remains authoritative.

AI failure must never result in an application status being changed incorrectly.

---

# Auditability

When AI contributes to processing, the audit record should contain:

- Whether AI was used
- AI model
- AI prompt/version
- AI-extracted values
- AI confidence
- Deterministic values
- Conflicts between deterministic and AI values
- Final resolved values
- Source of final values
- Final event classification
- Reason for status transition

Example:

`aiUsed = true`

`aiModel = ...`

`deterministicCompany = ScaleOps`

`aiCompany = ScaleOps`

`finalCompany = ScaleOps`

`aiEvent = REJECTION`

`aiConfidence = 0.94`

`finalEvent = REJECTION`

`reason = AI classification confirmed by explicit rejection language`

---

# Testing

## AI should improve extraction

Add tests for:

- Natural-language interview invitations
- Recruiter requests for availability
- Natural-language rejection messages
- Interview rescheduling
- Interview date extraction
- Interview time extraction
- Interview timezone extraction
- Interview link extraction
- Recruiter/contact extraction
- Requisition ID extraction
- Application ID extraction
- Job-title extraction
- Application-stage extraction
- Requested-action extraction
- Unrelated email classification

---

## AI must not override deterministic evidence

Test:

Deterministic company:

`ScaleOps`

AI company:

`POSIT`

Expected:

`final company = ScaleOps`

---

## AI must not create false interviews

Test:

Subject:

`Thank you for your application!`

Expected:

`event != INTERVIEW`

The application must not be moved to Interview.

---

## AI must not create false follow-ups

Test:

From:

`vercel[bot] <notifications@github.com>`

Subject:

`Re: [SagiEv/JobPilot] Status Update Flow Improvements (PR #69)`

Expected:

`event = UNRELATED`

`status = unchanged`

---

## AI failure

Given:

`AI unavailable`

Expected:

`deterministic pipeline continues`

---

## AI timeout

Given:

`AI request times out`

Expected:

`deterministic pipeline continues`

---

## Invalid AI response

Given:

`AI returns invalid structured data`

Expected:

`AI result rejected`

`deterministic result preserved`

---

## Low AI confidence

Given:

`AI event confidence = low`

Expected:

`AI result does not override deterministic result`

---

## Conflicting Data

Test conflicts between:

- Deterministic company vs AI company
- Deterministic job title vs AI job title
- Deterministic event vs AI event
- Deterministic application ID vs AI application ID
- Deterministic requisition ID vs AI requisition ID

Expected:

Strong deterministic evidence wins.

---

# Acceptance Criteria

- [ ] AI analysis is optional.
- [ ] Deterministic processing works without AI.
- [ ] AI never directly mutates application state.
- [ ] AI returns structured data.
- [ ] AI returns confidence information.
- [ ] Deterministic evidence takes priority over AI evidence.
- [ ] AI can enrich missing email information.
- [ ] AI can extract job titles.
- [ ] AI can extract company names.
- [ ] AI can extract requisition IDs.
- [ ] AI can extract application IDs.
- [ ] AI can extract interview dates.
- [ ] AI can extract interview times.
- [ ] AI can extract interview timezones.
- [ ] AI can extract interview links.
- [ ] AI can extract recruiter information.
- [ ] AI can classify natural-language events.
- [ ] AI can identify unrelated emails.
- [ ] Generic application acknowledgements cannot become interviews.
- [ ] GitHub/Vercel notifications cannot become recruiting follow-ups.
- [ ] AI failures do not break email processing.
- [ ] AI timeouts do not break email processing.
- [ ] Invalid AI responses are safely rejected.
- [ ] Low-confidence AI results cannot override strong deterministic evidence.
- [ ] AI results are stored separately from canonical application state.
- [ ] AI analysis is auditable.
- [ ] AI model/version is recorded.
- [ ] Field source is recorded where applicable.
- [ ] Deterministic-vs-AI conflict resolution is implemented.
- [ ] AI regression tests are added.
- [ ] Existing deterministic tests continue to pass.

---

# Definition of Done

- [ ] AI email analysis service implemented
- [ ] Structured AI response schema implemented
- [ ] AI confidence implemented
- [ ] Deterministic/AI field merging implemented
- [ ] Conflict resolution implemented
- [ ] AI output validation implemented
- [ ] Optional AI execution implemented
- [ ] AI failure handling implemented
- [ ] AI timeout handling implemented
- [ ] AI audit data implemented
- [ ] AI versioning implemented
- [ ] Field source tracking implemented
- [ ] Natural-language interview extraction tests added
- [ ] Rejection extraction tests added
- [ ] Follow-up extraction tests added
- [ ] Unrelated email tests added
- [ ] Deterministic-vs-AI conflict tests added
- [ ] AI failure tests added
- [ ] AI timeout tests added
- [ ] Invalid AI response tests added
- [ ] Low-confidence tests added
- [ ] Backend tests pass
- [ ] Integration tests pass
- [ ] E2E tests pass



Add Optional AI Email Analysis to the Email Processing Pipeline #83

Description

Add Optional AI Email Analysis to the Email Processing Pipeline

Problem

Goal

Non-Goal

Why AI Is Useful

AI Responsibilities

Suggested AI Output

AI as a Fallback for Missing Information

AI-Assisted Application Matching

Preserve Role Differences

AI Event Classification

Interview

Rejection

Follow-up

Generic Application Acknowledgements

Unrelated Emails

AI Must Not Create False Follow-Ups

AI Analysis Is Optional

Suggested Integration

Conflict Resolution

Strong deterministic evidence

Missing deterministic information

Field Source Tracking

AI Result Storage

AI Versioning

AI Output Validation

Status Mutation Rules

AI Failure Handling

AI unavailable

AI timeout

Invalid AI response

Low-confidence AI result

Auditability

Testing

AI should improve extraction

AI must not override deterministic evidence

AI must not create false interviews

AI must not create false follow-ups

AI failure

AI timeout

Invalid AI response

Low AI confidence

Conflicting Data

Acceptance Criteria

Definition of Done

Activity

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions