feat: ML-based merchant normalization for Phase 2 auto-categorization

[moltboie]

Follow-up to PR #133 (auto-categorization Phase 1).

Background

Phase 1 uses hardcoded patterns to normalize transaction names (Square, PayPal, Amazon, etc.) based on documented Plaid transaction formats. This works well for known processor prefixes and noise patterns.

The hardcoded approach has limits: novel merchant formats, regional processors, and evolving Plaid naming conventions may not be covered.

Proposed approach for Phase 2

Investigate and implement a lightweight ML-based or rule-learning approach to merchant normalization:

1. Option A: Bayesian-style frequency analysis — track which normalization transformations correlate with high-confidence category assignments across users
2. Option B: Pre-trained tokenizer — use an open-source rule-based tokenizer trained on financial transaction data (e.g. spaCy NER with financial corpus)
3. Option C: Pattern extraction from data — analyze real transaction names in the database to discover new noise patterns automatically, periodically updating the pattern list

Acceptance criteria

• Normalization accuracy improves on edge cases not covered by Phase 1 patterns
• No regression on existing covered patterns (test suite must still pass)
• Approach is explainable / auditable (no black-box ML)

Dependencies

• Depends on Phase 1 landing (PR feat: auto-categorization Phase 1 - data model & merchant normalization #133)
• Requires sufficient transaction data to train/test

References

• Phase 1 design: PR feat: auto-categorization Phase 1 - data model & merchant normalization #133
• Hardcoded patterns reference: Plaid transaction naming conventions

[moltboie]

Status Update

The ML-based normalization described here has been implemented directly in PR #133 (auto-categorization Phase 1).

Specifically, hoiekim requested dynamic ML detection during the PR review (instead of hardcoded patterns), and I implemented a TF-IDF token frequency model in commit dc236f5:

1. buildTokenModel(transactionNames) — learns token frequencies from all transaction names at startup
2. Noise detection — tokens appearing in >70% of transaction names are flagged as universal noise and stripped
3. Purely numeric tokens are stripped (order IDs, store numbers)
4. No hardcoded patterns — the model learns from the user's own transaction history

This covers Option C from the original proposal (pattern extraction from data). Once PR #133 merges, this issue can be closed as resolved.

[moltboie]

Implemented in PR #133 (merged 2026-03-21). The TF-IDF token frequency model (Option C from the proposal) was implemented as part of auto-cat Phase 1. Closing as resolved.