feat(nlp): zero-dep NLP compression phase, +11pp token savings

[zamalali]

Phase 2.5: Zero-dependency NLP compression

Adds a new NLP compression phase to the pipeline that runs between word abbreviation (Phase 2) and pattern detection (Phase 3). Pure JavaScript, zero external dependencies.

What it does

• Clause deduplication: Detects semantically similar clauses (Jaccard similarity on stemmed tokens) and removes redundant ones
• Redundant modifier stripping: Removes filler modifiers while protecting quoted strings, code blocks, and technical terms
• Importance-weighted sentence compression: TF-IDF + NPMI collocations to identify and compress low-information sentences
• Porter stemmer + stopword list: Purpose-built for LLM prompt text (preserves intent words like "must", "never", "always")

Benchmark results (same 5 official prompts, gpt-tokenizer cl100k_base)

Prompt	Original	v2.0 (no NLP)	v2.1 (NLP)	v2.0 %	v2.1 %	Lift
Dev assistant (verbose)	408	317	269	22.3%	34.1%	+11.8pp
Code review prompt	210	165	137	21.4%	34.8%	+13.4pp
Medical notes	151	110	97	27.2%	35.8%	+8.6pp
Business requirements	143	103	89	28.0%	37.8%	+9.8pp
Minimal filler	77	77	70	0.0%	9.1%	+9.1pp
Total	989	772	662	21.9%	33.1%	+11.2pp

Safety guarantees (83 stress-test invariants)

• NLP output is never longer than input
• NLP-on is never worse than NLP-off
• Quoted strings, SQL, code blocks are never modified
• Intent words (must, never, always, required) are always preserved
• Output is deterministic
• Higher aggressiveness produces same or shorter output
• Stemmer is idempotent

Files changed (8)

• sdk/src/nlp.js - NLP algorithms (Porter stemmer, NPMI, TF-IDF, clause dedup, modifier stripping)
• sdk/src/engine.js - Phase 2.5 integration (12 lines added)
• sdk/src/index.js - Re-export NLP functions
• app/lib/compression/nlp.js - App-layer re-export (matches existing pattern)
• tests/nlp.test.js - 59 tests covering every exported function
• tests/integration.test.js - Updated threshold test for NLP-improved compression
• README.md - Updated benchmarks and added Phase 2.5 to How it works
• sdk/src/nlp.js line-length fix for codebase consistency

Backwards compatible

NLP is enabled by default (nlp: true) but can be disabled per-call:

compress(text, { nlp: false })

324 tests pass. All existing behavior preserved.

[vercel]

@zamalali is attempting to deploy a commit to the chatde's projects Team on Vercel.

A member of the Team first needs to authorize it.