feat(vocab): k-gram TF-IDF/LSA differential (orders 3-7) — confirm stopword candidates (task #13) - #261
Merged
Merged
Conversation
There was a problem hiding this comment.
Pull request overview
Adds a new fixture-backed vocabulary analysis tool to confirm stopword candidates by measuring whether their domain-specific n-gram signal persists across k-gram orders 3–7, and wires it into the repo’s validation “teeth” so CI can gate on expected confirmation outcomes.
Changes:
- Introduces
tools/kgram_tfidf_differential.pyto compute per-candidate TF-IDF-by-order (with unigram-specificity discount) plus reported LSA top-component energy. - Adds a fixture-based CI validator
tools/validate_kgram_differential.pyand Makefile targetmake validate-kgram-differential(installs numpy). - Documents the method in
specs/kgram-tfidf-differential.mdand notes it inCHANGELOG.md.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tools/validate_kgram_differential.py | New “teeth” validator asserting expected confirmed/unconfirmed outcomes on the fixture corpus. |
| tools/kgram_tfidf_differential.py | Implements the k-gram TF-IDF/LSA differential calculation and confirmation verdicting across orders 3–7. |
| specs/kgram-tfidf-differential.md | Normative write-up for the method and expected validation behavior. |
| Makefile | Adds validate-kgram-differential target and includes it in validate. |
| fixtures/kgram-differential/domains/narrative.json | Fixture domain corpus emphasizing stylistic high-frequency function-word usage. |
| fixtures/kgram-differential/domains/formal.json | Fixture domain corpus emphasizing domain terms like set/class/state in repeated phrases. |
| fixtures/kgram-differential/candidates.json | Fixture candidate list for the validator/tool. |
| CHANGELOG.md | Records the addition of the k-gram differential tool + validation target. |
Comment on lines
+82
to
+87
| out = {} | ||
| for w in candidates: | ||
| rows = [idx[g] for g in vocab if w in g] | ||
| tfidf = float(M[rows].max() / col_max) if rows else 0.0 # strongest domain-specific phrase w heads | ||
| energy = float(loading[rows].sum() / total_energy) if rows else 0.0 # share of latent variation from w's n-grams | ||
| out[w] = {"tfidf": round(tfidf, 3), "lsaEnergy": round(energy, 3)} |
Comment on lines
+103
to
+106
| def differential(domains: dict[str, str], candidates: list[str]) -> dict: | ||
| domain_tokens = {d: raw_tokenize(t) for d, t in domains.items()} | ||
| by_order = {n: order_signal(domain_tokens, n, candidates) for n in ORDERS} | ||
| uspec = {w: _unigram_specificity(domain_tokens, w) for w in candidates} |
Comment on lines
+17
to
+19
| A candidate word's signal at order n = the strongest domain-specific TF-IDF among the n-grams that | ||
| contain it (and its share of the LSA top-component energy). The DIFFERENTIAL across 3..7 is the | ||
| discriminator: |
mdheller
added a commit
that referenced
this pull request
Aug 3, 2026
…<2-domain guard - confirmation now uses BOTH signals of the differential (docstring promised it): TF-IDF must persist across orders AND the word must load on the dominant cross-domain LSA axis (lsaParticipates). Matches 'tf-idf lsa differential'; 'and' now fails both signals. - precompute candidate->row indices in the single pass over vocab instead of rescanning the whole vocab per candidate (O(vocab) once, not O(vocab*candidates) per order). - differential() raises on <2 domains (TF-IDF/LSA are cross-domain measures — fail fast, don't emit a meaningless result). Teeth: single-domain rejected. 6 teeth total.
…opword candidates
The confirmation stage for the stopword deviation analysis. tools/kgram_tfidf_differential.py
measures a candidate word's domain-specificity across n-gram orders 3..7: per order it builds the
n-gram x domain TF-IDF matrix (n-grams over raw tokens, so phrase-embedded stopwords are captured),
takes the LSA (truncated SVD) top component as the cross-domain axis, and scores the candidate by
its strongest domain-specific n-gram DISCOUNTED by intrinsic unigram specificity (so 'the' can't
borrow 'the state machine' specificity). The differential across orders is the discriminator: a
true term PERSISTS (confirmed-term); a concentrated-but-diffuse word ('and') or a borrowed-signal
word ('the') stays unconfirmed.
validate-kgram-differential teeth: set/class/state confirmed across 3..7; 'and' unconfirmed;
'the' stripped by the unigram discount; orders are exactly [3,4,5,6,7]. Uses numpy (installed in
the make recipe). Closes the two-stage stopword design.
…<2-domain guard - confirmation now uses BOTH signals of the differential (docstring promised it): TF-IDF must persist across orders AND the word must load on the dominant cross-domain LSA axis (lsaParticipates). Matches 'tf-idf lsa differential'; 'and' now fails both signals. - precompute candidate->row indices in the single pass over vocab instead of rescanning the whole vocab per candidate (O(vocab) once, not O(vocab*candidates) per order). - differential() raises on <2 domains (TF-IDF/LSA are cross-domain measures — fail fast, don't emit a meaningless result). Teeth: single-domain rejected. 6 teeth total.
mdheller
force-pushed
the
feat/kgram-tfidf-differential
branch
from
August 3, 2026 00:05
de3229c to
d0ebb13
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Confirm stopword candidates by compositional scale
The confirmation stage for the stopword deviation analysis (#260). That tool flags a dropped word as a term-candidate on the bigram floor (concentrated + repeated bigram collocations). This measures the candidate's domain-specificity across n-gram orders 3..7 and takes the differential.
Per order
n:The candidate's per-order signal = strongest domain-specific TF-IDF among n-grams containing it, discounted by its intrinsic unigram specificity — otherwise
theborrows "the state machine"'s specificity. The discount strips that.The differential across 3..7 discriminates:
ngrows) → confirmed-termTeeth (
make validate-kgram-differential)set/class/stateconfirmed across all orders (signal persists, not a single spike)and(concentrated but diffuse n-grams) → unconfirmedthe(borrows phrase specificity) → stripped by the unigram discount, unconfirmed[3,4,5,6,7]Closes the two-stage design: stopword deviation (#260, candidates) → k-gram differential (confirmation). Uses numpy (installed in the make recipe, like
jsonschema).