From 2c289950af27fc18d91d79e8c99f7b99e5cc6677 Mon Sep 17 00:00:00 2001 From: Rahul R Date: Fri, 7 Aug 2026 23:10:49 -0500 Subject: [PATCH] Add retrieval evaluation, fix the bugs it found, add TypeScript MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit TokenHack had ~20 hand-tuned scoring constants and no way to tell whether any of them helped. This adds the measurement and then acts on it. ## tests/ — the harness 72 questions across 6 OSS repos (spring-framework, netty, Signal-Android, DuckDuckGo iOS, ownCloud Android, a small JS app), each paired with the file that answers it. Built blind: the questions and answers were derived by reading the repos only, with the router and its index off-limits, so the set is not shaped around what this tool happens to be good at. 45 of 72 are deliberately vocabulary-mismatch questions. Public repos are pinned in tests/repos.json and tests/fetch_repos.py reproduces the whole thing from scratch. Every scoring constant now reads an optional TOKENHACK_ environment override so a change can be ablated instead of argued about. ## What the first run found hit@5 = 0.19. Four questions in five got nothing useful into the top five. 1. graph_prop was 94% of the winning score. On a netty query the winner scored 354.76, of which 333.01 was import-graph propagation against a lexical score of 22 — and the file that answered the question ranked 1,716th of 3,522. The filename gate meant to bound this is trivially cleared by a natural-language question, which contains enough words that any well-connected file matches two of them. Propagation is evidence that a *neighbour* matched, which is weaker than matching yourself: it is now always log-compressed and capped at the file's own query-derived score. 2. The prose channel was empty. ETA is the highest weight in the scoring function and only the Python adapter ever populated it — Java, Kotlin, Swift and JavaScript returned nothing, so the highest-weighted term multiplied zero across 0/3512 netty files, 0/9487 Spring files, 2/5534 Signal files. Doc-comment extraction (javadoc/KDoc/Swift doc/JSDoc) now ships in _base.py and all adapters use it; coverage is 80% on Spring, 60% on netty. With the channel actually populated, ETA 1.2 was too low — measured 0.181 at ETA=0, 0.208 at 1.2, 0.264 at 2.4, flat above that. Default raised to 2.4. 3. Tests outranked implementations. Tests restate a concept's vocabulary more densely than the code does, which is what BM25 rewards; netty's pipeline question returned two testsuite files and a microbenchmark, with DefaultChannelPipeline.java absent. Test/benchmark/fixture paths are now demoted unless the question is about tests. Paired results were also labelled backwards — the implementation was printed as "test for" its own test. 4. Structural matches ignored IDF. filename_match and path_affinity counted 1.0 per token regardless of the token, so `read` and `socket` scored the same as `idempotency` and `backoff`. Both are now IDF-weighted; worth ~1.4 points of hit@5 on its own. Query tokens below MIN_QUERY_TOKEN_IDF are dropped as corpus-wide filler. ## Result hit@5 0.19 -> 0.26, recall@5 0.16 -> 0.19, MRR 0.13 -> 0.17. On the vocabulary-mismatch slice, where lexical retrieval is expected to lose, 0.13 -> 0.22. Baseline committed so --check can catch regressions. These are in-sample numbers on six repos and most questions still miss. That is now a fact with a number attached rather than an unknown. ## Also - TypeScript/TSX adapter. The JS walk moved into walk_source() and typescript.py reuses it, adding interface/type-alias/enum declarations and dispatching .tsx to the JSX grammar. Removes the single biggest adoption blocker. - The router now warns when more than a quarter of a repo's source files had no language adapter. The indexer had always recorded this and nothing read it, so an unsupported-language repo got confident rankings over a fragment of itself. - Light plural folding on structural matches, so "bytes" reaches ByteBuf and "credentials" reaches Credential. - A workflow_dispatch/PR-triggered CI job that clones the pinned repos and fails on regression against the baseline. --- .claude/skills/tokenhack/README.md | 74 ++++- .claude/skills/tokenhack/adapters/__init__.py | 3 +- .claude/skills/tokenhack/adapters/_base.py | 91 ++++++ .../skills/tokenhack/adapters/javascript.py | 28 +- .claude/skills/tokenhack/adapters/jvm.py | 7 +- .claude/skills/tokenhack/adapters/swift.py | 7 +- .../skills/tokenhack/adapters/typescript.py | 62 ++++ .claude/skills/tokenhack/requirements.txt | 1 + .claude/skills/tokenhack/router.py | 278 +++++++++++++++--- .github/workflows/tokenhack-eval.yml | 67 +++++ .gitignore | 3 + README.md | 25 +- ROADMAP.md | 16 +- tests/README.md | 94 ++++++ tests/baseline.json | 69 +++++ tests/eval.py | 244 +++++++++++++++ tests/fetch_repos.py | 87 ++++++ tests/gold/DDG.json | 119 ++++++++ tests/gold/OwnCloud.json | 121 ++++++++ tests/gold/Signal-Android.json | 119 ++++++++ tests/gold/blinkit.json | 127 ++++++++ tests/gold/netty.json | 120 ++++++++ tests/gold/spring-framework.json | 121 ++++++++ tests/repos.json | 36 +++ 24 files changed, 1838 insertions(+), 81 deletions(-) create mode 100644 .claude/skills/tokenhack/adapters/typescript.py create mode 100644 .github/workflows/tokenhack-eval.yml create mode 100644 tests/README.md create mode 100644 tests/baseline.json create mode 100644 tests/eval.py create mode 100644 tests/fetch_repos.py create mode 100644 tests/gold/DDG.json create mode 100644 tests/gold/OwnCloud.json create mode 100644 tests/gold/Signal-Android.json create mode 100644 tests/gold/blinkit.json create mode 100644 tests/gold/netty.json create mode 100644 tests/gold/spring-framework.json create mode 100644 tests/repos.json diff --git a/.claude/skills/tokenhack/README.md b/.claude/skills/tokenhack/README.md index e33df82..80ab6da 100644 --- a/.claude/skills/tokenhack/README.md +++ b/.claude/skills/tokenhack/README.md @@ -13,12 +13,13 @@ Contributor-facing docs. For the user-facing pitch and install, see the ├── requirements.txt # indexer deps (NOT needed on dev machines) ├── README.md # this file ├── adapters/ -│ ├── _base.py # Symbol, ExtractResult, shared helpers +│ ├── _base.py # Symbol, ExtractResult, doc-comment extraction │ ├── __init__.py # adapter registry, get_adapter(filepath) │ ├── python.py # .py │ ├── jvm.py # .java, .kt, .kts (shared adapter) │ ├── swift.py # .swift -│ └── javascript.py # .js, .mjs, .cjs, .jsx (JSX included) +│ ├── javascript.py # .js, .mjs, .cjs, .jsx (JSX included) +│ └── typescript.py # .ts, .tsx, .mts, .cts (reuses the JS walk) └── index/ ├── symbols.json # built by CI; committed to the repo └── meta.json # file hashes for incremental rebuilds @@ -29,20 +30,40 @@ Contributor-facing docs. For the user-facing pitch and install, see the The router combines a BM25 baseline with structural and heuristic priors: ``` -tokens = camel_snake_split(query) − code_stopwords + fuzzy(close_matches) +tokens = camel_snake_split(query) − code_stopwords − corpus_filler + fuzzy(close_matches) score(file, query) = - BM25(tokens, symbols ∪ refs ∪ path_tokens) # lexical - + η · BM25(tokens, docstrings ∪ markdown_paragraphs) # prose channel - + α · filename_match(tokens, basename(file)) - + β · path_affinity(tokens, dirname(file)) - + γ · recency(mtime(file)) # exponential decay - + ε · symbol_popularity(file) # incoming imports + [ BM25(tokens, symbols ∪ refs ∪ path_tokens) # lexical + + η · BM25(tokens, doc_comments ∪ markdown_paragraphs) # prose channel + + α · Σ idf_weight(t) for t in filename_match(tokens, basename(file)) + + β · Σ idf_weight(t) for t in path_affinity(tokens, dirname(file)) + + γ · recency(mtime(file)) # exponential decay + + ε · symbol_popularity(file) # incoming imports + ζ · definition_bonus(tokens, defs_in_file) - + δ₁ · graph_propagation_1hop(file, top_seeds) - + δ₂ · graph_propagation_2hop(file, top_seeds) # decayed + + impl_pattern_boost(file) + ] · (test_penalty if file is a test and the query isn't about tests) + + min( graph_prop(file), own_lexical_score + floor ) # capped, see below ``` +**Corpus filler removal.** Query tokens whose IDF falls below +`MIN_QUERY_TOKEN_IDF` are dropped. A natural-language question carries words +that are technically in the vocabulary but carry no discrimination — `netty` in +netty has an IDF of 0.01. BM25 already discounted them; the structural signals +did not. + +**IDF-weighted structural matches.** `filename_match` and `path_affinity` used +to count 1.0 per matching token regardless of the token. A question mentioning +`read` and `socket` therefore scored the same filename hit as one mentioning +`idempotency` and `backoff`. Both are now weighted by token IDF, normalised +against `IDF_REFERENCE` so `ALPHA` / `BETA` keep their existing meaning. + +**Test/benchmark demotion.** Tests restate a concept's vocabulary far more +densely than the implementation does, which is exactly what BM25 rewards — on +netty, the top three results for a pipeline question were two testsuite files +and a microbenchmark, and the implementation was absent. Test-path files are +multiplied by `TEST_PATH_PENALTY` unless the question is explicitly about +tests, and surface instead in the paired-results section. + Hub suppression: files imported by more than `HUB_FRACTION` of the corpus do not propagate scores to their neighbours (utility hubs like `utils.py` would otherwise drag the whole repo in). @@ -68,6 +89,17 @@ Graph-propagation credit is *restricted* to the caller set in this mode, so unrelated files that happen to match common tokens like `get` cannot accumulate inflated graph_prop and bury the actual target. +**Capped graph propagation.** The filename gate below was meant to bound +`graph_prop`, but a natural-language question contains many words, so any +well-connected file with two of them in its basename cleared it and received +*raw* credit. Measured on netty, `SocketReadPendingTest.java` accumulated a +`graph_prop` of 333 against a lexical score of 22 — propagation was 94% of the +winning score and the file that answered the question ranked 1,716th. +Propagation is evidence that a *neighbour* matched, which is weaker than +matching yourself, so it is now always log-compressed and then capped at +`GRAPH_PROP_CAP_RATIO ×` the file's own query-derived score plus a small floor. +It can promote a plausible file; it can no longer invent one. + **Lexical-only graph propagation.** Recency and popularity are still summed into the final ranking but are not allowed to *seed* the import graph. In an actively-modified repo nearly every file has high recency, @@ -88,7 +120,7 @@ in 300 files at once) from collectively dominating the graph. | `ZETA` | 0.5 | Bonus per matched definition name | | `DELTA_1HOP` | 1.0 | 1-hop import-graph propagation | | `DELTA_2HOP` | 0.3 | 2-hop propagation (decayed) | -| `ETA` | 1.2 | Prose channel weight | +| `ETA` | 2.4 | Prose channel weight (doc comments + markdown) — the highest-value signal in the ranker; see the measurement note in `router.py` | | `BM25_K1` | 1.5 | BM25 saturation parameter | | `BM25_B` | 0.75 | BM25 length-normalization parameter | | `RECENCY_HALFLIFE_DAYS` | 30 | Files this old contribute half their recency score | @@ -111,7 +143,23 @@ in 300 files at once) from collectively dominating the graph. | `WIDE_SPAN_FRACTION` | 0.5 | A matched span covering more than this fraction of the file is a container (class / extension), not a target — dropped when a tighter span also matched | | `TOP_K` | 5 | How many ranked results to emit | -Open a PR with rationale + measurements if you change any of these. +| `TEST_PATH_PENALTY` | 0.35 | Multiplier on a test/benchmark/fixture file when the query isn't about tests | +| `GRAPH_PROP_CAP_RATIO` | 1.0 | graph_prop ceiling as a multiple of the file's own lexical score | +| `GRAPH_PROP_CAP_FLOOR` | 1.0 | Additive floor so a zero-lexical neighbour can still surface | +| `IDF_REFERENCE` | 3.0 | IDF normaliser for structural (filename / path) match weighting | +| `IDF_WEIGHT_CAP` | 2.0 | Max weight a single very-rare token can carry structurally | +| `MIN_QUERY_TOKEN_IDF` | 0.35 | Query tokens below this IDF are corpus-wide filler and get dropped | + +Every constant here is overridable at runtime as `TOKENHACK_`, so you can +ablate it against the gold set: + +```bash +TOKENHACK_ETA=0 python3 tests/eval.py +``` + +**Open a PR with rationale + measurements if you change any of these.** Since +`tests/` exists there is no excuse for "felt better" — run `python3 +tests/eval.py` before and after and put both numbers in the PR body. ## Adding a new language adapter diff --git a/.claude/skills/tokenhack/adapters/__init__.py b/.claude/skills/tokenhack/adapters/__init__.py index c266f40..3e9fe11 100644 --- a/.claude/skills/tokenhack/adapters/__init__.py +++ b/.claude/skills/tokenhack/adapters/__init__.py @@ -20,8 +20,9 @@ from . import jvm as _jvm from . import swift as _swift from . import javascript as _javascript +from . import typescript as _typescript -_MODULES = (_python, _jvm, _swift, _javascript) +_MODULES = (_python, _jvm, _swift, _javascript, _typescript) def _build_ext_map(): diff --git a/.claude/skills/tokenhack/adapters/_base.py b/.claude/skills/tokenhack/adapters/_base.py index b070913..340d38e 100644 --- a/.claude/skills/tokenhack/adapters/_base.py +++ b/.claude/skills/tokenhack/adapters/_base.py @@ -56,6 +56,97 @@ def declaration_line(source: bytes, name_node) -> str: return source[left:right].decode("utf-8", errors="replace").strip() +# ---------------------------------------------------------------------- +# Doc-comment extraction (the prose channel) +# ---------------------------------------------------------------------- +# +# The router scores a separate "prose" BM25 channel at the highest weight in the +# formula (ETA = 1.2), because docs are written in the words a human question +# uses while identifiers are written in the words a compiler needs. Until now +# only the Python adapter populated it — Java, Kotlin, Swift and JavaScript +# returned nothing — so on netty, Spring, Signal, DuckDuckGo and ownCloud the +# highest-weighted channel in the scoring function was empty for 100% of files. +# Javadoc/KDoc/Swift-doc is exactly the natural-language bridge those repos +# needed. + +DOC_PREFIXES = ("/**", "/*!", "///", "//!") + +_DOC_LINE_NOISE = ("/**", "/*!", "*/", "///", "//!") +_JAVADOC_TAG = None # compiled lazily; adapters import this module a lot + + +def _doc_regexes(): + global _JAVADOC_TAG + if _JAVADOC_TAG is None: + import re + _JAVADOC_TAG = ( + re.compile(r"\{@\w+\s+([^}]*)\}"), # {@link Foo#bar} -> Foo#bar + re.compile(r"^\s*@\w+"), # @param / @return line + re.compile(r"<[^>]+>"), #

, , + re.compile(r"\s+"), + ) + return _JAVADOC_TAG + + +def clean_doc_comment(text: str) -> str: + """Turn a raw /** ... */ or /// block into a plain prose paragraph. + + Drops comment syntax, inline javadoc markup ({@link X} keeps X), HTML tags, + and standalone block-tag lines (@param, @return, @throws), which are + structure rather than description and mostly repeat identifiers the code + channel already indexes. + """ + inline_tag, block_tag, html, ws = _doc_regexes() + out = [] + for raw in text.splitlines(): + line = raw.strip() + for noise in _DOC_LINE_NOISE: + if line.startswith(noise): + line = line[len(noise):] + line = line.strip() + if line.startswith("*"): + line = line[1:].strip() + if line.endswith("*/"): + line = line[:-2].strip() + if not line or block_tag.match(line): + continue + out.append(line) + joined = " ".join(out) + joined = inline_tag.sub(r"\1", joined) + joined = html.sub(" ", joined) + return ws.sub(" ", joined).strip() + + +def collect_doc_comments(root, source: bytes, max_paragraphs=8, max_chars=240): + """Return (paragraphs, summary) from the doc comments in a parsed tree. + + Any node whose type contains "comment" is considered; only blocks opening + with a doc marker are kept, so ordinary `//` implementation notes and + license headers don't flood the channel. Language-agnostic on purpose — + Java, Kotlin, Swift and JavaScript all spell doc comments the same way. + """ + paragraphs = [] + stack = [root] + while stack and len(paragraphs) < max_paragraphs: + node = stack.pop() + if "comment" in node.type: + text = get_text(node, source).lstrip() + if text.startswith(DOC_PREFIXES): + cleaned = clean_doc_comment(text) + # Skip license/copyright boilerplate, which every file repeats + # and which would otherwise dominate the corpus term stats. + low = cleaned[:120].lower() + if cleaned and len(cleaned) > 24 and not ( + "copyright" in low or "licensed under" in low + or "license, version" in low or "all rights reserved" in low + ): + paragraphs.append(cleaned[:max_chars]) + for child in reversed(node.children): + stack.append(child) + summary = first_line(paragraphs[0]) if paragraphs else "" + return paragraphs, summary[:max_chars] + + def strip_string_quotes(s: str) -> str: """Strip leading/trailing string quote markers (single, double, triple).""" s = s.strip() diff --git a/.claude/skills/tokenhack/adapters/javascript.py b/.claude/skills/tokenhack/adapters/javascript.py index 92028f5..6cfc7d7 100644 --- a/.claude/skills/tokenhack/adapters/javascript.py +++ b/.claude/skills/tokenhack/adapters/javascript.py @@ -1,9 +1,11 @@ """JavaScript adapter for TokenHack (JSX included). -Note: TypeScript is intentionally NOT covered in v1 — see CONTRIBUTING -section of the top-level README for the planned first-PR scope. +The AST walk lives in `walk_source` so the TypeScript adapter can reuse it — +the grammars share the great majority of their node types, and TS only needs to +add its own declaration forms (interface, type alias, enum) on top. """ -from ._base import Symbol, ExtractResult, get_text, declaration_line +from ._base import (Symbol, ExtractResult, get_text, declaration_line, + collect_doc_comments) LANGUAGE_NAME = "javascript" FILE_EXTENSIONS = [".js", ".mjs", ".cjs", ".jsx"] @@ -27,11 +29,9 @@ } -def extract(source: bytes, filepath: str) -> ExtractResult: - if not AVAILABLE: - return ExtractResult() - - tree = _PARSER.parse(source) +def walk_source(parser, source: bytes, def_types) -> ExtractResult: + """Shared JS/TS extraction: definitions, imports, call references, docs.""" + tree = parser.parse(source) root = tree.root_node symbols = [] @@ -43,7 +43,7 @@ def extract(source: bytes, filepath: str) -> ExtractResult: node = stack.pop() nt = node.type - if nt in _DEF_TYPES: + if nt in def_types: name_node = node.child_by_field_name("name") if name_node is not None: name = get_text(name_node, source) @@ -84,8 +84,18 @@ def extract(source: bytes, filepath: str) -> ExtractResult: for child in reversed(node.children): stack.append(child) + prose, summary = collect_doc_comments(root, source) + return ExtractResult( symbols=symbols, imports=list(dict.fromkeys(imports)), references=sorted(r for r in references if r), + docstring_summary=summary, + prose_paragraphs=prose, ) + + +def extract(source: bytes, filepath: str) -> ExtractResult: + if not AVAILABLE: + return ExtractResult() + return walk_source(_PARSER, source, _DEF_TYPES) diff --git a/.claude/skills/tokenhack/adapters/jvm.py b/.claude/skills/tokenhack/adapters/jvm.py index 1590dbe..a654ac9 100644 --- a/.claude/skills/tokenhack/adapters/jvm.py +++ b/.claude/skills/tokenhack/adapters/jvm.py @@ -3,7 +3,8 @@ Java and Kotlin share enough AST structure that one adapter handles both, dispatching to the right grammar by file extension. """ -from ._base import Symbol, ExtractResult, get_text, declaration_line +from ._base import (Symbol, ExtractResult, get_text, declaration_line, + collect_doc_comments) LANGUAGE_NAME = "jvm" FILE_EXTENSIONS = [".java", ".kt", ".kts"] @@ -120,8 +121,12 @@ def extract(source: bytes, filepath: str) -> ExtractResult: for child in reversed(node.children): stack.append(child) + prose, summary = collect_doc_comments(root, source) + return ExtractResult( symbols=symbols, imports=list(dict.fromkeys(imports)), references=sorted(r for r in references if r), + docstring_summary=summary, + prose_paragraphs=prose, ) diff --git a/.claude/skills/tokenhack/adapters/swift.py b/.claude/skills/tokenhack/adapters/swift.py index d14c7de..7bd9cb3 100644 --- a/.claude/skills/tokenhack/adapters/swift.py +++ b/.claude/skills/tokenhack/adapters/swift.py @@ -1,5 +1,6 @@ """Swift adapter for TokenHack.""" -from ._base import Symbol, ExtractResult, get_text, declaration_line +from ._base import (Symbol, ExtractResult, get_text, declaration_line, + collect_doc_comments) LANGUAGE_NAME = "swift" FILE_EXTENSIONS = [".swift"] @@ -74,8 +75,12 @@ def extract(source: bytes, filepath: str) -> ExtractResult: for child in reversed(node.children): stack.append(child) + prose, summary = collect_doc_comments(root, source) + return ExtractResult( symbols=symbols, imports=list(dict.fromkeys(imports)), references=sorted(r for r in references if r), + docstring_summary=summary, + prose_paragraphs=prose, ) diff --git a/.claude/skills/tokenhack/adapters/typescript.py b/.claude/skills/tokenhack/adapters/typescript.py new file mode 100644 index 0000000..590593f --- /dev/null +++ b/.claude/skills/tokenhack/adapters/typescript.py @@ -0,0 +1,62 @@ +"""TypeScript / TSX adapter for TokenHack. + +The TypeScript grammar is a superset of the JavaScript one, so the AST walk is +imported wholesale from `javascript.py` and only the extra declaration forms +are added here: interfaces, type aliases, enums, abstract classes, and the +class-property / constructor-parameter shapes TS introduces. + +`.ts` and `.tsx` need *different* parsers — tree-sitter-typescript ships two +grammars, because `` is a type assertion in .ts and a JSX element in .tsx. +Dispatching on extension is the whole reason this file isn't three lines. +""" +from .javascript import _DEF_TYPES as _JS_DEF_TYPES, walk_source +from ._base import ExtractResult + +LANGUAGE_NAME = "typescript" +FILE_EXTENSIONS = [".ts", ".tsx", ".mts", ".cts"] + +try: + from tree_sitter import Parser, Language + import tree_sitter_typescript + _TS_PARSER = Parser(Language(tree_sitter_typescript.language_typescript())) + _TSX_PARSER = Parser(Language(tree_sitter_typescript.language_tsx())) + AVAILABLE = True +except Exception: + _TS_PARSER = None + _TSX_PARSER = None + AVAILABLE = False + +# TS-only declaration forms. `interface_declaration` and `type_alias_declaration` +# matter most: in a typed codebase the domain vocabulary lives in the types, and +# a question about "the order payload" is far likelier to hit `interface Order` +# than any function name. +_TS_EXTRA_DEF_TYPES = { + "interface_declaration", + "type_alias_declaration", + "enum_declaration", + "abstract_class_declaration", + "module", # `declare module Foo` / namespace + "internal_module", + "public_field_definition", # class properties, incl. arrow-function members + "abstract_method_signature", + "method_signature", + "property_signature", + "function_signature", +} + +_DEF_TYPES = _JS_DEF_TYPES | _TS_EXTRA_DEF_TYPES + +_TSX_EXTENSIONS = (".tsx",) + + +def _parser_for(filepath: str): + return _TSX_PARSER if filepath.lower().endswith(_TSX_EXTENSIONS) else _TS_PARSER + + +def extract(source: bytes, filepath: str) -> ExtractResult: + if not AVAILABLE: + return ExtractResult() + parser = _parser_for(filepath) + if parser is None: + return ExtractResult() + return walk_source(parser, source, _DEF_TYPES) diff --git a/.claude/skills/tokenhack/requirements.txt b/.claude/skills/tokenhack/requirements.txt index f314c52..ab08d4c 100644 --- a/.claude/skills/tokenhack/requirements.txt +++ b/.claude/skills/tokenhack/requirements.txt @@ -6,3 +6,4 @@ tree-sitter-java>=0.23 tree-sitter-javascript>=0.23 tree-sitter-kotlin>=0.3 tree-sitter-swift>=0.6 +tree-sitter-typescript>=0.23 diff --git a/.claude/skills/tokenhack/router.py b/.claude/skills/tokenhack/router.py index d4033e6..1139e38 100644 --- a/.claude/skills/tokenhack/router.py +++ b/.claude/skills/tokenhack/router.py @@ -25,41 +25,63 @@ # ---------------------------------------------------------------------- # Tunable scoring constants — community-iterable. See README "Scoring". +# +# Every constant below can be overridden per-invocation with an environment +# variable named TOKENHACK_, e.g. TOKENHACK_ETA=2.0. That exists so +# `tests/eval.py` can ablate one signal at a time against the gold set — if you +# change a weight, you should be able to show the recall number that justifies +# it — and so a team can tune for their repo without forking the file. # ---------------------------------------------------------------------- -ALPHA = 1.5 # filename match weight -BETA = 1.0 # path-affinity weight -GAMMA = 0.5 # recency weight (file mtime decay) -EPSILON = 0.3 # symbol popularity weight (incoming imports) -ZETA = 0.5 # definition bonus (per matched def name) -DELTA_1HOP = 1.0 # 1-hop import-graph propagation -DELTA_2HOP = 0.3 # 2-hop import-graph propagation (decayed) -ETA = 1.2 # prose channel weight (docstrings + README/docs) - -BM25_K1 = 1.5 -BM25_B = 0.75 - -RECENCY_HALFLIFE_DAYS = 30 -HUB_FRACTION = 0.10 # files imported by >10% of corpus = hubs -LOW_CONFIDENCE_SCORE = 0.5 # top score below this → flag low-confidence -STALE_INDEX_FILE_THRESHOLD = 5 # N+ changed files since index build → warn +def _tunable(name, default): + raw = os.environ.get("TOKENHACK_" + name) + if raw is None: + return default + try: + return type(default)(raw) + except (TypeError, ValueError): + return default + + +ALPHA = _tunable("ALPHA", 1.5) # filename match weight +BETA = _tunable("BETA", 1.0) # path-affinity weight +GAMMA = _tunable("GAMMA", 0.5) # recency weight (file mtime decay) +EPSILON = _tunable("EPSILON", 0.3) # symbol popularity weight (incoming imports) +ZETA = _tunable("ZETA", 0.5) # definition bonus (per matched def name) +DELTA_1HOP = _tunable("DELTA_1HOP", 1.0) # 1-hop import-graph propagation +DELTA_2HOP = _tunable("DELTA_2HOP", 0.3) # 2-hop import-graph propagation (decayed) +# Prose channel weight (doc comments + markdown). Raised from 1.2 to 2.4 after +# doc-comment extraction shipped: with the channel actually populated it is the +# single most valuable signal in the ranker, because docs are written in the +# words a question uses and identifiers are written in the words a compiler +# needs. Measured on the 72-query gold set: ETA=0 -> hit@5 0.181, ETA=1.2 -> +# 0.208, ETA=2.4 -> 0.264. Flat from 2.4 through 6.0, so 2.4 is the knee. +ETA = _tunable("ETA", 2.4) + +BM25_K1 = _tunable("BM25_K1", 1.5) +BM25_B = _tunable("BM25_B", 0.75) + +RECENCY_HALFLIFE_DAYS = _tunable("RECENCY_HALFLIFE_DAYS", 30) +HUB_FRACTION = _tunable("HUB_FRACTION", 0.10) # files imported by >10% of corpus = hubs +LOW_CONFIDENCE_SCORE = _tunable("LOW_CONFIDENCE_SCORE", 0.5) # top score below this → flag low-confidence +STALE_INDEX_FILE_THRESHOLD = _tunable("STALE_INDEX_FILE_THRESHOLD", 5) # N+ changed files since index build → warn # Lexical-vs-prior balance. When the query has a strong term match somewhere # in the corpus, scale down the *query-independent* priors (recency, popularity) # so they act as tie-breakers rather than primary signals. -LEX_STRONG_THRESHOLD = 2.0 # max(bm25 + η·prose) ≥ this → strong lexical -LEX_STRONG_PRIOR_SCALE = 0.25 # multiplier applied to GAMMA, EPSILON when strong +LEX_STRONG_THRESHOLD = _tunable("LEX_STRONG_THRESHOLD", 2.0) # max(bm25 + η·prose) ≥ this → strong lexical +LEX_STRONG_PRIOR_SCALE = _tunable("LEX_STRONG_PRIOR_SCALE", 0.25) # multiplier applied to GAMMA, EPSILON when strong # Noise-hub penalty. A file with popularity > 0 but no query-derived signal # (no BM25 / prose hit, no filename match, no path affinity, no def bonus) is # the classic "noise hub" failure: a popular utility unrelated to the query # bubbling up purely on its import-count prior. Dock it instead of crediting. -NOISE_HUB_PENALTY = 0.8 +NOISE_HUB_PENALTY = _tunable("NOISE_HUB_PENALTY", 0.8) # Boost for files that match "callers of X" intent (either define X or import # a file that defines X). Tunable — kept above prior magnitudes so callers-of # results dominate the ranking when the mode fires. -CALLERS_BOOST = 3.0 +CALLERS_BOOST = _tunable("CALLERS_BOOST", 3.0) # Impl-pattern boost — for interface-heavy codebases (Java, JDK, much of # Spring/netty/JDBC) where the conceptual name lives on an interface and the @@ -73,8 +95,8 @@ # boost, since the user is explicitly asking for the impl, not the interface. IMPL_PREFIXES = ("default", "abstract", "base", "internal") IMPL_SUFFIXES = ("impl", "internal") -IMPL_BOOST = 0.8 # weight per overlapping core token -IMPL_INTENT_MULTIPLIER = 1.5 # extra when query has impl-walkthrough intent +IMPL_BOOST = _tunable("IMPL_BOOST", 0.8) # weight per overlapping core token +IMPL_INTENT_MULTIPLIER = _tunable("IMPL_INTENT_MULTIPLIER", 1.5) # extra when query has impl-walkthrough intent IMPL_INTENT_RE = re.compile( r"\b(implementation|implements?|walk\s*(?:me\s*)?through|how\s+does|how\s+is|" r"internals?|actually|under\s+the\s+hood|dispatch(?:es|er)?)\b", @@ -92,8 +114,8 @@ # so common-token explosions ("matched 'get' in 300 files") cannot # all participate simultaneously. Files below the cutoff still rank # by their own lexical signal — they just don't *radiate* it. -MIN_LEX_SEED = 1.0 -MAX_LEX_SEEDS = 25 +MIN_LEX_SEED = _tunable("MIN_LEX_SEED", 1.0) +MAX_LEX_SEEDS = _tunable("MAX_LEX_SEEDS", 25) # Filename-gated graph_prop compression. The raw graph_prop bonus accumulates # into the thousands for popular receivers, which is *useful* when that file @@ -108,12 +130,55 @@ # graph_prop is log-compressed so it acts as a tie-breaker rather than a # dominant signal. This keeps canonical-popular files at the top of symbol # lookups while preventing popular utilities from dominating concept queries. -GRAPH_FILENAME_GATE = 2.0 # filename-match threshold for ungated graph credit -GRAPH_PROP_SCALE = 2.0 # log-compression scale applied below the gate +GRAPH_FILENAME_GATE = _tunable("GRAPH_FILENAME_GATE", 2.0) # filename-match threshold for ungated graph credit +GRAPH_PROP_SCALE = _tunable("GRAPH_PROP_SCALE", 2.0) # log-compression scale applied below the gate -TOP_K = 5 -FUZZY_MAX_PER_TOKEN = 2 -FUZZY_CUTOFF = 0.85 +# Hard ceiling on graph propagation, relative to the file's OWN lexical score. +# +# The filename gate above was meant to keep graph_prop bounded, but it is +# trivially cleared: a natural-language question contains many words, so any +# well-connected file with two of them in its basename gets *raw* credit. On +# netty, "how many bytes does it ask for on each socket read" gave +# SocketReadPendingTest.java a graph_prop of 333 against a lexical score of 22 — +# graph propagation was 94% of the winning score, and the file that actually +# answers the question ranked 1,716th. +# +# Propagation is evidence that a *neighbour* matched, which is weaker than +# matching yourself. It may promote a plausible file; it must not invent one. +# So graph_prop is now always log-compressed AND capped at a multiple of the +# file's own query-derived score. +GRAPH_PROP_CAP_RATIO = _tunable("GRAPH_PROP_CAP_RATIO", 1.0) # graph_prop <= this * (own lexical score + floor) +GRAPH_PROP_CAP_FLOOR = _tunable("GRAPH_PROP_CAP_FLOOR", 1.0) # lets a zero-lexical neighbour still surface, barely + +# Test / benchmark / fixture demotion. +# +# Tests restate a concept's vocabulary far more densely than the implementation +# does, which is precisely what BM25 rewards — so on netty the top three results +# for a pipeline question were two testsuite files and a microbenchmark. Tests +# are useful context, but they belong in the paired-test slot, not the headline. +# Suppressed unless the question is explicitly about tests. +TEST_PATH_PENALTY = _tunable("TEST_PATH_PENALTY", 0.35) # multiplier applied to a test file's positive score +TEST_PATH_RE = re.compile( + r"(^|/)(tests?|testsuite|androidTest|__tests__|__mocks__|spec|specs|" + r"microbench|benchmarks?|fixtures?|examples?|testFixtures)(/|$)" + r"|(^|/)src/test/" + r"|[._-](test|tests|spec|benchmark)\.[A-Za-z0-9]+$" + r"|(Test|Tests|TestCase|Spec|Benchmark)\.[A-Za-z0-9]+$", + re.I, +) +TEST_INTENT_RE = re.compile(r"\b(test|tests|testing|spec|specs|benchmark|fixture|mock)\w*\b", re.I) + +TOP_K = _tunable("TOP_K", 5) +FUZZY_MAX_PER_TOKEN = _tunable("FUZZY_MAX_PER_TOKEN", 2) + +# Structural matches (filename, path) are scaled by token IDF so a hit on a +# distinctive word counts for more than a hit on `read` or `data`. IDF_REFERENCE +# is roughly the IDF of an ordinarily-informative term in a mid-sized corpus; +# dividing by it keeps the weight near 1.0 there, so ALPHA / BETA keep meaning. +IDF_REFERENCE = _tunable("IDF_REFERENCE", 3.0) +IDF_WEIGHT_CAP = _tunable("IDF_WEIGHT_CAP", 2.0) # a very rare token is worth at most 2 ordinary ones +MIN_QUERY_TOKEN_IDF = _tunable("MIN_QUERY_TOKEN_IDF", 0.35) # below this a query token is corpus-wide filler +FUZZY_CUTOFF = _tunable("FUZZY_CUTOFF", 0.85) CODE_STOPWORDS = set(""" the is a an and or but in on at to for of with by from as this that these those @@ -133,6 +198,29 @@ _SPLIT_RE = re.compile(r"[_\-/.:\\]+") +def stem_variant(token: str) -> str: + """Fold a token to a crude singular/base form for structural matching. + + Deliberately minimal — no Porter stemmer, no dictionary. It exists because + English questions and code identifiers disagree on number and tense far more + often than on the root: a question says "bytes", "credentials", "retries", + "uploading" while the code says `ByteBuf`, `Credential`, `retry`, `upload`. + Those near-misses cost real recall and one string operation fixes most of + them. Applied only to structural signals (filename / path / span selection), + never to the BM25 channels, where changing the term distribution would + invalidate the IDF table computed over the corpus. + """ + if len(token) < 4: + return token + # Plural folding only. A gerund rule was tried and removed: "string" -> "str" + # and "setting" -> "sett" collide with real identifiers, and the recall it + # bought did not cover that. + for suffix, repl in (("ies", "y"), ("es", ""), ("s", "")): + if token.endswith(suffix) and len(token) - len(suffix) >= 3: + return token[:len(token) - len(suffix)] + repl + return token + + def split_identifier(token: str): """Split CamelCase, snake_case, kebab-case, dot/slash-separated.""" out = [] @@ -372,21 +460,40 @@ def bm25_score(query_tokens, doc_tokens, idf, avgdl, matched_out=None): return score -def filename_match(query_tokens, rel_path): - """Count query tokens that appear in the file's basename stem. +def _idf_weight(token, idf): + """Scale a structural (non-BM25) match by how informative the token is. + + Filename and path matches used to count 1.0 per token regardless of what + the token was, so a question mentioning `read` and `socket` scored the same + filename hit as one mentioning `idempotency` and `backoff`. In a long + natural-language question most tokens are near-worthless — on netty, `netty` + itself has an IDF of 0.01 — and counting them equally is what let generic + files win. Normalised against IDF_REFERENCE so weights stay near 1.0 for an + ordinary term and the existing ALPHA / BETA defaults keep their meaning. + """ + return min(idf.get(token, 0.0) / IDF_REFERENCE, IDF_WEIGHT_CAP) + + +def filename_match(query_tokens, rel_path, idf=None): + """IDF-weighted count of query tokens appearing in the file's basename stem. The original (mixed-case) stem is fed to `split_identifier` so it can find CamelCase boundaries; the lowercased form is added for whole-name - matches. + matches. `idf=None` falls back to unweighted counting (used by the graph + gate, which wants a raw token count rather than a score). """ stem = os.path.splitext(os.path.basename(rel_path))[0] stem_tokens = set(split_identifier(stem)) stem_tokens.add(stem.lower()) - return float(sum(1 for q in query_tokens if q in stem_tokens)) + stem_tokens |= {stem_variant(t) for t in stem_tokens} + if idf is None: + return float(sum(1 for q in query_tokens if q in stem_tokens)) + return float(sum(_idf_weight(q, idf) for q in query_tokens + if q in stem_tokens or stem_variant(q) in stem_tokens)) -def path_affinity(query_tokens, rel_path): - """Count query tokens that appear in any path-segment of the file.""" +def path_affinity(query_tokens, rel_path, idf=None): + """IDF-weighted count of query tokens appearing in any path segment.""" parts = [] for chunk in re.split(r"[\\/]+", os.path.dirname(rel_path)): if not chunk: @@ -396,7 +503,15 @@ def path_affinity(query_tokens, rel_path): if not parts: return 0.0 parts_set = set(parts) - return float(sum(1 for q in query_tokens if q in parts_set)) + parts_set |= {stem_variant(t) for t in parts_set} + if idf is None: + return float(sum(1 for q in query_tokens if q in parts_set)) + return float(sum(_idf_weight(q, idf) for q in query_tokens + if q in parts_set or stem_variant(q) in parts_set)) + + +def is_test_path(rel_path): + return bool(TEST_PATH_RE.search(rel_path)) def recency_score(mtime, now): @@ -668,6 +783,18 @@ def rank(query: str, index: dict): now = time.time() hub_set = identify_hubs(rev, len(files)) has_impl_intent = bool(IMPL_INTENT_RE.search(query)) + wants_tests = bool(TEST_INTENT_RE.search(query)) + + # Drop corpus-wide filler from the query. A natural-language question carries + # words that are technically in the vocabulary but carry no discrimination + # ("netty" in netty has IDF 0.01, "data" in a data layer, "app" in an app). + # BM25 already discounts them, but they were still counted at full weight by + # filename_match and path_affinity. Keep at least one token so a query made + # entirely of common words still retrieves something. + informative = [t for t in query_tokens + if max(idf_code.get(t, 0.0), idf_prose.get(t, 0.0)) >= MIN_QUERY_TOKEN_IDF] + if informative: + query_tokens = informative # Pass 1 — collect raw per-file signals so we can derive a corpus-wide # lexical-strength measurement before applying prior weights. @@ -677,8 +804,8 @@ def rank(query: str, index: dict): matched: set = set() bm = bm25_score(query_tokens, channels["code"], idf_code, avgdl_code, matched_out=matched) prose_bm = bm25_score(query_tokens, channels["prose"], idf_prose, avgdl_prose, matched_out=matched) - fn = filename_match(query_tokens, rel) - pa = path_affinity(query_tokens, rel) + fn = filename_match(query_tokens, rel, idf_code) + pa = path_affinity(query_tokens, rel, idf_code) rc = recency_score(entry.get("mtime", 0), now) pop = popularity_score(entry, max_pop) defb = definition_bonus(query_tokens, entry) @@ -686,6 +813,12 @@ def rank(query: str, index: dict): "bm25": bm, "prose": prose_bm, "filename": fn, "path_aff": pa, "recency": rc, "popularity": pop, "def_bonus": defb, "matched": matched, + # Unweighted token count, used only by the graph gate below, which + # asks "how many query words are in this name" rather than "how good + # is this name" — a threshold on a weighted score would drift with + # corpus size. + "filename_tokens": filename_match(query_tokens, rel), + "is_test": is_test_path(rel), } # Fix #2 — strong-lexical detection. When something in the corpus matches @@ -710,6 +843,7 @@ def rank(query: str, index: dict): # path / def / callers_boost) we keep priors as tie-breakers in the # final ranking while preventing them from poisoning propagation. lexical_seeds = {} + lex_by_file = {} for rel, r in raw.items(): impl_b = impl_pattern_boost(query_tokens, rel, has_impl_intent) @@ -758,8 +892,19 @@ def rank(query: str, index: dict): + impl_b + callers_boost ) + + # Demote tests / benchmarks / fixtures unless the question is about them. + # Applied to positive scores only, so a penalised file can't be pushed + # below a file with no signal at all. + is_test = r["is_test"] and not wants_tests + if is_test: + if total > 0: + total *= TEST_PATH_PENALTY + lex *= TEST_PATH_PENALTY + if (lex >= MIN_LEX_SEED and has_query_signal) or callers_boost > 0: lexical_seeds[rel] = lex + lex_by_file[rel] = lex if total > 0 or r["matched"] or callers_boost > 0 or impl_b > 0: breakdowns[rel] = { @@ -771,6 +916,7 @@ def rank(query: str, index: dict): "callers_boost": callers_boost, "impl_boost": impl_b, "noise_hub": noise_hub, + "is_test": is_test, } base_scores[rel] = total @@ -794,19 +940,29 @@ def rank(query: str, index: dict): # Filename-gated compression: if the file *also* matches the query # by name (likely the canonical popular file), keep graph_prop raw. # Otherwise log-compress so generic utility hubs can't dominate. - fn_score = raw.get(rel, {}).get("filename", 0.0) + fn_tokens = raw.get(rel, {}).get("filename_tokens", 0.0) if raw_b <= 0: b = 0.0 - elif fn_score >= GRAPH_FILENAME_GATE: - b = raw_b else: + # Always compress. The old code handed out *raw* graph_prop to any + # file clearing the filename gate, which let a well-connected test + # file accumulate 333 points against a 22-point lexical score. b = GRAPH_PROP_SCALE * math.log1p(raw_b) + if fn_tokens >= GRAPH_FILENAME_GATE: + b *= 2.0 # the canonical-popular case: name also matches + # Then cap against the file's own query-derived evidence, so + # propagation can promote a plausible neighbour but never invent one. + own_lex = max(lex_by_file.get(rel, 0.0), 0.0) + b = min(b, GRAPH_PROP_CAP_RATIO * own_lex + GRAPH_PROP_CAP_FLOOR) + if raw.get(rel, {}).get("is_test") and not wants_tests: + b *= TEST_PATH_PENALTY if rel not in breakdowns: breakdowns[rel] = { "bm25": 0.0, "prose": 0.0, "filename": 0.0, "path_aff": 0.0, "recency": 0.0, "popularity": 0.0, "def_bonus": 0.0, "graph_prop": b, "matched_tokens": set(), "callers_boost": 0.0, "impl_boost": 0.0, "noise_hub": False, + "is_test": False, } else: breakdowns[rel]["graph_prop"] = b @@ -827,7 +983,7 @@ def rank(query: str, index: dict): # Symbol-span staging — point Claude at the matched defs, not whole files # ---------------------------------------------------------------------- -MAX_SPANS_PER_FILE = 4 # cap so a broad query can't list every def in a file +MAX_SPANS_PER_FILE = _tunable("MAX_SPANS_PER_FILE", 4) # cap so a broad query can't list every def in a file # A matched *class* (or Swift `extension`, or a top-level Kotlin object) has an # end_line at the end of the class — i.e. essentially the whole file. Staging @@ -844,7 +1000,7 @@ def rank(query: str, index: dict): # Signal-Android, DuckDuckGo iOS, OwnCloud, blinkit): staging every matched # span read 68% of the whole-file bytes; dropping container spans in favour of # the tight ones reads 16%. -WIDE_SPAN_FRACTION = 0.5 +WIDE_SPAN_FRACTION = _tunable("WIDE_SPAN_FRACTION", 0.5) def _file_line_extent(entry): @@ -929,6 +1085,22 @@ def format_output(query, index, ranked, breakdowns, meta, index_path): n_files = index.get("n_files_indexed", 0) lines.append(f"[tokenhack: index built {built_at}, {n_files} files indexed]") + # Coverage warning. The indexer records how many files it had to skip and + # why, but nothing ever read it — so on a repo written in a language with no + # adapter (TypeScript before this shipped, Go, Rust, Ruby, C#) the router + # would confidently rank the handful of files it *could* parse and give no + # hint that it had ignored most of the codebase. Silent partial coverage is + # worse than no coverage: it looks like an answer. + skipped_no_adapter = (index.get("skip_reasons") or {}).get("no-adapter", 0) + total_seen = n_files + skipped_no_adapter + if skipped_no_adapter and total_seen and skipped_no_adapter / total_seen >= 0.25: + lines.append( + f"[coverage warning: {skipped_no_adapter} of {total_seen} source files " + f"had no language adapter and are NOT in the index — results below cover " + f"only {100 * n_files // total_seen}% of this repo. Prefer grep for anything " + f"in an unindexed language]" + ) + # Staleness warning (conditional) root = index_path.parent.parent.parent.parent.parent if index_path else None if root and root.exists(): @@ -1036,10 +1208,22 @@ def format_output(query, index, ranked, breakdowns, meta, index_path): lines.append(f" ↳ read {format_span(start, end)}{suffix}") if test_pairs: - lines.append("") - lines.append("Paired test files:") - for src, tst in test_pairs: - lines.append(f" - {tst} (test for {src})") + # Label by what each file actually IS. When a test ranked in the top-K + # its partner is the *implementation*, and the old wording printed + # "DefaultChannelPipeline.java (test for DefaultChannelPipelineTest.java)" + # — naming the file that answers the question as the test of its own test. + impl_pairs = [(r, p) for r, p in test_pairs if is_test_path(r)] + tst_pairs = [(r, p) for r, p in test_pairs if not is_test_path(r)] + if impl_pairs: + lines.append("") + lines.append("Implementations of the tests above (usually the real answer):") + for tst, impl in impl_pairs: + lines.append(f" - {impl} (implementation under test in {tst})") + if tst_pairs: + lines.append("") + lines.append("Paired test files:") + for src, tst in tst_pairs: + lines.append(f" - {tst} (test for {src})") return "\n".join(lines) diff --git a/.github/workflows/tokenhack-eval.yml b/.github/workflows/tokenhack-eval.yml new file mode 100644 index 0000000..cac4cbf --- /dev/null +++ b/.github/workflows/tokenhack-eval.yml @@ -0,0 +1,67 @@ +name: tokenhack-eval + +# Retrieval evaluation against the gold sets in tests/gold/. +# +# Not on every push: reproducing the scores means cloning and indexing several +# large repositories (spring-framework alone is 9,487 indexed files), which is +# minutes of CI time. Run it on demand, and always when a PR touches the +# scoring constants or the adapters — a change to those without a before/after +# recall number is a change nobody can review. + +on: + workflow_dispatch: + inputs: + repos: + description: "Comma-separated repo names, or blank for all pinned repos" + type: string + default: "" + pull_request: + paths: + - '.claude/skills/tokenhack/router.py' + - '.claude/skills/tokenhack/adapters/**' + - 'tests/**' + +jobs: + evaluate: + runs-on: ubuntu-latest + timeout-minutes: 45 + steps: + - name: Checkout TokenHack + uses: actions/checkout@v4 + + - name: Set up Python + uses: actions/setup-python@v5 + with: + python-version: '3.11' + + - name: Install indexer dependencies + run: pip install -r .claude/skills/tokenhack/requirements.txt + + - name: Clone and index the evaluation repos + run: | + ARGS="" + if [[ -n "${{ inputs.repos }}" ]]; then + IFS=',' read -ra NAMES <<< "${{ inputs.repos }}" + for n in "${NAMES[@]}"; do ARGS="$ARGS --repo $(echo "$n" | xargs)"; done + fi + python3 tests/fetch_repos.py --dest "$RUNNER_TEMP/evalrepos" $ARGS + + - name: Evaluate + run: | + python3 tests/eval.py --repos-dir "$RUNNER_TEMP/evalrepos" --markdown \ + | tee "$RUNNER_TEMP/eval.txt" + + - name: Check for regression against the committed baseline + run: | + python3 tests/eval.py --repos-dir "$RUNNER_TEMP/evalrepos" --check + + - name: Publish results to the job summary + if: always() + run: | + { + echo '## TokenHack retrieval evaluation' + echo + echo '```' + cat "$RUNNER_TEMP/eval.txt" || echo "(no results)" + echo '```' + } >> "$GITHUB_STEP_SUMMARY" diff --git a/.gitignore b/.gitignore index 0eed2b9..8da3d7a 100644 --- a/.gitignore +++ b/.gitignore @@ -33,3 +33,6 @@ dist/ .claude/settings.local.json # Note: do NOT ignore .claude/skills/tokenhack/index/ — the index is the point. + +# Article drafts (not part of the tool) +medium-*.html diff --git a/README.md b/README.md index 4e9efcd..167859a 100644 --- a/README.md +++ b/README.md @@ -37,7 +37,9 @@ The first time you set it up — and on every push afterwards — an *indexer* w The mental model is the index at the back of a textbook. It's not a search engine. It doesn't understand what your code *does*. It's just a list of what's where, written down once so the assistant doesn't have to flip through the whole book every time it wants to find something. -To extract that list, the indexer uses [tree-sitter](https://tree-sitter.github.io/tree-sitter/) — a parser that actually understands code structure rather than just pattern-matching text. Out of the box, it handles Python, Java, Kotlin, JavaScript/JSX, and Swift. Files in unsupported languages are skipped without breaking the build; the rest of the index still gets written. +To extract that list, the indexer uses [tree-sitter](https://tree-sitter.github.io/tree-sitter/) — a parser that actually understands code structure rather than just pattern-matching text. Out of the box, it handles **Python, TypeScript/TSX, JavaScript/JSX, Java, Kotlin, and Swift**. Files in unsupported languages are skipped without breaking the build; the rest of the index still gets written, and the router warns you in its header when a large share of the repo had no adapter, so partial coverage never silently masquerades as an answer. + +It also captures **doc comments** — javadoc, KDoc, Swift doc comments, JSDoc, Python docstrings — into a separate prose channel. That turns out to matter more than anything else in the ranker, because docs are written in the words a human question uses while identifiers are written in the words a compiler needs. --- @@ -304,6 +306,24 @@ The three-gate nudge ruleset (next section) is what keeps the skill silent in th --- +## Does it actually retrieve the right file? + +For most of this project's life the honest answer was "I don't know." Now there's a number. + +`tests/` holds a **72-question gold set across 6 open-source repos** — spring-framework, netty, Signal-Android, DuckDuckGo iOS, ownCloud Android and a small full-stack JS app — each question paired with the file that actually answers it. It was built blind: the questions and answers were produced by agents that explored each repo with only glob/grep/read and were forbidden from running the router or reading its index, so the set isn't shaped around what TokenHack happens to be good at. 45 of the 72 are deliberately *vocabulary-mismatch* questions, where the question's words don't appear in the answer's name — the case lexical retrieval is expected to lose. + +```bash +pip install -r .claude/skills/tokenhack/requirements.txt +python3 tests/fetch_repos.py # clone + index the pinned public repos +python3 tests/eval.py +``` + +The scores are not flattering, and they're the point. See [`tests/README.md`](tests/README.md) for what each metric means and how to add questions. + +**If you change a scoring constant, put the before/after numbers in the PR.** Every constant is overridable as `TOKENHACK_` precisely so you can. + +--- + ## The three-gate nudge ruleset After `/tokenhack` is invoked once in a session, Claude is instructed to suggest re-invoking it only when **all three** of the following gates pass: @@ -349,8 +369,7 @@ The full roadmap — near-term, medium-term, and out-of-scope items — lives in **Sharply scoped first-PR opportunities** (each is a few hours of work and has a clear, testable surface): -- **TypeScript adapter.** Extend `adapters/javascript.py` to handle `.ts` and `.tsx`. The grammars share ~95% of node types. *(Highest-impact contribution — the majority of modern React / Vue codebases.)* -- **Vue.js adapter.** Add `adapters/vue.py` using `tree-sitter-vue` to handle `.vue` SFCs. Reuse the JS/TS adapter internally for `