Add retrieval evaluation, fix the bugs it found, add TypeScript

.claude/skills/tokenhack/README.md

@@ -13,12 +13,13 @@ Contributor-facing docs. For the user-facing pitch and install, see the
 ├── requirements.txt      # indexer deps (NOT needed on dev machines)
 ├── README.md             # this file
 ├── adapters/
-│   ├── _base.py          # Symbol, ExtractResult, shared helpers
+│   ├── _base.py          # Symbol, ExtractResult, doc-comment extraction
 │   ├── __init__.py       # adapter registry, get_adapter(filepath)
 │   ├── python.py         # .py
 │   ├── jvm.py            # .java, .kt, .kts  (shared adapter)
 │   ├── swift.py          # .swift
-│   └── javascript.py     # .js, .mjs, .cjs, .jsx  (JSX included)
+│   ├── javascript.py     # .js, .mjs, .cjs, .jsx  (JSX included)
+│   └── typescript.py     # .ts, .tsx, .mts, .cts (reuses the JS walk)
 └── index/
     ├── symbols.json      # built by CI; committed to the repo
     └── meta.json         # file hashes for incremental rebuilds
@@ -29,20 +30,40 @@ Contributor-facing docs. For the user-facing pitch and install, see the
 The router combines a BM25 baseline with structural and heuristic priors:
 
 ```
-tokens = camel_snake_split(query) − code_stopwords + fuzzy(close_matches)
+tokens = camel_snake_split(query) − code_stopwords − corpus_filler + fuzzy(close_matches)
 
 score(file, query) =
-    BM25(tokens, symbols ∪ refs ∪ path_tokens)        # lexical
-  + η · BM25(tokens, docstrings ∪ markdown_paragraphs) # prose channel
-  + α · filename_match(tokens, basename(file))
-  + β · path_affinity(tokens, dirname(file))
-  + γ · recency(mtime(file))                          # exponential decay
-  + ε · symbol_popularity(file)                       # incoming imports
+  [ BM25(tokens, symbols ∪ refs ∪ path_tokens)          # lexical
+  + η · BM25(tokens, doc_comments ∪ markdown_paragraphs) # prose channel
+  + α · Σ idf_weight(t) for t in filename_match(tokens, basename(file))
+  + β · Σ idf_weight(t) for t in path_affinity(tokens, dirname(file))
+  + γ · recency(mtime(file))                            # exponential decay
+  + ε · symbol_popularity(file)                         # incoming imports
   + ζ · definition_bonus(tokens, defs_in_file)
-  + δ₁ · graph_propagation_1hop(file, top_seeds)
-  + δ₂ · graph_propagation_2hop(file, top_seeds)      # decayed
+  + impl_pattern_boost(file)
+  ] · (test_penalty if file is a test and the query isn't about tests)
+  + min( graph_prop(file), own_lexical_score + floor )  # capped, see below
 ```
 
+**Corpus filler removal.** Query tokens whose IDF falls below
+`MIN_QUERY_TOKEN_IDF` are dropped. A natural-language question carries words
+that are technically in the vocabulary but carry no discrimination — `netty` in
+netty has an IDF of 0.01. BM25 already discounted them; the structural signals
+did not.
+
+**IDF-weighted structural matches.** `filename_match` and `path_affinity` used
+to count 1.0 per matching token regardless of the token. A question mentioning
+`read` and `socket` therefore scored the same filename hit as one mentioning
+`idempotency` and `backoff`. Both are now weighted by token IDF, normalised
+against `IDF_REFERENCE` so `ALPHA` / `BETA` keep their existing meaning.
+
+**Test/benchmark demotion.** Tests restate a concept's vocabulary far more
+densely than the implementation does, which is exactly what BM25 rewards — on
+netty, the top three results for a pipeline question were two testsuite files
+and a microbenchmark, and the implementation was absent. Test-path files are
+multiplied by `TEST_PATH_PENALTY` unless the question is explicitly about
+tests, and surface instead in the paired-results section.
+
 Hub suppression: files imported by more than `HUB_FRACTION` of the corpus
 do not propagate scores to their neighbours (utility hubs like `utils.py`
 would otherwise drag the whole repo in).
@@ -68,6 +89,17 @@ Graph-propagation credit is *restricted* to the caller set in this mode,
 so unrelated files that happen to match common tokens like `get` cannot
 accumulate inflated graph_prop and bury the actual target.
 
+**Capped graph propagation.** The filename gate below was meant to bound
+`graph_prop`, but a natural-language question contains many words, so any
+well-connected file with two of them in its basename cleared it and received
+*raw* credit. Measured on netty, `SocketReadPendingTest.java` accumulated a
+`graph_prop` of 333 against a lexical score of 22 — propagation was 94% of the
+winning score and the file that answered the question ranked 1,716th.
+Propagation is evidence that a *neighbour* matched, which is weaker than
+matching yourself, so it is now always log-compressed and then capped at
+`GRAPH_PROP_CAP_RATIO ×` the file's own query-derived score plus a small floor.
+It can promote a plausible file; it can no longer invent one.
+
 **Lexical-only graph propagation.** Recency and popularity are still
 summed into the final ranking but are not allowed to *seed* the import
 graph. In an actively-modified repo nearly every file has high recency,
@@ -88,7 +120,7 @@ in 300 files at once) from collectively dominating the graph.
 | `ZETA` | 0.5 | Bonus per matched definition name |
 | `DELTA_1HOP` | 1.0 | 1-hop import-graph propagation |
 | `DELTA_2HOP` | 0.3 | 2-hop propagation (decayed) |
-| `ETA` | 1.2 | Prose channel weight |
+| `ETA` | 2.4 | Prose channel weight (doc comments + markdown) — the highest-value signal in the ranker; see the measurement note in `router.py` |
 | `BM25_K1` | 1.5 | BM25 saturation parameter |
 | `BM25_B` | 0.75 | BM25 length-normalization parameter |
 | `RECENCY_HALFLIFE_DAYS` | 30 | Files this old contribute half their recency score |
@@ -111,7 +143,23 @@ in 300 files at once) from collectively dominating the graph.
 | `WIDE_SPAN_FRACTION` | 0.5 | A matched span covering more than this fraction of the file is a container (class / extension), not a target — dropped when a tighter span also matched |
 | `TOP_K` | 5 | How many ranked results to emit |
 
-Open a PR with rationale + measurements if you change any of these.
+| `TEST_PATH_PENALTY` | 0.35 | Multiplier on a test/benchmark/fixture file when the query isn't about tests |
+| `GRAPH_PROP_CAP_RATIO` | 1.0 | graph_prop ceiling as a multiple of the file's own lexical score |
+| `GRAPH_PROP_CAP_FLOOR` | 1.0 | Additive floor so a zero-lexical neighbour can still surface |
+| `IDF_REFERENCE` | 3.0 | IDF normaliser for structural (filename / path) match weighting |
+| `IDF_WEIGHT_CAP` | 2.0 | Max weight a single very-rare token can carry structurally |
+| `MIN_QUERY_TOKEN_IDF` | 0.35 | Query tokens below this IDF are corpus-wide filler and get dropped |
+
+Every constant here is overridable at runtime as `TOKENHACK_<NAME>`, so you can
+ablate it against the gold set:
+
+```bash
+TOKENHACK_ETA=0 python3 tests/eval.py
+```
+
+**Open a PR with rationale + measurements if you change any of these.** Since
+`tests/` exists there is no excuse for "felt better" — run `python3
+tests/eval.py` before and after and put both numbers in the PR body.
 
 ## Adding a new language adapter
 

.claude/skills/tokenhack/adapters/__init__.py

@@ -20,8 +20,9 @@
 from . import jvm as _jvm
 from . import swift as _swift
 from . import javascript as _javascript
+from . import typescript as _typescript
 
-_MODULES = (_python, _jvm, _swift, _javascript)
+_MODULES = (_python, _jvm, _swift, _javascript, _typescript)
 
 
 def _build_ext_map():

.claude/skills/tokenhack/adapters/_base.py

@@ -56,6 +56,97 @@ def declaration_line(source: bytes, name_node) -> str:
     return source[left:right].decode("utf-8", errors="replace").strip()
 
 
+# ----------------------------------------------------------------------
+# Doc-comment extraction (the prose channel)
+# ----------------------------------------------------------------------
+#
+# The router scores a separate "prose" BM25 channel at the highest weight in the
+# formula (ETA = 1.2), because docs are written in the words a human question
+# uses while identifiers are written in the words a compiler needs. Until now
+# only the Python adapter populated it — Java, Kotlin, Swift and JavaScript
+# returned nothing — so on netty, Spring, Signal, DuckDuckGo and ownCloud the
+# highest-weighted channel in the scoring function was empty for 100% of files.
+# Javadoc/KDoc/Swift-doc is exactly the natural-language bridge those repos
+# needed.
+
+DOC_PREFIXES = ("/**", "/*!", "///", "//!")
+
+_DOC_LINE_NOISE = ("/**", "/*!", "*/", "///", "//!")
+_JAVADOC_TAG = None   # compiled lazily; adapters import this module a lot
+
+
+def _doc_regexes():
+    global _JAVADOC_TAG
+    if _JAVADOC_TAG is None:
+        import re
+        _JAVADOC_TAG = (
+            re.compile(r"\{@\w+\s+([^}]*)\}"),       # {@link Foo#bar} -> Foo#bar
+            re.compile(r"^\s*@\w+"),                  # @param / @return line
+            re.compile(r"<[^>]+>"),                   # <p>, <code>, </pre>
+            re.compile(r"\s+"),
+        )
+    return _JAVADOC_TAG
+
+
+def clean_doc_comment(text: str) -> str:
+    """Turn a raw /** ... */ or /// block into a plain prose paragraph.
+
+    Drops comment syntax, inline javadoc markup ({@link X} keeps X), HTML tags,
+    and standalone block-tag lines (@param, @return, @throws), which are
+    structure rather than description and mostly repeat identifiers the code
+    channel already indexes.
+    """
+    inline_tag, block_tag, html, ws = _doc_regexes()
+    out = []
+    for raw in text.splitlines():
+        line = raw.strip()
+        for noise in _DOC_LINE_NOISE:
+            if line.startswith(noise):
+                line = line[len(noise):]
+        line = line.strip()
+        if line.startswith("*"):
+            line = line[1:].strip()
+        if line.endswith("*/"):
+            line = line[:-2].strip()
+        if not line or block_tag.match(line):
+            continue
+        out.append(line)
+    joined = " ".join(out)
+    joined = inline_tag.sub(r"\1", joined)
+    joined = html.sub(" ", joined)
+    return ws.sub(" ", joined).strip()
+
+
+def collect_doc_comments(root, source: bytes, max_paragraphs=8, max_chars=240):
+    """Return (paragraphs, summary) from the doc comments in a parsed tree.
+
+    Any node whose type contains "comment" is considered; only blocks opening
+    with a doc marker are kept, so ordinary `//` implementation notes and
+    license headers don't flood the channel. Language-agnostic on purpose —
+    Java, Kotlin, Swift and JavaScript all spell doc comments the same way.
+    """
+    paragraphs = []
+    stack = [root]
+    while stack and len(paragraphs) < max_paragraphs:
+        node = stack.pop()
+        if "comment" in node.type:
+            text = get_text(node, source).lstrip()
+            if text.startswith(DOC_PREFIXES):
+                cleaned = clean_doc_comment(text)
+                # Skip license/copyright boilerplate, which every file repeats
+                # and which would otherwise dominate the corpus term stats.
+                low = cleaned[:120].lower()
+                if cleaned and len(cleaned) > 24 and not (
+                    "copyright" in low or "licensed under" in low
+                    or "license, version" in low or "all rights reserved" in low
+                ):
+                    paragraphs.append(cleaned[:max_chars])
+        for child in reversed(node.children):
+            stack.append(child)
+    summary = first_line(paragraphs[0]) if paragraphs else ""
+    return paragraphs, summary[:max_chars]
+
+
 def strip_string_quotes(s: str) -> str:
     """Strip leading/trailing string quote markers (single, double, triple)."""
     s = s.strip()

.claude/skills/tokenhack/adapters/javascript.py

@@ -1,9 +1,11 @@
 """JavaScript adapter for TokenHack (JSX included).
 
-Note: TypeScript is intentionally NOT covered in v1 — see CONTRIBUTING
-section of the top-level README for the planned first-PR scope.
+The AST walk lives in `walk_source` so the TypeScript adapter can reuse it —
+the grammars share the great majority of their node types, and TS only needs to
+add its own declaration forms (interface, type alias, enum) on top.
 """
-from ._base import Symbol, ExtractResult, get_text, declaration_line
+from ._base import (Symbol, ExtractResult, get_text, declaration_line,
+                    collect_doc_comments)
 
 LANGUAGE_NAME = "javascript"
 FILE_EXTENSIONS = [".js", ".mjs", ".cjs", ".jsx"]
@@ -27,11 +29,9 @@
 }
 
 
-def extract(source: bytes, filepath: str) -> ExtractResult:
-    if not AVAILABLE:
-        return ExtractResult()
-
-    tree = _PARSER.parse(source)
+def walk_source(parser, source: bytes, def_types) -> ExtractResult:
+    """Shared JS/TS extraction: definitions, imports, call references, docs."""
+    tree = parser.parse(source)
     root = tree.root_node
 
     symbols = []
@@ -43,7 +43,7 @@ def extract(source: bytes, filepath: str) -> ExtractResult:
         node = stack.pop()
         nt = node.type
 
-        if nt in _DEF_TYPES:
+        if nt in def_types:
             name_node = node.child_by_field_name("name")
             if name_node is not None:
                 name = get_text(name_node, source)
@@ -84,8 +84,18 @@ def extract(source: bytes, filepath: str) -> ExtractResult:
         for child in reversed(node.children):
             stack.append(child)
 
+    prose, summary = collect_doc_comments(root, source)
+
     return ExtractResult(
         symbols=symbols,
         imports=list(dict.fromkeys(imports)),
         references=sorted(r for r in references if r),
+        docstring_summary=summary,
+        prose_paragraphs=prose,
     )
+
+
+def extract(source: bytes, filepath: str) -> ExtractResult:
+    if not AVAILABLE:
+        return ExtractResult()
+    return walk_source(_PARSER, source, _DEF_TYPES)

.claude/skills/tokenhack/adapters/jvm.py

@@ -3,7 +3,8 @@
 Java and Kotlin share enough AST structure that one adapter handles both,
 dispatching to the right grammar by file extension.
 """
-from ._base import Symbol, ExtractResult, get_text, declaration_line
+from ._base import (Symbol, ExtractResult, get_text, declaration_line,
+                    collect_doc_comments)
 
 LANGUAGE_NAME = "jvm"
 FILE_EXTENSIONS = [".java", ".kt", ".kts"]
@@ -120,8 +121,12 @@ def extract(source: bytes, filepath: str) -> ExtractResult:
         for child in reversed(node.children):
             stack.append(child)
 
+    prose, summary = collect_doc_comments(root, source)
+
     return ExtractResult(
         symbols=symbols,
         imports=list(dict.fromkeys(imports)),
         references=sorted(r for r in references if r),
+        docstring_summary=summary,
+        prose_paragraphs=prose,
     )

.claude/skills/tokenhack/adapters/swift.py

@@ -1,5 +1,6 @@
 """Swift adapter for TokenHack."""
-from ._base import Symbol, ExtractResult, get_text, declaration_line
+from ._base import (Symbol, ExtractResult, get_text, declaration_line,
+                    collect_doc_comments)
 
 LANGUAGE_NAME = "swift"
 FILE_EXTENSIONS = [".swift"]
@@ -74,8 +75,12 @@ def extract(source: bytes, filepath: str) -> ExtractResult:
         for child in reversed(node.children):
             stack.append(child)
 
+    prose, summary = collect_doc_comments(root, source)
+
     return ExtractResult(
         symbols=symbols,
         imports=list(dict.fromkeys(imports)),
         references=sorted(r for r in references if r),
+        docstring_summary=summary,
+        prose_paragraphs=prose,
     )

.claude/skills/tokenhack/adapters/typescript.py

@@ -0,0 +1,62 @@
+"""TypeScript / TSX adapter for TokenHack.
+
+The TypeScript grammar is a superset of the JavaScript one, so the AST walk is
+imported wholesale from `javascript.py` and only the extra declaration forms
+are added here: interfaces, type aliases, enums, abstract classes, and the
+class-property / constructor-parameter shapes TS introduces.
+
+`.ts` and `.tsx` need *different* parsers — tree-sitter-typescript ships two
+grammars, because `<T>` is a type assertion in .ts and a JSX element in .tsx.
+Dispatching on extension is the whole reason this file isn't three lines.
+"""
+from .javascript import _DEF_TYPES as _JS_DEF_TYPES, walk_source
+from ._base import ExtractResult
+
+LANGUAGE_NAME = "typescript"
+FILE_EXTENSIONS = [".ts", ".tsx", ".mts", ".cts"]
+
+try:
+    from tree_sitter import Parser, Language
+    import tree_sitter_typescript
+    _TS_PARSER = Parser(Language(tree_sitter_typescript.language_typescript()))
+    _TSX_PARSER = Parser(Language(tree_sitter_typescript.language_tsx()))
+    AVAILABLE = True
+except Exception:
+    _TS_PARSER = None
+    _TSX_PARSER = None
+    AVAILABLE = False
+
+# TS-only declaration forms. `interface_declaration` and `type_alias_declaration`
+# matter most: in a typed codebase the domain vocabulary lives in the types, and
+# a question about "the order payload" is far likelier to hit `interface Order`
+# than any function name.
+_TS_EXTRA_DEF_TYPES = {
+    "interface_declaration",
+    "type_alias_declaration",
+    "enum_declaration",
+    "abstract_class_declaration",
+    "module",                       # `declare module Foo` / namespace
+    "internal_module",
+    "public_field_definition",      # class properties, incl. arrow-function members
+    "abstract_method_signature",
+    "method_signature",
+    "property_signature",
+    "function_signature",
+}
+
+_DEF_TYPES = _JS_DEF_TYPES | _TS_EXTRA_DEF_TYPES
+
+_TSX_EXTENSIONS = (".tsx",)
+
+
+def _parser_for(filepath: str):
+    return _TSX_PARSER if filepath.lower().endswith(_TSX_EXTENSIONS) else _TS_PARSER
+
+
+def extract(source: bytes, filepath: str) -> ExtractResult:
+    if not AVAILABLE:
+        return ExtractResult()
+    parser = _parser_for(filepath)
+    if parser is None:
+        return ExtractResult()
+    return walk_source(parser, source, _DEF_TYPES)

.claude/skills/tokenhack/requirements.txt

@@ -6,3 +6,4 @@ tree-sitter-java>=0.23
 tree-sitter-javascript>=0.23
 tree-sitter-kotlin>=0.3
 tree-sitter-swift>=0.6
+tree-sitter-typescript>=0.23