Skip to content

feat(ingest): extract JS/TS arrow and function-expression definitions - #142

Merged
jmagly merged 3 commits into
elder-plinius:mainfrom
lyubomir-bozhinov:feat/ingest-arrow-fn-extraction
Aug 1, 2026
Merged

feat(ingest): extract JS/TS arrow and function-expression definitions#142
jmagly merged 3 commits into
elder-plinius:mainfrom
lyubomir-bozhinov:feat/ingest-arrow-fn-extraction

Conversation

@lyubomir-bozhinov

Copy link
Copy Markdown
Contributor

Closes #141.

White-box ingest extracted JS/TS declarations only. Every definition bound to a value — the dominant shape in modern Node/TS — produced no CodeBlock, so its body never reached sink classification, priority scoring, entry-point detection or the context pack. The file parsed fine and simply yielded nothing, so the hole was silent.

const runCmd = (cmd) => exec(cmd);              // exec() sink went unranked
module.exports.handler = async (e) => run(e);   // Lambda entry point invisible
const routes = { upload: (p) => open(p) };      // not extracted
class C { #handler = (x) => eval(x); }          // not extracted

This is the follow-up to #75, where the limit was documented in-code when the multi-language ingest landed.

The change

Query-only, in the .js / .ts / .tsx def-queries (src/recon/ts-grammars.ts). No parser, pipeline or CodeBlock-shape change: nodeToCodeBlock already handles any captured @def/@name/@params triple and kindOf maps these to function.

Form Node
const f = (a) => {} / = function (a) {} variable_declarator
const f = a => … (un-parenthesized single param) variable_declarator
obj.f = …, module.exports.handler = … assignment_expression (member)
onmessage = e => … (worker/global handler) assignment_expression (identifier)
{ f: (a) => {} } pair
class K { f = (a) => {} }, #f = … field_definition / public_field_definition

Property-assigned and object-literal forms are named by the property, consistent with how method_definition is named — that is the name buildCallGraph and the entry-point heuristic match on. All three dialects share one pattern set; only the class-field node name differs (field_definition property: vs public_field_definition name:).

Deliberately not matched, because there is no name-to-function binding and inventing one is worse than honest absence:

  • HOC-wrapped values — const h = withAuth(async (req) => …) (the value is a call_expression)
  • anonymous callbacks — app.get('/x', (req, res) => …)

The whitebox.ts caveat block is updated in the same commit to say exactly this.

Also fixed here (found by adversarial review of the change)

Same-line block-id collision. A block id is path::name@line, so two same-named definitions starting on the same line collide — and buildCallGraph keys on that id, so the two silently share one callers/callees entry even when only one carries a sink. Rare for declarations; routine once value-bound functions are extracted, since minified/bundled JS packs many { handler: … } onto one line. The later block is now disambiguated by column. The first keeps the plain id, so no existing id changes and the Python path is untouched.

Verification

RED first — the test commit fails on main (26 failures across all three dialects), passes after the query change.

  • npm test708 passed, 65 files (was 675 before this branch; +33 new). Green.
  • npm run test:coverage100% statements/branches/functions/lines on the three coverage-gated recon files, including the new branch in ts-parse.ts.
  • npm run typecheck — clean. eslint on every changed file — 0 errors.
  • npm run doctor — PASS (warnings are pre-existing optional-tooling/offline-API items).

The new suite pins both directions: each form extracted with the right name, params and span, and the excluded shapes (const x = 42, IIFE, destructured bindings, HOC-wrapped) asserted absent. The full-set equality assertion catches a missed form and an over-matching pattern in one check. An end-to-end test runs the real ingestRepository and asserts an arrow-bound sink ranks attack_surface, an arrow-bound handler becomes an entry point, and the call-graph edge between them resolves.

Scale check against this repo's own 1.24 MB / 21k-line SPA script (docs/index.html), old query vs new, same parse tree:

blocks query match time
before 560 32 ms
after 679 28 ms

+119 real definitions recovered, no measurable match-time cost, and zero matches with an unbound @name/@def capture across that corpus — an unbound capture would throw and fail-open the whole file to [], so this was checked explicitly rather than assumed.

Verified on macOS (Apple Silicon) only — that is the only platform I can test on. The change is a tree-sitter query string, so it carries no platform-specific behaviour, but I am not claiming Linux/Windows verification I did not do.

Commits

f8988e0 test(ingest): RED baseline — fails on main
c8b0e20 feat(ingest): the query change + caveat-block update
7f34c47 fix(ingest): #private fields + same-line id disambiguation (review findings)

…re not extracted

Pins issue elder-plinius#141 in both directions: every named-value function form must be
extracted with its name, parameters and span, and the shapes carrying no
name-to-function binding (IIFE, destructured, HOC-wrapped) must stay unmatched.
Also covers the end-to-end consequence — an arrow-bound sink ranking
attack_surface, and an arrow-bound handler becoming an entry point.

Fails on main: the .js/.ts/.tsx def-queries bind declarations only.

Refs elder-plinius#141
The .js/.ts/.tsx def-queries bound declarations only, so every definition bound
to a value produced no CodeBlock and its sinks never reached classification or
ranking. Adds the name-to-function-value shapes to all three queries:

  const f = (a) => {} / function (a) {}   variable_declarator
  const f = a => ...                      un-parenthesized single param
  obj.f = ... / module.exports.handler    assignment_expression
  onmessage = e => ...                    bare-identifier assignment
  { f: (a) => {} }                        pair
  class K { f = (a) => {} }               field_definition / public_field_definition

Query-only — no parser, pipeline or CodeBlock-shape change. nodeToCodeBlock
already handles any @def/@name/@params triple and kindOf maps these to
'function'. Property-assigned and object-literal forms are named by the
property, consistent with how method_definition is named: that is the name
buildCallGraph and the entry-point heuristic match on.

Not matched, by design: HOC-wrapped values (const h = withAuth(fn)) and
anonymous callbacks (app.get('/x', (req,res) => ...)) have no name-to-function
binding, and inventing one is worse than honest absence. whitebox.ts caveat
block updated to match.

Refs elder-plinius#141
…k ids distinct

Two defects found by adversarial review of the arrow-extraction change:

- `class C { #handler = (x) => eval(x) }` yielded nothing: a private field's
  name is a `private_property_identifier`, not the `property_identifier` the
  class-field pattern required. A sink in a private handler is exactly as
  dangerous as one in a public handler, so it is captured (name keeps the `#`).

- A block id is `path::name@line`, so two same-named definitions starting on the
  same line collide, and buildCallGraph keys on that id — the two blocks silently
  share one callers/callees entry even when only one carries a sink. Rare for
  declarations, routine once value-bound functions are extracted (minified
  bundles pack many `{ handler: … }` onto one line). The later block is now
  disambiguated by column; the first keeps the plain id, so no existing id
  changes and the Python path is untouched.

Refs elder-plinius#141

@jmagly jmagly left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 7f34c47b97b0605dfb7dad5edaff646659eccee2.

No blocking findings. The query patterns cover the named-value forms without broadening into anonymous or wrapped callbacks, the same-line ID disambiguation preserves existing first-occurrence IDs, and the regression suite exercises extraction, exclusions, security ranking, entry-point elevation, and reachability.

Verification on a synthetic merge into current main (a606adf):

  • npm run typecheck
  • focused regression suite: 33/33
  • full suite: 740/740 Vitest, 11/11 ops preflight, 19/19 model matrix
  • hosted CI: pass

@jmagly
jmagly merged commit 9020e5f into elder-plinius:main Aug 1, 2026
1 check passed
@lyubomir-bozhinov
lyubomir-bozhinov deleted the feat/ingest-arrow-fn-extraction branch August 3, 2026 08:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

White-box ingest: JS/TS arrow functions and function expressions are never extracted

2 participants