Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,10 @@ not published here — see `docs/EVALS.md` for the instruments behind the headli

## [Unreleased]

### Added — Clojure (`.clj`/`.cljs`/`.cljc`/`.bb`)

Vendored Yogthos' ABI-15 `tree-sitter-clojure` parser and added syntax-level definitions and direct-call captures for the four Clojure-family extensions. The grammar comparison parsed 4,930 of 4,933 files (99.94%, 55.5 MB) from Metabase, clojure-lsp, clj-kondo and Babashka checkouts in 2.24 seconds; two failures were intentionally invalid clj-kondo fixtures and one was a Metabase test. This measures parser completion on that corpus, not semantic call-resolution accuracy. Namespace/import resolution, macro expansion and exact multi-arity matching remain outside this first pass; see `docs/ARCHITECTURE.md` for the extraction limits.

### Changed — Python imports use the shared capture vocabulary (#358, Python slice)

Python dependency targets now come from `@import.path`; named clauses and their optional local
Expand Down
10 changes: 9 additions & 1 deletion CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -271,6 +271,7 @@ endmacro()
set(_ripwire_query_names
bash
c
clojure
cpp
csharp
dart
Expand Down Expand Up @@ -552,6 +553,11 @@ add_ts_grammar(kotlin https://github.com/fwcd/tree-sitter-kotlin.git 1852ea17b7f
# gaps, NOT patched here (guardrail G3 — this repo carries no grammar fork).
add_ts_grammar(gdscript https://github.com/PrestonKnopp/tree-sitter-gdscript.git c5c8fa4861b5a4f04a7e60d97587fc3b6cc5639e) # master, ABI 14

# Yogthos' fork retains Sogaiu's grammar rules and regenerates an ABI-15 parser.
# On 4,933 real files it parsed 4,930 (99.94%) in 2.24 s; the Sogaiu ABI-14
# generated parser repeatedly stalled on the same corpus. CC0; no scanner.
add_ts_grammar(clojure https://github.com/yogthos/tree-sitter-clojure.git 8ec8407eada5f29728d746a46cbe6115938b5422) # v0.0.14, ABI 15

# PHP (.php): the official tree-sitter org grammar. Like TypeScript this repo hosts TWO sub-grammars
# in subdirectories — `php/` (the full one: HTML text interleaved with `<?php … ?>` blocks) and
# `php_only/` (no surrounding text). ONLY `php/` is vendored and built: a real .php file in a real
Expand Down Expand Up @@ -657,6 +663,7 @@ set(RIPWIRE_TS_OBJECTS
$<TARGET_OBJECTS:ts_dart>
$<TARGET_OBJECTS:ts_kotlin>
$<TARGET_OBJECTS:ts_gdscript>
$<TARGET_OBJECTS:ts_clojure>
$<TARGET_OBJECTS:ts_elixir>
$<TARGET_OBJECTS:ts_php>
$<TARGET_OBJECTS:ts_toml>
Expand Down Expand Up @@ -892,7 +899,7 @@ endif()
set(RIPWIRE_GRAMMAR_TARGETS
ts_cpp ts_python ts_go ts_rust ts_tsts ts_tstsx ts_swift ts_objc
ts_javascript ts_bash ts_java ts_ruby ts_json ts_toml ts_yaml ts_csharp ts_c ts_cuda ts_markdown
ts_php ts_lua ts_elixir ts_dart ts_kotlin ts_gdscript)
ts_php ts_lua ts_elixir ts_dart ts_kotlin ts_gdscript ts_clojure)
set(RIPWIRE_RUNTIME_COMPILE_TARGETS ripwire_probe ripwire tree-sitter ${RIPWIRE_GRAMMAR_TARGETS})
set(RIPWIRE_RUNTIME_LINK_TARGETS ripwire_probe ripwire)
# The C++ subset of the compile list — the only targets that include a C++ standard library at all, and so
Expand Down Expand Up @@ -1437,6 +1444,7 @@ if(RIPWIRE_FUZZ)
add_ripwire_fuzzer(lua tree_sitter_lua lua)
add_ripwire_fuzzer(kotlin tree_sitter_kotlin kotlin)
add_ripwire_fuzzer(gdscript tree_sitter_gdscript gdscript)
add_ripwire_fuzzer(clojure tree_sitter_clojure clojure)
add_custom_target(ripwire_fuzzers DEPENDS ${RIPWIRE_FUZZ_TARGETS})

# ---- reader fuzzers: ripwire's OWN parsers of bytes it did not create (test/fuzz/readers/) ----
Expand Down
15 changes: 8 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -136,7 +136,7 @@ claim cannot quietly drift. The row-by-row ledger is
</details>

**Languages:** Rust · C++ · Objective-C/C++ · C · Metal · CUDA · Python · Go · Swift · TypeScript ·
JavaScript · Java · Ruby · PHP · Lua · Elixir · Dart · Kotlin · GDScript · Bash · C# · JSON · TOML · YAML · Markdown — see
JavaScript · Java · Ruby · PHP · Lua · Elixir · Dart · Kotlin · GDScript · Clojure · Bash · C# · JSON · TOML · YAML · Markdown — see
[language support and limits](#languages).

**Latest: 0.6.7** — more complete answers, faster calls, fewer bytes. [Release notes](#release-notes) ·
Expand Down Expand Up @@ -2120,9 +2120,9 @@ wrong, and it has. These are the results that say so, all in-tree, all published
### In the tests

<details>
<summary><b>683 gate scripts</b>, five contracts no unit test can hold, and the house rule: write the gate before the code it measures</summary> <!-- gatecount -->
<summary><b>684 gate scripts</b>, five contracts no unit test can hold, and the house rule: write the gate before the code it measures</summary> <!-- gatecount -->

`test/regression.sh` names **683 gate scripts** and is the authoritative list; <!-- gatecount -->
`test/regression.sh` names **684 gate scripts** and is the authoritative list; <!-- gatecount -->
`python3 test/pargates.py . ./build/ripwire -j 6` runs the same set in parallel. On top of them sit the
contracts that do not fit a unit test: two runs byte-identical, warm output identical to cold, output
that pipes clean through `xmllint --noout`, a sanitizer build with `-fno-sanitize-recover=all`, and a
Expand Down Expand Up @@ -2313,7 +2313,7 @@ are one contributor's corpus away from being measurably better, and we cannot se
## Languages

<details>
<summary><b>25</b> vendored grammars, and what each parser does and does not see — CUDA launch edges, PHP dynamic dispatch, Lua metatables</summary>
<summary><b>26</b> vendored grammars, and what each parser does and does not see — CUDA launch edges, PHP dynamic dispatch, Lua metatables</summary>

C, C++, Objective-C / Objective-C++, **Metal** (Metal Shading Language, `.metal` — indexed with the
C++ grammar, since MSL is a C++14 dialect, so a dual-compile header's symbols resolve from both the
Expand All @@ -2331,7 +2331,7 @@ a runtime call with no syntax to read, so a Lua corpus reports no inheritance ed
implied), **Dart** (`.dart` — classes, mixins, extensions, enums, typedefs, functions, methods, getters/setters; `recv.m()`, `recv?.m()` and cascade `..m()` invocations are edges. Two stated floors: named constructors and factories index under the CLASS name, so `C()`, `C.seeded()` and `factory C.fromA()` are overloads of `C`; and `noSuchMethod` dynamic dispatch names its callee at run time. The grammar makes a function body a SIBLING of its signature rather than a child, so the definition span is extended through it at capture time — without that, every call in a body attributes to the enclosing class), **Elixir** (`.ex`/`.exs` — nested modules, structs, protocols and implementations, functions, macros, guards,
delegates, types, callbacks, attributes and literal ExUnit tests; module/name/arity resolution with lexical aliases,
filtered imports, default arguments, captures and pipes; see the
[static-analysis limits](docs/ARCHITECTURE.md#elixir-extraction)), **Kotlin** (`.kt` — classes, objects, companion objects, interfaces, enum classes and functions, extension functions included; bare and navigation calls and constructor delegation are call-graph edges; an import is a dependency edge only (role="import" on `--uses`, never a `--callers`/`--impact` edge — T13/fix3, 2026-09-20). Kotlin and Java share one call graph, and a call reaches the other language only when its own defines no candidate of that name, so adding `.kt` files never moves a Java edge. Stated floors: an explicit receiver (`A.f()`) does not narrow candidates; a multiplatform `expect`/`actual` type pair is two candidates; `.kts` is not indexed; and a file nesting string templates past 128 levels is refused and listed by `--skipped` — see the [Kotlin limits](docs/ARCHITECTURE.md#kotlin-extraction)), **GDScript** (`.gd` — a Godot file is a class body: `class_name` names it and its file-scope `func`/`var` are its members, with inner classes, constants, enums and their members, signals and call edges; `preload`/`load` produce no dependency edge yet, and `.tscn`/`.tres`/`.gdshader` are not indexed — see the [GDScript notes](docs/ARCHITECTURE.md#gdscript-extraction)), Bash, Go, Rust, Swift, C#, JSON + TOML + YAML (config keys — a
[static-analysis limits](docs/ARCHITECTURE.md#elixir-extraction)), **Kotlin** (`.kt` — classes, objects, companion objects, interfaces, enum classes and functions, extension functions included; bare and navigation calls and constructor delegation are call-graph edges; an import is a dependency edge only (role="import" on `--uses`, never a `--callers`/`--impact` edge — T13/fix3, 2026-09-20). Kotlin and Java share one call graph, and a call reaches the other language only when its own defines no candidate of that name, so adding `.kt` files never moves a Java edge. Stated floors: an explicit receiver (`A.f()`) does not narrow candidates; a multiplatform `expect`/`actual` type pair is two candidates; `.kts` is not indexed; and a file nesting string templates past 128 levels is refused and listed by `--skipped` — see the [Kotlin limits](docs/ARCHITECTURE.md#kotlin-extraction)), **GDScript** (`.gd` — a Godot file is a class body: `class_name` names it and its file-scope `func`/`var` are its members, with inner classes, constants, enums and their members, signals and call edges; `preload`/`load` produce no dependency edge yet, and `.tscn`/`.tres`/`.gdshader` are not indexed — see the [GDScript notes](docs/ARCHITECTURE.md#gdscript-extraction)), **Clojure** (`.clj`/`.cljs`/`.cljc`/`.bb` — namespaces, vars, functions, macros, multimethods, protocols, records and types; direct symbolic list-head calls are edges. Quoted, syntax-quoted and reader-discarded forms are excluded. Macro expansion, dynamic invocation, namespace aliases/import resolution, protocol/type method symbols, and exact multi-arity signatures are stated floors — see the [Clojure limits](docs/ARCHITECTURE.md#clojure-extraction)), Bash, Go, Rust, Swift, C#, JSON + TOML + YAML (config keys — a
`[tool.ruff.lint]` table is one symbol under its full dotted name, and
`pyproject.toml`/`Cargo.toml`/CI workflows become greppable), and **Markdown** (`.md`/`.markdown` —
the DOC tier: every heading, ATX or setext, is a section symbol whose span runs to the next
Expand Down Expand Up @@ -2961,7 +2961,7 @@ The CLI write verbs and the MCP write verbs use the same safety contract:

### 12. Supported languages and formats

The tool vendors 25 tree-sitter grammars. A table maps a file extension to a grammar and a query
The tool vendors 26 tree-sitter grammars. A table maps a file extension to a grammar and a query
file. One query engine runs over every language. A new language requires a vendored grammar, a query
file, and one row in the extension table.

Expand All @@ -2973,9 +2973,10 @@ file, and one row in the extension table.
| Metal (MSL) | `.metal` | Indexed with the C++ grammar. |
| CUDA | `.cu`, `.cuh` | `<<<>>>` launch sites are call edges. |
| Python | `.py` | |
| TypeScript / JavaScript | `.ts`, `.tsx`, `.mts`, `.cts`, `.js`, `.jsx`, `.mjs`, `.cjs` | Named imports and default imports resolve. One vendored dependency supplies two of the 25 grammars, `typescript` and `tsx`. `.astro` frontmatter rides this same `Lang` — see its own row below. |
| TypeScript / JavaScript | `.ts`, `.tsx`, `.mts`, `.cts`, `.js`, `.jsx`, `.mjs`, `.cjs` | Named imports and default imports resolve. One vendored dependency supplies two of the 26 grammars, `typescript` and `tsx`. `.astro` frontmatter rides this same `Lang` — see its own row below. |
| Java | `.java` | Qualified `new` calls resolve in a precise tier. |
| Kotlin | `.kt` | Shares one call graph with Java. A file with string templates past 128 levels is refused and listed by `--skipped`. |
| Clojure | `.clj`, `.cljs`, `.cljc`, `.bb` | Structural defining forms and direct symbolic list-head calls; quoted/discarded syntax is excluded. Macro expansion and namespace/import resolution are not inferred. |
| Ruby | `.rb` | Superclasses, mixins, `autoload`, and constant receivers are read. |
| PHP | `.php`, `.phtml` | Dynamic dispatch is a stated floor. |
| Lua | `.lua` | Metatable inheritance produces no inheritance edge. `require` is a function call. A string-literal `require` that names exactly one file in the tree adds a dependency edge. |
Expand Down
5 changes: 5 additions & 0 deletions THIRD_PARTY.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,7 @@ is why the sizes are what they are — `parser.c` is one big static table, not h
| `deps/markdown` | tree-sitter-markdown (v0.5.3; the block grammar `tree-sitter-markdown/` only) | Matthias Deiml | MIT | `f969cd3ae3f9fbd4e43205431d0ae286014c05b5` | https://github.com/tree-sitter-grammars/tree-sitter-markdown | 2.1 MB |
| `deps/kotlin` | tree-sitter-kotlin (ABI 14) | fwcd | MIT | `1852ea17b7f60fb3f9d84e0b1555d56b46b39fb1` | https://github.com/fwcd/tree-sitter-kotlin | 32 MB |
| `deps/gdscript` | tree-sitter-gdscript (master) | Preston Knopp | MIT | `c5c8fa4861b5a4f04a7e60d97587fc3b6cc5639e` | https://github.com/PrestonKnopp/tree-sitter-gdscript | 2624 KB |
| `deps/clojure` | tree-sitter-clojure v0.0.14 (ABI 15) | Sogaiu, Yogthos and contributors | CC0-1.0 | `8ec8407eada5f29728d746a46cbe6115938b5422` | https://github.com/yogthos/tree-sitter-clojure | 0.8 MB |
| `deps/doctest` | doctest (v2.4.12) | Viktor Kirilov | MIT | `1da23a3e8119ec5cce4f9388e91b065e20bf06f5` | https://github.com/doctest/doctest | 0.7 MB |

Notes:
Expand All @@ -91,6 +92,10 @@ Notes:
- `deps/kotlin` is pinned to a bare commit rather than the last tag (`v0.3.8`, Aug 2024) because
that tag predates a scanner segfault fix (upstream #136) present on `main`; the pinned commit is
`main`'s tip as of this vendoring, chosen for the fix, not for being a release.
- `deps/clojure` comes from Yogthos' fork of Sogaiu's grammar. The comparison used Sogaiu upstream
commit `e43eff80d17cf34852dcd92ca5e6986d23a7040f` against the Yogthos pin above. At those revisions the grammar
rules match; Yogthos' regenerated ABI-15 parser completed the 4,933-file qualification corpus
while Sogaiu's older ABI-14 generated parser repeatedly stalled.
- `deps/ts_typescript` keeps `common/scanner.h`, which both sub-grammars' `src/scanner.c` include.
- `deps/php` keeps the repo-relative layout `common/scanner.h` + `php/src/…` for the same reason, and
for one more: upstream hosts TWO sub-grammars (`php/`, `php_only/`) whose `src/scanner.c` each
Expand Down
19 changes: 19 additions & 0 deletions docs/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -250,6 +250,25 @@ triple-quoted string that ends in an escaped `$`). `test/kotlincheck.sh`
covers extraction, both bridge directions in flat and split layouts, the Java-edge invariant, the
bodyless-type collapse, hostile nesting, metrics and determinism.

<a id="clojure-extraction"></a>
**Clojure extraction.** The vendored Yogthos parser is a fork of Sogaiu's grammar. The grammar
rules match at the pinned revisions, while Yogthos' generated ABI-15 parser completed 4,930 of 4,933
files (99.94%, 55.5 MB across Metabase, clojure-lsp, clj-kondo and Babashka) in 2.24 s. Sogaiu's
ABI-14 generated parser repeatedly stalled on the same corpus. Two failures were intentionally
invalid clj-kondo corpus files; the third was a Metabase test file.

`queries/clojure/tags.scm` captures structural two-symbol defining lists and direct symbolic list
heads. `ingest_clojure.h` classifies namespaces, vars, functions, multimethods, macros, protocols,
records and types, and rejects quoted, syntax-quoted, reader-discarded and `(comment …)` subtrees. Syntax-quote
`~`/`~@` live holes and shared inert-reference accounting are deferred to issue #357. Macro-generated
definitions and calls, dynamic invocation, namespace aliases/import resolution and protocol/type
method symbols are not inferred. Multi-arity functions report inexact arity, so call resolution does
not narrow on a guess. Clojure does not claim dependency capability until namespace forms have a
corpus-index-backed resolver. The shared clone/readability scanner does not yet strip Clojure's `;`
comments; this can hide clone matches or inflate lexical readability counts. `test/clojurecheck.sh`
covers the four extensions, definition kinds, calls, quote/discard negatives, cache parity, root
spelling, skipped census and dependency/nonlocal-state disclosures.

Elixir extraction landed at revision 78 (rich 79) — `kParserVer` in `src/ingest_cache.h`, mirrored by
`kIngestParserVerMirror` in `src/quality.h`. The required `qschemetrip` source-change pin is refreshed
for this extraction change; snapshot scheme 8 is unchanged.
Expand Down
6 changes: 3 additions & 3 deletions docs/EVALS.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ section, and it is not an afterthought.
| **Co-change / known-item evals** | `--eval`, `--eval-retrieval` (see `bench/ANSWERQUALITY.md`) | Whether the tool surfaces the other files a real historical commit touched; and known-item retrieval across four rankers. |
| **Ensemble calibration harness** | `bench/ensemblecal/` | Whether `--ensemble`'s four evidence families are actually orthogonal, how often each fires, how stable each is across commits — and the preset ladder derived from that (§9). |
| **Differential argv harness** | `test/argvdiffcheck.sh` | That a refactor changed *nothing observable*: two binaries, every argv vector, stdout + stderr + exit code byte-identical. |
| **The gate suite** | `test/regression.sh`, `test/pargates.py` | 683 gate scripts plus the determinism, cache-transparency and golden contracts. <!-- gatecount --> |
| **The gate suite** | `test/regression.sh`, `test/pargates.py` | 684 gate scripts plus the determinism, cache-transparency and golden contracts. <!-- gatecount --> |
| **`--quality-delta`** | `src/quality.h` | Ten measured code-quality failure modes, reported only where a change made them worse. |

### The labeling protocol (why the held-out eval is allowed to disagree with the ranker)
Expand Down Expand Up @@ -5888,7 +5888,7 @@ copy here would be exactly the dialect divergence that gate exists to catch. Com
tags, wrap, stable-order defaults), seven individually invoked standalone gates (`g1freshcheck`,
`skillscan`, `htmlexport`, `compresscheck`, `handoffcheck`, `releaseinstallcheck`,
`taskroutecheck`), and a single loop
naming **683 gate scripts**, all of which exist on disk. <!-- gatecount -->
naming **684 gate scripts**, all of which exist on disk. <!-- gatecount -->

`python3 test/pargates.py . ./build/ripwire -j 6` runs the same scripts in parallel so a full
verification fits in one sitting. It does not modify `regression.sh`.
Expand Down Expand Up @@ -7005,7 +7005,7 @@ Listed because the reason is more useful than the silence.
shipped**. See `bench/locbench/anchorhop_calib.json`. The mention anchor's reproducible numbers are
the ablations in §4.
- **A single round gate-count.** Two in-tree numbers disagree (`test/pargates.py`'s docstring says
~210; `test/argvdiffcheck.sh` says 200+), while the loop in `test/regression.sh` names 683. The <!-- gatecount -->
~210; `test/argvdiffcheck.sh` says 200+), while the loop in `test/regression.sh` names 684. The <!-- gatecount -->
loop is the authority; the stale docstrings are a known drift. Since 2026-09-10 the number is not
written by hand anywhere: `docs/gatecount_build.py` derives it from the loop and rewrites every
published site, `test/gatecountcheck.sh` fails if any of them drifts, and `test/manifestcheck.sh`
Expand Down
Loading
Loading