From 986540cf66b4b2c335f39c6f85125a56b9737a47 Mon Sep 17 00:00:00 2001 From: joyful-ii-V-I Date: Mon, 28 Sep 2026 08:14:33 -0400 Subject: [PATCH 1/2] docs(lang): adding a language is data first; common mechanisms stay common prompts/add-a-language.md STEP 3 told contributors to model a new language on src/ingest_elixir.h, so the per-language-file pattern spread (Clojure followed it faithfully). STEP 3 now leads with tags.scm in the shared capture vocabulary, a table of the shared mechanisms to extend instead of copy (enclosingScopeOf, VarDecl shadowing, Binding/Include imports, the qualified- name resolver), the refuse-don't-ladder rule for unresolved qualified calls, and names the real gap honestly: tags-pass predicates do not run yet, which is why keyword-headed languages keep a small C++ table for now. STEP 4 asks for the reserved kParserVer instead of the next integer. CONTRIBUTING gains the matching short rule. Co-Authored-By: Claude Opus 5.5 --- CONTRIBUTING.md | 17 ++++++++++++++++ prompts/add-a-language.md | 42 +++++++++++++++++++++++++++++++++------ 2 files changed, 53 insertions(+), 6 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 437ec3f62..5bf9d216c 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -472,6 +472,23 @@ every input the tool ever sees, costs nothing in release, and tells the optimize ordered container); and `unordered_dense` invalidates references on insert (values live in one vector), so never hold a `T&` into it across an insert. +### Adding a language: common stays common + +A new language is **data plus, at most, small per-language pieces for semantics that genuinely differ** — +never a copy of another language's mechanism. The step-by-step path is `prompts/add-a-language.md`; the +rules it enforces: + +- **Data, not branches.** A `kLangTable` row, a `queries//tags.scm` in the shared capture + vocabulary, and a row in each exhaustive `switch( Lang )` table. No language-named capture kinds. +- **Reuse the shared mechanism.** Scope walks, shadowing (`VarDecl` bindings with spans), imports + (`Binding`/`Include`) and qualified-name resolution each exist once. Extend them; a second copy under a + new language's name is a review finding, and `--clones` shows it. +- **Refuse, don't guess.** A call whose qualifier (`Mod.f`, `ns/f`) cannot be resolved is counted as + unresolved, never laddered to every definition of that short name. +- **Known gap.** Tags-pass predicates (`#eq?`/`#any-of?`) do not run yet, so keyword-headed definitions + (Elixir, Clojure) keep a small keyword table in C++. That table belongs in `tags.scm` once the + mechanism lands; it is not a pattern to extend. + ### Interfaces - **Structured-binding returns** over out-params: `auto [ nodes, edges ] = build( … );`. diff --git a/prompts/add-a-language.md b/prompts/add-a-language.md index 46a74b5b5..95799a8eb 100644 --- a/prompts/add-a-language.md +++ b/prompts/add-a-language.md @@ -60,12 +60,39 @@ The build must still complete **with the network off**. That is the point of ven ## STEP 3 — extraction -- `src/ingest_LANG.h` — the extraction module. Read `src/ingest_elixir.h` first; it is the newest - and the most conventional. -- `src/ingest.cpp` — include it and dispatch to it. -- `src/ingest_sidecap.h` — sidecar capture, if LANG has doc comments worth carrying. -- `src/ingest_metrics.h`, `src/clones.h`, `src/lintcatalog.h`, `src/htmlexport.h`, `src/cli.h` — each - has a per-language arm. Grep for the newest language name to find every one. +**A language is data first, and C++ only where its semantics truly differ.** What is common must stay +common: twenty languages each carrying a private copy of the same idea is twenty places to fix one bug. + +- `queries/LANG/tags.scm` — this is where most of a language lives. Use **only the shared capture + vocabulary**: `@name`, `@definition.{function,method,class,struct,interface,var,constant,macro,module,type,field}`, + `@reference.{call,import,implementation,type}`. Do not invent a language-named capture + (`@definition.LANG`, `@LANG.head`); every consumer downstream has to learn a new name for it. +- The exhaustive `switch( Lang )` tables (`langTag`, `isCodeLang`, `dependencyCapable`, …) — the compiler + lists every one you missed (`-Werror=switch`). A row each, no logic. +- Per-language C++ is for semantics the grammar cannot express. **Before writing a helper, look for the + shared one:** + + | You need | Use (do not copy) | + |---|---| + | the scope a definition sits in | `enclosingScopeOf` (`src/ingest_names.h`) | + | a local that shadows a global name (`let`, params) | a `VarDecl` binding with its block span — the generic shadow pass (`suppressShadowedReferences`) does the rest | + | imports / aliases / `require … :as` | `Binding` / `Include` facts (`src/model.h`), as JS imports and Elixir aliases already do | + | module-qualified calls (`Mod.f`, `ns/f`) | extend the qualified-name resolver in `src/elixir_resolve.h`; do not write a `LANG_resolve.h` beside it | + + Then run `ripwire --clones` on your tree: your new code should add no clone group. +- **A qualified call you cannot resolve is counted as unresolved — never sent to the global name ladder.** + `(str/join …)` laddered to every `join` in the corpus mints edges that are not real (the PR #81 decision: + the ladder gave one Elixir function 55 callers where 3 were real). + +**A known gap, so you are not surprised by it:** the tags pass does not evaluate `#eq?`/`#any-of?` +predicates yet (see the notes in `queries/javascript/tags.scm`). A language whose definitions are *a call +with a keyword head* — Elixir `def`, Clojure `defn` — therefore has to keep that keyword list in C++ for +now. That is the reason `src/ingest_elixir.h` exists; it is **not a template to copy**. If you hit this, keep +the list a small `constexpr` table in one place, and say in the PR that it belongs in `tags.scm` once +predicates run. Maintainers will move it; you are not asked to build the mechanism. + +If the shared mechanism you need does not exist, say so in the PR. We would rather add it with you than +see the pattern forked. Extract **definitions with spans** first. Call edges are a separate round. @@ -82,6 +109,9 @@ Miss the mirror and `qextractionkeycheck` fails reporting a version mismatch tha totally unrelated bug. Bump on **any** grammar, `.scm`, or extraction change — the cache is keyed on it, and a stale cache serving old symbols for new code is the worst failure this tool has. +**Do not take the next integer blindly.** Parallel pull requests each need their own number, and the +maintainers keep a reservation list. Ask in your PR which number is yours. + Then regenerate `test/qschemetrip.hash` and update `test/printf_parity.manifest`. --- From 456e6fb5ad6d5572559a4611e428c5a6c07a83fc Mon Sep 17 00:00:00 2001 From: joyful-ii-V-I Date: Wed, 30 Sep 2026 04:01:25 -0400 Subject: [PATCH 2/2] docs(add-a-language): correct the resolver, grep-step, Clojure and parser-version claims - module-qualified calls: elixir_resolve.h is Elixir-specific; no language-neutral resolver exists yet, raise generalising it in the PR - STEP 3: -Werror=switch only covers switch tables; restore the grep step for lists and masks the compiler cannot see - Clojure is not in this tree; say "if added" - ingest_elixir.h is "one reason" the keyword table lives in C++ - STEP 4: maintainers assign the parser version number in the PR Co-Authored-By: Claude Sonnet 5 --- CONTRIBUTING.md | 9 +++++---- prompts/add-a-language.md | 12 ++++++++---- 2 files changed, 13 insertions(+), 8 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 5bf9d216c..1eb0dabc4 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -480,13 +480,14 @@ rules it enforces: - **Data, not branches.** A `kLangTable` row, a `queries//tags.scm` in the shared capture vocabulary, and a row in each exhaustive `switch( Lang )` table. No language-named capture kinds. -- **Reuse the shared mechanism.** Scope walks, shadowing (`VarDecl` bindings with spans), imports - (`Binding`/`Include`) and qualified-name resolution each exist once. Extend them; a second copy under a - new language's name is a review finding, and `--clones` shows it. +- **Reuse the shared mechanism.** Scope walks, shadowing (`VarDecl` bindings with spans) and imports + (`Binding`/`Include`) each exist once. Extend them; a second copy under a new language's name is a + review finding, and `--clones` shows it. No language-neutral qualified-call resolver exists yet + (`src/elixir_resolve.h` is Elixir-specific); raise generalising it in your PR. - **Refuse, don't guess.** A call whose qualifier (`Mod.f`, `ns/f`) cannot be resolved is counted as unresolved, never laddered to every definition of that short name. - **Known gap.** Tags-pass predicates (`#eq?`/`#any-of?`) do not run yet, so keyword-headed definitions - (Elixir, Clojure) keep a small keyword table in C++. That table belongs in `tags.scm` once the + (Elixir today; a Lisp-style `defn` if one is added) keep a small keyword table in C++. That table belongs in `tags.scm` once the mechanism lands; it is not a pattern to extend. ### Interfaces diff --git a/prompts/add-a-language.md b/prompts/add-a-language.md index 95799a8eb..74fda434f 100644 --- a/prompts/add-a-language.md +++ b/prompts/add-a-language.md @@ -69,6 +69,10 @@ common: twenty languages each carrying a private copy of the same idea is twenty (`@definition.LANG`, `@LANG.head`); every consumer downstream has to learn a new name for it. - The exhaustive `switch( Lang )` tables (`langTag`, `isCodeLang`, `dependencyCapable`, …) — the compiler lists every one you missed (`-Werror=switch`). A row each, no logic. +- Then the arms the compiler cannot see: `grep` for a recently added language (`Lang::Elixir`) and read + every hit. Lists and masks are not switches — e.g. `kAllCatalogLangs` in `src/lintcatalog.h`, the + `langBit` masks in `src/clones.h`, `wordLang` in `src/ingest_metrics.h`. Semantic hooks (arity, + params, attributes) live today in the per-language branches of `src/ingest_sidecap.h`. - Per-language C++ is for semantics the grammar cannot express. **Before writing a helper, look for the shared one:** @@ -77,7 +81,7 @@ common: twenty languages each carrying a private copy of the same idea is twenty | the scope a definition sits in | `enclosingScopeOf` (`src/ingest_names.h`) | | a local that shadows a global name (`let`, params) | a `VarDecl` binding with its block span — the generic shadow pass (`suppressShadowedReferences`) does the rest | | imports / aliases / `require … :as` | `Binding` / `Include` facts (`src/model.h`), as JS imports and Elixir aliases already do | - | module-qualified calls (`Mod.f`, `ns/f`) | extend the qualified-name resolver in `src/elixir_resolve.h`; do not write a `LANG_resolve.h` beside it | + | module-qualified calls (`Mod.f`, `ns/f`) | no language-neutral resolver exists yet; `src/elixir_resolve.h` is Elixir-specific and the only one. Raise generalising it in your PR rather than writing a `LANG_resolve.h` beside it | Then run `ripwire --clones` on your tree: your new code should add no clone group. - **A qualified call you cannot resolve is counted as unresolved — never sent to the global name ladder.** @@ -86,8 +90,8 @@ common: twenty languages each carrying a private copy of the same idea is twenty **A known gap, so you are not surprised by it:** the tags pass does not evaluate `#eq?`/`#any-of?` predicates yet (see the notes in `queries/javascript/tags.scm`). A language whose definitions are *a call -with a keyword head* — Elixir `def`, Clojure `defn` — therefore has to keep that keyword list in C++ for -now. That is the reason `src/ingest_elixir.h` exists; it is **not a template to copy**. If you hit this, keep +with a keyword head* — Elixir `def`, or a Lisp-style grammar (e.g. Clojure's `defn`, if added) — therefore has to keep that keyword list in C++ for +now. That is one reason `src/ingest_elixir.h` exists (it also holds arity, params, attribute and alias logic); it is **not a template to copy**. If you hit this, keep the list a small `constexpr` table in one place, and say in the PR that it belongs in `tags.scm` once predicates run. Maintainers will move it; you are not asked to build the mechanism. @@ -110,7 +114,7 @@ totally unrelated bug. Bump on **any** grammar, `.scm`, or extraction change — it, and a stale cache serving old symbols for new code is the worst failure this tool has. **Do not take the next integer blindly.** Parallel pull requests each need their own number, and the -maintainers keep a reservation list. Ask in your PR which number is yours. +maintainers assign you a parser version number in the PR. Then regenerate `test/qschemetrip.hash` and update `test/printf_parity.manifest`.