diff --git a/.changeset/custom-syntax-grammars.md b/.changeset/custom-syntax-grammars.md new file mode 100644 index 000000000..3928ef86a --- /dev/null +++ b/.changeset/custom-syntax-grammars.md @@ -0,0 +1,5 @@ +--- +"hunkdiff": minor +--- + +Let extensions register bounded data-only TextMate grammars for custom file languages. diff --git a/bun.lock b/bun.lock index 333ce24c8..c81dfb585 100644 --- a/bun.lock +++ b/bun.lock @@ -34,6 +34,7 @@ "oxfmt": "^0.41.0", "oxlint": "^1.56.0", "react": "^19.2.4", + "shiki": "3.23.0", "simple-git-hooks": "^2.13.1", "tuistory": "^0.11.0", "typescript": "^5.9.3", diff --git a/docs/extension-architecture.md b/docs/extension-architecture.md index e386a2ff4..227d547f7 100644 --- a/docs/extension-architecture.md +++ b/docs/extension-architecture.md @@ -43,15 +43,20 @@ load issue and costs only that extension. The rules themselves are stated in ## One registry, one apply path -Registrations (session behavior, themes, file languages, VCS adapters, +Registrations (session behavior, themes, file languages, syntax grammars, VCS adapters, changeset transforms, panes, interactive commands, top-level CLI commands, lifecycle/UI events, and inter-extension bus listeners) collect into one `ExtensionRegistry` (`src/extensions/types.ts`) and are resolved/applied through `src/extensions/apply.ts` on both startup and reload. File-language registrations stay as declarative extension, filename, or glob selectors until `fileLanguageLookup.ts` resolves them; Hunk then pins that answer into Pierre's metadata so rendering cannot re-derive a conflicting -language. A live reload replaces the compiled selector generation while preparing its changeset -and restores the previous generation if any pre-commit step fails. Staged external-VCS bootstrap +language. Custom syntax grammars are bounded, deeply frozen TextMate data rather than retained +extension loaders. `core/changeset/syntaxGrammar.ts` owns their generation and digest; the UI sends +that snapshot through the versioned highlight-worker configure handshake before matching jobs. +Grammar changes replace the worker and participate in rendered-result cache identity. Custom regexes +never run on the terminal thread, so worker failure or compiled-Windows worker unavailability falls +back to plaintext without poisoning bundled languages. A live reload replaces the selector and +grammar generations while preparing its changeset and restores both if any pre-commit step fails. Staged external-VCS bootstrap retains the provisional candidate/config snapshot: a final pass that only appends repo candidates extends the same registry, while a changed prefix receives bounded `shutdown` before being rebuilt. Live registry replacement uses diff --git a/docs/extensions.md b/docs/extensions.md index f80dff034..0e0ac9355 100644 --- a/docs/extensions.md +++ b/docs/extensions.md @@ -280,8 +280,9 @@ new instances and run that shutdown/startup pair around the replacement. ### `hunk.apiVersion` -The API generation this Hunk speaks (currently `16`). Branch on it if you want -one file to support several Hunk versions. Version 16 adds pane-wide +The API generation this Hunk speaks (currently `17`). Branch on it if you want +one file to support several Hunk versions. Version 17 adds bounded data-only custom TextMate +syntax grammars; version 16 added pane-wide `onActivate`; version 15 added `{ side, line }` to opted-in pane `currentLine` paint; version 14 added structured `rangeEndpoints` to two-revision VCS diff requests; version 13 added saved-note parent identities and @@ -428,8 +429,41 @@ filenames take precedence over globs, which take precedence over extensions. The extension wins, and later registrations win ties within each category. Direct attempts to register those two reserved extensions are skipped with a notice. -This API selects a language already available to Pierre/Shiki. It does not load a new syntax -grammar; an unknown language remains plain text. +This API selects a language already available to Pierre/Shiki or registered by the same load pass. +An unknown language remains plain text. + +### `hunk.registerSyntaxGrammar(grammar)` + +Register a data-only TextMate grammar, then map files to its language id separately: + +```ts +hunk.registerSyntaxGrammar({ + id: "mydsl", + scopeName: "source.mydsl", + patterns: [ + { match: "\\b(component|contract)\\b", name: "keyword.control.mydsl" }, + { include: "#strings" }, + ], + repository: { + strings: { begin: '"', end: '"', name: "string.quoted.double.mydsl" }, + }, +}); +hunk.registerFileLanguage(".mydsl", "mydsl"); +``` + +API v17 accepts a closed serializable subset: `match`, `begin`/`end`/`while`, captures, nested +`patterns`, and local repository includes. Includes may use only `#name`, `$self`, or `$base`. +External includes, embedded languages, injections, loaders, functions, unknown keys, and bundled +language ids are refused. Hunk bounds serialized bytes, nesting depth, node count, and individual +strings, then deeply copies and freezes accepted data. + +Custom regexes run only in the killable highlight worker. A timeout or grammar failure renders that +file as plaintext and does not poison bundled languages. Hunk keeps custom grammars plaintext when +worker offload is unavailable, including compiled Windows builds. Reloading atomically replaces the +complete grammar generation and invalidates worker and rendered-result caches; the first extension +to claim an id wins, and retired registries retain no executable loader authority. + +See `examples/extensions/archlang-syntax/` for a complete grammar and `.arch` mapping. ### `hunk.registerVcsAdapter(adapter)` diff --git a/examples/extensions/archlang-syntax/README.md b/examples/extensions/archlang-syntax/README.md new file mode 100644 index 000000000..33f077077 --- /dev/null +++ b/examples/extensions/archlang-syntax/README.md @@ -0,0 +1,11 @@ +# Archlang syntax grammar example + +This folder extension adds a bounded, data-only TextMate grammar for `.arch` files. Run it against a +repository containing Archlang input: + +```bash +hunk --extension ./examples/extensions/archlang-syntax diff +``` + +The grammar deliberately uses only local repository includes. Hunk validates, copies, freezes, and +sends the data to its killable highlight worker; it never retains an extension loader callback. diff --git a/examples/extensions/archlang-syntax/index.ts b/examples/extensions/archlang-syntax/index.ts new file mode 100644 index 000000000..9cffd11a4 --- /dev/null +++ b/examples/extensions/archlang-syntax/index.ts @@ -0,0 +1,40 @@ +import type { HunkExtensionAPI } from "hunkdiff/extension"; + +/** Register a compact Archlang TextMate grammar through API v17. */ +export default function archlangSyntax(hunk: HunkExtensionAPI) { + hunk.registerSyntaxGrammar({ + id: "archlang", + scopeName: "source.archlang", + patterns: [ + { include: "#comments" }, + { include: "#strings" }, + { include: "#keywords" }, + { include: "#operators" }, + { include: "#numbers" }, + { include: "#variables" }, + ], + repository: { + comments: { match: "//.*$", name: "comment.line.double-slash.archlang" }, + strings: { + begin: '"', + end: '"', + name: "string.quoted.double.archlang", + patterns: [ + { + match: '\\\\(?:[\\\\"nrt]|u[0-9a-fA-F]{4})', + name: "constant.character.escape.archlang", + }, + ], + }, + keywords: { + match: + "\\b(?:arch|system|let|source|component|contract|type|record|tagged|operation|async|throws|bind|using|process|in_process|via|edges|rule|paths|components|excluding|cycles|forbidden|property|on|claim|verify|static|interface|probe|advisory|existing|version|compatibility|observed|constructed|provides|requires|roots|exclude|entrypoints)\\b", + name: "keyword.control.archlang", + }, + operators: { match: "-/>|->|[:=$]", name: "keyword.operator.archlang" }, + numbers: { match: "\\b\\d+(?:_\\d+)*\\b", name: "constant.numeric.archlang" }, + variables: { match: "\\$[a-z][a-z0-9_]*", name: "variable.other.archlang" }, + }, + }); + hunk.registerFileLanguage(".arch", "archlang"); +} diff --git a/examples/extensions/archlang-syntax/package.json b/examples/extensions/archlang-syntax/package.json new file mode 100644 index 000000000..29643fe98 --- /dev/null +++ b/examples/extensions/archlang-syntax/package.json @@ -0,0 +1,12 @@ +{ + "name": "hunk-archlang-syntax-example", + "version": "0.0.0", + "private": true, + "description": "Example data-only Archlang syntax grammar for Hunk", + "hunk": { + "apiVersion": 17, + "extensions": [ + "./index.ts" + ] + } +} diff --git a/package.json b/package.json index 7c0f453aa..b4be6d60a 100644 --- a/package.json +++ b/package.json @@ -61,6 +61,7 @@ "install:bin": "bun run ./scripts/install-bin.ts", "generate:skill": "bun run ./scripts/generate-skill.ts", "generate:theme-colors": "bun run ./scripts/generate-theme-diff-colors.ts", + "generate:syntax-languages": "bun run ./scripts/generate-bundled-syntax-languages.ts", "generate:docs": "bun run ./scripts/generate-docs.ts", "generate:changelog": "bun run ./scripts/generate-changelog.ts", "generate:og": "bun run ./website/scripts/generate-og.ts", @@ -151,6 +152,7 @@ "oxfmt": "^0.41.0", "oxlint": "^1.56.0", "react": "^19.2.4", + "shiki": "3.23.0", "simple-git-hooks": "^2.13.1", "tuistory": "^0.11.0", "typescript": "^5.9.3" diff --git a/scripts/check-pack.ts b/scripts/check-pack.ts index 0edcf5d9c..e9709da04 100644 --- a/scripts/check-pack.ts +++ b/scripts/check-pack.ts @@ -39,6 +39,7 @@ import type { ExtensionPaneSize, ExtensionReviewSelection, ExtensionSessionOptions, + ExtensionSyntaxGrammar, ExtensionVerticalPane, ExtensionVcsAdapter, ExtensionVcsDiffInput, @@ -86,6 +87,12 @@ export default function (hunk: HunkExtensionAPI) { target: "path", }; hunk.registerFileLanguage(generatedTypeScript, "typescript"); + const grammar: ExtensionSyntaxGrammar = { + id: "mydsl", + scopeName: "source.mydsl", + patterns: [{ match: "component", name: "keyword.control.mydsl" }], + }; + hunk.registerSyntaxGrammar(grammar); const pane = (props: ExtensionPaneProps) => { hunk.log(\`\${props.placement}:\${props.width}x\${props.height}\`); diff --git a/scripts/generate-bundled-syntax-languages.ts b/scripts/generate-bundled-syntax-languages.ts new file mode 100644 index 000000000..22f207322 --- /dev/null +++ b/scripts/generate-bundled-syntax-languages.ts @@ -0,0 +1,21 @@ +#!/usr/bin/env bun + +import { writeFileSync } from "node:fs"; +import { resolve } from "node:path"; +import { bundledLanguages } from "shiki"; + +/** Render the checked-in collision list in formatter-stable TypeScript. */ +export function renderBundledSyntaxLanguages(): string { + const ids = Object.keys(bundledLanguages).sort(); + const entries = ids.map((id) => ` ${JSON.stringify(id)},`).join("\n"); + return `/** Language ids bundled by the pinned Pierre release; run bun generate:syntax-languages. */ +export const BUNDLED_SYNTAX_LANGUAGE_IDS: ReadonlySet = new Set([\n${entries}\n]); +`; +} + +if (import.meta.main) { + writeFileSync( + resolve(import.meta.dir, "../src/core/changeset/bundledSyntaxLanguages.generated.ts"), + renderBundledSyntaxLanguages(), + ); +} diff --git a/skills/hunk-extensions/SKILL.md b/skills/hunk-extensions/SKILL.md index 9bb72282d..77f4f192f 100644 --- a/skills/hunk-extensions/SKILL.md +++ b/skills/hunk-extensions/SKILL.md @@ -98,6 +98,7 @@ bad or duplicate id is skipped with a startup notice. | Keep demo/training view settings temporary | `hunk.configureSession(options)` | | Add a selectable color theme | `hunk.registerTheme(theme)` | | Highlight an extension, exact filename, or filename glob | `hunk.registerFileLanguage(matcher, lang)` | +| Add a bounded data-only TextMate syntax grammar | `hunk.registerSyntaxGrammar(grammar)` | | Support another VCS (`git`/`jj`/`sl` are reserved) | `hunk.registerVcsAdapter(adapter)` | | Add a navigation/list/status pane beside the review | `hunk.registerPane(pane)` | | Present a file as something other than a raw diff | `hunk.registerFileView(view)` (experimental) | @@ -110,7 +111,7 @@ bad or duplicate id is skipped with a startup notice. | Coordinate with another loaded extension | `hunk.events.emit` / `hunk.events.on` | | Read user-supplied settings | `hunk.config` (`[extension.]` table) | | Snapshot stable files and every saved review note | `ctx.review.snapshot()` in a command | -| Branch on the API generation (currently `15`) | `hunk.apiVersion` | +| Branch on the API generation (currently `17`) | `hunk.apiVersion` | Registration is only valid while the factory runs — Hunk seals the API object afterwards. @@ -190,6 +191,10 @@ through untouched. Most extension bugs are one of these: +- **Custom syntax grammars are data, not loaders.** Register the grammar during the factory, map + files separately with `registerFileLanguage`, and use only local includes. Hunk bounds and freezes + the grammar, runs its regexes in the killable worker, and leaves custom languages plain when worker + offload is unavailable. - **Registering a surface does not show it.** Panes need `defaultOpen`, `replaces: "hunk:files"`, or a command that opens them. File views remain raw until selected from the **View** menu. diff --git a/src/app/sessionBootstrap.test.ts b/src/app/sessionBootstrap.test.ts index cad35584f..cfe8bf6f8 100644 --- a/src/app/sessionBootstrap.test.ts +++ b/src/app/sessionBootstrap.test.ts @@ -1,6 +1,10 @@ import { describe, expect, test } from "bun:test"; import { fileLanguageForPath } from "../core/changeset/fileLanguageLookup"; import { replaceExtensionFileLanguages } from "../core/changeset/fileLanguage"; +import { + replaceExtensionSyntaxGrammars, + syntaxGrammarSnapshot, +} from "../core/changeset/syntaxGrammar"; import type { HunkConfigResolution } from "../core/run/config"; import type { AppBootstrap } from "../core/bootstrap"; import type { CliInput } from "../core/run/commandInputs"; @@ -58,15 +62,33 @@ describe("loadConfiguredSessionBootstrap", () => { expect(result.bootstrap.viewPreferencesConfigPath).toBe("/tmp/hunk-config.toml"); }); - test("restores the active file-language selectors when bootstrap loading fails", async () => { + test("restores file-language and syntax-grammar generations when bootstrap loading fails", async () => { replaceExtensionFileLanguages([ { matcher: { kind: "filename", value: "CurrentHunkfile" }, language: "python", }, ]); + replaceExtensionSyntaxGrammars([ + { + extensionId: "current", + grammar: Object.freeze({ + id: "current", + scopeName: "source.current", + patterns: Object.freeze([{ match: "current" }]), + }), + }, + ]); const input = createTestInput(); const extensions = createEmptyExtensionLoadResult(); + extensions.registry.syntaxGrammars.push({ + extensionId: "replacement", + grammar: Object.freeze({ + id: "replacement", + scopeName: "source.replacement", + patterns: Object.freeze([{ match: "replacement" }]), + }), + }); extensions.registry.fileLanguages.push({ extensionId: "replacement", matcher: { kind: "filename", value: "ReplacementHunkfile" }, @@ -80,6 +102,7 @@ describe("loadConfiguredSessionBootstrap", () => { extensions, loadAppBootstrapImpl: async () => { expect(fileLanguageForPath("ReplacementHunkfile")).toBe("ruby"); + expect(syntaxGrammarSnapshot().grammars.map(({ id }) => id)).toEqual(["replacement"]); throw new Error("load failed"); }, }), @@ -87,6 +110,8 @@ describe("loadConfiguredSessionBootstrap", () => { expect(fileLanguageForPath("CurrentHunkfile")).toBe("python"); expect(fileLanguageForPath("ReplacementHunkfile")).toBe("text"); + expect(syntaxGrammarSnapshot().grammars.map(({ id }) => id)).toEqual(["current"]); replaceExtensionFileLanguages([]); + replaceExtensionSyntaxGrammars([]); }); }); diff --git a/src/app/sessionBootstrap.ts b/src/app/sessionBootstrap.ts index eb36b7ed7..58ea870c5 100644 --- a/src/app/sessionBootstrap.ts +++ b/src/app/sessionBootstrap.ts @@ -3,6 +3,11 @@ import { restoreFileLanguageRegistrations, type FileLanguageRegistrationSnapshot, } from "../core/changeset/fileLanguage"; +import { + restoreSyntaxGrammars, + syntaxGrammarSnapshot, + type SyntaxGrammarSnapshot, +} from "../core/changeset/syntaxGrammar"; import type { HunkConfigResolution } from "../core/run/config"; import { isVcsReviewInput } from "../core/vcs"; import type { VcsCatalog } from "../core/vcs/types"; @@ -37,6 +42,8 @@ export interface SessionBootstrapResult { bootstrap: AppBootstrap; /** Selector set to restore if a live reload fails before its commit gate. */ previousFileLanguages: FileLanguageRegistrationSnapshot; + /** Grammar set to restore if a live reload fails before its commit gate. */ + previousSyntaxGrammars: SyntaxGrammarSnapshot; input: CliInput; sessionThemes: ReturnType; sessionVcs: ReturnType; @@ -59,6 +66,7 @@ export async function loadConfiguredSessionBootstrap({ baseVcsCatalog = getBundledVcsCatalog(), }: SessionBootstrapOptions): Promise { const previousFileLanguages = fileLanguageRegistrationSnapshot(); + const previousSyntaxGrammars = syntaxGrammarSnapshot(); try { const sessionThemes = collectSessionCustomThemes( @@ -91,9 +99,18 @@ export async function loadConfiguredSessionBootstrap({ bootstrap.viewPreferencesConfigPath = configured.viewPreferencesConfigPath; bootstrap.keybindings = configured.keybindings; - return { applied, bootstrap, input, previousFileLanguages, sessionThemes, sessionVcs }; + return { + applied, + bootstrap, + input, + previousFileLanguages, + previousSyntaxGrammars, + sessionThemes, + sessionVcs, + }; } catch (error) { restoreFileLanguageRegistrations(previousFileLanguages); + restoreSyntaxGrammars(previousSyntaxGrammars); throw error; } } diff --git a/src/core/changeset/bundledSyntaxLanguages.generated.ts b/src/core/changeset/bundledSyntaxLanguages.generated.ts new file mode 100644 index 000000000..a2fd364d2 --- /dev/null +++ b/src/core/changeset/bundledSyntaxLanguages.generated.ts @@ -0,0 +1,335 @@ +/** Language ids bundled by the pinned Pierre release; run bun generate:syntax-languages. */ +export const BUNDLED_SYNTAX_LANGUAGE_IDS: ReadonlySet = new Set([ + "1c", + "1c-query", + "abap", + "actionscript-3", + "ada", + "adoc", + "angular-html", + "angular-ts", + "apache", + "apex", + "apl", + "applescript", + "ara", + "asciidoc", + "asm", + "astro", + "awk", + "ballerina", + "bash", + "bat", + "batch", + "be", + "beancount", + "berry", + "bibtex", + "bicep", + "bird", + "bird2", + "blade", + "bsl", + "c", + "c#", + "c++", + "c3", + "cadence", + "cairo", + "cdc", + "cjs", + "clarity", + "clj", + "clojure", + "closure-templates", + "cmake", + "cmd", + "cobol", + "codeowners", + "codeql", + "coffee", + "coffeescript", + "common-lisp", + "console", + "coq", + "cpp", + "cql", + "crystal", + "cs", + "csharp", + "css", + "csv", + "cts", + "cue", + "cypher", + "d", + "dart", + "dax", + "desktop", + "diff", + "docker", + "dockerfile", + "dotenv", + "dream-maker", + "edge", + "elisp", + "elixir", + "elm", + "emacs-lisp", + "erb", + "erl", + "erlang", + "f", + "f#", + "f03", + "f08", + "f18", + "f77", + "f90", + "f95", + "fennel", + "fish", + "fluent", + "for", + "fortran-fixed-form", + "fortran-free-form", + "fs", + "fsharp", + "fsl", + "ftl", + "gd", + "gdresource", + "gdscript", + "gdshader", + "genie", + "gherkin", + "git-commit", + "git-rebase", + "gjs", + "gleam", + "glimmer-js", + "glimmer-ts", + "glsl", + "gn", + "gnuplot", + "go", + "gql", + "graphql", + "groovy", + "gts", + "hack", + "haml", + "handlebars", + "haskell", + "haxe", + "hbs", + "hcl", + "hjson", + "hlsl", + "hs", + "html", + "html-derivative", + "http", + "hurl", + "hxml", + "hy", + "imba", + "ini", + "jade", + "java", + "javascript", + "jinja", + "jison", + "jl", + "js", + "json", + "json5", + "jsonc", + "jsonl", + "jsonnet", + "jssm", + "jsx", + "julia", + "just", + "kdl", + "kotlin", + "kql", + "kt", + "kts", + "kusto", + "latex", + "lean", + "lean4", + "less", + "liquid", + "lisp", + "lit", + "llvm", + "log", + "logo", + "lua", + "luau", + "make", + "makefile", + "markdown", + "marko", + "matlab", + "mbt", + "mbti", + "md", + "mdc", + "mdx", + "mediawiki", + "mermaid", + "mips", + "mipsasm", + "mjs", + "mmd", + "mojo", + "moonbit", + "move", + "mts", + "nar", + "narrat", + "nextflow", + "nextflow-groovy", + "nf", + "nginx", + "nim", + "nix", + "nu", + "nushell", + "objc", + "objective-c", + "objective-cpp", + "ocaml", + "odin", + "openscad", + "pascal", + "perl", + "perl6", + "php", + "pkl", + "plsql", + "po", + "polar", + "postcss", + "pot", + "potx", + "powerquery", + "powershell", + "prisma", + "prolog", + "properties", + "proto", + "protobuf", + "ps", + "ps1", + "pug", + "puppet", + "purescript", + "py", + "python", + "ql", + "qml", + "qmldir", + "qss", + "r", + "racket", + "raku", + "razor", + "rb", + "reg", + "regex", + "regexp", + "rel", + "riscv", + "ron", + "rosmsg", + "rs", + "rst", + "ruby", + "rust", + "sas", + "sass", + "scad", + "scala", + "scheme", + "scss", + "sdbl", + "sh", + "shader", + "shaderlab", + "shell", + "shellscript", + "shellsession", + "smalltalk", + "solidity", + "soy", + "sparql", + "spl", + "splunk", + "sql", + "ssh-config", + "stata", + "styl", + "stylus", + "surql", + "surrealql", + "svelte", + "swift", + "system-verilog", + "systemd", + "talon", + "talonscript", + "tasl", + "tcl", + "templ", + "terraform", + "tex", + "tf", + "tfvars", + "toml", + "tres", + "ts", + "ts-tags", + "tscn", + "tsp", + "tsv", + "tsx", + "turtle", + "twig", + "typ", + "typescript", + "typespec", + "typst", + "v", + "vala", + "vb", + "verilog", + "vhdl", + "vim", + "viml", + "vimscript", + "vue", + "vue-html", + "vue-vine", + "vy", + "vyper", + "wasm", + "wenyan", + "wgsl", + "wiki", + "wikitext", + "wit", + "wl", + "wolfram", + "xml", + "xsl", + "yaml", + "yml", + "zenscript", + "zig", + "zsh", + "文言", +]); diff --git a/src/core/changeset/syntaxGrammar.test.ts b/src/core/changeset/syntaxGrammar.test.ts new file mode 100644 index 000000000..a5ce21431 --- /dev/null +++ b/src/core/changeset/syntaxGrammar.test.ts @@ -0,0 +1,41 @@ +import { readFileSync } from "node:fs"; +import { resolve } from "node:path"; +import { describe, expect, test } from "bun:test"; +import { bundledLanguages } from "shiki"; +import { renderBundledSyntaxLanguages } from "../../../scripts/generate-bundled-syntax-languages"; +import { BUNDLED_SYNTAX_LANGUAGE_IDS } from "./bundledSyntaxLanguages.generated"; +import { + replaceExtensionSyntaxGrammars, + restoreSyntaxGrammars, + syntaxGrammarSnapshot, +} from "./syntaxGrammar"; + +describe("syntax grammar registry", () => { + test("keeps the generated bundled-id collision list current and formatter-stable", () => { + expect([...BUNDLED_SYNTAX_LANGUAGE_IDS].sort()).toEqual(Object.keys(bundledLanguages).sort()); + expect( + readFileSync(resolve(import.meta.dir, "bundledSyntaxLanguages.generated.ts"), "utf8"), + ).toBe(renderBundledSyntaxLanguages()); + }); + + test("changes digest only when normalized grammar bytes change", () => { + replaceExtensionSyntaxGrammars([]); + const empty = syntaxGrammarSnapshot(); + const registration = { + extensionId: "demo", + grammar: Object.freeze({ + id: "demo", + scopeName: "source.demo", + patterns: Object.freeze([{ match: "demo", name: "keyword.demo" }]), + }), + }; + const first = replaceExtensionSyntaxGrammars([registration]); + const same = replaceExtensionSyntaxGrammars([registration]); + expect(same).toBe(first); + expect(first.digest).not.toBe(empty.digest); + + restoreSyntaxGrammars(empty); + expect(syntaxGrammarSnapshot().digest).toBe(empty.digest); + expect(syntaxGrammarSnapshot().generation).toBeGreaterThan(first.generation); + }); +}); diff --git a/src/core/changeset/syntaxGrammar.ts b/src/core/changeset/syntaxGrammar.ts new file mode 100644 index 000000000..aa46fb01c --- /dev/null +++ b/src/core/changeset/syntaxGrammar.ts @@ -0,0 +1,76 @@ +import { createHash } from "node:crypto"; +import type { ExtensionSyntaxGrammar } from "../../extension-api/types"; + +/** One accepted custom grammar attributed to its owning extension. */ +export interface SyntaxGrammarRegistration { + extensionId: string; + grammar: ExtensionSyntaxGrammar; +} + +/** Immutable data handed to main-thread and worker highlighters. */ +export interface SyntaxGrammarSnapshot { + generation: number; + digest: string; + grammars: readonly ExtensionSyntaxGrammar[]; +} + +const EMPTY_DIGEST = createHash("sha256").update("[]").digest("hex"); +let activeSnapshot: SyntaxGrammarSnapshot = Object.freeze({ + generation: 0, + digest: EMPTY_DIGEST, + grammars: Object.freeze([]), +}); +const grammarChangeListeners = new Set<() => void>(); + +/** Subscribe a loaded worker adapter to grammar replacement without importing the UI into core. */ +export function subscribeSyntaxGrammarChanges(listener: () => void) { + grammarChangeListeners.add(listener); + return () => grammarChangeListeners.delete(listener); +} + +/** Notify loaded adapters synchronously so retired grammar data cannot remain executable. */ +function notifyGrammarChange() { + for (const listener of grammarChangeListeners) listener(); +} + +/** Hash normalized grammar data for cache identity and worker configuration. */ +function grammarDigest(grammars: readonly ExtensionSyntaxGrammar[]) { + return createHash("sha256").update(JSON.stringify(grammars)).digest("hex"); +} + +/** Replace all custom grammar data, invalidating consumers only when bytes changed. */ +export function replaceExtensionSyntaxGrammars( + registrations: readonly SyntaxGrammarRegistration[], +): SyntaxGrammarSnapshot { + const grammars = Object.freeze(registrations.map(({ grammar }) => grammar)); + const digest = grammarDigest(grammars); + if (digest === activeSnapshot.digest) return activeSnapshot; + activeSnapshot = Object.freeze({ + generation: activeSnapshot.generation + 1, + digest, + grammars, + }); + notifyGrammarChange(); + return activeSnapshot; +} + +/** Restore grammar data captured before a failed session bootstrap. */ +export function restoreSyntaxGrammars(snapshot: SyntaxGrammarSnapshot): void { + if (snapshot.digest === activeSnapshot.digest) return; + activeSnapshot = Object.freeze({ + generation: activeSnapshot.generation + 1, + digest: snapshot.digest, + grammars: snapshot.grammars, + }); + notifyGrammarChange(); +} + +/** Return the active immutable grammar generation. */ +export function syntaxGrammarSnapshot(): SyntaxGrammarSnapshot { + return activeSnapshot; +} + +/** Return whether a language id currently names a custom grammar. */ +export function isCustomSyntaxLanguage(language: string | undefined): boolean { + return activeSnapshot.grammars.some((grammar) => grammar.id === language); +} diff --git a/src/extension-api/index.ts b/src/extension-api/index.ts index dbc86e2fe..e1fae78a0 100644 --- a/src/extension-api/index.ts +++ b/src/extension-api/index.ts @@ -45,6 +45,9 @@ export type { ExtensionDiffHunk, ExtensionFileChangeRange, ExtensionFileLanguageMatcher, + ExtensionSyntaxGrammar, + ExtensionSyntaxGrammarCapture, + ExtensionSyntaxGrammarRule, ExtensionFileSide, ExtensionFileView, ExtensionFileViewControls, diff --git a/src/extension-api/types.ts b/src/extension-api/types.ts index b162f6944..9d1276373 100644 --- a/src/extension-api/types.ts +++ b/src/extension-api/types.ts @@ -21,7 +21,7 @@ * Extensions can branch on `hunk.apiVersion` so a newer Hunk can keep loading * older extensions without guessing at their expectations. */ -export const HUNK_EXTENSION_API_VERSION = 16; +export const HUNK_EXTENSION_API_VERSION = 17; export type HunkExtensionApiVersion = typeof HUNK_EXTENSION_API_VERSION; export type ExtensionNotifyType = "info" | "warning" | "error"; @@ -36,6 +36,45 @@ export type ExtensionFileLanguageMatcher = readonly target: "basename" | "path"; }; +/** One capture group inside a serializable TextMate grammar rule. */ +export interface ExtensionSyntaxGrammarCapture { + readonly name?: string; + readonly contentName?: string; + readonly patterns?: readonly ExtensionSyntaxGrammarRule[]; +} + +/** + * One rule in the safe, serializable TextMate subset accepted by Hunk. + * + * Includes may reference only this grammar (`#name`, `$self`, or `$base`). External grammars, + * injections, embedded languages, and executable loaders are deliberately outside API v17. + */ +export interface ExtensionSyntaxGrammarRule { + readonly include?: string; + readonly name?: string; + readonly contentName?: string; + readonly match?: string; + readonly begin?: string; + readonly end?: string; + readonly while?: string; + readonly captures?: Readonly>; + readonly beginCaptures?: Readonly>; + readonly endCaptures?: Readonly>; + readonly whileCaptures?: Readonly>; + readonly patterns?: readonly ExtensionSyntaxGrammarRule[]; + readonly applyEndPatternLast?: 0 | 1; +} + +/** A data-only TextMate grammar contributed by one trusted extension. */ +export interface ExtensionSyntaxGrammar { + /** Stable language id passed separately to `registerFileLanguage`. */ + readonly id: string; + /** TextMate scope rooted below `source.` or `text.`. */ + readonly scopeName: string; + readonly patterns: readonly ExtensionSyntaxGrammarRule[]; + readonly repository?: Readonly>; +} + /** Capability object handed to every extension event handler and transform. */ export interface ExtensionContext { cwd: string; @@ -1986,6 +2025,8 @@ export interface HunkExtensionAPI { registerTheme(theme: ExtensionThemeConfig): void; /** Map a file extension, exact filename, or glob to a syntax-highlighting language. */ registerFileLanguage(matcher: string | ExtensionFileLanguageMatcher, language: string): void; + /** Register a bounded, data-only TextMate grammar for later file-language mappings. */ + registerSyntaxGrammar(grammar: ExtensionSyntaxGrammar): void; /** Contribute one additional VCS backend. */ registerVcsAdapter(adapter: ExtensionVcsAdapter): void; /** diff --git a/src/extensions/apply.test.ts b/src/extensions/apply.test.ts index bff56e08e..52dcad887 100644 --- a/src/extensions/apply.test.ts +++ b/src/extensions/apply.test.ts @@ -8,13 +8,16 @@ import { HUNK_DEFAULT_VCS_DETECTION_PRIORITY, } from "../extension-api/types"; import type { Changeset, DiffFile } from "../core/changeset/model"; +import { replaceExtensionSyntaxGrammars } from "../core/changeset/syntaxGrammar"; import { extendVcsCatalog } from "../core/vcs"; import type { VcsAdapter } from "../core/vcs/types"; import { getBundledVcsCatalog } from "../app/vcsCatalog"; import { HUNK_FILES_PANE_KEY } from "./extensionIds"; +import { SYNTAX_GRAMMAR_LIMITS } from "./syntaxGrammars"; import { applyExtensionChangesetTransforms, applyExtensionFileLanguages, + applyExtensionSyntaxGrammars, createExtensionApplyNotices, reportExtensionApplyIssues, createUnknownVcsNotice, @@ -44,6 +47,7 @@ function catalogWith(adapters: readonly VcsAdapter[] = []) { } afterEach(() => { + replaceExtensionSyntaxGrammars([]); for (const dir of tempDirs.splice(0)) { rmSync(dir, { recursive: true, force: true }); } @@ -711,3 +715,60 @@ describe("resolveSessionVcsId", () => { expect(notice.message).not.toContain("\u001b"); }); }); + +describe("extension syntax grammars", () => { + const registered = (extensionId: string, id: string) => ({ + extensionId, + grammar: Object.freeze({ + id, + scopeName: `source.${id}`, + patterns: Object.freeze([{ match: "x", name: `keyword.${id}` }]), + }), + }); + + test("keeps the first owner and refuses bundled language ids", async () => { + const { syntaxGrammarSnapshot } = await import("../core/changeset/syntaxGrammar"); + const registry = createEmptyExtensionLoadResult("/repo").registry; + registry.syntaxGrammars.push( + registered("first", "custom"), + registered("second", "custom"), + registered("shadow", "typescript"), + ); + + const issues = applyExtensionSyntaxGrammars(registry); + expect(syntaxGrammarSnapshot().grammars.map(({ id }) => id)).toEqual(["custom"]); + expect(issues.map(({ extensionId }) => extensionId)).toEqual(["second", "shadow"]); + }); + + test("accepts the exact session limit and attributes only the overflow", async () => { + const { syntaxGrammarSnapshot } = await import("../core/changeset/syntaxGrammar"); + const registry = createEmptyExtensionLoadResult("/repo").registry; + registry.syntaxGrammars.push( + ...Array.from({ length: SYNTAX_GRAMMAR_LIMITS.grammarsPerSession + 1 }, (_, index) => + registered(`owner-${index}`, `custom-${index}`), + ), + ); + + const issues = applyExtensionSyntaxGrammars(registry); + expect(syntaxGrammarSnapshot().grammars).toHaveLength(SYNTAX_GRAMMAR_LIMITS.grammarsPerSession); + expect(issues).toEqual([ + { + extensionId: `owner-${SYNTAX_GRAMMAR_LIMITS.grammarsPerSession}`, + message: `Skipped syntax grammar "custom-${SYNTAX_GRAMMAR_LIMITS.grammarsPerSession}" from extension owner-${SYNTAX_GRAMMAR_LIMITS.grammarsPerSession} • the session grammar limit was reached`, + }, + ]); + }); + + test("atomically removes retired grammar data", async () => { + const { syntaxGrammarSnapshot } = await import("../core/changeset/syntaxGrammar"); + const registry = createEmptyExtensionLoadResult("/repo").registry; + registry.syntaxGrammars.push(registered("temporary", "temporary")); + applyExtensionSyntaxGrammars(registry); + const previous = syntaxGrammarSnapshot(); + expect(previous.grammars).toHaveLength(1); + + applyExtensionSyntaxGrammars(createEmptyExtensionLoadResult("/repo").registry); + expect(syntaxGrammarSnapshot().grammars).toEqual([]); + expect(syntaxGrammarSnapshot().generation).toBeGreaterThan(previous.generation); + }); +}); diff --git a/src/extensions/apply.ts b/src/extensions/apply.ts index 05e79c3e7..53240f209 100644 --- a/src/extensions/apply.ts +++ b/src/extensions/apply.ts @@ -3,11 +3,17 @@ import { replaceExtensionFileLanguages, type FileLanguageRegistration, } from "../core/changeset/fileLanguage"; +import { BUNDLED_SYNTAX_LANGUAGE_IDS } from "../core/changeset/bundledSyntaxLanguages.generated"; +import { + replaceExtensionSyntaxGrammars, + type SyntaxGrammarRegistration, +} from "../core/changeset/syntaxGrammar"; import type { StartupNotice } from "../core/process/startupNotice"; import type { Changeset } from "../core/changeset/model"; import { detectVcs, extendVcsCatalog, getDefaultVcsAdapter } from "../core/vcs"; import type { VcsAdapter, VcsCatalog } from "../core/vcs/types"; import { sanitizeTerminalLine } from "../lib/terminalText"; +import { SYNTAX_GRAMMAR_LIMITS } from "./syntaxGrammars"; import type { ExtensionContext, ExtensionLoadResult, @@ -67,6 +73,43 @@ export function applyExtensionFileLanguages(registry: ExtensionRegistry): Extens return issues; } +/** Apply data-only custom grammars with deterministic first-owner-wins semantics. */ +export function applyExtensionSyntaxGrammars(registry: ExtensionRegistry): ExtensionApplyIssue[] { + const issues: ExtensionApplyIssue[] = []; + const registrations: SyntaxGrammarRegistration[] = []; + const claimed = new Set(); + + for (const registered of registry.syntaxGrammars) { + const { id } = registered.grammar; + if (BUNDLED_SYNTAX_LANGUAGE_IDS.has(id) || id === "text" || id === "ansi") { + issues.push({ + extensionId: registered.extensionId, + message: `Skipped syntax grammar "${id}" from extension ${registered.extensionId} • Pierre bundles or reserves that language id`, + }); + continue; + } + if (claimed.has(id)) { + issues.push({ + extensionId: registered.extensionId, + message: `Skipped syntax grammar "${id}" from extension ${registered.extensionId} • another extension already registered it`, + }); + continue; + } + if (registrations.length >= SYNTAX_GRAMMAR_LIMITS.grammarsPerSession) { + issues.push({ + extensionId: registered.extensionId, + message: `Skipped syntax grammar "${id}" from extension ${registered.extensionId} • the session grammar limit was reached`, + }); + continue; + } + claimed.add(id); + registrations.push(registered); + } + + replaceExtensionSyntaxGrammars(registrations); + return issues; +} + /** Extension VCS adapters that may join detection and lookup, plus the ones skipped. */ export interface ResolvedExtensionVcsAdapters { adapters: VcsAdapter[]; @@ -355,9 +398,11 @@ export function applyExtensionRegistrations( ): AppliedExtensionRegistrations { if (!result) { replaceExtensionFileLanguages([]); + replaceExtensionSyntaxGrammars([]); return { vcsAdapters: [], vcsCatalog: baseCatalog, issues: [] }; } + const grammarIssues = applyExtensionSyntaxGrammars(result.registry); const languageIssues = applyExtensionFileLanguages(result.registry); const vcs = resolveExtensionVcsAdapters(result.registry, baseCatalog); // Resolved again where the UI consumes them; consulted here so skipped @@ -372,6 +417,7 @@ export function applyExtensionRegistrations( vcsAdapters: vcs.adapters, vcsCatalog: extendVcsCatalog(baseCatalog, vcs.adapters), issues: [ + ...grammarIssues, ...languageIssues, ...vcs.issues, ...panes.issues, diff --git a/src/extensions/runExtension.test.ts b/src/extensions/runExtension.test.ts index 86d546818..68231a7d9 100644 --- a/src/extensions/runExtension.test.ts +++ b/src/extensions/runExtension.test.ts @@ -1,6 +1,7 @@ import { describe, expect, test } from "bun:test"; import { resolveExtensionPanes } from "./apply"; import { runExtensionFactory, toInternalVcsAdapter } from "./runExtension"; +import { normalizeSyntaxGrammar, SYNTAX_GRAMMAR_LIMITS } from "./syntaxGrammars"; import { createEmptyExtensionRegistry, HUNK_EXTENSION_API_VERSION, @@ -947,3 +948,135 @@ describe("toInternalVcsAdapter detection ids", () => { ]); }); }); + +describe("registerSyntaxGrammar", () => { + const grammar = () => ({ + id: "archlang", + scopeName: "source.archlang", + patterns: [ + { match: "\\b(component|contract|property)\\b", name: "keyword.control.archlang" }, + { include: "#strings" }, + ], + repository: { + strings: { begin: '"', end: '"', name: "string.quoted.double.archlang" }, + }, + }); + + test("deeply copies and freezes bounded data registrations", () => { + const registry = createEmptyExtensionRegistry(); + const source = grammar(); + runExtensionFactory({ + metadata: bundledMetadata("archlang"), + registry, + issues: [], + factory: (hunk) => hunk.registerSyntaxGrammar(source), + }); + + source.patterns[0]!.name = "changed"; + const stored = registry.syntaxGrammars[0]!.grammar; + expect(stored.patterns[0]?.name).toBe("keyword.control.archlang"); + expect(Object.isFrozen(stored)).toBe(true); + expect(Object.isFrozen(stored.patterns)).toBe(true); + expect(Object.isFrozen(stored.patterns[0])).toBe(true); + }); + + test("rejects unsafe grammar features, external includes, and excessive input", () => { + const registry = createEmptyExtensionRegistry(); + const failures: string[] = []; + for (const candidate of [ + { ...grammar(), injections: {} }, + { ...grammar(), patterns: [{ include: "source.typescript" }] }, + { ...grammar(), patterns: [{ match: "x".repeat(40_000) }] }, + { ...grammar(), id: "text" }, + ]) { + runExtensionFactory({ + metadata: bundledMetadata(`bad-${failures.length}`), + registry, + issues: [], + factory: (hunk) => { + try { + hunk.registerSyntaxGrammar(candidate as never); + } catch (error) { + failures.push(String(error)); + } + }, + }); + } + expect(failures).toHaveLength(4); + expect(failures.join("\n")).toContain("does not support grammar.injections"); + expect(failures.join("\n")).toContain("#local"); + expect(failures.join("\n")).toContain("string-size limit"); + expect(registry.syntaxGrammars).toEqual([]); + }); + + test("requires local includes to name own repository rules", () => { + for (const inheritedName of ["constructor", "toString", "hasOwnProperty"]) { + expect(() => + normalizeSyntaxGrammar({ + id: "prototype-check", + scopeName: "source.prototype-check", + patterns: [{ include: `#${inheritedName}` }], + }), + ).toThrow(`references missing local rule #${inheritedName}`); + } + + expect(() => + normalizeSyntaxGrammar({ + id: "owned-constructor", + scopeName: "source.owned-constructor", + patterns: [{ include: "#constructor" }], + repository: { constructor: { match: "constructor" } }, + }), + ).not.toThrow(); + }); + + test("accepts exact depth and node limits and rejects the next grammar node", () => { + const nestedRule = (depth: number): Record => + depth === 1 ? { match: "x" } : { patterns: [nestedRule(depth - 1)] }; + const grammarWithPatterns = (patterns: Record[]) => ({ + id: "boundary", + scopeName: "source.boundary", + patterns, + }); + + expect(() => + normalizeSyntaxGrammar(grammarWithPatterns([nestedRule(SYNTAX_GRAMMAR_LIMITS.depth)])), + ).not.toThrow(); + expect(() => + normalizeSyntaxGrammar(grammarWithPatterns([nestedRule(SYNTAX_GRAMMAR_LIMITS.depth + 1)])), + ).toThrow("grammar-depth limit"); + + const allowedRules = Array.from({ length: SYNTAX_GRAMMAR_LIMITS.nodes - 1 }, () => ({ + match: "x", + })); + expect(() => normalizeSyntaxGrammar(grammarWithPatterns(allowedRules))).not.toThrow(); + expect(() => + normalizeSyntaxGrammar(grammarWithPatterns([...allowedRules, { match: "x" }])), + ).toThrow("grammar-node limit"); + }); + + test("rolls back and seals grammar registration with the rest of the factory", () => { + const registry = createEmptyExtensionRegistry(); + let escaped: ((value: ReturnType) => void) | undefined; + runExtensionFactory({ + metadata: bundledMetadata("broken-grammar"), + registry, + issues: [], + factory: (hunk) => { + hunk.registerSyntaxGrammar(grammar()); + throw new Error("rollback"); + }, + }); + expect(registry.syntaxGrammars).toEqual([]); + + runExtensionFactory({ + metadata: bundledMetadata("sealed-grammar"), + registry, + issues: [], + factory: (hunk) => { + escaped = hunk.registerSyntaxGrammar.bind(hunk); + }, + }); + expect(() => escaped?.(grammar())).toThrow("can only be called while the extension is loading"); + }); +}); diff --git a/src/extensions/runExtension.ts b/src/extensions/runExtension.ts index 6345da02b..a9522f55b 100644 --- a/src/extensions/runExtension.ts +++ b/src/extensions/runExtension.ts @@ -17,6 +17,7 @@ import { type ExtensionPane, type ExtensionSidebarView, type ExtensionSessionOptions, + type ExtensionSyntaxGrammar, type ExtensionFileView, type ExtensionKeyboardMode, type ExtensionLineHighlighter, @@ -35,6 +36,7 @@ import { isValidExtensionCliCommandName, } from "../core/run/cliCommandNames"; import { copyExtensionCliCommand } from "./cliCommands"; +import { normalizeSyntaxGrammar, SYNTAX_GRAMMAR_LIMITS } from "./syntaxGrammars"; /** * Running one extension factory into the shared registry. @@ -270,6 +272,7 @@ interface RegistrySnapshot { sessionOptions: number; themes: number; fileLanguages: number; + syntaxGrammars: number; vcsAdapters: number; changesetTransforms: number; panes: number; @@ -294,6 +297,7 @@ function snapshotRegistry(registry: ExtensionRegistry): RegistrySnapshot { sessionOptions: registry.sessionOptions.length, themes: registry.themes.length, fileLanguages: registry.fileLanguages.length, + syntaxGrammars: registry.syntaxGrammars.length, vcsAdapters: registry.vcsAdapters.length, changesetTransforms: registry.changesetTransforms.length, panes: registry.panes.length, @@ -318,6 +322,7 @@ function rollbackRegistry(registry: ExtensionRegistry, snapshot: RegistrySnapsho registry.sessionOptions.length = snapshot.sessionOptions; registry.themes.length = snapshot.themes; registry.fileLanguages.length = snapshot.fileLanguages; + registry.syntaxGrammars.length = snapshot.syntaxGrammars; registry.vcsAdapters.length = snapshot.vcsAdapters; registry.changesetTransforms.length = snapshot.changesetTransforms; registry.panes.length = snapshot.panes; @@ -414,6 +419,19 @@ export function createExtensionApi( language, }); }, + registerSyntaxGrammar(grammar: ExtensionSyntaxGrammar) { + assertOpen("registerSyntaxGrammar"); + if ( + registry.syntaxGrammars.filter(({ extensionId }) => extensionId === metadata.id).length >= + SYNTAX_GRAMMAR_LIMITS.grammarsPerExtension + ) { + throw new Error("registerSyntaxGrammar exceeds the per-extension grammar limit."); + } + registry.syntaxGrammars.push({ + extensionId: metadata.id, + grammar: normalizeSyntaxGrammar(grammar), + }); + }, registerVcsAdapter(adapter: ExtensionVcsAdapter) { assertOpen("registerVcsAdapter"); assertNonEmptyString(adapter?.id, "registerVcsAdapter requires an adapter with an id."); diff --git a/src/extensions/syntaxGrammars.ts b/src/extensions/syntaxGrammars.ts new file mode 100644 index 000000000..aadd8c0bd --- /dev/null +++ b/src/extensions/syntaxGrammars.ts @@ -0,0 +1,273 @@ +import type { + ExtensionSyntaxGrammar, + ExtensionSyntaxGrammarCapture, + ExtensionSyntaxGrammarRule, +} from "../extension-api/types"; + +/** Bounds extension grammar input before it reaches the highlighting worker. */ +export const SYNTAX_GRAMMAR_LIMITS = Object.freeze({ + bytes: 256 * 1024, + depth: 32, + nodes: 10_000, + stringLength: 32_768, + grammarsPerExtension: 16, + grammarsPerSession: 64, +}); + +const ID_PATTERN = /^[a-z][a-z0-9_-]*$/; +const SCOPE_PATTERN = /^(?:source|text)\.[a-z0-9_.-]+$/; +const RULE_KEYS = new Set([ + "include", + "name", + "contentName", + "match", + "begin", + "end", + "while", + "captures", + "beginCaptures", + "endCaptures", + "whileCaptures", + "patterns", + "applyEndPatternLast", +]); +const CAPTURE_KEYS = new Set(["name", "contentName", "patterns"]); +const GRAMMAR_KEYS = new Set(["id", "scopeName", "patterns", "repository"]); + +/** Report whether one unknown value is a plain record. */ +function isRecord(value: unknown): value is Record { + return typeof value === "object" && value !== null && !Array.isArray(value); +} + +/** Reject unknown object keys so the public subset stays closed. */ +function assertOnlyKeys( + value: Record, + allowed: ReadonlySet, + where: string, +) { + for (const key of Object.keys(value)) { + if (!allowed.has(key)) { + throw new Error(`registerSyntaxGrammar does not support ${where}.${key}.`); + } + } +} + +/** Copy one bounded string. */ +function copyString(value: unknown, where: string, optional = false): string | undefined { + if (value === undefined && optional) return undefined; + if (typeof value !== "string" || value.length === 0) { + throw new Error(`registerSyntaxGrammar requires ${where} to be a non-empty string.`); + } + if (value.length > SYNTAX_GRAMMAR_LIMITS.stringLength) { + throw new Error(`registerSyntaxGrammar ${where} exceeds the string-size limit.`); + } + return value; +} + +interface CopyState { + nodes: number; +} + +/** Count one grammar node and enforce nesting bounds. */ +function countNode(state: CopyState, depth: number) { + state.nodes += 1; + if (state.nodes > SYNTAX_GRAMMAR_LIMITS.nodes) { + throw new Error("registerSyntaxGrammar exceeds the grammar-node limit."); + } + if (depth > SYNTAX_GRAMMAR_LIMITS.depth) { + throw new Error("registerSyntaxGrammar exceeds the grammar-depth limit."); + } +} + +/** Copy captures without retaining extension-owned mutable objects. */ +function copyCaptures( + value: unknown, + where: string, + state: CopyState, + depth: number, +): Readonly> | undefined { + if (value === undefined) return undefined; + if (!isRecord(value)) { + throw new Error(`registerSyntaxGrammar requires ${where} to be an object.`); + } + const result: Record = {}; + for (const [key, candidate] of Object.entries(value)) { + if (!/^\d+$/.test(key) || !isRecord(candidate)) { + throw new Error(`registerSyntaxGrammar requires ${where} capture keys to be integers.`); + } + countNode(state, depth); + assertOnlyKeys(candidate, CAPTURE_KEYS, `${where}.${key}`); + const name = copyString(candidate.name, `${where}.${key}.name`, true); + const contentName = copyString(candidate.contentName, `${where}.${key}.contentName`, true); + const patterns = copyPatterns( + candidate.patterns, + `${where}.${key}.patterns`, + state, + depth + 1, + true, + ); + result[key] = Object.freeze({ + ...(name && { name }), + ...(contentName && { contentName }), + ...(patterns && { patterns }), + }); + } + return Object.freeze(result); +} + +/** Copy one TextMate rule in Hunk's closed data-only subset. */ +function copyRule( + value: unknown, + where: string, + state: CopyState, + depth: number, +): ExtensionSyntaxGrammarRule { + if (!isRecord(value)) { + throw new Error(`registerSyntaxGrammar requires ${where} to be an object.`); + } + countNode(state, depth); + assertOnlyKeys(value, RULE_KEYS, where); + + const include = copyString(value.include, `${where}.include`, true); + if (include && include !== "$self" && include !== "$base" && !include.startsWith("#")) { + throw new Error( + "registerSyntaxGrammar includes may reference only #local, $self, or $base rules.", + ); + } + const name = copyString(value.name, `${where}.name`, true); + const contentName = copyString(value.contentName, `${where}.contentName`, true); + const match = copyString(value.match, `${where}.match`, true); + const begin = copyString(value.begin, `${where}.begin`, true); + const end = copyString(value.end, `${where}.end`, true); + const whilePattern = copyString(value.while, `${where}.while`, true); + const patterns = copyPatterns(value.patterns, `${where}.patterns`, state, depth + 1, true); + const captures = copyCaptures(value.captures, `${where}.captures`, state, depth + 1); + const beginCaptures = copyCaptures( + value.beginCaptures, + `${where}.beginCaptures`, + state, + depth + 1, + ); + const endCaptures = copyCaptures(value.endCaptures, `${where}.endCaptures`, state, depth + 1); + const whileCaptures = copyCaptures( + value.whileCaptures, + `${where}.whileCaptures`, + state, + depth + 1, + ); + const applyEndPatternLast = value.applyEndPatternLast; + if (applyEndPatternLast !== undefined && applyEndPatternLast !== 0 && applyEndPatternLast !== 1) { + throw new Error(`registerSyntaxGrammar requires ${where}.applyEndPatternLast to be 0 or 1.`); + } + if (!include && !match && !begin && !patterns) { + throw new Error( + `registerSyntaxGrammar requires ${where} to declare include, match, begin, or patterns.`, + ); + } + + return Object.freeze({ + ...(include && { include }), + ...(name && { name }), + ...(contentName && { contentName }), + ...(match && { match }), + ...(begin && { begin }), + ...(end && { end }), + ...(whilePattern && { while: whilePattern }), + ...(captures && { captures }), + ...(beginCaptures && { beginCaptures }), + ...(endCaptures && { endCaptures }), + ...(whileCaptures && { whileCaptures }), + ...(patterns && { patterns }), + ...(applyEndPatternLast !== undefined && { applyEndPatternLast }), + }); +} + +/** Copy a rule list, optionally accepting omission. */ +function copyPatterns( + value: unknown, + where: string, + state: CopyState, + depth: number, + optional = false, +): readonly ExtensionSyntaxGrammarRule[] | undefined { + if (value === undefined && optional) return undefined; + if (!Array.isArray(value)) { + throw new Error(`registerSyntaxGrammar requires ${where} to be an array.`); + } + return Object.freeze( + value.map((rule, index) => copyRule(rule, `${where}[${index}]`, state, depth)), + ); +} + +/** Collect local includes so misspelled repository references fail during registration. */ +function collectLocalIncludes(rule: ExtensionSyntaxGrammarRule, includes: Set): void { + if (rule.include?.startsWith("#")) includes.add(rule.include.slice(1)); + for (const nested of rule.patterns ?? []) collectLocalIncludes(nested, includes); + for (const captures of [ + rule.captures, + rule.beginCaptures, + rule.endCaptures, + rule.whileCaptures, + ]) { + for (const capture of Object.values(captures ?? {})) { + for (const nested of capture.patterns ?? []) collectLocalIncludes(nested, includes); + } + } +} + +/** Validate, deeply copy, and freeze one public syntax grammar. */ +export function normalizeSyntaxGrammar(value: unknown): ExtensionSyntaxGrammar { + let serialized: string; + try { + serialized = JSON.stringify(value); + } catch { + throw new Error("registerSyntaxGrammar requires serializable grammar data."); + } + if ( + serialized === undefined || + Buffer.byteLength(serialized, "utf8") > SYNTAX_GRAMMAR_LIMITS.bytes + ) { + throw new Error("registerSyntaxGrammar exceeds the serialized-size limit."); + } + if (!isRecord(value)) { + throw new Error("registerSyntaxGrammar requires a grammar object."); + } + assertOnlyKeys(value, GRAMMAR_KEYS, "grammar"); + const id = copyString(value.id, "grammar.id")!; + if (!ID_PATTERN.test(id) || id === "text" || id === "ansi") { + throw new Error( + "registerSyntaxGrammar grammar.id must be a lowercase language id and cannot be text or ansi.", + ); + } + const scopeName = copyString(value.scopeName, "grammar.scopeName")!; + if (!SCOPE_PATTERN.test(scopeName)) { + throw new Error( + "registerSyntaxGrammar grammar.scopeName must start with source. or text. and use portable characters.", + ); + } + const state = { nodes: 1 }; + const patterns = copyPatterns(value.patterns, "grammar.patterns", state, 1)!; + let repository: Record | undefined; + if (value.repository !== undefined) { + if (!isRecord(value.repository)) { + throw new Error("registerSyntaxGrammar requires grammar.repository to be an object."); + } + repository = {}; + for (const [key, rule] of Object.entries(value.repository)) { + if (!ID_PATTERN.test(key)) { + throw new Error("registerSyntaxGrammar repository keys must be lowercase portable ids."); + } + repository[key] = copyRule(rule, `grammar.repository.${key}`, state, 1); + } + Object.freeze(repository); + } + const includes = new Set(); + for (const rule of patterns) collectLocalIncludes(rule, includes); + for (const rule of Object.values(repository ?? {})) collectLocalIncludes(rule, includes); + for (const include of includes) { + if (repository === undefined || !Object.hasOwn(repository, include)) { + throw new Error(`registerSyntaxGrammar references missing local rule #${include}.`); + } + } + return Object.freeze({ id, scopeName, patterns, ...(repository && { repository }) }); +} diff --git a/src/extensions/types.ts b/src/extensions/types.ts index 1a399b57f..6c2608930 100644 --- a/src/extensions/types.ts +++ b/src/extensions/types.ts @@ -17,6 +17,7 @@ import type { ExtensionNotifyType, ExtensionPane, ExtensionSessionOptions, + ExtensionSyntaxGrammar, ExtensionThemeConfig, } from "../extension-api/types"; import { createExtensionNotificationHub, type ExtensionNotificationHub } from "./notifications"; @@ -71,6 +72,9 @@ export type { ExtensionPaneControls, ExtensionSidebarView, ExtensionSessionOptions, + ExtensionSyntaxGrammar, + ExtensionSyntaxGrammarCapture, + ExtensionSyntaxGrammarRule, ExtensionThemeConfig, ExtensionVcsAdapter, ExtensionWorkspace, @@ -123,6 +127,11 @@ export interface RegisteredFileLanguage { language: string; } +export interface RegisteredSyntaxGrammar { + extensionId: string; + grammar: ExtensionSyntaxGrammar; +} + export interface RegisteredVcsAdapter { extensionId: string; adapter: VcsAdapter; @@ -208,6 +217,7 @@ export interface ExtensionRegistry { sessionOptions: RegisteredSessionOptions[]; themes: RegisteredTheme[]; fileLanguages: RegisteredFileLanguage[]; + syntaxGrammars: RegisteredSyntaxGrammar[]; vcsAdapters: RegisteredVcsAdapter[]; changesetTransforms: RegisteredChangesetTransform[]; panes: RegisteredPane[]; @@ -297,6 +307,7 @@ export function createEmptyExtensionRegistry(): ExtensionRegistry { sessionOptions: [], themes: [], fileLanguages: [], + syntaxGrammars: [], vcsAdapters: [], changesetTransforms: [], panes: [], diff --git a/src/ui/AppHost.extensions.test.tsx b/src/ui/AppHost.extensions.test.tsx index b777021af..1b0913f41 100644 --- a/src/ui/AppHost.extensions.test.tsx +++ b/src/ui/AppHost.extensions.test.tsx @@ -18,6 +18,10 @@ import type { AppBootstrap } from "../app/types"; import { getBundledVcsCatalog } from "../app/vcsCatalog"; import { loadAppBootstrap as loadCoreAppBootstrap } from "../core/changeset/loaders"; import { fileLanguageForPath } from "../core/changeset/fileLanguageLookup"; +import { + replaceExtensionSyntaxGrammars, + syntaxGrammarSnapshot, +} from "../core/changeset/syntaxGrammar"; import type { CliInput } from "../core/run/commandInputs"; import type { HunkSessionBrokerClient } from "../session/broker/brokerClient"; @@ -57,6 +61,7 @@ const tempDirs: string[] = []; const originalXdgConfigHome = process.env.XDG_CONFIG_HOME; afterEach(async () => { + replaceExtensionSyntaxGrammars([]); for (const dir of tempDirs.splice(0)) { await removeTestDirectory(dir); } @@ -128,14 +133,22 @@ function useTempConfigHome(configToml?: string) { } /** Write an extension that appends every lifecycle event it sees to a log file. */ -function writeProbeExtension(path: string, logPath: string, languageExtension?: string) { +function writeProbeExtension( + path: string, + logPath: string, + languageExtension?: string, + syntaxGrammarId?: string, +) { mkdirSync(join(path, ".."), { recursive: true }); writeFileSync( path, `import { appendFileSync } from "node:fs";\n` + `export default function (hunk) {\n` + + (syntaxGrammarId + ? ` hunk.registerSyntaxGrammar({ id: ${JSON.stringify(syntaxGrammarId)}, scopeName: ${JSON.stringify(`source.${syntaxGrammarId}`)}, patterns: [{ match: "x", name: "keyword.${syntaxGrammarId}" }] });\n` + : "") + (languageExtension - ? ` hunk.registerFileLanguage(${JSON.stringify(languageExtension)}, "python");\n` + ? ` hunk.registerFileLanguage(${JSON.stringify(languageExtension)}, ${JSON.stringify(syntaxGrammarId ?? "python")});\n` : "") + ` appendFileSync(${JSON.stringify(logPath)}, "factory\\n");\n` + ` hunk.on("startup", () => {\n` + @@ -159,6 +172,8 @@ function writeDelayedReplacementExtension(path: string, logPath: string, release `import { appendFileSync, existsSync } from "node:fs";\n` + `export default function (hunk) {\n` + ` const replacement = existsSync(${JSON.stringify(logPath)});\n` + + ` const grammarId = replacement ? "replacementgategrammar" : "currentgategrammar";\n` + + ` hunk.registerSyntaxGrammar({ id: grammarId, scopeName: "source." + grammarId, patterns: [{ match: "x", name: "keyword." + grammarId }] });\n` + ` appendFileSync(${JSON.stringify(logPath)}, "factory\\n");\n` + ` hunk.transformChangeset(async (changeset) => {\n` + ` while (replacement && !existsSync(${JSON.stringify(releasePath)})) {\n` + @@ -500,7 +515,7 @@ describe("reload keeps launch extension authority", () => { const repo = createTestRepo("hunk-apphost-broker-replacement-failure-"); const logPath = join(repo, "probe.log"); const extPath = join(repo, "ext.ts"); - writeProbeExtension(extPath, logPath, "currenthunksyntax"); + writeProbeExtension(extPath, logPath, "currenthunksyntax", "currenthunksyntax"); useTempConfigHome(); const bootstrap = await launchInSubdirectory(repo, { extensionPaths: [extPath] }); @@ -519,8 +534,9 @@ describe("reload keeps launch extension authority", () => { { producerId: "broker-failure" }, ); const initialGeneration = producer.getPublication().generation; - expect(fileLanguageForPath("example.currenthunksyntax")).toBe("python"); - writeProbeExtension(extPath, logPath, "replacementhunksyntax"); + expect(fileLanguageForPath("example.currenthunksyntax")).toBe("currenthunksyntax"); + expect(syntaxGrammarSnapshot().grammars.map(({ id }) => id)).toEqual(["currenthunksyntax"]); + writeProbeExtension(extPath, logPath, "replacementhunksyntax", "replacementhunksyntax"); await withAppHost( bootstrap, @@ -547,8 +563,9 @@ describe("reload keeps launch extension authority", () => { expect(events.filter((line) => line === "factory")).toHaveLength(2); expect(events.filter((line) => line === "startup")).toHaveLength(1); expect(events.filter((line) => line === "shutdown")).toHaveLength(1); - expect(fileLanguageForPath("example.currenthunksyntax")).toBe("python"); + expect(fileLanguageForPath("example.currenthunksyntax")).toBe("currenthunksyntax"); expect(fileLanguageForPath("example.replacementhunksyntax")).toBe("text"); + expect(syntaxGrammarSnapshot().grammars.map(({ id }) => id)).toEqual(["currenthunksyntax"]); }, broker.client, { reviewProducer: producer }, @@ -666,6 +683,8 @@ describe("reload keeps launch extension authority", () => { cwd: join(repo, "sub"), cliExtensionPaths: [extPath], }); + applyExtensionRegistrations(bootstrap.extensions, getBundledVcsCatalog()); + expect(syntaxGrammarSnapshot().grammars.map(({ id }) => id)).toEqual(["currentgategrammar"]); const broker = createTestBrokerClient(); const quitController = new AbortController(); let quits = 0; @@ -705,6 +724,9 @@ describe("reload keeps launch extension authority", () => { expect(broker.replacementCount()).toBe(0); expect(events.filter((line) => line === "startup")).toHaveLength(1); expect(events.filter((line) => line === "session_reload")).toHaveLength(0); + expect(syntaxGrammarSnapshot().grammars.map(({ id }) => id)).toEqual([ + "currentgategrammar", + ]); }, broker.client, { externalQuitSignal: quitController.signal, onQuit: () => (quits += 1) }, diff --git a/src/ui/AppHost.tsx b/src/ui/AppHost.tsx index 9e6fd2e78..33588a9e9 100644 --- a/src/ui/AppHost.tsx +++ b/src/ui/AppHost.tsx @@ -4,6 +4,7 @@ import { ReviewProducer } from "../app/review/producer"; import { loadConfiguredSessionBootstrap } from "../app/sessionBootstrap"; import { getBundledVcsCatalog } from "../app/vcsCatalog"; import { restoreFileLanguageRegistrations } from "../core/changeset/fileLanguage"; +import { restoreSyntaxGrammars } from "../core/changeset/syntaxGrammar"; import { resolveConfiguredCliInput } from "../core/run/config"; import { resolveRuntimeCliInput } from "../core/process/terminal"; import type { StartupNotice } from "../core/process/startupNotice"; @@ -324,6 +325,7 @@ export function AppHost({ // Quit therefore linearizes either wholly before or wholly after adoption. if (quitRequestedRef.current) { restoreFileLanguageRegistrations(loaded.previousFileLanguages); + restoreSyntaxGrammars(loaded.previousSyntaxGrammars); await retirePreparedExtensionReplacement(replacementExtensions); throw reloadRefusedDuringShutdown(); } @@ -372,6 +374,7 @@ export function AppHost({ } } catch (error) { restoreFileLanguageRegistrations(loaded.previousFileLanguages); + restoreSyntaxGrammars(loaded.previousSyntaxGrammars); await retirePreparedExtensionReplacement(replacementExtensions); throw error; } diff --git a/src/ui/diff/diffRows.test.ts b/src/ui/diff/diffRows.test.ts index 87b1b7f75..78a60a065 100644 --- a/src/ui/diff/diffRows.test.ts +++ b/src/ui/diff/diffRows.test.ts @@ -2,6 +2,7 @@ import { describe, expect, test } from "bun:test"; import { parseDiffFromFile, parsePatchFiles } from "@pierre/diffs"; import { createTwoFilesPatch } from "diff"; import type { DiffFile } from "../../core/changeset/model"; +import { replaceExtensionSyntaxGrammars } from "../../core/changeset/syntaxGrammar"; import { buildSplitRows, buildStackRows, @@ -316,6 +317,55 @@ describe("Pierre diff rows", () => { ); }); + test("offloads a small custom language and returns colored compact spans", async () => { + replaceExtensionSyntaxGrammars([ + { + extensionId: "archlang", + grammar: Object.freeze({ + id: "archlang", + scopeName: "source.archlang", + patterns: Object.freeze([ + { match: "\\b(?:component|property|export)\\b", name: "keyword.control.archlang" }, + ]), + }), + }, + ]); + const file = createDiffFile(); + file.language = "archlang"; + file.path = "architecture.arch"; + file.metadata.lang = "archlang" as never; + const theme = resolveTheme("github-dark-default", null); + + expect(shouldOffloadHighlight(file.metadata, theme, {}, file.language)).toBe(true); + const highlighted = await loadHighlightedDiff(file, theme); + expect(highlighted.compact?.payload.foregroundPalette.length).toBeGreaterThan(0); + replaceExtensionSyntaxGrammars([]); + }, 30_000); + + test("keeps bundled worker highlighting healthy after custom grammar use", async () => { + replaceExtensionSyntaxGrammars([ + { + extensionId: "broken", + grammar: Object.freeze({ + id: "broken", + scopeName: "source.broken", + patterns: Object.freeze([{ match: "(", name: "keyword.broken" }]), + }), + }, + ]); + const broken = createDiffFile(); + broken.language = "broken"; + broken.metadata.lang = "broken" as never; + const theme = resolveTheme("github-dark-default", null); + await expect(loadHighlightedDiff(broken, theme)).resolves.toHaveProperty("compact"); + + const bundled = await loadHighlightedDiff(createWorkerEligibleDiffFile(), theme, { + offloadLargeDiff: true, + }); + expect(bundled.compact).toBeDefined(); + replaceExtensionSyntaxGrammars([]); + }, 30_000); + test("matches inline spans when an eligible bundled-theme diff uses the worker", async () => { const file = createWorkerEligibleDiffFile(); const theme = resolveTheme("github-dark-default", null); diff --git a/src/ui/diff/diffRows.ts b/src/ui/diff/diffRows.ts index f50bccf1e..d90879a14 100644 --- a/src/ui/diff/diffRows.ts +++ b/src/ui/diff/diffRows.ts @@ -48,6 +48,8 @@ import { ensureSyntaxHighlightThemeRegistered, syntaxHighlightThemeName, } from "./syntaxHighlightTheme"; +import { isCustomSyntaxLanguage } from "../../core/changeset/syntaxGrammar"; +import { activeSyntaxGrammarDigest } from "./syntaxGrammarRuntime"; type HighlightThemeInput = AppTheme | AppTheme["appearance"]; @@ -443,7 +445,7 @@ function collapsedGapRow( async function prepareHighlighter(language: string | undefined, theme: HighlightThemeInput) { const resolvedLanguage = language ?? "text"; const syntaxTheme = ensureSyntaxHighlightThemeRegistered(theme); - const cacheKey = `${syntaxTheme}:${resolvedLanguage}`; + const cacheKey = `${activeSyntaxGrammarDigest()}:${syntaxTheme}:${resolvedLanguage}`; const options = highlighterOptionsByKey.get(cacheKey) ?? getHighlighterOptions(resolvedLanguage, { @@ -600,15 +602,18 @@ export function shouldOffloadHighlight( metadata: FileDiffMetadata, theme: HighlightThemeInput, options: LoadHighlightedDiffOptions, + language: string | undefined = metadata.lang, ) { + const customLanguage = isCustomSyntaxLanguage(language); return ( - options.offloadLargeDiff === true && + (options.offloadLargeDiff === true || customLanguage) && supportsHighlightWorkerOffload() && typeof theme !== "string" && Object.keys(theme.syntaxScopeOverrides ?? {}).length === 0 && shouldHighlightMetadata(metadata) && - Math.max(metadata.deletionLines.length, metadata.additionLines.length) >= - HIGHLIGHT_WORKER_MIN_LINES + (customLanguage || + Math.max(metadata.deletionLines.length, metadata.additionLines.length) >= + HIGHLIGHT_WORKER_MIN_LINES) ); } @@ -668,7 +673,10 @@ export async function loadHighlightedDiff( sourcePlan && shouldHighlightMetadata(sourcePlan.metadata) ? sourcePlan : null; const metadata = highlightSourcePlan?.metadata ?? file.metadata; - if (typeof theme !== "string" && shouldOffloadHighlight(metadata, theme, options)) { + if ( + typeof theme !== "string" && + shouldOffloadHighlight(metadata, theme, options, file.language) + ) { try { return await loadWorkerHighlightedDiff(file, metadata, theme, highlightSourcePlan); } catch { @@ -678,6 +686,13 @@ export async function loadHighlightedDiff( } } + // Extension regexes run only in the killable worker. Compiled Windows builds and custom syntax + // themes keep the existing main-thread fallback for bundled languages, but custom grammars stay + // plaintext rather than gaining authority over the terminal event loop. + if (isCustomSyntaxLanguage(file.language)) { + return UNHIGHLIGHTED_DIFF; + } + try { const highlighter = await prepareHighlighter(file.language, theme); try { @@ -714,6 +729,9 @@ export async function loadHighlightedSourceLines({ text: string; theme?: HighlightThemeInput; }): Promise { + if (isCustomSyntaxLanguage(file.language)) { + return { lines: [] }; + } try { const highlighter = await prepareHighlighter(file.language, theme); return queueHighlightedWork(() => { diff --git a/src/ui/diff/syntaxGrammarRuntime.ts b/src/ui/diff/syntaxGrammarRuntime.ts new file mode 100644 index 000000000..a8f47edf4 --- /dev/null +++ b/src/ui/diff/syntaxGrammarRuntime.ts @@ -0,0 +1,6 @@ +import { syntaxGrammarSnapshot } from "../../core/changeset/syntaxGrammar"; + +/** Return the grammar digest used by rendered-result caches. */ +export function activeSyntaxGrammarDigest() { + return syntaxGrammarSnapshot().digest; +} diff --git a/src/ui/diff/useHighlightedDiff.test.ts b/src/ui/diff/useHighlightedDiff.test.ts index 3a9e0bdb5..a9c62d987 100644 --- a/src/ui/diff/useHighlightedDiff.test.ts +++ b/src/ui/diff/useHighlightedDiff.test.ts @@ -4,6 +4,7 @@ import { resolveTheme } from "../themes"; import { HIGHLIGHT_WORKER_MIN_LINES } from "./diffRows"; import { prefetchHighlightedDiff, highlightedDiffCacheKey } from "./useHighlightedDiff"; import { registerHighlightWorker } from "./worker"; +import { replaceExtensionSyntaxGrammars } from "../../core/changeset/syntaxGrammar"; /** Build one file large enough to qualify for worker highlighting. */ function createLargeHighlightTestFile(id: string) { @@ -45,6 +46,24 @@ function registerFailingHighlightWorkerForTest() { } describe("highlighted diff cache", () => { + test("invalidates cached results when custom grammar data changes", () => { + const file = createLargeHighlightTestFile("grammar-cache"); + const theme = resolveTheme("github-dark-default", null); + replaceExtensionSyntaxGrammars([]); + const before = highlightedDiffCacheKey(theme, file); + replaceExtensionSyntaxGrammars([ + { + extensionId: "custom", + grammar: Object.freeze({ + id: "custom", + scopeName: "source.custom", + patterns: Object.freeze([{ match: "x", name: "keyword.custom" }]), + }), + }, + ]); + expect(highlightedDiffCacheKey(theme, file)).not.toBe(before); + replaceExtensionSyntaxGrammars([]); + }); test("does not reuse stale highlighted text for patches that collide under sampling", async () => { const firstPatch = createAdversarialPatch("a"); const secondPatch = createAdversarialPatch("b"); diff --git a/src/ui/diff/useHighlightedDiff.ts b/src/ui/diff/useHighlightedDiff.ts index bd5fe41a9..1510070a0 100644 --- a/src/ui/diff/useHighlightedDiff.ts +++ b/src/ui/diff/useHighlightedDiff.ts @@ -5,6 +5,7 @@ import type { AppTheme } from "../themes"; import { loadHighlightedDiff, type HighlightedDiffCode } from "./diffRows"; import { createHighlightedDiffCache } from "./highlightedDiffCache"; import { syntaxHighlightThemeName } from "./syntaxHighlightTheme"; +import { activeSyntaxGrammarDigest } from "./syntaxGrammarRuntime"; const SHARED_HIGHLIGHTED_DIFF_CACHE = createHighlightedDiffCache(); const SHARED_HIGHLIGHT_PROMISES = new Map>(); @@ -61,7 +62,7 @@ function sourceFetcherFingerprint(file: DiffFile) { /** Cache key that includes every content and source-provider input to highlighted rendering. */ export function highlightedDiffCacheKey(theme: AppTheme, file: DiffFile) { - return `${theme.id}:${syntaxHighlightThemeName(theme)}:${file.id}:${file.language ?? "text"}:${highlightedContentFingerprint(file)}:${sourceFetcherFingerprint(file)}`; + return `${activeSyntaxGrammarDigest()}:${theme.id}:${syntaxHighlightThemeName(theme)}:${file.id}:${file.language ?? "text"}:${highlightedContentFingerprint(file)}:${sourceFetcherFingerprint(file)}`; } /** diff --git a/src/ui/diff/worker/highlightWorker.ts b/src/ui/diff/worker/highlightWorker.ts index 7da20daed..b129a44bc 100644 --- a/src/ui/diff/worker/highlightWorker.ts +++ b/src/ui/diff/worker/highlightWorker.ts @@ -2,15 +2,19 @@ /** * Highlights diff metadata away from the terminal event loop. * - * This worker accepts only bundled Pierre themes. The main thread keeps custom-theme registration, - * source loading, result mapping, and every terminal rendering concern local. + * The main thread sends bounded data-only custom grammars before matching highlight jobs. Grammar + * changes dispose the worker-local shared highlighter and cache before any new result is produced. */ import { + RegisteredCustomLanguages, + disposeHighlighter, getHighlighterOptions, getSharedHighlighter, + registerCustomLanguage, renderDiffWithHighlighter, type FileDiffMetadata, } from "@pierre/diffs"; +import type { ExtensionSyntaxGrammar } from "../../../extension-api/types"; import { aliasContextHighlightLines } from "./highlightContext"; import { cloneCompactHighlightedDiff, @@ -22,26 +26,36 @@ import { import { HighlightWorkerCache } from "./highlightWorkerCache"; import { highlightWorkerCacheKey } from "./highlightWorkerIdentity"; -interface HighlightWorkerRequest { - version: 3; - id: number; - aliasContext: boolean; - metadata: FileDiffMetadata; - appearance: "dark" | "light"; - language: string; - theme: string; -} - -type HighlightWorkerResponse = +type HighlightWorkerRequest = | { - version: 3; - id: number; - ok: true; - code: CompactHighlightedDiff; + version: 4; + type: "configure"; + generation: number; + digest: string; + grammars: readonly ExtensionSyntaxGrammar[]; } - | { version: 3; id: number; ok: false; message: string }; + | { + version: 4; + type: "highlight"; + id: number; + grammarGeneration: number; + aliasContext: boolean; + metadata: FileDiffMetadata; + appearance: "dark" | "light"; + language: string; + theme: string; + }; + +type HighlightWorkerResponse = + | { version: 4; type: "configured"; generation: number; ok: true } + | { version: 4; type: "configured"; generation: number; ok: false; message: string } + | { version: 4; type: "highlight"; id: number; ok: true; code: CompactHighlightedDiff } + | { version: 4; type: "highlight"; id: number; ok: false; message: string }; const highlightedDiffCache = new HighlightWorkerCache(); +let grammarGeneration = -1; +let grammarDigest = ""; +let customGrammarIds: readonly string[] = []; /** Build the fixed Pierre render options shared with the terminal highlighter. */ function workerRenderOptions(theme: string) { @@ -59,17 +73,87 @@ function errorMessage(error: unknown) { return error instanceof Error ? error.message : String(error); } +/** Reject malformed internal configuration instead of corrupting the worker registry. */ +function assertGrammarConfiguration( + grammars: unknown, +): asserts grammars is ExtensionSyntaxGrammar[] { + if (!Array.isArray(grammars) || grammars.length > 64) { + throw new Error("Invalid syntax grammar configuration."); + } + const ids = new Set(); + for (const grammar of grammars) { + if ( + typeof grammar !== "object" || + grammar === null || + typeof grammar.id !== "string" || + typeof grammar.scopeName !== "string" || + !Array.isArray(grammar.patterns) || + ids.has(grammar.id) + ) { + throw new Error("Invalid syntax grammar configuration."); + } + ids.add(grammar.id); + } +} + +/** Apply one complete grammar generation to this worker. */ +async function configureGrammars(request: Extract) { + if (request.generation === grammarGeneration && request.digest === grammarDigest) return; + assertGrammarConfiguration(request.grammars); + await disposeHighlighter(); + for (const id of customGrammarIds) RegisteredCustomLanguages.delete(id); + for (const grammar of request.grammars) { + const registration = { + name: grammar.id, + scopeName: grammar.scopeName, + patterns: grammar.patterns, + repository: grammar.repository ?? {}, + }; + registerCustomLanguage(grammar.id, async () => ({ default: [registration] as never[] })); + } + customGrammarIds = request.grammars.map(({ id }) => id); + grammarGeneration = request.generation; + grammarDigest = request.digest; + highlightedDiffCache.clear(); +} + declare const self: Worker; self.onmessage = async (event: MessageEvent) => { - const { aliasContext, appearance, id, language, metadata, theme, version } = event.data; + const request = event.data; + if (request.version !== 4) return; + + if (request.type === "configure") { + try { + await configureGrammars(request); + const response: HighlightWorkerResponse = { + version: 4, + type: "configured", + generation: request.generation, + ok: true, + }; + self.postMessage(response); + } catch (error) { + const response: HighlightWorkerResponse = { + version: 4, + type: "configured", + generation: request.generation, + ok: false, + message: errorMessage(error), + }; + self.postMessage(response); + } + return; + } - if (version !== 3) { + const { aliasContext, appearance, id, language, metadata, theme } = request; + if (request.grammarGeneration !== grammarGeneration) { const response: HighlightWorkerResponse = { - version: 3, + version: 4, + type: "highlight", id, ok: false, - message: `Unsupported highlight worker protocol version: ${String(version)}`, + message: "Syntax grammar configuration changed before highlighting.", }; self.postMessage(response); return; @@ -83,8 +167,6 @@ self.onmessage = async (event: MessageEvent) => { metadata, theme, }); - // A transferred response detaches its buffers. Cache hits therefore return a fresh typed-array - // copy, while the worker retains its own compact payload for a later request. let code = highlightedDiffCache.get(cacheKey); if (!code) { const highlighter = await getSharedHighlighter({ @@ -100,24 +182,16 @@ self.onmessage = async (event: MessageEvent) => { aliasContext ? aliasContextHighlightLines(metadata, highlighted) : highlighted, appearance, ); - - // Oversized payloads stay uncached and transfer their only copy, avoiding a temporary - // second typed-array payload that would violate the worker cache's memory bound. code = highlightedDiffCache.set(cacheKey, cachedCode) ? cloneCompactHighlightedDiff(cachedCode) : cachedCode; } - - const response: HighlightWorkerResponse = { - version: 3, - id, - ok: true, - code, - }; + const response: HighlightWorkerResponse = { version: 4, type: "highlight", id, ok: true, code }; self.postMessage(response, compactHighlightTransferList(code)); } catch (error) { const response: HighlightWorkerResponse = { - version: 3, + version: 4, + type: "highlight", id, ok: false, message: errorMessage(error), diff --git a/src/ui/diff/worker/highlightWorkerCache.ts b/src/ui/diff/worker/highlightWorkerCache.ts index 1b695faae..ede671702 100644 --- a/src/ui/diff/worker/highlightWorkerCache.ts +++ b/src/ui/diff/worker/highlightWorkerCache.ts @@ -64,6 +64,12 @@ export class HighlightWorkerCache { return true; } + /** Drop every retained result when grammar configuration changes. */ + clear() { + this.entries.clear(); + this.cachedBytes = 0; + } + /** Reports retained payload bytes for focused cache tests. */ getCachedBytes() { return this.cachedBytes; diff --git a/src/ui/diff/worker/highlightWorkerClient.test.ts b/src/ui/diff/worker/highlightWorkerClient.test.ts index 46556a170..60a53141a 100644 --- a/src/ui/diff/worker/highlightWorkerClient.test.ts +++ b/src/ui/diff/worker/highlightWorkerClient.test.ts @@ -1,6 +1,7 @@ import { afterEach, describe, expect, test } from "bun:test"; import { createTestDiffFile } from "../../../../test/helpers/diff-helpers"; import { supportsHighlightWorkerOffload } from "../../../highlightWorkerClient"; +import { replaceExtensionSyntaxGrammars } from "../../../core/changeset/syntaxGrammar"; import type { CompactHighlightedDiff } from "./highlightCompact"; import { disposeHighlightWorker, @@ -8,11 +9,9 @@ import { registerHighlightWorker, } from "./highlightWorkerClient"; -interface TestWorkerRequest { - version: 3; - id: number; - aliasContext: boolean; -} +type TestWorkerRequest = + | { version: 4; type: "configure"; generation: number; grammars: readonly unknown[] } + | { version: 4; type: "highlight"; id: number; aliasContext: boolean }; /** Build the smallest valid compact worker response. */ function emptyCompactResponse(): CompactHighlightedDiff { @@ -63,6 +62,13 @@ function createTestHighlightWorker({ throwOnPost }: { throwOnPost?: Error } = {} reply(data: unknown) { worker.onmessage?.({ data } as MessageEvent); }, + acknowledgeConfiguration() { + const request = state.messages.at(-1); + if (!request || request.type !== "configure") throw new Error("Expected configure request"); + worker.onmessage?.({ + data: { version: 4, type: "configured", generation: request.generation, ok: true }, + } as MessageEvent); + }, }; } @@ -79,6 +85,7 @@ function requestHighlight(aliasContext = false) { afterEach(() => { disposeHighlightWorker(); + replaceExtensionSyntaxGrammars([]); }); describe("highlight worker client", () => { @@ -111,24 +118,31 @@ describe("highlight worker client", () => { const second = requestHighlight(); expect(control.state.unrefCalls).toBe(1); expect(control.state.messages).toHaveLength(1); - expect(control.state.messages[0]?.aliasContext).toBe(true); + expect(control.state.messages[0]?.type).toBe("configure"); + control.acknowledgeConfiguration(); + expect(control.state.messages).toHaveLength(2); + const firstRequest = control.state.messages[1]; + expect(firstRequest?.type === "highlight" && firstRequest.aliasContext).toBe(true); - control.reply({ version: 2, id: control.state.messages[0]?.id, ok: true }); + control.reply({ version: 3, type: "highlight", id: 1, ok: true }); await Promise.resolve(); - expect(control.state.messages).toHaveLength(1); + expect(control.state.messages).toHaveLength(2); control.reply({ - version: 3, - id: control.state.messages[0]?.id, + version: 4, + type: "highlight", + id: firstRequest?.type === "highlight" ? firstRequest.id : -1, ok: true, code: emptyCompactResponse(), }); await expect(first).resolves.toEqual(emptyCompactResponse()); - expect(control.state.messages).toHaveLength(2); + expect(control.state.messages).toHaveLength(3); + const secondRequest = control.state.messages[2]; control.reply({ - version: 3, - id: control.state.messages[1]?.id, + version: 4, + type: "highlight", + id: secondRequest?.type === "highlight" ? secondRequest.id : -1, ok: false, message: "highlight rejected", }); @@ -160,11 +174,102 @@ describe("highlight worker client", () => { const recovered = createTestHighlightWorker(); registerHighlightWorker(recovered.worker); const pending = requestHighlight(); - const request = recovered.state.messages[0]!; - recovered.reply({ version: 3, id: request.id, ok: true, code: emptyCompactResponse() }); + recovered.acknowledgeConfiguration(); + const request = recovered.state.messages[1]!; + recovered.reply({ + version: 4, + type: "highlight", + id: request.type === "highlight" ? request.id : -1, + ok: true, + code: emptyCompactResponse(), + }); await expect(pending).resolves.toEqual(emptyCompactResponse()); }); + test("rejects every pending request when worker configuration is rejected", async () => { + const control = createTestHighlightWorker(); + registerHighlightWorker(control.worker); + const active = requestHighlight(); + const queued = requestHighlight(); + const configure = control.state.messages[0]; + if (!configure || configure.type !== "configure") throw new Error("Expected configure request"); + + control.reply({ + version: 4, + type: "configured", + generation: configure.generation, + ok: false, + message: "grammar rejected", + }); + + await expect(active).rejects.toThrow("grammar rejected"); + await expect(queued).rejects.toThrow("grammar rejected"); + expect(control.state.terminateCalls).toBe(1); + }); + + test("times out stalled worker configuration without waiting for the production budget", async () => { + const control = createTestHighlightWorker(); + registerHighlightWorker(control.worker, { timeoutMs: 10 }); + + await expect(requestHighlight()).rejects.toThrow( + "Syntax grammar configuration timed out after 10ms", + ); + expect(control.state.terminateCalls).toBe(1); + }); + + test("sends grammar data before work and retires a worker on generation change", async () => { + replaceExtensionSyntaxGrammars([ + { + extensionId: "custom", + grammar: Object.freeze({ + id: "custom", + scopeName: "source.custom", + patterns: Object.freeze([{ match: "x", name: "keyword.custom" }]), + }), + }, + ]); + const first = createTestHighlightWorker(); + registerHighlightWorker(first.worker); + const pending = requestHighlight(); + const configure = first.state.messages[0]; + expect(configure?.type).toBe("configure"); + expect(configure?.type === "configure" && configure.grammars).toHaveLength(1); + + replaceExtensionSyntaxGrammars([]); + await expect(pending).rejects.toThrow("configuration changed"); + expect(first.state.terminateCalls).toBe(1); + + const replacement = createTestHighlightWorker(); + registerHighlightWorker(replacement.worker); + const replacementPending = requestHighlight(); + expect(replacement.state.messages[0]?.type).toBe("configure"); + disposeHighlightWorker(); + await expect(replacementPending).rejects.toThrow("disposed"); + }); + + test("retires an active highlight when its configured grammar generation changes", async () => { + replaceExtensionSyntaxGrammars([ + { + extensionId: "custom", + grammar: Object.freeze({ + id: "custom", + scopeName: "source.custom", + patterns: Object.freeze([{ match: "x", name: "keyword.custom" }]), + }), + }, + ]); + const control = createTestHighlightWorker(); + registerHighlightWorker(control.worker); + const active = requestHighlight(); + control.acknowledgeConfiguration(); + expect(control.state.messages[1]?.type).toBe("highlight"); + + replaceExtensionSyntaxGrammars([]); + + await expect(active).rejects.toThrow("configuration changed"); + expect(control.state.terminateCalls).toBe(1); + }); + test("disposal terminates the worker and rejects active plus queued work", async () => { const control = createTestHighlightWorker(); registerHighlightWorker(control.worker); diff --git a/src/ui/diff/worker/highlightWorkerClient.ts b/src/ui/diff/worker/highlightWorkerClient.ts index 68f9b3851..fee8814c9 100644 --- a/src/ui/diff/worker/highlightWorkerClient.ts +++ b/src/ui/diff/worker/highlightWorkerClient.ts @@ -1,31 +1,50 @@ /** * Brokers terminal syntax-highlighting jobs through Bun's compiled-entrypoint worker support. * - * The first eligible request starts the worker through the root-level Bun resolver, then later - * requests share its serialized queue. UI callers consume the public worker-folder entrypoint. + * Grammar generations configure each worker before its first matching job. A changed generation + * replaces the worker, so stale grammar code and responses cannot cross an extension reload. */ import type { FileDiffMetadata } from "@pierre/diffs"; +import { + subscribeSyntaxGrammarChanges, + syntaxGrammarSnapshot, +} from "../../../core/changeset/syntaxGrammar"; +import type { ExtensionSyntaxGrammar } from "../../../extension-api/types"; import { createHighlightWorker } from "../../../highlightWorkerClient"; import type { CompactHighlightedDiff } from "./highlightCompact"; export type WorkerHighlightedDiffCode = CompactHighlightedDiff; +export const HIGHLIGHT_WORKER_TIMEOUT_MS = 15_000; -interface HighlightWorkerRequest { - version: 3; - id: number; - aliasContext: boolean; - metadata: FileDiffMetadata; - appearance: "dark" | "light"; - language: string; - theme: string; -} +type HighlightWorkerRequest = + | { + version: 4; + type: "configure"; + generation: number; + digest: string; + grammars: readonly ExtensionSyntaxGrammar[]; + } + | { + version: 4; + type: "highlight"; + id: number; + grammarGeneration: number; + aliasContext: boolean; + metadata: FileDiffMetadata; + appearance: "dark" | "light"; + language: string; + theme: string; + }; type HighlightWorkerResponse = - | { version: 3; id: number; ok: true; code: WorkerHighlightedDiffCode } - | { version: 3; id: number; ok: false; message: string }; + | { version: 4; type: "configured"; generation: number; ok: true } + | { version: 4; type: "configured"; generation: number; ok: false; message: string } + | { version: 4; type: "highlight"; id: number; ok: true; code: WorkerHighlightedDiffCode } + | { version: 4; type: "highlight"; id: number; ok: false; message: string }; interface PendingHighlightRequest { id: number; + grammarGeneration: number; aliasContext: boolean; metadata: FileDiffMetadata; appearance: "dark" | "light"; @@ -37,80 +56,98 @@ interface PendingHighlightRequest { let worker: Worker | null = null; let activeRequest: PendingHighlightRequest | null = null; +let configuredGeneration = -1; +let configurationInFlight: number | null = null; +let requestTimer: ReturnType | undefined; +let requestTimeoutMs = HIGHLIGHT_WORKER_TIMEOUT_MS; let nextRequestId = 1; const queuedRequests: PendingHighlightRequest[] = []; +/** Clear the kill timer shared by configuration and highlighting. */ +function clearRequestTimer() { + if (requestTimer) clearTimeout(requestTimer); + requestTimer = undefined; +} + +/** Kill a worker whose grammar or highlight regexes stop making progress. */ +function armRequestTimer(label: string) { + clearRequestTimer(); + requestTimer = setTimeout(() => { + resetWorker(new Error(`${label} timed out after ${requestTimeoutMs}ms.`)); + }, requestTimeoutMs); + requestTimer.unref?.(); +} + /** Attach the one message/error protocol every worker instance uses. */ function useHighlightWorker(nextWorker: Worker) { - // Bun workers otherwise keep a static command or test process alive after its last request. (nextWorker as Worker & { unref?: () => void }).unref?.(); nextWorker.onmessage = handleWorkerMessage; nextWorker.onerror = handleWorkerError; worker = nextWorker; + configuredGeneration = -1; + configurationInFlight = null; return nextWorker; } -/** Register a caller-provided worker, such as a deterministic test double. */ -export function registerHighlightWorker(nextWorker: Worker) { - if (worker && worker !== nextWorker) { +/** Register a caller-provided worker, with an optional short timeout for deterministic tests. */ +export function registerHighlightWorker(nextWorker: Worker, options: { timeoutMs?: number } = {}) { + if (worker && worker !== nextWorker) resetWorker(new Error("The syntax highlighting worker was replaced.")); - } + requestTimeoutMs = options.timeoutMs ?? HIGHLIGHT_WORKER_TIMEOUT_MS; return useHighlightWorker(nextWorker); } /** Return one reusable worker without keeping short-lived Bun processes alive. */ function getHighlightWorker() { - if (worker) { - return worker; - } - - // Construction runs inside `runNextRequest`'s try/catch, so unavailable workers leave the - // visible diff plain rather than aborting the interactive application. - return useHighlightWorker(createHighlightWorker()); + return worker ?? useHighlightWorker(createHighlightWorker()); } -/** Resolve or reject the active job and advance the serialized message queue. */ +/** Resolve or reject the active job and advance the serialized queue. */ function settleActiveRequest(settle: (request: PendingHighlightRequest) => void) { + clearRequestTimer(); const request = activeRequest; activeRequest = null; - if (request) { - settle(request); - } + if (request) settle(request); runNextRequest(); } -/** Receive replies from the one worker and ignore no-longer-relevant messages. */ +/** Receive configuration acknowledgements and highlight results. */ function handleWorkerMessage(event: MessageEvent) { const response = event.data; - const request = activeRequest; - if (!request || response.version !== 3 || response.id !== request.id) { - return; - } - - if (response.ok) { - settleActiveRequest((active) => active.resolve(response.code)); + if (response.version !== 4) return; + if (response.type === "configured") { + if (response.generation !== configurationInFlight) return; + clearRequestTimer(); + configurationInFlight = null; + if (!response.ok) { + resetWorker(new Error(response.message)); + return; + } + configuredGeneration = response.generation; + runNextRequest(); return; } - settleActiveRequest((active) => active.reject(new Error(response.message))); + const request = activeRequest; + if (!request || response.id !== request.id) return; + if (response.ok) settleActiveRequest((active) => active.resolve(response.code)); + else settleActiveRequest((active) => active.reject(new Error(response.message))); } /** Drop a broken worker and fail every request rather than leaving stale work behind. */ function resetWorker(error: Error) { + clearRequestTimer(); const currentWorker = worker; worker = null; - if (currentWorker) { - void currentWorker.terminate(); - } - + configuredGeneration = -1; + configurationInFlight = null; + if (currentWorker) void currentWorker.terminate(); const pending = [activeRequest, ...queuedRequests].filter( (request): request is PendingHighlightRequest => request !== null, ); activeRequest = null; queuedRequests.length = 0; - for (const request of pending) { - request.reject(error); - } + for (const request of pending) request.reject(error); } /** Fail pending work when Bun reports a worker startup or runtime error. */ @@ -118,22 +155,43 @@ function handleWorkerError(event: ErrorEvent) { resetWorker(new Error(event.message || "The syntax highlighting worker failed.")); } -/** Post the next job only after the previous reply has been processed. */ -function runNextRequest() { - if (activeRequest || queuedRequests.length === 0) { - return; - } +/** Configure the worker before posting a job for the active grammar generation. */ +function configureWorker() { + const snapshot = syntaxGrammarSnapshot(); + if (configuredGeneration === snapshot.generation || configurationInFlight !== null) return; + const currentWorker = getHighlightWorker(); + configurationInFlight = snapshot.generation; + const message: HighlightWorkerRequest = { + version: 4, + type: "configure", + generation: snapshot.generation, + digest: snapshot.digest, + grammars: snapshot.grammars, + }; + currentWorker.postMessage(message); + armRequestTimer("Syntax grammar configuration"); +} - const request = queuedRequests.shift(); - if (!request) { +/** Post the next job only after the worker has matching grammar data. */ +function runNextRequest() { + if (activeRequest || queuedRequests.length === 0) return; + const request = queuedRequests[0]!; + if (configuredGeneration !== request.grammarGeneration) { + try { + configureWorker(); + } catch (error) { + resetWorker(error instanceof Error ? error : new Error(String(error))); + } return; } - + queuedRequests.shift(); activeRequest = request; try { const message: HighlightWorkerRequest = { - version: 3, + version: 4, + type: "highlight", id: request.id, + grammarGeneration: request.grammarGeneration, aliasContext: request.aliasContext, metadata: request.metadata, appearance: request.appearance, @@ -141,6 +199,7 @@ function runNextRequest() { theme: request.theme, }; getHighlightWorker().postMessage(message); + armRequestTimer("Syntax highlighting"); } catch (error) { resetWorker(error instanceof Error ? error : new Error(String(error))); } @@ -160,9 +219,21 @@ export function highlightDiffInWorker({ metadata: FileDiffMetadata; theme: string; }) { + const generation = syntaxGrammarSnapshot().generation; + const pendingGeneration = + activeRequest?.grammarGeneration ?? queuedRequests[0]?.grammarGeneration; + if ( + worker && + (pendingGeneration !== undefined + ? pendingGeneration !== generation + : configurationInFlight !== null && configurationInFlight !== generation) + ) { + resetWorker(new Error("Syntax grammar configuration changed.")); + } return new Promise((resolve, reject) => { queuedRequests.push({ id: nextRequestId++, + grammarGeneration: generation, aliasContext, appearance, language, @@ -171,11 +242,22 @@ export function highlightDiffInWorker({ resolve, reject, }); - runNextRequest(); + try { + runNextRequest(); + } catch (error) { + resetWorker(error instanceof Error ? error : new Error(String(error))); + } }); } +subscribeSyntaxGrammarChanges(() => { + if (worker || activeRequest || queuedRequests.length > 0) { + resetWorker(new Error("Syntax grammar configuration changed.")); + } +}); + /** Terminate the shared worker when a controlled caller needs to release it. */ export function disposeHighlightWorker() { resetWorker(new Error("The syntax highlighting worker was disposed.")); + requestTimeoutMs = HIGHLIGHT_WORKER_TIMEOUT_MS; } diff --git a/test/pty/extensions-integration.test.ts b/test/pty/extensions-integration.test.ts index b5fb49338..5c6f72e3c 100644 --- a/test/pty/extensions-integration.test.ts +++ b/test/pty/extensions-integration.test.ts @@ -2,7 +2,7 @@ import { afterEach, describe, expect, setDefaultTimeout, test } from "bun:test"; import { existsSync, readFileSync } from "node:fs"; import { join, resolve } from "node:path"; import { fileURLToPath } from "node:url"; -import { createPtyHarness, dragMouse, lineIndexOf } from "./harness"; +import { createPtyHarness, dragMouse, lineIndexOf, sleep } from "./harness"; const harness = createPtyHarness(); const REVIEW_TRIAGE_EXTENSION = resolve( @@ -17,6 +17,9 @@ const REVIEW_SNAPSHOT_EXPORT_EXTENSION = resolve( const VIM_NAVIGATION_EXTENSION = resolve( fileURLToPath(new URL("../../examples/extensions/vim-navigation", import.meta.url)), ); +const ARCHLANG_SYNTAX_EXTENSION = resolve( + fileURLToPath(new URL("../../examples/extensions/archlang-syntax", import.meta.url)), +); /** Give PTY-backed startup, reloads, and redraws headroom on slower CI machines. */ setDefaultTimeout(30_000); @@ -279,6 +282,42 @@ const DIALOG_EXTENSION_SOURCE = `export default function (hunk) { `; describe("PTY extensions", () => { + test("a custom syntax grammar highlights its mapped file through the worker", async () => { + const configHome = harness.createIsolatedConfigHome(); + const fixture = harness.createRepoExtensionFixture("export default () => {};", "fixture.ts", [ + { + path: "architecture.arch", + before: 'component old: model observed {\n roots ["src/old"]\n}\n', + after: + 'component current: model observed {\n roots ["src/current"]\n}\nproperty stable on component current {\n advisory\n}\n', + }, + ]); + const session = await harness.launchHunk({ + args: ["diff", "--mode", "stack", "--extension", ARCHLANG_SYNTAX_EXTENSION], + cwd: fixture.dir, + env: { XDG_CONFIG_HOME: configHome }, + cols: 110, + rows: 28, + }); + + await session.waitForText(/component current/); + let highlighted = false; + for (let attempt = 0; attempt < 40 && !highlighted; attempt += 1) { + const line = session.getTerminalData().lines.find((candidate) => + candidate.spans + .map((span) => span.text) + .join("") + .includes("component current"), + ); + const codeColors = line?.spans + .filter((span) => span.text.includes("component") || span.text.includes("current")) + .map((span) => span.fg) + .filter(Boolean); + highlighted = new Set(codeColors).size > 1; + if (!highlighted) await sleep(50); + } + expect(highlighted).toBe(true); + }); test("trust prompt runs repo extensions after the user trusts the repository", async () => { const configHome = harness.createIsolatedConfigHome(); const fixture = harness.createRepoExtensionFixture(TRANSFORM_EXTENSION_SOURCE); diff --git a/website/src/content/docs/docs/extend/extension-api.md b/website/src/content/docs/docs/extend/extension-api.md index df94e725d..472e3a925 100644 --- a/website/src/content/docs/docs/extend/extension-api.md +++ b/website/src/content/docs/docs/extend/extension-api.md @@ -7,8 +7,9 @@ The extension factory receives one API object. Registration calls are only valid ## `hunk.apiVersion` -The API generation this Hunk speaks (currently `16`). Branch on it if you want -one file to support several Hunk versions. Version 16 adds pane-wide +The API generation this Hunk speaks (currently `17`). Branch on it if you want +one file to support several Hunk versions. Version 17 adds bounded data-only custom TextMate +syntax grammars; version 16 added pane-wide `onActivate`; version 15 added `{ side, line }` to opted-in pane `currentLine` paint; version 14 added structured two-revision VCS diff endpoints; version 13 added saved-note parent identities @@ -122,7 +123,33 @@ hunk.registerFileLanguage( Filename and glob matching is case-sensitive. Filename selectors match at any directory depth; globs explicitly target the basename or review path exactly as decoded. `/` is the path separator, backslashes stay literal, and filename/glob whitespace is preserved. Globs reject NUL and skip NUL-bearing decoded patch paths; exact filenames can still match them. VCS review paths are normally repo-relative, while generic patches may carry absolute paths. Hunk's reserved `.mts` and `.cts` mappings run first and cannot be overridden. Otherwise, exact filenames take precedence over globs, then extensions. Later registrations win ties. -This selects a grammar already available to Pierre/Shiki; it does not load a new syntax grammar. +This selects a grammar already available to Pierre/Shiki or registered by the same load pass. + +## `hunk.registerSyntaxGrammar(grammar)` + +Register a bounded data-only TextMate grammar, then map files to it separately: + +```ts +hunk.registerSyntaxGrammar({ + id: "mydsl", + scopeName: "source.mydsl", + patterns: [ + { match: "\\b(component|contract)\\b", name: "keyword.control.mydsl" }, + { include: "#strings" }, + ], + repository: { + strings: { begin: '"', end: '"', name: "string.quoted.double.mydsl" }, + }, +}); +hunk.registerFileLanguage(".mydsl", "mydsl"); +``` + +The v17 subset supports ordinary match and begin/end/while rules, captures, nested patterns, and +local `#name`, `$self`, or `$base` includes. Hunk rejects external includes, embeddings, injections, +loaders, unknown keys, oversized/deep grammars, and bundled language ids. Grammar regexes run only +inside the bounded highlight worker; failures and unsupported compiled-Windows offload render plain +text without affecting built-in languages. Reload replaces the full grammar generation and drops +stale cached results. See `examples/extensions/archlang-syntax/` for a complete example. ## `hunk.registerVcsAdapter(adapter)`