Skip to content

feat: tressiter for colored codeblocks - #625

Draft
eszlamczyk wants to merge 15 commits into
feat/51/code-block-highligherfrom
feat/51/code-block-highlighter-treesiter-vendor
Draft

feat: tressiter for colored codeblocks#625
eszlamczyk wants to merge 15 commits into
feat/51/code-block-highligherfrom
feat/51/code-block-highlighter-treesiter-vendor

Conversation

@eszlamczyk

@eszlamczyk eszlamczyk commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator

What/Why?

Second part of #51closes #51.

Part one landed the code-block segmentation/rendering seam. This PR wires the
actual native syntax highlighter behind it: fenced code blocks are now
highlighted on iOS and Android via a vendored tree-sitter
runtime plus a curated set of grammars, foreground-only (it only recolors
tokens, never changes text metrics, so a block's measured height always matches
its drawn height).

Highlighting is on by default with a curated language set, and can be trimmed or
fully disabled per platform to control binary size.

The vendoring problem it solves

Tree-sitter grammars ship as machine-generated parser.c parse tables — the full
supported set is ~177 MB / ~1.1M LOC of generated C. Committing that would
bloat every clone, dominate PR diffs, and swamp GitHub's language stats. It also
has no business living in git: it is 100% reproducible from the pinned grammar
packages. The same is true of the tree-sitter runtime (~21k lines) and the
codegen'd registry — all machine-generated, all reproducible from pins.

So none of the vendored C is committed. The entire vendor/ tree is gitignored
and reproduced on demand from vendor/grammar-versions.json:

  • vendor/grammars/ — the ~177 MB of parser.c/scanner.c tables, copied
    from the grammar devDependencies (version-pinned in the lockfile).
  • vendor/tree-sitter/ — the ~21k-line runtime, fetched and sha256-verified
    from the pinned GitHub release tarball (runtime.tarball + runtime.sha256 in
    the manifest). No native build, no compiler — just lib/src + lib/include.
  • vendor/generated/ — the default registry, codegen'd from the vendored
    grammars by gen-registry.mjs.

Restore is idempotent (.stamp fingerprints per runtime pin and per grammar set),
wired into prepare (so yarn install self-heals) and prepack (so the published
tarball ships the full set — consumers install it prebaked and never fetch).

Net effect: the branch reads as ~2k lines of genuinely reviewable diff — all
the vendored/generated C is out of git entirely — while a git clone +
yarn install (or an npm install of the published tarball) still ends up with
the complete set.

New scripts / infra

  • vendor/vendor-grammars.mjs — one idempotent restore command. Fetches +
    sha256-verifies the runtime from the pinned tarball, copies each grammar's
    minimal source set (only parser.c, scanner.c when present, tree_sitter/*.h,
    highlights.scm, LICENSE) from the grammar devDependencies, and regenerates
    the default registry. Separate .stamp fingerprints (runtime ref+sha; pinned
    grammar versions) make repeated yarn installs a no-op — no re-fetch, no
    rewrite of 177 MB. --force rebuilds regardless; --only a,b scopes to specific
    grammars; --runtime-src <lib> vendors the runtime from a local checkout offline
    instead of the network.
  • vendor/gen-registry.mjs — offline, build-time codegen. Emits
    generated_queries.h + generated_registry.cpp for exactly the selected
    language subset, so the link step is valid no matter how many grammars are
    vendored. Invoked by the Android CMake configure step, the iOS podspec, and by
    vendor-grammars.mjs to (re)generate the default set. Resolves
    ; inherits: directives (e.g. cpp inherits c) by inlining parent highlights.
  • vendor/grammar-versions.json — single pin manifest: the tree-sitter
    runtime ref (v0.26.3, ABI 13-15) + tarball URL + sha256, and every grammar's
    package/version/scanner flags plus which form the curated default set.
  • scripts/prepare-npm-publish.shprepack restores the whole vendor tree,
    copies core/cpp into the RN package, and ships gen-registry.mjs +
    grammar-versions.json alongside so a consumer can compile a custom language
    set; postpack restores the symlink and cleans up.
  • prepare hook now runs vendor-grammars:install before the bob build, so a
    fresh yarn install self-heals the entire gitignored vendor tree.

Consumers can pick their language set (and shrink the binary) via
ENRICHED_MARKDOWN_CODE_HIGHLIGHT_LANGUAGES (Podfile ENV) /
enrichedMarkdown.codeHighlightLanguages (gradle.properties) / the Expo config
plugin - see docs/CODE_HIGHLIGHT.md.

Testing

Quick summary of what was verified:

  • Fresh yarn install regenerates the gitignored vendor/grammars/ from the
    devDependencies; a second install is a .stamp no-op.
  • gen-registry.mjs produces a registry for both the default set and an arbitrary
    --only/--languages subset; the link step stays valid.
  • Highlighting renders correctly in the example app on iOS and Android for the
    default languages, and respects codeBlock.syntaxColors.
  • Language-subset overrides (Podfile ENV / gradle property / config plugin) build
    and drop the unselected grammars from the binary.
  • yarn workspace react-native-enriched-markdown pack produces a tarball that
    contains the full vendored grammar set (prepack restore), while the git
    working tree does not — verified by installing the tarball into a scratch RN app
    and comparing sizes.

PR Checklist

  • Code compiles and runs on iOS
  • Code compiles and runs on Android
  • Updated documentation/README if applicable
  • Ran example app to verify changes
  • E2E tests are passing
  • Required E2E tests have been added (if applicable)

@eszlamczyk
eszlamczyk requested a review from Copilot August 3, 2026 12:57

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot wasn't able to review this pull request because it exceeds the maximum number of lines (20,000). Try reducing the number of changed lines and requesting a review from Copilot again.

@eszlamczyk
eszlamczyk force-pushed the feat/51/code-block-highlighter-treesiter-vendor branch from 55c0b2a to f7a8103 Compare August 3, 2026 13:10

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 20 out of 25 changed files in this pull request and generated no new comments.

Suppressed comments (2)

packages/react-native-enriched-markdown/plugin/src/withIosCodeHighlight.ts:18

  • The early return for the default case (enabled && !hasLanguages) means the plugin won’t remove previously injected ENRICHED_MARKDOWN_* lines if a project switches back to defaults after customizing. That can leave stale Podfile ENV overrides in place and make the app build behave differently than the Expo config.

Running the mod unconditionally (but injecting nothing for the default case) keeps the Podfile consistent with config changes.

  const hasLanguages = Array.isArray(languages) && languages.length > 0;
  // Default (enabled, curated set) needs no Podfile ENV lines.
  if (enabled && !hasLanguages) {
    return config;
  }

packages/react-native-enriched-markdown/plugin/src/withAndroidCodeHighlight.ts:34

  • Even when the config returns to the default (enabled && !hasLanguages), it’s still useful to run the mod so any previously-added enrichedMarkdown.* properties get removed. With the current early return, switching from a customized config back to defaults leaves stale gradle.properties entries behind.

Consider always dropping existing keys, and only re-adding properties when the config is non-default.

  // Default (enabled, curated set) needs no gradle property.
  if (enabled && !hasLanguages) {
    return config;
  }
  return withGradleProperties(config, (gradleConfig) => {

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support Code Block Highliter

2 participants