[Feature] Syntax highlighting for code blocks that persist into reports

[chrismaddalena]

Summary

Add language-aware syntax highlighting to code blocks in the Tiptap editor and carry equivalent formatting into generated reports.

Installing Tiptap’s CodeBlockLowlight extension alone is insufficient because it uses editor-only ProseMirror decorations. Those decorations are not included in serialized HTML and therefore cannot reach report exporters.

Existing support

Ghostwriter already has much of the required foundation:

• javascript/src/tiptap_gw/codeblock.ts defines a custom code block that permits marks.
• Tiptap code blocks support a language attribute serialized as <code class="language-python">.
• Collaborative content is serialized to ordinary HTML by javascript/src/collab_server/yjs_converters.ts.
• The DOCX and PPTX converters already support colored text runs.
• DOCX code blocks already use the CodeBlock paragraph style.
• XLSX currently converts rich text to plain text.

No database migration should be necessary.

Recommended approach

Keep only the code’s source text and canonical language in Tiptap. Treat syntax colors as derived presentation rather than stored editor content.

Editor

• Extend or replace FormattedCodeblock with CodeBlockLowlight.
• Preserve the existing support for marks and <code> parsing.
• Add a language selector beside the code-block toolbar control.
• Register a curated set of Lowlight languages.
• Default to plaintext; avoid automatic detection unless the detected language is persisted.
• Add editor CSS for the selected highlighting theme.

Report exports

Add Pygments as an explicit Python dependency and tokenize code after Jinja rendering.

• Read the language from code[class^="language-"].
• Look up a lexer using a controlled language/alias registry.
• Convert tokens into styled DOCX and PPTX runs.
• Preserve the existing CodeBlock paragraph formatting.
• Fall back to plain code for missing or unsupported languages.
• Use a light, print-safe palette for generated reports.

The TypeScript and Python implementations should share a canonical language registry or have tests that ensure their mappings remain synchronized.

Report preview

Either:

• Run Lowlight client-side over language-tagged code blocks, or
• Apply Pygments while rendering the preview.

Highlighting should not be persisted as generated token spans.

XLSX

Keep XLSX output plain initially. Rich syntax highlighting is possible with XlsxWriter’s write_rich_string(), but it would require redesigning the current write_string() pipeline and handling Excel’s rich-string constraints.

Alternatives considered

Persist syntax colors as Tiptap marks

This would let the existing exporter consume colored spans, but it would:

• Add substantial collaborative document churn.
• Bake a theme into stored content.
• Require rewriting documents when themes or tokenizers change.
• Conflict with manual formatting inside code blocks.

Not recommended.

Shiki

Shiki offers high-fidelity TextMate/VS Code highlighting, but Ghostwriter’s report generation is Python-based. It would require Node preprocessing, a separate service, or invoking Node during report generation.

Consider only if exact VS Code-style fidelity is required.

CodeMirror node view

CodeMirror would add line numbers, folding, bracket matching, and autocomplete, but significantly increases Tiptap/Yjs integration complexity and still requires a separate export solution.

This should be treated as a future embedded-code-editor feature.

Design considerations

• Explicit user formatting should override derived syntax styling.
• Unknown languages must safely fall back to plain text.
• Do not dynamically import a language based solely on user input.
• Apply reasonable size limits or plain-text fallback for unusually large blocks.
• Highlight only after Jinja templates have rendered.
• Pin CodeBlockLowlight to the repository’s Tiptap version or upgrade all Tiptap packages together.
• Editor dark mode and report output may use different palettes, but token categories should remain visually consistent.

Proposed scope

Initial implementation

• Add a shared language list and aliases.
• Add the Tiptap language selector.
• Add Lowlight editor highlighting.
• Preserve language through HTML/Yjs round trips.
• Add Pygments-based DOCX highlighting.
• Add highlighting to HTML report previews.
• Add tests for known, unknown, and unlabeled languages.
• Verify whitespace, tabs, and line breaks remain unchanged.

Follow-up

• Add PPTX token coloring using the same export token stream.
• Evaluate rich-string highlighting for XLSX.
• Consider additional languages based on user demand.

Acceptance criteria

• A user can select a language for a Tiptap code block.
• The editor highlights that block without modifying its source text.
• The selected language survives save, reload, and collaboration.
• DOCX output contains searchable and copyable syntax-colored text.
• Unsupported or legacy code blocks continue to render as plain code.
• Syntax highlighting does not interfere with Jinja rendering.
• Existing code-block formatting and report templates remain compatible.