Serialize Google Docs documents to Markdown - #581
Open
kcarnold wants to merge 1 commit into
Open
Conversation
The document's only consumer is an LLM prompt, but Apps Script was flattening it to bare text, which lost more than it appeared to: - Headings arrived as ordinary sentences. Nothing marked structure. - Lists arrived as prose. Docs renders bullets and numbers from list formatting, so the glyphs are not characters and getText() never returns them; nesting was invisible too. - Paragraphs ran together with no separator, because DocumentApp text carries no trailing newline and the extractor concatenated it directly. getParagraphs() splits on \n, so it returned the whole document as one paragraph. serializeBody() now emits Markdown: headings as #, list items as - or sequentially-numbered N., four spaces per nesting level, blank lines between blocks. Where Docs is richer than Markdown we flatten to the nearest form rather than invent notation — Title and Heading 1 both become #, Subtitle becomes a paragraph (it is not an outline level), and all ordered glyphs become N. Positioning had to change with it. The old code searched the flattened text for the selected string, which resolved a repeated phrase to its first occurrence and cannot work at all against prefixes, since a heading's selected text does not contain its "## ". serializeBody now returns an index of where each child's text landed, and positions are looked up from the element the cursor is actually in. This restores the invariant every caller assumes: beforeCursor + selectedText + afterCursor is exactly the document. Testing, in two layers: - Unit tests run the real Code.gs, read from disk and evaluated with a stubbed DocumentApp, so they cover the file that ships rather than a copy that can drift. Iterating the glyph table no longer costs a clasp push and a sidebar reload. - A Debug page (lab tier) renders the context as the model receives it, cursor marked inline. It is the only way to see the real round-trip, and it works on every surface. Not flag-gated: flags come from the URL, and neither the Word task pane nor the Google Docs sidebar has an addressable one, so a flag would hide it exactly where it is needed. Comments are untouched — DocumentApp exposes none, and reading them needs a Drive scope. Tracked separately. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015urg1xXgUsRk3U8bUzFzC3
Contributor
Author
|
New positioning logic needs testing! Also, we'll need to apply the analogous thing to Word; see #155 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Google Docs document reaches the model as Markdown instead of bare text, so structure the writer marked in Docs survives into the prompt.
Why
The document's only consumer is an LLM prompt, but Apps Script was flattening it to characters, which lost more than it looked like:
Paragraph.getHeading()was never called, so nothing marked structure.getText()on a list item returns only the words. Nesting depth was invisible too.DocumentApptext carries no trailing newline (the paragraph boundary is the newline; onlyBody.getText()inserts it), andextractTextOnlyconcatenated paragraph text directly. SincegetParagraphs()splits the result on\n, it returned the entire document as a single paragraph on this surface.What changed
serializeBody()inCode.gsemits Markdown: headings as#…######, list items as-or sequentially-numberedN., four spaces per nesting level, blank lines between blocks. Empty paragraphs are spacing rather than content and emit nothing.Where Docs is richer than Markdown, we flatten to the nearest form rather than invent notation — a private convention is one more thing for the model to misread:
#…#######-1.Tables and images are still skipped, as they were before.
Positioning had to change with it
The old code searched the flattened text for the selected string. That resolved a phrase occurring twice to its first occurrence, and it cannot work at all against prefixes — a heading's selected text does not contain the
##in front of it.serializeBody()now returns an index of where each top-level child's text landed, and positions are looked up from the element the cursor is actually in (walking up from the Text run viagetParent()/getChildIndex()). This restores the invariant every caller assumes andgetDocText()is built on:beforeCursor + selectedText + afterCursoris exactly the document. A wholly-selected heading includes its##, which is what keeps that exact.Unmappable positions (an element outside the body — a header, a footnote) fall back to "the whole document is before the cursor" rather than throwing, so a serialization edge case can't take the sidebar down.
Testing
Unit tests (21, in
googleDocsMarkdown.test.ts) run the realCode.gs— read from disk and evaluated with a stubbedDocumentApp— so they cover the file that actually ships rather than a copy that can drift from it. The alternative, extracting a shared module, would mean either a second npm package for a folderclasppushes verbatim or amodule.exportsguard in code that has no module system.This matters for the dev loop: iterating a glyph table through
clasp push+ sidebar reload costs minutes per attempt, and here it costs milliseconds. Every positioning test also asserts the concatenation invariant.A Debug page (
labtier, Labs ··· menu) renders the context exactly as the model receives it, with the cursor marked inline, plus the paragraph list and character count. Nothing else in the sidebar surfacesgetDocContext()— every page folds it into a prompt, where a bad serialization is indistinguishable from the model just being confused. It works on Word and the standalone editor too, which makes it the quickest way to compare how the two editors serialize the same document.Deliberately not flag-gated: flags are set from the URL, and neither the Word task pane nor the Google Docs sidebar has an addressable one (see
flags.ts), so a flag would hide it exactly where it's needed. Happy to gate it if you'd rather it not sit in the Labs menu for beta users.npm test→ 243 passed.tsc --noEmitandeslintclean.Not in scope
Comments — filed as #580.
DocumentAppexposes none, and neither does the Docs REST API; they live in the Drive API, so reading them means adding a Drive scope and enabling the Advanced Drive Service. That's an authorization-surface change, not a serialization one.Also noted as gaps in the docs: tables/images are dropped, and inline formatting (bold, italic, links) isn't carried — Docs stores it as attributes over text ranges rather than as characters.
Docs
docs/google-docs-document-serialization.md— full mapping, the flattening rationale, positioning, and the three ways to check itfrontend/CLAUDE.md— what the model sees, and the Word/Docs divergence a new page needs to handlegoogle-docs-addon/README.md— points at bothDeployment note
Code.gschanged, so this needs aclasp pushto take effect — the bundle and the add-on deploy separately. Until then the installed add-on keeps returning bare text, which the frontend still handles (it's just a document with no Markdown in it).Generated by Claude Code