From 10437bcbe1b142003d54ba451fb930cdca7519bc Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 30 Jul 2026 19:12:58 +0000 Subject: [PATCH] Serialize Google Docs documents to Markdown MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The document's only consumer is an LLM prompt, but Apps Script was flattening it to bare text, which lost more than it appeared to: - Headings arrived as ordinary sentences. Nothing marked structure. - Lists arrived as prose. Docs renders bullets and numbers from list formatting, so the glyphs are not characters and getText() never returns them; nesting was invisible too. - Paragraphs ran together with no separator, because DocumentApp text carries no trailing newline and the extractor concatenated it directly. getParagraphs() splits on \n, so it returned the whole document as one paragraph. serializeBody() now emits Markdown: headings as #, list items as - or sequentially-numbered N., four spaces per nesting level, blank lines between blocks. Where Docs is richer than Markdown we flatten to the nearest form rather than invent notation — Title and Heading 1 both become #, Subtitle becomes a paragraph (it is not an outline level), and all ordered glyphs become N. Positioning had to change with it. The old code searched the flattened text for the selected string, which resolved a repeated phrase to its first occurrence and cannot work at all against prefixes, since a heading's selected text does not contain its "## ". serializeBody now returns an index of where each child's text landed, and positions are looked up from the element the cursor is actually in. This restores the invariant every caller assumes: beforeCursor + selectedText + afterCursor is exactly the document. Testing, in two layers: - Unit tests run the real Code.gs, read from disk and evaluated with a stubbed DocumentApp, so they cover the file that ships rather than a copy that can drift. Iterating the glyph table no longer costs a clasp push and a sidebar reload. - A Debug page (lab tier) renders the context as the model receives it, cursor marked inline. It is the only way to see the real round-trip, and it works on every surface. Not flag-gated: flags come from the URL, and neither the Word task pane nor the Google Docs sidebar has an addressable one, so a flag would hide it exactly where it is needed. Comments are untouched — DocumentApp exposes none, and reading them needs a Drive scope. Tracked separately. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_015urg1xXgUsRk3U8bUzFzC3 --- docs/google-docs-document-serialization.md | 160 +++++++ frontend/CLAUDE.md | 28 ++ .../api/__tests__/googleDocsMarkdown.test.ts | 362 +++++++++++++++ .../src/api/__tests__/helpers/appsScript.ts | 316 +++++++++++++ frontend/src/api/googleDocsEditorAPI.ts | 21 +- frontend/src/contexts/pageContext.tsx | 1 + frontend/src/pages/debug/index.tsx | 166 +++++++ frontend/src/pages/debug/styles.module.css | 95 ++++ frontend/src/pages/registry.tsx | 13 + google-docs-addon/Code.gs | 438 +++++++++++++----- google-docs-addon/README.md | 15 + 11 files changed, 1499 insertions(+), 116 deletions(-) create mode 100644 docs/google-docs-document-serialization.md create mode 100644 frontend/src/api/__tests__/googleDocsMarkdown.test.ts create mode 100644 frontend/src/api/__tests__/helpers/appsScript.ts create mode 100644 frontend/src/pages/debug/index.tsx create mode 100644 frontend/src/pages/debug/styles.module.css diff --git a/docs/google-docs-document-serialization.md b/docs/google-docs-document-serialization.md new file mode 100644 index 00000000..8129a718 --- /dev/null +++ b/docs/google-docs-document-serialization.md @@ -0,0 +1,160 @@ +# Google Docs: how the document reaches the model + +The Google Docs add-on serializes the document to **Markdown** before it leaves +Apps Script. This note records what survives that trip, what is deliberately +flattened, what is dropped, and how to check any of it. + +The code is `serializeBody` in `google-docs-addon/Code.gs`; the tests are +`frontend/src/api/__tests__/googleDocsMarkdown.test.ts`. + +## Why Markdown + +The only consumer of the document text is an LLM prompt, and Markdown is the +notation those models read most fluently. The previous serialization was bare +text, which lost more than it looked like: + +- A heading arrived as an ordinary sentence. Nothing marked it as a heading, so + a document's structure was invisible to the model. +- A bulleted list arrived as prose. Docs renders bullets and numbers *from list + formatting*, so the glyphs are not characters — `getText()` on a list item + returns only the words, never `•` or `1.`, and nesting depth is likewise + invisible. +- Paragraphs ran together with no separator at all. `DocumentApp` text carries + no trailing newline (the paragraph boundary *is* the newline, and only + `Body.getText()` inserts it), and the old extractor concatenated paragraph + text directly. Two paragraphs reached the model as one run-on sentence, and + `getParagraphs()` on the frontend — which splits the result on `\n` — returned + the entire document as a single paragraph. + +## What survives + +| In Docs | In the serialized text | +|---|---| +| Heading 1–6 | `#` … `######` | +| Title | `#` | +| Subtitle | plain paragraph | +| Body text | plain paragraph | +| Bullet (round, hollow, square) | `- ` | +| Numbered / lettered / roman | `1. `, numbered sequentially | +| List nesting | four spaces per level | +| Paragraph break | blank line | +| Empty paragraph | nothing (it is spacing, not content) | +| Table | **dropped** | +| Image | **dropped** | +| Comments | **not available** — see below | + +## Deliberate flattening + +Where Docs is richer than Markdown we map to the nearest Markdown form rather +than invent notation, because a private convention is one more thing for the +model to misread. + +- **Title and Heading 1 both become `#`.** A document using both loses that + distinction. Markdown's top level *is* the document title, and the overlap is + rare enough not to justify a custom marker. +- **Subtitle becomes a plain paragraph.** It is not an outline level in Docs — + it doesn't appear in the document outline — so giving it a `#` would claim a + section boundary that isn't there. +- **Hollow and square bullets become `-`,** like round ones. Markdown's `*`, + `-`, and `+` are interchangeable renderers' choices, not distinct meanings. +- **All ordered glyphs become `N.`.** Latin and roman numbering are a display + choice in Docs; the ordinal is the meaning. + +Ordered items are numbered sequentially rather than all emitted as `1.` (which +Markdown also accepts). The numbers are what make an item referenceable — a +writer asking about "the third step" and a model answering about it need the +same label. + +## Cursor and selection positioning + +`getDocContext()` returns `beforeCursor` / `selectedText` / `afterCursor`, and +those three **always concatenate back to exactly the serialized document**. +Callers rely on it (`getDocText` is that concatenation), and the tests assert it +on every positioning case. + +Getting there required dropping the old approach. It searched the flattened text +for the selected string, which had two failure modes: a phrase occurring twice +resolved to the first occurrence, and once Markdown prefixes existed the search +could not succeed at all — the selected text of a heading does not contain the +`## ` in front of it. + +Positions are now derived structurally. `serializeBody` returns, alongside the +Markdown, an index recording where each top-level child's own text landed. +`topLevelChildIndex` walks up from whatever element the cursor points at +(usually a Text run inside a paragraph) to the body child containing it, and the +offset is looked up rather than searched for. + +A wholly-selected element starts at its Markdown prefix rather than after it, so +selecting a heading yields `## Heading` — the prefix is part of that span of the +document, and including it is what keeps the concatenation exact. + +If a position can't be placed (an element outside the body, such as a header or +footnote), the whole document is returned as `beforeCursor` rather than +throwing. A serialization error must not take the sidebar down. + +## Comments are not available + +`DocumentApp` has no comments API at all — not content, not authors, not +anchors. Neither does the Docs REST API v1. + +Comments live in the **Drive API** (`comments.list` on the file), which returns +`content`/`htmlContent`, `replies`, `author`, `resolved`, and +`quotedFileContent.value` — the text the comment is anchored to. The `anchor` +field is an opaque `kix`-style region string with no supported mapping to a +document offset, so locating a comment in practice means searching for its +quoted text, much as `selectPhrase()` already does. + +The blocker is scope. `appsscript.json` requests only +`documents.currentonly`, `script.container.ui`, and `userinfo.email`, and +`Code.gs` is annotated `@OnlyCurrentDoc`. Reading comments means adding a Drive +scope and enabling the Advanced Drive Service — a visible expansion of what +users are asked to authorize. Tracked separately; not part of this +serialization. + +## How to check it + +Three layers, cheapest first. + +**1. Unit tests** — `npm test` in `frontend/`. These run the real `Code.gs`: +`frontend/src/api/__tests__/helpers/appsScript.ts` reads the file and evaluates +it with a stubbed `DocumentApp`, so the tests cover the file that actually ships +rather than a copy that can drift from it. Fakes implement only the element +methods `Code.gs` calls, so anything the serializer starts calling has to be +added there. + +This is where mapping logic belongs. Iterating a glyph table through `clasp +push` + sidebar reload costs minutes per attempt; here it costs milliseconds. + +**2. The Debug page** — a `lab`-tier page (`frontend/src/pages/debug/`), +reachable from the Labs (···) menu on every surface. It shows the document +exactly as the model receives it, with the cursor position marked inline, plus +the paragraph list and character count. This is the only way to verify the real +round-trip — real document, real Apps Script, real bridge — and it works on Word +and the standalone editor too, which makes it the quickest way to compare how +the two editors serialize the same document. + +It is deliberately *not* behind a feature flag. Flags are set from the URL, and +neither the Word task pane nor the Google Docs sidebar has an addressable URL +(see `frontend/src/pages/flags.ts`), so gating it would leave it reachable only +where it is least needed. + +**3. The devtools console** — zero code, for a one-off poke. Open devtools on +the sidebar, switch the frame selector to the sandboxed userHtmlFrame, then: + +```js +await window.GoogleAppsScript.getDocContext(); +``` + +Useful when something looks wrong *below* the bridge and you want to see the +raw Apps Script response. Not a substitute for either layer above, since every +call is a full round-trip and nothing is asserted. + +## Known gaps + +- **Tables and images are dropped.** They were dropped before Markdown too, so + this is not a regression, but a Markdown serializer that silently omits a + table is a reasonable thing to fix — Markdown has table syntax. +- **Comments** — see above. +- **Text formatting** (bold, italic, links) is not carried. Docs stores it as + attributes over text ranges rather than as characters, so mapping it means + walking attribute runs within each paragraph. diff --git a/frontend/CLAUDE.md b/frontend/CLAUDE.md index 8cbd40a0..2b2cfb6d 100644 --- a/frontend/CLAUDE.md +++ b/frontend/CLAUDE.md @@ -45,6 +45,34 @@ task pane's URL comes from `manifest.xml`, and the Google Docs bundle runs insid an Apps Script sandbox iframe with no addressable URL. The Labs menu is the cross-surface way in; flags are for keeping something out of even that. +### What the model sees of the document + +`getDocContext()` returns `beforeCursor` / `selectedText` / `afterCursor`, and +those three always concatenate to the whole document — `getDocText()` is that +concatenation, so anything that breaks the invariant breaks the corpus. + +On **Google Docs** the document is serialized to Markdown in Apps Script +(`serializeBody` in `google-docs-addon/Code.gs`): headings become `#`, list +items `-` / `1.`, paragraphs separated by a blank line. Docs renders list glyphs +from formatting rather than storing them as characters, so without this the +model saw a list as unmarked prose and a heading as an ordinary sentence. +Positions are derived from the element the cursor is in, never by searching the +text for the selected string — that mislocated repeated phrases and cannot work +once prefixes exist, since a heading's selected text doesn't contain its `## `. +Tables, images, comments, and inline formatting do not survive; see +`docs/google-docs-document-serialization.md`. + +**Word** returns plain text with no such markup. A page that comes to depend on +Markdown structure needs to handle both, or the two surfaces will diverge. + +The **Debug** page (Labs menu, `src/pages/debug/`) renders the context exactly as +the model receives it, with the cursor marked inline. It is the only place the +serialization is observable — every other page folds it into a prompt, where a +bad serialization is indistinguishable from the model being confused. Reach for +it on any change to document reading. Unit tests for the Apps Script side live in +`src/api/__tests__/googleDocsMarkdown.test.ts` and run the real `Code.gs` against +a stubbed `DocumentApp`, so iterating the mapping needs no `clasp push`. + ### Document-scoped settings `EditorAPI` has `getDocumentSetting`/`setDocumentSetting` for small values that diff --git a/frontend/src/api/__tests__/googleDocsMarkdown.test.ts b/frontend/src/api/__tests__/googleDocsMarkdown.test.ts new file mode 100644 index 00000000..81e3e3f1 --- /dev/null +++ b/frontend/src/api/__tests__/googleDocsMarkdown.test.ts @@ -0,0 +1,362 @@ +/** + * Tests for the Google Docs add-on's Markdown serialization + * (`google-docs-addon/Code.gs`), run against the real file — see + * `helpers/appsScript.ts` for how and why. + * + * What these pin down is the mapping from what a writer marked in Docs to what + * the model reads. Two properties matter beyond any individual glyph: + * + * 1. Structure the writer marked survives. A heading arrives as a heading. + * 2. `beforeCursor + selectedText + afterCursor` is exactly the document. + * Every caller assumes it (`getDocText` concatenates the three), and the + * old string-search positioning quietly broke it. + */ +import { describe, expect, it } from 'vitest'; + +import { + body, + cursor, + documentWith, + GlyphType, + listItem, + loadAddonScript, + paragraph, + ParagraphHeading, + rangeElement, + selection, + table, +} from './helpers/appsScript'; + +describe('headings', () => { + it('maps each Docs heading level to its Markdown depth', () => { + const script = loadAddonScript(); + const markdown = script.bodyToMarkdown( + body([ + paragraph('Chapter', ParagraphHeading.HEADING1), + paragraph('Section', ParagraphHeading.HEADING2), + paragraph('Sub', ParagraphHeading.HEADING3), + paragraph('Deeper', ParagraphHeading.HEADING4), + paragraph('Deepest', ParagraphHeading.HEADING6), + ]), + ); + + expect(markdown).toBe( + [ + '# Chapter', + '## Section', + '### Sub', + '#### Deeper', + '###### Deepest', + ].join('\n\n'), + ); + }); + + it('leaves body text unprefixed', () => { + const script = loadAddonScript(); + const markdown = script.bodyToMarkdown( + body([paragraph('Just some prose.', ParagraphHeading.NORMAL)]), + ); + + expect(markdown).toBe('Just some prose.'); + }); + + it('flattens Title to a top-level heading and Subtitle to a paragraph', () => { + // Deliberate lossiness: Markdown has no subtitle, and Docs' Subtitle is + // not an outline level, so it must not claim a section boundary. + const script = loadAddonScript(); + const markdown = script.bodyToMarkdown( + body([ + paragraph('The Title', ParagraphHeading.TITLE), + paragraph('A subtitle', ParagraphHeading.SUBTITLE), + ]), + ); + + expect(markdown).toBe('# The Title\n\nA subtitle'); + }); + + it('treats an unrecognized named style as body text', () => { + const script = loadAddonScript(); + // Anything a future Docs version adds must degrade to a plain paragraph + // rather than crash the whole serialization. + expect(script.headingPrefix('SOMETHING_NEW' as never)).toBe(''); + }); +}); + +describe('lists', () => { + it('maps every bullet glyph to a dash', () => { + const script = loadAddonScript(); + const markdown = script.bodyToMarkdown( + body([ + listItem('Round', { glyph: GlyphType.BULLET }), + listItem('Hollow', { glyph: GlyphType.HOLLOW_BULLET }), + listItem('Square', { glyph: GlyphType.SQUARE_BULLET }), + ]), + ); + + expect(markdown).toBe('- Round\n- Hollow\n- Square'); + }); + + it('numbers ordered items sequentially', () => { + const script = loadAddonScript(); + const markdown = script.bodyToMarkdown( + body([ + listItem('First', { glyph: GlyphType.NUMBER }), + listItem('Second', { glyph: GlyphType.NUMBER }), + listItem('Third', { glyph: GlyphType.NUMBER }), + ]), + ); + + expect(markdown).toBe('1. First\n2. Second\n3. Third'); + }); + + it('maps latin and roman glyphs to ordinary numbering', () => { + const script = loadAddonScript(); + const markdown = script.bodyToMarkdown( + body([ + listItem('Alpha', { glyph: GlyphType.LATIN_LOWER }), + listItem('Beta', { glyph: GlyphType.ROMAN_UPPER }), + ]), + ); + + expect(markdown).toBe('1. Alpha\n2. Beta'); + }); + + it('indents nested items and restarts deeper numbering', () => { + const script = loadAddonScript(); + const markdown = script.bodyToMarkdown( + body([ + listItem('One', { glyph: GlyphType.NUMBER, level: 0 }), + listItem('One A', { glyph: GlyphType.NUMBER, level: 1 }), + listItem('One B', { glyph: GlyphType.NUMBER, level: 1 }), + listItem('Two', { glyph: GlyphType.NUMBER, level: 0 }), + listItem('Two A', { glyph: GlyphType.NUMBER, level: 1 }), + ]), + ); + + expect(markdown).toBe( + [ + '1. One', + ' 1. One A', + ' 2. One B', + '2. Two', + ' 1. Two A', + ].join('\n'), + ); + }); + + it('counts separate lists separately', () => { + const script = loadAddonScript(); + const markdown = script.bodyToMarkdown( + body([ + listItem('A1', { glyph: GlyphType.NUMBER, listId: 'a' }), + listItem('A2', { glyph: GlyphType.NUMBER, listId: 'a' }), + paragraph('An interruption.'), + listItem('B1', { glyph: GlyphType.NUMBER, listId: 'b' }), + ]), + ); + + expect(markdown).toBe('1. A1\n2. A2\n\nAn interruption.\n\n1. B1'); + }); + + it('keeps list items adjacent but separates them from prose', () => { + // A blank line between items would end the list; a missing one between + // the list and the paragraph would swallow the paragraph into it. + const script = loadAddonScript(); + const markdown = script.bodyToMarkdown( + body([ + paragraph('Before:'), + listItem('One'), + listItem('Two'), + paragraph('After.'), + ]), + ); + + expect(markdown).toBe('Before:\n\n- One\n- Two\n\nAfter.'); + }); +}); + +describe('block separation', () => { + it('separates paragraphs with a blank line', () => { + // The bug this replaces: paragraph text was concatenated with no + // separator at all, because DocumentApp text carries no trailing + // newline. Two paragraphs arrived as one run-on sentence. + const script = loadAddonScript(); + const markdown = script.bodyToMarkdown( + body([paragraph('First para.'), paragraph('Second para.')]), + ); + + expect(markdown).toBe('First para.\n\nSecond para.'); + }); + + it('does not emit extra blank lines for spacing paragraphs', () => { + const script = loadAddonScript(); + const markdown = script.bodyToMarkdown( + body([paragraph('First.'), paragraph(''), paragraph('Second.')]), + ); + + expect(markdown).toBe('First.\n\nSecond.'); + }); + + it('skips tables and other non-text children', () => { + const script = loadAddonScript(); + const markdown = script.bodyToMarkdown( + body([paragraph('Before.'), table(), paragraph('After.')]), + ); + + expect(markdown).toBe('Before.\n\nAfter.'); + }); + + it('serializes an empty document to an empty string', () => { + const script = loadAddonScript(); + expect(script.bodyToMarkdown(body([]))).toBe(''); + }); +}); + +describe('getDocContext positioning', () => { + /** Every context must reassemble into exactly the serialized document. */ + function expectPartition( + context: { + beforeCursor: string; + selectedText: string; + afterCursor: string; + }, + markdown: string, + ) { + expect( + context.beforeCursor + context.selectedText + context.afterCursor, + ).toBe(markdown); + } + + it('places a cursor at its offset within the paragraph text', () => { + const script = loadAddonScript(); + const intro = paragraph('Hello world'); + const rest = paragraph('Second.'); + const docBody = body([intro, rest]); + script.setActiveDocument( + documentWith(docBody, { cursor: cursor(intro.text, 5) }), + ); + + const context = script.getDocContext(); + + expect(context.beforeCursor).toBe('Hello'); + expect(context.selectedText).toBe(''); + expect(context.afterCursor).toBe(' world\n\nSecond.'); + expectPartition(context, script.bodyToMarkdown(docBody)); + }); + + it('counts the Markdown prefix as before the cursor', () => { + // The cursor's offset is into the writer's text, which starts after the + // `## ` we inserted. Adding the prefix length is what keeps them aligned. + const script = loadAddonScript(); + const heading = paragraph('Section title', ParagraphHeading.HEADING2); + const docBody = body([heading]); + script.setActiveDocument( + documentWith(docBody, { cursor: cursor(heading.text, 7) }), + ); + + const context = script.getDocContext(); + + expect(context.beforeCursor).toBe('## Section'); + expect(context.afterCursor).toBe(' title'); + }); + + it('locates a partial selection precisely', () => { + const script = loadAddonScript(); + const only = paragraph('The quick brown fox'); + const docBody = body([only]); + script.setActiveDocument( + documentWith(docBody, { + selection: selection([ + rangeElement(only.text, { start: 4, endInclusive: 8 }), + ]), + }), + ); + + const context = script.getDocContext(); + + expect(context.beforeCursor).toBe('The '); + expect(context.selectedText).toBe('quick'); + expect(context.afterCursor).toBe(' brown fox'); + expectPartition(context, script.bodyToMarkdown(docBody)); + }); + + it('spans multiple blocks, prefixes included', () => { + const script = loadAddonScript(); + const heading = paragraph('Title here', ParagraphHeading.HEADING2); + const item = listItem('A point'); + const docBody = body([ + paragraph('Intro.'), + heading, + item, + paragraph('End.'), + ]); + script.setActiveDocument( + documentWith(docBody, { + selection: selection([ + rangeElement(heading), + rangeElement(item), + ]), + }), + ); + + const context = script.getDocContext(); + + expect(context.beforeCursor).toBe('Intro.\n\n'); + expect(context.selectedText).toBe('## Title here\n\n- A point'); + expect(context.afterCursor).toBe('\n\nEnd.'); + expectPartition(context, script.bodyToMarkdown(docBody)); + }); + + it('does not mislocate a phrase that occurs twice', () => { + // The old implementation searched the document for the selected string, + // so selecting the *second* "the plan" reported the first one's position. + const script = loadAddonScript(); + const first = paragraph('We discussed the plan'); + const second = paragraph('We revisited the plan'); + const docBody = body([first, second]); + script.setActiveDocument( + documentWith(docBody, { + selection: selection([ + rangeElement(second.text, { start: 13, endInclusive: 20 }), + ]), + }), + ); + + const context = script.getDocContext(); + + expect(context.selectedText).toBe('the plan'); + expect(context.beforeCursor).toBe( + 'We discussed the plan\n\nWe revisited ', + ); + expect(context.afterCursor).toBe(''); + }); + + it('returns the whole document as "before" when nothing is focused', () => { + const script = loadAddonScript(); + const docBody = body([paragraph('Alone.')]); + script.setActiveDocument(documentWith(docBody)); + + const context = script.getDocContext(); + + expect(context.beforeCursor).toBe('Alone.'); + expect(context.selectedText).toBe(''); + expect(context.afterCursor).toBe(''); + }); + + it('falls back to the whole document when a position cannot be placed', () => { + // An element outside the body (a header, a footnote) has no offset in the + // body's Markdown. Degrading to "everything is before" beats throwing, + // which would take the sidebar down. + const script = loadAddonScript(); + const orphan = paragraph('Detached'); + const docBody = body([paragraph('Real content.')]); + script.setActiveDocument( + documentWith(docBody, { cursor: cursor(orphan.text, 2) }), + ); + + const context = script.getDocContext(); + + expect(context.beforeCursor).toBe('Real content.'); + expect(context.selectedText).toBe(''); + }); +}); diff --git a/frontend/src/api/__tests__/helpers/appsScript.ts b/frontend/src/api/__tests__/helpers/appsScript.ts new file mode 100644 index 00000000..1a12030e --- /dev/null +++ b/frontend/src/api/__tests__/helpers/appsScript.ts @@ -0,0 +1,316 @@ +/** + * A test harness for `google-docs-addon/Code.gs`. + * + * The add-on's Markdown serialization is the only genuinely tricky logic in the + * Google Docs surface, and it is also the logic that is most expensive to check + * by hand: verifying it in place means `clasp push`, reloading the sidebar, and + * reading the result through a UI. That loop is far too slow to iterate a glyph + * table on. + * + * So we run the real file here. `Code.gs` is plain ES2015 that only touches + * Apps Script through globals, so evaluating it with a stubbed `DocumentApp` + * gives us the shipped source under test — not a copy that can drift from it. + * The alternative, extracting the serializer into a module both the add-on and + * the tests import, would mean either a second npm package for a folder clasp + * pushes verbatim or a `module.exports` guard in code that has no module + * system. Evaluating the file avoids both and keeps `Code.gs` readable as what + * it is: an Apps Script file. + * + * The fakes below implement only the element methods `Code.gs` actually calls. + * They are duck types, not a DocumentApp emulator — anything the serializer + * starts calling has to be added here, which is the intended pressure. + */ +import { readFileSync } from 'node:fs'; +import { dirname, resolve } from 'node:path'; +import { fileURLToPath } from 'node:url'; + +const CODE_GS = resolve( + dirname(fileURLToPath(import.meta.url)), + '../../../../../google-docs-addon/Code.gs', +); + +/** Enum values are compared by identity in Code.gs, so plain strings suffice. */ +export const ElementType = { + BODY_SECTION: 'BODY_SECTION', + PARAGRAPH: 'PARAGRAPH', + LIST_ITEM: 'LIST_ITEM', + TEXT: 'TEXT', + TABLE: 'TABLE', + INLINE_IMAGE: 'INLINE_IMAGE', +} as const; + +export const ParagraphHeading = { + NORMAL: 'NORMAL', + TITLE: 'TITLE', + SUBTITLE: 'SUBTITLE', + HEADING1: 'HEADING1', + HEADING2: 'HEADING2', + HEADING3: 'HEADING3', + HEADING4: 'HEADING4', + HEADING5: 'HEADING5', + HEADING6: 'HEADING6', +} as const; + +export const GlyphType = { + BULLET: 'BULLET', + HOLLOW_BULLET: 'HOLLOW_BULLET', + SQUARE_BULLET: 'SQUARE_BULLET', + NUMBER: 'NUMBER', + LATIN_UPPER: 'LATIN_UPPER', + LATIN_LOWER: 'LATIN_LOWER', + ROMAN_UPPER: 'ROMAN_UPPER', + ROMAN_LOWER: 'ROMAN_LOWER', +} as const; + +type Heading = (typeof ParagraphHeading)[keyof typeof ParagraphHeading]; +type Glyph = (typeof GlyphType)[keyof typeof GlyphType]; + +export interface FakeElement { + getType(): string; + getParent(): FakeElement | null; + getText?(): string; +} + +interface Mutable { + parent: FakeElement | null; +} + +/** + * The Text run inside a paragraph. Real cursors point at one of these rather + * than at the paragraph, which is why `topLevelChildIndex` has to walk up. + */ +export interface FakeText extends FakeElement { + getText(): string; +} + +export interface FakeParagraph extends FakeElement { + asParagraph(): FakeParagraph; + getHeading(): Heading; + getText(): string; + /** The Text run a cursor would be placed in. */ + text: FakeText; +} + +export interface FakeListItem extends FakeElement { + asListItem(): FakeListItem; + getText(): string; + getGlyphType(): Glyph; + getNestingLevel(): number; + getListId(): string; + text: FakeText; +} + +export interface FakeBody extends FakeElement { + getNumChildren(): number; + getChild(index: number): FakeElement; + getChildIndex(child: FakeElement): number; +} + +function makeTextRun(owner: () => FakeElement, read: () => string): FakeText { + return { + getType: () => ElementType.TEXT, + getParent: () => owner(), + getText: read, + }; +} + +/** A paragraph, optionally carrying a named style. */ +export function paragraph( + content: string, + heading: Heading = ParagraphHeading.NORMAL, +): FakeParagraph { + const state: Mutable = { parent: null }; + const self = { + getType: () => ElementType.PARAGRAPH, + getParent: () => state.parent, + asParagraph: () => self, + getHeading: () => heading, + getText: () => content, + } as FakeParagraph; + self.text = makeTextRun( + () => self, + () => content, + ); + // biome-ignore lint/suspicious/noExplicitAny: attach the mutable slot + (self as any).__state = state; + return self; +} + +/** A list item. `listId` groups items Docs considers one list. */ +export function listItem( + content: string, + options: { + glyph?: Glyph; + level?: number; + listId?: string; + } = {}, +): FakeListItem { + const glyph = options.glyph ?? GlyphType.BULLET; + const level = options.level ?? 0; + const listId = options.listId ?? 'list-1'; + const state: Mutable = { parent: null }; + const self = { + getType: () => ElementType.LIST_ITEM, + getParent: () => state.parent, + asListItem: () => self, + getText: () => content, + getGlyphType: () => glyph, + getNestingLevel: () => level, + getListId: () => listId, + } as FakeListItem; + self.text = makeTextRun( + () => self, + () => content, + ); + // biome-ignore lint/suspicious/noExplicitAny: attach the mutable slot + (self as any).__state = state; + return self; +} + +/** A non-text child, to check that it is skipped rather than serialized. */ +export function table(): FakeElement { + const state: Mutable = { parent: null }; + const self = { + getType: () => ElementType.TABLE, + getParent: () => state.parent, + } as FakeElement; + // biome-ignore lint/suspicious/noExplicitAny: attach the mutable slot + (self as any).__state = state; + return self; +} + +/** Assembles children into a body and wires up their parent links. */ +export function body(children: FakeElement[]): FakeBody { + const self = { + getType: () => ElementType.BODY_SECTION, + getParent: () => null, + getNumChildren: () => children.length, + getChild: (index: number) => children[index], + getChildIndex: (child: FakeElement) => { + const index = children.indexOf(child); + if (index === -1) + throw new Error('element is not a child of this body'); + return index; + }, + } as FakeBody; + for (const child of children) { + // biome-ignore lint/suspicious/noExplicitAny: read back the mutable slot + const state = (child as any).__state as Mutable | undefined; + if (state) state.parent = self; + } + return self; +} + +/** A cursor, as `Document.getCursor()` returns it. */ +export function cursor(element: FakeElement, offset: number) { + return { + getElement: () => element, + getOffset: () => offset, + }; +} + +/** + * One element of a selection. A partial range covers part of an element's text; + * a non-partial range covers the whole element. + */ +export function rangeElement( + element: FakeElement, + range?: { start: number; endInclusive: number }, +) { + return { + getElement: () => element, + isPartial: () => range !== undefined, + getStartOffset: () => range?.start ?? -1, + getEndOffsetInclusive: () => range?.endInclusive ?? -1, + }; +} + +/** A selection, as `Document.getSelection()` returns it. */ +export function selection(elements: ReturnType[]) { + return { getRangeElements: () => elements }; +} + +export interface DocContext { + beforeCursor: string; + selectedText: string; + afterCursor: string; +} + +/** The subset of Code.gs the tests drive. */ +export interface AddonScript { + bodyToMarkdown(body: FakeBody): string; + serializeBody(body: FakeBody): { + markdown: string; + blocks: Array<{ + start: number; + prefixLength: number; + textLength: number; + } | null>; + }; + getDocContext(): DocContext; + headingPrefix(heading: Heading): string; + setActiveDocument(document: { + getBody(): FakeBody; + getSelection(): unknown; + getCursor(): unknown; + }): void; +} + +/** + * Evaluates `Code.gs` against a stubbed `DocumentApp` and returns the functions + * under test. Read once, re-evaluated per call so tests never share state. + */ +export function loadAddonScript(): AddonScript { + const source = readFileSync(CODE_GS, 'utf8'); + + let activeDocument: unknown = null; + const DocumentApp = { + ElementType, + ParagraphHeading, + GlyphType, + getActiveDocument: () => activeDocument, + }; + + // Evaluating the source is the whole point: it is what makes these tests + // cover the file that actually ships to Apps Script rather than a duplicate + // of it. The input is a repo file read from disk at a fixed path, never user + // or network data, so the usual injection concern behind this rule does not + // apply. `DocumentApp` is passed as a parameter rather than left global so + // the stub cannot leak into other tests in the same worker. + // eslint-disable-next-line @typescript-eslint/no-implied-eval + const factory = new Function( + 'DocumentApp', + `${source} + return { + bodyToMarkdown: bodyToMarkdown, + serializeBody: serializeBody, + getDocContext: getDocContext, + headingPrefix: headingPrefix, + };`, + ); + + // eslint-disable-next-line @typescript-eslint/no-unsafe-call + const exported = factory(DocumentApp) as Omit< + AddonScript, + 'setActiveDocument' + >; + + return { + ...exported, + setActiveDocument(document) { + activeDocument = document; + }, + }; +} + +/** A document with no cursor and no selection. */ +export function documentWith( + docBody: FakeBody, + options: { cursor?: unknown; selection?: unknown } = {}, +) { + return { + getBody: () => docBody, + getSelection: () => options.selection ?? null, + getCursor: () => options.cursor ?? null, + }; +} diff --git a/frontend/src/api/googleDocsEditorAPI.ts b/frontend/src/api/googleDocsEditorAPI.ts index ec8d8452..17e800d5 100644 --- a/frontend/src/api/googleDocsEditorAPI.ts +++ b/frontend/src/api/googleDocsEditorAPI.ts @@ -317,17 +317,32 @@ export const googleDocsEditorAPI: EditorAPI = { } }, - /** Full document text, used for the corpus and the `view` tool. */ + /** + * Full document text, used for the corpus and the `view` tool. + * + * Apps Script serializes the document to Markdown (`serializeBody` in + * `google-docs-addon/Code.gs`), so headings arrive as `##` and list items as + * `-`/`N.` rather than as unmarked prose. The three context pieces always + * concatenate back to the whole document. + */ async getDocText(): Promise { const ctx = await window.GoogleAppsScript.getDocContext(); return `${ctx.beforeCursor || ''}${ctx.selectedText || ''}${ctx.afterCursor || ''}`; }, - /** Paragraphs in order — the coordinate system for `view` and inserts. */ + /** + * Paragraphs in order — the coordinate system for `view` and inserts. + * + * Markdown separates blocks with a blank line but keeps the items of one + * list on adjacent lines, so splitting on a single newline is what yields + * one entry per authored block. The blank lines that survive that split are + * separators rather than content, and dropping them keeps a list item and a + * paragraph counted the same way. + */ async getParagraphs(): Promise { const ctx = await window.GoogleAppsScript.getDocContext(); const text = `${ctx.beforeCursor || ''}${ctx.selectedText || ''}${ctx.afterCursor || ''}`; - return text.split('\n'); + return text.split('\n').filter((line) => line.trim() !== ''); }, // TODO(my-words): bridge to Apps Script (selectPhrase + replaceSelection for diff --git a/frontend/src/contexts/pageContext.tsx b/frontend/src/contexts/pageContext.tsx index 91afce0b..ec6f7860 100644 --- a/frontend/src/contexts/pageContext.tsx +++ b/frontend/src/contexts/pageContext.tsx @@ -7,6 +7,7 @@ export enum PageName { TagLinker = 'tag-linker', MyWords = 'my-words', Tools = 'tools', + Debug = 'debug', } export enum OverallMode { diff --git a/frontend/src/pages/debug/index.tsx b/frontend/src/pages/debug/index.tsx new file mode 100644 index 00000000..4199a664 --- /dev/null +++ b/frontend/src/pages/debug/index.tsx @@ -0,0 +1,166 @@ +/** + * Debug page — shows the document context exactly as the model receives it. + * + * Nothing else in the sidebar surfaces `getDocContext()`. Every page folds it + * into a prompt and shows you the model's reply, so when the serialization is + * wrong the only symptom is an answer that seems slightly confused about the + * document — which is indistinguishable from the model simply being wrong. + * That made the Google Docs Markdown mapping effectively unobservable on the + * surface it matters most. + * + * This page is deliberately unstyled-looking and read-only: it is an inspection + * window, not a feature. It works on every surface (Word, Google Docs, the + * standalone editor), which also makes it the quickest way to compare how the + * two editors serialize the same document. + */ +import { useCallback, useContext, useEffect, useState } from 'react'; +import { Button } from 'reshaped'; +import { detectPlatform } from '@/api'; +import { EditorContext } from '@/contexts/editorContext'; +import classes from './styles.module.css'; + +/** + * Where the cursor sits, or where the selection begins and ends. Rendered into + * the text because a caret is otherwise invisible in a `
` — and "is the
+ * cursor where I think it is" is most of what this page is for.
+ */
+const CURSOR_MARK = '⟦cursor⟧';
+const SELECTION_OPEN = '⟦selection⟧';
+const SELECTION_CLOSE = '⟦/selection⟧';
+
+interface Snapshot {
+	platform: string;
+	documentLabel: string;
+	annotated: string;
+	docText: string;
+	paragraphs: string[];
+	capturedAt: string;
+}
+
+/** Renders the context back into one string with the cursor made visible. */
+function annotate(context: DocContext): string {
+	const { beforeCursor, selectedText, afterCursor } = context;
+	if (selectedText) {
+		return `${beforeCursor}${SELECTION_OPEN}${selectedText}${SELECTION_CLOSE}${afterCursor}`;
+	}
+	return `${beforeCursor}${CURSOR_MARK}${afterCursor}`;
+}
+
+export default function Debug() {
+	const editorAPI = useContext(EditorContext);
+	const [snapshot, setSnapshot] = useState(null);
+	const [error, setError] = useState(null);
+	const [loading, setLoading] = useState(false);
+	const [copied, setCopied] = useState(false);
+
+	const capture = useCallback(async () => {
+		setLoading(true);
+		setError(null);
+		setCopied(false);
+		try {
+			// Sequential rather than parallel: on Google Docs each of these is an
+			// Apps Script round-trip, and firing them together has them race for
+			// the same execution slot.
+			const context = await editorAPI.getDocContext();
+			const docText = await editorAPI.getDocText();
+			const paragraphs = await editorAPI.getParagraphs();
+
+			setSnapshot({
+				platform: detectPlatform(),
+				documentLabel: context.documentLabel ?? '(none)',
+				annotated: annotate(context),
+				docText,
+				paragraphs,
+				capturedAt: new Date().toLocaleTimeString(),
+			});
+		} catch (e) {
+			setError(e instanceof Error ? e.message : String(e));
+		} finally {
+			setLoading(false);
+		}
+	}, [editorAPI]);
+
+	useEffect(() => {
+		void capture();
+	}, [capture]);
+
+	const copy = useCallback(async () => {
+		if (!snapshot) return;
+		try {
+			await navigator.clipboard.writeText(snapshot.docText);
+			setCopied(true);
+		} catch {
+			// Clipboard access is blocked in some sandboxed iframes. The text is
+			// on screen and selectable either way, so this is not worth an error.
+			setCopied(false);
+		}
+	}, [snapshot]);
+
+	return (
+		
+

+ The document exactly as the model receives it. On Google Docs + that is Markdown — headings as #, list items as{' '} + - or 1. +

+ +
+ + +
+ + {error ?

{error}

: null} + + {snapshot ? ( + <> +
+
Platform
+
{snapshot.platform}
+
Document
+
{snapshot.documentLabel}
+
Characters
+
{snapshot.docText.length}
+
Paragraphs
+
{snapshot.paragraphs.length}
+
Read at
+
{snapshot.capturedAt}
+
+ +
+

Context

+
+							{snapshot.annotated}
+						
+
+ +
+

Paragraphs

+
    + {snapshot.paragraphs.map((paragraph, index) => ( + // Position is the identity here — two paragraphs can hold the + // same text, and the list is rebuilt wholesale on every read. + // biome-ignore lint/suspicious/noArrayIndexKey: index is the identity +
  1. + {paragraph} +
  2. + ))} +
+
+ + ) : null} +
+ ); +} diff --git a/frontend/src/pages/debug/styles.module.css b/frontend/src/pages/debug/styles.module.css new file mode 100644 index 00000000..1997c506 --- /dev/null +++ b/frontend/src/pages/debug/styles.module.css @@ -0,0 +1,95 @@ +.container { + padding: 0.75rem 1rem 1.5rem; + display: flex; + flex-direction: column; + gap: 1rem; +} + +.intro { + margin: 0; + font-size: 0.9rem; + color: var(--rs-color-foreground-neutral-faded, #555); + line-height: 1.4; +} + +.intro code { + font-size: 0.85em; +} + +.actions { + display: flex; + gap: 0.5rem; +} + +.error { + background: #fdecea; + color: #b71c1c; + border: 1px solid #f5c6cb; + border-radius: 6px; + padding: 0.5rem 0.75rem; + margin: 0; + font-size: 0.85rem; +} + +.facts { + display: grid; + grid-template-columns: auto 1fr; + gap: 0.15rem 0.75rem; + margin: 0; + font-size: 0.8rem; +} + +.facts dt { + color: var(--rs-color-foreground-neutral-faded, #666); +} + +.facts dd { + margin: 0; + font-variant-numeric: tabular-nums; + /* Document titles can be long and have no break opportunities. */ + overflow-wrap: anywhere; +} + +.heading { + margin: 0 0 0.35rem; + font-size: 0.8rem; + text-transform: uppercase; + letter-spacing: 0.04em; + color: var(--rs-color-foreground-neutral-faded, #666); +} + +/* + * `pre-wrap` rather than `pre`: the sidebar is ~300px and the point is to read + * the text, so wrapping beats a horizontal scrollbar. The block still scrolls + * vertically so a long document can't push the controls off screen. + */ +.output { + margin: 0; + padding: 0.5rem 0.6rem; + background: var(--rs-color-background-neutral-faded, #f6f6f6); + border: 1px solid var(--rs-color-border-neutral-faded, #e0e0e0); + border-radius: 6px; + font-size: 0.75rem; + line-height: 1.45; + white-space: pre-wrap; + overflow-wrap: anywhere; + max-height: 24rem; + overflow-y: auto; +} + +.paragraphs { + margin: 0; + padding-left: 1.5rem; + max-height: 16rem; + overflow-y: auto; + font-size: 0.75rem; + line-height: 1.5; +} + +.paragraphs li { + margin-bottom: 0.2rem; +} + +.paragraphs code { + overflow-wrap: anywhere; +} diff --git a/frontend/src/pages/registry.tsx b/frontend/src/pages/registry.tsx index 5c781c01..453c64df 100644 --- a/frontend/src/pages/registry.tsx +++ b/frontend/src/pages/registry.tsx @@ -30,6 +30,7 @@ import type React from 'react'; import { PageName } from '@/contexts/pageContext'; import Chat from './chat'; +import Debug from './debug'; import Draft from './draft'; import { isFlagEnabled } from './flags'; import MyWords from './my-words'; @@ -101,6 +102,18 @@ export const PAGES: PageDef[] = [ enabled: () => isFlagEnabled('my-words'), render: () => , }, + { + name: PageName.Debug, + title: 'Debug', + hint: 'See what the model sees', + tier: 'lab', + // Deliberately not behind a flag, unlike My Words. A flag is set from the + // URL, and the two surfaces whose document handling this page exists to + // check — the Word task pane and the Google Docs sidebar — have no + // addressable URL (see flags.ts). Gating it would leave it reachable only + // where it is least needed. The Labs menu is the cross-surface way in. + render: () => , + }, ]; /** The page shown when nothing is selected, or when the selection isn't visible. */ diff --git a/google-docs-addon/Code.gs b/google-docs-addon/Code.gs index 2cf433b9..07da106a 100644 --- a/google-docs-addon/Code.gs +++ b/google-docs-addon/Code.gs @@ -110,137 +110,349 @@ function showSidebar() { // Document Operations (called from sidebar via google.script.run) // ============================================================================= +// ============================================================================= +// Markdown serialization +// ============================================================================= +// +// The document is serialized to Markdown rather than bare text because its only +// consumer is an LLM prompt, and Markdown is the notation those models are most +// fluent in. Bare text threw away everything the writer had marked: a heading +// arrived as an ordinary sentence, and a bulleted list arrived as prose — Docs +// renders bullets and numbers from list formatting, so the glyphs are not +// characters and `getText()` never returns them. +// +// Where Docs is richer than Markdown we flatten to the nearest Markdown form +// rather than inventing notation, because a fabricated convention is one more +// thing for the model to misread: +// +// - Title and Heading 1 both become `#`. A document using both loses that +// distinction; Markdown's top level is the document title and the overlap +// is rare enough not to justify a private convention. +// - Subtitle becomes a plain paragraph. It is not an outline level in Docs +// (it doesn't appear in the document outline), so giving it a `#` would +// claim a section boundary that isn't there. +// - Hollow and square bullets become `-`, like round ones. Ordered glyphs +// (numbers, latin, roman) all become `N.` and are numbered sequentially. +// +// Tables and images are still skipped entirely — see `serializeBody`. + +/** One nesting level of list indentation. */ +const LIST_INDENT = ' '; + /** - * Gets the document context: text before cursor, selected text, and text after cursor. - * This mirrors the DocContext interface from the frontend. - * + * The Markdown prefix for a paragraph's named style, `''` for body text. + * + * @param {ParagraphHeading} heading - The value of `Paragraph.getHeading()` + * @returns {string} A Markdown ATX heading prefix, or '' for an ordinary paragraph + */ +function headingPrefix(heading) { + const headings = DocumentApp.ParagraphHeading; + switch (heading) { + case headings.TITLE: + case headings.HEADING1: + return '# '; + case headings.HEADING2: + return '## '; + case headings.HEADING3: + return '### '; + case headings.HEADING4: + return '#### '; + case headings.HEADING5: + return '##### '; + case headings.HEADING6: + return '###### '; + default: + // NORMAL, SUBTITLE, and anything a future Docs version adds. + return ''; + } +} + +/** + * Whether a list glyph is a counter (1., a., i.) rather than a bullet. + * + * @param {GlyphType} glyph - The value of `ListItem.getGlyphType()` + * @returns {boolean} True for ordered glyphs + */ +function isOrderedGlyph(glyph) { + const glyphs = DocumentApp.GlyphType; + return ( + glyph === glyphs.NUMBER || + glyph === glyphs.LATIN_UPPER || + glyph === glyphs.LATIN_LOWER || + glyph === glyphs.ROMAN_UPPER || + glyph === glyphs.ROMAN_LOWER + ); +} + +/** + * The Markdown prefix for one list item: indentation plus its marker. + * + * Ordered items are numbered rather than all emitted as `1.`, which Markdown + * would also accept. The numbers are what make an item referenceable — a writer + * asking about "the third step" and a model answering about it need the same + * label — and they cost only the counter bookkeeping here. + * + * @param {ListItem} item - The list item to label + * @param {Object} counters - Per-list counters, carried across calls and mutated + * @returns {string} Indentation followed by `- ` or `N. ` + */ +function listItemPrefix(item, counters) { + const level = item.getNestingLevel(); + let indent = ''; + for (let i = 0; i < level; i++) indent += LIST_INDENT; + + if (!isOrderedGlyph(item.getGlyphType())) { + return indent + '- '; + } + + // Numbering is per list and per depth: `getListId()` is shared by every item + // Docs considers one list, and returning to a shallower level restarts the + // deeper ones, exactly as the editor renders them. + const listId = item.getListId(); + let counts = counters[listId]; + if (!counts) { + counts = []; + counters[listId] = counts; + } + counts.length = level + 1; + counts[level] = (counts[level] || 0) + 1; + return indent + counts[level] + '. '; +} + +/** + * Serializes a body (or tab body) to Markdown, along with an index recording + * where each top-level child's own text landed in the output. + * + * The index is what lets `getDocContext` place the cursor exactly. The previous + * implementation searched the flattened text for the selected string, which + * mislocated repeated phrases and failed outright once prefixes existed — the + * selected text of a heading does not contain the `## ` we inserted in front of + * it. Positions are now derived from the element the cursor is actually in. + * + * Tables and images are skipped, as they were before Markdown: their text never + * reached the model, and turning them into Markdown tables is its own change. + * + * @param {Body} body - The body element to serialize + * @returns {{markdown: string, blocks: Array}} + * `blocks` is indexed by body child index; entries are null for skipped children. + */ +function serializeBody(body) { + const blocks = []; + const counters = {}; + const numChildren = body.getNumChildren(); + + let markdown = ''; + let previousWasListItem = false; + + for (let i = 0; i < numChildren; i++) { + const child = body.getChild(i); + const type = child.getType(); + + let prefix = ''; + let text = ''; + let isListItem = false; + + if (type === DocumentApp.ElementType.LIST_ITEM) { + const item = child.asListItem(); + prefix = listItemPrefix(item, counters); + text = item.getText(); + isListItem = true; + } else if (type === DocumentApp.ElementType.PARAGRAPH) { + const paragraph = child.asParagraph(); + prefix = headingPrefix(paragraph.getHeading()); + text = paragraph.getText(); + } else { + // Tables, images, page breaks, anything unsupported. + blocks.push(null); + continue; + } + + // An empty paragraph is spacing the writer inserted, not content. Emitting + // it would add a second blank line on top of the one that already separates + // blocks, so record its position (the cursor can sit in one) and emit + // nothing. + if (text === '' && !isListItem) { + blocks.push({ start: markdown.length, prefixLength: 0, textLength: 0 }); + continue; + } + + if (markdown !== '') { + // Consecutive list items are one list, so a single newline. Everything + // else needs the blank line that separates Markdown blocks. + markdown += previousWasListItem && isListItem ? '\n' : '\n\n'; + } + + const start = markdown.length; + markdown += prefix + text; + blocks.push({ + start: start, + prefixLength: prefix.length, + textLength: text.length + }); + previousWasListItem = isListItem; + } + + return { markdown: markdown, blocks: blocks }; +} + +/** The Markdown rendering of a body, without the position index. */ +function bodyToMarkdown(body) { + return serializeBody(body).markdown; +} + +/** + * Finds which top-level body child contains an element, walking up from + * wherever the cursor or range actually points (usually a Text run inside a + * paragraph). + * + * @param {Body} body - The body to index into + * @param {Element} element - Any element beneath it + * @returns {?number} The body child index, or null if the element isn't under it + */ +function topLevelChildIndex(body, element) { + let current = element; + + // Bounded because a malformed parent chain must not hang the 6-minute + // execution budget; real documents nest a handful of levels at most. + for (let depth = 0; current && depth < 64; depth++) { + let parent; + try { + parent = current.getParent(); + } catch (e) { + return null; + } + if (!parent) return null; + + if (parent.getType() === DocumentApp.ElementType.BODY_SECTION) { + try { + return body.getChildIndex(current); + } catch (e) { + return null; + } + } + current = parent; + } + + return null; +} + +/** + * Maps a position inside a document element to an offset in the serialized + * Markdown. + * + * @param {Body} body - The serialized body + * @param {Object} serialized - The result of `serializeBody` + * @param {Element} element - The element the position is in + * @param {number} offsetInText - Offset within that element's text + * @returns {?number} An offset into `serialized.markdown`, or null if unmappable + */ +function markdownOffset(body, serialized, element, offsetInText) { + const index = topLevelChildIndex(body, element); + if (index === null) return null; + + const block = serialized.blocks[index]; + if (!block) return null; + + const clamped = Math.max(0, Math.min(offsetInText, block.textLength)); + return block.start + block.prefixLength + clamped; +} + +/** + * Maps one end of a selection to an offset in the serialized Markdown. + * + * A wholly-selected element starts at its Markdown prefix rather than after it, + * so selecting a heading yields `## Heading` — the prefix is part of that span + * of the document, and including it keeps + * `beforeCursor + selectedText + afterCursor` equal to the full Markdown. + * + * @param {Body} body - The serialized body + * @param {Object} serialized - The result of `serializeBody` + * @param {RangeElement} rangeElement - One element of the selection + * @param {boolean} isEnd - True for the trailing edge of the selection + * @returns {?number} An offset into `serialized.markdown`, or null if unmappable + */ +function selectionEdgeOffset(body, serialized, rangeElement, isEnd) { + const element = rangeElement.getElement(); + const index = topLevelChildIndex(body, element); + if (index === null) return null; + + const block = serialized.blocks[index]; + if (!block) return null; + + if (rangeElement.isPartial()) { + const offset = isEnd + ? rangeElement.getEndOffsetInclusive() + 1 + : rangeElement.getStartOffset(); + const clamped = Math.max(0, Math.min(offset, block.textLength)); + return block.start + block.prefixLength + clamped; + } + + return isEnd + ? block.start + block.prefixLength + block.textLength + : block.start; +} + +/** + * Gets the document context: Markdown before the cursor, the selected Markdown, + * and the Markdown after it. This mirrors the DocContext interface from the + * frontend, and the three pieces always concatenate back to the whole document. + * * @returns {Object} DocContext object with beforeCursor, selectedText, afterCursor */ function getDocContext() { const doc = DocumentApp.getActiveDocument(); const body = doc.getBody(); - - // Extract only text content, ignoring images, tables, etc. - const fullText = extractTextOnly(body); - + + const serialized = serializeBody(body); + const markdown = serialized.markdown; + const selection = doc.getSelection(); const cursor = doc.getCursor(); - - let beforeCursor = ''; - let selectedText = ''; - let afterCursor = ''; - + if (selection) { - // There's a selection const elements = selection.getRangeElements(); - if (elements.length > 0) { - // Get selected text - const selectedParts = []; - for (let i = 0; i < elements.length; i++) { - const element = elements[i]; - const text = element.getElement().asText(); - if (text) { - if (element.isPartial()) { - selectedParts.push(text.getText().substring( - element.getStartOffset(), - element.getEndOffsetInclusive() + 1 - )); - } else { - selectedParts.push(text.getText()); - } - } - } - selectedText = selectedParts.join(''); - - // Find the selection in the full text to determine before/after - const selectionStart = fullText.indexOf(selectedText); - if (selectionStart !== -1) { - beforeCursor = fullText.substring(0, selectionStart); - afterCursor = fullText.substring(selectionStart + selectedText.length); + const start = selectionEdgeOffset(body, serialized, elements[0], false); + const end = selectionEdgeOffset( + body, + serialized, + elements[elements.length - 1], + true + ); + + if (start !== null && end !== null && end >= start) { + return { + beforeCursor: markdown.substring(0, start), + selectedText: markdown.substring(start, end), + afterCursor: markdown.substring(end) + }; } } } else if (cursor) { - // There's just a cursor, no selection - const cursorElement = cursor.getElement(); - const cursorOffset = cursor.getOffset(); - - // Get the text element containing the cursor - const textElement = cursorElement.asText ? cursorElement.asText() : null; - - if (textElement) { - // Find the position in the full document - const textContent = textElement.getText(); - const textStart = fullText.indexOf(textContent); - - if (textStart !== -1) { - const absolutePosition = textStart + cursorOffset; - beforeCursor = fullText.substring(0, absolutePosition); - afterCursor = fullText.substring(absolutePosition); - } + const position = markdownOffset( + body, + serialized, + cursor.getElement(), + cursor.getOffset() + ); + if (position !== null) { + return { + beforeCursor: markdown.substring(0, position), + selectedText: '', + afterCursor: markdown.substring(position) + }; } - } else { - // No selection and no cursor - return full document as "before" - beforeCursor = fullText; } - + + // No cursor, no selection, or a position we couldn't place: the whole + // document is "before", which is what an append-at-the-end caller expects. return { - beforeCursor: beforeCursor, - selectedText: selectedText, - afterCursor: afterCursor + beforeCursor: markdown, + selectedText: '', + afterCursor: '' }; } -/** - * Extracts only text content from a document element, ignoring images, tables, etc. - * - * @param {Element} element - The document element to extract text from - * @returns {string} Text content only - */ -function extractTextOnly(element) { - let text = ''; - - // Get child elements - const numChildren = element.getNumChildren(); - for (let i = 0; i < numChildren; i++) { - const child = element.getChild(i); - const elementType = child.getType(); - - // Only process text-containing elements - if (elementType === DocumentApp.ElementType.PARAGRAPH || - elementType === DocumentApp.ElementType.LIST_ITEM || - elementType === DocumentApp.ElementType.TEXT) { - // Try to get text - try { - const childText = child.asText ? child.asText() : null; - if (childText) { - text += childText.getText(); - } - } catch (e) { - // Skip elements that can't be converted to text - continue; - } - } else if (elementType === DocumentApp.ElementType.TABLE) { - // Skip tables - continue; - } else if (elementType === DocumentApp.ElementType.INLINE_IMAGE || - elementType === DocumentApp.ElementType.IMAGE || - elementType === DocumentApp.ElementType.UNSUPPORTED) { - // Skip images and unsupported elements - continue; - } else { - // For other container types, recursively extract text - try { - text += extractTextOnly(child); - } catch (e) { - continue; - } - } - } - - return text; -} - /** * Escapes a literal string for findText, which takes a regular expression * rather than plain text. Without this, any quote containing punctuation the @@ -513,7 +725,7 @@ function getAllTabs() { return tabs.map(function(tab) { var text = ''; try { - text = extractTextOnly(tab.asDocumentTab().getBody()); + text = bodyToMarkdown(tab.asDocumentTab().getBody()); } catch (e) { /* skip */ } return { id: tab.getId(), title: tab.getTitle(), text: text }; }); @@ -522,7 +734,7 @@ function getAllTabs() { return [{ id: doc.getId(), title: 'Main', - text: extractTextOnly(doc.getBody()) + text: bodyToMarkdown(doc.getBody()) }]; } } diff --git a/google-docs-addon/README.md b/google-docs-addon/README.md index 7ca906db..847c295d 100644 --- a/google-docs-addon/README.md +++ b/google-docs-addon/README.md @@ -146,6 +146,20 @@ There is no runtime dev/prod flag — the two modes are just two versions of `dist-gdocs/sidebar-bundled.html` with all JS/CSS inlined; that file is what you `clasp push` for a real deployment. +## Document serialization + +`getDocContext()` and `getAllTabs()` return the document as **Markdown**, not +bare text: headings become `#`, list items become `-` or `1.`, and paragraphs +are separated by a blank line. The document's only consumer is an LLM prompt, +and structure the writer marked in Docs is otherwise invisible there — Docs +renders list glyphs from formatting, so `getText()` never returns them. + +See [docs/google-docs-document-serialization.md](../docs/google-docs-document-serialization.md) +for the full mapping, what is deliberately flattened, and how to test it. In +short: unit tests (`npm test` in `frontend/` — they run this file with a stubbed +`DocumentApp`, so no `clasp push` is needed to iterate) for the mapping, and the +**Debug** page in the sidebar's Labs (···) menu for the real round-trip. + ## Key Differences from Word Add-in | Aspect | Word Add-in | Google Docs Add-on | @@ -153,6 +167,7 @@ There is no runtime dev/prod flag — the two modes are just two versions of | Document API | Office.js (client-side) | DocumentApp (server-side via Apps Script) | | UI Framework | HTML/JS in taskpane | HTML/JS in sidebar | | Selection events | Native Office events | Polling (no native events) | +| Document text | Plain text | Markdown (headings, lists) | ## OAuth Scopes