diff --git a/docs/google-docs-document-serialization.md b/docs/google-docs-document-serialization.md new file mode 100644 index 00000000..8129a718 --- /dev/null +++ b/docs/google-docs-document-serialization.md @@ -0,0 +1,160 @@ +# Google Docs: how the document reaches the model + +The Google Docs add-on serializes the document to **Markdown** before it leaves +Apps Script. This note records what survives that trip, what is deliberately +flattened, what is dropped, and how to check any of it. + +The code is `serializeBody` in `google-docs-addon/Code.gs`; the tests are +`frontend/src/api/__tests__/googleDocsMarkdown.test.ts`. + +## Why Markdown + +The only consumer of the document text is an LLM prompt, and Markdown is the +notation those models read most fluently. The previous serialization was bare +text, which lost more than it looked like: + +- A heading arrived as an ordinary sentence. Nothing marked it as a heading, so + a document's structure was invisible to the model. +- A bulleted list arrived as prose. Docs renders bullets and numbers *from list + formatting*, so the glyphs are not characters — `getText()` on a list item + returns only the words, never `•` or `1.`, and nesting depth is likewise + invisible. +- Paragraphs ran together with no separator at all. `DocumentApp` text carries + no trailing newline (the paragraph boundary *is* the newline, and only + `Body.getText()` inserts it), and the old extractor concatenated paragraph + text directly. Two paragraphs reached the model as one run-on sentence, and + `getParagraphs()` on the frontend — which splits the result on `\n` — returned + the entire document as a single paragraph. + +## What survives + +| In Docs | In the serialized text | +|---|---| +| Heading 1–6 | `#` … `######` | +| Title | `#` | +| Subtitle | plain paragraph | +| Body text | plain paragraph | +| Bullet (round, hollow, square) | `- ` | +| Numbered / lettered / roman | `1. `, numbered sequentially | +| List nesting | four spaces per level | +| Paragraph break | blank line | +| Empty paragraph | nothing (it is spacing, not content) | +| Table | **dropped** | +| Image | **dropped** | +| Comments | **not available** — see below | + +## Deliberate flattening + +Where Docs is richer than Markdown we map to the nearest Markdown form rather +than invent notation, because a private convention is one more thing for the +model to misread. + +- **Title and Heading 1 both become `#`.** A document using both loses that + distinction. Markdown's top level *is* the document title, and the overlap is + rare enough not to justify a custom marker. +- **Subtitle becomes a plain paragraph.** It is not an outline level in Docs — + it doesn't appear in the document outline — so giving it a `#` would claim a + section boundary that isn't there. +- **Hollow and square bullets become `-`,** like round ones. Markdown's `*`, + `-`, and `+` are interchangeable renderers' choices, not distinct meanings. +- **All ordered glyphs become `N.`.** Latin and roman numbering are a display + choice in Docs; the ordinal is the meaning. + +Ordered items are numbered sequentially rather than all emitted as `1.` (which +Markdown also accepts). The numbers are what make an item referenceable — a +writer asking about "the third step" and a model answering about it need the +same label. + +## Cursor and selection positioning + +`getDocContext()` returns `beforeCursor` / `selectedText` / `afterCursor`, and +those three **always concatenate back to exactly the serialized document**. +Callers rely on it (`getDocText` is that concatenation), and the tests assert it +on every positioning case. + +Getting there required dropping the old approach. It searched the flattened text +for the selected string, which had two failure modes: a phrase occurring twice +resolved to the first occurrence, and once Markdown prefixes existed the search +could not succeed at all — the selected text of a heading does not contain the +`## ` in front of it. + +Positions are now derived structurally. `serializeBody` returns, alongside the +Markdown, an index recording where each top-level child's own text landed. +`topLevelChildIndex` walks up from whatever element the cursor points at +(usually a Text run inside a paragraph) to the body child containing it, and the +offset is looked up rather than searched for. + +A wholly-selected element starts at its Markdown prefix rather than after it, so +selecting a heading yields `## Heading` — the prefix is part of that span of the +document, and including it is what keeps the concatenation exact. + +If a position can't be placed (an element outside the body, such as a header or +footnote), the whole document is returned as `beforeCursor` rather than +throwing. A serialization error must not take the sidebar down. + +## Comments are not available + +`DocumentApp` has no comments API at all — not content, not authors, not +anchors. Neither does the Docs REST API v1. + +Comments live in the **Drive API** (`comments.list` on the file), which returns +`content`/`htmlContent`, `replies`, `author`, `resolved`, and +`quotedFileContent.value` — the text the comment is anchored to. The `anchor` +field is an opaque `kix`-style region string with no supported mapping to a +document offset, so locating a comment in practice means searching for its +quoted text, much as `selectPhrase()` already does. + +The blocker is scope. `appsscript.json` requests only +`documents.currentonly`, `script.container.ui`, and `userinfo.email`, and +`Code.gs` is annotated `@OnlyCurrentDoc`. Reading comments means adding a Drive +scope and enabling the Advanced Drive Service — a visible expansion of what +users are asked to authorize. Tracked separately; not part of this +serialization. + +## How to check it + +Three layers, cheapest first. + +**1. Unit tests** — `npm test` in `frontend/`. These run the real `Code.gs`: +`frontend/src/api/__tests__/helpers/appsScript.ts` reads the file and evaluates +it with a stubbed `DocumentApp`, so the tests cover the file that actually ships +rather than a copy that can drift from it. Fakes implement only the element +methods `Code.gs` calls, so anything the serializer starts calling has to be +added there. + +This is where mapping logic belongs. Iterating a glyph table through `clasp +push` + sidebar reload costs minutes per attempt; here it costs milliseconds. + +**2. The Debug page** — a `lab`-tier page (`frontend/src/pages/debug/`), +reachable from the Labs (···) menu on every surface. It shows the document +exactly as the model receives it, with the cursor position marked inline, plus +the paragraph list and character count. This is the only way to verify the real +round-trip — real document, real Apps Script, real bridge — and it works on Word +and the standalone editor too, which makes it the quickest way to compare how +the two editors serialize the same document. + +It is deliberately *not* behind a feature flag. Flags are set from the URL, and +neither the Word task pane nor the Google Docs sidebar has an addressable URL +(see `frontend/src/pages/flags.ts`), so gating it would leave it reachable only +where it is least needed. + +**3. The devtools console** — zero code, for a one-off poke. Open devtools on +the sidebar, switch the frame selector to the sandboxed userHtmlFrame, then: + +```js +await window.GoogleAppsScript.getDocContext(); +``` + +Useful when something looks wrong *below* the bridge and you want to see the +raw Apps Script response. Not a substitute for either layer above, since every +call is a full round-trip and nothing is asserted. + +## Known gaps + +- **Tables and images are dropped.** They were dropped before Markdown too, so + this is not a regression, but a Markdown serializer that silently omits a + table is a reasonable thing to fix — Markdown has table syntax. +- **Comments** — see above. +- **Text formatting** (bold, italic, links) is not carried. Docs stores it as + attributes over text ranges rather than as characters, so mapping it means + walking attribute runs within each paragraph. diff --git a/frontend/CLAUDE.md b/frontend/CLAUDE.md index 8cbd40a0..2b2cfb6d 100644 --- a/frontend/CLAUDE.md +++ b/frontend/CLAUDE.md @@ -45,6 +45,34 @@ task pane's URL comes from `manifest.xml`, and the Google Docs bundle runs insid an Apps Script sandbox iframe with no addressable URL. The Labs menu is the cross-surface way in; flags are for keeping something out of even that. +### What the model sees of the document + +`getDocContext()` returns `beforeCursor` / `selectedText` / `afterCursor`, and +those three always concatenate to the whole document — `getDocText()` is that +concatenation, so anything that breaks the invariant breaks the corpus. + +On **Google Docs** the document is serialized to Markdown in Apps Script +(`serializeBody` in `google-docs-addon/Code.gs`): headings become `#`, list +items `-` / `1.`, paragraphs separated by a blank line. Docs renders list glyphs +from formatting rather than storing them as characters, so without this the +model saw a list as unmarked prose and a heading as an ordinary sentence. +Positions are derived from the element the cursor is in, never by searching the +text for the selected string — that mislocated repeated phrases and cannot work +once prefixes exist, since a heading's selected text doesn't contain its `## `. +Tables, images, comments, and inline formatting do not survive; see +`docs/google-docs-document-serialization.md`. + +**Word** returns plain text with no such markup. A page that comes to depend on +Markdown structure needs to handle both, or the two surfaces will diverge. + +The **Debug** page (Labs menu, `src/pages/debug/`) renders the context exactly as +the model receives it, with the cursor marked inline. It is the only place the +serialization is observable — every other page folds it into a prompt, where a +bad serialization is indistinguishable from the model being confused. Reach for +it on any change to document reading. Unit tests for the Apps Script side live in +`src/api/__tests__/googleDocsMarkdown.test.ts` and run the real `Code.gs` against +a stubbed `DocumentApp`, so iterating the mapping needs no `clasp push`. + ### Document-scoped settings `EditorAPI` has `getDocumentSetting`/`setDocumentSetting` for small values that diff --git a/frontend/src/api/__tests__/googleDocsMarkdown.test.ts b/frontend/src/api/__tests__/googleDocsMarkdown.test.ts new file mode 100644 index 00000000..81e3e3f1 --- /dev/null +++ b/frontend/src/api/__tests__/googleDocsMarkdown.test.ts @@ -0,0 +1,362 @@ +/** + * Tests for the Google Docs add-on's Markdown serialization + * (`google-docs-addon/Code.gs`), run against the real file — see + * `helpers/appsScript.ts` for how and why. + * + * What these pin down is the mapping from what a writer marked in Docs to what + * the model reads. Two properties matter beyond any individual glyph: + * + * 1. Structure the writer marked survives. A heading arrives as a heading. + * 2. `beforeCursor + selectedText + afterCursor` is exactly the document. + * Every caller assumes it (`getDocText` concatenates the three), and the + * old string-search positioning quietly broke it. + */ +import { describe, expect, it } from 'vitest'; + +import { + body, + cursor, + documentWith, + GlyphType, + listItem, + loadAddonScript, + paragraph, + ParagraphHeading, + rangeElement, + selection, + table, +} from './helpers/appsScript'; + +describe('headings', () => { + it('maps each Docs heading level to its Markdown depth', () => { + const script = loadAddonScript(); + const markdown = script.bodyToMarkdown( + body([ + paragraph('Chapter', ParagraphHeading.HEADING1), + paragraph('Section', ParagraphHeading.HEADING2), + paragraph('Sub', ParagraphHeading.HEADING3), + paragraph('Deeper', ParagraphHeading.HEADING4), + paragraph('Deepest', ParagraphHeading.HEADING6), + ]), + ); + + expect(markdown).toBe( + [ + '# Chapter', + '## Section', + '### Sub', + '#### Deeper', + '###### Deepest', + ].join('\n\n'), + ); + }); + + it('leaves body text unprefixed', () => { + const script = loadAddonScript(); + const markdown = script.bodyToMarkdown( + body([paragraph('Just some prose.', ParagraphHeading.NORMAL)]), + ); + + expect(markdown).toBe('Just some prose.'); + }); + + it('flattens Title to a top-level heading and Subtitle to a paragraph', () => { + // Deliberate lossiness: Markdown has no subtitle, and Docs' Subtitle is + // not an outline level, so it must not claim a section boundary. + const script = loadAddonScript(); + const markdown = script.bodyToMarkdown( + body([ + paragraph('The Title', ParagraphHeading.TITLE), + paragraph('A subtitle', ParagraphHeading.SUBTITLE), + ]), + ); + + expect(markdown).toBe('# The Title\n\nA subtitle'); + }); + + it('treats an unrecognized named style as body text', () => { + const script = loadAddonScript(); + // Anything a future Docs version adds must degrade to a plain paragraph + // rather than crash the whole serialization. + expect(script.headingPrefix('SOMETHING_NEW' as never)).toBe(''); + }); +}); + +describe('lists', () => { + it('maps every bullet glyph to a dash', () => { + const script = loadAddonScript(); + const markdown = script.bodyToMarkdown( + body([ + listItem('Round', { glyph: GlyphType.BULLET }), + listItem('Hollow', { glyph: GlyphType.HOLLOW_BULLET }), + listItem('Square', { glyph: GlyphType.SQUARE_BULLET }), + ]), + ); + + expect(markdown).toBe('- Round\n- Hollow\n- Square'); + }); + + it('numbers ordered items sequentially', () => { + const script = loadAddonScript(); + const markdown = script.bodyToMarkdown( + body([ + listItem('First', { glyph: GlyphType.NUMBER }), + listItem('Second', { glyph: GlyphType.NUMBER }), + listItem('Third', { glyph: GlyphType.NUMBER }), + ]), + ); + + expect(markdown).toBe('1. First\n2. Second\n3. Third'); + }); + + it('maps latin and roman glyphs to ordinary numbering', () => { + const script = loadAddonScript(); + const markdown = script.bodyToMarkdown( + body([ + listItem('Alpha', { glyph: GlyphType.LATIN_LOWER }), + listItem('Beta', { glyph: GlyphType.ROMAN_UPPER }), + ]), + ); + + expect(markdown).toBe('1. Alpha\n2. Beta'); + }); + + it('indents nested items and restarts deeper numbering', () => { + const script = loadAddonScript(); + const markdown = script.bodyToMarkdown( + body([ + listItem('One', { glyph: GlyphType.NUMBER, level: 0 }), + listItem('One A', { glyph: GlyphType.NUMBER, level: 1 }), + listItem('One B', { glyph: GlyphType.NUMBER, level: 1 }), + listItem('Two', { glyph: GlyphType.NUMBER, level: 0 }), + listItem('Two A', { glyph: GlyphType.NUMBER, level: 1 }), + ]), + ); + + expect(markdown).toBe( + [ + '1. One', + ' 1. One A', + ' 2. One B', + '2. Two', + ' 1. Two A', + ].join('\n'), + ); + }); + + it('counts separate lists separately', () => { + const script = loadAddonScript(); + const markdown = script.bodyToMarkdown( + body([ + listItem('A1', { glyph: GlyphType.NUMBER, listId: 'a' }), + listItem('A2', { glyph: GlyphType.NUMBER, listId: 'a' }), + paragraph('An interruption.'), + listItem('B1', { glyph: GlyphType.NUMBER, listId: 'b' }), + ]), + ); + + expect(markdown).toBe('1. A1\n2. A2\n\nAn interruption.\n\n1. B1'); + }); + + it('keeps list items adjacent but separates them from prose', () => { + // A blank line between items would end the list; a missing one between + // the list and the paragraph would swallow the paragraph into it. + const script = loadAddonScript(); + const markdown = script.bodyToMarkdown( + body([ + paragraph('Before:'), + listItem('One'), + listItem('Two'), + paragraph('After.'), + ]), + ); + + expect(markdown).toBe('Before:\n\n- One\n- Two\n\nAfter.'); + }); +}); + +describe('block separation', () => { + it('separates paragraphs with a blank line', () => { + // The bug this replaces: paragraph text was concatenated with no + // separator at all, because DocumentApp text carries no trailing + // newline. Two paragraphs arrived as one run-on sentence. + const script = loadAddonScript(); + const markdown = script.bodyToMarkdown( + body([paragraph('First para.'), paragraph('Second para.')]), + ); + + expect(markdown).toBe('First para.\n\nSecond para.'); + }); + + it('does not emit extra blank lines for spacing paragraphs', () => { + const script = loadAddonScript(); + const markdown = script.bodyToMarkdown( + body([paragraph('First.'), paragraph(''), paragraph('Second.')]), + ); + + expect(markdown).toBe('First.\n\nSecond.'); + }); + + it('skips tables and other non-text children', () => { + const script = loadAddonScript(); + const markdown = script.bodyToMarkdown( + body([paragraph('Before.'), table(), paragraph('After.')]), + ); + + expect(markdown).toBe('Before.\n\nAfter.'); + }); + + it('serializes an empty document to an empty string', () => { + const script = loadAddonScript(); + expect(script.bodyToMarkdown(body([]))).toBe(''); + }); +}); + +describe('getDocContext positioning', () => { + /** Every context must reassemble into exactly the serialized document. */ + function expectPartition( + context: { + beforeCursor: string; + selectedText: string; + afterCursor: string; + }, + markdown: string, + ) { + expect( + context.beforeCursor + context.selectedText + context.afterCursor, + ).toBe(markdown); + } + + it('places a cursor at its offset within the paragraph text', () => { + const script = loadAddonScript(); + const intro = paragraph('Hello world'); + const rest = paragraph('Second.'); + const docBody = body([intro, rest]); + script.setActiveDocument( + documentWith(docBody, { cursor: cursor(intro.text, 5) }), + ); + + const context = script.getDocContext(); + + expect(context.beforeCursor).toBe('Hello'); + expect(context.selectedText).toBe(''); + expect(context.afterCursor).toBe(' world\n\nSecond.'); + expectPartition(context, script.bodyToMarkdown(docBody)); + }); + + it('counts the Markdown prefix as before the cursor', () => { + // The cursor's offset is into the writer's text, which starts after the + // `## ` we inserted. Adding the prefix length is what keeps them aligned. + const script = loadAddonScript(); + const heading = paragraph('Section title', ParagraphHeading.HEADING2); + const docBody = body([heading]); + script.setActiveDocument( + documentWith(docBody, { cursor: cursor(heading.text, 7) }), + ); + + const context = script.getDocContext(); + + expect(context.beforeCursor).toBe('## Section'); + expect(context.afterCursor).toBe(' title'); + }); + + it('locates a partial selection precisely', () => { + const script = loadAddonScript(); + const only = paragraph('The quick brown fox'); + const docBody = body([only]); + script.setActiveDocument( + documentWith(docBody, { + selection: selection([ + rangeElement(only.text, { start: 4, endInclusive: 8 }), + ]), + }), + ); + + const context = script.getDocContext(); + + expect(context.beforeCursor).toBe('The '); + expect(context.selectedText).toBe('quick'); + expect(context.afterCursor).toBe(' brown fox'); + expectPartition(context, script.bodyToMarkdown(docBody)); + }); + + it('spans multiple blocks, prefixes included', () => { + const script = loadAddonScript(); + const heading = paragraph('Title here', ParagraphHeading.HEADING2); + const item = listItem('A point'); + const docBody = body([ + paragraph('Intro.'), + heading, + item, + paragraph('End.'), + ]); + script.setActiveDocument( + documentWith(docBody, { + selection: selection([ + rangeElement(heading), + rangeElement(item), + ]), + }), + ); + + const context = script.getDocContext(); + + expect(context.beforeCursor).toBe('Intro.\n\n'); + expect(context.selectedText).toBe('## Title here\n\n- A point'); + expect(context.afterCursor).toBe('\n\nEnd.'); + expectPartition(context, script.bodyToMarkdown(docBody)); + }); + + it('does not mislocate a phrase that occurs twice', () => { + // The old implementation searched the document for the selected string, + // so selecting the *second* "the plan" reported the first one's position. + const script = loadAddonScript(); + const first = paragraph('We discussed the plan'); + const second = paragraph('We revisited the plan'); + const docBody = body([first, second]); + script.setActiveDocument( + documentWith(docBody, { + selection: selection([ + rangeElement(second.text, { start: 13, endInclusive: 20 }), + ]), + }), + ); + + const context = script.getDocContext(); + + expect(context.selectedText).toBe('the plan'); + expect(context.beforeCursor).toBe( + 'We discussed the plan\n\nWe revisited ', + ); + expect(context.afterCursor).toBe(''); + }); + + it('returns the whole document as "before" when nothing is focused', () => { + const script = loadAddonScript(); + const docBody = body([paragraph('Alone.')]); + script.setActiveDocument(documentWith(docBody)); + + const context = script.getDocContext(); + + expect(context.beforeCursor).toBe('Alone.'); + expect(context.selectedText).toBe(''); + expect(context.afterCursor).toBe(''); + }); + + it('falls back to the whole document when a position cannot be placed', () => { + // An element outside the body (a header, a footnote) has no offset in the + // body's Markdown. Degrading to "everything is before" beats throwing, + // which would take the sidebar down. + const script = loadAddonScript(); + const orphan = paragraph('Detached'); + const docBody = body([paragraph('Real content.')]); + script.setActiveDocument( + documentWith(docBody, { cursor: cursor(orphan.text, 2) }), + ); + + const context = script.getDocContext(); + + expect(context.beforeCursor).toBe('Real content.'); + expect(context.selectedText).toBe(''); + }); +}); diff --git a/frontend/src/api/__tests__/helpers/appsScript.ts b/frontend/src/api/__tests__/helpers/appsScript.ts new file mode 100644 index 00000000..1a12030e --- /dev/null +++ b/frontend/src/api/__tests__/helpers/appsScript.ts @@ -0,0 +1,316 @@ +/** + * A test harness for `google-docs-addon/Code.gs`. + * + * The add-on's Markdown serialization is the only genuinely tricky logic in the + * Google Docs surface, and it is also the logic that is most expensive to check + * by hand: verifying it in place means `clasp push`, reloading the sidebar, and + * reading the result through a UI. That loop is far too slow to iterate a glyph + * table on. + * + * So we run the real file here. `Code.gs` is plain ES2015 that only touches + * Apps Script through globals, so evaluating it with a stubbed `DocumentApp` + * gives us the shipped source under test — not a copy that can drift from it. + * The alternative, extracting the serializer into a module both the add-on and + * the tests import, would mean either a second npm package for a folder clasp + * pushes verbatim or a `module.exports` guard in code that has no module + * system. Evaluating the file avoids both and keeps `Code.gs` readable as what + * it is: an Apps Script file. + * + * The fakes below implement only the element methods `Code.gs` actually calls. + * They are duck types, not a DocumentApp emulator — anything the serializer + * starts calling has to be added here, which is the intended pressure. + */ +import { readFileSync } from 'node:fs'; +import { dirname, resolve } from 'node:path'; +import { fileURLToPath } from 'node:url'; + +const CODE_GS = resolve( + dirname(fileURLToPath(import.meta.url)), + '../../../../../google-docs-addon/Code.gs', +); + +/** Enum values are compared by identity in Code.gs, so plain strings suffice. */ +export const ElementType = { + BODY_SECTION: 'BODY_SECTION', + PARAGRAPH: 'PARAGRAPH', + LIST_ITEM: 'LIST_ITEM', + TEXT: 'TEXT', + TABLE: 'TABLE', + INLINE_IMAGE: 'INLINE_IMAGE', +} as const; + +export const ParagraphHeading = { + NORMAL: 'NORMAL', + TITLE: 'TITLE', + SUBTITLE: 'SUBTITLE', + HEADING1: 'HEADING1', + HEADING2: 'HEADING2', + HEADING3: 'HEADING3', + HEADING4: 'HEADING4', + HEADING5: 'HEADING5', + HEADING6: 'HEADING6', +} as const; + +export const GlyphType = { + BULLET: 'BULLET', + HOLLOW_BULLET: 'HOLLOW_BULLET', + SQUARE_BULLET: 'SQUARE_BULLET', + NUMBER: 'NUMBER', + LATIN_UPPER: 'LATIN_UPPER', + LATIN_LOWER: 'LATIN_LOWER', + ROMAN_UPPER: 'ROMAN_UPPER', + ROMAN_LOWER: 'ROMAN_LOWER', +} as const; + +type Heading = (typeof ParagraphHeading)[keyof typeof ParagraphHeading]; +type Glyph = (typeof GlyphType)[keyof typeof GlyphType]; + +export interface FakeElement { + getType(): string; + getParent(): FakeElement | null; + getText?(): string; +} + +interface Mutable { + parent: FakeElement | null; +} + +/** + * The Text run inside a paragraph. Real cursors point at one of these rather + * than at the paragraph, which is why `topLevelChildIndex` has to walk up. + */ +export interface FakeText extends FakeElement { + getText(): string; +} + +export interface FakeParagraph extends FakeElement { + asParagraph(): FakeParagraph; + getHeading(): Heading; + getText(): string; + /** The Text run a cursor would be placed in. */ + text: FakeText; +} + +export interface FakeListItem extends FakeElement { + asListItem(): FakeListItem; + getText(): string; + getGlyphType(): Glyph; + getNestingLevel(): number; + getListId(): string; + text: FakeText; +} + +export interface FakeBody extends FakeElement { + getNumChildren(): number; + getChild(index: number): FakeElement; + getChildIndex(child: FakeElement): number; +} + +function makeTextRun(owner: () => FakeElement, read: () => string): FakeText { + return { + getType: () => ElementType.TEXT, + getParent: () => owner(), + getText: read, + }; +} + +/** A paragraph, optionally carrying a named style. */ +export function paragraph( + content: string, + heading: Heading = ParagraphHeading.NORMAL, +): FakeParagraph { + const state: Mutable = { parent: null }; + const self = { + getType: () => ElementType.PARAGRAPH, + getParent: () => state.parent, + asParagraph: () => self, + getHeading: () => heading, + getText: () => content, + } as FakeParagraph; + self.text = makeTextRun( + () => self, + () => content, + ); + // biome-ignore lint/suspicious/noExplicitAny: attach the mutable slot + (self as any).__state = state; + return self; +} + +/** A list item. `listId` groups items Docs considers one list. */ +export function listItem( + content: string, + options: { + glyph?: Glyph; + level?: number; + listId?: string; + } = {}, +): FakeListItem { + const glyph = options.glyph ?? GlyphType.BULLET; + const level = options.level ?? 0; + const listId = options.listId ?? 'list-1'; + const state: Mutable = { parent: null }; + const self = { + getType: () => ElementType.LIST_ITEM, + getParent: () => state.parent, + asListItem: () => self, + getText: () => content, + getGlyphType: () => glyph, + getNestingLevel: () => level, + getListId: () => listId, + } as FakeListItem; + self.text = makeTextRun( + () => self, + () => content, + ); + // biome-ignore lint/suspicious/noExplicitAny: attach the mutable slot + (self as any).__state = state; + return self; +} + +/** A non-text child, to check that it is skipped rather than serialized. */ +export function table(): FakeElement { + const state: Mutable = { parent: null }; + const self = { + getType: () => ElementType.TABLE, + getParent: () => state.parent, + } as FakeElement; + // biome-ignore lint/suspicious/noExplicitAny: attach the mutable slot + (self as any).__state = state; + return self; +} + +/** Assembles children into a body and wires up their parent links. */ +export function body(children: FakeElement[]): FakeBody { + const self = { + getType: () => ElementType.BODY_SECTION, + getParent: () => null, + getNumChildren: () => children.length, + getChild: (index: number) => children[index], + getChildIndex: (child: FakeElement) => { + const index = children.indexOf(child); + if (index === -1) + throw new Error('element is not a child of this body'); + return index; + }, + } as FakeBody; + for (const child of children) { + // biome-ignore lint/suspicious/noExplicitAny: read back the mutable slot + const state = (child as any).__state as Mutable | undefined; + if (state) state.parent = self; + } + return self; +} + +/** A cursor, as `Document.getCursor()` returns it. */ +export function cursor(element: FakeElement, offset: number) { + return { + getElement: () => element, + getOffset: () => offset, + }; +} + +/** + * One element of a selection. A partial range covers part of an element's text; + * a non-partial range covers the whole element. + */ +export function rangeElement( + element: FakeElement, + range?: { start: number; endInclusive: number }, +) { + return { + getElement: () => element, + isPartial: () => range !== undefined, + getStartOffset: () => range?.start ?? -1, + getEndOffsetInclusive: () => range?.endInclusive ?? -1, + }; +} + +/** A selection, as `Document.getSelection()` returns it. */ +export function selection(elements: ReturnType[]) { + return { getRangeElements: () => elements }; +} + +export interface DocContext { + beforeCursor: string; + selectedText: string; + afterCursor: string; +} + +/** The subset of Code.gs the tests drive. */ +export interface AddonScript { + bodyToMarkdown(body: FakeBody): string; + serializeBody(body: FakeBody): { + markdown: string; + blocks: Array<{ + start: number; + prefixLength: number; + textLength: number; + } | null>; + }; + getDocContext(): DocContext; + headingPrefix(heading: Heading): string; + setActiveDocument(document: { + getBody(): FakeBody; + getSelection(): unknown; + getCursor(): unknown; + }): void; +} + +/** + * Evaluates `Code.gs` against a stubbed `DocumentApp` and returns the functions + * under test. Read once, re-evaluated per call so tests never share state. + */ +export function loadAddonScript(): AddonScript { + const source = readFileSync(CODE_GS, 'utf8'); + + let activeDocument: unknown = null; + const DocumentApp = { + ElementType, + ParagraphHeading, + GlyphType, + getActiveDocument: () => activeDocument, + }; + + // Evaluating the source is the whole point: it is what makes these tests + // cover the file that actually ships to Apps Script rather than a duplicate + // of it. The input is a repo file read from disk at a fixed path, never user + // or network data, so the usual injection concern behind this rule does not + // apply. `DocumentApp` is passed as a parameter rather than left global so + // the stub cannot leak into other tests in the same worker. + // eslint-disable-next-line @typescript-eslint/no-implied-eval + const factory = new Function( + 'DocumentApp', + `${source} + return { + bodyToMarkdown: bodyToMarkdown, + serializeBody: serializeBody, + getDocContext: getDocContext, + headingPrefix: headingPrefix, + };`, + ); + + // eslint-disable-next-line @typescript-eslint/no-unsafe-call + const exported = factory(DocumentApp) as Omit< + AddonScript, + 'setActiveDocument' + >; + + return { + ...exported, + setActiveDocument(document) { + activeDocument = document; + }, + }; +} + +/** A document with no cursor and no selection. */ +export function documentWith( + docBody: FakeBody, + options: { cursor?: unknown; selection?: unknown } = {}, +) { + return { + getBody: () => docBody, + getSelection: () => options.selection ?? null, + getCursor: () => options.cursor ?? null, + }; +} diff --git a/frontend/src/api/googleDocsEditorAPI.ts b/frontend/src/api/googleDocsEditorAPI.ts index ec8d8452..17e800d5 100644 --- a/frontend/src/api/googleDocsEditorAPI.ts +++ b/frontend/src/api/googleDocsEditorAPI.ts @@ -317,17 +317,32 @@ export const googleDocsEditorAPI: EditorAPI = { } }, - /** Full document text, used for the corpus and the `view` tool. */ + /** + * Full document text, used for the corpus and the `view` tool. + * + * Apps Script serializes the document to Markdown (`serializeBody` in + * `google-docs-addon/Code.gs`), so headings arrive as `##` and list items as + * `-`/`N.` rather than as unmarked prose. The three context pieces always + * concatenate back to the whole document. + */ async getDocText(): Promise { const ctx = await window.GoogleAppsScript.getDocContext(); return `${ctx.beforeCursor || ''}${ctx.selectedText || ''}${ctx.afterCursor || ''}`; }, - /** Paragraphs in order — the coordinate system for `view` and inserts. */ + /** + * Paragraphs in order — the coordinate system for `view` and inserts. + * + * Markdown separates blocks with a blank line but keeps the items of one + * list on adjacent lines, so splitting on a single newline is what yields + * one entry per authored block. The blank lines that survive that split are + * separators rather than content, and dropping them keeps a list item and a + * paragraph counted the same way. + */ async getParagraphs(): Promise { const ctx = await window.GoogleAppsScript.getDocContext(); const text = `${ctx.beforeCursor || ''}${ctx.selectedText || ''}${ctx.afterCursor || ''}`; - return text.split('\n'); + return text.split('\n').filter((line) => line.trim() !== ''); }, // TODO(my-words): bridge to Apps Script (selectPhrase + replaceSelection for diff --git a/frontend/src/contexts/pageContext.tsx b/frontend/src/contexts/pageContext.tsx index 91afce0b..ec6f7860 100644 --- a/frontend/src/contexts/pageContext.tsx +++ b/frontend/src/contexts/pageContext.tsx @@ -7,6 +7,7 @@ export enum PageName { TagLinker = 'tag-linker', MyWords = 'my-words', Tools = 'tools', + Debug = 'debug', } export enum OverallMode { diff --git a/frontend/src/pages/debug/index.tsx b/frontend/src/pages/debug/index.tsx new file mode 100644 index 00000000..4199a664 --- /dev/null +++ b/frontend/src/pages/debug/index.tsx @@ -0,0 +1,166 @@ +/** + * Debug page — shows the document context exactly as the model receives it. + * + * Nothing else in the sidebar surfaces `getDocContext()`. Every page folds it + * into a prompt and shows you the model's reply, so when the serialization is + * wrong the only symptom is an answer that seems slightly confused about the + * document — which is indistinguishable from the model simply being wrong. + * That made the Google Docs Markdown mapping effectively unobservable on the + * surface it matters most. + * + * This page is deliberately unstyled-looking and read-only: it is an inspection + * window, not a feature. It works on every surface (Word, Google Docs, the + * standalone editor), which also makes it the quickest way to compare how the + * two editors serialize the same document. + */ +import { useCallback, useContext, useEffect, useState } from 'react'; +import { Button } from 'reshaped'; +import { detectPlatform } from '@/api'; +import { EditorContext } from '@/contexts/editorContext'; +import classes from './styles.module.css'; + +/** + * Where the cursor sits, or where the selection begins and ends. Rendered into + * the text because a caret is otherwise invisible in a `
` — and "is the
+ * cursor where I think it is" is most of what this page is for.
+ */
+const CURSOR_MARK = '⟦cursor⟧';
+const SELECTION_OPEN = '⟦selection⟧';
+const SELECTION_CLOSE = '⟦/selection⟧';
+
+interface Snapshot {
+	platform: string;
+	documentLabel: string;
+	annotated: string;
+	docText: string;
+	paragraphs: string[];
+	capturedAt: string;
+}
+
+/** Renders the context back into one string with the cursor made visible. */
+function annotate(context: DocContext): string {
+	const { beforeCursor, selectedText, afterCursor } = context;
+	if (selectedText) {
+		return `${beforeCursor}${SELECTION_OPEN}${selectedText}${SELECTION_CLOSE}${afterCursor}`;
+	}
+	return `${beforeCursor}${CURSOR_MARK}${afterCursor}`;
+}
+
+export default function Debug() {
+	const editorAPI = useContext(EditorContext);
+	const [snapshot, setSnapshot] = useState(null);
+	const [error, setError] = useState(null);
+	const [loading, setLoading] = useState(false);
+	const [copied, setCopied] = useState(false);
+
+	const capture = useCallback(async () => {
+		setLoading(true);
+		setError(null);
+		setCopied(false);
+		try {
+			// Sequential rather than parallel: on Google Docs each of these is an
+			// Apps Script round-trip, and firing them together has them race for
+			// the same execution slot.
+			const context = await editorAPI.getDocContext();
+			const docText = await editorAPI.getDocText();
+			const paragraphs = await editorAPI.getParagraphs();
+
+			setSnapshot({
+				platform: detectPlatform(),
+				documentLabel: context.documentLabel ?? '(none)',
+				annotated: annotate(context),
+				docText,
+				paragraphs,
+				capturedAt: new Date().toLocaleTimeString(),
+			});
+		} catch (e) {
+			setError(e instanceof Error ? e.message : String(e));
+		} finally {
+			setLoading(false);
+		}
+	}, [editorAPI]);
+
+	useEffect(() => {
+		void capture();
+	}, [capture]);
+
+	const copy = useCallback(async () => {
+		if (!snapshot) return;
+		try {
+			await navigator.clipboard.writeText(snapshot.docText);
+			setCopied(true);
+		} catch {
+			// Clipboard access is blocked in some sandboxed iframes. The text is
+			// on screen and selectable either way, so this is not worth an error.
+			setCopied(false);
+		}
+	}, [snapshot]);
+
+	return (
+		
+

+ The document exactly as the model receives it. On Google Docs + that is Markdown — headings as #, list items as{' '} + - or 1. +

+ +
+ + +
+ + {error ?

{error}

: null} + + {snapshot ? ( + <> +
+
Platform
+
{snapshot.platform}
+
Document
+
{snapshot.documentLabel}
+
Characters
+
{snapshot.docText.length}
+
Paragraphs
+
{snapshot.paragraphs.length}
+
Read at
+
{snapshot.capturedAt}
+
+ +
+

Context

+
+							{snapshot.annotated}
+						
+
+ +
+

Paragraphs

+
    + {snapshot.paragraphs.map((paragraph, index) => ( + // Position is the identity here — two paragraphs can hold the + // same text, and the list is rebuilt wholesale on every read. + // biome-ignore lint/suspicious/noArrayIndexKey: index is the identity +
  1. + {paragraph} +
  2. + ))} +
+
+ + ) : null} +
+ ); +} diff --git a/frontend/src/pages/debug/styles.module.css b/frontend/src/pages/debug/styles.module.css new file mode 100644 index 00000000..1997c506 --- /dev/null +++ b/frontend/src/pages/debug/styles.module.css @@ -0,0 +1,95 @@ +.container { + padding: 0.75rem 1rem 1.5rem; + display: flex; + flex-direction: column; + gap: 1rem; +} + +.intro { + margin: 0; + font-size: 0.9rem; + color: var(--rs-color-foreground-neutral-faded, #555); + line-height: 1.4; +} + +.intro code { + font-size: 0.85em; +} + +.actions { + display: flex; + gap: 0.5rem; +} + +.error { + background: #fdecea; + color: #b71c1c; + border: 1px solid #f5c6cb; + border-radius: 6px; + padding: 0.5rem 0.75rem; + margin: 0; + font-size: 0.85rem; +} + +.facts { + display: grid; + grid-template-columns: auto 1fr; + gap: 0.15rem 0.75rem; + margin: 0; + font-size: 0.8rem; +} + +.facts dt { + color: var(--rs-color-foreground-neutral-faded, #666); +} + +.facts dd { + margin: 0; + font-variant-numeric: tabular-nums; + /* Document titles can be long and have no break opportunities. */ + overflow-wrap: anywhere; +} + +.heading { + margin: 0 0 0.35rem; + font-size: 0.8rem; + text-transform: uppercase; + letter-spacing: 0.04em; + color: var(--rs-color-foreground-neutral-faded, #666); +} + +/* + * `pre-wrap` rather than `pre`: the sidebar is ~300px and the point is to read + * the text, so wrapping beats a horizontal scrollbar. The block still scrolls + * vertically so a long document can't push the controls off screen. + */ +.output { + margin: 0; + padding: 0.5rem 0.6rem; + background: var(--rs-color-background-neutral-faded, #f6f6f6); + border: 1px solid var(--rs-color-border-neutral-faded, #e0e0e0); + border-radius: 6px; + font-size: 0.75rem; + line-height: 1.45; + white-space: pre-wrap; + overflow-wrap: anywhere; + max-height: 24rem; + overflow-y: auto; +} + +.paragraphs { + margin: 0; + padding-left: 1.5rem; + max-height: 16rem; + overflow-y: auto; + font-size: 0.75rem; + line-height: 1.5; +} + +.paragraphs li { + margin-bottom: 0.2rem; +} + +.paragraphs code { + overflow-wrap: anywhere; +} diff --git a/frontend/src/pages/registry.tsx b/frontend/src/pages/registry.tsx index 5c781c01..453c64df 100644 --- a/frontend/src/pages/registry.tsx +++ b/frontend/src/pages/registry.tsx @@ -30,6 +30,7 @@ import type React from 'react'; import { PageName } from '@/contexts/pageContext'; import Chat from './chat'; +import Debug from './debug'; import Draft from './draft'; import { isFlagEnabled } from './flags'; import MyWords from './my-words'; @@ -101,6 +102,18 @@ export const PAGES: PageDef[] = [ enabled: () => isFlagEnabled('my-words'), render: () => , }, + { + name: PageName.Debug, + title: 'Debug', + hint: 'See what the model sees', + tier: 'lab', + // Deliberately not behind a flag, unlike My Words. A flag is set from the + // URL, and the two surfaces whose document handling this page exists to + // check — the Word task pane and the Google Docs sidebar — have no + // addressable URL (see flags.ts). Gating it would leave it reachable only + // where it is least needed. The Labs menu is the cross-surface way in. + render: () => , + }, ]; /** The page shown when nothing is selected, or when the selection isn't visible. */ diff --git a/google-docs-addon/Code.gs b/google-docs-addon/Code.gs index 2cf433b9..07da106a 100644 --- a/google-docs-addon/Code.gs +++ b/google-docs-addon/Code.gs @@ -110,137 +110,349 @@ function showSidebar() { // Document Operations (called from sidebar via google.script.run) // ============================================================================= +// ============================================================================= +// Markdown serialization +// ============================================================================= +// +// The document is serialized to Markdown rather than bare text because its only +// consumer is an LLM prompt, and Markdown is the notation those models are most +// fluent in. Bare text threw away everything the writer had marked: a heading +// arrived as an ordinary sentence, and a bulleted list arrived as prose — Docs +// renders bullets and numbers from list formatting, so the glyphs are not +// characters and `getText()` never returns them. +// +// Where Docs is richer than Markdown we flatten to the nearest Markdown form +// rather than inventing notation, because a fabricated convention is one more +// thing for the model to misread: +// +// - Title and Heading 1 both become `#`. A document using both loses that +// distinction; Markdown's top level is the document title and the overlap +// is rare enough not to justify a private convention. +// - Subtitle becomes a plain paragraph. It is not an outline level in Docs +// (it doesn't appear in the document outline), so giving it a `#` would +// claim a section boundary that isn't there. +// - Hollow and square bullets become `-`, like round ones. Ordered glyphs +// (numbers, latin, roman) all become `N.` and are numbered sequentially. +// +// Tables and images are still skipped entirely — see `serializeBody`. + +/** One nesting level of list indentation. */ +const LIST_INDENT = ' '; + /** - * Gets the document context: text before cursor, selected text, and text after cursor. - * This mirrors the DocContext interface from the frontend. - * + * The Markdown prefix for a paragraph's named style, `''` for body text. + * + * @param {ParagraphHeading} heading - The value of `Paragraph.getHeading()` + * @returns {string} A Markdown ATX heading prefix, or '' for an ordinary paragraph + */ +function headingPrefix(heading) { + const headings = DocumentApp.ParagraphHeading; + switch (heading) { + case headings.TITLE: + case headings.HEADING1: + return '# '; + case headings.HEADING2: + return '## '; + case headings.HEADING3: + return '### '; + case headings.HEADING4: + return '#### '; + case headings.HEADING5: + return '##### '; + case headings.HEADING6: + return '###### '; + default: + // NORMAL, SUBTITLE, and anything a future Docs version adds. + return ''; + } +} + +/** + * Whether a list glyph is a counter (1., a., i.) rather than a bullet. + * + * @param {GlyphType} glyph - The value of `ListItem.getGlyphType()` + * @returns {boolean} True for ordered glyphs + */ +function isOrderedGlyph(glyph) { + const glyphs = DocumentApp.GlyphType; + return ( + glyph === glyphs.NUMBER || + glyph === glyphs.LATIN_UPPER || + glyph === glyphs.LATIN_LOWER || + glyph === glyphs.ROMAN_UPPER || + glyph === glyphs.ROMAN_LOWER + ); +} + +/** + * The Markdown prefix for one list item: indentation plus its marker. + * + * Ordered items are numbered rather than all emitted as `1.`, which Markdown + * would also accept. The numbers are what make an item referenceable — a writer + * asking about "the third step" and a model answering about it need the same + * label — and they cost only the counter bookkeeping here. + * + * @param {ListItem} item - The list item to label + * @param {Object} counters - Per-list counters, carried across calls and mutated + * @returns {string} Indentation followed by `- ` or `N. ` + */ +function listItemPrefix(item, counters) { + const level = item.getNestingLevel(); + let indent = ''; + for (let i = 0; i < level; i++) indent += LIST_INDENT; + + if (!isOrderedGlyph(item.getGlyphType())) { + return indent + '- '; + } + + // Numbering is per list and per depth: `getListId()` is shared by every item + // Docs considers one list, and returning to a shallower level restarts the + // deeper ones, exactly as the editor renders them. + const listId = item.getListId(); + let counts = counters[listId]; + if (!counts) { + counts = []; + counters[listId] = counts; + } + counts.length = level + 1; + counts[level] = (counts[level] || 0) + 1; + return indent + counts[level] + '. '; +} + +/** + * Serializes a body (or tab body) to Markdown, along with an index recording + * where each top-level child's own text landed in the output. + * + * The index is what lets `getDocContext` place the cursor exactly. The previous + * implementation searched the flattened text for the selected string, which + * mislocated repeated phrases and failed outright once prefixes existed — the + * selected text of a heading does not contain the `## ` we inserted in front of + * it. Positions are now derived from the element the cursor is actually in. + * + * Tables and images are skipped, as they were before Markdown: their text never + * reached the model, and turning them into Markdown tables is its own change. + * + * @param {Body} body - The body element to serialize + * @returns {{markdown: string, blocks: Array}} + * `blocks` is indexed by body child index; entries are null for skipped children. + */ +function serializeBody(body) { + const blocks = []; + const counters = {}; + const numChildren = body.getNumChildren(); + + let markdown = ''; + let previousWasListItem = false; + + for (let i = 0; i < numChildren; i++) { + const child = body.getChild(i); + const type = child.getType(); + + let prefix = ''; + let text = ''; + let isListItem = false; + + if (type === DocumentApp.ElementType.LIST_ITEM) { + const item = child.asListItem(); + prefix = listItemPrefix(item, counters); + text = item.getText(); + isListItem = true; + } else if (type === DocumentApp.ElementType.PARAGRAPH) { + const paragraph = child.asParagraph(); + prefix = headingPrefix(paragraph.getHeading()); + text = paragraph.getText(); + } else { + // Tables, images, page breaks, anything unsupported. + blocks.push(null); + continue; + } + + // An empty paragraph is spacing the writer inserted, not content. Emitting + // it would add a second blank line on top of the one that already separates + // blocks, so record its position (the cursor can sit in one) and emit + // nothing. + if (text === '' && !isListItem) { + blocks.push({ start: markdown.length, prefixLength: 0, textLength: 0 }); + continue; + } + + if (markdown !== '') { + // Consecutive list items are one list, so a single newline. Everything + // else needs the blank line that separates Markdown blocks. + markdown += previousWasListItem && isListItem ? '\n' : '\n\n'; + } + + const start = markdown.length; + markdown += prefix + text; + blocks.push({ + start: start, + prefixLength: prefix.length, + textLength: text.length + }); + previousWasListItem = isListItem; + } + + return { markdown: markdown, blocks: blocks }; +} + +/** The Markdown rendering of a body, without the position index. */ +function bodyToMarkdown(body) { + return serializeBody(body).markdown; +} + +/** + * Finds which top-level body child contains an element, walking up from + * wherever the cursor or range actually points (usually a Text run inside a + * paragraph). + * + * @param {Body} body - The body to index into + * @param {Element} element - Any element beneath it + * @returns {?number} The body child index, or null if the element isn't under it + */ +function topLevelChildIndex(body, element) { + let current = element; + + // Bounded because a malformed parent chain must not hang the 6-minute + // execution budget; real documents nest a handful of levels at most. + for (let depth = 0; current && depth < 64; depth++) { + let parent; + try { + parent = current.getParent(); + } catch (e) { + return null; + } + if (!parent) return null; + + if (parent.getType() === DocumentApp.ElementType.BODY_SECTION) { + try { + return body.getChildIndex(current); + } catch (e) { + return null; + } + } + current = parent; + } + + return null; +} + +/** + * Maps a position inside a document element to an offset in the serialized + * Markdown. + * + * @param {Body} body - The serialized body + * @param {Object} serialized - The result of `serializeBody` + * @param {Element} element - The element the position is in + * @param {number} offsetInText - Offset within that element's text + * @returns {?number} An offset into `serialized.markdown`, or null if unmappable + */ +function markdownOffset(body, serialized, element, offsetInText) { + const index = topLevelChildIndex(body, element); + if (index === null) return null; + + const block = serialized.blocks[index]; + if (!block) return null; + + const clamped = Math.max(0, Math.min(offsetInText, block.textLength)); + return block.start + block.prefixLength + clamped; +} + +/** + * Maps one end of a selection to an offset in the serialized Markdown. + * + * A wholly-selected element starts at its Markdown prefix rather than after it, + * so selecting a heading yields `## Heading` — the prefix is part of that span + * of the document, and including it keeps + * `beforeCursor + selectedText + afterCursor` equal to the full Markdown. + * + * @param {Body} body - The serialized body + * @param {Object} serialized - The result of `serializeBody` + * @param {RangeElement} rangeElement - One element of the selection + * @param {boolean} isEnd - True for the trailing edge of the selection + * @returns {?number} An offset into `serialized.markdown`, or null if unmappable + */ +function selectionEdgeOffset(body, serialized, rangeElement, isEnd) { + const element = rangeElement.getElement(); + const index = topLevelChildIndex(body, element); + if (index === null) return null; + + const block = serialized.blocks[index]; + if (!block) return null; + + if (rangeElement.isPartial()) { + const offset = isEnd + ? rangeElement.getEndOffsetInclusive() + 1 + : rangeElement.getStartOffset(); + const clamped = Math.max(0, Math.min(offset, block.textLength)); + return block.start + block.prefixLength + clamped; + } + + return isEnd + ? block.start + block.prefixLength + block.textLength + : block.start; +} + +/** + * Gets the document context: Markdown before the cursor, the selected Markdown, + * and the Markdown after it. This mirrors the DocContext interface from the + * frontend, and the three pieces always concatenate back to the whole document. + * * @returns {Object} DocContext object with beforeCursor, selectedText, afterCursor */ function getDocContext() { const doc = DocumentApp.getActiveDocument(); const body = doc.getBody(); - - // Extract only text content, ignoring images, tables, etc. - const fullText = extractTextOnly(body); - + + const serialized = serializeBody(body); + const markdown = serialized.markdown; + const selection = doc.getSelection(); const cursor = doc.getCursor(); - - let beforeCursor = ''; - let selectedText = ''; - let afterCursor = ''; - + if (selection) { - // There's a selection const elements = selection.getRangeElements(); - if (elements.length > 0) { - // Get selected text - const selectedParts = []; - for (let i = 0; i < elements.length; i++) { - const element = elements[i]; - const text = element.getElement().asText(); - if (text) { - if (element.isPartial()) { - selectedParts.push(text.getText().substring( - element.getStartOffset(), - element.getEndOffsetInclusive() + 1 - )); - } else { - selectedParts.push(text.getText()); - } - } - } - selectedText = selectedParts.join(''); - - // Find the selection in the full text to determine before/after - const selectionStart = fullText.indexOf(selectedText); - if (selectionStart !== -1) { - beforeCursor = fullText.substring(0, selectionStart); - afterCursor = fullText.substring(selectionStart + selectedText.length); + const start = selectionEdgeOffset(body, serialized, elements[0], false); + const end = selectionEdgeOffset( + body, + serialized, + elements[elements.length - 1], + true + ); + + if (start !== null && end !== null && end >= start) { + return { + beforeCursor: markdown.substring(0, start), + selectedText: markdown.substring(start, end), + afterCursor: markdown.substring(end) + }; } } } else if (cursor) { - // There's just a cursor, no selection - const cursorElement = cursor.getElement(); - const cursorOffset = cursor.getOffset(); - - // Get the text element containing the cursor - const textElement = cursorElement.asText ? cursorElement.asText() : null; - - if (textElement) { - // Find the position in the full document - const textContent = textElement.getText(); - const textStart = fullText.indexOf(textContent); - - if (textStart !== -1) { - const absolutePosition = textStart + cursorOffset; - beforeCursor = fullText.substring(0, absolutePosition); - afterCursor = fullText.substring(absolutePosition); - } + const position = markdownOffset( + body, + serialized, + cursor.getElement(), + cursor.getOffset() + ); + if (position !== null) { + return { + beforeCursor: markdown.substring(0, position), + selectedText: '', + afterCursor: markdown.substring(position) + }; } - } else { - // No selection and no cursor - return full document as "before" - beforeCursor = fullText; } - + + // No cursor, no selection, or a position we couldn't place: the whole + // document is "before", which is what an append-at-the-end caller expects. return { - beforeCursor: beforeCursor, - selectedText: selectedText, - afterCursor: afterCursor + beforeCursor: markdown, + selectedText: '', + afterCursor: '' }; } -/** - * Extracts only text content from a document element, ignoring images, tables, etc. - * - * @param {Element} element - The document element to extract text from - * @returns {string} Text content only - */ -function extractTextOnly(element) { - let text = ''; - - // Get child elements - const numChildren = element.getNumChildren(); - for (let i = 0; i < numChildren; i++) { - const child = element.getChild(i); - const elementType = child.getType(); - - // Only process text-containing elements - if (elementType === DocumentApp.ElementType.PARAGRAPH || - elementType === DocumentApp.ElementType.LIST_ITEM || - elementType === DocumentApp.ElementType.TEXT) { - // Try to get text - try { - const childText = child.asText ? child.asText() : null; - if (childText) { - text += childText.getText(); - } - } catch (e) { - // Skip elements that can't be converted to text - continue; - } - } else if (elementType === DocumentApp.ElementType.TABLE) { - // Skip tables - continue; - } else if (elementType === DocumentApp.ElementType.INLINE_IMAGE || - elementType === DocumentApp.ElementType.IMAGE || - elementType === DocumentApp.ElementType.UNSUPPORTED) { - // Skip images and unsupported elements - continue; - } else { - // For other container types, recursively extract text - try { - text += extractTextOnly(child); - } catch (e) { - continue; - } - } - } - - return text; -} - /** * Escapes a literal string for findText, which takes a regular expression * rather than plain text. Without this, any quote containing punctuation the @@ -513,7 +725,7 @@ function getAllTabs() { return tabs.map(function(tab) { var text = ''; try { - text = extractTextOnly(tab.asDocumentTab().getBody()); + text = bodyToMarkdown(tab.asDocumentTab().getBody()); } catch (e) { /* skip */ } return { id: tab.getId(), title: tab.getTitle(), text: text }; }); @@ -522,7 +734,7 @@ function getAllTabs() { return [{ id: doc.getId(), title: 'Main', - text: extractTextOnly(doc.getBody()) + text: bodyToMarkdown(doc.getBody()) }]; } } diff --git a/google-docs-addon/README.md b/google-docs-addon/README.md index 7ca906db..847c295d 100644 --- a/google-docs-addon/README.md +++ b/google-docs-addon/README.md @@ -146,6 +146,20 @@ There is no runtime dev/prod flag — the two modes are just two versions of `dist-gdocs/sidebar-bundled.html` with all JS/CSS inlined; that file is what you `clasp push` for a real deployment. +## Document serialization + +`getDocContext()` and `getAllTabs()` return the document as **Markdown**, not +bare text: headings become `#`, list items become `-` or `1.`, and paragraphs +are separated by a blank line. The document's only consumer is an LLM prompt, +and structure the writer marked in Docs is otherwise invisible there — Docs +renders list glyphs from formatting, so `getText()` never returns them. + +See [docs/google-docs-document-serialization.md](../docs/google-docs-document-serialization.md) +for the full mapping, what is deliberately flattened, and how to test it. In +short: unit tests (`npm test` in `frontend/` — they run this file with a stubbed +`DocumentApp`, so no `clasp push` is needed to iterate) for the mapping, and the +**Debug** page in the sidebar's Labs (···) menu for the real round-trip. + ## Key Differences from Word Add-in | Aspect | Word Add-in | Google Docs Add-on | @@ -153,6 +167,7 @@ There is no runtime dev/prod flag — the two modes are just two versions of | Document API | Office.js (client-side) | DocumentApp (server-side via Apps Script) | | UI Framework | HTML/JS in taskpane | HTML/JS in sidebar | | Selection events | Native Office events | Polling (no native events) | +| Document text | Plain text | Markdown (headings, lists) | ## OAuth Scopes