Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
72 changes: 72 additions & 0 deletions docs/parsed-results-citation-ux.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
# Parsed Results + citation UX

Recorded 2026-08-18 so we do not relitigate the Figma / issue discussion.

Sources:

- Figma [Knowhere (Copy) / brain](https://www.figma.com/design/0jjj7SQ1N0zo4D9WzJoNJh/Knowhere--Copy-?node-id=4000-37742) (`4000:37742`)
- GitHub [knowhere#222](https://github.com/Ontos-AI/knowhere/issues/222), [knowhere#223](https://github.com/Ontos-AI/knowhere/issues/223)
- Product lock from that conversation (issues and Figma notes are partly stale)

This work lands in **Knowhere Notebook**. Region geometry later lands in **Knowhere retrieval** (`Ontos-AI/knowhere`) and is only consumed here when the payload exists.

## Locked product

Vision / page-image mode only. There is **no original PDF preview** as a citation destination.

| Topic | Decision |
| --- | --- |
| Layout | Sources \| Parsed Results \| Assistant (today’s Notebook columns). Ignore the Figma frame that put Assistant in the middle. |
| Rename | Knowhere Notebook → **Knowhere Brain** |
| Canvas | Title **Parsed Results**, subtitle **From {source title}** |
| Page card | **Page N + parse path**, page image, keywords. No keywords → hide the keyword row, still show the image. Screen 2 “text” is a crop overlay on the page image, not a text chunk. |
| List / Tree | Keep **both**. Tree is the **existing** section tree, also in page-image mode. **Default Tree**. Citation click may switch to List so the page card is visible. |
| Inline chips | One chip **per citation**, not per page. Label `{source title}/pN` (same name as the Sources row). Max width 250px, truncate. Same page can be two chips; later they get different crop boxes. |
| Chip / `pN` click | Switch the **viewed** source if needed (not query checkboxes), open that file’s parsed page, scroll to it. Chat may cite any workspace file while the canvas shows another. |
| Footer SOURCES | Numbered list, one row per file, `{title}` + cited page links `p25 p26 p27`. Those links do the **same jump** as chips. |
| Copy / download | Copy answer / code / tables. Download **this assistant turn** as **Markdown and PDF**, **without citations** (no chips, no SOURCES list). |
| Region box | **Required product**, **not this slice**. When retrieval sends a crop, draw `#8E51FF` / 25% / 8px radius on that page image (flash twice, 3s, fade). Until then, page jump only. Do not fake a full-page box. |

Out of this slice: original-file viewer for citations, region overlay, faking geometry.

## Current code (gaps)

Almost all of this is UI in this repo. Page images, keywords, page numbers, and source-title chips already exist in pieces.

| Area | Today | Need |
| --- | --- | --- |
| Branding | “Knowhere Notebook” in `top-nav.tsx`, `lib/app-metadata.ts`, login, `global-error.tsx` | Brain |
| Canvas copy | “Parsed Chunks” / “Showing all parsed chunks from …” | Parsed Results / From {title} |
| Original preview | Hidden for `page-assets` sources (`chunks-panel.tsx`) | Keep hidden. Citations must not reopen it. |
| List / Tree | Tree **hidden** for page-asset sources (`!isPageAssetSource`). Default mode is already `"tree"` but unused in page mode. | Show the toggle for page mode; Tree uses existing `ChunkSectionTree`. |
| Page cards | `PageChunkCard` already shows page image + `ChunkKeywords` | Match Figma title (`Page N` + path). Image-first. Hide keywords when empty (already returns null). |
| Inline chips | `[Source N: …]` tokens are **stripped** (`buildCitationContentMarkdown`). Chips live in a footer `AssistantSources`. | Replace tokens with inline `{title}/pN` chips in the answer. |
| Chip label | Filename / source title only, **no `/pN`**. | `{title}/pN` |
| Dedupe | `getDisplayCitations` and `selectCitationRawResults` drop same-key citations. | Keep one chip per citation so two regions on one page stay two chips. |
| Footer SOURCES | Flat chips | Numbered per-file list + page links |
| Click target | `handleCitationClick` focuses a **chunk**. `focusedPage` is plumbed to `ChunksPanel` but **not used**. | Focus/scroll the **page** card; `onSelectSource` for file B while viewing A already exists. |
| Export | None | Per-turn copy + MD/PDF without citations |

Later API (do not block Notebook UI):

- Retrieval / agent citation payload has page number and page image today (`pageCitationPageNumber`, `pageCitationAssetUrl`).
- It does **not** have a citation crop box. Worker bboxes are for asset crops, not “this quote lives here.”
- When Knowhere adds a region on the citation, extend `ChatCitationView` + persistence, then overlay on `PageCitationAssetImage`.

## Implementation slices (Notebook)

Do these in order. Tests follow existing ownership: model tests for labels/grouping, component tests for chips/cards/export, panel tests for click → source + page.

1. **Chrome** — rename Brain; Parsed Results header/subtitle; enable List/Tree on page-asset sources; default Tree.
2. **Page cards** — Figma card header (page + path); image + optional keywords; citation click forces List (already does via `citationListViewRequestId`) and scrolls to the page chunk.
3. **Citations** — stop stripping without replacement; inline chips `{title}/pN`; stop collapsing same-page citations; numbered SOURCES with `pN` links; click switches viewed source and page-focuses.
4. **Copy / download** — message actions: clipboard of visible answer (code/tables included as rendered); MD + PDF of this turn with the same citation-stripped body we already produce for display (`buildCitationContentMarkdown` / equivalent). PDF is client-side from that Markdown, not the original document.
5. **Later** — region overlay when retrieval sends a box. Keep citation identity stable (do not key chips by `title+page` only).

## Do not

- Route citation clicks to original PDF preview.
- Deduplicate inline chips by page.
- Draw a highlight without retrieval geometry.
- Change which sources are **checked for the next query** when jumping the canvas.
- Put this feature in archived `knowhere-api`; new retrieval fields go in `Ontos-AI/knowhere`.
4 changes: 2 additions & 2 deletions e2e/chunk-original-preview.e2e.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ test("reopens the demo PDF preview after returning from original to parsed chunk
}) => {
await page.goto("/")
await expect(
page.getByRole("heading", { name: "Parsed Chunks" }),
page.getByRole("heading", { name: "Parsed Results" }),
).toBeVisible()

await openFirstChunk(page)
Expand All @@ -18,7 +18,7 @@ test("reopens the demo PDF preview after returning from original to parsed chunk
.getByRole("button", { name: "Parsed", exact: true })
.click()
await expect(
page.getByRole("heading", { name: "Parsed Chunks" }),
page.getByRole("heading", { name: "Parsed Results" }),
).toBeVisible()

await openFirstChunk(page)
Expand Down
21 changes: 21 additions & 0 deletions e2e/citation-dedupe.e2e.ts
Original file line number Diff line number Diff line change
Expand Up @@ -87,3 +87,24 @@ test("keeps duplicate source labels clickable for separate documents", async ({
await expect(page.getByText("First report source content.")).toBeVisible()
expect(firstSourceChunkRequests).toBeGreaterThan(0)
})

test("keeps two chips to the same title/pN as separate buttons", async ({
context,
page,
}) => {
await context.addCookies([
{
name: "better-auth.session_token",
value: "playwright",
url: "http://localhost:3000",
},
])

await page.goto("/e2e/citation-same-page")

const chatPanel = page.getByTestId("desktop-chat-panel")
await expect(chatPanel.getByTestId("citation-chip")).toHaveCount(2)
await expect(
chatPanel.getByRole("button", { name: "Open source spacex-s1.pdf/p26" }),
).toHaveCount(2)
})
71 changes: 71 additions & 0 deletions src/agent-harness/ledger.test.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
import { describe, expect, it } from "vitest"
import type {
KnowledgeGrepMatch,
KnowledgeGrepResponse,
KnowledgeReadResponse,
RetrievalQueryResponse,
Expand Down Expand Up @@ -111,6 +112,76 @@ describe("createEvidenceLedger", () => {
}),
])
})

it("copies page metadata onto grep matches from the same chunk id", () => {
const ledger = createEvidenceLedger()
ledger.addReadChunksResponse(makeReadResponse())

const snapshot = ledger.addGrepChunksResponse({
...makeGrepResponse(),
matches: [
{
position: 1,
chunkId: "chunk_page_1",
chunkType: "page",
sectionPath: "Root / Page 1",
sourceChunkPath: "pages/page-1.md",
startOffset: 0,
endOffset: 12,
snippet: "Full page snip",
},
],
})

expect(snapshot.chunks[1]).toEqual(
expect.objectContaining({
ref: "grep1:match:1",
kind: "grep_match",
chunkId: "chunk_page_1",
metadata: expect.objectContaining({
pageNums: [1],
position: 1,
startOffset: 0,
endOffset: 12,
}),
}),
)
})

it("copies pageNumbers from the grep match when the SDK provides them", () => {
const ledger = createEvidenceLedger()

const snapshot = ledger.addGrepChunksResponse({
...makeGrepResponse(),
matches: [
{
position: 1,
chunkId: "chunk_page_4",
chunkType: "page",
sectionPath: "FINANCIAL SUMMARY",
sourceChunkPath: "pages/page-4.md",
startOffset: 0,
endOffset: 12,
snippet: "automotive revenues",
pageNumbers: [4],
} as KnowledgeGrepMatch,
],
})

expect(snapshot.chunks[0]).toEqual(
expect.objectContaining({
ref: "grep1:match:1",
kind: "grep_match",
chunkId: "chunk_page_4",
metadata: expect.objectContaining({
pageNums: [4],
position: 1,
startOffset: 0,
endOffset: 12,
}),
}),
)
})
})

function makeRetrievalResponse(): RetrievalQueryResponse {
Expand Down
36 changes: 36 additions & 0 deletions src/agent-harness/ledger.ts
Original file line number Diff line number Diff line change
Expand Up @@ -255,6 +255,19 @@ function addChunkFromGrepMatch(input: {
readonly match: KnowledgeGrepMatch
readonly ref: string
}): void {
const matchWithPages = input.match as KnowledgeGrepMatch & {
readonly pageNumbers?: readonly number[]
readonly metadata?: Readonly<Record<string, unknown>>
}
const donor = input.ledger.chunks.find(
(chunk) =>
chunk.chunkId === input.match.chunkId && hasPageMetadata(chunk.metadata),
)
const pageNums =
matchWithPages.pageNumbers && matchWithPages.pageNumbers.length > 0
? [...matchWithPages.pageNumbers]
: undefined

addChunk({
ledger: input.ledger,
chunk: {
Expand All @@ -268,6 +281,9 @@ function addChunkFromGrepMatch(input: {
sourceChunkPath: input.match.sourceChunkPath,
filePath: input.match.filePath,
metadata: {
...(donor?.metadata ?? {}),
...(matchWithPages.metadata ?? {}),
...(pageNums ? { pageNums } : {}),
position: input.match.position,
startOffset: input.match.startOffset,
endOffset: input.match.endOffset,
Expand All @@ -282,6 +298,26 @@ function addChunkFromGrepMatch(input: {
})
}

function hasPageMetadata(
metadata: Readonly<Record<string, unknown>> | undefined,
): boolean {
if (!metadata) return false
const values = [
metadata.pageNums,
metadata.page_nums,
metadata.pageNum,
metadata.page_num,
metadata.pageAssets,
metadata.page_assets,
]
return values.some((value) => {
if (Array.isArray(value)) return value.length > 0
if (typeof value === "number") return Number.isSafeInteger(value) && value > 0
if (typeof value === "string") return value.trim().length > 0
return false
})
}

function addChunk(input: {
readonly ledger: MutableLedger
readonly chunk: Omit<EvidenceChunk, "assetRef">
Expand Down
9 changes: 9 additions & 0 deletions src/agent-harness/runtime.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,15 @@ describe("agent harness runtime", () => {
expect(prompt).not.toContain("must match the selected evidence ref exactly")
})

it("tells the agent to emit [[cite:n]] markers instead of title/pN or [1]", () => {
const prompt = buildHarnessSystemPrompt(makeTurnInput())

expect(prompt).toContain("[[cite:n]]")
expect(prompt).toContain("1-based index into the citations array")
expect(prompt).toContain("Do not write title/pN, [1], Markdown footnotes")
expect(prompt).toContain("Do not collapse same-page citations")
})

it("passes only outer retrieval parameters to KNOWHERE without planning-tool gating", async () => {
const query = vi.fn<KnowhereToolRuntime["search"]>().mockResolvedValue(
makeRetrievalResponse(),
Expand Down
3 changes: 3 additions & 0 deletions src/agent-harness/runtime.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1176,6 +1176,9 @@ export function buildHarnessSystemPrompt(turn: AgentTurnInput): string {
"- artifacts with display=true are the exact images/tables shown. Never display every candidate; honor constraints.desiredCount / maxCount.",
"- Use type=derived_table only for tables you create from evidence; every derived_table.sourceRefs entry must reference evidence in the ledger.",
"- Prefer citation and selected image/table artifact refs returned by Knowhere tools in the evidence ledger.",
"- Place [[cite:n]] immediately after the supported claim. n is the 1-based index into the citations array passed to finalize.",
"- Do not write title/pN, [1], Markdown footnotes, or [Source N: ...] in the answer text. Notebook renders chips from [[cite:n]] and citation metadata.",
"- Repeat [[cite:n]] when another claim uses the same page. Do not collapse same-page citations to one row.",
"- Citation label and source metadata are optional. Notebook resolves citation metadata from evidence refs when possible.",
"- If evidence is relevant but you cannot identify a supporting evidence ref, answer with unresolved issues instead of fabricating a ref.",
"- inspectImage observations are inspection notes, not new source refs. Final citations and displayed image artifacts must use the original retrieved image asset refs.",
Expand Down
78 changes: 78 additions & 0 deletions src/app/e2e/citation-same-page/page.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
import { notFound } from "next/navigation"

import { WorkspaceShell } from "@/components/workspace-shell"
import type { ChatMessageView, ChatThreadView } from "@/domains/chat/types"
import type { SourceView } from "@/domains/sources/types"

const sources: SourceView[] = [
{
id: "source_spacex",
title: "spacex-s1.pdf",
mimeType: "application/pdf",
status: "ready",
documentId: "doc_spacex",
chunkCount: 1,
documentPresentation: { kind: "page-assets", pageCount: 26 },
},
]

const chatThreads: ChatThreadView[] = [
{
id: "thread_1",
title: "Same page citations",
createdAt: "2026-05-17T00:00:00.000Z",
updatedAt: "2026-05-17T00:00:00.000Z",
},
]

const chatMessages: ChatMessageView[] = [
{
id: "assistant_1",
role: "assistant",
content: "Revenue grew [[cite:1]] and later expanded [[cite:2]].",
citations: [
{
chunkType: "page",
score: 0.91,
pageCitationPageNumber: 26,
source: {
documentId: "doc_spacex",
sourceFileName: "spacex-s1.pdf",
sectionPath: "Page 26",
},
},
{
chunkType: "page",
score: 0.89,
pageCitationPageNumber: 26,
source: {
documentId: "doc_spacex",
sourceFileName: "spacex-s1.pdf",
sectionPath: "Page 26",
},
},
],
},
]

export default function CitationSamePageTestPage() {
if (process.env.NODE_ENV === "production") notFound()

return (
<WorkspaceShell
user={{
id: "user_playwright",
name: "Playwright",
email: "playwright@example.com",
}}
workspace={{
id: "workspace_playwright",
namespace: "notebook-playwright",
}}
sources={sources}
chatThreads={chatThreads}
activeChatThreadId="thread_1"
chatMessages={chatMessages}
/>
)
}
4 changes: 2 additions & 2 deletions src/app/global-error.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -15,11 +15,11 @@ export default function GlobalError({
return (
<html lang="en">
<body>
<title>Something went wrong - Knowhere Notebook</title>
<title>Something went wrong - Knowhere Brain</title>
<main className="flex min-h-screen items-center justify-center bg-background px-6 py-16 font-sans text-foreground antialiased">
<div className="w-full max-w-md text-center">
<p className="text-xs font-bold uppercase tracking-widest text-muted-foreground">
Notebook error
Brain error
</p>
<h1 className="mt-3 text-2xl font-semibold tracking-normal">
Something went wrong
Expand Down
2 changes: 1 addition & 1 deletion src/app/login/page.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ export async function LoginContent() {
<NotebookLogoMark width={28} />
</div>
<h1 className="mb-8 text-2xl font-bold tracking-tight">
Knowhere Notebook
Knowhere Brain
</h1>
<Link
href={loginHref}
Expand Down
Loading
Loading