Skip to content

CLAUDE.md

Latest commit

 

History

History
68 lines (52 loc) · 9.48 KB

CLAUDE.md

File metadata and controls

68 lines (52 loc) · 9.48 KB
Raw

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

What this is

vectorless-js, a JavaScript/TypeScript port of PageIndex (vectorless, reasoning-based RAG). It builds a hierarchical "table of contents" tree from a document and lets an LLM navigate it with tools, instead of embeddings/chunking/vector search. Runs in the browser (via WASM) and anywhere JS runs (Bun, Node, Cloudflare Workers).

The repo is a publishable library (dist-lib/), plus a SolidJS demo app (dist-app/), a docs site (docs/), and a runnable example (examples/quick-start/). Published to npm as vectorless-js (0.5.2, MIT). Git remote is ancs21/vectorless-js (public). Docs are live at vectorless.js.org (js.org subdomain), served from Cloudflare Pages vectorless-js.pages.dev (bunx wrangler pages deploy <build dir> --project-name=vectorless-js). The js.org subdomain is a Pages custom domain added via the API, not the dashboard (Cloudflare Pages requires the API for this); it is already active.

Commands (Bun)

bun install
bun test                       # bun:test, verifies the deterministic tree/tool logic
bun test -t "code fences"      # single test by name
bun run typecheck              # tsc --noEmit over src (catches real bugs; run before done)
bun run build:lib              # library → dist-lib/ (ESM .js + .d.ts) via tsconfig.build.json
bun run build                  # demo app → dist-app/ via Vite
bun run dev                    # Vite dev server (app) on :5173
bun run docs:dev               # Blume docs site (content in docs/)
bun run docs:build             # docs → dist/ (blume has no outDir option, hence lib lives in dist-lib/)
bunx blume validate            # docs link check

examples/quick-start/ is a standalone project with its own install; it consumes the published package, not the local source. It ships a fictional lease.pdf so bun run ask.ts ./lease.pdf "..." works out of the box.

bunx blume build fails while blume dev is running ("would corrupt its .blume runtime"). Use bunx blume build --isolated to verify docs without stopping the dev server.

Architecture, the big picture

Data flow. Everything except the LLM call runs client-side, or anywhere:

pdfToMarkdown (liteparse-wasm)  →  buildIndex (deterministic tree)  →  ask (LLM tool-use loop)
   src/parse.ts                     src/pageindex.ts                    src/agent.ts
                                                                          │ fetch (prompts only)
                                                                          ▼
                                                    any OpenAI-compatible endpoint
  • src/pageindex.ts is a faithful 1:1 port of PageIndex's Python page_index_md.py (tree build + summaries) + retrieve.py (the 3 tools). buildIndex + the tools are fully deterministic, no LLM and no runtime imports (verified byte-for-byte against Python md_to_tree). addNodeSummaries/countTokens are the exception: they lazy-import() gpt-tokenizer (o200k, matching PageIndex's litellm.token_counter) only when called, so bundlers tree-shake it out for consumers who don't summarize. Treat parity with the Python as an invariant; keep the verification diff empty.
  • src/agent.ts is the retrieval loop: an OpenAI-compatible tool-use conversation over get_document / get_document_structure / get_page_content. Tools execute locally against the in-browser tree, so the document never leaves the client except as prompt/message text the LLM chose to fetch. Loop caps at 12 hops.
  • A browser cannot call a provider directly (CORS blocks it, and the key would be exposed in the page), so browser deployments need a proxy in front. The library does not ship one; any OpenAI-compatible URL works, and docs/browser.mdx shows a minimal proxy inline.

Non-obvious invariants and gotchas

  • LLM config mirrors the OpenAI client, deliberately. LLMConfig is { baseURL, apiKey?, model?, maxTokens?, headers? } and the documented env vars are OPENAI_BASE_URL / OPENAI_API_KEY / OPENAI_MODEL. Do not invent env names or rename baseURL back to endpoint. A bare string is shorthand for { baseURL }. baseURL gets /chat/completions appended unless it already ends in it. Default model is gpt-4o (DEFAULT_MODEL in src/agent.ts).
  • Reasoning tokens count against max_tokens and are invisible in the response. ask() uses 4096; complete() uses 16384. Measured: asking Gemini to level 318 headings in one call burned 15.7k tokens thinking, leaving so little room that the answer was truncated mid-array with finish_reason: length. A starved cap does not error, it returns an unparseable answer. Don't lower either default, and prefer many small calls over one large one.
  • A flat tree has two causes with very different costs. buildIndexAuto routes on which. Headings that all parsed at one level (a PDF with no font-size hierarchy) keep correct titles and lines, so assignHeadingLevels sends headings only, batched 60 per call. No headings at all means reconstructTree, which sends the whole line-numbered document: a 448 KB filing is ~133k tokens, over gpt-4o's 128k window. Measured on FinanceBench: 80 of 84 filings tree deterministically; the flat ones were 3 General Mills 10-Ks (uniform headings) and an earnings deck (none).
  • LLM answers are aligned by key, never by position. assignHeadingLevels parses number:depth pairs, because models miscount long lists (307 returned for 318 asked) and a positional list would shift every heading after an omission. Missing depths default to top-level, degrading to the flat tree rather than a wrong one. Note new Array(n) is sparse and map/flat skip holes; it must be .fill(undefined).
  • .js extensions on relative imports are required, not optional (e.g. import … from "./pageindex.js"). The library builds with moduleResolution: NodeNext (tsconfig.build.json), which mandates them. Bun/Vite resolve .js→.ts fine. Do not strip them.
  • src/parse.ts must stay environment-agnostic. It never hardcodes Vite's ?url import; the caller passes the wasm source to initParser(). The Vite ?url import lives only in src/main.tsx (the app). Keeping this out of the lib is what lets it run in Bun/Node/Workers.
  • Package exports: . = core + agent; ./parse = liteparse (an optional peer dependency). Importing the core must not pull in the WASM bundle, so parse.ts is never re-exported from src/index.ts. gpt-tokenizer is the only runtime dependency. solid-js is app-only and must stay in devDependencies; it once sat in dependencies and shipped SolidJS to every library consumer.
  • Three build outputs, kept separate: lib → dist-lib/ (tsc), app → dist-app/ (Vite build.outDir), docs site → dist/ (Blume). Blume hard-codes dist/ with no outDir option, which is why the library lives in dist-lib/. Don't move it back. files: ["dist-lib", "README.md", "LICENSE"] is what gets published.
  • LICENSE must keep Vectify AI's notice. Upstream PageIndex is MIT, and src/pageindex.ts is a substantial port of it, so MIT obliges us to ship their copyright notice. LICENSE carries both grants; don't reduce it to a bare MIT. Source comments marking what was ported are provenance for maintainers, and stay.
  • Our brand is vectorless-js, everywhere a user or model can see it. The agent's system prompt says "You are vectorless-js"; it must never claim to be PageIndex. Attribution belongs in LICENSE, README, and source comments, not in runtime output.
  • No em dashes or en dashes in docs, README, or new code comments. This is a hard constraint the repo owner enforces, not a style preference. (Older src/ comments still contain them.)
  • docs/meta.ts controls nav order via defineMeta({ pages: [...] }). Adding a page without listing it there leaves it in default order.
  • A React Doctor pre-commit hook fires on this repo. It is a React linter scanning a SolidJS app and plain TypeScript, so its findings are mostly noise. Its "chained array iterations" rule targets src/pageindex.ts, where changing the code would break the Python parity invariant for no measurable gain.
  • A pre-push hook rejects "claude" in commit subjects. Keep it out of the subject line; body trailers pass.

Scope / status

  • Multi-document (small case) built. ask(docOrCorpus, …) takes one IndexedDoc or a Corpus (Record<doc_id, IndexedDoc>); the agent calls list_documents, picks a doc_id, then drills in. Tools route via runTool(corpus, name, input) (retrieve.py's documents[doc_id]). Corpus-scale routing uses routeDocuments(corpus, query, complete), LLM description-based (vectorless) picking of relevant doc_ids; compose as route, narrow corpus, then ask. Tens of thousands of documents is unsupported: the router prompt blows up.
  • OCR is off (born-digital PDFs only); liteparse's isComplex()/needsOcr flags scanned pages that would need a JS ocrEngine (e.g. tesseract.js).
  • There is no cloud/hosted service. The library is client-side plus, for browsers, a proxy you supply. Don't add server-side storage, async indexing APIs, or multi-tenant plumbing unless asked.
  • The FinanceBench harness and sample PDFs were removed from the repo. There is no in-repo regression check for the tree-quality work above; the measurements quoted here came from that harness while it existed, and reproducing them means rebuilding it.