Skip to content

Modernize Discourse Lens: new frontend, topic model, streaming Ask, claims view - #2

Merged
Meerav29 merged 16 commits into
mainfrom
modernize
Oct 16, 2026
Merged

Modernize Discourse Lens: new frontend, topic model, streaming Ask, claims view#2
Meerav29 merged 16 commits into
mainfrom
modernize

Conversation

@Meerav29

@Meerav29 Meerav29 commented Oct 16, 2026

Copy link
Copy Markdown
Owner

An overhaul of Discourse Lens in six steps, one or more commits each. Every step was tested against real maps as well as the test suites.

What changes for users

  • New interface (SvelteKit 3, Svelte 5, TypeScript, Tailwind 4). One app served from the API at localhost:8001, light and dark themes, works at phone width. Selection lives in the URL, so links and reloads land on the same view, topic and article.
  • Many maps, not one corpus. Every map is kept and can be opened, cancelled while running, or deleted. A live feed shows each step as it happens.
  • Better topics. K-means is replaced by BERTopic's recipe without PyTorch (UMAP, HDBSCAN, c-TF-IDF), run on articles rather than chunks. Each topic gets a name, a one-line summary and key terms from one Claude request. Articles that fit nowhere are grouped as Unclustered instead of being forced into a topic.
  • Ask streams answers with numbered citations you can click. Search is hybrid (vectors plus keywords), follow-up questions build on earlier answers, and conversations are saved with the map.
  • Claims view (new). It lists the points two or more articles take a position on, and whether each article agrees, qualifies or disputes them. It runs only when you click Find claims: about $0.27 for 52 articles, about a minute.
  • More articles read. For academic pages that block the crawler, and for PDF links, it looks up open-access text or the abstract through OpenAlex, Crossref and Europe PMC. On a test crawl this raised the number of readable articles from 44 to 72.
  • Optional source review. You can check the searches and pages before anything is read: edit the searches, untick pages, or add your own.
  • No Ollama needed. Embeddings run inside the app with fastembed, using the same model as Ollama. Its vectors are identical (cosine 1.0000), so maps made with Ollama still work. Ollama and OpenAI remain options.
  • SearXNG can replace Brave for search (SEARCH_PROVIDER=searxng).
  • Export to Obsidian is kept.

Under the hood

  • uv with a lockfile, pydantic-settings, and anthropic 1.x on claude-sonnet-5-5 with structured outputs.
  • The SQLite schema migrates automatically from v1 to v8 on first start. Older databases are backed up first, and existing maps keep working (see the README's upgrade notes).
  • A typed frontend client generated from the OpenAPI schema.
  • Quality fixes from step 1: dedup, nomic task prefixes, RAG ranking order, outlier distances.

Testing

  • 128 backend tests (pytest) and 20 frontend unit tests (vitest).
  • 10 Playwright browser tests against a seeded demo map. Run uv run python -m scripts.seed_demo to explore it without API keys.
  • New GitHub Actions CI with backend, frontend and e2e jobs.

Known gaps

  • SearXNG has only been tested against mocked responses, not a live instance.
  • If you switch EMBEDDING_MODE between OpenAI and the local models, maps made with the other one can't be searched. The app now says which setting to use instead of failing with a cryptic error.

🤖 Generated with Claude Code

Meerav29 and others added 16 commits October 11, 2026 11:47
@Meerav29 @claude
Fix analysis-quality bugs in dedup, embeddings, RAG and outliers
fee0db4 
- Dedup chunks across the whole corpus instead of per article, so
  syndicated copies of a story on different sites are caught
- Add nomic-embed-text task prefixes (search_document / search_query)
- Restore vector-search rank order before picking one chunk per article
  in RAG (SQLite IN returned rows in arbitrary order)
- Outliers: one row per article, cluster filter applied server-side
- Sanitize LLM markdown with DOMPurify before {@html}

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Meerav29 @claude
Move to uv, pydantic-settings and current dependencies
b2b4b06 
- pyproject.toml + uv.lock replace requirements.txt; Python pinned to 3.12
- backend/config.py: one pydantic-settings Settings object replaces
  os.getenv calls scattered across modules
- FastAPI lifespan replaces deprecated on_event("startup")
- anthropic 0.34 -> 1.x; default model claude-sonnet-5-5. New
  claude_client.complete() collects text blocks (current models think
  by default, so content[0] may not be text), raises on refusal, and uses
  server-side fallback on the Anthropic API. Label/query calls run at low
  effort with room for thinking (max_tokens 20 would be consumed by it)
- lancedb 0.9 -> 0.40 (paginated list_tables), trafilatura 2.x,
  openai, numpy 2, pandas, scikit-learn upgrades
- ruff config; zip(strict=True) so a short embedding batch fails loudly

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Meerav29 @claude
Support multiple analyses instead of one global corpus
215af0a 
- Schema v2: jobs -> analyses; articles, chunks and projections carry an
  analysis_id with ON DELETE CASCADE. Article ids hash analysis_id:url so
  the same URL can appear in several analyses. WAL + busy timeout so the
  pipeline thread and API reads don't block each other
- One LanceDB table per analysis; drop the unused cluster_id column and
  its O(n^2) full-table rewrite on every run
- Migration from v1 turns the existing corpus into one analysis (keyed by
  the latest finished job), backing up the DB and vectors to *.v1.bak.
  Handles early v1 databases without the jobs.warning column
- Analyses left running by a server restart are marked as errors
- Split main.py into backend/db, backend/pipeline and the routes
- REST API under /api/analyses/{id}/...: list, get, delete (409 while
  running), stream, corpus, clusters, outliers, articles, chat, export.
  Pydantic response models so the frontend can generate a typed client
- Errors use string `detail`; the frontend was reading a key that never
  existed, so every error showed as a bare "HTTP <status>"
- RAG raises instead of returning error text as an "answer"; chat maps
  missing corpus to 409 and model/embedding failures to 502
- Old frontend pointed at the new routes until the rebuild

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Meerav29 @claude
Cancellable analyses with detailed progress and a live event feed
7f0d550 
- pipeline.start/cancel keep a registry of running analyses. Cancellation
  is cooperative: stages check a flag between queries, pages, articles,
  embedding batches and cluster labels. Cancelled is a BaseException so
  the stages' per-item `except Exception` handlers can't swallow it. A
  cancelled run discards its partial data and ends in status 'cancelled'
- Each stage owns a progress band; stage_detail carries text such as
  "Crawled 47 of 112 pages" (schema v3). Progress writes are throttled
- analysis_events table logs queries, indexed articles, skipped pages
  with the reason (extractor now raises SkipArticle), clusters, warnings
  and errors. SSE sends them as `log` events with ids so EventSource
  resumes from Last-Event-ID; GET /events?after=N for polling
- POST /api/analyses/{id}/cancel; shutdown cancels running analyses
- Clear errors when search finds nothing or every page is filtered
- Commit each extracted article before reporting it: an open write
  transaction made the event/progress writes hit "database is locked"
- Schema migration splits statements with comments stripped, so a ';'
  in a SQL comment can't break it

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Meerav29 @claude
Serve the frontend build from the API server and tidy the OpenAPI schema
a5eaa11 
- spa_router serves frontend/build with an index.html fallback for client
  routes; registered after the API so it never shadows /api; refuses
  paths that resolve outside the build directory
- Operation ids are route function names, so the generated client reads
  listAnalyses() rather than listAnalysesApiAnalysesGet()
- scripts/export_openapi.py writes the schema for client generation
- Query events read "query (16 new results)"

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Meerav29 @claude
Rebuild the frontend: SvelteKit 3, Svelte 5 runes, TypeScript, Tailwi…
8032b99 
…nd 4

A fresh app replacing the single-page Svelte 4 UI.

Structure
- Real routes: / (topic input + every map), /analysis/[id] with Map,
  Timeline, Outliers, Key terms and Ask views. Cluster selection and the
  open article live in the URL, so views link and survive reloads
- Typed API client generated from the backend's OpenAPI schema
  (@hey-api/openapi-ts); same-origin /api, proxied by Vite in dev
- adapter-static SPA, served by FastAPI in production
- bits-ui for accessible dialog/sheet/alert primitives; mode-watcher,
  svelte-sonner, lucide icons

Design: a survey chart
- Cool chart paper / night chart themes, Mona Sans with the width axis
  as the display voice, one accent for primary actions and focus
- Map drawn with per-cluster density contours, direct ink labels with a
  colour swatch, zoom that survives selection changes, one scale for both
  axes, and a Table view as the accessible alternative
- Categorical palette validated on both surfaces; the 8 largest clusters
  get hues, the rest share a neutral ink but keep their labels

Behaviour
- Live progress view: stage detail, progress, kept/skipped counts and the
  pipeline's event feed over SSE; cancel; failed/cancelled states with
  "map this topic again"
- Articles take their majority cluster (and its mean position) instead of
  the first chunk's; outlier scores use the same model so every view
  names clusters consistently
- Timeline as stacked bars per month or year (sparse data isn't
  interpolated); key terms ignore URLs
- Chat history per map survives switching views; markdown sanitised
  with DOMPurify; citations open the in-app reader when the source is in
  the corpus

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Meerav29 @claude
Replace k-means with an article-level topic model
01d7022 
Chunks of one article are near-identical in embedding space, so clustering
chunks grouped text by source rather than by argument (HDBSCAN found ~one
"topic" per article). Topics are now found among articles.

- backend/topics.py: BERTopic's recipe without its PyTorch dependency.
  Article vectors -> UMAP 5D -> HDBSCAN (leaf selection) -> merge splits
  members can't tell apart -> noise joins a topic only if it would pass as
  a member, else stays unclustered (-1) -> cap at 8 topics (the palette)
  -> c-TF-IDF key terms (needing 2+ articles per term; boilerplate,
  citation and month words filtered). Separate looser 2D UMAP for the map,
  tuned on real corpora for ~2.5x more room per topic
- One Claude call names all topics with structured output, so names
  contrast (the per-topic calls produced duplicates such as two "Urban
  Heat Mitigation"); falls back to key terms if the call fails
- Schema v4: placements (one row per article: x, y, topic, outlier
  distance) and clusters (label, description, size, terms) replace
  chunk-level projections; v3 data converted on startup
- API: GET /analyses/{id}/articles and /clusters (with terms and
  descriptions); /corpus and /outliers removed
- Whole-article near-duplicate detection (MinHash on word shingles)
  catches syndicated copies whose intros differ; marked 'duplicate'
- scripts/recluster.py rebuilds maps from stored embeddings, no re-crawl
- complete() accepts a JSON schema (structured outputs)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Meerav29 @claude
Show topics from the server: names, summaries, key terms, unclustered
0ebf2af 
- Load /articles and /clusters; drop client-side chunk aggregation and
  TF-IDF (corpus payload no longer carries chunk text)
- Map draws one dot per article with contours per topic; unclustered
  articles shown in grey without a centre label; more padding for contours
- Key terms page shows each topic's summary and c-TF-IDF terms
- Cluster key shows the selected topic's summary

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Meerav29 @claude
Streaming, citable, conversational Ask
1a5a689 
- backend/synthesis/chat.py replaces rag.py: hybrid retrieval (vector +
  BM25 full-text, reciprocal-rank fusion; LanceDB FTS index built on first
  use), one excerpt per article, follow-up questions embed the previous
  question too
- Citation numbers are stable across a conversation: the excerpts the
  previous answer cited are shown again under the same numbers and new
  excerpts continue the count. Without this, a follow-up's [6] meant a
  different article and the model couldn't re-check its own evidence
- POST /chat streams server-sent events (sources, delta, done, error);
  GET /chat returns the saved conversation; DELETE /chat clears it.
  A turn is saved only when its answer completes, so a failure or a
  stopped answer leaves nothing half-written (schema v5: chat_messages)
- stream_text() in the Claude client (server-side fallback kept);
  CHAT_EFFORT setting, default medium: measured first text in ~1s and
  ~9s per answer at both low and medium
- Article dedup threshold 0.8 -> 0.5, with candidates confirmed by
  estimated Jaccard: across real maps, republished copies scored 0.52-1.0
  (an edited WLRN reprint missed at 0.8 scored 0.77), distinct pages
  below 0.4

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Meerav29 @claude
Ask view: streamed answers with clickable citations
593663a 
- Answers stream in with a caret; sources appear as soon as retrieval
  finishes ("Reading 8 articles…"); Stop ends an answer (nothing saved)
- [n] citations become small buttons (titled with the source) that open
  the article reader; sources the answer cited are listed, the rest fold
  under "Also searched, not cited"
- Conversation loads from the server, survives reloads, keeps streaming
  if you switch views, and can be cleared
- fetch-based SSE reader (EventSource can't POST), unit-tested along with
  citation linking

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Meerav29 @claude
Recover academic articles that publishers block
03e3974 
Academic publishers serve headless browsers a bot check or paywall, so
~45% of crawled pages on research-heavy topics were discarded as too
short. backend/ingestion/scholarly.py looks them up instead:

- Identify the work from the URL (DOI, PMC, PubMed, Elsevier PII via
  Crossref, arXiv, MDPI's ISSN/volume/issue/article scheme) or citation
  meta tags
- Text, best first: Europe PMC full text; open-access PDFs from OpenAlex
  locations (several tried: publisher PDFs are often blocked too, while
  repository copies work); else the abstract. PDF links the crawler
  skipped are read directly. Capped at 8,000 words per paper
- Runs 8 lookups at a time after extraction, cancellable, with events;
  articles record where their text came from (source_note, schema v6) and
  the reader shows it. RECOVER_ACADEMIC=false turns it off

Measured on real maps: 51 of 126 previously skipped pages recovered from
the event logs; a fresh urban-heat crawl indexed 72 articles instead of 44
(36 of 47 academic pages recovered in 25 s).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Meerav29 @claude
Add a demo map, end-to-end tests and CI
850ff2f 
- scripts/seed_demo.py builds a small synthetic map (articles, topics with
  terms and summaries, activity log, a saved conversation) with no network
  or API keys: for trying the app, and as end-to-end test data
- Playwright tests (frontend/e2e) drive the real backend serving the built
  app, seeded into a throwaway .e2e/ folder: home, map and key, topic
  selection across views, table and reader, timeline, key terms, saved
  chat with citations, theme, not-found
- GitHub Actions: backend (uv, ruff, pytest), frontend (svelte-check,
  eslint, vitest, build), then e2e. npm pinned to 11 (npm 10's resolver
  crashes on this dependency tree)
- .gitattributes keeps LF line endings across Windows and Linux

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Meerav29 @claude
Optional source review before crawling; SearXNG search provider
eea206b 
- "Let me check the sources before they're read" pauses a run after
  searching: edit and re-run the searches, untick pages, add your own,
  then approve (new reviewing status, /plan endpoints, schema v7)
- SEARCH_PROVIDER=searxng with SEARXNG_URL uses a self-hosted SearXNG
  instance instead of Brave

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Meerav29 @claude
In-process local embeddings with fastembed (no Ollama needed)
4b9fc8a 
- EMBEDDING_MODE=fastembed (new default) runs nomic-embed-text v1.5 via
  ONNX; vectors are normalised and match Ollama's (cosine 1.0000 on real
  chunks, same speed on CPU), so existing maps stay searchable
- ollama keeps working ("local" is still accepted as its old name)
- Asking about a map embedded with a different-sized model now explains
  which EMBEDDING_MODE to use instead of failing inside LanceDB

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Meerav29 @claude
Claims view: where the sources agree and disagree
c0376e2 
- On demand ("Find claims"): Claude lists up to 4 claims from each
  article's opening (10 articles per request), then one request groups
  claims from different articles into propositions, marking each claim
  as agreeing, qualifying or disputing. Validated so every proposition
  has two or more articles and each claim is used once.
- Runs in the background with progress on the analysis row (schema v8:
  claims, propositions, proposition_claims); a restart marks it
  interrupted. GET/POST /api/analyses/{id}/claims.
- New Claims tab: most debated (or most articles) first, stance bars on
  one scale with counts written out, expand a point to read each
  article's claim and open the article.
- Measured: $0.27 for a 52-article map, about a minute.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Meerav29 @claude
Let the topic-recovery test tolerate an unclustered edge point
d36ae02 
UMAP's floating point differs between Windows and Linux; on CI one or
two edge points stayed unclustered (by design), failing an assertion
that every point be clustered. The test now requires the four topics,
an exact split of the clustered points, and at least 90% clustered.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Meerav29
Meerav29 merged commit 3c49c6e into main Oct 16, 2026
3 checks passed
@Meerav29 Meerav29 mentioned this pull request Oct 16, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@Meerav29