diff --git a/.env.example b/.env.example index 6866357..4fd9ee7 100644 --- a/.env.example +++ b/.env.example @@ -1,35 +1,36 @@ -# Required -BRAVE_API_KEY=your_key_here +# Copy to .env and fill in. Only the search key and the Claude key are needed to start. -# Claude backend — choose one: anthropic | vertex -CLAUDE_BACKEND=anthropic +# ── Web search ────────────────────────────────────────────────────────────── +SEARCH_PROVIDER=brave # brave | searxng +BRAVE_API_KEY=your_key_here # needed when SEARCH_PROVIDER=brave +SEARXNG_URL= # e.g. http://localhost:8888 when SEARCH_PROVIDER=searxng -# Anthropic API (used when CLAUDE_BACKEND=anthropic) +# ── Claude ────────────────────────────────────────────────────────────────── +# Writes search queries, names topics, finds claims and answers questions. +CLAUDE_BACKEND=anthropic # anthropic | vertex + +# Anthropic API (CLAUDE_BACKEND=anthropic) ANTHROPIC_API_KEY=your_key_here -CLAUDE_MODEL=claude-sonnet-5-5 # override default model +CLAUDE_MODEL=claude-sonnet-5-5 -# Vertex AI (used when CLAUDE_BACKEND=vertex) +# Vertex AI (CLAUDE_BACKEND=vertex) # Prerequisites: uv sync --extra vertex && gcloud auth application-default login GOOGLE_CLOUD_PROJECT=your_gcp_project_id VERTEX_REGION=global # global, a multi-region (us / eu), or a specific region -VERTEX_CLAUDE_MODEL=claude-sonnet-5-5 # override Vertex model +VERTEX_CLAUDE_MODEL=claude-sonnet-5-5 -# Embeddings — choose one mode +# ── Embeddings ────────────────────────────────────────────────────────────── EMBEDDING_MODE=fastembed # fastembed (in-process) | ollama | openai FASTEMBED_CACHE=./data/models # where fastembed keeps its model (~520 MB) OLLAMA_BASE_URL=http://localhost:11434 # only needed if EMBEDDING_MODE=ollama OPENAI_API_KEY=your_key_here # only needed if EMBEDDING_MODE=openai -# Storage paths +# ── Storage ───────────────────────────────────────────────────────────────── DB_PATH=./data/discourse.db LANCE_PATH=./data/vectors -# Optional tuning -MAX_ARTICLES=120 # cap on total articles to crawl -UMAP_N_NEIGHBORS=15 -KMEANS_K=auto # auto | integer (e.g. 7) -RAG_TOP_K=15 -CHAT_EFFORT=medium # low | medium | high: reasoning before chat answers -RECOVER_ACADEMIC=true # look up open-access copies of pages academic publishers block -SEARCH_PROVIDER=brave # brave | searxng -SEARXNG_URL= # e.g. http://localhost:8888 when SEARCH_PROVIDER=searxng +# ── Optional tuning ───────────────────────────────────────────────────────── +MAX_ARTICLES=120 # most pages read per map +RECOVER_ACADEMIC=true # look up open-access copies of pages academic publishers block +RAG_TOP_K=15 # excerpts searched per question in Ask +CHAT_EFFORT=medium # low | medium | high: reasoning before Ask answers diff --git a/Planning.md b/Planning.md index 886bbe9..6554bee 100644 --- a/Planning.md +++ b/Planning.md @@ -1,6 +1,8 @@ # Discourse Lens — Full Project Documentation > Research Intelligence Tool | v1.0 +> **This is the original v1 plan (April 2026), kept for history.** The app has since been rebuilt (October 2026): many maps instead of one corpus, a topic model in place of k-means, a new SvelteKit 3 frontend, a Claims view, and a different API (`/api/analyses/...`). For how it works now, see [README.md](README.md); for how to check it, see [validation.md](validation.md). + --- ## Table of Contents diff --git a/README.md b/README.md index 1574338..5c4e61c 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,27 @@ # discourse-lens -Discourse Lens takes a topic, reads ~100 relevant articles from the web, and maps the conversation: which camps exist, how coverage has moved over time, which articles don't fit, and what each camp talks about. You can then ask questions answered only from those articles. Everything runs locally except web search and the Claude API calls. +Discourse Lens takes a topic, reads around a hundred articles about it from the web, and maps the conversation: which camps exist, where the sources agree and disagree, how coverage has moved over time, and which articles don't fit. You can then ask questions answered only from those articles, with citations. Everything runs on your machine except web search and the Claude requests. + +![The map view: articles about mechanistic interpretability grouped into three named topics](docs/screenshots/map.png) + +--- + +## What you get + +Type a topic and Discourse Lens searches the web, reads the pages, and builds a map in 3 to 8 minutes. Every finished map has six views, plus a reader for any article. + +| | | +|---|---| +| ![Claims view](docs/screenshots/claims.png) | **Claims.** The points two or more articles take a position on, and whether each one agrees, qualifies or disputes them, most debated first. | +| ![Ask view](docs/screenshots/ask.png) | **Ask.** Answers drawn only from the map's articles, streamed as they're written, with numbered citations you can click. | +| ![Key terms view](docs/screenshots/terms.png) | **Key terms.** Each topic's one-line summary and the words it uses far more than the others. | +| ![Timeline view](docs/screenshots/timeline.png) | **Timeline.** Articles per month or year, by topic. | +| ![Article reader](docs/screenshots/reader.png) | **Reader.** Any article's text in a side panel, from the map, the table or a citation. | +| ![Source review](docs/screenshots/review.png) | **Source review (optional).** See the searches and pages before anything is read; edit, untick or add your own. | + +The map works in light and dark themes, and picking a topic focuses every view on it: + +![The map in dark mode with one topic selected and its summary shown](docs/screenshots/map-dark.png) --- @@ -8,14 +29,16 @@ Discourse Lens takes a topic, reads ~100 relevant articles from the web, and map - Python 3.12 and [uv](https://docs.astral.sh/uv/) - Node.js 20.19+ (Vite 8) -- A [Brave Search API key](https://brave.com/search/api) (free tier: 2000 queries/month), or a self-hosted [SearXNG](https://docs.searxng.org) instance with the JSON format enabled (`SEARCH_PROVIDER=searxng`, `SEARXNG_URL=http://localhost:8888`) -- An [Anthropic API key](https://console.anthropic.com) (for chat and cluster labelling) +- A [Brave Search API key](https://brave.com/search/api) (free tier: 2000 queries a month), or a self-hosted [SearXNG](https://docs.searxng.org) instance with the JSON format enabled +- An [Anthropic API key](https://console.anthropic.com), or Claude on Google Cloud Vertex AI + +No GPU, Ollama or OpenAI key is needed: embeddings run inside the app. --- ## Setup -**1. Clone and install Python dependencies** +**1. Install the Python dependencies** ```bash git clone https://github.com/Meerav29/discourse-lens.git @@ -35,22 +58,13 @@ npm run build cd .. ``` -Embeddings run in-process with [fastembed](https://github.com/qdrant/fastembed) (nomic-embed-text v1.5, no GPU or PyTorch needed). The first map downloads the model once (about 520 MB) to `data/models`. Already running [Ollama](https://ollama.com)? `EMBEDDING_MODE=ollama` uses its `nomic-embed-text` instead. It's the same model with the same vectors, so maps work with either. `EMBEDDING_MODE=openai` uses OpenAI's API, but maps made that way can only be searched with OpenAI. - -**3. Configure API keys** - -Copy `.env.example` to `.env` and fill in your keys: +**3. Add your keys** ```bash cp .env.example .env ``` -Open `.env` and set: - -``` -BRAVE_API_KEY=your_key_here -ANTHROPIC_API_KEY=your_key_here -``` +Open `.env` and set `BRAVE_API_KEY` and `ANTHROPIC_API_KEY`. Everything else has a working default; see [Settings](#settings). --- @@ -60,15 +74,76 @@ ANTHROPIC_API_KEY=your_key_here uv run uvicorn backend.api.main:app --port 8001 ``` -Then open [http://localhost:8001](http://localhost:8001). The backend serves the built app and the API from one address. +Then open [http://localhost:8001](http://localhost:8001). The backend serves the app and the API from one address. + +The first map downloads the embedding model once (about 520 MB, into `data/models`). -**No API keys yet?** Load a small synthetic demo map to explore the interface: +**No API keys yet?** Load a small made-up demo map to explore the interface: ```bash uv run python -m scripts.seed_demo ``` -### Working on the frontend +--- + +## Using it + +1. On the home page, type a topic and choose **Map this topic**. Tick **Let me check the sources before they're read** to see the searches and pages first. +2. While the map is built you see each step: the searches, pages read, articles kept or skipped (and why), and topics as they're named. You can leave the page or **Cancel this map**. +3. When it's done, switch between the views above. **Map** has a **Table** toggle for a plain list. **Outliers** lists the articles furthest from the middle of their topic. +4. Pick a topic in the **Clusters** panel to focus every view on it and read its summary. Articles that don't fit any topic well are grouped as **Unclustered** rather than forced into one. The address bar keeps the view, topic and open article, so links and reloads land in the same place. +5. **Claims** runs only when you choose **Find claims**, because it costs Claude requests: about half a cent per article (roughly $0.30 for a 50-article map), and it takes about a minute. +6. In **Ask**, follow-up questions build on earlier answers, and a citation number means the same article for the whole conversation. Conversations are saved with the map. +7. **Export to Obsidian** downloads a vault: a Canvas of the map plus one note per article. +8. Every map is kept. **All maps** returns to the list, where you can open or delete them. + +### Good topics to start with + +| Topic | Why it works well | +|---|---| +| `mechanistic interpretability in large language models` | Active research field with distinct camps | +| `longevity science and aging reversal` | Real disagreements for the Claims view | +| `urban heat islands and city design` | Recent policy debate, good timeline | +| `jevons paradox and energy efficiency` | Older topic with a long timeline | + +Avoid topics that are too broad (`AI`, `climate change`) or too narrow (fewer than about 20 articles exist on the web). + +--- + +## How it works + +1. **Search.** Claude writes six search queries for the topic; Brave or SearXNG runs them (`backend/ingestion/search.py`). With source review on, the run pauses here. +2. **Read.** Playwright loads each page and trafilatura extracts the article (`backend/ingestion/crawler.py`, `extractor.py`). Academic publishers often block automated reading, so for those pages and for PDF links the app looks up open-access text or the abstract through OpenAlex, Crossref and Europe PMC (`backend/ingestion/scholarly.py`). The reader says where recovered text came from. +3. **Clean.** Articles under 200 words, non-English ones and near-duplicates are dropped (MinHash), and articles are split into chunks (`backend/processing/`). +4. **Embed.** Chunks are embedded with nomic-embed-text v1.5 via fastembed and stored in LanceDB, one table per map (`backend/embedding/`). +5. **Map.** BERTopic's recipe without PyTorch: each article's chunk vectors are averaged, reduced with UMAP, grouped with HDBSCAN, and described with c-TF-IDF key terms. One Claude request names and summarizes all topics so the names contrast (`backend/topics.py`, `backend/embedding/project.py`). +6. **Ask.** Hybrid search (vector similarity plus BM25 keywords, fused by reciprocal rank) finds the best excerpt per article, and Claude answers from those excerpts only (`backend/synthesis/chat.py`). +7. **Claims.** Claude lists each article's main claims from its opening, then groups claims from different articles into shared points with a stance for each (`backend/synthesis/claims.py`). + +Data lives in `data/`: SQLite for articles, maps and conversations (`discourse.db`), LanceDB for vectors (`vectors/`). + +--- + +## Settings + +All settings go in `.env`; `.env.example` lists them with comments. + +| Setting | Default | What it does | +|---|---|---| +| `SEARCH_PROVIDER` | `brave` | `brave` (needs `BRAVE_API_KEY`) or `searxng` (needs `SEARXNG_URL`) | +| `CLAUDE_BACKEND` | `anthropic` | `anthropic` (needs `ANTHROPIC_API_KEY`) or `vertex` (needs `GOOGLE_CLOUD_PROJECT`) | +| `CLAUDE_MODEL` | `claude-sonnet-5-5` | Model for queries, topic names, claims and answers | +| `EMBEDDING_MODE` | `fastembed` | `fastembed` (in the app), `ollama` (same model and vectors, served by [Ollama](https://ollama.com)) or `openai` | +| `MAX_ARTICLES` | `120` | Most pages read per map | +| `RECOVER_ACADEMIC` | `true` | Look up open-access copies of pages academic publishers block | +| `CHAT_EFFORT` | `medium` | How much Claude reasons before an Ask answer: `low`, `medium` or `high` | +| `RAG_TOP_K` | `15` | Excerpts searched per question | + +Maps made with `EMBEDDING_MODE=openai` can only be searched with OpenAI, and the other way round; Ask tells you which setting a map needs. + +--- + +## Development Run the backend as above, plus the Vite dev server with hot reload (it forwards `/api` to port 8001): @@ -84,10 +159,6 @@ uv run python -m scripts.export_openapi cd frontend && npm run gen:api ``` -How answers are found: `backend/synthesis/chat.py` (hybrid search: vector similarity plus BM25 keyword search, fused by reciprocal rank). - -How maps are built: `backend/topics.py` (article vectors → UMAP → HDBSCAN → c-TF-IDF, BERTopic's recipe without PyTorch) and `backend/embedding/project.py` (naming topics with Claude). - Checks (CI runs all of these on every push and pull request): ```bash @@ -96,7 +167,7 @@ uv run pytest cd frontend && npm run check && npm run lint && npx vitest run ``` -End-to-end tests drive the real app (backend + built frontend) against the demo map in a throwaway `.e2e/` folder, with no network or API keys: +End-to-end tests drive the real app (backend and built frontend) against the demo map in a throwaway `.e2e/` folder, with no network or API keys: ```bash cd frontend && npm run build && npx playwright install chromium && npm run test:e2e @@ -104,51 +175,21 @@ cd frontend && npm run build && npx playwright install chromium && npm run test: --- -## Using it - -1. On the home page, type a topic (e.g. `mechanistic interpretability in large language models`) and choose **Map this topic**. Tick **Let me check the sources before they're read** to see the searches and pages first: edit the searches, untick pages, or add your own before anything is read. -2. A map takes **3–8 minutes**. Academic publishers often block automated reading; for those pages (and for PDF links) Discourse Lens looks up an open-access copy or the abstract through OpenAlex and Europe PMC, and the article reader says where the text came from. You'll see each step as it happens: the search queries, pages crawled, articles kept or skipped (and why), and clusters as they're named. You can leave the page, or **Cancel this map**. -3. When it's done you get six views of the same map: - - **Map**: articles that make similar arguments sit close together, grouped into topics (camps) with contour lines and a name for each. Switch to **Table** for a plain list. - - **Timeline**: articles per month or year, by cluster. - - **Outliers**: articles furthest from the middle of their cluster. - - **Key terms**: each topic's one-line summary and the words it uses far more than the others. - - **Claims**: the points two or more articles take a position on, and whether each article agrees, qualifies or disputes them, with the most debated first. Click a point to see each article's claim and open the article. This view runs only when you choose **Find claims**, because it costs Claude requests: about half a cent per article (roughly $0.30 for a 50-article map; about a minute). - - **Ask**: questions answered only from this map's articles. Answers stream in as they're written, with numbered citations like **[3]**; click one to read that article. Follow-up questions build on earlier answers, and a citation number means the same article for the whole conversation. Conversations are saved with the map. -4. Pick a topic in the **Clusters** panel to focus every view on it (and read its summary). Articles that don't fit any topic well are grouped as **Unclustered** rather than forced into one. Click any article to read it in a side panel. The address bar keeps the view, cluster and open article, so links and reloads land in the same place. -5. **Export to Obsidian** downloads a vault: a Canvas of the map plus one note per article. -6. Every map is kept. **All maps** returns to the list, where you can open or delete them. - ---- - -## Good topics to start with - -| Topic | Why it works well | -|---|---| -| `mechanistic interpretability in large language models` | Rich, active discourse, clear camps | -| `longevity science and aging reversal` | Strong consensus vs. fringe split | -| `urban heat islands and city design` | Good temporal drift, recent policy debate | -| `jevons paradox and energy efficiency` | Older topic, good for testing the temporal view | - -Avoid topics that are too broad (`AI`, `climate change`) or too narrow (fewer than ~20 articles exist on the web). - ---- - ## Troubleshooting -**Don't want the open-access lookups?** Set `RECOVER_ACADEMIC=false` in `.env`; academic pages a publisher blocks are then skipped. +**A map fails or finds no pages while searching** — Check `BRAVE_API_KEY` (or `SEARXNG_URL` if you use SearXNG) in `.env`. -**Pipeline stalls at "Crawling"** — Check your `BRAVE_API_KEY` in `.env`. +**Topic names look like lists of words ("Roof, albedo, paint")** — Naming the topics with Claude failed, so the app fell back to each topic's top key terms. Check `ANTHROPIC_API_KEY`. -**Cluster labels show as "cluster 0", "cluster 1"...** — The Anthropic API call for labelling failed. Check your `ANTHROPIC_API_KEY`. +**Embedding is slow** — Local embedding runs on the CPU: about 3 seconds per 10 chunks, so a few minutes for a typical map. `EMBEDDING_MODE=openai` is faster, for new maps only (see [Settings](#settings)). -**Embeddings are slow** — Local embedding runs on the CPU: about 3 seconds per 10 chunks, so a few minutes for a typical map. For speed, set `EMBEDDING_MODE=openai` and `OPENAI_API_KEY=your_key` in `.env` (new maps only; see Setup). +**Answers are slow to start or too brief** — Set `CHAT_EFFORT` to `low` or `high`. In testing, `medium` started answering in about a second, the same as `low`. -**Answers are slow to start or too brief** — `CHAT_EFFORT` in `.env` sets how much the model reasons before answering (`low`, `medium` (default), `high`). In testing, `medium` started answering in about a second, same as `low`. +**Ask says "This analysis has no indexed corpus yet"** — The map hasn't finished, or no articles passed the quality filter. Try a different topic. -**Chat says "This analysis has no indexed corpus yet"** — The analysis hasn't completed, or no articles passed the quality filter. Try re-running with a different topic. +**Don't want the open-access lookups?** Set `RECOVER_ACADEMIC=false`; pages a publisher blocks are then skipped. -**Upgrading from an older version** — On first start the backend migrates the database. Maps made before topic modelling keep their old clusters, without key terms or summaries, until you rebuild them from the stored embeddings (no re-crawl; one Claude request per map): +**Upgrading from an older version** — On first start the backend migrates the database. Maps made before topic modelling keep their old clusters, without key terms or summaries, until you rebuild them from the stored vectors (no re-crawl; one Claude request per map): ```bash uv run python -m scripts.recluster --all diff --git a/backend/config.py b/backend/config.py index eaf8223..fdab99c 100644 --- a/backend/config.py +++ b/backend/config.py @@ -47,8 +47,6 @@ class Settings(BaseSettings): lance_path: str = "./data/vectors" # Tuning - umap_n_neighbors: int = 15 - kmeans_k: str = "auto" # "auto" | integer string rag_top_k: int = 15 # Thinking effort for chat answers: low answers sooner, medium reasons more. chat_effort: Literal["low", "medium", "high"] = "medium" diff --git a/backend/ingestion/search.py b/backend/ingestion/search.py index 436fdeb..d3147b6 100644 --- a/backend/ingestion/search.py +++ b/backend/ingestion/search.py @@ -9,7 +9,9 @@ from __future__ import annotations +import html import logging +import re import time from collections.abc import Callable from dataclasses import dataclass @@ -23,6 +25,13 @@ RESULTS_PER_QUERY = 20 +_TAG = re.compile(r"<[^>]+>") + + +def plain(text: str | None) -> str: + """Titles and snippets arrive as HTML ( around matches, ').""" + return " ".join(html.unescape(_TAG.sub("", text or "")).split()) + @dataclass class SearchResult: @@ -78,7 +87,7 @@ def brave_search(query: str, count: int = RESULTS_PER_QUERY) -> list[SearchResul headers={"Accept": "application/json", "X-Subscription-Token": settings.brave_api_key}, ) return [ - SearchResult(r["url"], r.get("title") or "", r.get("description") or "", query) + SearchResult(r["url"], plain(r.get("title")), plain(r.get("description")), query) for r in resp.json().get("web", {}).get("results", []) if "url" in r ][:count] @@ -93,7 +102,7 @@ def searxng_search(query: str, count: int = RESULTS_PER_QUERY) -> list[SearchRes headers={"Accept": "application/json"}, ) return [ - SearchResult(r["url"], r.get("title") or "", r.get("content") or "", query) + SearchResult(r["url"], plain(r.get("title")), plain(r.get("content")), query) for r in resp.json().get("results", []) if "url" in r ][:count] diff --git a/docs/screenshots/ask.png b/docs/screenshots/ask.png new file mode 100644 index 0000000..faa8899 Binary files /dev/null and b/docs/screenshots/ask.png differ diff --git a/docs/screenshots/claims.png b/docs/screenshots/claims.png new file mode 100644 index 0000000..66c76b9 Binary files /dev/null and b/docs/screenshots/claims.png differ diff --git a/docs/screenshots/home.png b/docs/screenshots/home.png new file mode 100644 index 0000000..b19304c Binary files /dev/null and b/docs/screenshots/home.png differ diff --git a/docs/screenshots/map-dark.png b/docs/screenshots/map-dark.png new file mode 100644 index 0000000..2200856 Binary files /dev/null and b/docs/screenshots/map-dark.png differ diff --git a/docs/screenshots/map.png b/docs/screenshots/map.png new file mode 100644 index 0000000..c409be4 Binary files /dev/null and b/docs/screenshots/map.png differ diff --git a/docs/screenshots/reader.png b/docs/screenshots/reader.png new file mode 100644 index 0000000..b617ab8 Binary files /dev/null and b/docs/screenshots/reader.png differ diff --git a/docs/screenshots/review.png b/docs/screenshots/review.png new file mode 100644 index 0000000..13fad37 Binary files /dev/null and b/docs/screenshots/review.png differ diff --git a/docs/screenshots/terms.png b/docs/screenshots/terms.png new file mode 100644 index 0000000..66420fc Binary files /dev/null and b/docs/screenshots/terms.png differ diff --git a/docs/screenshots/timeline.png b/docs/screenshots/timeline.png new file mode 100644 index 0000000..952e75e Binary files /dev/null and b/docs/screenshots/timeline.png differ diff --git a/frontend/src/lib/citations.spec.ts b/frontend/src/lib/citations.spec.ts index 8fbde06..41d9325 100644 --- a/frontend/src/lib/citations.spec.ts +++ b/frontend/src/lib/citations.spec.ts @@ -25,7 +25,15 @@ describe('linkCitations', () => { it('leaves markdown links untouched', () => { const text = 'See [the report](https://x.test) [2].'; expect(linkCitations(text, sources)).toMatch( - /^See \[the report\]\(https:\/\/x\.test\) `; - }) - .join(''); + return markdown.replace(RUN, (run) => { + let linked = false; + const html = run.replace(MARKER, (match, group: string) => { + const ns = numbersIn(group); + if (!ns.length || !ns.every((n) => byN.has(n))) return match; + linked = true; + return ns + .map((n) => { + const s = byN.get(n)!; + const label = escapeAttr(`${s.title} (${s.domain})`); + return ``; + }) + .join(''); + }); + return linked ? `${html}` : run; }); } diff --git a/frontend/src/lib/components/Answer.svelte b/frontend/src/lib/components/Answer.svelte index 5b950df..21c8c67 100644 --- a/frontend/src/lib/components/Answer.svelte +++ b/frontend/src/lib/components/Answer.svelte @@ -134,6 +134,9 @@ line-height: 1; cursor: pointer; } + .answer :global(.cite-group) { + white-space: nowrap; + } .answer :global(.cite:hover) { background: var(--accent); color: var(--accent-ink); diff --git a/frontend/src/routes/+page.svelte b/frontend/src/routes/+page.svelte index d36d55a..b9b0571 100644 --- a/frontend/src/routes/+page.svelte +++ b/frontend/src/routes/+page.svelte @@ -88,7 +88,8 @@

Map a conversation.

Enter a topic. Discourse Lens reads around a hundred articles about it and charts the camps, - the outliers and how the conversation has moved. A map takes 3 to 8 minutes. + where they agree and disagree, and how the conversation has moved. A map takes 3 to 8 + minutes.

diff --git a/sanity-check.md b/sanity-check.md index 1d16d9d..553f69f 100644 --- a/sanity-check.md +++ b/sanity-check.md @@ -1,18 +1,20 @@ # Discourse Lens — Personal Sanity Check -Last updated: v1.0 +Last updated: October 2026, after the modernization (v0.2) --- ## What We're Building (Plain English) -You type a topic. The tool searches the web, reads ~100 articles, and builds a semantic map of the whole discourse. You get four views: +You type a topic. The tool searches the web, reads ~100 articles, and builds a semantic map of the whole discourse. You get six views: -- **Cluster map** — what camps / schools of thought exist -- **Temporal drift** — how the topic has evolved over time -- **Outlier finder** — the genuinely novel or fringe takes -- **Chat interface** — ask questions, get answers with cited sources +- **Map**: what camps / schools of thought exist, as named topics +- **Timeline**: how coverage has moved over time +- **Outliers**: the genuinely novel or fringe takes +- **Key terms**: what each camp talks about that the others don't +- **Claims**: where the sources agree, qualify or dispute each other +- **Ask**: questions answered only from the articles, with clickable citations -Everything runs locally. No accounts, no cloud infra beyond API calls. +Everything runs locally. No accounts, no cloud infra beyond the search and Claude requests. > **The insight:** most research tools help you *retrieve*. This one helps you *orient*. It answers "what is the shape of this conversation?" not just "what does this article say?" @@ -22,44 +24,47 @@ Everything runs locally. No accounts, no cloud infra beyond API calls. | Thing | What it is | How to start it | |-------|-----------|-----------------| -| FastAPI (Python) | Backend — all the heavy lifting | `uvicorn backend.api.main:app --reload` | -| SvelteKit (Node) | Frontend UI | `cd frontend && npm run dev` | -| SQLite | Stores articles + chunks as text | Auto-created at `data/discourse.db` | -| LanceDB | Stores embedding vectors | Auto-created at `data/vectors/` | -| Brave Search API | Finds articles to crawl | Get free key → brave.com/search/api | -| Anthropic API | Powers the chat synthesis | Your existing key | -| Ollama / nomic-embed-text | Turns text into vectors locally | `ollama pull nomic-embed-text` | +| FastAPI (Python) | Backend, and it serves the built app too | `uv run uvicorn backend.api.main:app --port 8001` | +| SvelteKit (Node) | Frontend UI, built into `frontend/build` | `cd frontend && npm run build` (or `npm run dev` on :5173 while editing) | +| SQLite | Maps, articles, chunks, claims, conversations | Auto-created at `data/discourse.db` | +| LanceDB | Embedding vectors, one table per map | Auto-created at `data/vectors/` | +| fastembed / nomic-embed-text | Turns text into vectors inside the app | Downloads itself once (~520 MB) to `data/models/` | +| Brave Search API (or SearXNG) | Finds articles to read | Free key at brave.com/search/api | +| Anthropic API | Queries, topic names, claims, answers | Your existing key | --- -## The Six Stages +## How It Was Built -| Stage | Name | What gets built | Done when... | -|-------|------|----------------|--------------| -| 0 | Scaffold | Folder structure, env vars, health endpoint | `curl /health` returns 200 | -| 1 | Ingestion | Crawls ~100 articles from topic string | 70+ rows in `articles` table | -| 2 | Processing | Chunks + deduplicates article text | 300+ rows in `chunks` table | -| 3 | Embedding | Embeds chunks, runs UMAP, labels clusters | Scatter plot shows distinct clusters | -| 4 | Visualization | All 4 views working in browser | Cross-view filtering works | -| 5 | Synthesis | RAG chat answers questions with citations | Chat cites 3+ articles | -| 6 | Polish | Progress bars, error handling, export | You'd show this to someone | +The original six stages (scaffold → ingestion → processing → embedding → visualization → synthesis → polish) were finished in April 2026. In October 2026 it was overhauled in six steps (merged as PR #2): -**Current stage:** `[ fill this in ]` +| Step | What changed | +|------|--------------| +| 1 | Quality bug fixes: dedup, embedding prefixes, RAG ranking, outliers | +| 2 | Backend restructure: uv, many maps instead of one corpus, cancellable runs with a live feed | +| 3 | Frontend rebuilt: SvelteKit 3, Svelte 5, TypeScript, Tailwind 4 | +| 4 | Topic model: BERTopic's recipe without PyTorch, named topics with summaries | +| 5 | Ask: streaming answers, hybrid search, stable citations, saved conversations | +| 6 | Academic page recovery, source review, SearXNG, fastembed, Claims view, e2e tests + CI | + +**Current state:** all done and merged to `main`. --- ## Decisions That Are Locked -Don't let Claude Code or yourself relitigate these. They're in the spec for good reasons. +Don't let Claude Code or yourself relitigate these. They're here for good reasons. | Decision | Why it's locked | |----------|----------------| -| UMAP runs server-side in Python | Browser UMAP is slow and inconsistent for 200+ points | -| Two-level chunking (1500 parent / 300 child) | Small chunks for retrieval precision, large context for LLM | +| UMAP and clustering run server-side in Python | Browser UMAP is slow and inconsistent | +| Cluster articles, not chunks | Chunk-level clustering just gave one topic per article | +| No PyTorch | BERTopic's recipe works without it; keeps installs small and CPU-only | +| Two-level chunking (1500 parent / 300 child) | Small chunks for retrieval precision, large context for the answer | | LanceDB for vectors, SQLite for everything else | Zero infra, no Postgres, no Redis — keep it simple | -| Frontend never calls Anthropic directly | Security + easier to add auth later | -| Embedding mode is configurable via `EMBEDDING_MODE` | Free local (Ollama) or fast API (OpenAI) | -| No user accounts in v1 | Single active session. State lives in DB files. | +| Frontend never calls Anthropic directly | Keys stay on the server | +| Claims run only when asked | They cost Claude requests on every map | +| No user accounts | Local tool; state lives in the `data/` folder | --- @@ -67,22 +72,23 @@ Don't let Claude Code or yourself relitigate these. They're in the spec for good | Problem | Likely cause | Fix | |---------|-------------|-----| -| Crawl gets < 50 articles | Paywalls, bot detection, dead links | Increase query count; add retry logic; use archive.org fallback | -| UMAP clusters look like a blob | Not enough articles or topic too broad | Try 150+ articles; narrow the topic string | -| Embeddings are slow | Running local model on CPU | Switch to `EMBEDDING_MODE=openai` temporarily | -| Publish dates all null | Sites don't use standard meta tags | Check URL slug extraction fallback in `metadata.py` | -| RAG answers are vague | Chunks too small or context window too short | Increase parent chunk size to 2000 tokens; raise `RAG_TOP_K` to 20 | -| D3 scatter is slow at 200+ nodes | Too many SVG DOM nodes | Switch cluster map to canvas renderer | +| Crawl keeps < 40 articles | Paywalls, bot detection, dead links | Academic pages are already recovered via OpenAlex / Europe PMC; try a broader topic or turn on source review and add pages | +| Map looks like one blob | Topic too broad or too few articles | Narrow the topic; articles that fit nowhere go to Unclustered on purpose | +| Embeddings are slow | Running on CPU | It's ~3 s per 10 chunks; `EMBEDDING_MODE=openai` is faster for new maps | +| Publish dates all null | Sites don't use standard meta tags | Check the URL fallback in `backend/processing/metadata.py` | +| Answers are vague | Not enough context | Raise `RAG_TOP_K`, or `CHAT_EFFORT=high` | +| Ask errors about a different model | Map made with another `EMBEDDING_MODE` | Set the mode the message names | +| A test passes on Windows but fails in CI | UMAP floating point differs by platform | Assert on the meaning, not exact labels | --- ## How to Work with Claude Code -- Start each session: `"I am working on Stage N of Discourse Lens. Read PLANNING.md first."` -- Keep sessions focused on **one stage at a time** — context bleed causes drift -- If Claude Code suggests a different tech choice, ask it to justify against the spec before agreeing -- Always run the stage verification command before moving to the next stage -- If something breaks, share the error message **and** the relevant section of `PLANNING.md` together +- Start with: `"Read README.md and validation.md first."` They describe the current app; `Planning.md` is the original v1 plan. +- Keep sessions focused on **one change at a time**; context bleed causes drift +- If Claude Code suggests a different tech choice, ask it to justify it against the locked decisions above +- Run the checks in `validation.md` before calling something done +- If something breaks, share the error message **and** what you were doing --- @@ -92,21 +98,22 @@ Don't let Claude Code or yourself relitigate these. They're in the spec for good |-----|----------------|-----------|--------| | `BRAVE_API_KEY` | brave.com/search/api | Yes — 2000 queries/month | `[ ]` | | `ANTHROPIC_API_KEY` | console.anthropic.com | No — pay per token | `[ ]` | -| `OPENAI_API_KEY` | platform.openai.com | No — optional anyway | `[ ]` | +| `OPENAI_API_KEY` | platform.openai.com | No — optional, only for `EMBEDDING_MODE=openai` | `[ ]` | + +Rough costs: a map is a few cents of Claude; Claims is about half a cent per article; an Ask answer is a cent or two. --- ## Session Checklist **Before each session:** -- [ ] Backend running? (`uvicorn backend.api.main:app --reload`) -- [ ] Frontend running? (`cd frontend && npm run dev`) -- [ ] Which stage am I on? -- [ ] Did I verify the previous stage's output? +- [ ] `git pull` on `main` +- [ ] Backend running? (`uv run uvicorn backend.api.main:app --port 8001`) +- [ ] Frontend rebuilt if it changed? (`cd frontend && npm run build`) **After each session:** -- [ ] Run the stage verification command -- [ ] `git commit -m "Stage N complete: [what works]"` +- [ ] `uv run pytest` and `cd frontend && npm run check && npm run lint` +- [ ] Commit on a branch, open a PR, let CI run - [ ] Note anything surprising — it will matter later --- @@ -118,8 +125,8 @@ Start with these — they have clear discourse structure and good web coverage. | Topic string | Why it's good | |---|---| | `mechanistic interpretability in large language models` | Rich, active, 2022–present, clear camps | -| `longevity science and aging reversal` | Lots of camps, clear consensus vs. fringe | -| `urban heat islands and city design` | Good temporal drift, recent policy debate | -| `jevons paradox and energy efficiency` | Older topic, good for testing temporal view | +| `longevity science and aging reversal` | Real disagreements, great for Claims | +| `urban heat islands and city design` | Good timeline, recent policy debate | +| `jevons paradox and energy efficiency` | Older topic, long timeline | -**Avoid:** Topics too broad (`AI`, `climate change`) or too narrow (< 20 articles exist). The sweet spot is a specific sub-topic with 50–200 good articles. \ No newline at end of file +**Avoid:** Topics too broad (`AI`, `climate change`) or too narrow (< 20 articles exist). The sweet spot is a specific sub-topic with 50–200 good articles. diff --git a/tests/test_search.py b/tests/test_search.py index 8edae7e..5a18fb7 100644 --- a/tests/test_search.py +++ b/tests/test_search.py @@ -56,3 +56,9 @@ def fake(query): assert [r.url for r in results] == ["https://x.test/1", "https://x.test/2"] assert [r.query for r in results] == ["one", "one"] assert progress == [("one", 2), ("broken", None), ("two", 0)] + + +def test_titles_and_snippets_become_plain_text(): + assert search.plain("Labor leaders' four-day week") == ( + "Labor leaders' four-day week") + assert search.plain(None) == "" diff --git a/validation.md b/validation.md index 80a0d56..65d6980 100644 --- a/validation.md +++ b/validation.md @@ -1,314 +1,144 @@ # Discourse Lens — Validation Guide -This file explains how to run a fresh pipeline job and verify each completed stage. +How to check that Discourse Lens works: the automated checks first, then a real map built step by step through the API, with what to look for at each stage. --- -## Prerequisites +## 1. Automated checks -```bash -# Activate the virtual environment (Windows/Git Bash) -source venv/Scripts/activate - -# Start the backend (port 8001) -python -m uvicorn backend.api.main:app --reload --port 8001 -``` - -The backend initialises the SQLite schema on startup — safe to restart at any time. - ---- - -## Running a fresh job - -If you want a clean slate before testing (e.g. to validate Stage 2 on a fresh corpus): - -```bash -# Delete the existing database so all tables start empty -rm data/discourse.db -``` - -Then submit a new job: +These need no API keys and no network. CI runs all of them on every push and pull request. ```bash -curl -X POST localhost:8001/api/jobs \ - -H "Content-Type: application/json" \ - -d '{"topic": "mechanistic interpretability in large language models"}' -``` - -**Windows PowerShell alternative:** - -```powershell -Invoke-RestMethod -Method Post -Uri http://localhost:8001/api/jobs ` - -ContentType "application/json" ` - -Body '{"topic": "mechanistic interpretability in large language models"}' +uv run ruff check backend tests scripts # lint +uv run pytest # backend: ~130 tests +cd frontend +npm run check && npm run lint # types and formatting +npx vitest run # frontend unit tests +npm run build && npm run test:e2e # browser tests against the demo map ``` -The response contains a `job_id`: - -```json -{"job_id": "some-uuid-here"} -``` +The browser tests start their own backend on port 8011 with a throwaway `.e2e/` data folder, so they never touch your maps. --- -## Polling job status +## 2. Look around without keys ```bash -curl localhost:8001/api/jobs/ -``` - -**Windows PowerShell alternative:** - -```powershell -Invoke-RestMethod -Uri http://localhost:8001/api/jobs/ +uv run python -m scripts.seed_demo # add --reset to rebuild it +uv run uvicorn backend.api.main:app --port 8001 ``` -The `status` field progresses through these values: - -| Status | What is happening | -|--------|------------------| -| `queued` | Job created, not started yet | -| `crawling` | Generating search queries and scraping URLs | -| `processing` | Extracting text, chunking, deduplicating | -| `embedding` | Embedding non-duplicate chunks and writing to LanceDB | -| `projecting` | Running UMAP, k-means clustering, and labelling clusters | -| `done` | Pipeline complete | -| `error` | Something failed — check `error_msg` | - -Poll every 10–15 seconds. The full pipeline typically takes 3–8 minutes depending on network speed and article count. +Open [http://localhost:8001](http://localhost:8001) and open **Demo: urban heat and city design**. It has 12 made-up articles in three topics plus one unclustered article, a saved conversation and two claims. Every view should render; Ask can show the saved conversation but can't answer new questions without keys. --- -## Stage verification commands - -Run these after the job reaches `status: "done"`. +## 3. Build a real map -### Stage 0 — Health check +With `BRAVE_API_KEY` and `ANTHROPIC_API_KEY` in `.env` and the backend running: ```bash -curl localhost:8001/api/health -# Expected: {"status":"ok"} +curl -X POST localhost:8001/api/analyses \ + -H "Content-Type: application/json" \ + -d '{"topic": "mechanistic interpretability in large language models"}' ``` -**Windows PowerShell alternative:** +**Windows PowerShell:** ```powershell -Invoke-RestMethod -Uri http://localhost:8001/api/health -# Expected: status ok -``` - -### Stage 1 — Ingestion - -```bash -sqlite3 data/discourse.db "SELECT COUNT(*) FROM articles WHERE status='ok';" -# Expected: 70 or more -``` - -To see a breakdown of all statuses: - -```bash -sqlite3 data/discourse.db "SELECT status, COUNT(*) FROM articles GROUP BY status;" -``` - -### Stage 2 — Processing (chunking + deduplication) - -```bash -# Unique (non-duplicate) child chunks — should be 300+ -sqlite3 data/discourse.db "SELECT COUNT(*) FROM chunks WHERE is_duplicate=0;" - -# Duplicate child chunks — should be > 0 (proves dedup ran) -sqlite3 data/discourse.db "SELECT COUNT(*) FROM chunks WHERE is_duplicate=1;" -``` - -To inspect a sample chunk: - -```bash -sqlite3 data/discourse.db \ - "SELECT id, article_id, chunk_index, token_count, is_duplicate, substr(child_text,1,120) \ - FROM chunks LIMIT 5;" -``` - -To confirm parent/child structure is correct: - -```bash -sqlite3 data/discourse.db \ - "SELECT article_id, COUNT(*) as total_chunks, \ - SUM(is_duplicate) as dupes \ - FROM chunks GROUP BY article_id LIMIT 10;" +Invoke-RestMethod -Method Post -Uri http://localhost:8001/api/analyses ` + -ContentType "application/json" ` + -Body '{"topic": "mechanistic interpretability in large language models"}' ``` -### Stage 3 — Embeddings & projections +The response is the new analysis; keep its `id`. Add `"review": true` to pause after searching (see [Source review](#source-review)). -First confirm the job reached `status: "done"` with `embedding` and `projecting` stages completing (you can watch progress tick through 80% → 90% → 100% while polling). - -**Check the projections table is populated:** +### Watch it run ```bash -sqlite3 data/discourse.db "SELECT COUNT(*) FROM projections;" -# Expected: 300 or more rows +curl localhost:8001/api/analyses/ # status, progress (0-100), stage_detail +curl localhost:8001/api/analyses//events # every step so far ``` -**Check cluster IDs are assigned (should see multiple distinct values):** +The app shows the same feed live (it streams `/api/analyses//stream`). `status` moves through: -```bash -sqlite3 data/discourse.db \ - "SELECT cluster_id, cluster_label, COUNT(*) as chunks \ - FROM projections GROUP BY cluster_id ORDER BY cluster_id;" -# Expected: 5–10 rows, each with a 2-3 word label and a chunk count -``` +| Status | Progress | What is happening | +|---|---|---| +| `queued` | 0 | Created, about to start | +| `crawling` | 0–55 | Claude writes 6 search queries, the search provider runs them, then pages are loaded | +| `reviewing` | 10 | Only with `"review": true`: waiting for you to approve pages | +| `processing` | 55–68 | Extracting text, recovering blocked academic pages, dropping duplicates, chunking | +| `embedding` | 68–88 | Embedding chunks into LanceDB | +| `projecting` | 88–99 | Topic model, map layout, one Claude request to name the topics | +| `done` | 100 | Finished; `article_count` is the number of articles on the map | +| `error` / `cancelled` | | `error_msg` says what went wrong | -**Check x/y coordinates look reasonable (should not all be 0.0):** +A map takes 3 to 8 minutes. Cancel with `curl -X POST localhost:8001/api/analyses//cancel`. -```bash -sqlite3 data/discourse.db \ - "SELECT MIN(x), MAX(x), MIN(y), MAX(y) FROM projections;" -# Expected: non-zero ranges spanning several units, e.g. -6.3 | 8.1 | -5.0 | 7.4 -``` +### Check each stage -**Hit the /api/corpus endpoint and confirm shape:** +These use Python's built-in `sqlite3`, so no extra tools are needed. Replace `` in each. ```bash -curl localhost:8001/api/corpus | python3 -c \ - "import sys,json; d=json.load(sys.stdin); \ - print(f'{len(d)} points, clusters: {set(p[\"cluster_id\"] for p in d)}')" -# Expected: 300+ points with multiple cluster IDs, e.g.: -# 412 points, clusters: {0, 1, 2, 3, 4, 5, 6} -``` +uv run python - <<'EOF' +import sqlite3 +db = sqlite3.connect("data/discourse.db") +a = "" +q = lambda sql: db.execute(sql, (a,)).fetchall() -**Windows PowerShell alternative:** - -```powershell -$d = Invoke-RestMethod -Uri http://localhost:8001/api/corpus -Write-Host "$($d.Count) points" -$d | Select-Object -ExpandProperty cluster_id | Sort-Object -Unique -# Expected: 300+ points, several distinct cluster IDs listed -``` - -**Inspect a sample corpus point to confirm all fields are present:** - -```bash -curl -s localhost:8001/api/corpus | python3 -c \ - "import sys,json; d=json.load(sys.stdin); import pprint; pprint.pprint(d[0])" -# Expected: dict with chunk_id, article_id, url, title, domain, publish_date, -# x, y, cluster_id, cluster_label, distance_from_centroid, word_count +print("articles by status:", q("SELECT status, COUNT(*) FROM articles WHERE analysis_id=? GROUP BY status")) +print("recovered:", q("SELECT source_note, COUNT(*) FROM articles WHERE analysis_id=? AND source_note IS NOT NULL GROUP BY source_note")) +print("chunks (unique, duplicate):", q("SELECT SUM(is_duplicate=0), SUM(is_duplicate=1) FROM chunks WHERE analysis_id=?")) +print("placed on the map:", q("SELECT COUNT(*) FROM placements WHERE analysis_id=?")) +print("topics:", q("SELECT cluster_id, label, article_count FROM clusters WHERE analysis_id=? ORDER BY cluster_id")) +print("unclustered:", q("SELECT COUNT(*) FROM placements WHERE analysis_id=? AND cluster_id=-1")) +EOF ``` -**Windows PowerShell alternative:** +What good looks like: -```powershell -$d = Invoke-RestMethod -Uri http://localhost:8001/api/corpus -$d[0] | Format-List -# Expected: all fields listed above present on the first item -``` - -**Check LanceDB vectors directory was created:** +- **Ingestion:** 40 or more articles with status `ok`. `filtered` pages were too short, not English or unreadable; `duplicate` ones repeated another article. On academic topics, some articles have a `source_note` such as "Full text from Europe PMC". +- **Chunks:** usually a few thousand unique chunks (around 50 per article). +- **Map:** every `ok` article has a placement. 2 to 8 topics, each with a short descriptive name (not a list of words, which means naming fell back to key terms), plus a handful unclustered at most. +- **Vectors:** `data/vectors/` contains a table named `emb_`. -```bash -ls data/vectors/ -# Expected: directory exists and contains LanceDB files (embeddings.lance/ etc.) -``` +Over the API, `GET /api/analyses//articles` returns each article's position and topic, and `GET /api/analyses//clusters` returns topic names, summaries and key terms. -### Stage 6 — Polish (export + chat retry) +### Source review -**Run the unit tests:** +Create the analysis with `"review": true`. When `status` is `reviewing`: ```bash -source venv/Scripts/activate -pytest tests/test_export.py -v -# Expected: 12 passed +curl localhost:8001/api/analyses//plan # {queries, results[{url, title, snippet, query}]} ``` -**Test the export endpoint directly:** - -```bash -curl -o test-export.zip localhost:8001/api/export -# Expected: file downloads (non-zero size) -# If no corpus: {"error": "No corpus found. Run an analysis first."} - -unzip -l test-export.zip -# Expected: entries like: -# discourse-/cluster-map.canvas -# discourse-/vault/_index.md -# discourse-/vault/.md (one per article) -``` - -**Windows PowerShell alternative:** - -```powershell -Invoke-WebRequest -Uri http://localhost:8001/api/export -OutFile test-export.zip -# Then list the contents: -Add-Type -AssemblyName System.IO.Compression.FileSystem -[System.IO.Compression.ZipFile]::OpenRead("$PWD\test-export.zip").Entries | Select-Object FullName -# Expected: cluster-map.canvas, vault/_index.md, vault/.md entries -``` - -**Verify the canvas file is valid JSON:** - -```bash -unzip -p test-export.zip "*/cluster-map.canvas" | python3 -c \ - "import sys, json; d=json.load(sys.stdin); \ - print(f'{len(d[\"nodes\"])} nodes, edges: {len(d[\"edges\"])}')" -# Expected: N nodes (one per article), 0 edges -``` - -**Verify a vault article has correct frontmatter:** - -```bash -unzip -p test-export.zip "*/vault/_index.md" | head -20 -# Expected: starts with "# ", contains cluster table and [[wikilinks]] -``` - -**Test in Obsidian:** - -1. Unzip the export -2. Open `cluster-map.canvas` in Obsidian — article cards should appear color-coded by cluster, spatially arranged by UMAP position -3. Open the `vault/` folder as an Obsidian vault — graph view should show `_index.md` linking to all article notes -4. Install the Dataview plugin and run: `TABLE cluster, domain FROM "vault" SORT publish_date` — should return rows with frontmatter fields - -**Test chat retry:** - -1. Stop the backend server -2. In the UI, submit a chat query — an error message should appear -3. A "Try again" button should appear below the error -4. Restart the backend and click "Try again" — query should re-submit successfully +In the app, untick a group and add one of your own addresses, then **Read N pages**. The events should say `Approved N pages`, then `Crawling N pages` with the same N. Titles and snippets should be plain text (no `'` or ``). --- -## Useful ad-hoc queries +## 4. Check the views -```bash -# Total tokens across all non-duplicate chunks -sqlite3 data/discourse.db \ - "SELECT SUM(token_count) FROM chunks WHERE is_duplicate=0;" - -# Articles with the most chunks -sqlite3 data/discourse.db \ - "SELECT a.title, COUNT(c.id) as chunks \ - FROM chunks c JOIN articles a ON c.article_id=a.id \ - WHERE c.is_duplicate=0 \ - GROUP BY c.article_id ORDER BY chunks DESC LIMIT 10;" - -# Duplicate rate per article -sqlite3 data/discourse.db \ - "SELECT article_id, \ - ROUND(100.0 * SUM(is_duplicate) / COUNT(*), 1) as dup_pct \ - FROM chunks GROUP BY article_id ORDER BY dup_pct DESC LIMIT 10;" -``` +Open the map in the app and go through each view: + +- **Map:** dots grouped by topic with contour lines and names. Drag, zoom, click a dot to open the reader. **Table** lists every article. +- **Clusters panel:** picking a topic fades the others in every view and shows its summary; the address bar gains `?cluster=`. +- **Timeline:** bars per year or month, coloured by topic. +- **Outliers:** articles sorted by distance from their topic's centre. +- **Key terms:** one block per topic with a summary and weighted terms. +- **Claims:** choose **Find claims** (about half a cent per article). Progress shows "Read N of M articles", then "Comparing N claims". When it's done, each point has a bar and counts, the most debated first; expanding one lists each article's claim, and clicking an article opens it. +- **Ask:** ask a question. Sources appear first, then the answer streams in with numbered citations; a citation and the punctuation after it never split across lines. Clicking a citation opens that article. Ask a follow-up: new sources continue the numbering. Reload: the conversation is still there. +- **Export to Obsidian:** downloads `discourse-.zip` containing `discourse-/cluster-map.canvas` and `discourse-/vault/` (an `_index.md` plus one note per article). Opening the canvas in Obsidian shows article cards coloured by topic, laid out like the map. +- **Theme:** the moon/sun button switches dark and light, and the choice survives a reload. --- ## Troubleshooting -**Job stuck at `crawling`** — Brave Search API key missing or quota exceeded. Check `.env` and `BRAVE_API_KEY`. - -**`article_count` is 0 after `done`** — trafilatura filtered all articles. Try a broader topic with more web coverage. - -**`chunks` table is empty after `done`** — The pipeline ran before Stage 2 was wired in. Delete the DB and run a fresh job. +**Fails or finds nothing while searching** — Check `BRAVE_API_KEY` (or `SEARXNG_URL`) in `.env`, and your Brave quota. -**Job fails at `embedding` status** — If using `EMBEDDING_MODE=local`, confirm Ollama is running (`ollama serve`) and `nomic-embed-text` is pulled (`ollama pull nomic-embed-text`). If using `EMBEDDING_MODE=openai`, check `OPENAI_API_KEY` in `.env`. +**`article_count` is 0 after `done`** — Every page was filtered. Try a topic with more coverage on the web. -**`projections` table is empty after `done`** — Embedding step may have produced no rows (check `data/vectors/` exists and has content). Also check `error_msg` on the job — UMAP requires at least `n_neighbors + 1` data points. +**Fails while embedding** — With `EMBEDDING_MODE=fastembed`, the first run downloads the model (about 520 MB); check your connection and disk space. With `ollama`, confirm Ollama is running and `nomic-embed-text` is pulled. With `openai`, check `OPENAI_API_KEY`. -**Cluster labels are all `"cluster N"`** — Claude API call failed during labelling (likely missing or invalid `ANTHROPIC_API_KEY`). The projections are still valid; only the text labels fall back to the default. +**Topic names are lists of words** — Naming with Claude failed and the app fell back to key terms. Check `ANTHROPIC_API_KEY`; `uv run python -m scripts.recluster ` renames without re-crawling. -**`sqlite3: command not found` (Windows)** — Use the SQLite CLI bundled with Git Bash, or install it via `winget install SQLite.SQLite`. +**Ask says the map was embedded with a different model** — The map's vectors came from another `EMBEDDING_MODE`; the message says which one to set.