Modernize Discourse Lens: new frontend, topic model, streaming Ask, claims view

.env.example

@@ -6,17 +6,18 @@ CLAUDE_BACKEND=anthropic
 
 # Anthropic API (used when CLAUDE_BACKEND=anthropic)
 ANTHROPIC_API_KEY=your_key_here
-CLAUDE_MODEL=claude-sonnet-4-6  # override default model
+CLAUDE_MODEL=claude-sonnet-5-5  # override default model
 
 # Vertex AI (used when CLAUDE_BACKEND=vertex)
-# Prerequisites: pip install anthropic[vertex]  &&  gcloud auth application-default login
+# Prerequisites: uv sync --extra vertex  &&  gcloud auth application-default login
 GOOGLE_CLOUD_PROJECT=your_gcp_project_id
-VERTEX_REGION=us-east5          # region where Claude is deployed in your project
-VERTEX_CLAUDE_MODEL=claude-sonnet-4-0@20250514  # override Vertex model
+VERTEX_REGION=global            # global, a multi-region (us / eu), or a specific region
+VERTEX_CLAUDE_MODEL=claude-sonnet-5-5  # override Vertex model
 
 # Embeddings — choose one mode
-EMBEDDING_MODE=local            # local | openai
-OLLAMA_BASE_URL=http://localhost:11434
+EMBEDDING_MODE=fastembed        # fastembed (in-process) | ollama | openai
+FASTEMBED_CACHE=./data/models   # where fastembed keeps its model (~520 MB)
+OLLAMA_BASE_URL=http://localhost:11434  # only needed if EMBEDDING_MODE=ollama
 OPENAI_API_KEY=your_key_here    # only needed if EMBEDDING_MODE=openai
 
 # Storage paths
@@ -28,3 +29,7 @@ MAX_ARTICLES=120                # cap on total articles to crawl
 UMAP_N_NEIGHBORS=15
 KMEANS_K=auto                   # auto | integer (e.g. 7)
 RAG_TOP_K=15
+CHAT_EFFORT=medium             # low | medium | high: reasoning before chat answers
+RECOVER_ACADEMIC=true         # look up open-access copies of pages academic publishers block
+SEARCH_PROVIDER=brave         # brave | searxng
+SEARXNG_URL=                  # e.g. http://localhost:8888 when SEARCH_PROVIDER=searxng

.gitattributes

@@ -0,0 +1,4 @@
+# Keep line endings consistent across Windows and Linux (CI, tooling)
+* text=auto eol=lf
+*.png binary
+*.svg text eol=lf

.github/workflows/ci.yml

@@ -0,0 +1,75 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+concurrency:
+  group: ci-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  backend:
+    name: Backend (lint, tests)
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v6
+        with:
+          enable-cache: true
+      - run: uv python install 3.12
+      - run: uv sync --locked
+      - run: uv run ruff check backend tests scripts
+      - run: uv run pytest -q
+
+  frontend:
+    name: Frontend (types, lint, unit tests, build)
+    runs-on: ubuntu-latest
+    defaults:
+      run:
+        working-directory: frontend
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: 22
+          cache: npm
+          cache-dependency-path: frontend/package-lock.json
+      # npm 10's resolver crashes on this dependency tree; npm 11 is fine
+      - run: npm install -g npm@11
+      - run: npm ci
+      - run: npm run check
+      - run: npm run lint
+      - run: npx vitest run
+      - run: npm run build
+
+  e2e:
+    name: End-to-end (demo map, no API keys)
+    runs-on: ubuntu-latest
+    needs: [backend, frontend]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v6
+        with:
+          enable-cache: true
+      - run: uv python install 3.12
+      - run: uv sync --locked
+      - uses: actions/setup-node@v4
+        with:
+          node-version: 22
+          cache: npm
+          cache-dependency-path: frontend/package-lock.json
+      - working-directory: frontend
+        run: |
+          npm install -g npm@11
+          npm ci
+          npx playwright install --with-deps chromium
+          npm run build
+          npx playwright test
+      - uses: actions/upload-artifact@v4
+        if: failure()
+        with:
+          name: playwright-traces
+          path: frontend/test-results/
+          retention-days: 7

.gitignore

@@ -17,3 +17,6 @@ frontend/build/
 
 # Claude
 .claude/
+.venv/
+.playwright-mcp/
+.e2e/

.python-version

@@ -0,0 +1 @@
+3.12

README.md

@@ -1,15 +1,14 @@
 # discourse-lens
 
-Discourse Lens takes a topic string as input, crawls ~100 relevant articles from the web, and surfaces patterns in the discourse through semantic visualization. You get five views — a cluster map, temporal drift, outlier finder, consensus bars, and a full-width chat interface — all running locally.
+Discourse Lens takes a topic, reads ~100 relevant articles from the web, and maps the conversation: which camps exist, how coverage has moved over time, which articles don't fit, and what each camp talks about. You can then ask questions answered only from those articles. Everything runs locally except web search and the Claude API calls.
 
 ---
 
 ## Requirements
 
-- Python 3.10+
-- Node.js 18+
-- [Ollama](https://ollama.com) (for local embeddings) — or an OpenAI API key
-- A [Brave Search API key](https://brave.com/search/api) (free tier: 2000 queries/month)
+- Python 3.12 and [uv](https://docs.astral.sh/uv/)
+- Node.js 20.19+ (Vite 8)
+- A [Brave Search API key](https://brave.com/search/api) (free tier: 2000 queries/month), or a self-hosted [SearXNG](https://docs.searxng.org) instance with the JSON format enabled (`SEARCH_PROVIDER=searxng`, `SEARXNG_URL=http://localhost:8888`)
 - An [Anthropic API key](https://console.anthropic.com) (for chat and cluster labelling)
 
 ---
@@ -21,27 +20,24 @@ Discourse Lens takes a topic string as input, crawls ~100 relevant articles from
 ```bash
 git clone https://github.com/Meerav29/discourse-lens.git
 cd discourse-lens
-python -m venv venv
-source venv/Scripts/activate   # Windows Git Bash
-# source venv/bin/activate     # Mac/Linux
-pip install -r requirements.txt
+uv sync                        # creates .venv from uv.lock
+uv run playwright install chromium
 ```
 
-**2. Install frontend dependencies**
+Using Vertex AI for Claude? `uv sync --extra vertex` instead.
+
+**2. Build the frontend**
 
 ```bash
 cd frontend
 npm install
+npm run build
 cd ..
 ```
 
-**3. Pull the embedding model** (skip if using OpenAI)
-
-```bash
-ollama pull nomic-embed-text
-```
+Embeddings run in-process with [fastembed](https://github.com/qdrant/fastembed) (nomic-embed-text v1.5, no GPU or PyTorch needed). The first map downloads the model once (about 520 MB) to `data/models`. Already running [Ollama](https://ollama.com)? `EMBEDDING_MODE=ollama` uses its `nomic-embed-text` instead. It's the same model with the same vectors, so maps work with either. `EMBEDDING_MODE=openai` uses OpenAI's API, but maps made that way can only be searched with OpenAI.
 
-**4. Configure API keys**
+**3. Configure API keys**
 
 Copy `.env.example` to `.env` and fill in your keys:
 
@@ -60,39 +56,68 @@ ANTHROPIC_API_KEY=your_key_here
 
 ## Running
 
-Open two terminals.
+```bash
+uv run uvicorn backend.api.main:app --port 8001
+```
+
+Then open [http://localhost:8001](http://localhost:8001). The backend serves the built app and the API from one address.
 
-**Terminal 1 — Backend:**
+**No API keys yet?** Load a small synthetic demo map to explore the interface:
 
 ```bash
-source venv/Scripts/activate   # Windows Git Bash / source venv/bin/activate on Mac/Linux
-python -m uvicorn backend.api.main:app --reload --port 8001
+uv run python -m scripts.seed_demo
 ```
 
-**Terminal 2 — Frontend:**
+### Working on the frontend
+
+Run the backend as above, plus the Vite dev server with hot reload (it forwards `/api` to port 8001):
 
 ```bash
 cd frontend
 npm run dev
 ```
 
-Then open [http://localhost:5173](http://localhost:5173) in your browser.
+and open [http://localhost:5173](http://localhost:5173). After changing the backend API, regenerate the typed client:
+
+```bash
+uv run python -m scripts.export_openapi
+cd frontend && npm run gen:api
+```
+
+How answers are found: `backend/synthesis/chat.py` (hybrid search: vector similarity plus BM25 keyword search, fused by reciprocal rank).
+
+How maps are built: `backend/topics.py` (article vectors → UMAP → HDBSCAN → c-TF-IDF, BERTopic's recipe without PyTorch) and `backend/embedding/project.py` (naming topics with Claude).
+
+Checks (CI runs all of these on every push and pull request):
+
+```bash
+uv run ruff check backend tests scripts
+uv run pytest
+cd frontend && npm run check && npm run lint && npx vitest run
+```
+
+End-to-end tests drive the real app (backend + built frontend) against the demo map in a throwaway `.e2e/` folder, with no network or API keys:
+
+```bash
+cd frontend && npm run build && npx playwright install chromium && npm run test:e2e
+```
 
 ---
 
 ## Using it
 
-1. Type a topic into the input field (e.g. `mechanistic interpretability in large language models`) and click **Analyze Discourse**
-2. The pipeline runs in the background — crawling, embedding, and clustering ~100 articles. This takes **3–8 minutes** depending on your connection and machine
-3. A progress bar shows the current stage. Once complete, five tabs appear:
-   - **Cluster Map** — semantic clusters of the discourse, plotted in 2D
-   - **Temporal Drift** — how the discourse has shifted over time
-   - **Outliers** — articles furthest from any cluster centroid
-   - **Consensus** — most common terms per cluster, via TF-IDF
-   - **Ask** — full-width chat interface for natural language queries against the indexed corpus; answers render as formatted markdown with cited sources
-4. Click any point on the cluster map to open the source article. Click a cluster label to filter the outliers sidebar. Use the **Ask** tab to query the corpus with natural language
-5. Click **New analysis** in the top bar to start over with a different topic. You'll be asked to confirm — this clears the current corpus entirely before the new pipeline runs
-6. Click **Export** in the top bar to download an Obsidian-compatible zip — a Canvas file with all articles plotted at their UMAP positions, and a vault folder with one Markdown note per article
+1. On the home page, type a topic (e.g. `mechanistic interpretability in large language models`) and choose **Map this topic**. Tick **Let me check the sources before they're read** to see the searches and pages first: edit the searches, untick pages, or add your own before anything is read.
+2. A map takes **3–8 minutes**. Academic publishers often block automated reading; for those pages (and for PDF links) Discourse Lens looks up an open-access copy or the abstract through OpenAlex and Europe PMC, and the article reader says where the text came from. You'll see each step as it happens: the search queries, pages crawled, articles kept or skipped (and why), and clusters as they're named. You can leave the page, or **Cancel this map**.
+3. When it's done you get six views of the same map:
+   - **Map**: articles that make similar arguments sit close together, grouped into topics (camps) with contour lines and a name for each. Switch to **Table** for a plain list.
+   - **Timeline**: articles per month or year, by cluster.
+   - **Outliers**: articles furthest from the middle of their cluster.
+   - **Key terms**: each topic's one-line summary and the words it uses far more than the others.
+   - **Claims**: the points two or more articles take a position on, and whether each article agrees, qualifies or disputes them, with the most debated first. Click a point to see each article's claim and open the article. This view runs only when you choose **Find claims**, because it costs Claude requests: about half a cent per article (roughly $0.30 for a 50-article map; about a minute).
+   - **Ask**: questions answered only from this map's articles. Answers stream in as they're written, with numbered citations like **[3]**; click one to read that article. Follow-up questions build on earlier answers, and a citation number means the same article for the whole conversation. Conversations are saved with the map.
+4. Pick a topic in the **Clusters** panel to focus every view on it (and read its summary). Articles that don't fit any topic well are grouped as **Unclustered** rather than forced into one. Click any article to read it in a side panel. The address bar keeps the view, cluster and open article, so links and reloads land in the same place.
+5. **Export to Obsidian** downloads a vault: a Canvas of the map plus one note per article.
+6. Every map is kept. **All maps** returns to the list, where you can open or delete them.
 
 ---
 
@@ -111,12 +136,22 @@ Avoid topics that are too broad (`AI`, `climate change`) or too narrow (fewer th
 
 ## Troubleshooting
 
+**Don't want the open-access lookups?** Set `RECOVER_ACADEMIC=false` in `.env`; academic pages a publisher blocks are then skipped.
+
 **Pipeline stalls at "Crawling"** — Check your `BRAVE_API_KEY` in `.env`.
 
 **Cluster labels show as "cluster 0", "cluster 1"...** — The Anthropic API call for labelling failed. Check your `ANTHROPIC_API_KEY`.
 
-**Embeddings are slow** — You're running Ollama on CPU. Switch to OpenAI embeddings by setting `EMBEDDING_MODE=openai` and `OPENAI_API_KEY=your_key` in `.env`.
+**Embeddings are slow** — Local embedding runs on the CPU: about 3 seconds per 10 chunks, so a few minutes for a typical map. For speed, set `EMBEDDING_MODE=openai` and `OPENAI_API_KEY=your_key` in `.env` (new maps only; see Setup).
+
+**Answers are slow to start or too brief** — `CHAT_EFFORT` in `.env` sets how much the model reasons before answering (`low`, `medium` (default), `high`). In testing, `medium` started answering in about a second, same as `low`.
+
+**Chat says "This analysis has no indexed corpus yet"** — The analysis hasn't completed, or no articles passed the quality filter. Try re-running with a different topic.
 
-**Chat returns "No corpus indexed yet"** — The analysis hasn't completed, or no articles passed the quality filter. Try re-running with a different topic.
+**Upgrading from an older version** — On first start the backend migrates the database. Maps made before topic modelling keep their old clusters, without key terms or summaries, until you rebuild them from the stored embeddings (no re-crawl; one Claude request per map):
+
+```bash
+uv run python -m scripts.recluster --all
+```
 
-**Second analysis returns stale data** — This should no longer happen. Each new job deletes all rows from `articles`, `chunks`, `projections` and drops the LanceDB vectors table before running. If you see stale data, stop the backend, delete `data/discourse.db` and `data/vectors/`, and restart.
+The very first migration (from the single-corpus version) copies `data/discourse.db` and `data/vectors/` to `*.v1.bak`; delete those once you're happy.