Signal Intelligence For Technologists
A local-first AI intelligence workstation. Harvest what matters from X, 小红书, Hacker News, arXiv, GitHub and your feeds — then score it, search it in either language, and turn it into briefings. Everything runs on your machine, in one SQLite file.
Quick start · How it works · Signal scoring · Extension · Architecture · Website · Screenshot tour
Following AI properly has become a part-time job. Six timelines, forty feeds, three languages. The same launch reported twenty times over. Engagement bait ranked above the benchmark table that actually answers your question. Bookmarks you will never open again, because nothing can search them.
Every tool solves one slice — a reader that cannot rank, an aggregator that cannot search your highlights, a scraper with no opinion about what is worth reading. Sift is the whole loop, and it runs locally, so your reading history is nobody's dataset.
| Collects | 8 built-in keyless connectors + a browser extension for X and 小红书 |
| Ranks | 6 explainable components, per-source trust, engagement-bait penalty |
| Dedupes | SimHash LSH for reposts, embedding similarity for the same story retold |
| Searches | BM25 + concept expansion + vector KNN, fused by reciprocal rank — in English and Chinese |
| Synthesises | Cited briefings, retrieval-grounded chat over your own corpus |
| Costs | Nothing. No account, no API key required, no telemetry |
Requires Node 24+ and pnpm. Nothing else — no database to provision, no key to obtain.
git clone https://github.com/micaho26/sift && cd sift
pnpm install # zero native modules, so this cannot fail on a compile step
pnpm seed # optional: 37 realistic items to judge it before configuring
pnpm dev # starts the API + app and opens your browserThat is it. pnpm dev prints something like:
◆ Sift · starting
api :4471 web :4470
──────────────────────────────────────────────────────────
▸ Sift is running http://127.0.0.1:4470
37 items · hash/384d · AI: not configured
──────────────────────────────────────────────────────────
Other commands
pnpm start # production mode: build once, serve app + API on one port
pnpm build:extension # build the Chrome/Edge/Brave extension
pnpm test # run every test suite
pnpm typecheck # typecheck every package
pnpm seed # (re)load the demo corpus — idempotent
pnpm db:reset # delete the local database (asks for confirmation)
pnpm screenshots # regenerate the screenshots in assets/Environment variables, all optional:
| Variable | Default | Purpose |
|---|---|---|
SIFT_PORT |
4471 |
API port |
SIFT_WEB_PORT |
4470 |
Vite dev-server port |
SIFT_DB |
./data/sift.db |
Database location |
SIFT_DATA_DIR |
./data |
Data directory |
SIFT_LOG |
info |
debug | info | quiet |
SIFT_NO_SCHEDULER |
— | 1 disables background polling |
SIFT_NO_OPEN |
— | 1 stops pnpm dev opening a browser |
┌─────────────────────────────────────────────┐
browser │ extension (MV3) │
┌──────────┐ │ ┌────────────────┐ ┌──────────────────┐ │
│ x.com │────▶│ │ MAIN world │──▶│ isolated content │ │
│ 小红书 │ │ │ observes the │ │ script extracts │ │
│ any page │ │ │ page's own │ │ items │ │
└──────────┘ │ │ fetch / XHR │ └────────┬─────────┘ │
│ └────────────────┘ │ │
└────────────────────────────────┼────────────┘
│ coalesced batches
┌─────────────────────────────────┐ ▼
│ connectors (server-side) │ ┌──────────────────────┐
│ HN · arXiv · GitHub · HF │─────▶│ ingestion pipeline │
│ Reddit · RSS/Atom │ │ │
└─────────────────────────────────┘ │ canonicalise URL │
│ mute filters │
│ topics + entities │
│ SimHash fingerprint │
│ ── dedup (2 tiers) │
│ embed (384-d) │
│ score (6 components) │
└──────────┬───────────┘
▼
┌──────────────────────────────────────────────────────────────────┐
│ SQLite · ./data/sift.db │
│ items · items_fts (BM25 + CJK bigrams) · item_bands (LSH) │
│ item_topics · item_entities · collections · highlights │
│ + an in-process flat Float32Array vector index (exact KNN) │
└──────────────────────────────────────────────────────────────────┘
▼
┌──────────────────────────────────────────────────────────────────┐
│ Hono API (loopback only) · SSE for live updates │
└──────────────────────────────────────────────────────────────────┘
▼
┌──────────────────────────────────────────────────────────────────┐
│ React 19 app — feed · reader · search · trends · ask · briefing │
└──────────────────────────────────────────────────────────────────┘
Full detail in docs/ARCHITECTURE.md.
Every item gets a 0–100 score answering one question: should this interrupt you? Six components, each squashed so no single axis can be gamed into dominance.
| Component | Weight | What it measures |
|---|---|---|
| Relevance | 26% | Cosine to your interest clusters + explicit keyword hits |
| Velocity | 22% | Attention per hour, normalised against the source's own p95 |
| Depth | 15% | Evidence: numbers, code, citations, artefacts, hedged claims |
| Novelty | 14% | 1 − similarity to everything already in your library |
| Authority | 13% | Author reach × how often you keep their work |
| Recency | 10% | Exponential decay, half-life you choose (default 36h) |
Then multiplied by a per-source trust factor, and reduced by up to 30% for engagement-bait patterns — emoji density, hashtag stuffing, giveaway language, in both English and Chinese.
The score always shows its work. Press W on any item:
Measured on the bundled demo corpus: a thread with benchmark numbers, an ablation and a repo link scores 66. A post with 32,000 likes, nine hashtags and "this will blow your mind" scores 31 — the depth term and the noise penalty both bite.
Every weight is a slider in Settings. Move one and the whole library rescores in the background. Formulas in docs/SCORING.md.
Four retrievers, fused by Reciprocal Rank Fusion rather than score normalisation, because BM25 and cosine live on incomparable scales:
- Keyword — FTS5 BM25 over title / body / author / tags, with column weights.
- Concept — the query is run through the same 24-topic AI taxonomy the items were, so "making models cheaper to run" retrieves the
efficiencytopic with no neural model involved. - Semantic — exact cosine KNN over an in-memory flat matrix.
- Browse — no query, one indexed scan. The common case pays for none of the above.
SQLite's unicode61 tokeniser treats an entire Chinese sentence as one token, so Chinese search silently returns nothing. Sift indexes a parallel column of character bigrams and translates queries into ordered bigram phrases, which is exact substring matching:
| You type | Becomes | Finds |
|---|---|---|
量化 |
cjk : "量化" |
「M4 Max 本地跑 DeepSeek-V4 量化版实测」 |
vLLM 推理 |
cjk : "推理" AND "vllm"* |
mixed script splits cleanly |
怎么降低推理成本 |
affixes stripped → OR-ed 4-char windows | 「大模型推理成本优化实战」 |
Exact-URL matching catches the same link twice. It does not catch the twenty accounts that all posted "OpenAI just shipped X" in different words — which is the actual failure mode of an AI feed.
Tier 1 — SimHash + LSH. A 64-bit fingerprint over weighted word bigrams, split into four 16-bit bands. By the pigeonhole principle any two prints within Hamming distance 3 must collide on at least one band, so an equality index gives complete recall from one query instead of a full scan.
Tier 2 — embedding similarity. Measured on our own corpus, a genuine repost sits at Hamming 22 while unrelated items sit at 27–31 — far too narrow to threshold safely. Cosine separates the same pair 0.60 vs 0.29. So tier 2 uses the vector, gated on a shared named entity and a 72-hour window, which stops topical resemblance alone from folding two distinct stories together.
Nothing is ever deleted. Echoes fold behind their canonical item with a count, and stay fully searchable.
|
Keyboard-first triage
|
Ask your own library Retrieval-grounded chat scoped strictly to what you have collected. Citations arrive before the first token and link back. If the answer is not in your corpus, it says so. |
|
Trends that say something Not "which topics are big" — always the same few — but which are accelerating, against the previous window, with smoothing so a fresh install does not claim 17× growth on three items. |
Cited briefings Themed daily or weekly digests where every claim carries the bracketed number of the item it came from. Works without an LLM: the template path clusters and assembles locally. |
|
Collections, highlights, tags Boards with manual ordering, highlight-with-note, and tags that survive a re-crawl. Re-ingesting an item never clobbers anything you did by hand. |
Real export Markdown with your highlights, JSON with every field, CSV (BOM'd so Excel opens Chinese correctly), OPML for your feeds, and a share card rendered locally as SVG → PNG. |
More screenshots
Some sources need your own logged-in session. The extension covers those.
pnpm build:extension
# chrome://extensions → developer mode → Load unpacked
# → apps/extension/.output/chrome-mv3
How it gets the data. A MAIN-world script observes the page's own fetch and XMLHttpRequest responses and forwards the JSON to the extension. Nothing is re-requested, no credentials are touched, and no scraping traffic is generated — the extension reads what your browser already received, at the rate you browse. The DOM is scraped only as a fallback when a response shape changes.
The JSON walkers are shape-tolerant by construction: rather than hardcode data.home.home_timeline_urt.instructions[].entries[]…, they recursively look for anything that resembles a tweet (legacy.full_text + an id) or a note (a 24-hex id + a title). A platform reshuffling its envelope does not break capture.
Permissions are narrow. Three hosts plus loopback. No <all_urls>, no tabs, no cookie access. Page capture rides on activeTab, so it can only ever run where you invoked it. Content scripts never talk to the Sift server — everything routes through the worker, which verifies a service: "sift" handshake before posting anything.
Per-site auto-collection toggles with ⌥⇧H; ⌥⇧S captures the current page (or just your text selection).
Detail in docs/EXTENSION.md.
Sift is complete without an API key.
| Feature | Without a provider | With one |
|---|---|---|
| Semantic search | Built-in hashing embedder — instant, offline | Real paraphrase matching |
| Summaries | Local extractive summarisation | Written prose |
| Takeaways | Top-scoring sentences | Extracted claims |
| Ask | Returns the matching items | A cited written answer |
| Briefings | Clustered and assembled locally | Themed prose with citations |
| Translation | unavailable | zh ↔ en |
Providers: Anthropic, OpenAI-compatible, Ollama. Keys are stored in the local database and never returned by any endpoint — the API only reports whether one is set.
The default embedder hashes character n-grams into 384 dimensions. It captures lexical and sub-word similarity rather than true paraphrase, which is an honest trade for needing no download and working offline on the very first run. Switch to a transformer, Ollama or OpenAI in Settings and the library re-embeds in the background.
- One file. Everything is in
./data/sift.db. Back it up by copying it. - Loopback only. The server binds
127.0.0.1. CORS admits extension origins and localhost; nothing else. - No telemetry. Not opt-out — absent. There is no analytics code in this repository.
- Keys are write-only. Paste-in, never read back.
data/is gitignored. Your corpus cannot be committed by accident.
pnpm install failing on node-gyp is the most common reason a local tool never gets run. So Sift has no native modules at all: SQLite comes from Node 24's built-in node:sqlite, which ships FTS5, WAL and window functions. The vector index is a hand-written flat scan — one contiguous Float32Array, exact cosine KNN, unrolled by four.
At the scale of a personal corpus this is not a compromise. 100k items × 384 dims is ~38M multiply-adds, around 25 ms — and there is no index to rebuild, no recall cliff, and no stale-index class of bug. An ANN library would be faster and strictly worse here.
Runtime dependencies, in total:
| Package | Runtime deps |
|---|---|
@sift/core |
zod |
sift-server |
hono, @hono/node-server, @hono/zod-validator, zod |
sift-web |
react, react-dom, TanStack Query/Virtual, cmdk, sonner, lucide-react, motion |
sift-extension |
react, react-dom |
sift/
├── apps/
│ ├── server/ Hono API · connectors · pipeline · AI · SQLite
│ │ └── src/
│ │ ├── db/ schema, access layer, flat vector index
│ │ ├── connectors/ HN, arXiv, GitHub, HF, Reddit, RSS + scheduler
│ │ ├── pipeline/ ingest, rescore, interest profile
│ │ ├── repo/ items, library, sources, settings
│ │ ├── ai/ provider, prompts, digest
│ │ ├── routes/ the HTTP surface
│ │ └── share/ SVG share cards
│ ├── web/ React 19 + Vite 8 + Tailwind 4
│ ├── extension/ WXT · MV3 · X + 小红书 + article extractors
│ └── site/ Astro marketing site
├── packages/core/ types · scoring · SimHash · RRF · taxonomy · embedder
├── scripts/ dev, start, screenshots, postinstall
└── docs/ architecture, scoring, extension, API
pnpm test79 tests, no mocks of our own logic:
packages/core(39) — URL canonicalisation across platform aliases, SimHash discrimination, CJK query construction, the full scoring model including degenerate and adversarial inputs, RRF and MMR, the hashing embedder's determinism and ordering, taxonomy precision.apps/server(27) — the schema against a realnode:sqlitefile: every table and index the repo queries, foreign-key cascade, upsert-preserves-user-state, the FTS5 Chinese bigram path end to end (including a natural-language question), the pigeonhole completeness of LSH banding at the duplicate threshold, and the vector index's exact-KNN ordering, tombstoned deletes, row reuse, facet filtering and dimension guards. Plus the default source list itself: every GitHub search URL is asserted free of a bareOR, because GitHub answers 422 to a logical operator between qualifiers and the shipped three-topic target was hitting it on every poll.apps/extension(13) — the JSON walkers against realistic X and Xiaohongshu envelopes, including cyclic and over-deep input, Chinese count parsing (1.2万→ 12000), and endpoint-matcher precision.
The server tests deliberately do not mock SQLite. Every bug that layer has actually produced was a SQL-level one — a stray backtick inside a template literal, a column that did not exist, a tokenizer that swallowed a whole Chinese sentence — and none of those are reachable from a test that stubs the driver.
The scoring tests assert behaviour, not values — e.g. "a substantive thread outranks viral bait", "the fortieth copy of a story ranks below the first", "clock skew cannot push a score out of range".
Test scripts name their files explicitly rather than globbing. node --test 'test/*.test.ts' exits 0 when the glob matches nothing, so a package whose test directory goes missing reports "0 tests, 0 failures" and CI stays green over it — which is exactly how apps/server came to have a test script and no tests. An explicit path fails loudly instead.
Issues and PRs welcome. See CONTRIBUTING.md. The codebase is written to be read: every non-obvious decision has a comment explaining why, and there is no build step between the source and what runs in development.
Good first areas: more connectors (apps/server/src/connectors/), more entries in the AI lexicon (packages/core/src/taxonomy.ts), a Firefox build of the extension, better article extraction.
The ideas here are not new — the implementation is. Reciprocal Rank Fusion is Cormack, Clarke & Buettcher (SIGIR 2009). SimHash is Charikar (2002), with the band scheme from Manku, Jain & Das Sarma (WWW 2007). MMR is Carbonell & Goldstein (SIGIR 1998). The gravity term in the velocity component is Hacker News's.
MIT.
Built to be read, forked and disagreed with.