Awareness

A private index of the public web — built on your laptop, queried like a table.

Point it at a date range and it backfills historical public text. Leave it running and it tails the web live. Everything lands in one local lake you can search from a dashboard, a terminal, or SQL. One Python process. No Spark, no Kafka, no cloud account, nothing leaves the machine.

Awareness ingests text and text-oriented metadata only. No images, no binary media, no login-gated content, no paywall circumvention. robots.txt is obeyed and per-domain politeness is enforced on every live fetch. What you point it at is your responsibility; what it stores is plain, auditable JSON on your disk.

---

The two things it does

BODY — backfill. Ask for a historical window and Awareness plans it into partitions, pulls the matching public text (Common Crawl, HuggingFace FineWeb, GDELT, seed feeds), de-duplicates it, and writes it to the lake. Bounded, resumable, cancellable.

TAIL — live capture. Give it a list of RSS/Atom/sitemap seeds and it watches them, fetching newly published articles as they appear until you stop it. Politeness and robots apply; crashes resume where they left off.

Both feed the same durable schema (DocCapture) and the same query surface. A backfilled 2024 archive and a document captured thirty seconds ago sit in one table.

---

Quickstart

git clone https://github.com/nazmiefearmutcu/awareness && cd awareness
uv venv --python 3.13 --seed
uv pip install -e '.[dev]'

awareness init                                   # create the local lake + state DB
awareness-api                                    # dashboard at http://127.0.0.1:8085

Open the dashboard, hit Start on the live tail, and watch real articles land. Or drive it entirely from the shell:

awareness tail start                             # live capture from configs/tail_seeds.yaml
awareness search "central bank" --limit 20       # BM25 over everything captured
awareness browse                                 # read captured text in the terminal

---

The workbench

A single-process FastAPI server ships a hand-written vanilla-JS SPA at / — no build step, no runtime dependencies, no external requests (fonts are self-hosted; it renders fully offline). Six sections, keyboard shortcuts 1–6, a ⌘K command palette, and a light/dark theme that follows your system and remembers your choice.

Dashboard — the corpus at a glance, plus deep process telemetry: fetch latency percentiles, robots-cache hit rate, dedup fold ratio, Iceberg row counts, staging lag, per-source health.

Captures — full-text search (BM25, prefix, or substring) with source / domain / language / date filters, duplicate collapsing, and a reader for the stored text.

Pipeline — submit a backfill and watch every run: progress bars, task/doc/fold counters, and status badges you can read by colour, not just by squinting at a label.

Tail — the live-capture console: queue depth, what's fetching now, what just landed, what's backing off on retry, and recent storage commits.

Settings — every knob in one place, written straight to awareness.yaml: sources, politeness, storage routing, search behaviour, the tail seed list, and a job-search profile — with live runtime status underneath.

(A Work section reuses the same engine to search public job boards against a saved profile.)

---

How it works

flowchart TD
    user(["CLI / API / dashboard"]) -->|"plan a window"| planner[Planner]
    planner -->|partitions| state[("state DB<br/>jobs · tasks · manifests")]
    state --> worker["Worker engine<br/>asyncio · backpressure"]
    worker -->|"runs partition"| adapters

    subgraph adapters["Source adapters"]
      cc["Common Crawl<br/>WET · CDX · WARC"]
      fw["HuggingFace<br/>FineWeb · FineWeb-2"]
      rss["RSS · Atom · sitemap"]
      tail["Tail recrawl<br/>politeness · robots"]
      gd["GDELT"]
    end

    adapters -->|DocCapture| norm["Normalize → Dedup<br/>trafilatura · xxh3<br/>128-bit SimHash · 32-band"]
    norm --> jsonl["JSONL staging (atomic)<br/>data/jsonl/captures/Y/M/D/"]
    jsonl -.->|optional| iceberg[("Apache Iceberg<br/>PyIceberg warehouse")]
    jsonl --> duckdb["DuckDB<br/>BM25 search · range scans · analytics"]
    iceberg --> duckdb

    classDef src fill:#0e9b8d18,stroke:#0e9b8d,color:#0e9b8d
    classDef store fill:#1f6bff18,stroke:#1f6bff,color:#1f6bff
    class cc,fw,rss,tail,gd src
    class state,iceberg store

Loading

The design is deliberately boring where it should be: JSONL on disk is the source of truth and is written atomically, so a kill -9 never corrupts the lake. Iceberg is an optional durable copy. DuckDB reads both — the same SQL engine answers a ranked search, a date-range scan, and an Iceberg analytics query, so there's one query surface instead of three.

Layer	Module	Job
Sources	awareness.sources.*	one adapter per data tier, all emitting DocCapture
Normalize	awareness.normalize.{text,html}	trafilatura extraction + cleanup
Dedup	awareness.dedup.engine	exact (xxh3) + canonical-URL + near-dup (SimHash)
Storage	awareness.storage.{jsonl,iceberg,duckdb_index,state}	staging · durable · query · state
Planner / Workers	awareness.{planner,workers}	window → partitions → tasks → async execution
Tail	awareness.tail.engine	live-capture lifecycle, resume, politeness
API / CLI	awareness.{api,cli}	the human surface

The single durable record is DocCapture — every adapter produces it, Iceberg mirrors it. Timestamps are UTC; provenance lives in source_*; identity in doc_id / capture_id; dedup grouping in parent_doc_or_dup_group.

---

From the terminal

The dashboard is optional. The CLI is the full control surface:

awareness backfill submit --start 2024-06-01 --end 2024-06-14 --max-tasks 5
awareness backfill run  <JOB_ID>            # execute in-process (or `awareness-worker`)
awareness backfill status <JOB_ID>

awareness inspect --start 2024-06-01 --end now --limit 25
awareness counts  --start 2024-06-01 --end now          # by source / domain / language
awareness search  "sanctions" --mode fts --limit 20     # BM25, prefix, or substring
awareness browse                                        # interactive terminal reader
awareness export  --out corpus.jsonl                    # or --raw-text for a folder of .txt
awareness compact                                       # fold JSONL staging into Iceberg
awareness hf-push  <dataset>                             # publish to the HF Hub

awareness tui                                           # a full-screen terminal dashboard
awareness shell                                         # interactive REPL over every command
awareness stats                                         # storage / DB / ingestion telemetry

awareness configure walks you through where the engine writes before you start a tail. awareness commands prints the categorised map of everything.

---

Benchmarks — measured, not asserted

Awareness is benchmarked head-to-head against the de-facto peer in each space, on one machine, over a deterministic synthetic corpus (fixed seed — accuracy reproduces exactly; throughput drifts with hardware). Where a result trailed the standard, the gap was closed with a real code change and re-measured. Nothing here is tuned to flatter a single number.

uv pip install -e '.[bench]'
python -m benchmarks.run_all      # writes docs/benchmarks/results.json
python -m benchmarks.plot         # renders the charts

_{Apple Silicon (arm64), Python 3.11, single core. Peers: datasketch 1.10, BLAKE3 1.0, trafilatura, DuckDB FTS, SQLite FTS5.}

Near-duplicate detection — precision-first, resource-frugal

Awareness folds near-duplicates with a 128-bit frequency-weighted Charikar SimHash under a Hamming threshold, retrieved through Manku/Jain pigeonhole banding (32 bands × 4 bits — so any pair within Hamming ≤ 31 is guaranteed to share a band, covering the default merge threshold of 24 with exact recall). The peer is datasketch MinHashLSH (num_perm=128), compared on the full end-to-end pipeline (retrieval + threshold + grouping), the way text-dedup and datasketch report — not an all-pairs oracle.

End-to-end pipeline	Awareness DedupEngine	datasketch MinHashLSH
Precision	1.00 — never false-merges	0.999
F1 (default · tuned)	0.84 · 0.96	0.998
Throughput	≈5,200 docs/s (≈3.3×)	≈1,600 docs/s
Signature footprint	16 B/doc (64× smaller)	1,024 B/doc

The 64-bit fingerprint the engine started with had fine separability (0.99, on par with MinHash) but its coarse index retrieved almost nothing at a realistic near-dup radius — end-to-end recall was ~2%. Widening to 128 bits and finer banding fixed the retrieval, not the fingerprint.

Honest verdict: MinHashLSH wins recall and needs no per-corpus tuning — the classic SimHash↔MinHash trade. Awareness picks the other corner on purpose: identical precision at 3.3× the throughput and 64× less memory, and because dedup only ever sets a grouping hint and never drops a row, lower recall costs a little less folding — never data. Full numbers and the other three benchmarks (xxh3 hashing, trafilatura extraction quality, the search/ingest speedups shipped while measuring) are in docs/benchmarks/.

---

What it is, and what it isn't

Honesty beats a feature matrix. This is the part most READMEs leave out.

It does	It does not
Run entirely local: SQLite state, JSONL on disk, Iceberg on disk via PyIceberg.	Touch the cloud. Nothing leaves your machine. The ops/compose Postgres + MinIO + Redpanda + ClickHouse stack is opt-in scaffolding — off by default, and the code never writes to it unless you point it there.
Poll for live updates: the dashboard refreshes on an interval and the tail view every few seconds.	Push over SSE/WebSocket. If the tail is idle (nothing new to discover), the numbers simply don't move.
Enforce robots.txt + per-domain politeness on every live fetch, with crawl-delay honoured.	Surface per-fetch robots decisions in the UI yet.
Cap the planner's initial fan-out with --max-tasks.	Cap sub-partitions that discovery adapters enqueue. One GDELT 15-minute slot can fan out into 1000+ downstream fetches — --max-tasks won't stop that.
Store text and text-oriented metadata, converted from HTML and audited on disk.	Store images, binary media, or anything behind a login or paywall.

---

Configuration & layout

configs/awareness.yaml is the config file; the Settings screen writes to it. Common overrides also read from the environment:

Env	Meaning
AW_DATA_DIR	where the lake (Iceberg + JSONL + state) lives
AW_STATE_DB_URL	SQLAlchemy URL — SQLite by default, Postgres works
AW_API_PORT	dashboard/API port (default 8085)
AW_USER_AGENT	the bot identity sent on every fetch
AW_PER_DOMAIN_CONCURRENCY	live-fetch concurrency cap per domain
AW_TAIL_POLL_SECONDS	feed re-poll interval
AW_ENABLE_ICEBERG	toggle the durable Iceberg copy (JSONL is always on)

data/
├── jsonl/captures/YYYY/MM/DD/captures-*.jsonl   ← atomic staging (source of truth)
├── iceberg/                                     ← PyIceberg warehouse + catalog
├── state/awareness.sqlite                       ← jobs · tasks · manifests · dedup index
├── warc/                                        ← cached WET/WARC bytes (TTL-able)
├── dlq/  cache/  checkpoints/  logs/            ← dead-letter · robots cache · resume · logs

For the analytics-grade environment (Postgres + Redpanda + MinIO + ClickHouse), the same binary points at ops/compose/docker-compose.yml via env vars — no code change. See docs/runbook.md.

---

Develop

pytest                    # full suite
pytest -m smoke           # smoke only
pytest -m integration     # integration only
ruff check . && mypy src  # lint + types

Architecture notes live in docs/architecture.md; the field-by-field record layout in docs/data_dictionary.md; contribution norms in CONTRIBUTING.md. Bugs and security reports: SECURITY.md.

MIT-licensed. Use it responsibly — you are accountable for what you point it at.