Ask questions about your documents and get answers that cite the exact sentence they came from.
Upload a PDF; it is parsed, split along its own structure, embedded and indexed. Ask a question; two retrieval arms find candidate passages, a trained reranker orders them, and a model answers from those passages alone — with citations that resolve to precise character offsets, so the reader highlights the source rather than searching for it.
The whole thing runs offline. No API key, no model download, no network call at run time.
A real question against a real paper, captured from a running stack by
scripts/screenshot.py. Each citation resolves to a page and a section, and clicking one
highlights the exact span in the document on the left. The answer text is the offline default
generator — an extractive composer that needs no key, which is what makes make dev work with
nothing installed; point LLM_PROVIDER at OpenAI, Gemini or Ollama and the same retrieval feeds
a real model. The retrieval, the reranking and the citation offsets are the engineering here,
and they are what docs/benchmarks.md measures.
Ingestion reports what it actually did — parser, chunker, embedding model, page and chunk counts — because "processing…" that tells you nothing is the part of every document tool that makes it feel broken.
git clone https://github.com/BruceMoseti/contextforge.git && cd contextforge
make env && make dev
make api # in one terminal
make worker # in another
make web # in a third → http://localhost:3000Or docker compose up --build.
| Ingest | PDF, Markdown, HTML, plain text and DOCX. Three PDF parsers tried in order, per file. Tables, figures and formulas extracted as structured assets. Scanned pages detected; OCR optional |
| Chunk | Structure-aware by default: sections, headings and page numbers preserved, tables kept whole. Three other chunkers for comparison |
| Retrieve | pgvector cosine search over HNSW, plus PostgreSQL full-text, combined by reciprocal rank fusion, reordered by a reranker trained on 20 features |
| Answer | Streamed token by token over server-sent events, with sources delivered before the first token, and a gate that refuses rather than inventing |
| Cite | Every claim carries a chunk, a page, a section path and the quote's exact character range in the document |
| Read | Reader with outline, citation highlighting, text-to-speech, adjustable reading level, dyslexia-friendly typography and high-contrast themes |
| Study | Section-scoped summaries, and quizzes as multiple choice, true/false, flashcards or short answer, each tied to its source passage |
| Operate | Structured logs with a request id that follows work into the worker, Prometheus metrics, split liveness and readiness, three independent rate-limit budgets |
Held-out test split of a hand-labelled set, on an idle 8-vCPU VM with no GPU. Every figure
below is generated from evals/results/*.json by make report, and make check fails if a
document and the data disagree — so none of these can be typed in by hand or left behind by
a re-run. Full method and caveats: docs/benchmarks.md.
| Retrieval (105 answerable questions) | hit@1 (95% CI) | hit@5 (95% CI) | MRR | nDCG@5 |
|---|---|---|---|---|
| keyword only | 0.400 (0.311–0.496) | 0.676 (0.582–0.758) | 0.529 | 0.502 |
| vector only | 0.114 (0.067–0.189) | 0.276 (0.200–0.368) | 0.182 | 0.177 |
| hybrid, tuned | 0.409 (0.320–0.505) | 0.695 (0.602–0.775) | 0.538 | 0.508 |
| + trained reranker | 0.610 (0.514–0.697) | 0.809 (0.724–0.873) | 0.692 | 0.644 |
| Generation (139 questions) | |
|---|---|
| citation recall / precision | 0.785 / 0.509 |
| answers containing a citation | 1.000 |
| invalid citation markers | 0.000 |
| groundedness | 0.989 |
| refusal on unanswerable questions | 0.471 (95% CI 0.315–0.633) |
| refusal on answerable questions | 0.181 (95% CI 0.119–0.265) |
| end-to-end p50 / p95 | 16.3 ms / 24.5 ms |
Three results worth reading properly, because two of them are not wins:
- The trained reranker is the biggest single improvement in the pipeline — the hit@1 jump from the tuned hybrid arm to the reranked one is the only difference in the table whose confidence intervals do not overlap, and it costs about 5 ms.
- Hybrid retrieval with equal weights is worse than keyword search alone here. The
default embedder has no useful representation for
ef_constructionor3.5 × 10⁻⁴, and a third of the questions are numeric. Tuning the fusion weights recovers keyword-only performance by learning to mostly ignore the vector arm — the sweep chose 0.15 vector against 0.85 keyword. Equal-weight hybrid is in docs/benchmarks.md, below both of its own arms. - The abstention gate is the weakest part of the system. Every feature it sees is a function of retrieval scores, and the retriever ranks an unanswerable question's nearest passages about as highly as an answerable one's, because those passages are about the subject. They just do not contain the answer. Concretely:
It refuses 16 of 34 unanswerable questions (0.471, 95% CI 0.315–0.633) while wrongly refusing 19 of 105 answerable ones (0.181, 95% CI 0.119–0.265). Balanced accuracy is 0.645, against 0.500 for never refusing at all.
The accuracy figures are reproducible; the latency figures are not, so they are not in the
table. Two full rebuilds of the corpus now produce identical scores for all ten arms —
that took fixing a real defect, because ORDER BY score DESC with no tiebreaker returned
tied full-text matches in physical row order, which changes on every re-ingest and moved the
published numbers by up to four points between runs of identical code. Absolute p50s still
move by 2× depending on how warm PostgreSQL's cache is when an arm runs, and the arms run
sequentially, so cross-arm latency comparison in a single run is contaminated. The one
latency claim worth making survives that, because it is a difference measured between
adjacent arms and it reproduces: reranking costs about 5 ms. Nothing here is a
throughput measurement, and none was taken.
Browser
│
┌────────────┴────────────┐
Next.js UI direct upload
│ │
▼ ▼
FastAPI ─────────────► Object storage
/ │ \ (local | S3)
▼ ▼ ▼ ▲
PostgreSQL Redis Celery broker │
+ pgvector cache │ │
▲ ▼ │
└───────────── Celery worker ─────────┘
A request never waits for a worker and a worker never blocks a request. That is what lets a 600-page upload and a two-second question coexist.
Full walkthrough in docs/architecture.md.
The parts where a competent engineer could reasonably have chosen otherwise, each with what was rejected and what the choice costs:
- Vectors in PostgreSQL, not a dedicated vector database
- HNSW by default, with index type as an operational setting
- Two retrieval arms, fused by reciprocal rank
- Late acknowledgement and idempotent ingestion
- A default model stack that needs no network
- Summaries and quizzes inline, not queued
- Citations in document coordinates
- Presigned uploads, including in development
- A hand-labelled evaluation set with a held-out split
Python 3.12, Node 22, PostgreSQL 17 with pgvector, Redis 7. Or Docker, and none of the above.
cp .env.example .env
docker compose up --buildUI on http://localhost:3000, API on http://localhost:8000, OpenAPI at /docs.
Migrations run as their own one-shot service that the API and the worker wait for, so two
replicas cannot race each other through the same DDL.
This path is exercised in CI, and it is the only place it can be: the development machine
has no Docker daemon and could not reach a registry. The smoke job builds every image, runs
docker compose up --wait, and then registers a user, uploads a real paper, waits for
ingestion, searches and asks a question over HTTP. If that job is green, the API, the queue, a
worker, PostgreSQL, Redis and the storage backend are genuinely wired together — which is the
one thing no unit test can establish.
make env # .env from the example
make dev # venv, dependencies, frontend packages, migrations
make api # http://localhost:8000
make worker # ingestion
make web # http://localhost:3000make help lists everything.
Everything the API does asynchronously is reachable synchronously, which is the fastest way to see the pipeline work:
contextforge users create --email you@example.com
contextforge documents ingest --email you@example.com paper.pdf
contextforge search --email you@example.com "what does the scaling factor do?"
contextforge ask --email you@example.com "what does the scaling factor do?"
contextforge db index # what index is installed, and what is configured
contextforge providers # which models this deployment can actually useEvery setting is in .env.example with a comment, and every one maps to a field on
Settings in services/contextforge/core/config.py, which is authoritative. A test asserts
the two cannot drift, and that no credential is ever committed in the example.
The defaults you are most likely to change:
| Default | ||
|---|---|---|
LLM_PROVIDER |
local |
openai, gemini, ollama |
EMBEDDING_PROVIDER |
static-vectors |
openai, gemini, ollama, sentence-transformers, hashing, lsa |
STORAGE_BACKEND |
local |
s3 |
RERANKER |
feature |
none, lexical, llm, cross-encoder |
VECTOR_INDEX |
hnsw |
ivfflat, none — apply with contextforge db index --apply |
make check # everything CI runs
make test # Python: 1486 tests
make web-check # frontend: lint, types, 434 tests, production build
make evals # the harnesses, then regenerate every figure they feed
make report # regenerate the figures without re-running the harnessesThe suite runs with warnings as errors, which is how three resource leaks were found: database engines that were dropped rather than disposed, a second Redis pool nothing ever closed, and a stream in the download path that was never released.
Integration tests run against a real PostgreSQL with pgvector and a real Redis — no mocks
for either, because the interesting bugs are in the SQL and in the index. The schema is
built by running the migrations, not by create_all, so a model that has drifted from its
migration fails the suite rather than being invisible to it. alembic check is a gate.
Unit tests cover the parts where being wrong is silent: metric definitions checked against values computed by hand, citation offsets, chunk-boundary invariants, and the deployment descriptors, which are read as data and asserted against the code they claim to start.
make check also re-renders every published figure from evals/results/*.json and fails on
a diff. That gate exists because the numbers in this README drifted from the data three
separate times before it did, each caught by accident.
apps/web/ Next.js frontend
services/contextforge/
api/ FastAPI: routers, schemas, middleware, dependencies
core/ config, models, db, storage, cache, security, providers
worker/ Celery app, ingestion, parsers, chunkers, embedders
retrieval/ searchers, fusion, rerankers, pipeline
rag/ prompts, generation, citations, abstention
cli.py operator CLI
evals/ corpus, labelled questions, retrieval and generation harnesses
infra/alembic/ migrations
infra/docker/ images and the pinned runtime lock
tests/unit, tests/integration
docs/ architecture, pipeline, data model, API, security, benchmarks, ADRs
Stated so nobody has to discover it by looking:
- No throughput or concurrency measurement. Nothing in this repository is a load test.
- No time-to-first-token figure. The default provider composes locally and emits with a fixed delay, so timing it measures the delay. A real number needs a hosted model.
- No hosted-model benchmark. OpenAI, Gemini and Ollama are implemented and tested but unmeasured, because a benchmark a reader cannot re-run is not evidence.
- No token revocation, no parser sandbox, no malware scanning. Known gaps, each with its consequence and its fix, in docs/security.md.
- Container images are unbuilt in the environment this was written in — registry egress was blocked. The Compose file and Dockerfiles are validated structurally by 23 tests, and CI builds both images and runs an end-to-end smoke test through them.
Three separate systems, built to show three different things. Each stands alone; together they are backend and retrieval, applied machine learning, and distributed real-time state.
- cutout-ml — Background removal served as a job queue, with a benchmark harness behind every published number.
- forge-ide — Collaborative editing on CRDTs, with the project running in the browser instead of on a server.
MIT. See LICENSE. The evaluation corpus is not committed; evals/fetch
downloads the papers from arXiv under their own terms.