Atlas Knowledge

Compare dense, sparse, hybrid and reranked retrieval while inspecting every passage behind the answer.

Atlas is a reusable RAG workbench, not a chat mockup. It ingests PDF, Markdown, text, DOCX, HTML, URL and CSV sources into named dense and sparse Qdrant indexes, supports collection filters, fuses semantic and lexical candidates, optionally reranks them with a local cross-encoder, and exposes the retrieval trace beside the cited answer. A deterministic local mode runs without credentials.

Try the working system

The Codespace installs the project, uses deterministic offline retrieval, starts the API, and opens the web interface on port 8000. Six fictional policy documents are already indexed, so ask:

Can enterprise customers cancel mid-contract, and what approval is required?

The Render blueprint uses the same no-key mode. Render's free services can take time to wake and their local filesystem is ephemeral; use Docker or a persistent disk for durable uploads.

See the document library

What is implemented

• a replaceable parser registry for PDF, Markdown, text, DOCX, HTML, URL content, and CSV
• measured parser routing: pypdf for ordinary text PDFs, Docling for scanned, DOCX and structured HTML
• local-folder and URL connectors behind one discover/fetch/describe contract
• connector synchronization with stable source IDs, checksum skipping, immutable versions, a documented archive/delete policy, and idempotent repeat runs
• connector security boundaries: configured roots only, no traversal or symlink escape, http/https only, no embedded credentials, blocked private and cloud metadata targets, revalidated redirects, bounded size and timeouts
• immutable source IDs, checksums, source URIs, and document version history
• replacement/re-indexing that retires stale vectors before the new version is searchable
• durable asynchronous ingestion jobs with progress, cancellation, retry, and dead-letter states
• tenant-scoped metadata, vector search, and duplicate detection
• tenant-and-owner-scoped ingestion idempotency keys
• tenant-visible or restricted document ACLs for principals and groups
• owner-only replacement, deletion, and ingestion-job controls
• correlated JSON request logs and OpenTelemetry spans across RAG and ingestion
• optional pinned Phoenix trace viewer through a Docker Compose overlay
• deterministic direct/indirect prompt-injection checks and source risk flags
• named dense and sparse Qdrant indexes with reciprocal-rank fusion
• selectable dense, sparse, hybrid, and hybrid-reranked query profiles
• optional local BGE cross-encoder and ColBERT reranking through FastEmbed
• configurable chunking, candidate limits, score gates, and collection filters
• persistent Qdrant indexes plus SQLite document metadata
• source-ranked answers with inline citations and inspectable passages
• an execution trace with profile, fusion, candidate count, and stage latency
• a generation trace with provider, context size, stage latency, and provider-reported token use that is never estimated when the provider does not report it
• a versioned golden set and command-line profile comparison
• a no-key deterministic mode for tests and regression checks
• FastEmbed semantic embeddings using BAAI/bge-small-en-v1.5
• optional OpenAI-compatible answer generation
• upload, search, query, and delete flows in the browser
• typed FastAPI endpoints, automated tests, Docker, and GitHub Actions

The seeded documents are fictional. Uploaded files are processed in a temporary directory and are not retained after indexing.

Run locally

Requirements: Python 3.11.

python -m venv .venv

Activate the environment, then:

python -m pip install -e ".[dev]"
uvicorn app.main:app --reload

Open http://localhost:8000. Interactive API documentation is available at http://localhost:8000/docs.

The default answer generator is local and extractive, so it needs no API key. FastEmbed downloads BAAI/bge-small-en-v1.5 on first launch and then uses the local model cache.

The optional embedding comparison adds the much larger BGE-M3 model:

python -m pip install -e ".[embedding-benchmark]"

Lightweight offline retrieval

To skip the embedding-model download, copy .env.example to .env and set:

ATLAS_EMBEDDING_PROVIDER=hash

Hash mode is deterministic and useful for development and tests, but provides lexical rather than semantic similarity. Remove data/runtime before switching embedding providers so vectors are rebuilt with the matching representation.

Retrieval profiles

The request can select a profile without rebuilding the index:

Profile	Pipeline	Best use
dense	dense nearest-neighbor search	semantic baseline
sparse	hashed lexical vectors with collection-level IDF	exact terms, identifiers, and lexical baseline
hybrid	dense + hashed lexical vectors + RRF	mixed semantic and identifier-heavy corpora when evaluation justifies the extra work
hybrid-reranked	hybrid candidate set + configured reranker	quality-focused queries where added latency is acceptable

Sparse is the measured default for the bundled policy corpus. The configured reranker is BAAI/bge-reranker-base. It is loaded only when a reranked query is requested. The committed bake-off keeps this as a high-latency experiment, not a recommendation for interactive queries. Set ATLAS_RERANKER_PROVIDER=lexical for a small deterministic fallback.

Use an OpenAI-compatible model

Set these values in .env:

ATLAS_GENERATION_PROVIDER=openai-compatible
ATLAS_LLM_BASE_URL=https://your-provider.example/v1
ATLAS_LLM_API_KEY=your-key
ATLAS_LLM_MODEL=your-model

Only the question and retrieved passages are sent to the configured endpoint. The application rejects generated answers that do not contain source citations. Secrets and runtime data are excluded from Git.

Docker

docker compose up --build

The Docker image persists SQLite and Qdrant data in the atlas-data volume.

Run the same application with a local OpenTelemetry/Phoenix trace viewer:

docker compose \
  -f docker-compose.yml \
  -f docker-compose.observability.yml \
  up --build

Phoenix opens on http://localhost:6006. See docs/observability.md for the emitted spans, data minimization boundary, collector configuration, and security limitations.

Try these questions

• Can enterprise customers cancel mid-contract, and what approval is required?
• When is an unused annual plan eligible for a refund?
• What process applies after a serious security incident affects a contract?

You can also disable collections in the sidebar to see how retrieval scope changes the result.

API

Method	Endpoint	Purpose
GET	/api/health	Provider modes and index status
GET	/api/health/live	Process liveness
GET	/api/health/ready	SQLite, job store, and vector-index readiness
GET	/api/documents	List indexed documents
GET	/api/documents/{id}/versions	Read immutable version history for a source
POST	/api/documents	Extract, chunk, embed, and index a file
PUT	/api/documents/{id}	Replace the current document with a new immutable version
DELETE	/api/documents/{id}	Remove every version and vector for the source
GET	/api/connectors	List connectors, configured root names, and supported formats
POST	/api/connectors/local-folder/sync	Queue a folder synchronization job
POST	/api/connectors/url/sync	Queue a URL synchronization job
POST	/api/ingestion-jobs	Persist an upload and return a queued job
GET	/api/ingestion-jobs	List recent ingestion jobs
GET	/api/ingestion-jobs/{id}	Read progress, attempts, errors, and result ID
POST	/api/ingestion-jobs/{id}/cancel	Cancel queued or cooperatively stop running work
POST	/api/ingestion-jobs/{id}/retry	Retry a transient failure within its attempt budget
POST	/api/ingestion-jobs/{id}/replay	Explicitly replay a dead-letter job
POST	/api/query	Retrieve passages and generate a cited answer
POST	/api/evaluations/compare	Run the same question through two to four retrieval profiles

Uploads accept an optional source_uri form field. A replacement keeps the stable source_id, increments version, records supersedes_document_id, and marks the prior metadata row as superseded. Retrieval excludes retired vectors before ranking. Every returned passage identifies the exact source URI, document version, and SHA-256 checksum used for the answer.

Connector synchronization

Configure the folders a connector may ever read, by name:

export ATLAS_CONNECTOR_LOCAL_ROOTS='{"handbook": "/srv/atlas/handbook"}'

curl -X POST localhost:8000/api/connectors/local-folder/sync \
  -H 'Content-Type: application/json' -H 'Idempotency-Key: handbook-nightly' \
  -d '{"root": "handbook", "collection": "Operations"}'

The response is a queued job. GET /api/ingestion-jobs/{id} returns its progress and, once finished, a sync_report counting created, updated, unchanged, removed, skipped, and failed items with a per-item breakdown. A repeated run over unchanged content performs no vector work. Changed content creates a new immutable version and retires the previous vectors. An item that disappears upstream follows ATLAS_CONNECTOR_DELETION_POLICY.

URL synchronization takes an explicit list and rejects unsafe targets before a job is created:

curl -X POST localhost:8000/api/connectors/url/sync \
  -H 'Content-Type: application/json' \
  -d '{"urls": ["https://example.com/handbook.md"], "instance_id": "handbook-mirror"}'

The connector reference documents the parser routing table, the SSRF controls, and the deletion policy.

Tenant and ACL demo contract

Every API route accepts X-Atlas-Tenant, X-Atlas-Principal, and optional comma-separated X-Atlas-Groups headers. The no-header browser demo defaults to tenant demo and principal demo-user. Uploads accept visibility (tenant or restricted) plus comma-separated allowed_principals and allowed_groups form fields.

These headers model identity already verified by a trusted API gateway. They are convenient for a runnable portfolio demo; exposing them directly to untrusted clients is not authentication. In production, derive the same AccessContext from a validated OIDC/JWT session or gateway-signed identity. Do not let callers choose their tenant header.

Tenant and ACL predicates are applied to Qdrant before dense or sparse retrieval. SQLite reads also require the tenant and ACL, while replacement, deletion, and job controls require the owning principal. Unauthorized direct IDs return 404 to avoid confirming whether a resource exists.

The asynchronous endpoint accepts the same multipart fields and an optional Idempotency-Key header. Job state is stored in SQLite independently of the request, while the uploaded input is retained only until success or cancellation. Interrupted running jobs return to the queue at startup unless their attempt budget is exhausted.

The bundled worker deliberately needs no Redis or broker, which keeps Docker and Codespaces one-command demos. It is a single-process runner, not a distributed queue claim. For multi-worker or multi-server client deployments, keep the API and job schema while replacing the runner with RQ/Celery or a durable workflow platform. FastAPI itself recommends a larger queue tool for heavy work that should run outside the web process: https://fastapi.tiangolo.com/tutorial/background-tasks/

Example query:

curl -X POST http://localhost:8000/api/query \
  -H "Content-Type: application/json" \
  -d '{"question":"Who approves a commercial exception below 50,000 USD?","collections":["Customer contracts"]}'

Retrieval evaluation

The frozen evaluation corpus lives in evals/golden.jsonl. It contains 56 reviewed cases: 50 runnable retrieval cases and six explicit capability gates for document versions, prompt injection, and tenant isolation. Cases are labeled by category and split into development and held-out sets.

Run the fast, deterministic comparison and retain both aggregate and per-case evidence:

python -m app.evaluation \
  --profiles dense sparse hybrid hybrid-reranked \
  --output artifacts/baseline.json \
  --raw-output artifacts/baseline-cases.jsonl

Run the semantic and cross-encoder providers used by the full local setup:

python -m app.evaluation \
  --embedding-provider fastembed \
  --reranker-provider cross-encoder \
  --profiles dense sparse hybrid hybrid-reranked

Use --split development while tuning and --split held_out for the final comparison. The runner records MRR@5, Recall@5, nDCG@5, abstention accuracy, p50/p95 latency, index time, traced Python peak memory, dataset hashes, and raw per-case outcomes. The committed deterministic baseline, semantic baseline, and decision report make the quality/latency tradeoff inspectable.

The result is a regression gate on a small fictional corpus, not a quality claim for arbitrary client documents. Replace the corpus with representative client questions and expected sources before selecting a production profile.

Parser bake-off

Atlas also includes a reproducible PDF parser comparison using four upstream document/reference pairs. The normal application keeps its lightweight dependency set; the larger parser stack is an experiment-only extra.

pip install -e ".[parsing-benchmark]"
python -m app.parsing_benchmark \
  --output artifacts/parsing.json \
  --artifacts-dir artifacts/parsing-output

The committed parser decision report, raw benchmark result, and extracted Markdown make the tradeoff inspectable. The seven-fixture comparison covers normal and scanned PDFs, DOCX, and HTML. The measured decision is a fast pypdf path for ordinary text PDFs and an asynchronous Docling quality path for scanned, layout-heavy, and structured office/web documents. Unstructured-fast retained text but flattened the tested DOCX and HTML tables.

Chunking bake-off

Fixed windows, heading-aware chunks, parent-child retrieval, and Docling's tokenizer-aware HybridChunker are compared on the same 50 frozen cases. The runner can isolate sparse retrieval or use dense+sparse hybrid retrieval:

python -m app.chunking_benchmark \
  --embedding-provider fastembed \
  --retrieval-profile hybrid \
  --output artifacts/chunking.json \
  --raw-output artifacts/chunking-cases.jsonl

The committed chunking decision report links the sparse and hybrid aggregate results and every per-case outcome. Fixed chunking remains the default because it is the only method with complete held-out source recall in both runs; Docling hybrid was also roughly an order of magnitude slower to index this corpus. Set ATLAS_CHUNKING_PROFILE to heading-aware, parent-child, or docling-hybrid before ingestion to reproduce an alternative. docling-hybrid requires the parsing-benchmark extra. Rebuild an existing index after changing the profile; the setting does not rewrite already indexed documents.

Embedding bake-off

The embedding runner holds the fixed chunker, frozen questions, retrieval settings, generator, and reranker constant while comparing BGE small and BGE-M3 locally. It also defines a credential-gated, 1,024-dimension text-embedding-3-large run:

python -m app.embedding_benchmark \
  --output artifacts/embedding.json \
  --raw-output artifacts/embedding-cases.jsonl

Set ATLAS_EMBEDDING_API_KEY to execute the hosted candidate. Without a key, the report records credential_required and no quality or cost result. The committed embedding decision report, aggregate result, and per-case evidence explain why BGE small remains the local default while BGE-M3 remains selectable for multilingual or longer-document client work.

Answer and citation evaluation

The release evaluator runs the held-out split through the measured sparse profile, validates every citation rank and extractive span, and uses the pinned open HHEM model to score each claim against its cited passage:

python -m pip install -e ".[semantic-eval]"
python -m app.semantic_evaluation \
  --output artifacts/semantic-evaluation.json \
  --raw-output artifacts/semantic-evaluation-cases.jsonl

The evaluator needs no API key. Supported and deliberately unsupported control pairs must separate correctly before the gate can pass. See the committed semantic evaluation report, aggregate artifact, raw case records, and profile winner matrix. Citation faithfulness passes; the report also preserves the held-out abstention failure instead of hiding it inside the aggregate score.

Reranker bake-off

The reranker runner holds the fixed chunker, BGE small embeddings, hybrid candidates, and frozen cases constant while comparing no reranker, MiniLM, BGE, and ColBERT:

python -m app.reranker_benchmark \
  --output artifacts/reranker.json \
  --raw-output artifacts/reranker-cases.jsonl

The decision report, aggregate result, and per-case outcomes retain both gains and regressions. No learned reranker passed the interactive-default gate.

Quality checks

ruff check .
pytest --cov=app --cov-report=term-missing

CI runs both commands on every push and pull request. Tests cover all retrieval profiles, trace output, abstention, collection filtering, document upload and persistence, immutable replacement history, stale-vector exclusion, duplicate detection, deletion, invalid file handling, durable job recovery, idempotent submission, cooperative cancellation, retry, and dead-letter replay. The authorization suite also covers cross-tenant canary retrieval, restricted principal/group access, tenant-scoped hashes and idempotency, and unauthorized direct object requests. Security regressions also cover correlation IDs, response headers, declared body limits, structured-log redaction, direct prompt overrides, indirect source instructions, and remote-resource output markup. Evaluation tests cover typed side-by-side profile comparison, citation validity/completeness, exact passage support, and semantic-judge controls.

Architecture

The application deliberately keeps the retrieval pipeline visible:

file -> extraction -> chunking -> dense + sparse vectors -> Qdrant
                                                     |
question -> dense + lexical retrieval -> RRF -> optional rerank
                                             |
                                  evidence trace -> cited answer
                                             |
                                  SQLite document metadata

See docs/architecture.md for data boundaries, provider modes, and explicit non-goals.

Limitations

• Authentication remains an extension layer. The included tenant/ACL adapter assumes a trusted gateway has already authenticated the request identity.
• Local-folder and URL connectors are implemented. Drive, SharePoint and database sources are extension points behind the same contract; no OAuth application is bundled.
• Image-only PDFs are detected and refused unless the Docling quality extra is installed. The core install never indexes an empty document instead.
• The DOCX and HTML readers in the core install are reported as degraded: they recover every tested anchor but lose merged table rows that Docling keeps.
• Hash fallback mode is lexical, not a semantic model.
• The golden set is intentionally small and must be replaced or expanded for a client corpus.
• Source display makes evidence auditable; it is not a hallucination detector.
• Changing embedding dimensions requires rebuilding the local index.

License

MIT