rag-cpp is layered so that each concern is a swappable seam. The type system
enforces the boundaries: nominal strong types for identity, structural concepts
for extension points, and Result<T> for totality.
┌───────────┐
│ Engine │ one-call facade: add · build · search
└─────┬─────┘
┌───────────┴───────────┐
▼ ▼
┌──────────┐ ┌─────────────┐
│ Pipeline │ │ Corpus │ ground truth + indexes
└────┬─────┘ └──────┬──────┘
┌──────────┼──────────┐ ┌─────┼───────┬─────────┐
▼ ▼ ▼ ▼ ▼ ▼ ▼
Retrieve Rerank Expand/ BM25 HNSW Embedder store
(hybrid) (crossenc) Stitch (lexical)(ANN) (dense) (.ragdb)
│ │ │ │
└── fusion (RRF/RSF) ──────────────┘ ▼
HttpTransport ── network seam
StrongId<Tag> (phantom-typed DocId/ChunkId/TermId), Score/Similarity
newtypes, Result<T> = expected<T, Error> with a closed Errc sum type, and
the algebraic records Document/Chunk/SearchResult. The concepts
Embedder, Retriever, Ranker, Tokenizer define the extension points
structurally — implement the shape, get accepted by the templates.
Tokenizer (lowercase, stopwords, Porter stemmer), a prose chunker (semantic
line-aligned with heading breadcrumbs for contextual retrieval), and — in
loaders/ — a code-aware chunker that splits on definition boundaries.
Okapi BM25 over an inverted index with smoothed idf. Serializes to a versioned blob. Always available (no dependencies), so retrieval never hard-fails.
The Embedder concept + the injectable HttpTransport seam (the library's only
outbound I/O). Backends: Ollama, OpenAI-compatible (+ Together/TEI presets),
llama.cpp, and a deterministic local Hash embedder. Decorators RetryingEmbedder
(bounded retries) and FallbackEmbedder (primary → secondary) compose over
any embedder. SIMD kernels (AVX2 / NEON / scalar) for dot, cosine, sign-packing,
Hamming.
HNSW ANN (Malkov & Yashunin) with Matryoshka truncation and binary quantization,
plus filtered search — a metadata predicate pushed into the graph walk
(pre-filter) so a selective filter still returns k results. Corpus owns the
chunks + both indexes, does incremental ingest, and persists via store/.
Reciprocal Rank Fusion (weighted) and Relative Score Fusion.
An explicit document graph (nodes = documents) with two edge kinds — link
edges (markdown [..](..), bare URLs, [[wikilinks]] mined from the text and
resolved against document URIs/titles) and similarity edges (dense centroid
cosine when embedded, else lexical Jaccard; k-NN sparsified). Communities are
found by deterministic synchronous label propagation (Raghavan 2007) and
summarized extractively by default (highest-centrality sentences — free, no
model) with an optional abstractive Summarizer seam. Two retrieval entries:
local (hybrid seed → Personalized-PageRank expansion over the graph →
chunks re-scored by their document's restart-biased centrality) and global
(rank community summaries against the query — GraphRAG's map-reduce over
community reports, reduced to its retrieval core).
The retrieval-side machinery of four landmark recipes, generator-agnostic (the
LM call is an injected seam): RAG/REPLUG ensemble weights (temperature-scaled
softmax p(z|x) over retrieval scores) + a replug_combine distribution mixer;
RETRO chunked-neighbour retrieval (per-stride neighbours with their
continuation — the following chunk); In-Context RALM stride schedule with a
rerank hook; and grounded, source-attributed prompt assembly.
An inverted-index retriever with saturated impact weighting (log(1+tf)·idf)
and query-time term EXPANSION from a corpus co-occurrence graph — SPLADE's
two ideas (learned weights + expansion) approximated model-free, or driven by a
real SPLADE model through the encoder seam. BM25 speed, dense-like recall.
Token-level MaxSim scoring: every query token softly matches its best document token and the matches sum. Precomputable document token embeddings make it a fast reranker between bi-encoders and cross-encoders. Ships a deterministic hashed-n-gram token embedder; a real ColBERT checkpoint plugs into the seam.
RAPTOR (Sarthi 2024): recursively cluster → summarize chunks bottom-up into a tree of increasing abstraction, then retrieve over the collapsed tree (all levels pooled) so a query matches a fine leaf or a high-level synopsis. Dense or lexical agglomerative clustering; extractive summaries with an abstractive seam.
HyDE (Gao 2022): embed an LLM-hallucinated hypothetical answer document to close the query-document distribution gap. Multi-query / RAG-Fusion: retrieve for several paraphrases and RRF-fuse. Both degrade to plain dense search with no generator.
A retrieval evaluator grades the retrieved set's confidence and triggers
CRAG's three actions (correct / ambiguous / incorrect) with decompose-recompose
knowledge strips and an external-source fallback seam. Self-RAG reflection:
per-passage relevance filtering and an answer groundedness (IsSupported)
score. Model-free graders by default; learned graders via seams.
OnnxEmbedder (ONNX Runtime + WordPiece) and GgufEmbedder (llama.cpp) run
embedding models INSIDE the process — no server, no network. Gated behind
RAGCPP_WITH_ONNX / RAGCPP_WITH_LLAMA; absent the dep, load() returns
unavailable (the same graceful-degradation contract as the HTTP backends).
Loads the standard BEIR format (corpus/queries jsonl + qrels tsv) and computes nDCG@k, Recall@k, Precision@k, MAP, MRR — the measurement that turns "SOTA" from a claim into a number you can diff.
Maximal Marginal Relevance (Carbonell & Goldstein 1998): greedily trade
relevance against novelty (λ·rel − (1−λ)·max-sim-to-chosen) so the top-k stop
being k paraphrases of the same passage. Cosine or lexical similarity; a pipeline
stage too.
Compress embeddings 4–64×: split each vector into m sub-vectors, k-means each
subspace to 256 centroids, store one byte per sub-vector. Scoring is ADC
(Asymmetric Distance Computation): a per-subspace query·centroid lookup table,
then m table lookups per vector — no decompression. Serializable.
The production funnel as one call: hybrid retrieve (top-N₀) → ColBERT late interaction (top-N₁) → cross-encoder rerank (top-N₂) → top-k, each stage with a candidate BUDGET so the expensive stages never blow up. Stages are optional and degrade gracefully (an unavailable stage is skipped).
Bounded thread-safe LRU caches. EmbeddingCache keys on (embedder identity, text) so a model swap never returns a stale vector; QueryCache memoizes
(query, k) → hits (clear on corpus mutation). Both are the cheapest latency win
in a RAG loop.
Semantic chunking: embed sentences and break where consecutive-sentence similarity drops below a percentile — chunks are topically coherent runs. Proposition chunking (dense-x): atomic self-contained statements. Lexical fallback with no embedder.
Prepend to each chunk a short blurb SITUATING it in its document before indexing, so both BM25 and dense representations carry the disambiguating context. LLM-generated via a seam, or a deterministic extractive default (title + best- overlap sentence).
HNSW gains tombstone soft-delete (remove/is_deleted/compact): deleted
nodes stay in the graph for connectivity but never surface in results; re-adding
an id resurrects it; compact() rebuilds without tombstones. Corpus:: remove_document tombstones a doc's chunks across BM25 + HNSW with stable ids
(so the chunk meta-pointer invariant holds).
A turnkey binary: ragcpp index <dir> <out.ragdb>, query <db> "...",
eval <beir-dir>, info <db> — build and search a corpus with no C++.
Cross-encoder reranking over HTTP (TEI /rerank and Cohere/Jina /v1/rerank
wire formats) and a local ScoreFnReranker for in-process models. Adapts into a
pipeline stage via make_rerank_stage(), blending cross-encoder score with the
fused score.
Composable RetrievalStages: PrfExpand (RM3-lite query expansion) →
HybridRetrieve (BM25 + dense + fusion) → Filter → cross-encoder rerank →
ParentStitch (small-to-big) → TopK. Every stage has the same interface, so
they compose in any order. Stages are runtime-polymorphic; the hot scoring loops
they call stay non-virtual inside Corpus. Three pre-assembled factories ship:
Pipeline::standard() (hybrid → filter → feature-rerank → top-k, the default),
Pipeline::quality() (+MMR diversity), and Pipeline::context() (+ParentStitch
small-to-big). The two extras are opt-in because their good is coverage, not
accuracy, and each carries a measured justification at its declaration.
The stable, versioned, CRC-checked .ragdb container (see FORMAT.md).
A flat opaque-handle C API (rag/c/rag.h) so Python/Rust/Go/any language can
drive the engine. Errors cross as status codes, never exceptions.
rag-cpp as a conformant RCP/1 server. A
header-only framework layer (rag/rcp/) over the RCP C++ SDK, layered like the
rest of the library:
error.hpp— the one total map betweenrag::Error(Errc) and the wire's JSON-RPC codes (rcp::errc), exhaustively switched.convert.hpp— pure wire↔engine translation:retrieveparam parsing with the funnel invariant (§3.3),SearchResult→Hit(id/score/citation/trust/vector), and the §8 filter tree → aMetaFilterpredicate (validated against advertised fields, precise-32602otherwise).handler.hpp—EngineHandlersatisfies the SDKHandlerconcept over anEngine&. Capabilities are data (Options), advertised only when honestly backable (e.g.embediff an embedder is attached); disabled methods are unreachable via-32003before the hook runs.Hookslets a host override or extend any method with astd::function— no subclassing.server.hpp—ServerBuilder+serve_stdio/serve_httpfree functions.
The engine is borrowed by reference (host owns lifetime + ingestion); the server reads the live corpus. Certified at conformance level L2.
- The network is optional and injected.
HttpTransportis the single outbound seam; everything degrades to BM25 when it's unavailable. - Generic hot path, dynamic cold path. Concepts constrain the templated
scoring code (no vtables per token); type-erased
AnyEmbedder/AnyReranker/RetrievalStagecarry the runtime-assembled pipeline. - Totality. No exceptions for expected failure. Every fallible call returns
Result<T>and composes viaand_then/transform. - Everything serializes. BM25, HNSW, and the whole corpus round-trip to a documented on-disk contract; reopening never rebuilds.