Semantic Cache Server

A tiered caching layer that sits in front of an LLM and avoids redundant calls by detecting semantically similar prior questions, not just identical ones.

Screenshots

Five requests against the same underlying fact, fired back-to-back through the dashboard's Try it panel — watch the hit rate climb from 0% to 80% as the cache learns to recognize paraphrases it's never seen verbatim before, and the cost-saved counter grows without a single extra LLM call.

1. Cache miss — first time this question is asked, nothing cached yet, so it calls the LLM (~10s):

2. L1 exact match — the exact same question again — instant Redis hit (3.5ms, $0.000032 saved):

3. L2 semantic match — a paraphrase, never asked before — vector search finds it anyway (similarity 0.942):

4. L2 semantic match — a different phrasing again (similarity 0.908) — hit rate now 75%:

5. L2 semantic match, and the latency payoff — even a typo'd fragment ("what capital of India ?") still matches (similarity 0.972); the latency-by-cache-layer chart now shows all three tiers side by side — exact/semantic hits in milliseconds vs. the miss bar reaching into seconds, at an 85.7% hit rate:

• L1 — Redis exact-match: hash of the normalized query, sub-5ms lookups for literal repeats.
• L2 — Vector similarity search: embeds the query and searches for a near-duplicate above a similarity threshold, so paraphrases hit the cache too.
• On a full miss, the LLM is called, and the result is written back to both layers.

The vector store, embedding provider, and LLM provider are all swappable via a single environment variable, using a common adapter interface per category:

Category	Adapters
Vector store	Chroma (local default), Qdrant, Pinecone, pgvector
Embeddings	local (sentence-transformers), OpenAI
LLM	Mock (default, no API key needed), Anthropic Claude, OpenAI

The project runs out of the box with zero external API keys (mock LLM + local embeddings + Chroma). Real providers are opt-in via .env.

Currently implemented: the tiered L1/L2 cache loop, Redis-backed stats + a built-in dashboard at /dashboard, and all four vector store adapters (Chroma, Qdrant, Pinecone, pgvector) — switching between the three self-hostable ones is a single VECTOR_STORE_PROVIDER env var change, verified to produce identical cache-hit behavior across all three. Real LLM/embedding providers (OpenAI, Anthropic) are also wired in, gated behind your own API key.

Production-hardening pieces are in too:

• Request coalescing — concurrent identical cache-miss requests collapse into a single upstream LLM call (Redis lock; followers poll the leader's result instead of duplicating the call).
• Retry + circuit breaker — tenacity retries transient failures within a call; a small native-asyncio circuit breaker (app/core/circuit_breaker.py) opens after sustained failures and fails fast with a 503 instead of continuing to hammer a downed provider.
• L2 staleness pruning — Redis cache:valid:{id} keys are the source of truth for whether a semantic-match entry is still fresh; a background TtlPruner sweeps stale entries out of the vector store.
• Rate limiting — Redis fixed-window counter per client IP.
• Prometheus metrics at /metrics (HTTP-level via prometheus-fastapi-instrumentator + custom business counters for cache hit types, LLM calls, and estimated cost saved).

Tech stack

Category	Packages	Used for
Web framework	fastapi 0.140, uvicorn 0.51	Async API server
Config	pydantic-settings 2.14	Typed, validated env-var configuration
Cache / infra	redis (redis-py) 8.0	L1 exact-match cache, request coalescing lock, rate limiting, live stats
Logging	structlog 26.1	Structured JSON logs with request-id correlation
Vector stores	chromadb 1.5, qdrant-client 1.18, pinecone 9.1, psycopg 3.3 + pgvector 0.5	Four interchangeable L2 semantic-search backends behind one interface
Embeddings	sentence-transformers 5.6, openai 2.48	Local (zero-key) and hosted embedding generation
LLM providers	anthropic 0.120, openai 2.48	Claude, GPT, and OpenRouter-compatible gateways
Resilience	tenacity 9.1	Retry with exponential backoff on transient provider failures
Observability	prometheus-client 0.26, prometheus-fastapi-instrumentator 8.1	/metrics — HTTP-level + custom business metrics
Linting	ruff 0.16	Style/lint enforcement, wired into CI

One deliberate substitution worth calling out: the original plan used pybreaker for the circuit breaker, but its async support turns out to require Tornado's coroutine style under the hood — not usable in a plain-asyncio app without pulling in a whole second async framework. Replaced with a ~40-line native-asyncio circuit breaker (app/core/circuit_breaker.py) instead.

Full direct-dependency list is in pyproject.toml; a fully pinned snapshot of the entire dependency tree (for exact reproducibility) is in requirements.txt. For a package-by-package breakdown of exactly which files use what (and which of the 154 pinned packages are transitive rather than a deliberate choice), see DEPENDENCIES.md.

Table of contents

• Screenshots
• Tech stack
• Getting started
• Trying the other vector stores
• Request lifecycle
• Project structure
• Endpoints

Getting started

python -m venv .venv
source .venv/Scripts/activate  # .venv\Scripts\activate on Windows cmd/PowerShell
pip install -e ".[dev]"
cp .env.example .env

# Redis is required even in the zero-key default config (it's the L1 cache)
docker run -d --name semantic-cache-redis -p 6379:6379 redis:7.4-alpine

uvicorn app.main:app --reload --port 8010

Then:

curl -X POST http://127.0.0.1:8010/api/query \
  -H "Content-Type: application/json" \
  -d '{"query": "What is the capital of France?"}'

Ask the same question again for an instant L1 hit, or a paraphrase (e.g. "What's France's capital city?") for an L2 semantic hit — the response's cache_hit field shows which layer served it. Watch it happen live at http://127.0.0.1:8010/dashboard.

Windows + Docker Desktop note: use 127.0.0.1, not localhost, in REDIS_URL/QDRANT_URL/ PG_DSN — localhost can resolve to ::1 first and time out against the container's IPv4 port-forward.

Trying the other vector stores

cd docker && docker compose up -d   # brings up redis + qdrant + postgres(pgvector)

Then set VECTOR_STORE_PROVIDER=qdrant (or pgvector) in .env and restart the app — no code changes. Pinecone is code-complete but requires your own cloud account/API key (no local emulator exists for it); set VECTOR_STORE_PROVIDER=pinecone and PINECONE_API_KEY to use it.

Windows + pgvector note: run python run.py instead of uvicorn app.main:app when VECTOR_STORE_PROVIDER=pgvector. psycopg's async pool requires a selector-based event loop, but uvicorn forces ProactorEventLoop on Windows; run.py overrides uvicorn's loop factory to fix this. (Not needed for Chroma/Qdrant/Pinecone, and not needed at all on Linux/macOS.)

Request lifecycle

Request → normalize+hash → [L1 Redis exact?] --hit--> return (~ms)
                │miss
                ▼
          embed query → [L2 vector search ≥ threshold & still valid?] --hit--> promote to L1, return
                │miss
                ▼
     [Redis coalescing lock] → leader calls LLM (retry+breaker) → write L1 + L2 → return
                              → followers poll L1 briefly, then join as leader on timeout

Project structure

app/
├── main.py                    # FastAPI app, lifespan wiring (constructs every adapter/service)
├── config.py                  # pydantic-settings — every env var, provider selection
├── deps.py                    # FastAPI Depends() accessors
├── api/                       # routes_query, routes_stats, routes_admin, routes_health
├── adapters/
│   ├── vector_store/          # base.py (ABC) + chroma/qdrant/pinecone/pgvector + factory.py
│   ├── embeddings/            # base.py (ABC) + local/openai + factory.py
│   └── llm/                   # base.py (ABC) + mock/anthropic/openai + factory.py
├── core/                      # models, normalization, logging, resilience, circuit_breaker, pricing
├── services/                  # cache_orchestrator, coalescer, rate_limiter, ttl_pruner, stats_service
├── metrics/                   # prometheus_metrics.py
└── static/                    # dashboard.html/js/css + vendored Chart.js
docker/                        # Dockerfile, docker-compose.yml (redis + qdrant + postgres + app)
run.py                         # Windows+pgvector entrypoint (see note above)

Endpoints

Endpoint	Purpose
GET /	Redirects to /dashboard
POST /api/query	The main endpoint — rate-limited
GET /dashboard	Live built-in stats dashboard
GET /docs	Interactive Swagger UI (auto-generated by FastAPI; /redoc is the alternate read-only view)
GET /api/stats/summary, /top-queries, /latency-comparison	Data backing the dashboard
POST /api/admin/flush	Demo reset: clears L1 cache + stats counters
GET /healthz, /readyz	Liveness / readiness (readyz checks real Redis + vector store connectivity)
GET /metrics	Prometheus exposition