A production multi-agent platform running at fareeth.surf. Two LangGraph agents collaborate to answer queries: one retrieves, one generates. Self-hosted inference, 75% token cost reduction measured and documented.
Every time you send a query to a standard LLM setup, you're paying for the full context window — even if 90% of what you stuffed in is irrelevant to the question. Before this platform, my naive approach was loading entire document sets into each prompt. At scale, that's expensive and slow.
The platform solves this with a retrieval-first architecture: only the most relevant chunks reach the generator. Token cost per query dropped 75% once measured with tiktoken across a 50-query test set.
User Query
│
▼
FastAPI (Pydantic validation)
│
▼
LangGraph Orchestrator
│
├──► OpenClaw Agent (Retrieval)
│ ChromaDB vector search (top-20)
│ Cohere re-ranking (top-5)
│ Returns: ranked context chunks
│
└──► Hermes Agent (Generation)
Receives: query + compressed context
Inference: Ollama (self-hosted) or Claude API
Returns: grounded response with citations
│
▼
FastAPI response → Nginx → Cloudflare → User
Infrastructure: EC2 + Terraform provisioned, GitHub Actions CI/CD, Cloudflare DNS + SSL.
OpenClaw owns everything before generation. It takes the raw user query, embeds it, queries ChromaDB for the top 20 candidates by cosine similarity, then passes those 20 to Cohere's re-rank API. Cohere scores them by contextual relevance and returns the top 5. Those 5 chunks are the only context Hermes sees.
The two-stage approach matters because cosine similarity on embeddings is fast but coarse. You get recall in stage one, precision in stage two.
Hermes receives the compressed context from OpenClaw plus the original query. It generates a response constrained to that context — if the answer isn't in the retrieved chunks, it says so rather than hallucinating. Responses include source citations derived from ChromaDB metadata.
LLM routing: short, cheap queries go to local Ollama inference (zero cost). Queries requiring deeper reasoning route to Claude via API. The routing decision is a simple heuristic on query complexity score — this is what drives most of the cost reduction.
The two agents are nodes in a LangGraph state machine. State flows: receive_query → openclaw_retrieve → hermes_generate → format_response. LangGraph handles the async execution, state passing between agents, and retry logic if either agent errors.
Ingestion
- Documents chunked with
RecursiveCharacterTextSplitter(700 tokens, 100 overlap) - Token-aware splitting using tiktoken to avoid mid-sentence cuts
- Embeddings: OpenAI
text-embedding-3-smallvia OpenRouter - Stored in ChromaDB with source metadata (file, chunk index, token count)
Retrieval
- Stage 1: ChromaDB cosine similarity, k=20
- Stage 2: Cohere
rerank-english-v3.0, top_n=5 - Final payload to Hermes: ~700 tokens × 5 chunks = ~3,500 tokens max context
Why this chunking size: 700 tokens was the sweet spot after testing. Smaller chunks (300) lost context across sentence boundaries. Larger chunks (1,500) sent too much noise to Cohere and increased re-ranking latency.
def route_query(query: str, complexity_score: float) -> str:
if complexity_score < 0.4:
return "ollama" # local inference, zero API cost
return "claude" # API, higher capabilityComplexity score is calculated from: query length, presence of multi-step reasoning indicators, and whether the query touches multiple distinct topics. This single decision accounts for roughly half the token cost reduction — local inference is free.
| Layer | Tools |
|---|---|
| Orchestration | LangGraph, LangChain |
| API | FastAPI, Pydantic |
| Vector DB | ChromaDB |
| Re-ranking | Cohere rerank-english-v3.0 |
| Embeddings | OpenAI text-embedding-3-small |
| Local inference | Ollama |
| Cloud inference | Claude API (Anthropic) |
| Infra | AWS EC2, Terraform |
| CI/CD | GitHub Actions |
| Reverse proxy | Nginx |
| DNS / SSL | Cloudflare |
75% token cost reduction measured across a 50-query test set:
| Metric | Baseline (naive) | RAG Pipeline |
|---|---|---|
| Avg tokens per query | ~12,400 | ~4,100 |
| Cost per 1,000 queries (Claude API) | ~$3.72 | ~$0.93 |
| Avg response latency | 8.2s | 3.1s |
Measurement method: logged prompt_tokens from each API response. Baseline = same queries answered with the full document corpus in context. RAG = queries answered with 5 re-ranked chunks. Averaged across 50 representative queries. Token counts verified with tiktoken independently.
1. ChromaDB collection corruption on server restart
The DB was initialised inside the FastAPI startup event without checking if the collection already existed. On restart, it tried to create a duplicate collection and threw. Fix: get_or_create_collection() rather than create_collection().
2. Nginx proxy timeout on long agent runs
LangGraph orchestration with Ollama local inference sometimes takes 45–90 seconds on complex queries. Default Nginx proxy timeout is 60s. Fixed by setting proxy_read_timeout 120s in the Nginx config and adding a streaming response mode so the client sees partial output rather than waiting for the full response.
3. Cohere rate limiting at burst load Cohere's free tier rate limits at 100 calls/minute. Under load testing, burst queries would exceed this and the re-ranker would 429. Fixed with an exponential backoff retry wrapper around the Cohere call and a local LRU cache for identical queries within a 5-minute window.
4. LangGraph infinite loop on agent error
If Hermes errored partway through generation, the graph had no terminal state for that branch and would retry indefinitely. Fixed by adding an explicit error node with a maximum retry count of 3 before returning a graceful fallback response.
git clone https://github.com/DOWNEY7/fareeth-platform
cd fareeth-platform
cp .env.example .env
# Add: OPENROUTER_API_KEY, COHERE_API_KEY, ANTHROPIC_API_KEY
# Start with Docker Compose
docker compose up --build
# Ingest documents into ChromaDB
python src/ingest.py --docs-dir ./docs
# API is live at http://localhost:8000
# Docs at http://localhost:8000/docsFor production deployment, see terraform/ — provisions EC2, security groups, and IAM roles. CI/CD via .github/workflows/deploy.yml on push to main.
Async throughout. The current FastAPI setup mixes sync and async endpoints. Re-doing this with full async — async ChromaDB client, async Cohere calls, async LangGraph execution — would reduce latency by removing thread pool overhead.
LangSmith tracing from day one. I traced agent runs with print statements at first. LangSmith gives you a full execution trace with latency per node, token counts, and retry visibility. I added it eventually but should have started there.
Redis for session state. Currently, conversation history is in-memory per FastAPI worker. A restart loses all context and load-balancing across multiple workers is broken. Redis as an external session store fixes both.
Better chunking per document type. Right now everything uses the same chunk size. Code files, prose documentation, and structured data should each use different splitters. The current approach loses structure in tabular content.