A reference architecture and cost model for LLM request routing in Managed AI Pods. It includes a runnable, offline token-cost estimator and a reproducible benchmark.
This is a reference implementation, not a production library. It is an example project built by a Folder IT AI Pod: it documents how to structure LLM request routing and cost control, and lets anyone reproduce the cost figures for themselves. Maintained as a static reference.
- A documented architecture blueprint for the request lifecycle of an LLM-backed workflow: cache → route → local SLM or premium model → measure.
- A single runnable module — a deterministic token-cost estimator — with tests and a reproducible benchmark.
- A cost model you can run offline to estimate token spend under different routing strategies.
- Not a production framework and not middleware that intercepts live traffic.
- Not a service that calls any LLM. It runs fully offline and needs no API keys.
- Not a latency benchmark. It models cost, not measured wall-clock time.
Being explicit about the boundary is deliberate: the value here is a transparent, reproducible cost model, not a black box.
Most "AI teams" are traditional developers who added a chat assistant to their workflow. A Folder IT AI Pod is built differently from the ground up. One distinction matters most: "AI" describes how the Pod builds — with AI coding agents — not what it builds. A Pod delivers software of any kind, not only AI products. This repository is one example of what a Pod can build, chosen here because LLM cost control is a concrete, measurable problem.
Led by a Forward Deployed Engineer (FDE). Every Pod is anchored by a senior individual contributor who owns the problem end to end — running discovery, translating ambiguous business needs into a technical plan, building the solution, and validating it with your team. This is not a project manager coordinating developers; it is one high-autonomy senior profile accountable for the result.
Powered by AI coding agents. The FDE works with AI coding agents (such as Anthropic's Claude and OpenAI's Codex) as a core part of the execution stack, not as an occasional helper. Working spec-driven — precise specifications that agents generate, validate, and iterate against — gives one senior profile the throughput of a larger team without the coordination overhead.
Backed by the Folder AI Lab. Behind every Pod is an internal team of AI architects and senior specialists. The Lab does not run day-to-day delivery; it steps in when the problem calls for it: complex architectural decisions, model selection, security design, risk assessment, and technical review before a major deployment.
A traditional dev team runs on headcount, standups, and ticket queues. An AI Pod runs on ownership, AI leverage, and expert backup — the same outcome in a fraction of the time and cost.
The blueprint models four stages for every incoming request:
- Semantic cache check. If a near-identical request was answered before, return the cached answer at zero token cost.
- Intent routing. Route routine requests to a local Small Language Model (SLM) and reserve premium cloud models for requests that need heavier reasoning.
- Execution. The request is handled by the selected target.
- Measurement. Token consumption and estimated cost are recorded for every request.
Full diagrams and the decision records live in docs/architecture.md. In this repo, only stage-relevant cost estimation is implemented as code; the rest is documented as architecture.
The estimator is deterministic and offline. Given a set of prompts, it estimates token spend under two strategies and reports the difference:
- Baseline — every request goes to a premium cloud model.
- Routed — exact-match caching plus routing to a local SLM where the rules allow it.
It uses tiktoken for token counting and a dated, sourced pricing table. Because it has no randomness and no network calls, the same input always produces the same output.
Run the benchmark to regenerate the results table:
pip install -r requirements.txt
python -m cost_estimator.benchmark \
--dataset benchmarks/dataset.jsonl \
--cache benchmarks/cache_keys.txt \
--format markdownCommitted results (regenerated by the command above):
| Strategy | Requests | Tokens in | Tokens out | Estimated cost (USD) |
|---|---|---|---|---|
| Baseline (all premium) | N |
— |
— |
$— |
| Cache + routing | N |
— |
— |
$— |
Estimated savings: NN.N% · Pricing as of YYYY-MM-DD · Assumptions: docs/decision_records/assumptions.md
Every figure above is produced by the benchmark. Nothing is hand-entered. Pricing is a dated snapshot from official provider pages, listed in docs/decision_records/sources.md.
from cost_estimator.estimator import compare_strategies
import json
records = [json.loads(line) for line in open("benchmarks/dataset.jsonl")]
cache_keys = {line.strip() for line in open("benchmarks/cache_keys.txt")}
comparison = compare_strategies(records, cache_keys)
print(comparison["savings_pct"])This is a model, so it rests on stated assumptions. All of them are documented in docs/decision_records/assumptions.md. The two that matter most:
- Local SLM cost. Local inference is modeled at zero token cost because it runs on owned infrastructure. The real cost is compute and amortization, not per-token billing. The estimator accepts an optional per-request infrastructure cost so you can model this yourself rather than assume it is free.
- Output tokens. Output length per request is a declared estimate, not a measured value, since no model is actually called.
ai-pod-reference-architecture/
├── README.md
├── STATUS.md # reference, static, unsupported
├── LICENSE # MIT
├── requirements.txt
├── src/cost_estimator/ # the one runnable module
├── tests/
├── benchmarks/ # dataset, cache keys, results
└── docs/
├── architecture.md # request lifecycle diagrams + ADRs
└── decision_records/ # supporting reference docs
├── assumptions.md
├── sources.md # pricing sources, dated
└── glossary.md
This repository is a static reference. It is versioned and archived rather than actively developed. Issues may be left open as a reference template; there is no support commitment. Treat it as a blueprint to read and adapt, not a dependency to install in production.
Released under the MIT License. Copyright (c) 2026 Folder IT.