Design decisions from building and shipping AI systems in production — written down, with the trade-offs left in.
Most of what I build is commercial and client-facing, so the source code is private. The reasoning does not have to be.
These are notes on decisions I have actually had to make and defend — in regulated environments, in front of compliance reviewers, and occasionally in front of the consequences of getting them wrong. Each one states the problem, the choice, and what the choice cost, because a decision written up without its cost is marketing rather than engineering.
They are opinions formed from a particular vantage point: nine AI systems designed and shipped, five running live, mostly in financial services and adjacent regulated domains. Where I think a reasonable architect would disagree, I have tried to say so rather than smooth it over.
| # | Note | The question it answers |
|---|---|---|
| 01 | The reasoning layer holds no credential that can change anything | If an attacker controlled your model's output token for token, what could it actually do? |
| 02 | Deterministic gates around a probabilistic core | How do you make an LLM-driven decision reproducible enough for an auditor? |
| 03 | Hybrid retrieval: what vectors miss | When does a vector index stop being enough, and what earns a graph its place? |
| 04 | Why AI programmes fail | Across 30+ documented failures, what actually goes wrong — and when? |
Where a note has a runnable counterpart, it is open source:
- llm-gateway — multi-provider LLM routing with automatic failover, circuit breaking, a bounded latency budget and cost tracking that refuses to guess. One dependency, MIT.
Roughly in the order I expect to write them:
- Self-learning scoring without a training loop — feedback from business outcomes, not from labels
- Entity resolution as a first-class layer, not a preprocessing step
- Structured memory: what to persist between turns, and what to deliberately forget
- Unit economics of an agent — what a decision costs before you build it
- AI-assisted development at scale — what changes about engineering discipline past a few thousand files
If you think one of these is wrong, I would genuinely like to hear it — open an issue. Several of these positions were arrived at by being wrong first, and I assume that process is not finished.
Prose is licensed CC BY 4.0 — use it, quote it, build on it, with attribution. Any code snippets within the notes are MIT.
By Sivakumar Chandrasekaran — AI Solutions Architect, Abu Dhabi.