A map of how neural networks actually run on the devices in our pockets, on our wrists, in our ears, and in our cars — the silicon, the software, the vendors, and the trade-offs that shape what ships.
41 chapters across 12 parts. ~148 pages.
chapters/— one markdown file per chapter, numbered01-…through41-….outline.md— the working outline for the book, with design principles.build_book.py— assembles all chapters into a single HTML and renders a PDF via headless Chrome.mobile_inference_book.pdf— the built book.
The PDF is the intended reading format. If you'd rather read on GitHub, the chapter files are plain markdown and render cleanly.
Each chapter is designed to stand reasonably alone — dip in at a topic you care about — with explicit cross-references where prior context matters. The "What you need to remember" list at the end of each chapter is the TL;DR if you're scanning.
| Part | Chapters | Topic |
|---|---|---|
| I | 1 | Context — why on-device, why now, what "mobile" means |
| II | 2–5 | Inference fundamentals — transformers, tokens, prefill/decode/KV, quantization |
| III | 6–8 | Mental models — the 8-layer stack, four ecosystems, central asymmetries |
| IV | 9–12 | Silicon — SoC anatomy, CPU, GPU, NPU |
| V | 13–16 | Hardware APIs and shaders — Vulkan, OpenCL, Metal, NPU SDKs |
| VI | 17–21 | Vendor stacks — Qualcomm, MediaTek, Google, Apple, IP vendors |
| VII | 22–26 | Runtimes — llama.cpp, LiteRT, ORT, Core ML + MLX, compiler runtimes |
| VIII | 27–29 | Models and formats — quantization schemes, language/vision, speech/image |
| IX | 30–33 | Engineering — memory, scheduling, LoRA/speculative, profiling |
| X | 34–36 | Platform layer — walled gardens, privacy/regulation, app stores |
| XI | 37–39 | Future — compiler convergence, open drivers, research frontiers |
| XII | 40–41 | Deep dives — llama.cpp internals, inference optimization playbook |
Requirements:
- Python 3 with the
markdownpackage (pip install markdown) - Google Chrome (used in headless mode for rendering)
python3 build_book.pyOutput: mobile_inference_book.html (intermediate) and mobile_inference_book.pdf.
The build script handles cover, table of contents grouped by part, page numbering, and styling. Edit chapters in chapters/, re-run the script, and the PDF updates.
- Mechanism first, metaphor second. Every abstraction is cashed out in what the hardware actually does.
- Opinionated but sourced. Claims cite runtimes, commits, benchmarks, vendor docs. Opinions are marked as such.
- Real code, judiciously. Shader excerpts, ggml interfaces, delegate APIs — but only where the code shows something words cannot.
- Current as of its time. The field moves fast. The book is dated (2026). Stable parts are structural; transient parts are marked transient.
- Friendly to the dip-in reader. Every chapter stands reasonably alone.
Content: CC BY 4.0. Build scripts: MIT.