Inference Factory Simulator

inference-factory-sim is a trace-driven laboratory for testing a specific hypothesis:

Cheap frontier-model inference is an emergent property of high decode utilization, speculative generation, workload-aware MoE placement, tiered KV reuse, low precision, and prefill/decode specialization—not one magic kernel.

The repository converts that hypothesis into an inspectable model. It does not claim to reproduce DeepSeek's private infrastructure or provide cycle-accurate GPU simulation.

Read the companion long-form article: The model is not the product. The factory is.

What it models

• Continuous batching from request arrival rate and visible stream duration.
• Separate prefill and decode pools.
• Exact-prefix KV reuse across HBM, DRAM, and NVMe LRU tiers.
• Workload-dependent speculative-decoding acceptance.
• MoE expert hotspots, slowest-rank efficiency, and redundant hot experts.
• Low-precision, fused-kernel, and communication-overlap gains.
• API revenue, energy, capex depreciation, gross margin, and GPU payback.

Every optimization is independently switchable through TOML configuration.

Quick start

No third-party runtime dependencies are required.

python -m venv .venv
.venv/Scripts/activate
python -m pip install -e .
inference-factory compare --config configs/factory.toml

Without installation:

$env:PYTHONPATH = "src"
python -m inference_factory compare --config configs/factory.toml

Generate machine-readable output:

inference-factory compare --config configs/factory.toml --json
inference-factory compare --config configs/factory.toml --output results/comparison.json

Run one scenario rather than the built-in baseline comparison:

inference-factory simulate --config configs/factory.toml

How the comparison works

The compare command generates one deterministic request trace and runs it through two configurations:

1. baseline: no tiered cache, speculation, continuous batching, expert replication, low precision, fusion, communication overlap, or P/D split.
2. factory: the switches and factors from the supplied configuration.

Both sides therefore see identical prompts, outputs, prefix popularity, expert activation tendencies, and arrival times.

The principal decode relation is:

effective decode tok/s/GPU =
    base saturated throughput
  × batch efficiency
  × precision gain
  × fused-kernel gain
  × communication-overlap gain
  × realized speculative gain
  × slowest-rank expert efficiency

Speculative gain is discounted because accepted tokens per verification pass do not translate perfectly into wall-clock speedup. Draft overhead and divergence remain.

Reading the output

• cache hit rate is the fraction of all input tokens restored from reusable prefix KV, not request hit rate.
• batch efficiency approximates how fully the serving fleet amortizes model weight movement.
• speculative tokens/pass is realized throughput gain after overhead, not raw draft acceptance length.
• expert efficiency is mean GPU expert load divided by the slowest GPU load.
• required GPU-hours is useful accelerator work for the trace.
• modeled infra cost charges useful work at target_utilization; it assumes a shared fleet can absorb spare capacity.
• gross margin includes accelerator depreciation and electricity only. It excludes networking, storage capex, CPUs, staff, tax, R&D, and financing.
• payback is an inference-unit economic indicator, not an accounting forecast.

Repository map

src/inference_factory/
  cache.py       exclusive HBM/DRAM/NVMe KV hierarchy
  moe.py         expert activation and replica placement
  workload.py    deterministic mixed request traces
  simulator.py   throughput and economics engine
  config.py      strict TOML loader
  cli.py         command-line interface
configs/
  factory.toml   DeepSeek-like hypothesis, using public-style assumptions
  conservative.toml
tests/
docs/
  MODEL.md       equations, boundaries, and extension points

Tests

python -m unittest discover -s tests -v

Important boundaries

This is a comparative systems model. Absolute results depend strongly on the calibration values. In particular:

• Optimization factors overlap in real systems.
• Actual expert activation traces are model- and traffic-specific.
• Prefix reuse requires exact token prefixes.
• SSD is modeled as persistence/restoration, not as storage read on every decoding step.
• A provider's reported gross margin cannot be inferred from hardware throughput alone.

Use the simulator to ask “which combination could explain the economics?” and to run sensitivity analyses—not to assert undisclosed company facts.