Heterogeneous execution planning for PyTorch on a single machine.
Profile → plan → simulate → compile → run across CPUs, GPUs, memory, and storage.

Install · Quick start · Benchmarks · When to use · Docs · Contributing

---

When a model approaches or exceeds accelerator memory, hand-written .to(device) maps and ad-hoc offload get fragile. TensorTorrent profiles the host, searches placements in a native Rust planner, simulates finalists, then compiles and runs the winner — trading PCIe/host bandwidth for capacity when that trade makes sense.

Note

Alpha. CPU and virtual backends are covered by CI. Validate accelerators on the target host with tensortorrent validate-hardware.

Benchmarks

Primarily aimed at models that approach or exceed accelerator memory. Fit-in-VRAM auto is near eager (export-free GPU). Under memory pressure or beyond-VRAM, TensorTorrent can be competitive with host-offload runtimes.

Numbers below: RTX 3070 Ti Laptop (~7.66 GiB VRAM) · torch 2.13.0+cu130 · freeze fdbe974. Full tables, figures, raw JSON: benchmarks/evidence/ · methodology.

Qwen3-8B BF16 logits forward (seq_len=16, 16.38 GB params)

Fixed-shape forward only — not autoregressive generation.

Approach	Median ms	Peak VRAM	Notes
GPU eager	—	—	infeasible (params > VRAM)
CPU eager	3153	0	ok
TensorTorrent auto	1203	7.39 GB	transfer_evict · cosine 0.9997 · argmax 15/16
Accelerate (device_map=auto)	1625	6.64 GB	tested config only

DeepMLP 1.5× VRAM (12.35 GB params)

Approach	Median ms	Peak VRAM	Notes
GPU eager	—	—	OOM
CPU eager	429	0.08 GB	ok
TensorTorrent auto	434	0.00 GB	chose CPU (direct_export_free)
Accelerate (device_map=auto)	768	5.38 GB	tested config only

Fit-in-VRAM (export-free GPU, near eager)

Workload	Eager ms	TensorTorrent ms	Peak VRAM
MLP 512×8	0.23	0.23	26 MB
Transformer 256	0.26	0.29	24 MB
MLP 2048×8	0.70	0.69	152 MB

2× GPU / ROCm / XPU: SUPPORTED BUT UNMEASURED on this machine. Crossover 1.00×/1.10× and 2/4 GiB budgets: Transfer fail (see evidence).

Install

pip install torch
pip install tensortorrent

Linux · Python 3.10–3.13 · PyTorch 2.4+. Source builds and CUDA/ROCm/XPU notes: Installation.

Quick start

import torch
import torch.nn as nn
import tensortorrent as tt

model = nn.Sequential(
    nn.Linear(256, 1024),
    nn.GELU(),
    nn.Linear(1024, 256),
).eval()

x = torch.randn(32, 256)
compiled = tt.compile(model, example_inputs=(x,))

y = compiled(x)
torch.testing.assert_close(y, model(x), check_device=False)
print(compiled.explain())

Save/reload, objectives, and multi-module graphs: Quickstart.

When to use

Good fit: model does not fit one GPU · unequal devices · transfer cost matters · RAM/VRAM budgets · parameter streaming or activation spill · reproducible plans instead of hand-written device maps.

Not for: multi-node clusters · exhaustive placement search · “use every detected GPU” · replacing PyTorch kernels · treating discovery as production validation.

Full boundary: Product scope.

Docs

Getting started	Install · Quickstart
Architecture	Overview · Planner · Runtime
Ops	Large models · Deployment · FAQ

tensortorrent doctor
tensortorrent validate-hardware --output artifacts/validation_report.json
make check   # from a source checkout

License

Apache License 2.0. See LICENSE.

---

_{TensorTorrent}