Repository navigation
Tensor parallel across Metal and CUDA ranks (Mac + NVIDIA): tracker #1236
Description
We are building tensor parallel (TP) with one group that mixes a Metal rank (a Mac) and CUDA ranks (GB10), connected by RDMA: MCDMA on the Mac, RoCE on Linux. This issue tracks it and proposes the order and the accuracy rule.
What TensorFold has today
- The cluster layer already plans Metal and CUDA nodes (
zig/src/cluster), butserve --clusterrefuses to run on real hosts because the MCDMA endpoints are not linked into the build (zig/src/cluster/cli.zig). - A mixed TP group is refused on purpose:
zig/src/cluster/config.zigsays "Metal and CUDA kernels give different bits". zig/src/fabric/mcdma.zigaccepts only Thunderbolt links, so a Mac cannot reach a CUDA host through it yet.- CUDA TP and Metal TP each run within their own platform today.
Measured on our hardware (not a TensorFold collective)
- Linux host to GB10 over RoCE v2 through a switch, perftest: 4 KB half round trip 6.7 µs (8 KB 8.5, 16 KB 11.8). Only at RDMA MTU 1024 on that switch.
- Mac (MCDMA, ConnectX-5 over Thunderbolt) to GB10: 4 KiB write p50 about 10.5 µs, 28 to 50 Gbit/s.
- Estimate, not a run: a 60-layer model on two ranks needs about 120 syncs a token, so about 0.8 to 2.9 ms a token on the wire depending on the all-reduce (one hop for a one-shot exchange at 2 ranks, two for a ring), before GPU to NIC copies and kernel launches.
Accuracy rule (proposed)
A mixed group is not expected to match one box byte for byte: Metal uses fast fp32 functions and CUDA precise ones, and kernels add in their own order. How far it diverges is unmeasured, and we will measure and report it. So we propose:
- Exact transport: every collective equals a host reference on both vendors.
- The group agrees with itself: the same assignment gives the same bytes run after run.
- Accuracy no worse than the worse single box, measured against an FP64 reference.
- Divergence and greedy token agreement against one box are reported.
bf16 activations are kept; this is about matching rounding, not precision.
Steps
- MCDMA over Ethernet RDMA: let
mcdma.zigaccept a ConnectX link, not only Thunderbolt. - Link the MCDMA endpoints into the cluster layer, so
serve --clusterruns on real hosts. - An all-reduce and all-gather that a Metal rank and a CUDA rank both call, exact against the host reference.
- A small dense model split across a Mac and a GB10, with the receipts for rules 1 to 4. Dense first, since MoE expert choice can differ between ranks.
- Then MoE and larger models.
For you, @ashhart
- Is the cluster layer the right home for this, or
fabric/tp's wire? - Is the accuracy rule above acceptable for mixed groups?
Activity
ThinkOffApp commentedon Oct 24, 2026
/atc claim
ThinkOffApp commentedon Oct 24, 2026
/atc release
[-]Three-platform tensor parallel (Metal + CUDA + HIP) over MCDMA/RoCE[/-][+]Question: should one tensor-parallel group mix Metal and CUDA ranks?[/+][-]Question: should one tensor-parallel group mix Metal and CUDA ranks?[/-][+]Tensor parallel across Metal and CUDA ranks (Mac + NVIDIA): tracker[/+]akol1 commentedon Oct 27, 2026
/atc claim
ashhart commentedon Oct 28, 2026
Thanks @ThinkOffApp for the careful write-up and the wire measurements, and sorry this waited.
On the accuracy rule: we can't take it as proposed. For tensor parallel our bar is one box's tokens, bit for bit, not a comparison with FP64. TP=2 on CUDA is held to TP=1's tokens (#1442), and for the two-Mac 27B (#1432) we asked for a split by output rows, so that every sum stays on one rank. In a group with a Metal rank and a CUDA rank, even that split computes some rows with Metal's arithmetic and the rest with CUDA's, so the tokens would equal neither box. That's why
zig/src/cluster/config.zigrefuses a mixed tensor-parallel group, and it stays that way.On the home: MCDMA lives in
zig/src/fabric/mcdma.zig, and the cluster layer already plans Metal and CUDA nodes, so transport work belongs there.fabric/tpis the CUDA ranks' own wire (NCCL, mailbox, RoCE), and we'd keep it CUDA-only.Steps 1 and 2 (MCDMA over a ConnectX link, and linking the endpoints so
serve --clusterruns on real hosts) don't change any arithmetic. If you'd like to carry them on their own, please open an issue for them with your rule 1 as the gate: every collective equal to a host reference on both ends. Otherwise we'll close this tracker for now; it can always be reopened. @akol1, the same goes for any work you had planned here.ThinkOffApp commentedon Oct 29, 2026
Hi @ashhart Isn't the prefill / decode split you plan to use already outside the exactness contract? TF's cluster planner describes the prefill/decode split as "outside the exactness contract (a prefill computed elsewhere is not this node's own)". Could you accept mixed Metal + CUDA tensor parallel the same way, as an opt-in mode labelled outside the contract with every exact path unchanged? If you'd prefer not, I'd then make my own fork and share what I measure. I want to make TP across three platforms.
ashhart commentedon Oct 31, 2026
Thanks @ThinkOffApp, and you're right about the disaggregated mode: the planner labels it outside the exactness
contract, and it's opt-in.
We'd take it on the same terms as the disaggregated mode: off unless the cluster config asks for it by name, labelled
outside the exactness contract at start, and every exact path unchanged. Your rules 1, 2 and 4 become its gate:
every collective equal to a host reference on both vendors, the same bytes run after run, and divergence and greedy
agreement against each box reported. It lands after #1594's transport steps, dense first as you planned.
The transport work in #1594 comes first and is useful to any Mac and GB10 cluster. We'll keep this tracker open for the mixed mode itself.
ThinkOffApp commentedon Oct 31, 2026
/atc propose pair