Skip to content

feat: add distributed expert workers#551

Open
gauravsaini wants to merge 1 commit into
JustVugg:devfrom
gauravsaini:cluster-upstream
Open

feat: add distributed expert workers#551
gauravsaini wants to merge 1 commit into
JustVugg:devfrom
gauravsaini:cluster-upstream

Conversation

@gauravsaini

Copy link
Copy Markdown

Summary

  • add a coordinator-owned expert-worker data plane over persistent TCP
  • keep routing, token generation, and KV cache on the coordinator
  • batch each layer's routed expert union into one RPC to amortize network latency
  • add disk-backed worker execution using the same converted model and native FFN kernels
  • add a dependency-free registration/heartbeat/topology service via coli cluster coordinator
  • add coli cluster worker and coordinator discovery for coli serve

Correctness and scope

The single-machine path remains unchanged unless CLUSTER_WORKERS or --cluster-coordinator is configured. The control-plane tests cover registration, heartbeat, stale topology filtering, and expert discovery. Native C tests and make colibri pass on macOS.

This PR is intentionally the expert-worker slice only. Dense activation sharding is in #550; WebGPU/browser workers will be proposed separately. The original bundled PR #380 remains open as requested and is not this PR's merge unit.

@JustVugg

Copy link
Copy Markdown
Owner

Thanks for splitting this out. Two blockers, and I want to be straight about both. (1) Priority: distributed expert workers are breadth (multi-machine), and the focus right now is depth on single-consumer-box models — so this isn't where attention is going this week. (2) Correctness, and this is the real one: cluster_worker_run reimplements the expert FFN (expert_gate_upsilufe8_rot_rowsmatmul_qt) as a parallel copy of moe()'s inner compute, but test_cluster.py only exercises the Python HTTP registry — nothing proves the worker's output matches local moe(). A parallel expert path with no token-exact worker-vs-local gate can silently drift from the real kernel. Compare your own #550, which does ship that gate. Before this could land, it needs a correctness test proving worker output == local output. It's env-gated OFF so the default path is safe, but I won't merge a re-implemented core kernel on trust. Deferring for now on both counts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants