You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
add a coordinator-owned expert-worker data plane over persistent TCP
keep routing, token generation, and KV cache on the coordinator
batch each layer's routed expert union into one RPC to amortize network latency
add disk-backed worker execution using the same converted model and native FFN kernels
add a dependency-free registration/heartbeat/topology service via coli cluster coordinator
add coli cluster worker and coordinator discovery for coli serve
Correctness and scope
The single-machine path remains unchanged unless CLUSTER_WORKERS or --cluster-coordinator is configured. The control-plane tests cover registration, heartbeat, stale topology filtering, and expert discovery. Native C tests and make colibri pass on macOS.
This PR is intentionally the expert-worker slice only. Dense activation sharding is in #550; WebGPU/browser workers will be proposed separately. The original bundled PR #380 remains open as requested and is not this PR's merge unit.
Thanks for splitting this out. Two blockers, and I want to be straight about both. (1) Priority: distributed expert workers are breadth (multi-machine), and the focus right now is depth on single-consumer-box models — so this isn't where attention is going this week. (2) Correctness, and this is the real one:cluster_worker_run reimplements the expert FFN (expert_gate_up → siluf → e8_rot_rows → matmul_qt) as a parallel copy of moe()'s inner compute, but test_cluster.py only exercises the Python HTTP registry — nothing proves the worker's output matches local moe(). A parallel expert path with no token-exact worker-vs-local gate can silently drift from the real kernel. Compare your own #550, which does ship that gate. Before this could land, it needs a correctness test proving worker output == local output. It's env-gated OFF so the default path is safe, but I won't merge a re-implemented core kernel on trust. Deferring for now on both counts.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
coli cluster coordinatorcoli cluster workerand coordinator discovery forcoli serveCorrectness and scope
The single-machine path remains unchanged unless
CLUSTER_WORKERSor--cluster-coordinatoris configured. The control-plane tests cover registration, heartbeat, stale topology filtering, and expert discovery. Native C tests andmake colibripass on macOS.This PR is intentionally the expert-worker slice only. Dense activation sharding is in #550; WebGPU/browser workers will be proposed separately. The original bundled PR #380 remains open as requested and is not this PR's merge unit.