Repository navigation
Port three host fixes from the official TensorFold: RDMA GID auto-detect, NCCL cuMem off, page cache release - #22
Conversation
…ache after a weight load The GID index of each RDMA port is read from sysfs at start (lowest RoCE v2 IPv4 entry). TF_RDMA_GID_INDEX or NCCL_IB_GID_INDEX with a value of 0 or more still fixes it; unset, empty or -1 means detect. The gather takes one index per device. The engine sets NCCL_IB_GID_INDEX to the detected index when it is unset, and NCCL_CUMEM_ENABLE=0 and NCCL_CUMEM_HOST_ENABLE=0 unless the user set them. Shards.close and RankCache.close call posix_fadvise(DONTNEED) on the files they read; TF_DS_RELEASE_PAGE_CACHE=0 turns it off.
Elevatormusic
added
the
type: fix
|
@codex review |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Codex Review: Didn't find any major issues. Hooray! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2911757036
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
…1 bypass the variables
|
Codex Review: Didn't find any major issues. Bravo. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
@codex review |
|
Codex Review: Didn't find any major issues. Keep them coming! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
@codex review |
|
@codex security review |
|
Codex Review: Didn't find any major issues. Swish! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
1 participant
What this changes
Three host fixes for the DeepSeek-V4.1 TP path. Each one follows an idea from the official TensorFold (ashhart/TensorFold, Zig engine, Apache-2.0) and is written again in Python. No official code is copied. The notices are in
NOTICE,THIRD_PARTY_NOTICES.mdandtools/dsv41/ATTRIBUTION.md.NCCL_CUMEM_ENABLE=0andNCCL_CUMEM_HOST_ENABLE=0before NCCL starts, unless the user set them (official PR GLM-5.3-Flash on two CUDA ranks: linears packed to 4, 6 or 8 bits ashhart/TensorFold#995: on GB10 the driver ran out of memory at 8 to 16 streams, and later 2-rank starts failed until a reboot).rdma.pyread a fixed index (5, orNCCL_IB_GID_INDEX).cuda/roce_gid.pynow reads/sys/class/infiniband/<dev>/ports/1/gidsandgid_attrs/typesand takes the lowest RoCE v2 IPv4 entry of each port.TF_RDMA_GID_INDEXorNCCL_IB_GID_INDEXwith a value of 0 or more still fixes the index. Unset, empty or-1means detect. Before,-1went toibv_query_gidand failed. The gather takes one index per device. The engine also setsNCCL_IB_GID_INDEXto the detected index when it is unset and every port thatNCCL_IB_HCAlets NCCL use has the same one (a set value, also-1, stays).Shards.closeandRankCache.closecallposix_fadvise(DONTNEED)on the files they read, so the page cache of those files goes back to the node (the weight files are about 106 GB a rank when read from the rank cache).TF_DS_RELEASE_PAGE_CACHE=0turns it off. The load log now ends with the node's MemAvailable. A restart reads the files from disk, not from the page cache. The rank cache key includesweights.py, so the first start after this change writes the rank cache again.Refs Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark#21 (host memory only).
Type
type: fix: a bug fix, or a fix after review findingstype: feature: new code (a feature, tool or add-on)type: docs: docs onlytype: results: run results or a close-outtype: research: a study or failure research, with no fixtype: spec: a plan or spec before the workReceipt
deepseek-v41-tp220277f6 (base imagee2b282c50724, PyTorch 2.13.0a0 nv26.07), 2x DGX Spark (GB10), DeepSeek-V4.1-Flash-1017 EXL3 2.9 bpw,--vision,--parallel 4, 5 drafts, context 262144, prefill chunk 2048, rank cache on. The base image and the PR image were each started several times on the same two nodes.token_shaon the base image and on the PR image in every start:c2ab0d41bf03,16295b26740f,32288eae41c4,4202d7acb4b6(4 base starts, 16 PR starts).tools/bench_concurrent.py --alone --serialwas not run. No token path changes.tools/bench_openai.py. The decode path does not change.tools/prefill_cold.pywas not run.rocep1s0f1,roceP2p1s0f1): the log showsRDMA GID index of <dev> port 1: 3 (detected), and the engine setsNCCL_IB_GID_INDEX=3. Both nodes have index 3 now, so a node pair with different tables and a link flap were not tried. On the base image,NCCL_IB_GID_INDEX=-1makes both ranks refuse the gather (ibv_query_gid failed), and decode gathers fall back to NCCL. On the PR image the same setting starts the gather with detected indices. Other checks:TF_RDMA_GID_INDEX=3logs(fixed), a setNCCL_IB_GID_INDEXstays. On these nodesibv_query_gidwith index -1 returns an error, and with the old default 5 it returns an all-zero GID.NCCL_CUMEM_ENABLE=0 NCCL_CUMEM_HOST_ENABLE=0on both ranks. The 8 to 16 stream driver failure of official PR GLM-5.3-Flash on two CUDA ranks: linears packed to 4, 6 or 8 bits ashhart/TensorFold#995 was not tried, because it can need a node reboot.tests/test_dsv41_host_env.py, 37 tests. In the engine image (PyTorch): 37 pass, 0 skip. On a CPU host without PyTorch: 34 pass, 3 skip (the loader tests and the realposix_fadvisecall skip on Windows). CI checks pass.rufffinds nothing in the new files.Checklist
"draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. (No token path changes.token_shais compared with the base image in the Receipt.)AI disclosure
AI: Claude Sonnet 5.5 via Claude Code
Checked: read every line of the diff, wrote the tests, ran them on CPU and in the engine image, and ran the live check on the 2x Spark pair (20 starts, numbers in the Receipt).