Repository navigation
Ship a reproducible TP2 sidecar setup and maintenance engine - #5
Merged
Merged
Conversation
…failing NCCL init
fabric_hcas() adds the device behind the other PCIe domain on `active(dev)`
alone, which is right for GB10 - one 200G cable is two RDMA devices and filling
it needs both. But a rail can be ACTIVE while its netdev carries no address, so
it has no IPv4 RoCE v2 GID for NCCL to use, and start.sh then dies with
Failed to initialize NCCL
RDMA device 'roceP2p1s0f0' port 1: no RoCE v2 GID with an IPv4 address
after both ranks have already loaded the model. start.sh's own fabric_ports()
applies the IPv4 test but scans rocep* only, leaving the sibling untested.
Gate the sibling on having an IPv4 RoCE v2 GID and log when one is skipped, so
an operator who addressed a single interface per Spark gets a single-rail start
- and a visible reason - rather than a crash. Explicitly requested rails are
untouched.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add a copyable
.env.sidecars.examplefor TP28x262katGPU_MEMORY_UTILIZATION=0.70, with the existing .88 default preserved. The setup uses the same allocation on both ranks and explains the shared KV capacity and separate sidecar installation. A sanitized observation records the existing .70 co-residency checks without treating them as an eight-stream capacity guarantee.Pin the maintenance engine that preserves full blocking chat content alongside refusal metadata (Atlas-Inf/atlas#172) and recognizes compressed-tensors packed projection names. Add refusal and streaming regressions to the native image gates; the engine also includes packed-namespace regression tests. Include the reviewed optional auto-sibling GID fix from #3, preserving its author; explicitly requested rails remain exact.
Validation: 30 CPU installer/fabric tests pass, including the real top-level entrypoint loading the sidecar example and invoking both rank launchers. Native GB10 build passed, including 32 refusal and 43 streaming regression tests; 7 packed-namespace regressions passed separately. The anonymously accessible image is
ghcr.io/enntity/atlas-sparkqwen@sha256:ea6a2df03adfc8802a1baf6ab1d4b55c69dd76588f36bfd5d189781500e75830(install treeb151fbe840a1).A drained TP2 activation at 0.70 passed blocking/streaming refusal-content and gateway canaries, embeddings and 1024x1024 image generation. Two Runtime clients overlapped in all 18 samples with minimum 14.15/20.15 GiB available. Exact sanitized receipts and limitations are in
results/2026-10-13-maintenance/; this is not a full RC16 benchmark rerun. Atlas maintenance PR Enntity/atlas#2 is merged; its inherited fork-wide CI limitations are documented there.Closes #3.