Skip to content

start.sh: detect the fabric port on each Spark - #1

Merged
data-angel merged 1 commit into
mainfrom
fix/fabric-port-detect
Oct 12, 2026
Merged

data-angel merged 1 commit into
mainfrom
fix/fabric-port-detect

Conversation

@data-angel

Copy link
Copy Markdown
Contributor

Same fix as Enntity/sparkglm#45 (reported in Enntity/sparkglm#44): this launcher had the same hardcoded port.

start.sh hardcoded the fabric port to rocep1s0f0, so a pair cabled on the ConnectX-7's second port (enp1s0f1np1) failed with no hint. .env.example already claimed the port was detected.

Now, on each Spark:

  • start.sh lists the RoCE ports whose link is up and that have an IPv4 address.
  • It picks the pair of ports on the same subnet, which is the cable between the two Sparks.
  • It prints the choice and how to override it:
    == fabric: rocep1s0f1 (enp1s0f1np1, 192.168.100.1) here, rocep1s0f1 (enp1s0f1np1, 192.168.100.2) on <worker>; set FABRIC_HCA in .env to choose another port
  • If no port pair shares a subnet, it falls back to the first active port and says to set FABRIC_HCA.
  • FABRIC_HCA in .env still overrides both ranks. The engine still adds the sibling device behind the other PCIe domain of whichever port is chosen.

Tested:

  • The selection logic on: a cable on port 1, mixed ports, two cables on one Spark, and no shared subnet.
  • Real detection on our pair, where one Spark has two active ports (port 0 to the other Spark, port 1 to a third host). It picked port 0.

install/ is unchanged, so the image tag stays 656234c336af. The manual-install doc now mentions --fabric-hca rocep1s0f1 for the second port.

🤖 Generated with Claude Code

The default was hardcoded to rocep1s0f0, so a pair cabled on the second port
failed with no hint (reported against SparkGLM, Enntity/sparkglm#44; this
launcher has the same code). start.sh now lists each Spark's RoCE ports whose
link is up and that have an IPv4 address, picks the pair on the same subnet,
falls back to the first active port with a warning, and prints its choice and
how to override it. FABRIC_HCA in .env still overrides both ranks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@data-angel
data-angel merged commit 85325e5 into main Oct 12, 2026
1 check passed
@data-angel
data-angel deleted the fix/fabric-port-detect branch October 17, 2026 04:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant