Skip to content

start.sh: detect the fabric port on each Spark (fixes #44) - #45

Merged
data-angel merged 1 commit into
mainfrom
fix/fabric-port-detect
Oct 12, 2026
Merged

data-angel merged 1 commit into
mainfrom
fix/fabric-port-detect

Conversation

@data-angel

Copy link
Copy Markdown
Contributor

Fixes #44.

start.sh hardcoded the fabric port to rocep1s0f0, so a pair cabled on the ConnectX-7's second port (enp1s0f1np1) failed with no hint. .env.example already claimed the port was detected.

Now, on each Spark:

  • start.sh lists the RoCE ports whose link is up and that have an IPv4 address.
  • It picks the pair of ports on the same subnet, which is the cable between the two Sparks.
  • It prints the choice and how to override it:
    == fabric: rocep1s0f1 (enp1s0f1np1, 192.168.100.1) here, rocep1s0f1 (enp1s0f1np1, 192.168.100.2) on <worker>; set FABRIC_HCA in .env to choose another port
  • If no port pair shares a subnet, it falls back to the first active port and says to set FABRIC_HCA.
  • FABRIC_HCA in .env still overrides both ranks. The engine still adds the sibling device behind the other PCIe domain of whichever port is chosen.

Tested:

  • The selection logic on: a cable on port 1, mixed ports, two cables on one Spark, and no shared subnet.
  • Real detection on our pair, where one Spark has two active ports (port 0 to the other Spark, port 1 to a third host). It picked port 0.

install/ is unchanged, so the image tag stays 4046c81baa07. The manual-install doc now mentions --fabric-hca rocep1s0f1 for the second port.

🤖 Generated with Claude Code

The default was hardcoded to rocep1s0f0, so a pair cabled on the second port
(enp1s0f1np1) failed with no hint. start.sh now lists each Spark's RoCE ports
whose link is up and that have an IPv4 address, picks the pair on the same
subnet (the cable between them; with several cables this skips the others),
falls back to the first active port with a warning, and prints its choice and
how to override it. FABRIC_HCA in .env still overrides both ranks. Fixes #44.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@data-angel
data-angel merged commit 65091f5 into main Oct 12, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Need a notification on using start.sh

1 participant