I have two 100 gig link between S5148 and S4248-fbl. Media type is QSFP28 100GBASE LR.
Both ports are configure to be fanout to 4x1 10 g mode in dn_nas_fanout_confg.init file. At reboot of s4200, I run a script to bring up all interface. For link 1 opx-ethtool shows all four ports are UP in both side but for link 2 I have seen some link are UP and some are down and some are one side shows up and the peer device that shows down.
root@TR1:~# opx-ethtool e101-054-1
Settings for e101-054-1:
Channel ID: 1
Transceiver Status: Enable
Media Type: QSFP28 100GBASE LR4
Part Number: GMFC5
Serial Number: CN0GMFC55840019
Qualified: Yes
Administrative State: UP
Operational State: DOWN
Supported Speed (in Mbps): [25000, 10000]
Auto Negotiation : off
Configured Speed : auto
Operating Speed : 0
Duplex : full
root@AG2:~# opx-ethtool e101-047-1
Settings for e101-047-1:
Channel ID: 1
Transceiver Status: Enable
Media Type: QSFP28 100GBASE LR4
Part Number: GMFC5
Serial Number: CN0GMFC55830005
Qualified: Yes
Administrative State: UP
Operational State: DOWN
Supported Speed (in Mbps): [10000]
Auto Negotiation : off
Configured Speed : auto
Operating Speed : 0
Duplex : full
root@TR1:~# opx-ethtool e101-054-2
Settings for e101-054-2:
Channel ID: 2
Transceiver Status: Enable
Media Type: QSFP28 100GBASE LR4
Part Number: GMFC5
Serial Number: CN0GMFC55840019
Qualified: Yes
Administrative State: UP
Operational State: UP
Supported Speed (in Mbps): [25000, 10000]
Auto Negotiation : off
Configured Speed : auto
Operating Speed : False
Duplex : full
root@AG2:~# opx-ethtool e101-047-2
Settings for e101-047-2:
Channel ID: 2
Transceiver Status: Enable
Media Type: QSFP28 100GBASE LR4
Part Number: GMFC5
Serial Number: CN0GMFC55830005
Qualified: Yes
Administrative State: UP
Operational State: DOWN
Supported Speed (in Mbps): [10000]
Auto Negotiation : off
Configured Speed : auto
Operating Speed : 0
Duplex : full
root@TR1:~# opx-ethtool e101-054-3
Settings for e101-054-3:
Channel ID: 3
Transceiver Status: Enable
Media Type: QSFP28 100GBASE LR4
Part Number: GMFC5
Serial Number: CN0GMFC55840019
Qualified: Yes
Administrative State: UP
Operational State: UP
Supported Speed (in Mbps): [25000, 10000]
Auto Negotiation : off
Configured Speed : auto
Operating Speed : False
Duplex : full
root@AG2:~# opx-ethtool e101-047-3
Settings for e101-047-3:
Channel ID: 3
Transceiver Status: Enable
Media Type: QSFP28 100GBASE LR4
Part Number: GMFC5
Serial Number: CN0GMFC55830005
Qualified: Yes
Administrative State: UP
Operational State: UP
Supported Speed (in Mbps): [10000]
Auto Negotiation : off
Configured Speed : auto
Operating Speed : False
Duplex : full
root@TR1:~# opx-ethtool e101-054-4
Settings for e101-054-4:
Media Type: QSFP28 100GBASE LR4
Part Number: GMFC5
Serial Number: CN0GMFC55840019
Qualified: Yes
Administrative State: UP
Operational State: UP
Supported Speed (in Mbps): [25000, 10000]
Auto Negotiation : off
Configured Speed : auto
Operating Speed : False
Duplex : full
root@AG2:~# opx-ethtool e101-047-4
Settings for e101-047-4:
Media Type: QSFP28 100GBASE LR4
Part Number: GMFC5
Serial Number: CN0GMFC55830005
Qualified: Yes
Administrative State: UP
Operational State: DOWN
Supported Speed (in Mbps): [10000]
Auto Negotiation : off
Configured Speed : auto
Operating Speed : 0
Duplex : full
Hardware configuration:
S5148:
xpShell):linkMgr)print_all_port_status 0
==================================================================================================================================================================
ChassisPort# CNX-Port# :[PTGNum ]: Link Fault PTG-signalOK MAC-Mode FEC-Mode serdesId:(Loopback, signalOk, MediaStatus) LB
==================================================================================================================================================================
--------------------
--------------------
53/1: CNX- 36 :[PTG: 9 ]: UP OK OK MAC_MODE_4X10GB DISABLE [9 :(EXT, OK , C) Tx(Inv) : Rx(Inv)] ---
53/2: CNX- 37 :[PTG: 9 ]: UP OK OK MAC_MODE_4X10GB DISABLE [10 :(EXT, OK , C) Tx(Inv) : Rx ] ---
53/4: CNX- 38 :[PTG: 9 ]: UP OK OK MAC_MODE_4X10GB DISABLE [11 :(EXT, OK , C) Tx(Inv) : Rx(Inv)] ---
53/3: CNX- 39 :[PTG: 9 ]: UP OK OK MAC_MODE_4X10GB DISABLE [12 :(EXT, OK , C) Tx(Inv) : Rx ] ---
51/1: CNX- 40 :[PTG: 10]: UP OK OK MAC_MODE_4X10GB DISABLE [13 :(EXT, OK , C) Tx(Inv) : Rx ] ---
51/2: CNX- 41 :[PTG: 10]: DOWN FAULT NO_SIGNAL MAC_MODE_4X10GB DISABLE [14 :(EXT, OK , C) Tx(Inv) : Rx ] ---
51/4: CNX- 42 :[PTG: 10]: DOWN FAULT NO_SIGNAL MAC_MODE_4X10GB DISABLE [15 :(EXT, NO_SIGNAL, NC) Tx(Inv) : Rx(Inv)] ---
51/3: CNX- 43 :[PTG: 10]: DOWN FAULT NO_SIGNAL MAC_MODE_4X10GB DISABLE [16 :(EXT, NO_SIGNAL, NC) Tx(Inv) : Rx(Inv)] ---
49/1: CNX- 44 :[PTG: 11]: UP OK OK MAC_MODE_4X10GB DISABLE [17 :(EXT, OK , C) Tx(Inv) : Rx ] ---
49/2: CNX- 45 :[PTG: 11]: UP OK OK MAC_MODE_4X10GB DISABLE [18 :(EXT, OK , C) Tx(Inv) : Rx ] ---
49/4: CNX- 46 :[PTG: 11]: UP OK OK MAC_MODE_4X10GB DISABLE [19 :(EXT, OK , C) Tx(Inv) : Rx(Inv)] ---
49/3: CNX- 47 :[PTG: 11]: UP OK OK MAC_MODE_4X10GB DISABLE [20 :(EXT, OK , C) Tx(Inv) : Rx(Inv)] ---
40/1: CNX- 48 :[PTG: 12]: DOWN FAULT NO_SIGNAL MAC_MODE_4X25GB DISABLE [21 :(EXT, NO_SIGNAL, NC) Tx : Rx(Inv)] ---
47/1: CNX- 49 :[PTG: 12]: DOWN FAULT NO_SIGNAL MAC_MODE_4X25GB DISABLE [22 :(EXT, NO_SIGNAL, NC) Tx : Rx ] ---
34/1: CNX- 50 :[PTG: 12]: DOWN FAULT NO_SIGNAL MAC_MODE_4X25GB DISABLE [23 :(EXT, NO_SIGNAL, NC) Tx(Inv) : Rx ] ---
33/1: CNX- 51 :[PTG: 12]: DOWN FAULT NO_SIGNAL MAC_MODE_4X25GB DISABLE [24 :(EXT, NO_SIGNAL, NC) Tx(Inv) : Rx ] ---
54/2: CNX- 52 :[PTG: 13]: UP OK OK MAC_MODE_4X10GB DISABLE [25 :(EXT, OK , C) Tx : Rx ] ---
54/1: CNX- 53 :[PTG: 13]: DOWN FAULT(R) OK MAC_MODE_4X10GB DISABLE [26 :(EXT, OK , C) Tx : Rx ] ---
54/3: CNX- 54 :[PTG: 13]: UP OK OK MAC_MODE_4X10GB DISABLE [27 :(EXT, OK , C) Tx(Inv) : Rx ] ---
54/4: CNX- 55 :[PTG: 13]: UP OK OK MAC_MODE_4X10GB DISABLE [28 :(EXT, OK , C) Tx(Inv) : Rx ] ---
52/2: CNX- 56 :[PTG: 14]: DOWN FAULT NO_SIGNAL MAC_MODE_4X10GB DISABLE [29 :(EXT, NO_SIGNAL, NC) Tx(Inv) : Rx ] ---
52/1: CNX- 57 :[PTG: 14]: DOWN FAULT NO_SIGNAL MAC_MODE_4X10GB DISABLE Error: Mac serdes signalOK = 0 for portNum 2, Can not tune serdes.
[30 :(EXT, NO_SIGNAL, NC) Tx : Rx ] ---
52/3: CNX- 58 :[PTG: 14]: DOWN FAULT NO_SIGNAL MAC_MODE_4X10GB DISABLE [31 :(EXT, NO_SIGNAL, NC) Tx(Inv) : Rx ] ---
52/4: CNX- 59 :[PTG: 14]: DOWN FAULT NO_SIGNAL MAC_MODE_4X10GB DISABLE [32 :(EXT, NO_SIGNAL, NC) Tx(Inv) : Rx ] ---
50/2: CNX- 60 :[PTG: 15]: UP OK OK MAC_MODE_4X10GB DISABLE [33 :(EXT, OK , C) Tx : Rx ] ---
50/1: CNX- 61 :[PTG: 15]: UP OK OK MAC_MODE_4X10GB DISABLE [34 :(EXT, OK , C) Tx(Inv) : Rx ] ---
50/4: CNX- 62 :[PTG: 15]: UP OK OK MAC_MODE_4X10GB DISABLE [35 :(EXT, OK , C) Tx(Inv) : Rx(Inv)] ---
50/3: CNX- 63 :[PTG: 15]: UP OK OK MAC_MODE_4X10GB DISABLE [36 :(EXT, OK , C) Tx(Inv) : Rx ] ---
41/1: CNX- 64 :[PTG: 16]: DOWN FAULT NO_SIGNAL MAC_MODE_4X25GB DISABLE [37 :(EXT, NO_SIGNAL, NC) Tx : Rx(Inv)] ---
43/1: CNX- 65 :[PTG: 16]: DOWN FAULT NO_SIGNAL MAC_MODE_4X25GB DISABLE [38 :(EXT, NO_SIGNAL, NC) Tx(Inv) : Rx(Inv)] ---
44/1: CNX- 66 :[PTG: 16]: DOWN FAULT NO_SIGNAL MAC_MODE_4X25GB DISABLE [39 :(EXT, NO_SIGNAL, NC) Tx(Inv) : Rx ] ---
48/1: CNX- 67 :[PTG: 16]: DOWN FAULT NO_SIGNAL MAC_MODE_4X25GB DISABLE [40 :(EXT, NO_SIGNAL, NC) Tx : Rx(Inv)] ---
39/1: CNX- 68 :[PTG: 17]: DOWN FAULT NO_SIGNAL MAC_MODE_4X25GB DISABLE [41 :(EXT, NO_SIGNAL, NC) Tx : Rx(Inv)] ---
45/1: CNX- 69 :[PTG: 17]: DOWN FAULT NO_SIGNAL MAC_MODE_4X25GB DISABLE [42 :(EXT, NO_SIGNAL, NC) Tx(Inv) : Rx ] ---
46/1: CNX- 70 :[PTG: 17]: DOWN FAULT NO_SIGNAL MAC_MODE_4X25GB DISABLE [43 :(EXT, NO_SIGNAL, NC) Tx(Inv) : Rx(Inv)] ---
42/1: CNX- 71 :[PTG: 17]: DOWN FAULT NO_SIGNAL MAC_MODE_4X25GB DISABLE [44 :(EXT, NO_SIGNAL, NC) Tx(Inv) : Rx ] ---
(xpShell):linkMgr)
S4248-fbl:
Duplex : full
root@AG2:~# hshell -c "ps"
Entry not found
0:_bcm_dpp_gport_to_hw_resources: Entry not found
0:bcm_petra_port_discard_set: Entry not found
ena/ speed/ link auto STP lrn inter max loop
port link duplex scan neg? state pause discrd ops face frame back
-----------------
-----------------
xe65( 65) down 10G FD SW No Forward None FA XFI 9028
xe66( 66) down 10G FD SW No Forward None FA XFI 9028
xe67( 67) down 10G FD SW No Forward None FA XFI 9028
xe68( 68) down 10G FD SW No Forward None FA XFI 9028
xe69( 69) up 10G FD SW No Forward None FA XFI 9028
xe70( 70) up 10G FD SW No Forward None FA XFI 9028
xe71( 71) up 10G FD SW No Forward None FA XFI 9028
xe72( 72) up 10G FD SW No Forward None FA XFI 9028
Even though all links are down on hardware in s4248 device, yet opx-ethtool shows one link is up. As those links are part of the bond and karnel try to bring up those interface , I have seen a different result when I ran the ps for second time.
xe65( 65) down 10G FD SW No Forward None FA XFI 9028
xe66( 66) down 10G FD SW No Forward None FA XFI 9028
xe67( 67) up 10G FD SW No Forward None FA XFI 9028
xe68( 68) down 10G FD SW No Forward None FA XFI 9028
xe69( 69) up 10G FD SW No Forward None FA XFI 9028
xe70( 70) up 10G FD SW No Forward None FA XFI 9028
xe71( 71) up 10G FD SW No Forward None FA XFI 9028
xe72( 72) up 10G FD SW No Forward None FA XFI 9028
xe65( 65) up 10G FD SW No Forward None FA XFI 9028
xe66( 66) down 10G FD SW No Forward None FA XFI 9028
xe67( 67) up 10G FD SW No Forward None FA XFI 9028
xe68( 68) down 10G FD SW No Forward None FA XFI 9028
xe69( 69) up 10G FD SW No Forward None FA XFI 9028
xe70( 70) up 10G FD SW No Forward None FA XFI 9028
xe71( 71) up 10G FD SW No Forward None FA XFI 9028
xe72( 72) up 10G FD SW No Forward None FA XFI 9028
54/2: CNX- 52 :[PTG: 13]: UP OK OK MAC_MODE_4X10GB DISABLE [25 :(EXT, OK , C) Tx : Rx ] ---
54/1: CNX- 53 :[PTG: 13]: UP OK OK MAC_MODE_4X10GB DISABLE [26 :(EXT, OK , C) Tx : Rx ] ---
54/3: CNX- 54 :[PTG: 13]: UP OK OK MAC_MODE_4X10GB DISABLE [27 :(EXT, OK , C) Tx(Inv) : Rx ] ---
54/4: CNX- 55 :[PTG: 13]: UP OK OK MAC_MODE_4X10GB DISABLE [28 :(EXT, OK , C) Tx(Inv) : Rx ] ---
I have two 100 gig link between S5148 and S4248-fbl. Media type is QSFP28 100GBASE LR.
s5148 s4248-fbl
link 1: e101-053-0 -------------------------e101-048-0
link 2: e101-054-0--------------------------e101-047-0
Both ports are configure to be fanout to 4x1 10 g mode in dn_nas_fanout_confg.init file. At reboot of s4200, I run a script to bring up all interface. For link 1 opx-ethtool shows all four ports are UP in both side but for link 2 I have seen some link are UP and some are down and some are one side shows up and the peer device that shows down.
Even though all links are down on hardware in s4248 device, yet opx-ethtool shows one link is up. As those links are part of the bond and karnel try to bring up those interface , I have seen a different result when I ran the ps for second time.
After some times all links are up in hardware on S5148 side