Skip to content

Fail to bring up all interface between s5148 and s4248-fbl after fanout  #51

@waliulislam

Description

@waliulislam

I have two 100 gig link between S5148 and S4248-fbl. Media type is QSFP28 100GBASE LR.

s5148 s4248-fbl
link 1: e101-053-0 -------------------------e101-048-0
link 2: e101-054-0--------------------------e101-047-0

Both ports are configure to be fanout to 4x1 10 g mode in dn_nas_fanout_confg.init file. At reboot of s4200, I run a script to bring up all interface. For link 1 opx-ethtool shows all four ports are UP in both side but for link 2 I have seen some link are UP and some are down and some are one side shows up and the peer device that shows down.

root@TR1:~# opx-ethtool e101-054-1
Settings for e101-054-1:
    Channel ID:   1
    Transceiver Status: Enable
    Media Type: QSFP28 100GBASE LR4
    Part Number: GMFC5
    Serial Number: CN0GMFC55840019
    Qualified: Yes
    Administrative State: UP
    Operational State: DOWN
    Supported Speed (in Mbps):  [25000, 10000]
    Auto Negotiation : off
    Configured Speed   : auto
    Operating Speed   : 0
    Duplex   : full
root@AG2:~# opx-ethtool e101-047-1
Settings for e101-047-1:
    Channel ID:   1
    Transceiver Status: Enable
    Media Type: QSFP28 100GBASE LR4
    Part Number: GMFC5
    Serial Number: CN0GMFC55830005
    Qualified: Yes
    Administrative State: UP
    Operational State: DOWN
    Supported Speed (in Mbps):  [10000]
    Auto Negotiation : off
    Configured Speed   : auto
    Operating Speed   : 0
    Duplex   : full

root@TR1:~# opx-ethtool e101-054-2
Settings for e101-054-2:
    Channel ID:   2
    Transceiver Status: Enable
    Media Type: QSFP28 100GBASE LR4
    Part Number: GMFC5
    Serial Number: CN0GMFC55840019
    Qualified: Yes
    Administrative State: UP
    Operational State: UP
    Supported Speed (in Mbps):  [25000, 10000]
    Auto Negotiation : off
    Configured Speed   : auto
    Operating Speed   : False
    Duplex   : full
root@AG2:~# opx-ethtool e101-047-2
Settings for e101-047-2:
    Channel ID:   2
    Transceiver Status: Enable
    Media Type: QSFP28 100GBASE LR4
    Part Number: GMFC5
    Serial Number: CN0GMFC55830005
    Qualified: Yes
    Administrative State: UP
    Operational State: DOWN
    Supported Speed (in Mbps):  [10000]
    Auto Negotiation : off
    Configured Speed   : auto
    Operating Speed   : 0
    Duplex   : full

root@TR1:~# opx-ethtool e101-054-3
Settings for e101-054-3:
    Channel ID:   3
    Transceiver Status: Enable
    Media Type: QSFP28 100GBASE LR4
    Part Number: GMFC5
    Serial Number: CN0GMFC55840019
    Qualified: Yes
    Administrative State: UP
    Operational State: UP
    Supported Speed (in Mbps):  [25000, 10000]
    Auto Negotiation : off
    Configured Speed   : auto
    Operating Speed   : False
    Duplex   : full
root@AG2:~# opx-ethtool e101-047-3
Settings for e101-047-3:
    Channel ID:   3
    Transceiver Status: Enable
    Media Type: QSFP28 100GBASE LR4
    Part Number: GMFC5
    Serial Number: CN0GMFC55830005
    Qualified: Yes
    Administrative State: UP
    Operational State: UP
    Supported Speed (in Mbps):  [10000]
    Auto Negotiation : off
    Configured Speed   : auto
    Operating Speed   : False
    Duplex   : full

root@TR1:~# opx-ethtool e101-054-4
Settings for e101-054-4:
    Media Type: QSFP28 100GBASE LR4
    Part Number: GMFC5
    Serial Number: CN0GMFC55840019
    Qualified: Yes
    Administrative State: UP
    Operational State: UP
    Supported Speed (in Mbps):  [25000, 10000]
    Auto Negotiation : off
    Configured Speed   : auto
    Operating Speed   : False
    Duplex   : full
root@AG2:~# opx-ethtool e101-047-4
Settings for e101-047-4:
    Media Type: QSFP28 100GBASE LR4
    Part Number: GMFC5
    Serial Number: CN0GMFC55830005
    Qualified: Yes
    Administrative State: UP
    Operational State: DOWN
    Supported Speed (in Mbps):  [10000]
    Auto Negotiation : off
    Configured Speed   : auto
    Operating Speed   : 0
    Duplex   : full
Hardware configuration:
S5148:
xpShell):linkMgr)print_all_port_status 0


==================================================================================================================================================================
ChassisPort# CNX-Port# :[PTGNum ]:   Link     Fault    PTG-signalOK       MAC-Mode         FEC-Mode     serdesId:(Loopback, signalOk, MediaStatus)  LB
==================================================================================================================================================================
--------------------
--------------------
    53/1:     CNX- 36  :[PTG: 9 ]:     UP         OK             OK    MAC_MODE_4X10GB    DISABLE      [9  :(EXT, OK       ,  C) Tx(Inv) : Rx(Inv)]  ---
    53/2:     CNX- 37  :[PTG: 9 ]:     UP         OK             OK    MAC_MODE_4X10GB    DISABLE      [10 :(EXT, OK       ,  C) Tx(Inv) : Rx     ]  ---
    53/4:     CNX- 38  :[PTG: 9 ]:     UP         OK             OK    MAC_MODE_4X10GB    DISABLE      [11 :(EXT, OK       ,  C) Tx(Inv) : Rx(Inv)]  ---
    53/3:     CNX- 39  :[PTG: 9 ]:     UP         OK             OK    MAC_MODE_4X10GB    DISABLE      [12 :(EXT, OK       ,  C) Tx(Inv) : Rx     ]  ---
    51/1:     CNX- 40  :[PTG: 10]:     UP         OK             OK    MAC_MODE_4X10GB    DISABLE      [13 :(EXT, OK       ,  C) Tx(Inv) : Rx     ]  ---
    51/2:     CNX- 41  :[PTG: 10]:   DOWN      FAULT      NO_SIGNAL    MAC_MODE_4X10GB    DISABLE      [14 :(EXT, OK       ,  C) Tx(Inv) : Rx     ]  ---
    51/4:     CNX- 42  :[PTG: 10]:   DOWN      FAULT      NO_SIGNAL    MAC_MODE_4X10GB    DISABLE      [15 :(EXT, NO_SIGNAL, NC) Tx(Inv) : Rx(Inv)]  ---
    51/3:     CNX- 43  :[PTG: 10]:   DOWN      FAULT      NO_SIGNAL    MAC_MODE_4X10GB    DISABLE      [16 :(EXT, NO_SIGNAL, NC) Tx(Inv) : Rx(Inv)]  ---
    49/1:     CNX- 44  :[PTG: 11]:     UP         OK             OK    MAC_MODE_4X10GB    DISABLE      [17 :(EXT, OK       ,  C) Tx(Inv) : Rx     ]  ---
    49/2:     CNX- 45  :[PTG: 11]:     UP         OK             OK    MAC_MODE_4X10GB    DISABLE      [18 :(EXT, OK       ,  C) Tx(Inv) : Rx     ]  ---
    49/4:     CNX- 46  :[PTG: 11]:     UP         OK             OK    MAC_MODE_4X10GB    DISABLE      [19 :(EXT, OK       ,  C) Tx(Inv) : Rx(Inv)]  ---
    49/3:     CNX- 47  :[PTG: 11]:     UP         OK             OK    MAC_MODE_4X10GB    DISABLE      [20 :(EXT, OK       ,  C) Tx(Inv) : Rx(Inv)]  ---
    40/1:     CNX- 48  :[PTG: 12]:   DOWN      FAULT      NO_SIGNAL    MAC_MODE_4X25GB    DISABLE      [21 :(EXT, NO_SIGNAL, NC) Tx      : Rx(Inv)]  ---
    47/1:     CNX- 49  :[PTG: 12]:   DOWN      FAULT      NO_SIGNAL    MAC_MODE_4X25GB    DISABLE      [22 :(EXT, NO_SIGNAL, NC) Tx      : Rx     ]  ---
    34/1:     CNX- 50  :[PTG: 12]:   DOWN      FAULT      NO_SIGNAL    MAC_MODE_4X25GB    DISABLE      [23 :(EXT, NO_SIGNAL, NC) Tx(Inv) : Rx     ]  ---
    33/1:     CNX- 51  :[PTG: 12]:   DOWN      FAULT      NO_SIGNAL    MAC_MODE_4X25GB    DISABLE      [24 :(EXT, NO_SIGNAL, NC) Tx(Inv) : Rx     ]  ---
    54/2:     CNX- 52  :[PTG: 13]:     UP         OK             OK    MAC_MODE_4X10GB    DISABLE      [25 :(EXT, OK       ,  C) Tx      : Rx     ]  ---
    54/1:     CNX- 53  :[PTG: 13]:   DOWN   FAULT(R)             OK    MAC_MODE_4X10GB    DISABLE      [26 :(EXT, OK       ,  C) Tx      : Rx     ]  ---
    54/3:     CNX- 54  :[PTG: 13]:     UP         OK             OK    MAC_MODE_4X10GB    DISABLE      [27 :(EXT, OK       ,  C) Tx(Inv) : Rx     ]  ---
    54/4:     CNX- 55  :[PTG: 13]:     UP         OK             OK    MAC_MODE_4X10GB    DISABLE      [28 :(EXT, OK       ,  C) Tx(Inv) : Rx     ]  ---
    52/2:     CNX- 56  :[PTG: 14]:   DOWN      FAULT      NO_SIGNAL    MAC_MODE_4X10GB    DISABLE      [29 :(EXT, NO_SIGNAL, NC) Tx(Inv) : Rx     ]  ---
    52/1:     CNX- 57  :[PTG: 14]:   DOWN      FAULT      NO_SIGNAL    MAC_MODE_4X10GB    DISABLE      Error: Mac serdes signalOK = 0 for portNum 2, Can not tune serdes.
[30 :(EXT, NO_SIGNAL, NC) Tx      : Rx     ]  ---
    52/3:     CNX- 58  :[PTG: 14]:   DOWN      FAULT      NO_SIGNAL    MAC_MODE_4X10GB    DISABLE      [31 :(EXT, NO_SIGNAL, NC) Tx(Inv) : Rx     ]  ---
    52/4:     CNX- 59  :[PTG: 14]:   DOWN      FAULT      NO_SIGNAL    MAC_MODE_4X10GB    DISABLE      [32 :(EXT, NO_SIGNAL, NC) Tx(Inv) : Rx     ]  ---
    50/2:     CNX- 60  :[PTG: 15]:     UP         OK             OK    MAC_MODE_4X10GB    DISABLE      [33 :(EXT, OK       ,  C) Tx      : Rx     ]  ---
    50/1:     CNX- 61  :[PTG: 15]:     UP         OK             OK    MAC_MODE_4X10GB    DISABLE      [34 :(EXT, OK       ,  C) Tx(Inv) : Rx     ]  ---
    50/4:     CNX- 62  :[PTG: 15]:     UP         OK             OK    MAC_MODE_4X10GB    DISABLE      [35 :(EXT, OK       ,  C) Tx(Inv) : Rx(Inv)]  ---
    50/3:     CNX- 63  :[PTG: 15]:     UP         OK             OK    MAC_MODE_4X10GB    DISABLE      [36 :(EXT, OK       ,  C) Tx(Inv) : Rx     ]  ---
    41/1:     CNX- 64  :[PTG: 16]:   DOWN      FAULT      NO_SIGNAL    MAC_MODE_4X25GB    DISABLE      [37 :(EXT, NO_SIGNAL, NC) Tx      : Rx(Inv)]  ---
    43/1:     CNX- 65  :[PTG: 16]:   DOWN      FAULT      NO_SIGNAL    MAC_MODE_4X25GB    DISABLE      [38 :(EXT, NO_SIGNAL, NC) Tx(Inv) : Rx(Inv)]  ---
    44/1:     CNX- 66  :[PTG: 16]:   DOWN      FAULT      NO_SIGNAL    MAC_MODE_4X25GB    DISABLE      [39 :(EXT, NO_SIGNAL, NC) Tx(Inv) : Rx     ]  ---
    48/1:     CNX- 67  :[PTG: 16]:   DOWN      FAULT      NO_SIGNAL    MAC_MODE_4X25GB    DISABLE      [40 :(EXT, NO_SIGNAL, NC) Tx      : Rx(Inv)]  ---
    39/1:     CNX- 68  :[PTG: 17]:   DOWN      FAULT      NO_SIGNAL    MAC_MODE_4X25GB    DISABLE      [41 :(EXT, NO_SIGNAL, NC) Tx      : Rx(Inv)]  ---
    45/1:     CNX- 69  :[PTG: 17]:   DOWN      FAULT      NO_SIGNAL    MAC_MODE_4X25GB    DISABLE      [42 :(EXT, NO_SIGNAL, NC) Tx(Inv) : Rx     ]  ---
    46/1:     CNX- 70  :[PTG: 17]:   DOWN      FAULT      NO_SIGNAL    MAC_MODE_4X25GB    DISABLE      [43 :(EXT, NO_SIGNAL, NC) Tx(Inv) : Rx(Inv)]  ---
    42/1:     CNX- 71  :[PTG: 17]:   DOWN      FAULT      NO_SIGNAL    MAC_MODE_4X25GB    DISABLE      [44 :(EXT, NO_SIGNAL, NC) Tx(Inv) : Rx     ]  ---

(xpShell):linkMgr) 

S4248-fbl:
    Duplex   : full
root@AG2:~# hshell -c "ps"
Entry not found
0:_bcm_dpp_gport_to_hw_resources: Entry not found
0:bcm_petra_port_discard_set: Entry not found
                 ena/    speed/ link auto    STP                  lrn  inter   max  loop
           port  link    duplex scan neg?   state   pause  discrd ops   face frame  back
-----------------
-----------------
      xe65( 65)  down   10G  FD   SW  No   Forward          None   FA    XFI  9028
      xe66( 66)  down   10G  FD   SW  No   Forward          None   FA    XFI  9028
      xe67( 67)  down   10G  FD   SW  No   Forward          None   FA    XFI  9028
      xe68( 68)  down   10G  FD   SW  No   Forward          None   FA    XFI  9028
      xe69( 69)  up     10G  FD   SW  No   Forward          None   FA    XFI  9028
      xe70( 70)  up     10G  FD   SW  No   Forward          None   FA    XFI  9028
      xe71( 71)  up     10G  FD   SW  No   Forward          None   FA    XFI  9028
      xe72( 72)  up     10G  FD   SW  No   Forward          None   FA    XFI  9028

Even though all links are down on hardware in s4248 device, yet opx-ethtool shows one link is up. As those links are part of the bond and karnel try to bring up those interface , I have seen a different result when I ran the ps for second time.

      xe65( 65)  down   10G  FD   SW  No   Forward          None   FA    XFI  9028
      xe66( 66)  down   10G  FD   SW  No   Forward          None   FA    XFI  9028
      xe67( 67)  up     10G  FD   SW  No   Forward          None   FA    XFI  9028
      xe68( 68)  down   10G  FD   SW  No   Forward          None   FA    XFI  9028
      xe69( 69)  up     10G  FD   SW  No   Forward          None   FA    XFI  9028
      xe70( 70)  up     10G  FD   SW  No   Forward          None   FA    XFI  9028
      xe71( 71)  up     10G  FD   SW  No   Forward          None   FA    XFI  9028
      xe72( 72)  up     10G  FD   SW  No   Forward          None   FA    XFI  9028
      xe65( 65)  up     10G  FD   SW  No   Forward          None   FA    XFI  9028
      xe66( 66)  down   10G  FD   SW  No   Forward          None   FA    XFI  9028
      xe67( 67)  up     10G  FD   SW  No   Forward          None   FA    XFI  9028
      xe68( 68)  down   10G  FD   SW  No   Forward          None   FA    XFI  9028
      xe69( 69)  up     10G  FD   SW  No   Forward          None   FA    XFI  9028
      xe70( 70)  up     10G  FD   SW  No   Forward          None   FA    XFI  9028
      xe71( 71)  up     10G  FD   SW  No   Forward          None   FA    XFI  9028
      xe72( 72)  up     10G  FD   SW  No   Forward          None   FA    XFI  9028

After some times all links are up in hardware on S5148 side

    54/2:     CNX- 52  :[PTG: 13]:     UP         OK             OK    MAC_MODE_4X10GB    DISABLE      [25 :(EXT, OK       ,  C) Tx      : Rx     ]  ---
    54/1:     CNX- 53  :[PTG: 13]:     UP         OK             OK    MAC_MODE_4X10GB    DISABLE      [26 :(EXT, OK       ,  C) Tx      : Rx     ]  ---
    54/3:     CNX- 54  :[PTG: 13]:     UP         OK             OK    MAC_MODE_4X10GB    DISABLE      [27 :(EXT, OK       ,  C) Tx(Inv) : Rx     ]  ---
    54/4:     CNX- 55  :[PTG: 13]:     UP         OK             OK    MAC_MODE_4X10GB    DISABLE      [28 :(EXT, OK       ,  C) Tx(Inv) : Rx     ]  ---

Metadata

Metadata

Labels

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions