Topic
  • 8 replies
  • Latest Post - ‏2013-11-26T12:16:50Z by HendersonD
HendersonD
HendersonD
49 Posts

Pinned topic LACP across two stacked switches?

‏2013-09-25T23:16:23Z |

IBM Bladecenter H chassis
Two 10GB ethernet switch modules in the chassis, they are stacked and are running firmware 7.7.1 which is very new
Two ports on each switch connected to my core switch. These 4 total connections are setup as an LACP bundle

When I run "show lacp information" command this is what I see on the 4 links:

port    mode    adminkey  operkey  selected   prio  aggr  trunk  status  minlinks
---------------------------------------------------------------------------------------------------------
1:17    active        17            17         yes      32768    49     65     up         1
1:18    active        17            17         yes      32768    49     65     up         1
2:17    active        17            17         yes      32768    49     65   down       1
2:18    active        17            17         yes      32768    49     65   down       1

Basically only one switch (two ports) in the lacp bundle are up. Two questions:

  • Can an LACP bundle include ports on two different stacked switches?
  • If so, any idea why I have two ports down? How can I bring them back up?



 

 

  • SlowinskiS
    SlowinskiS
    28 Posts

    Re: LACP across two stacked switches?

    ‏2013-10-05T12:50:26Z  

    Hi ,

    can you please share config both IBM and core switches ?

     

  • HendersonD
    HendersonD
    49 Posts

    Re: LACP across two stacked switches?

    ‏2013-10-05T14:00:16Z  

    Hi ,

    can you please share config both IBM and core switches ?

     

    Here is the relevant portion of the Juniper config first. I am very confident this is correct, I ran it by a Juniper engineer and we have setup LACP bundles on the Juniper many times. If you scroll down I have included the IBM switch config after this one. Keep in mind, what I am really looking for is can an LACP bundle be made using links from two different IBM switches that are stacked. If the answer is no then the config does not matter. If the answer is yes, then the configs below become relevant.

    JUNIPER EX-4500 CONFIG

     

        xe-0/0/27 {.

            description "Bladecenter Bay7 BNT";

            ether-options {

                802.3ad ae12;

            }

        }

        xe-0/0/28 {

            description "Bladecenter Bay9 BNT";

            ether-options {

                802.3ad ae12;

            }

        }

        xe-1/0/27 {

            description "Bladecenter Bay7 BNT";

            ether-options {

                802.3ad ae12;

            }

        }

        xe-1/0/28 {

            description "Bladecenter Bay9 BNT";

            ether-options {

                802.3ad ae12;

            }

        }

        ae12 {

            description "Bladecenter BNT";

            mtu 9216;

            aggregated-ether-options {

                lacp {

                    active;

                }

            }

            unit 0 {

                family ethernet-switching {

                    port-mode trunk;

                    vlan {

                        members [ Servers Storage VDI ];

                    }

                }

            }

        }

     

    IBM 10GB SWITCH IN BLADECENTER

     

    interface port 1:17
            description "EXT1-Juniper-0/0/27"
            switchport mode trunk
            switchport trunk allowed vlan 1,125-126,200
            vlan dot1q tag native
            exit
    !
    interface port 1:18
            description "EXT2-Juniper-1/0/27"
            switchport mode trunk
            switchport trunk allowed vlan 1,125-126,200
            vlan dot1q tag native
            exit
    !
    interface port 2:17
            description "EXT1-Juniper-0/0/28"
            switchport mode trunk
            switchport trunk allowed vlan 1,125-126,200
            vlan dot1q tag native
            exit
    !
    interface port 2:18
            description "EXT2-Juniper-1/0/28"
            switchport mode trunk
            switchport trunk allowed vlan 1,125-126,200
            vlan dot1q tag native
            exit
    !
    interface port 1:17
            lacp mode active
            lacp key 17
    !
    interface port 1:18
            lacp mode active
            lacp key 17
    !
    interface port 2:17
            lacp mode active
            lacp key 17
    !
    interface port 2:18
            lacp mode active
            lacp key 17
    !
  • SlowinskiS
    SlowinskiS
    28 Posts

    Re: LACP across two stacked switches?

    ‏2013-10-05T14:09:34Z  

    Here is the relevant portion of the Juniper config first. I am very confident this is correct, I ran it by a Juniper engineer and we have setup LACP bundles on the Juniper many times. If you scroll down I have included the IBM switch config after this one. Keep in mind, what I am really looking for is can an LACP bundle be made using links from two different IBM switches that are stacked. If the answer is no then the config does not matter. If the answer is yes, then the configs below become relevant.

    JUNIPER EX-4500 CONFIG

     

        xe-0/0/27 {.

            description "Bladecenter Bay7 BNT";

            ether-options {

                802.3ad ae12;

            }

        }

        xe-0/0/28 {

            description "Bladecenter Bay9 BNT";

            ether-options {

                802.3ad ae12;

            }

        }

        xe-1/0/27 {

            description "Bladecenter Bay7 BNT";

            ether-options {

                802.3ad ae12;

            }

        }

        xe-1/0/28 {

            description "Bladecenter Bay9 BNT";

            ether-options {

                802.3ad ae12;

            }

        }

        ae12 {

            description "Bladecenter BNT";

            mtu 9216;

            aggregated-ether-options {

                lacp {

                    active;

                }

            }

            unit 0 {

                family ethernet-switching {

                    port-mode trunk;

                    vlan {

                        members [ Servers Storage VDI ];

                    }

                }

            }

        }

     

    IBM 10GB SWITCH IN BLADECENTER

     

    interface port 1:17
            description "EXT1-Juniper-0/0/27"
            switchport mode trunk
            switchport trunk allowed vlan 1,125-126,200
            vlan dot1q tag native
            exit
    !
    interface port 1:18
            description "EXT2-Juniper-1/0/27"
            switchport mode trunk
            switchport trunk allowed vlan 1,125-126,200
            vlan dot1q tag native
            exit
    !
    interface port 2:17
            description "EXT1-Juniper-0/0/28"
            switchport mode trunk
            switchport trunk allowed vlan 1,125-126,200
            vlan dot1q tag native
            exit
    !
    interface port 2:18
            description "EXT2-Juniper-1/0/28"
            switchport mode trunk
            switchport trunk allowed vlan 1,125-126,200
            vlan dot1q tag native
            exit
    !
    interface port 1:17
            lacp mode active
            lacp key 17
    !
    interface port 1:18
            lacp mode active
            lacp key 17
    !
    interface port 2:17
            lacp mode active
            lacp key 17
    !
    interface port 2:18
            lacp mode active
            lacp key 17
    !

    Yes, config looks good.

    Did you change on IBM side lacp timeout?

    Juniper is using by default short , IBm long, therefore please try following command:

    lacp timeout short

    Maybe this can help...

     

     

  • HendersonD
    HendersonD
    49 Posts

    Re: LACP across two stacked switches?

    ‏2013-10-06T15:09:22Z  

    Yes, config looks good.

    Did you change on IBM side lacp timeout?

    Juniper is using by default short , IBm long, therefore please try following command:

    lacp timeout short

    Maybe this can help...

     

     

    I made a few changes to my environment

    • Changed the IBM switch to lacp timeout short
    • I had two vstp vlans configured on my core switch, I took those out relying solely on rstp
    • Changed the IBM switch from pvrstp to rstp

    Now I have a very simple spanning tree environment

    • My Juniper core switch has a bridge priority of 20K making it the root
    • All other switches have the default priority of 32768
    • All Juniper switches (most of my environment) and the two IBM switches in the Bladecenter run RSTP
    • I do have two Cisco switches in the back of my Bladecenter and one other Cisco switch on campus that run Rapid-PVRSTP+ which is fully compatible with RSTP

    If you look at the diagram attached it shows the situation. Four links from my Juniper core to the two IBM switches. Both ends are configured as a single LACP bundle. The two green links are active, the two red links are not. I removed the two red links from the bundle and added them back in. The are still shown as disabled according to Spanning-Tree.

    Back to my original question, can links across two stacked IBM switches be included in an LACP bundle?

    I have read through all the documentation and there is a big list of items that are not compatible with stacked switches, LACP bundles is not mentioned

    Attachments

    Updated on 2013-10-06T15:16:03Z at 2013-10-06T15:16:03Z by HendersonD
  • SlowinskiS
    SlowinskiS
    28 Posts

    Re: LACP across two stacked switches?

    ‏2013-10-06T18:25:52Z  

    I made a few changes to my environment

    • Changed the IBM switch to lacp timeout short
    • I had two vstp vlans configured on my core switch, I took those out relying solely on rstp
    • Changed the IBM switch from pvrstp to rstp

    Now I have a very simple spanning tree environment

    • My Juniper core switch has a bridge priority of 20K making it the root
    • All other switches have the default priority of 32768
    • All Juniper switches (most of my environment) and the two IBM switches in the Bladecenter run RSTP
    • I do have two Cisco switches in the back of my Bladecenter and one other Cisco switch on campus that run Rapid-PVRSTP+ which is fully compatible with RSTP

    If you look at the diagram attached it shows the situation. Four links from my Juniper core to the two IBM switches. Both ends are configured as a single LACP bundle. The two green links are active, the two red links are not. I removed the two red links from the bundle and added them back in. The are still shown as disabled according to Spanning-Tree.

    Back to my original question, can links across two stacked IBM switches be included in an LACP bundle?

    I have read through all the documentation and there is a big list of items that are not compatible with stacked switches, LACP bundles is not mentioned

    Can you please provide sh int status and sh int info output from ibm?

     

     

     

  • HendersonD
    HendersonD
    49 Posts

    Re: LACP across two stacked switches?

    ‏2013-10-06T19:41:37Z  

    Can you please provide sh int status and sh int info output from ibm?

     

     

     

    Below is

    • sh int status
    • sh int info
    • sh lacp info

    Ports 1:17, 1:18, 2:17, and 2:18 are the four connections to my Juniper core switch

     

     

    BladeCenter-Bay7>sh int status
    -----------------------------------------------------------------------
    Alias   Port   Speed    Duplex     Flow Ctrl      Link     Description
    ------- ----   -----   --------  --TX-----RX--   ------   -------------
    1:1       1     10000     full     yes    yes       up       vSphere01
    1:2       2     10000     full     yes    yes       up       vSphere02
    1:3       3     10000     full     yes    yes       up       vSphere03
    1:4       4     10000     full     yes    yes       up       vSphere04
    1:5       5     10000     full     yes    yes       up       vSphere05
    1:6       6     10000     full     yes    yes       up       vSphere06
    1:7       7     10000     full     yes    yes       up       vSphere07
    1:8       8     10000     full     yes    yes       up       vSphere08
    1:9       9     1G/10G    full     yes    yes      down      1:9
    1:10      10    1G/10G    full     yes    yes      down      1:10
    1:11      11    1G/10G    full     yes    yes      down      1:11
    1:12      12    1G/10G    full     yes    yes      down      1:12
    1:13      13    1G/10G    full     yes    yes      down      1:13
    1:14      14    10000     full     yes    yes       up       ContentIndex
    1:15      15      100*    full*    yes*   yes*      up       1:15
    1:16      16      100*    full*    yes*   yes*     down      1:16
    1:17      17    10000     full      no     no       up       EXT1-Juniper-0/0/27
    1:18      18    10000     full      no     no       up       EXT2-Juniper-1/0/27
    1:19      19    1G/10G    full      no     no    disabled    1:19
    1:20      20    1G/10G    full      no     no    disabled    1:20
    1:21      21    1G/10G    full      no     no    disabled    1:21
    1:22      22    1G/10G    full      no     no    disabled    1:22
    1:23      23    1G/10G    full      no     no    disabled    1:23
    1:24      24    1G/10G    full      no     no    disabled    1:24
    1:25      25    10000     full      no     no       up       Stack to Bay9
    1:26      26    10000     full      no     no       up       Stack to Bay9
    1:27      27      any     full      no     no      down      1:27
    2:1       65    10000     full     yes    yes       up       vSphere01
    2:2       66    10000     full     yes    yes       up       vSphere02
    2:3       67    10000     full     yes    yes       up       vSphere03
    2:4       68    10000     full     yes    yes       up       vSphere04
    2:5       69    10000     full     yes    yes       up       vSphere05
    2:6       70    10000     full     yes    yes       up       vSphere06
    2:7       71    10000     full     yes    yes       up       vSphere07
    2:8       72    10000     full     yes    yes       up       vSphere08
    2:9       73    1G/10G    full     yes    yes      down      2:9
    2:10      74    1G/10G    full     yes    yes      down      2:10
    2:11      75    1G/10G    full     yes    yes      down      2:11
    2:12      76    1G/10G    full     yes    yes      down      2:12
    2:13      77    1G/10G    full     yes    yes      down      2:13
    2:14      78    10000     full     yes    yes       up       ContentIndex
    2:15      79      100*    full*    yes*   yes*      up       2:15
    2:16      80      100*    full*    yes*   yes*     down      2:16
    2:17      81    10000     full      no     no       up       EXT1-Juniper-0/0/28
    2:18      82    10000     full      no     no       up       EXT2-Juniper-1/0/28
    2:19      83    1G/10G*   full*     no*    no*   disabled    2:19
    2:20      84    1G/10G*   full*     no*    no*   disabled    2:20
    2:21      85    1G/10G*   full*     no*    no*   disabled    2:21
    2:22      86    1G/10G*   full*     no*    no*   disabled    2:22
    2:23      87    1G/10G*   full*     no*    no*   disabled    2:23
    2:24      88    1G/10G*   full*     no*    no*   disabled    2:24
    2:25      89    10000     full      no     no       up       Stack to Bay7
    2:26      90    10000     full      no     no       up       Stack to Bay7
    2:27      91      any     full      no     no      down      2:27

     

    BladeCenter-Bay7>sh int info              
    Alias   Port Tag    Type    RMON Lrn Fld PVID     DESCRIPTION              VLAN(s)
                 Trk                         NVLAN                                    
    ------- ---- --- ---------- ---- --- --- ------ -------------- -------------------------------
    1:1     1     y  Internal    d    e   e     1   vSphere01      1 125 126 127 4095 
    1:2     2     y  Internal    d    e   e     1   vSphere02      1 125 126 127 4095 
    1:3     3     y  Internal    d    e   e     1   vSphere03      1 125 126 127 4095 
    1:4     4     y  Internal    d    e   e     1   vSphere04      1 125 126 127 200 4095 
    1:5     5     y  Internal    d    e   e     1   vSphere05      1 125 126 127 200 4095 
    1:6     6     y  Internal    d    e   e     1   vSphere06      1 125 126 127 200 4095 
    1:7     7     y  Internal    d    e   e     1   vSphere07      1 125 126 127 200 4095 
    1:8     8     y  Internal    d    e   e     1   vSphere08      1 125 126 127 200 4095 
    1:9     9     y  Internal    d    e   e     1   1:9            1 4095 
    1:10    10    y  Internal    d    e   e     1   1:10           1 4095 
    1:11    11    y  Internal    d    e   e     1   1:11           1 4095 
    1:12    12    y  Internal    d    e   e     1   1:12           1 4095 
    1:13    13    y  Internal    d    e   e     1   1:13           1 4095 
    1:14    14    y  Internal    d    e   e     1   ContentIndex   1 125 126 4095 
    1:15    15    y  LocalMgmt   d    e   e  4095*  1:15           4095 
    1:16    16    y  LocalMgmt   d    e   e  4095*  1:16           4095 
    1:17    17    y  External    d    e   e     1*  EXT1-Juniper-0 1 125 126 200 
                                                    /0/27          
    1:18    18    y  External    d    e   e     1*  EXT2-Juniper-1 1 125 126 200 
                                                    /0/27          
    1:19    19    n  External    d    e   e     1   1:19           1 
    1:20    20    n  External    d    e   e     1   1:20           1 
    1:21    21    n  External    d    e   e     1   1:21           1 
    1:22    22    n  External    d    e   e     1   1:22           1 
    1:23    23    n  External    d    e   e     1   1:23           1 
    1:24    24    n  External    d    e   e     1   1:24           1 
    1:25    25    n  Stacking    d    e   e     1   Stack to Bay9  Stacking
    1:26    26    n  Stacking    d    e   e     1   Stack to Bay9  Stacking
    1:27    27    n  External    d    e   e     1   1:27           1 
    2:1     65    y  Internal    d    e   e     1   vSphere01      1 125 126 127 4095 
    2:2     66    y  Internal    d    e   e     1   vSphere02      1 125 126 127 4095 
    2:3     67    y  Internal    d    e   e     1   vSphere03      1 125 126 127 4095 
    2:4     68    y  Internal    d    e   e     1   vSphere04      1 125 126 127 200 4095 
    2:5     69    y  Internal    d    e   e     1   vSphere05      1 125 126 127 200 4095 
    2:6     70    y  Internal    d    e   e     1   vSphere06      1 125 126 127 200 4095 
    2:7     71    y  Internal    d    e   e     1   vSphere07      1 125 126 127 200 4095 
    2:8     72    y  Internal    d    e   e     1   vSphere08      1 125 126 127 200 4095 
    2:9     73    y  Internal    d    e   e     1   2:9            1 4095 
    2:10    74    y  Internal    d    e   e     1   2:10           1 4095 
    2:11    75    y  Internal    d    e   e     1   2:11           1 4095 
    2:12    76    y  Internal    d    e   e     1   2:12           1 4095 
    2:13    77    y  Internal    d    e   e     1   2:13           1 4095 
    2:14    78    y  Internal    d    e   e     1   ContentIndex   1 125 126 4095 
    2:15    79    y  RemoteMgmt  d    e   e  4095*  2:15           4095 
    2:16    80    y  RemoteMgmt  d    e   e  4095*  2:16           4095 
    2:17    81    y  External    d    e   e     1*  EXT1-Juniper-0 1 125 126 200 
                                                    /0/28          
    2:18    82    y  External    d    e   e     1*  EXT2-Juniper-1 1 125 126 200 
                                                    /0/28          
    2:19    83    n  External    d    e   e     1   2:19           1 
    2:20    84    n  External    d    e   e     1   2:20           1 
    2:21    85    n  External    d    e   e     1   2:21           1 
    2:22    86    n  External    d    e   e     1   2:22           1 
    2:23    87    n  External    d    e   e     1   2:23           1 
    2:24    88    n  External    d    e   e     1   2:24           1 
    2:25    89    n  Stacking    d    e   e     1   Stack to Bay7  Stacking
    2:26    90    n  Stacking    d    e   e     1   Stack to Bay7  Stacking
    2:27    91    n  External    d    e   e     1   2:27           1 

     

    BladeCenter-Bay7>sh lacp info             
    port    mode    adminkey  operkey  selected   prio  aggr  trunk  status  minlinks
    ---------------------------------------------------------------------------------
    1:1     off           33       33     no     32768    --     --    --        1
    1:2     off           34       34     no     32768    --     --    --        1
    1:3     off           35       35     no     32768    --     --    --        1
    1:4     off           36       36     no     32768    --     --    --        1
    1:5     off           37       37     no     32768    --     --    --        1
    1:6     off           38       38     no     32768    --     --    --        1
    1:7     off           39       39     no     32768    --     --    --        1
    1:8     off           40       40     no     32768    --     --    --        1
    1:9     off           41       41     no     32768    --     --    --        1
    1:10    off           42       42     no     32768    --     --    --        1
    1:11    off           43       43     no     32768    --     --    --        1
    1:12    off           44       44     no     32768    --     --    --        1
    1:13    off           45       45     no     32768    --     --    --        1
    1:14    off           46       46     no     32768    --     --    --        1
    1:17    active        17       17    yes     32768    49     65    up        1
    1:18    active        17       17    yes     32768    49     65    up        1
    1:19    off           51       51     no     32768    --     --    --        1
    1:20    off           52       52     no     32768    --     --    --        1
    1:21    off           53       53     no     32768    --     --    --        1
    1:22    off           54       54     no     32768    --     --    --        1
    1:23    off           55       55     no     32768    --     --    --        1
    1:24    off           56       56     no     32768    --     --    --        1
    1:27    off           59       59     no     32768    --     --    --        1
    2:1     off           65       65     no     32768    --     --    --        1
    2:2     off           66       66     no     32768    --     --    --        1
    2:3     off           67       67     no     32768    --     --    --        1
    2:4     off           68       68     no     32768    --     --    --        1
    2:5     off           69       69     no     32768    --     --    --        1
    2:6     off           70       70     no     32768    --     --    --        1
    2:7     off           71       71     no     32768    --     --    --        1
    2:8     off           72       72     no     32768    --     --    --        1
    2:9     off           73       73     no     32768    --     --    --        1
    2:10    off           74       74     no     32768    --     --    --        1
    2:11    off           75       75     no     32768    --     --    --        1
    2:12    off           76       76     no     32768    --     --    --        1
    2:13    off           77       77     no     32768    --     --    --        1
    2:14    off           78       78     no     32768    --     --    --        1
    2:17    active        17       17    yes     32768    49     65   down       1
    2:18    active        17       17    yes     32768    49     65   down       1
  • HendersonD
    HendersonD
    49 Posts

    Re: LACP across two stacked switches?

    ‏2013-10-27T16:46:49Z  

    Opened a ticket with IBM this past week and so far they cannot figure out why this LACP bundle does not work. The engineer did verify that an LACP bundle across two stacked IBM switches should work. I have also opened a ticket with Juniper on this, hope to hear back tomorrow (Monday) from them.

  • HendersonD
    HendersonD
    49 Posts

    Re: LACP across two stacked switches?

    ‏2013-11-26T12:16:50Z  

    Opened a ticket with IBM this past week and so far they cannot figure out why this LACP bundle does not work. The engineer did verify that an LACP bundle across two stacked IBM switches should work. I have also opened a ticket with Juniper on this, hope to hear back tomorrow (Monday) from them.

    Problem is solved

    One the four ports on the IBM switch, VLAN tagging and PVID tagging were turned on. Once PVID tagging was turned off, native vlan set to 1 on the IBM and Juniper side, all 4 links became part of the LACP bundle