Blade-based operational models are of tremendous value in the wired and wireless telecommunications domain for several reasons:
- Small footprints mean cost-effective use of data center space.
- Deployment meets NEBS requirements for distributed deployments. (NEBS, or Network Equipment Building System, is a set of criteria that networked equipment must meet to qualify for compatibility.)
- Cost-effective massive horizontal scalability lowers deployment costs for the telecommunications service provider.
- Centralized management support provides better OAM&P support for in-network deployment in service provider networks. (OAM&P stands for "operations, administration, maintenance, and provisioning." The term describes this collection of disciplines and specific software used to track it.)
- Built-in support for continuous-availability-based operational models—including upgrades and maintenance activities—avoids service downtime from a subscriber perspective.
These additional considerations are key in a telecommunications service provider environment, especially one with complex configurations:
- Multiple VLANs. These are used for CDN (Customer Data Network) and Management (OAM&P) traffic. Considering them separately ensures that subscriber QoS (Quality of Service) is effectively maintained across multiple LPARs (logical partitions).
- Micro-partitioning and virtualization. These strategies help maximize the capacity utilization and TCO (Total Cost Of Ownership).
- Existing network complexities. Existing networks can have a higher degree of load variability, requiring resource load balancing among multiple client LPARs.
In this article, we describe an implementation of a multiple VLAN configuration of a blade chassis using Cisco switches paired in an active/passive configuration. In our example, we configured networks to connect multiple VLANs on a BladeCenter® JS22 using Linux® on Power. The architecture consists of six Cisco catalyst switch modules with fourteen internal ports and four external 1GB ports each.
To properly leverage all six switches in the chassis, the blade requires six Ethernet interfaces. Ethernet interface ent0 on the blade maps to the first switch on the blade chassis; each consecutive Ethernet interface maps to the next available switch. This mapping creates a limitation, because it does not allow administrators to map the physical adapter on the blade to the switch they choose to map them to.
When creating network architecture for your blades, you must have one physical interface for each Cisco switch you want to use on the chassis. If some blades do not have the same number of adapters as the chassis, the switches that do not have a physical adapter associated with them on the blade cannot be used by the blade.
It is important to understand how to pair the Ethernet interfaces on the blade and the switches within the chassis. The first switch in the chassis is normally located in the upper left of the blade chassis, just below the power plug. It would map to ent0 on the blade since it is the first interface on the blade. Figure 1 shows the numbering of the switches in our configuration.
Figure 1. Physical adapter switches in the configuring we're using
Determining switch pairing is extremely important for high availability. In a typical configuration, one power distribution unit (PDU) supplies power to the top half of the blade chassis while another PDU provides power to the bottom half of the chassis. If you are creating a redundant solution, it's important to split the primary and secondary switch between the upper half and lower half of the chassis.
In our case, we created a solution with the following switch pairs: (1, 9) (2, 3) (4, 7). Because adapter pairs (ent0, ent1), (ent2, ent3), and (ent4, ent5) are on the same physical I/O card, we also needed to make sure that the network traffic for our target VLANs did not travel across the same I/O card. Our configuration splits traffic across multiple PDUs and across multiple interfaces.
Even though the pairing of the adapters and switches may seem simple, there are multiple steps to configure the IVM (Integrated Virtualization Manager), switches, and LPARs to take advantage of this architecture. Figure 2 represents the setup of one of our blades with the associated switch, trunking, and VLAN tagging. This configuration allows multiple VLANs to communicate over the same physical adapters to multiple switches.
Figure 2. One of the blades with the associated switch, trunking, and VLAN tagging
In this example, each LPAR has two Ethernet adapters that connect to one Virtual Ethernet Adapter (VEA) on the IVM. Notice that the VEA on the IVM has multiple VLANs associated with it. Traffic for each VLAN is separated on the LPARs by their adapters. The VEA trunks the VLANs together and sends them over the Shared Ethernet Adapter (SEA) through a Link Aggregation Device and out to the network via one of the chassis switches. The switches route the VLAN traffic to the appropriate network through the use of VLAN tagging.
Five steps to configuring VLANs
There are five main steps (and one optional step) for configuring VLANs for client LPARs on the IVM:
- Configure Cisco switches. (An optional step at this point may be to create a link aggregation device, commonly referred to as the LNAGG device.)
- Create 802.1q IEEE VLAN-aware interfaces on the IVM to support the VLANs you want to use on each interface. It is very important to design the VLANs prior to any work because it is not possible to modify the created VLAN-aware interfaces afterwards. You would need to delete them and re-add them, which means a big waste of time.
- Assign the Virtual Ethernet Adapter to the physical adapter (LNAGG) in the Virtual Ethernet Menu on the IVM.
- Modify the LPARs properties to map the new virtual adapters to LPAR adapters. Make sure the LPARs are inactive before performing changes to the network device properties.
- Boot each LPAR and configure the new interfaces.
The example in this article is intended for a fresh install of a blade.
Step 1: Configure the Cisco switches
You may choose to skip this step if the switches are already properly configured with the VLANs that you want to use. The example does not demonstrate how to configure spanning trees and switch priority for spanning trees that may be required. If you want to follow the example, then you should configure your switches to match the configuration that follows.
Log in to the switch
Type the following commands in this order:
switchport trunk native vlan 383
switchport trunk allowed vlan 373,383
switchport mode trunk
The commands configure port 1 on the switch for trunking. If traffic comes in from the blade to the switch or from the external ports of the blades to the switch and they are not tagged with a VLAN, the switch will tag them with VLAN 383. The switchport (external ports) will allow only traffic from 373 and 383, in this case, to route through to and from the IVM's VEAs. To change which VLANs need to access the port from the IVM, simply change the trunk-allowed VLANs numbers.
Splitting the traffic
Once the traffic is trunked and sent to the switch, the switch will determine which external port routes the VLAN traffic. In the example, we are sending external VLAN 373 traffic across port 17, and VLAN 383 over port 20.
To configure VLAN 373 traffic over port 17, type the following commands on the Cisco switch:
switchport access vlan 373
switchport mode access
To configure VLAN 383 traffic over port 20, type the following commands on the Cisco switch:
switchport access vlan 383
switchport mode access
After setting the external port configuration, type exit twice to
back out of the configuration mode of the command line. Then run the
show run command to see your configuration. The
show run command displays the actively running
switch configuration. The configuration has not been written to memory in
the switch, but you can see the changes that are currently running on the
switch. If you look in the configuration, you can see changes we made in
the steps above. Look for the Ethernet port for blade 1 as we configured
Listing 1. Displaying the configuration of switch port 1
interface GigabitEthernet0/1 description blade1 switchport trunk native vlan 383 switchport trunk allowed vlan 373,383 switchport mode trunk
If you issue a
show config command, you will
see the previous configuration of the interface, not the one we just
configured. If the switch is
rebooted in this state, the current configuration will be lost. To write
it to memory, type
write on the command
line of the switch. If you run a
again, the configuration stored on the switch for our interfaces will
match the running configuration.
Step optional: Create a Link Aggregation Adapter
A link aggregation device can be used to connect two physical adapters together to look like one adapter. This is useful for creating an active/passive configuration for failover.
In our example, we wanted to create an active/passive configuration on the blade linking adapter ent0 and ent5 together. Issue the following command to create link aggregation device (LNAGG) with a backup adapter on ent5 from the IVM's command line:
$ mkvdev -lnagg ent0 -attr backup_adapter=ent5
lsdev command to verify the creation of
the LNAGG device.
Listing 2. Verifying the creation of the LNAGG device
$lsdev |grep ^ent ent0 Available Logical Host Ethernet Port (lp-hea) ent1 Available Logical Host Ethernet Port (lp-hea) ent2 Available Gigabit Ethernet-SX Adapter (e414a816) ent3 Available Gigabit Ethernet-SX Adapter (e414a816) ent4 Available Gigabit Ethernet-SX PCI-X Adapter (14106703) ent5 Available Gigabit Ethernet-SX PCI-X Adapter (14106703) ent6 Available Virtual I/O Ethernet Adapter (l-lan) ent7 Available Virtual I/O Ethernet Adapter (l-lan) ent8 Available Virtual I/O Ethernet Adapter (l-lan) ent9 Available Virtual I/O Ethernet Adapter (l-lan) ent10 Available EtherChannel/IEEE 802.3ad Link Aggregation ent11 Available EtherChannel/IEEE 802.3ad Link Aggregation ent12 Available EtherChannel/IEEE 802.3ad Link Aggregation ent13 Available Virtual I/O Ethernet Adapter (l-lan) ent14 Available Virtual I/O Ethernet Adapter (l-lan) ent15 Available Virtual I/O Ethernet Adapter (l-lan) ent16 Available Shared Ethernet Adapter ent17 Available Shared Ethernet Adapter ent18 Available Shared Ethernet Adapter
In the output from our
lsdev command, you can
see that the link aggregation device (ent10 - ent12) looks like a physical
adapter. This allows you to map the link aggregation device to a virtual
device (ent13 - ent15) via a Shared Ethernet Adapter (ent16 - ent18). The
SEA treats the link aggregation device (LNAGG) as though it is a physical
adapter. The Virtual Ethernet Adapters (VEA) ent6 - ent9 are created by
default and are not VLAN-aware devices, nor can you modify them to become
VLAN aware. ent0 through ent5 are the physical adapters on the blade
attr flag on the
lsdev command to see how the LNAGG device maps
to its physical adapters:
$lsdev -dev ent10 -attr will produce mapping
as shown in Table 1.
Table 1. LNAGG to PA mapping with $lsdev -dev ent10 -attr
|Attribute||Value||Description||Is it user-settable?|
|adapter_names||ent0||EtherChannel Adapters||True < Primary|
|alt_addr||0x000000000000||Alternate EtherChannel Addr||True|
|auto_recovery||no||Enable automatic recovery after failover||True|
|backup_adapter||ent5||Adapter used when whole channel fails||True < Backup|
|hash_mode||default||Determines how outgoing adapter is chosen||True|
|mode||standard||EtherChannel mode of operation||True|
|netaddr||Address to ping||True|
|noloss_failover||yes||Enable lossless failover after ping failure||True|
|num_retries||8||Times to retry ping before failing||True|
|retry_time||1||Wait time (in seconds) between pings||True|
|use_alt_addr||no||Enable Alternate EtherChannel Address||True|
|use_jumbo_frame||no||Enable Gigabit Ethernet Jumbo Frames||True|
Step 2: Create virtual adapters on the IVM
In this example, we're going to configure network traffic to flow through the Virtual Ethernet Adapter for VLANs 373 and 383. The first step in configuring the solution is to create the virtual adapter on the IVM that is needed to transport traffic to and from the client LPARs. Let's create a virtual adapter with a primary port of 373 and a secondary port of 383.
To create the VLAN-aware interface on the IVM:
- Use your favorite (putty) telnet program, open a window to the IVM, and log in as padmin (the default password is "passw0rd" with a "0" instead of an "O").
- Use the
shwres -r virtualio --rsubtype eth --level lparcommand to list the Ethernet resources:
Listing 3. Listing the Ethernet resources
$lshwres -r virtualio --rsubtype eth --level lpar lpar_name=IVM_01,lpar_id=1,slot_num=3,state=1,ieee_virtual_eth=0,port_vlan_id=1, addl_vlan_ids=none,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=463337C4B503 lpar_name=IVM_01,lpar_id=1,slot_num=4,state=1,ieee_virtual_eth=0,port_vlan_id=2, addl_vlan_ids=none,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=463337C4B504 lpar_name=IVM_01,lpar_id=1,slot_num=5,state=1,ieee_virtual_eth=0,port_vlan_id=3, addl_vlan_ids=none,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=463337C4B505 lpar_name=IVM_01,lpar_id=1,slot_num=6,state=1,ieee_virtual_eth=0,port_vlan_id=4, addl_vlan_ids=none,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=463337C4B506
- Use the
chhwrescommand to create a virtual adapter on the IVM that supports IEEE VLAN awareness and the additional VLANs you want on that interface. To do this, issue the following command from the command line of the IVM:
$ chhwres -p IVM_01 -o a -r virtualio --rsubtype eth -s 15 -a\ '"ieee_virtual_eth=1","port_vlan_id=373","addl_vlan_ids=383","is_trunk=1",\
chhwrescommand tells the IVM how to construct a new VLAN-aware Virtual Ethernet Adapter.
There are some important features of the command that you need to know to create multiple virtual adapters on the IVM:
-p partition: In the command, we are telling the
chhwresthat there is a change to the IVM partition by issuing the
-s nn: This tells the IVM that we are going to use a particular slot number. If this parameter is not specified, the IVM will use the next available slot. The slot number is required when a device is removed from the IVM.
ieee_virtual_eth: A value of 1 informs the IVM that this adapter supports IEEE 802.1Q. This needs to be set to 1 if there are additional VLANs that are required.
port_vlan_id: This is the primary VLAN for the virtual adapter.
add_vlan_ids: If trunking is enabled, then this parameter accepts the additional VLANs.
is_trunk: This attribute must also be set to 1 if you are passing multiple VLANs.
trunk_priority: When trunking, the priority of the adapter must be set between 1-15.
- Ensure the creation is complete by re-running the
lshwrescommand and look for the new devices.
Listing 4. Displaying the new devices
$lshwres -r virtualio --rsubtype eth --level lpar lpar_name=IVM_01,lpar_id=1,slot_num=3,state=1,ieee_virtual_eth=0,port_vlan_id=1, addl_vlan_ids=none,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=463337C4B503 lpar_name=IVM_01,lpar_id=1,slot_num=4,state=1,ieee_virtual_eth=0,port_vlan_id=2, addl_vlan_ids=none,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=463337C4B504 lpar_name=IVM_01,lpar_id=1,slot_num=5,state=1,ieee_virtual_eth=0,port_vlan_id=3, addl_vlan_ids=none,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=463337C4B505 lpar_name=IVM_01,lpar_id=1,slot_num=6,state=1,ieee_virtual_eth=0,port_vlan_id=4, addl_vlan_ids=none,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=463337C4B506 lpar_name=IVM_01,lpar_id=1,slot_num=15,state=1,ieee_virtual_eth=1,port_vlan_id=383, addl_vlan_ids=378,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=463337C4B50F lpar_name=IVM_01,lpar_id=1,slot_num=16,state=1,ieee_virtual_eth=1,port_vlan_id=6, "addl_vlan_ids=22,23",is_trunk=1,trunk_priority=1,is_required=0, mac_addr=463337C4B510 lpar_name=IVM_01,lpar_id=1,slot_num=17,state=1,ieee_virtual_eth=1,port_vlan_id=7, "addl_vlan_ids=565,566,567,568",is_trunk=1,trunk_priority=1,is_required=0, mac_addr=463337C4B511
Step 3: Assign the virtual Ethernet ID to a physical adapter
Once the link aggregation device has been created, it needs to be mapped to a virtual adapter. This is easily accomplished via the IVM GUI as shown in Figure 3.
Figure 3. Mapping the LNAGG to the virtual adapter via the IVM GUI
After logging into the GUI, select the "View/Modify Virtual Ethernet" from the left-side navigation, then choose the "Virtual Ethernet Bridge" tab. From this menu you can see the virtual Ethernet adapter we created previously with the 383 primary VLAN and the 373 additional VLAN. From the drop-down box, you can select the link aggregation device we created in the previous step. Once the new device has been chosen, click Apply. This creates a Shared Ethernet Adapter (SEA) within the IVM.
Step 4: Modify the LPAR's properties
Once a physical adapter or LNAGG device has been mapped to a Virtual Ethernet ID on the IVM, then the virtual adapter needs to be created for each logical partition. The first step is to log into the GUI on the IVM (see Figure 4).
Figure 4. Creating the virtual adapter for each LPAR
After logging in, select View/Modify Partition in the upper left corner. Once the page refreshes, choose the LPAR you are going to modify.
Select the client LPAR in the menu and be sure that your browser supports pop-up menus. On the pop-up menu, select the "Ethernet" tab (Figure 5).
Figure 5. Remember to power-off LPAR to modify properties
In Figure 5, you can see that Virtual Ethernet pull-downs are grayed out; that is because the LPAR was running when the screenshot was taken. Make sure the client LPAR is powered off or inactive before modifying the properties.
Use the pull-downs to map the VLAN-aware VEAs to the client LPAR's adapter on this screen. Notice that the virtual adapter is associated with one VLAN—this allows the IVM to attach VLAN tags to the traffic as it comes in from the operating systems and sends it out the appropriate LNAGG device. If more adapters are needed, click Create Adapter as needed.
Step 5: Configure Linux to use the VLAN-aware interfaces
Once the IVM and Cisco switches have been configured, you may need to do one additional step if the configuration requires static IP addresses for your Linux partitions. From the IVM GUI, activate the LPAR.
- Log in to the box and change your user to root.
- Type the following:
If you perform an
ifconfig command, you will
see that the virtual adapter individual VLAN mapped to the LPAR. With your
favorite editor, change the interfaces parameters to meet your
configuration requirements. The following is an example of an interface,
ifcfg-eth0, with a static IP address:
Listing 5. An interface with a static IP address
DEVICE=eth0 BOOTPROTO=static BROADCAST=192.168.1.31 HWADDR=00:1A:64:8C:B2:32 IPADDR=192.168.1.44 NETMASK=255.255.255.0 ONBOOT=yes TYPE=Ethernet GATEWAY=192.168.1.1
Restart the interfaces using
As with any deployment, planning is crucial to success. To avoid costly rework, it's important to lay out your network design before implementing a complex network within the blade chassis. From our experience, we can tell you that reconfiguring a complex IVM-supported network requires considerable effort; in fact, the administrator usually must remove the previous configuration before reconfiguring.
Planning is also critical to the install because you cannot add new VLANs to the virtual adapters on the fly in the IVM. Since you can only have one IVM in the JS22, you cannot use SEA failover as you can in a traditional VIOS installation. Link aggregation provides a mechanism for routing traffic across multiple switches in the case of a network failure of a switch. When considering redundancy in the blade, remember that the top half of the blade is powered by one PDU, while the bottom half is powered by the other PDU.
All these considerations add up to a relatively complex network implementation, so our best advice is to plan before you act.
- The IBM Installation Toolkit simplifies the installation of Linux on POWER systems.
- In the developerWorks Linux zone, find more resources for Linux developers, and scan our most popular articles and tutorials.
- See all Linux tips and Linux tutorials on developerWorks.
- Stay current with developerWorks technical events and Webcasts.
Get products and technologies
- Take a look at what the IBM BladeCenter JS22 Express has to offer.
- With IBM trial software, available for download directly from developerWorks, build your next development project on Linux.
- Participate in the discussion forum.
- Get involved in the developerWorks community through blogs, forums, podcasts, and spaces.