It is ironic that only days after I wrote that 497 is the IT number of the beast, I learn that Linux has another unfortunate number: 208.
The reason for this is a defect in the internal Linux kernel used in recent firmware levels of SVC, Storwize V7000 and Storwize V7000 Unified nodes. This defect will cause each node to reboot after 208 days of uptime. This issue exists in unfixed versions of the 6.2 and 6.3 level of firmware, so a large number of users are going to need to take some action on this (except those who are still on a 4.x, 5.x, 6.0 or 6.1 release). If you have done a code update after June 2011, then you are probably affected. This means that if you are an IBM client you need to read this alert now and determine how far you are into that 208 day period. If you are an IBMer or an IBM Business Partner, you need to make sure your clients are aware of this issue, though hopefully they have signed up for IBM My Notifications and have already been notified by e-mail.
In short what needs to happen is that you must:
- Determine your current firmware level.
- Check the table in the alert to determine if you are affected at all, and if so, how far you are potentially into the 208 day period.
- Use the Software Upgrade Test Utility to confirm your actual uptime.
- Prior to the 208 day period finishing, either reboot your nodes (one at a time, with a decent interval between them) or install a fixed level of software (as detailed in the alert).
To give you an example of the process, my lab machine is on software version 188.8.131.52 which you can see in the screen capture below. So when I check the table in the alert, I see that version 184.108.40.206 was made available on January 24, 2012, which means the 208 day period cannot possibly end before August 19, 2012.
|Version Number||Release Date||Earliest possible date that a system running this release could hit the 208 day reboot.|
SAN Volume Controller and Storwize V7000 Version 6.3
|220.127.116.11||30 November 2011||25 June 2012|
|18.104.22.168||24 January 2012||19 August 2012|
Regardless, I need to know the uptime of my nodes, so I download the Software Upgrade Test Utility (in case you have an older copy, we need at least version 7.9) and run it using the Upgrade Wizard (NOTE! We are NOT updating anything here, just checking):
I Launch the Upgrade Wizard, use it to upload the tool and follow the prompts to run it, so that I get to see the output of that tool. The output in this example shows the uptime of each node is 56 days, so I have a maximum of 152 days remaining before I have to take any action. At this point I select Cancel. You can run this tool as often as you like to keep checking uptime.
Note if you are on 6.1 or 6.2 code you may see a timeout error when running the tool, especially for the first time. If you do see an error, please follow the instructions in the section titled "When running the the upgrade test utility v7.5 or later on Storwize V7000 v6.1 or v6.2" at the Test Utility download site.
As per the Alert:
- If you are running a 6.0 or 6.1 level of firmware, you are not affected.
- If you are running a 6.2 level of firmware, the fix level is v22.214.171.124 which is available here for Storwize V7000 and here for SVC.
- If you are running a 6.3 level of firmware, the fix level is v126.96.36.199 which is available here for Storwize V7000 and here for SVC.
- If you are using a Storwize V7000 Unified, the fix level is v188.8.131.52 which is available here.
You should keep checking the alert to find out any new details as they come to hand. If you are curious about Linux and 208 day bugs, try this Google search.
*** Updated April 4, 2012 with links to fix levels ***
If you have any questions or need help, please reach out to your IBM support team or leave me a comment or a tweet.
*** April 10: The IBM Web Alert has been updated with new information on what to do if your uptime has actually gone past 208 days without a reboot. In short you still need to take action. Please read the updated alert and follow the instructions given there. ***
As a follow up to my previous blog, there are also distance considerations for longwave 8 Gpbs SFPs.
To provide some background, there are three things that control our ability to reliably send data over longer distances than 300 meters.
- The intensity (brightness) of the transmitter (for long distances this is normally a laser as opposed to an LED). The transmit value is measured in dBm. There is a concept called a link budget that is arrived at by adding the number of joins and the length of the fibre to determine if the Tx (transmit) value will fall below the minimum Rx (receive) value of the SFP at the receiving side. If it does, we will have an optical quality issue. If its too bright on the other hand, we will need a device called an optical attenuator to dim the light. There is a good Wikipedia article here: Link Budget
- The wavelength of the laser. Traditional longwave is 1310nm (that value effectively being the 'colour' of the light). For real longhaul (like WDM and CWDM SFPs) we use SFPs in the 1500-1550nm range. The main point is that the SFPs at each end of a link need to use the same wavelength, or they won't be able to communicate with each other. Wikipedia again: Optical variants
- The number of BB (buffer to buffer) credits available to the link, especially if this is a cross site Inter Switch Link (ISL). This is not a big issue for director class switches, but can be a major gotcha if your using small/midrange switches. If you have a Brocade switches and dont have the Extended Fabrics license, you could be in trouble if your link is more than 10km long. The bad news is that this license is not free. The good news is that Brocade can provide an evaluation license so you can test to see if purchasing the license will really help. Cisco doesn't need an equivalent license, but the 4 Gbps capable MDS9124 and MDS9134 are limited in the maximum number of buffers that can be dedicated to one port (there are only 64 buffers per group of 4 ports). At 2 Gbps you need to have at least 1 buffer per kilometer of fibre. At 4 Gbps you need 2 buffers per km and at 8 Gbps you need 4 buffers per km. The 8 GBps capable Cisco MDS9148 has 128 buffers per port group meaning one port at 8 Gbps could utilize 32 km of fibre. This great article is about FICON but its an excellent read: BB Credits and FICON
Standard Longwave versus Extended Longwave
When you see a 10km SFP versus a 25km longwave SFP , the main difference between them is that one uses a more intense (brighter) transmit signal than the other.
Without intending to indicate a vendor preference, I will use the specs of Brocades SFPs to make a point. You can visit the specifications page here:http://www.brocade.com/products-solutions/products/tranceivers/specifications.page
Compare this 8 Gbps capable 10km range SFP:http://www.brocade.com/forms/getFile?p=documents/data_sheets/product_data_sheets/SFP_8GB_LWL10Km_DS_00.pdf
With this 8 Gbps capable 25km range SFP:http://www.brocade.com/forms/getFile?p=documents/data_sheets/product_data_sheets/SFP_8GB_ELWL_DS_00.pdf
The 10km SFPs transmits between -8.4 and 0.5 dBm and needs to receive a signal of at least -15.4dBm.
The 25km SFPs transmits between 0 to 5.0 dBm and needs to receive a signal of at least -13.8dBm. The higher transmit value gives you a bigger link budget, which is why the SFP can go 15 extra km.
Remember you can increase these distance with WDM or CWDM technology, but you need to ensure the available BB credits on the ISL will be able to fully utilize it.
For Cisco switches check out the specs here:http://www.cisco.com/en/US/docs/switches/datacenter/mds9000/hw/9500series/installation/guide/techspec.htmlDisplaying actual transmit and receive values
The great thing is that both Cisco and Brocade switches use SFPs that report diagnostic information that can be displayed using the relevant management GUI.
In the screen capture below you can see the output from a shortwave length (850 nm) 4 Gbps capable SFP installed in an IBM R18 (Brocade Silkworm 7500) switch. Because this is a very short connection, the Rx level is very close to the Tx level.
In the screen capture below you can see the output from a CWDM long haul (1530 nm) 4 Gbps capable SFP installed in a Cisco MDS9509. This screen capture is more interesting because it shows that the Rx level is too low (which is why I got the screen capture in the first place - the link wasn't working!) So while its a good example of failed link, it doesn't give an example of seeing how close to reality your link budget calculations were.
SDDPCM (Subsystem Device Driver Path Control Module) is the multi-pathing plug-in for AIX 5.3 and 6.1.
Customers who use IBM SVC, DS6000, DS8000 and/or ESS800 use this package to allow the operating system to handle multiple paths from OS to storage.
The good news is that this plug-in is supplied free of charge. The bad news is that it is not included with AIX fixpacks.
What this means is that while you may be dilligent with keeping AIX up to date, you may miss SDDPCM in the process.
There are two good reasons to keep SDDPCM in mind when planning updates:
1) Planning an upgrade from AIX 5.3 to 6.1
Before the AIX OS is upgraded, SDDPCM must be uninstalled and then reinstalled after the upgrade. There are cases when the host attachment script must also be uninstalled and reinstalled. This is explained in the SDD Users Guide found here:
If you have already upgraded from AIX 5.3 to 6.1 but you are still using the AIX 5.3 version of SDDPCM, you may need help from IBM before you can upgrade your SDDPCM to the AIX 6.1 version. This will come in the form of some special scripts.
2) General SDDPCM maintenance
As I noted in my previous blog entry, there are quite a few SDDPCM flashes out there right now. You need to check these out and ensure you are not exposed to the issues that are corrected by later versions of SDDPCM. Check out the flashes listed here (or read my previous blog entry):
What about SDD (Subsystem Device Driver) for AIX?
Prior to AIX 5.2 FP5, AIX did not offer native multi-pathing (MPIO). This meant that each hardware vendor had to offer their own third-party software to handle multiple paths. To achieve this with the ESS (Shark), IBM released a product called DPO (Data Path Optimiser). This product became SDD and was made available for a wide variety of operating systems.
When AIX offered MPIO, IBM then also offered a vendor plug-in (Path Contol Module) for native AIX MPIO which IBM called SDDPCM. This means you have two choices with AIX: SDD or SDDPCM. If your considering which is best, SDDPCM is my preference. This is because it is native to the operating system and also better supports the possibility of co-existence of multiple PCMs. Note that migrating from SDD to SDDPCM is not supported by the VIOS at this time, so if your running VIOS you will need to stay put for now.
Its been a busy few weeks.
I just spent a week in RTP North Carolina, with the STG Education team.
We ran through our first "Implementing the Storwize V7000" course in a "Teach the Teacher" format.
It was a lot of fun and I met some great fellow IBMers.
It gave me a great opportunity to drive the Storwize V7000 GUI and explore all the new possibilities it opens up.
First up.... the GUI is fantastic. Don't be fooled by the XIV Icons, its the smarts behind what the GUI does that makes it so powerful.
Its a 21st Century GUI following very strong principles of usability and simplicity.
Talking to client after client about this product, I get lots of great questions.
Two questions I get asked on a regular basis about Storwize V7000 are:
1) What is the smallest number of SSDs I can purchase?
The answer is that you can purchase just one. However with one disk you don't get any RAID.
So its better to buy two SSDs for a RAID1 pair. If you buy three SSDs you can form a RAID5 array.
2) Will the Storwize V7000 enforce the creation of hot spares?
The answer is that the pre-sets that the GUI offers you, will suggest the creation of spares.
For every 23 array members with the same drive class on a single
SAS chain which are not RAID 0 members, a single spare is created.
However the GUI will also allow you great flexibility.
You can specify that a smaller or larger number of spares get created.
You can choose to create NO hot spares at all.
You can convert a hot spare drive into a candidate drive ( a 'free' drive).
You can convert a candidate drive into a hot spare drive.
You can set a 'spare goal' to set a minimum number of spares that need to exist (or an event will be logged).
So what you get is a great level of flexibility.
Either follow the pre-sets and get IBM best practice... or choose your own desired spare levels.
If you choose to create no spares using SSDs the Storwize V7000 will use spinning disks to rebuild a failed SSD.
Then when the failed SSD is replaced, the contents of that drive will be failed back.
*** Updated 25/07/2011: The VAAI plugin can be downloaded from here: http://www-933.ibm.com/support/fixcentral/swg/selectFixes?parent=ibm~Storage_Disk&product=ibm/Storage_Disk/IBM+Storwize+V7000+(2076)&release=6.2&platform=All&function=all ***
The May 9 announcement that SVC and Storwize V7000 will support VAAI is very welcome news. The fundamental point is that the SVC and Storwize V7000 virtualise external storage. This means that the mountains of DS3000, DS4000, DS5000, AMS1000s, CX3s, etc, that are currently being virtualized behind these products, will inherit VAAI as soon as the virtualization layer supports it. This is yet another feature to add to the list of functions that IBM Storage virtualization can provide, such as: EasyTier; Thin Provisioning; multiple consistency groups; snapshots; remote mirroring; dynamic data relocation... the list goes on.
In addition we are releasing a plug-in for vCenter that enables VMware administrators to manage their SVC or Storwize V7000 from within the VMware management environment
Functions will include:
- Volume provisioning and resizing
- Displaying information about volumes
- Viewing general information about Storwize V7000 and SVC systems
- Receiving events and alerts for Storwize V7000 systems and SVC attached to vSphere
- The Storwize V7000 and SVC plug-in for vCenter will also supports virtualized external disk systems
The plug-in will be available at no charge on June 30 (for Version 6.1 software) and July 31 (Version 6.2). Here is a sneak peak of what it will look like:
And to get an independent viewpoint have a read of Stephen Fosketts blog entry here:
So your planning to attach your XIV at an AIX host?
Here are some best practices for you to follow.
1) Native XIV detection
The XIV uses a path control module (PCM) that plugs into AIX MPIO. Depending on your AIX level the XIV will be recognised natively by AIX without additional software.
This is nice because it means you can simply run cfgmgr and detect the XIV hdisks without doing any system changes.
If your on the following AIX levels (with TL and SP) then your AIX system will detect the XIV natively. Frankly its a good excuse to perform a system update.
AIX Release APAR Bundled in
AIX 5.3 TL 10 IZ69239 SP 3
AIX 5.3 TL 11 IZ59765 SP 0
AIX 6.1 TL3 IZ63292 SP 3
AIX 6.1 TL 4 IZ59789 SP 0
If your running VIOS, you need to be on VIOS v2.1.2 FP22 to recognise the XIV natively.
Natively detected XIV devices will look like this when displayed using the command:
lsdev -Cc disk
03-00-02 MPIO 2810 XIV Disk
2) XIV Host attachment kit
If you are not on the levels listed above, you can install the XIV Host Attachment Kit to get XIV support.
However at lower AIX and VIOS levels there are issues with queue depth and round robin (its limited to 1)
The following releases do not have the queue depth issue, so they are better levels to be on:
AIX 5.3 TL 10 SP 0,1 and 2
AIX 6.1 TL 4 SP 0,1 and 2
VIOS v2.1.1.x FP-21.x
If your on level lower than those, you can still install the Host Attachment Kit to get XIV device support.
To detect XIV volumes when using the XIV Host Attachment Kit, you use the command xiv_attach.
The very first time you run xiv_attach you will need to reboot the host. After that you can use xiv_attach or cfgmgr (without reboot).
XIV devices detected by the xiv_attach command will look like this when displayed using the command:
lsdev -Cc disk
hdisk3 Available 02-01-02 IBM 2810XIV Fibre
3) The xiv_devlist command
Regardless of what level of AIX your running, you should install the Host Attachment Kit (HAK) to get the wonderful xiv_devlist command.
The HAK uses a specially packaged version of Python which is renamed XPYV (to not get in the way of any system Python already installed).
Just installing the kit does not require a reboot.
The xiv_devlist command is the equivalent of what SDD gave you with datapath query device.
It lets you map an AIX device (an hdisk) to an XIV volume. Its a tool you don't want to live without.
In the example below you can see the hdisk number on the left,
but all the other information (volume size, number of paths, volume name, XIV host) all come from the XIV itself.
This is really useful information.
root@system] # xiv_devlist
Device Size Paths Vol Name Vol Id XIV Id XIV Host
/dev/hdisk26 204.0GB 6/6 PROD-3050 188 7802844 PROD-prd
/dev/hdisk27 42.9GB 6/6 PROD-3051 189 7802844 PROD-prd
In my next blog entries I will tell you about zoning and what fcs, fscsi and hdisk variable work best with XIV.
I will also share a great way to update them.
For those of you who have worked with SVC, you will be aware of a quirk when determining the WWPN of an SVC fibre channel port.
of each fibre channel port is based on: 50:05:07:68:01:Yz:zz:zz where z:zz:zz is unique for each machine and the Y
value is taken from the port position (the red numbers in the boxes shown below).
numbers (used for servicing) are always sequentially numbered left to right
(viewed from the rear of the node). So the port numbers make sense.
for example port 1 the left hand most port, contains a 4,
so a WWPN presented by this port would look like: 50:05:07:68:01:4z:zz:zz (because the
number in each box is the Y value).
The Storwize V7000 consists of two node canisters .
The WWNN of each node canister will be based on 50:05:07:68:02:0z:zz:zz where z:zz:zz is
unique for each node canister. It is unrelated to the WWNN of the other node canister (they may be sequential numbers, they may not).
WWPN of each Storwize V7000 fibre port is based on: 50:05:07:68:02:Yz:zz:zz where z:zz:zz is unique for each node canister and the Y
value is taken from the port position.
Note that the lower canister is upside down in relation to the upper
canister (which you can see from the image below).
number in each black box (which represents a fibre channel port) is the Y value.
It is also the port number, This
means the Y
value and the port number are the same number. So
for example, port 1 (the upper left hand most port), contains a 1, so
a WWPN presented by this port would look like: 50:05:07:68:02:1z:zz:zz.
This makes more sense and fixes a 'feature' introduced with the 4F2 model of the SVC. Remember, that the z:zz:zz values for each node canister will be different (just like two SVC nodes had different z:zz:zz values for each node).
One common question that I hear on a regular basis regards the availability of an SRA for VMware SRM 5.0 when using Storwize V7000 or IBM SVC running V6.3 firmware. This combination is currently unsupported as per the alert found here.
The good news is that there are now IBM SRAs available for clients running SRM in combination with V6.3 firmware. While this combination is still not listed on the VMware support matrix found here, you can download the SRAs direct from IBM if your need is urgent.
The IBM download site is here:
For SRM 4.0 with a Storwize V7000 or SVC on V6.3 you can use:
For SRM 5.0 with a Storwize V7000 or SVC on V6.3 you can use:
For both links please check the FTP site to see if there is a later version.
Please also keep checking the VMware support matrix for the official support statement. I can see at least one example of someone using the latest SRA here.
I love PuTTY, it is one of my favourite pieces of open source software. For those who don't know what PuTTY is, it is a free implementation of Telnet and SSH for Windows and Unix platforms. Personally I use PuTTY for connecting to Storwize V7000s, SVCs, SAN switches and Unix servers in my lab and at clients.
Being someone who does implementation services, I want to always be sure about what I do, what I did and what affect it had. So keeping logs of every time I connect to any device is very important to me. I want to be able to go back in time to any point at any client I have ever worked with (sounds like time travel, but hopefully you get the idea).
I achieve this by changing the default behaviour of Putty so that every new Session I save uses my preferred default settings. Note all the screen captures below use version 0.61 of PuTTY (which came out July 2011).
First start up PuTTY. We are going to change the Default Settings (which I have highlighted):
Now select Logging from the left hand panel and change all the indicated settings (2, 3 and 4). The folder I use is C:\PuttyLogs but you could of course choose a different one. Note the &H_&Y&M&D&T means the session file name will be the host name, year, month, day and time for that session.
Now go back to Session, highlight Default Settings and hit Save:
Now every new session you open and/or save will automatically log your session output to the specified folder. Existing Saved Sessions will not be changed. You will need to update each of these separately.
The Storwize V7000 and SVC release 6.1 introduced a new WEB GUI interface to assist with service issues, known as the Service Assistant. The Service Assistant interface is a browser-based GUI that is used to service your nodes. Much of what you traditionally did with the SVC front panel can all be done using the Service Assistant GUI. You can see a screen capture of the Service Assistant below:
While I would like to be optimistic and hope that you will never have to use the Service Assistant, you should always ensure your toolkit is equipped with every possible tool. I say this because one thing I have noted is that the majority of installs are not configuring the Service Assistant IP addresses. This is particularly apparent as clients upgrade their SVC clusters to release 6.1.
By default on Storwize V7000, the Service Assistant is accessible on IP addresseshttps://192.168.70.121 for node 1 and https://192.168.70.122 for node 2 (don't try and point your browser at them right now, as your network routing won't work - you would need to set your laptop IP address to the same subnet and be on the same switch. Details to do that are here). For SVC there are no default IP addresses, although we traditionally asked the client to configure one service address per cluster. The best thing for you to do is approach your network admin and ask for two more IP addresses for each Storwize V7000 and/or SVC I/O group. Once you have these two extra IP addresses, record them somewhere and then set them using the normal GUI.
Its an easy five step process as shown in the screen capture below. Go to the Configuration group and then choose Network (step 1). From there select Service IP addresses (step 2) and the relevant node canister (step 3). Choose port one or port two (step 4) and then set the IP address, mask and gateway (step 5).
You can also set them using CLI (replace the word panelname with the panel name of each node, which you can get using the svcinfo lsnode command).
satask chserviceip -serviceip 10.10.10.10 -gw 10.10.10.1 -mask 255.255.255.0 panelname
If you forget these IP addresses, you can reset them using the same CLI commands or using the Initialization tool as documented here.
Finally having set the IP addresses, visit the service assistant by pointing your browser at each address. This is just to confirm you can access it. You logon with your Superuser password. With the process complete, ensure the IP addresses are clearly documented and filed away. So now if requested, you will be able to perform recovery tasks (in the unlikely chance they are needed). If for some reason your browser keeps bringing you to the normal GUI rather than the Service Assistance GUI, just add /service to the URL, e.g. browse to https://10.10.10.10/service rather than https://10.10.10.10.
So what should you do now?
If your an SVC customer on SVC code version v5 and below, please get two IP addresses allocated for each SVC I/O group, so you can set them the moment you upgrade to V6. Do this once the upgrade is complete.
If your an existing Storwize V7000 client or an SVC client already on V6.1 or V6.2 code, then hopefully you should already have set the service IP addresses. If not, please do so and test them.
Once your SVC or Storwize V7000 is upgraded to version 6.3 you can start using LDAP for authentication. This means that when you logon, you authenticate with your domain user-id and password rather than a locally created user-id and password.
So why is this important?
- It saves you having to configure every user on every SVC or Storwize V7000. If you have multiple machines this makes it far more efficient to set up authentication.
- It means that when commands are executed on the SVC or Storwize V7000, the audit log will show the domain username that issued that command, rather than a local username, or worse just superuser (i.e. who mapped that volume? The superuser did.... who? )
- It gives you central control over access. If someone leaves the company you just need to remove access at the domain controller, meaning there won't be orphan user-ids left on your Storage equipment.
So as an exercise I added my lab Storwize V7000 to our domain to show how it is done. This example also applies to an SVC so don't be confused if I only refer to Storwize V7000 from now on.
The first task is to negotiate with your Domain administrator to get a new group setup on the domain. In this example I use a group called IBM_Storage_Admins which lets me use this group for various storage devices (such as an XIV or a SAN Switch).
To create this group we need to logon to the Domain Controller and configure Active Directory. An easy way to do this from the AD controller is to go to Start → Run and type dsa.msc and hit OK. The Active Directory Users and Computers Management Console should open.
Select the groups icon to create a new group.
Enter your group name, in my case: IBM_Storage_Admins and hit OK.
Now right select relevant users who need access to the storage and add them to the IBM_Storage_Admins group. In this example I have selected Anthony (which uses anthonyv as a username).
In this example we are adding anthony into the IBM_Storage_Admins group:
Now it is time to configure the Storwize V7000 so start the Web GUI and logon as Superuser.
Firstly we go to Settings → Directory Services:
We choose the button to Configure Remote Authentication:
We choose LDAP and hit next.
We choose Microsoft Active Directory with no Transport layer Security. We then expand the Advanced Settings. My lab domain is ad.mel.stg.ibm so I use the Administrator ID on the Domain Controller to authenticate access. You could use any user that has authority to query the LDAP directory. We then hit Next.
We then add the domain controller which in this example is 10.1.60.50 and the base domain name chopped into pieces (so ad.mel.stg.ibm becomes dc=ad,dc=mel,dc=stg,dc=ibm ) and hit Finish.
Provided the command completes successfully we have defined the domain controller to the Storwize V7000. Now we need to add a group. Go to Access → Users.
Select the option to add a New User Group.
In this example we want to add a group for users allowed full admin access to the Storwize V7000. This matches the group we created on the Domain Controller. So we call the group IBM_Storage_Admins and we use the Security Administrator role (which is the most powerful role) and tick the box to enable LDAP for this group.
Now to test, I logon to the Storwize V7000 using the domain user-id anthonyv with that users domain password. Remember this user is not defined on the Storwize V7000 itself and that if it all goes wrong, we can still logon as Superuser.
Now I create a volume and delete it. Then I check the audit log from Access → Audit log.
Sure enough, we see exactly who did that command.
This is a great outcome for security,auditing and for easy access administration.
If you have issues, from the Settings → Directory Services menu, use the Global Actions dropdown on the right hand side to Test LDAP Connections and Authentication or re-configure LDAP.
If you already have existing users (what we call Local users), configuring remote authentication using LDAP does not disable or invalidate those local user-ids. This means you can either logon with a local user-id or logon with a Domain user-id. This is handy if the domain controller fails but can confuse you if your local user name and your domain user name are the same name (for example both anthonyv). The Storwize V7000 will look you up in the local user name list first. I suggest removing all local users (except superuser) as this will reduce confusion but still leave you a backdoor in case remote authentication stops working.
If you see any mistakes or have suggestions to improve the way I described this, please let me know.
With Brocade's recent announcement of a 16 Gbps capable Fibre Channel Switchand Director, the question of which cable type to purchase becomes even more relevant. Do you buy OM3 or OM4 cable over OM2?
Now if your saying... OM-what? Let me start at the beginning...
Back when fibre channel was fresh and new and ran at 1 Gbps, the common multi-mode fibre cable that we used had a glass core that was 62.5 microns in diameter. This became known as OM1 type fibre cable. We rapidly switched to 50 micron cores because you could get a reliable signal across a longer distance, say 500 meters maximum rather than 300 meters. The 50 micron cable became known as OM2 type cable.
What has happened since then is that fibre channel speeds have moved from 1 Gbps to 2 GBps to 4 Gbps to 8 Gbps to 16 Gbps. This is exciting stuff, but with every increase in speed, we suffer a decrease in maximum distance. This means that something else needs to change... and that something is the quality of the cables, or more specifically, the modal bandwidth (the signalling rate per distance unit).
With the evolution of 10 Gbps ethernet, the industry produced a new standard of fibre cable which the fibre channel world can happily use. Its called laser optimized cable, or more correctly: OM3. Since then OM3 has been joined by an even higher standard known as OM4.
Lets look at the distances we can achieve with different cable types. You can see in the table below that the modal bandwidth (given in MHz times kilometers), improves as we move to higher quality glass. You can also see that single mode fibre (with the 9 micron core) has not suffered the same issue with decreasing maximum distances as speeds have increased. These numbers come from Brocades SFP specification sheets found here andhere (so there may be slight variations if you view specs from other vendors).
I didn't fill in the table for 1 Gbps and 2 Gbps using OM4 cable, simply because I couldn't find it... but the distances would be very large indeed.
So how can you tell what sort of cable you have? The first hint is the colour, the second is the printing on the cable. Cables that are 50 micron and orange are almost certainly OM2. Cables that are aqua in colour (don't call them green!) are either OM3 or OM4. In the example below I can clearly tell which cable is OM3.
Pictured below is a roll of OM3 cable, all ready for deployment with standard LC connectors. Note you can also get OM3 cable with a smaller LC type connector used on the mSFPs in the high density 64 port blades in the Brocade DCX. You can find additional information on identifying cables here.
So should you be buying OM3 cable over OM2? Or even considering OM4?
The reality is that in many cases, server and storage hardware is often in the same or adjacent racks to the switch hardware. If this is true for your site, OM2 will satisfy the vast bulk of requirements, because the distances are quite short. The most common cable I add to configurations is either 5 or 25 meters long. This is why OM2 is still IBM's cable of choice, since either length would satisfy 16 Gbps connectivity. Checking with some local cable vendors, OM2 cable also remains the cheaper alternative.
Clearly if your computer room is large enough to need cable runs of over 35 meters, then serious consideration should be given to future proofing parts of your cable infrastructure with OM3 (or even OM4). There is nothing wrong with having a mix of cable types - just don't join them together.
I would be curious to know how many sites are choosing to move to OM3? Feel free to comment either way. I think there will be more to come on this subject, and remember.... OM3 and OM4 cables are aqua not green or blue. #.
Would love to hear about your sites recent cabling purchases.
And if the word Aqua reminds you of a late 90s Scandinavian pop group, look no further:
IBM have offered Brocade switch modules for their BladeCenters for many years. One common question I get asked regards which of these Switch Modules are supported with the IBM Storwize V7000 or IBM XIV.
Your first port of call is the System Storage Interoperation center or SSIC. However that will only list the switch modules by the IBM Feature Part Number (for example 26K5601), which may confuse you. So to help you determine which switch module you possess, here is a history of Brocade SAN switch modules for IBM BladeCenter:
2 Gbps Brocade Switch Modules for IBM BladeCenter
There were two switches 26K5601 and 90P0165 but they are actually the same switch with different software features. The Enterprise version had extra licenses like Trunking and Extended Fabrics. These products were withdrawn from marketing on May 26, 2006.
4 Gbps Brocade Switch Modules for IBM BladeCenter
IBM offered two models which were physically identical. You could upgrade from the 10 port to the 20 port with a software activation key. These products were withdrawn from marketing on December 31, 2010.
8 Gbps Brocade Switch Modules for IBM BladeCenter
IBM Product Name
Brocade 10-port SAN Switch Module for IBM eServer™ BladeCenter
Brocade 20-port SAN Switch Module for IBM eServer BladeCenter
IBM Feature Part Number
Brocade Product Number
Brocade Switch Type Numer
IBM FRU Number
Brocade ASIC Type
Minimum FOS Version
Maximum FOS Version
There are actually three models available but I have collapsed them down to two. The 20 port switch is offered as both Enterprise and non-Enterprise. The 42C1828 switch module is the Enterprise version that has more licensed software features such as Trunking, Fabric Watch and Extended Fabrics (and is indicated with an *).
IBM Product Name
Brocade 10-port 8Gb SAN Switch Module for IBM BladeCenter
Brocade 20-port 8Gb SAN Switch Module for IBM BladeCenter
IBM Feature Part Number
Brocade Product Number
Brocade Switch Type Numer
IBM FRU Number
Brocade ASIC Type
Minimum FOS Version
Maximum FOS Version
| || |
At the moment the SSIC does not list every switch module. However support is available for many configurations via the IBM SCORE request system (sometimes called an RPQ). Your IBM pre-sales storage specialist can raise one of these. Depending on your request you may get a support statement as quickly as overnight.
If your wondering what an IBM FRU number is: a FRU or Field Replacement Unit is the part number used in the IBM spare parts system to replace your part under warranty or maintenance agreement.
If you want the information above in a spreadsheet format, you can find it here.
If you have combined vSphere 5.0 with XIV, then you may want to try out the new IBM Storage Provider for VMware VASA (vSphere Storage APIs for Storage Awareness). You can download the installation instructions, the release notes and the current version of the IBM VASA provider from here. Clearly because VASA is introduced in vSphere 5.0 your VMware vCenter also needs to be on version 5.0.
Now IBM have had a vCenter plugin for a very long time (which I have written about here, here and here) and while you still need that plugin if you want to do storage volume creation and mapping from within vCenter (as opposed to using the XIV GUI), the VASA provider makes storage awareness more native to vCenter. This is a very important step. It means instead of using vendor added icons and tabs (like the IBM Storage icon and the IBM Storage tab that are added by the IBM Storage Management Console for vCenter), you just use the default vCenter tabs.
Right now version 1.1.1 of the IBM VASA provider delivers information about storage topology, capabilities, and state, as well as events and alerts to VMware. This means you will see new additional information in three tabs: Storage Views, Alarms and Events.
After installing and setting up the VASA provider, in vCenter select your VMware cluster, go to the Storage Views tab and select the view Show all SCSI Volumes (LUNs) there are four columns with more information. The Committed, Thin Provisioned information, Storage Array and Identifier on Array (indicated with red arrows) comes straight from the XIV (hit the Update button at upper right if you are not seeing anything yet). This is really useful information as it lets you correlate the SCSI ID of a LUN to an actual volume on a source array. Here is a cut-down view of that extra information:
If you want a larger screen capture you can find one here.
The Task & Events and Alarms tabs will also now contain events reported by the VASA provider such as thin provisioning threshold alerts (although if you have just installed the provider you may see nothing new, as nothing has occurred yet to provoke an alert or event).
As usual I have some handy tips on the steps you will need to take to get VASA going:
- First up you will need to identify a virtual machine to run the provider on (or just create a new one). I chose to deploy a new instance of Windows 2008 from a template. Because the VASA provider communicates to vCenter via an Apache Tomcat server listening on port 8443, that port needs to be free and unblocked. This also means you should not run the VASA provider in the same instance of Windows as the vCenter server (see below for more information as to why).
- Download the IBM Storage Provider for VMware VASA as per the link above (use version 1.1.1, see the user comments in this post for details about a bug in version 1.1.0).
- Install the provider in the Windows VM you created in step 1. The tasks are detailed in the Installation Instructions, but it is a simple follow-your-nose application installation. As per most XIV software packages, it will install a runtime environment (xPYV which is Python) as part of the install.
- Now we need to define the credentials that VMware vCenter will use to authenticate to the IBM VASA Storage Provider. These should be unique (and are not an XIV userid and password - this is only between vCenter and the provider software). In my example I use xivvasa and pa55w0rd. The truststore password is used to encrypt the username and password details (so that they are not stored in plain text). Open a Windows command prompt (make sure to right select and open it as an Administrator) and enter the following commands:
cd "C:\Program Files (x86)\IBM\IBM Storage Provider for VMware VASA\bin"
vasa_util register -u xivvasa -p pa55w0rd -t changeit
Don't close the command prompt, because we now need to define the XIV to the IBM VASA provider.
- You need the IP address of your XIV and a valid user and password on the XIV that can be used to logon to the XIV. So in this example my XIV is using 10.1.60.100 and I am using the default admin username and password (which I know does not set a good example). This is the command you need to run:
vasa_util add -i 10.1.60.100 -u admin -p adminadmin
If this command fails, reporting your firmware is invalid, you are probably using the original 1.1.0 version of the VASA provider, go back to the IBM Fix Central website and make sure you have the latest version (at least version 1.1.1). If it reports the firmware cannot be read, make sure you are running the Command Prompt as an Administrator.
- Once you successfully added the XIV to the provider, you need to restart the Apache webserver. Do this by starting the services.msc panel and looking for the Apache Tomcat IBMVASA service as pictured below. Stop it and then start it. Once you have done that you can logoff from the VASA VM.
- Now connect to your vSphere Client (which needs to be on at least version 5.0.0) and from the Home panel, open the Storage Providers panel.Then select the option to Add a new provider. The URL needs to include the correct port number (by default 8443), so it will look something like this (where the provider is running on 10.1.60.193). Note also that the VASA provider version number is in the URL, so if you upgrade the provider you will need to change the URL (currently v1.1.1):
The Login and password should match the user id and password you defined in step 4 (remember it is not logging into the XIV, it is logging into the VASA provider).
- If you get a message saying your user id and password are wrong, you probably forgot to stop and start Apache in step 6 above. If you succeed you should see a new provider listed. Highlight the provider and select sync to update the last sync time.
Your setup tasks are now all completed. Now go and explore the panels I detailed above to see what new information you have available to your vCenter server.
Why a separate server for the VASA provider?
The IBM VASA provider uses Apache Tomcat, which by default listens on port 8443. However since vCenter already has a service listening on port 8443, it means we have a clash. I googled and found the Dell and Netapp VASA providers also listen on port 8443 and they also recommend separate servers. I noted Fujitsu's provider uses a different port but still requires a separate server. So it seems if you have multiple vendors you will either have to spin up a separate server for each vendors provider, or start playing with changing the port number. The installation instructions for the IBM VASA Provider explain how to change the default port number if you are truly keen.
One of the more common themes I blog about is the demand for Visio Stencils. In this regard I have some more good news.
Firstly the VisioCafe site has an updated IBM-Disk stencil set with the following additions:
IBM-SystemStorage-Disk.vss - Added EXP5060 Front, Front Open and Rear Views
- Added DS8x00 Cabinet Rear Views
Secondly, I have started using the collections function on the IBM developerWorks site to share my files with the public. To this end I have posted a collection of all the standard Visio documents I use to build XIV Solution Designs. You can find them here.
They tend to use two basic building blocks, which are the Fibre Channel patch panel and the iSCSI patch panel. For instance this is what the iSCSI patch panel Visio diagram looks like:
And this is an example of what a Fibre Channel Visio diagram looks like.
Your free to use any of my diagrams for any purpose you like. If you have suggestions, comments or donations, please feel free to share. The link again is here.