IBM AIX SAN Volume Controller update and migration
This article discusses the tasks to be performed on AIX when updating or migrating to a new IBM SVC release. You might be updating from SVC V6.2.x to SVC V6.4.x or you might be migrating from SVC V6.4.x to a new release, SVC V7.2.x. This article discusses migrating only SAN logical unit numbers (LUNs) or disks and do not concern tape drives. The article covers the process of migrating one AIX server and provides a script that can save you time in gathering the information that you need specially if you have many AIX servers to update or migrate. The information in this article can also be used when using other SAN beside IBM. They can also be used if you are updating or upgrading your SAN Brocade Switches as well. You can get the general ideas of what you need to do and how to do it.
Tasks to be performed before migration
If the SVC is being updated or upgraded, the first few things that you have to worry about are SDD, Subsystem Device Driver Path Control Module (SDDPCM), and your host bus adapter (HBA) firmware. In this article, I am covering only about SDDPCM because it is the most popular. Both SDDPCM and HBA are used to access the SAN LUNs or disks and have to be compatible with the new update or release of SVC. Along with the SAN team, you have to come up with a list of AIX servers that need SDDPCM and/or HBA firmware updated or upgraded. This is the first task you have to do in this project.
SDDPCM and host script update
You need to update both SDDPCM and the host attachment script. Here are the commands to find the current release for both of them:
=> lslpp -l|grep sdd devices.sddpcm.61.rte 184.108.40.206 COMMITTED IBM SDD PCM for AIX V61
=> lslpp -L devices.fcp.disk.ibm.mpio.rte|grep MPIO 220.127.116.11 C F IBM MPIO FCP Disk Device
From the above output, you can tell that SDDPCM is at level 18.104.22.168 and the host attachment script is at 22.214.171.124.
Let us say the SAN team is migrating to SVC 6.4.x. Then, you need to find the compatibility hardware list. Search in Google for svc 6.4 supported hardware list. In the search results, click IBM V6.4.x Supported Hardware List, Device Drivers. Scroll down the page and in the Hosts section, click IBM AIX Fibre Channel. A table as shown in Figure 1 is displayed.
Figure 1. SVC compatibility matrix
This is the SVC compatibility matrix. Here is how you can read this matrix. The first thing you have to keep in mind when you look at this compatibility matrix is SDDPCM. It is not AIX or IBM PowerHA® and so on that you are looking for. You are looking at this table for only one reason and that is to determine the SDDPCM release required for your AIX server. Everything else in the table is designed to help you make that determination. This table works as follows: If you have one of those, and one of those, and one of those, than you need one of those. The one of those that you need in this case is of course the SDDPCM release for your AIX server. If you look at the row highlighted in red, then you read it as follows. If you have AIX 5.3 TL12 (one of those), and you are not booting from the SAN (one of those), and you have PowerHA release 6.1 (one of those) then you need SDDPCM 126.96.36.199 (one of those). If your current SDDPCM is at 188.8.131.52, then you don't have to update it, and if lower, then you have to update it. If I have a choice of three SDDPCM releases, I usually go with the latest release (instead of going with a lower one and then ending up updating to the latest release). To download SDDPCM 184.108.40.206, click it under the Multipath Driver column. Scroll down to the Download package section, and within the section, scroll down to the SDDPCM Package for SAN Volume Controller (SVC) heading.
Look for the row with AIX 5.3 and SDDPCM 220.127.116.11. You may need to download the prerequisite which is SDDPCM 18.104.22.168 if yours is below that level and then download 22.214.171.124 under the DOWNLOAD PTF column.
If you have to update your SDDPCM, then you have to update your host attachment script as well. Let us say you have to update your SDDPCM to 126.96.36.199, then you need to find the corresponding release for the host attachment script. In order to find that, you can search in Google for "IBM Host Attachment for SDDPCM on AIX" and you would find "IBM Host Attachment for SDDPCM on AIX" as one of the search results.
Click the link and scroll down to the Download package section. In the Description column, you can find many Platform AIX entries. Look for your AIX release (which is 5.3 in this case) and SVC because you are migrating to a new SVC (in this case, it is release v188.8.131.52). You have three choices for AIX 5.3 but the first two do not include SVC (look under the LABEL column). Click HTTP corresponding to this entry under the Download Options column and you can save the download to your PC.
In summary, you need to update SDDPCM to release 184.108.40.206 and your host attachment
script to v220.127.116.11. After downloading, you can use the
update_all command to install them. You need to reboot your system for
the update to take effect. It is recommended to reboot as soon as possible.
HBA firmware update
To make sure that your HBA firmware works with the upgraded SVC, your best bet is to
make sure that you have the latest firmware level for your HBA. To get the latest
firmware level for your HBA, you need to get the feature code (FC) for it. You can
use the FC in IBM Fix Central and it gives you all the details for your HBA,
including the latest firmware level for it. You can use the
lscfg -vl fcs0 command to get the FC but most of the time, it may not be
provided in the output. Instead, you can use the customer card ID number (CCIN) to
actually find the FC. Here is the output of this command:
=> lscfg -vl fcs0 fcs0 U5791.001.99B0D90-P2-C05-T1 FC Adapter Part Number.................03N5029 EC Level....................A Serial Number...............1D61808067 Manufacturer................001D Customer Card ID Number.....5759 FRU Number..................03N5029 Device Specific.(ZM)........3 Network Address.............10000000C954BBE0 ROS Level and ID............02C82774 Device Specific.(Z0)........1036406D Device Specific.(Z1)........00000000 Device Specific.(Z2)........00000000 Device Specific.(Z3)........03000909 Device Specific.(Z4)........FFC01231 Device Specific.(Z5)........02C82774 Device Specific.(Z6)........06C12715 Device Specific.(Z7)........07C12774 Device Specific.(Z8)........20000000C954BBE0 Device Specific.(Z9)........BS2.71X4 Device Specific.(ZA)........B1F2.70A5 Device Specific.(ZB)........B2F2.71X4 Device Specific.(ZC)........00000000 Hardware Location Code......U5791.001.99B0D90-P2-C05-T1
In this case, the customer card ID number is 5759. After finding the FC, you can download the latest firmware from IBM Fix Central and get ready to install it.
I like to use Z9 from the output above to find the current firmware level of the HBA,
which in this case is 2.71X4 (you drop off the BS in this case).
Sometimes, this is not provided. In this case, you can always use the "lsmcode -cd
fcs0" command to find the current firmware level. Here is the output of the
=> lsmcode -cd fcs0
The current microcode level for fcs0 is 271304.
My script gets both values of Z9 and from
lsmcode. If Z9
is not good, then use the output from
If the FC is not provided by the
lscfg command as
indicated above, then use the CCIN to get the FC value. Visit the IBM Knowledge Center for more information about it.
In IBM Knowledge Center, in the Search Field in the upper-left corner, type the CCIN you have and press Enter. Let us say the CCIN in this case is 1910. In the search results, look for CCIN 1910 and that will be the HBA model you are looking for. Click 4 Gb Dual-Port Fibre Channel PCI-X 2.0 DDR adapter (FC 1910, 5759;CCIN 1910, 5759) to find more information about it.
Here, (FC 1910, 5759; CCIN 1910, 5759) means that CCIN 1910 has an FC of 1910 and CCIN 5759 has an FC of 5759. Most of the time, the CCIN and the FC values are the same, but not always. Now that you have the FC, you need to perform the following tasks in IBMFix Central to get the latest firmware for your HBA.
- On the Select Product tab, from the Product Group drop-down list, select System P.
- From the Product drop-down list, select Firmware and HMC.
- Retain the processor type as POWER4 and earlier.
- From the Firmware Type drop-down list, select System and device firmware.
- Click Continue.
- You will get to a new page with the 3) Search for Device Firmware by Feature Code option allowing to enter the FC to get you latest firmware and download it.
- Enter 1910 in the text field and press Enter.
Figure 2. Firmware download
- Note the first column that lists RPM and
RPM(Linux). We are not interest in Linux. So, in the
RPM row, click Desc. A page with the
following information at the top is displayed.
Microcode Level df1000fd-0002.271304 (2.71x4) with FCode Level 1.50x1 for FC 5759, 1910
Microcode Level df1000fd-002.271310 (2.71x10) with FCode Level 1.50x1 for FC 5759, 1910
The first line has the string "df1000fd-0002.271304 (2.71x4)".
"271304" refers to the latest firmware level (normally derived from the
lsmcode -cd fcs0 command ) and "2.71x4" also refers to the
latest firmware level (this stands for the Z9 value that we discussed earlier). If
you want to map 271304 to 2.71x4, you have to remove the period
and the x stands for 30 in this case.
You can become familiar with these two values as you work with them. The second line in the above output refers to "271210" and "2.71x4". Concerning this, here is what IBM says in the Overview section of this page.
Figure 3. Overview section
It is recommended to read this entire page and become familiar with the information in it. For example, if you are updating the firmware, you might have some APAR as requirement and on a Virtual Input/Output Server (VIOS) instance, you may have some fix packs as a requirements. You need to be careful with that.
The next step is to download your firmware. Go to the previous page on your website and perform the following tasks.
1. Select the RPM check box [and not RPM(linux)] and click Continue.
2. Click Continue again.
3. Select the I agree to abide by this terms check box and click Continue.
Now you are on the download page. You can either click HTTP or FTP and save the RPM in your PC. The name of the download in this case is df1000fd-0002-271304.aix.rpm. Note that 271304 refers to the firmware level. You need to get it on your AIX server. The following section provides instructions to install it in your system.
Installing the latest firmware for your HBA
Now that you downloaded the firmware, df1000fd-0002-271304.aix.rpm, you can install
it using the
diag command. Assuming that you have saved
df1000fd-0002-271304.aix.rpm in /tmp, you need to run the following commands to
install this firmware update.
1. mkdir /etc/microcode 2. mv /tmp/df1000fd-0002-271304.aix.rpm /etc/microcode 3. rpm -ihv --ignoreos df1000fd-0002-271304.aix.rpm 4. diag -d fcs0 -T download
The first command creates the directory /etc/microcode if you do not already have it. The second command moves the firmware from /tmp to /etc/microcode. The third command unpacks the files and adds them to /etc/microcode in preparation for the microcode install. The fourth step updates or upgrades the firmware for HBA fcs0.
Here are the steps you go through for the
diag command in
step 4 done on AIX 6.1 TL6.
1. Type "
diag -d fcs0 -T download" and press Enter.
2. Notice that fcs0 appears at the top and NOTICE at the bottom of this page. Press Enter.
3. Select /etc/microcode and press Enter.
4. The current microcode level is displayed at the top. At the bottom, select the microcode you want to install and press Enter.
The installation of the microcode begins now. It might take sometime to complete. It is recommended to perform this update during non-peak production periods. You don't need to stop other applications for this update.
Use the following commands to verify that the firmware had been updated successfully.
lsmcode -cd fcs0
lscfg -vl fcs0
To restore the previous firmware level, use the following command:
diag -d fcs0 -T "download -f -l previous"
SAN system administrator information
The SAN system administrators would appreciate it, if you give them the output from the following three commands (the output from each command is displayed).
1. => pcmpath query wwpn
This command provide the SAN SA with the WWPN of the HBA.
2. => pcmpath query adapter
Total Dual Active and Active/Asymmetric Adapters : 2
This command provide the SAN SA with the number of Paths.
3. => pcmpath query device
DEV#: 10 DEVICE NAME: hdisk10 TYPE: 2145 ALGORITHM: Load Balance
The third command could have many LUNs displayed, but I only have one displayed as an example. This command provides the SAN system administrator with all the SAN LUNs and the paths for each LUN.
Commands to run before SVC migration
You need to run the following commands before migrating SVC.
pcmpath query adapter >
pcmpath query device >
pcmpath query wwpn >
lspath > /home/user/lspath_before_052014
lspath | grep -v Enabled >
You might have to refer to the output of these commands after the SAN team update or migrate their SVC. This is the before output and you will get the after output after the SVC is updated or migrated. You can than compare the before and after output to make sure that everything looks good. This shows you if there are some zoning problems and if everything looks good after the migration.
Commands to run on the day of migration immediately after updating SVC
=> pcmpath query adapter
Total Dual Active and Active/Asymmetric Adapters : 2
The output from the above command is crucial. Pay particular attention to the number of "Paths" and "Active" which are in this case 80 and 72. Ideally, they should be the same numbers as before SVC was migrated. If the Paths number is not the same, then you might have some zoning issues. For example, if the Paths number before SVC migration is 80 and after migration is 64, then there are 16 Paths missing.
Review the output of the following command in this case:
=> pcmpath query device
DEV#: 4 DEVICE NAME: hdisk10 TYPE: 2145 ALGORITHM: Load Balance
Review this command before and after the migration. Here, I have provided only one entry, but you will have many. You might find that before migration, every hdisk had four paths as indicated above, two for fscis1 and two for fscsi3. Now, review the output of this command after the SVC migration. You might find that some hdisks have two paths instead of four. This would account for the missing paths. Pass the output to the SAN team and let them know that some paths are missing. You can pass them the output before and after of this command and they can find and fix the problem. In this case, the problem is with zoning and the SAN team can fix it.
The reason why we have 80 paths and only 72 are active according to the output above
is that because fscsi1 has two hdisks that are not assigned to a volume group (VG)
and each one of them has four paths and a total of 8 paths. These disks are not
active means that they have not been assigned to a VG and are not being
used. These hdisks will show in the
pcmpath query device
command as CLOSE NORMAL for STATE and MODE as opposed to OPEN NORMAL, which means
that they have been assigned to a VG and are being used.
I also would like to check the number of enabled paths before and after and make sure
that I have the same count. You can also browse quickly through the output from
before and after the
lspath | grep Enabled | wc -l (get a
count of the enabled paths. Should be same as before)
I also like to browse all the paths that are not enabled from before or after. You may have to deal with some problems here, but not likely, if everything else looked good.
lspath | grep -v Enabled (look for all
the paths that are not enabled)
You can use this script to get a report for many AIX servers with their SDDPCM and host attachment script releases along with their CCIN. It includes more useful info. You need to run it on a jump server (meaning an AIX server) from which you can use Secure Shell (SSH) to all AIX servers from where you want to get the report without having to enter the password. You also have to be the root user to run it. Refer to the script and its output.
The output of the script is in the comma separated value (CSV) format which means that you can copy it and paste it in a Microsoft Excel sheet. I usually like to highlight the label for all the fields. Here are the labels:
AIX SERVER, POWER, S/N, MODEL TYPE, AIX RELEASE, IOSLEVEL, SDDPCM, HAS, SAN DISK HBA, Z9, CURRENT FIRMWARE, LATEST FIRMWARE, CCIN, and FC.
See Table 1 to find the details.
Most of the labels in this table, such as "AIX RELEASE" are self explanatory.
"IOSLEVEL" will be reported if the server is a VIOS instance. HAS stands for host
attachment script. The label, "SAN DISK HBA", refers to fcs derived from the
pcmpath query adapter command. You have to fill in the FC
field and the LATEST FIRMWARE field as well. I have created a field in the Excel
sheet right after the label "SDDPCM" and labeled it as "UPDATE TO". If the current
SDDPCM release is fine, then I enter OK in this field. If it needs to be updated,
then I would enter the new release to which it needs to be updated and then I would
highlight the SDDPCM release and the "UPDATE TO" release to indicate that update or
upgrade is needed for this server. I would also highlight the release for "LATEST
FIRMWARE" if it needs to be updated as well.