IBM AIX SAN Volume Controller update and migration

This article discusses the tasks to be performed for IBM® AIX® when updating or migrating IBM System Storage® SAN Volume Controller (SVC) to a new release. It includes a script that makes it easier to prepare many AIX servers for update or migration.

Christopher Narcouzi (cnarcouz@gmail.com), Senior AIX Systems Administrator, IBM

Photo of Christopher Narcouzi Christopher is a Senior AIX System Administrator with 21 years of experience with AIX. He has been writing scripts in Korn shell (ksh) and Perl for many years now. He has also written some Expect scripts and has three scripts published on www.mtxia.com. One of them is to create a VIOS instance while another is to reset the passwords for many users on many AIX servers without having to provide the password.



19 June 2014

Introduction

This article discusses the tasks to be performed on AIX when updating or migrating to a new IBM SVC release. You might be updating from SVC V6.2.x to SVC V6.4.x or you might be migrating from SVC V6.4.x to a new release, SVC V7.2.x. This article discusses migrating only SAN logical unit numbers (LUNs) or disks and do not concern tape drives. The article covers the process of migrating one AIX server and provides a script that can save you time in gathering the information that you need specially if you have many AIX servers to update or migrate. The information in this article can also be used when using other SAN beside IBM. They can also be used if you are updating or upgrading your SAN Brocade Switches as well. You can get the general ideas of what you need to do and how to do it.


Tasks to be performed before migration

If the SVC is being updated or upgraded, the first few things that you have to worry about are SDD, Subsystem Device Driver Path Control Module (SDDPCM), and your host bus adapter (HBA) firmware. In this article, I am covering only about SDDPCM because it is the most popular. Both SDDPCM and HBA are used to access the SAN LUNs or disks and have to be compatible with the new update or release of SVC. Along with the SAN team, you have to come up with a list of AIX servers that need SDDPCM and/or HBA firmware updated or upgraded. This is the first task you have to do in this project.

SDDPCM and host script update

You need to update both SDDPCM and the host attachment script. Here are the commands to find the current release for both of them:

=> lslpp -l|grep sdd
devices.sddpcm.61.rte 2.6.4.0 COMMITTED IBM SDD PCM for AIX V61 
=> lslpp -L devices.fcp.disk.ibm.mpio.rte|grep MPIO 1.0.0.24 C F IBM MPIO FCP Disk Device

From the above output, you can tell that SDDPCM is at level 2.6.4.0 and the host attachment script is at 1.0.0.24.

Let us say the SAN team is migrating to SVC 6.4.x. Then, you need to find the compatibility hardware list. Search in Google for svc 6.4 supported hardware list. In the search results, click IBM V6.4.x Supported Hardware List, Device Drivers. Scroll down the page and in the Hosts section, click IBM AIX Fibre Channel. A table as shown in Figure 1 is displayed.

Figure 1. SVC compatibility matrix

This is the SVC compatibility matrix. Here is how you can read this matrix. The first thing you have to keep in mind when you look at this compatibility matrix is SDDPCM. It is not AIX or IBM PowerHA® and so on that you are looking for. You are looking at this table for only one reason and that is to determine the SDDPCM release required for your AIX server. Everything else in the table is designed to help you make that determination. This table works as follows: If you have one of those, and one of those, and one of those, than you need one of those. The one of those that you need in this case is of course the SDDPCM release for your AIX server. If you look at the row highlighted in red, then you read it as follows. If you have AIX 5.3 TL12 (one of those), and you are not booting from the SAN (one of those), and you have PowerHA release 6.1 (one of those) then you need SDDPCM 2.6.3.2 (one of those). If your current SDDPCM is at 2.6.3.2, then you don't have to update it, and if lower, then you have to update it. If I have a choice of three SDDPCM releases, I usually go with the latest release (instead of going with a lower one and then ending up updating to the latest release). To download SDDPCM 2.6.3.2, click it under the Multipath Driver column. Scroll down to the Download package section, and within the section, scroll down to the SDDPCM Package for SAN Volume Controller (SVC) heading.

Look for the row with AIX 5.3 and SDDPCM 2.6.3.2. You may need to download the prerequisite which is SDDPCM 2.6.3.0 if yours is below that level and then download 2.6.3.2 under the DOWNLOAD PTF column.

If you have to update your SDDPCM, then you have to update your host attachment script as well. Let us say you have to update your SDDPCM to 2.6.3.2, then you need to find the corresponding release for the host attachment script. In order to find that, you can search in Google for "IBM Host Attachment for SDDPCM on AIX" and you would find "IBM Host Attachment for SDDPCM on AIX" as one of the search results.

Click the link and scroll down to the Download package section. In the Description column, you can find many Platform AIX entries. Look for your AIX release (which is 5.3 in this case) and SVC because you are migrating to a new SVC (in this case, it is release v1.0.0.24). You have three choices for AIX 5.3 but the first two do not include SVC (look under the LABEL column). Click HTTP corresponding to this entry under the Download Options column and you can save the download to your PC.

In summary, you need to update SDDPCM to release 2.6.3.2 and your host attachment script to v1.0.0.24. After downloading, you can use the smitty update_all command to install them. You need to reboot your system for the update to take effect. It is recommended to reboot as soon as possible.

HBA firmware update

To make sure that your HBA firmware works with the upgraded SVC, your best bet is to make sure that you have the latest firmware level for your HBA. To get the latest firmware level for your HBA, you need to get the feature code (FC) for it. You can use the FC in IBM Fix Central and it gives you all the details for your HBA, including the latest firmware level for it. You can use the lscfg -vl fcs0 command to get the FC but most of the time, it may not be provided in the output. Instead, you can use the customer card ID number (CCIN) to actually find the FC. Here is the output of this command:

 => lscfg -vl fcs0 fcs0 U5791.001.99B0D90-P2-C05-T1 FC Adapter
                Part Number.................03N5029
                EC Level....................A
                Serial Number...............1D61808067
                Manufacturer................001D
                Customer Card ID Number.....5759
                FRU Number..................03N5029
                Device Specific.(ZM)........3
                Network Address.............10000000C954BBE0
                ROS Level and ID............02C82774
                Device Specific.(Z0)........1036406D
                Device Specific.(Z1)........00000000
                Device Specific.(Z2)........00000000
                Device Specific.(Z3)........03000909
                Device Specific.(Z4)........FFC01231
                Device Specific.(Z5)........02C82774
                Device Specific.(Z6)........06C12715
                Device Specific.(Z7)........07C12774
                Device Specific.(Z8)........20000000C954BBE0
                Device Specific.(Z9)........BS2.71X4
                Device Specific.(ZA)........B1F2.70A5
                Device Specific.(ZB)........B2F2.71X4
                Device Specific.(ZC)........00000000
                Hardware Location Code......U5791.001.99B0D90-P2-C05-T1

In this case, the customer card ID number is 5759. After finding the FC, you can download the latest firmware from IBM Fix Central and get ready to install it.

I like to use Z9 from the output above to find the current firmware level of the HBA, which in this case is 2.71X4 (you drop off the BS in this case). Sometimes, this is not provided. In this case, you can always use the "lsmcode -cd fcs0" command to find the current firmware level. Here is the output of the lsmcode command.

=> lsmcode -cd fcs0 
The current microcode level for fcs0 is 271304.

My script gets both values of Z9 and from lsmcode. If Z9 is not good, then use the output from lsmcode.

If the FC is not provided by the lscfg command as indicated above, then use the CCIN to get the FC value. Visit the IBM Knowledge Center for more information about it.

In IBM Knowledge Center, in the Search Field in the upper-left corner, type the CCIN you have and press Enter. Let us say the CCIN in this case is 1910. In the search results, look for CCIN 1910 and that will be the HBA model you are looking for. Click 4 Gb Dual-Port Fibre Channel PCI-X 2.0 DDR adapter (FC 1910, 5759;CCIN 1910, 5759) to find more information about it.

Here, (FC 1910, 5759; CCIN 1910, 5759) means that CCIN 1910 has an FC of 1910 and CCIN 5759 has an FC of 5759. Most of the time, the CCIN and the FC values are the same, but not always. Now that you have the FC, you need to perform the following tasks in IBMFix Central to get the latest firmware for your HBA.

  1. On the Select Product tab, from the Product Group drop-down list, select System P.
  2. From the Product drop-down list, select Firmware and HMC.
  3. Retain the processor type as POWER4 and earlier.
  4. From the Firmware Type drop-down list, select System and device firmware.
  5. Click Continue.
  6. You will get to a new page with the 3) Search for Device Firmware by Feature Code option allowing to enter the FC to get you latest firmware and download it.
  7. Enter 1910 in the text field and press Enter.

    Figure 2. Firmware download

  8. Note the first column that lists RPM and RPM(Linux). We are not interest in Linux. So, in the RPM row, click Desc. A page with the following information at the top is displayed.

    Microcode Level df1000fd-0002.271304 (2.71x4) with FCode Level 1.50x1 for FC 5759, 1910
    and
    Microcode Level df1000fd-002.271310 (2.71x10) with FCode Level 1.50x1 for FC 5759, 1910

The first line has the string "df1000fd-0002.271304 (2.71x4)". "271304" refers to the latest firmware level (normally derived from the lsmcode -cd fcs0 command ) and "2.71x4" also refers to the latest firmware level (this stands for the Z9 value that we discussed earlier). If you want to map 271304 to 2.71x4, you have to remove the period and the x stands for 30 in this case.

You can become familiar with these two values as you work with them. The second line in the above output refers to "271210" and "2.71x4". Concerning this, here is what IBM says in the Overview section of this page.


Figure 3. Overview section

It is recommended to read this entire page and become familiar with the information in it. For example, if you are updating the firmware, you might have some APAR as requirement and on a Virtual Input/Output Server (VIOS) instance, you may have some fix packs as a requirements. You need to be careful with that.

The next step is to download your firmware. Go to the previous page on your website and perform the following tasks.

1. Select the RPM check box [and not RPM(linux)] and click Continue.

2. Click Continue again.

3. Select the I agree to abide by this terms check box and click Continue.

Now you are on the download page. You can either click HTTP or FTP and save the RPM in your PC. The name of the download in this case is df1000fd-0002-271304.aix.rpm. Note that 271304 refers to the firmware level. You need to get it on your AIX server. The following section provides instructions to install it in your system.


Installing the latest firmware for your HBA

Now that you downloaded the firmware, df1000fd-0002-271304.aix.rpm, you can install it using the diag command. Assuming that you have saved df1000fd-0002-271304.aix.rpm in /tmp, you need to run the following commands to install this firmware update.

1. mkdir /etc/microcode
2. mv /tmp/df1000fd-0002-271304.aix.rpm /etc/microcode
3. rpm -ihv --ignoreos df1000fd-0002-271304.aix.rpm
4. diag -d fcs0 -T download

The first command creates the directory /etc/microcode if you do not already have it. The second command moves the firmware from /tmp to /etc/microcode. The third command unpacks the files and adds them to /etc/microcode in preparation for the microcode install. The fourth step updates or upgrades the firmware for HBA fcs0.

Here are the steps you go through for the diag command in step 4 done on AIX 6.1 TL6.

1. Type "diag -d fcs0 -T download" and press Enter.

2. Notice that fcs0 appears at the top and NOTICE at the bottom of this page. Press Enter.

3. Select /etc/microcode and press Enter.

4. The current microcode level is displayed at the top. At the bottom, select the microcode you want to install and press Enter.

The installation of the microcode begins now. It might take sometime to complete. It is recommended to perform this update during non-peak production periods. You don't need to stop other applications for this update.

Use the following commands to verify that the firmware had been updated successfully.

1. lsmcode -cd fcs0

2. lscfg -vl fcs0

To restore the previous firmware level, use the following command:

 diag -d fcs0 -T "download -f -l previous"

SAN system administrator information

The SAN system administrators would appreciate it, if you give them the output from the following three commands (the output from each command is displayed).

1. => pcmpath query wwpn

Adapter Name PortWWN
fscsi0 10000000C94E6119
fscsi1 10000000C94E68CA

This command provide the SAN SA with the WWPN of the HBA.

2. => pcmpath query adapter

Total Dual Active and Active/Asymmetric Adapters : 2

Adpt# Name State Mode Select Errors Paths Active
0 fscsi0 NORMAL ACTIVE 433837731 15 40 40
1 fscsi1 NORMAL ACTIVE 434324564 54 40 40

This command provide the SAN SA with the number of Paths.

3. => pcmpath query device

DEV#: 10 DEVICE NAME: hdisk10 TYPE: 2145 ALGORITHM: Load Balance

SERIAL: 6005076801918141580000000000017B

===========================================================================

Path#Adapter/Path NameStateModeSelectErrors
0*fscsi0/path0OPENNORMAL570
1*fscsi0/path1OPENNORMAL560
2fscsi0/path2OPENNORMAL2309340
3fscsi0/path3OPENNORMAL2300300
4*fscsi1/path4OPENNORMAL490
5fscsi1/path5OPENNORMAL2304000
6*fscsi1/path6OPENNORMAL480
7fscsi1/path7OPENNORMAL2307720

The third command could have many LUNs displayed, but I only have one displayed as an example. This command provides the SAN system administrator with all the SAN LUNs and the paths for each LUN.

Commands to run before SVC migration

You need to run the following commands before migrating SVC.

1. pcmpath query adapter > /home/user/pcmpath_query_adapter_before_052014

2. pcmpath query device > /home/user/pcmpath_query_device_before_052014

3. pcmpath query wwpn > /home/user/pcmpath_query_wwpn_before_052014

4. lspath > /home/user/lspath_before_052014

5. lspath | grep -v Enabled > /home/user/lspath_grep_-v_Enabled_before_052014

You might have to refer to the output of these commands after the SAN team update or migrate their SVC. This is the before output and you will get the after output after the SVC is updated or migrated. You can than compare the before and after output to make sure that everything looks good. This shows you if there are some zoning problems and if everything looks good after the migration.

Commands to run on the day of migration immediately after updating SVC

=> pcmpath query adapter

Total Dual Active and Active/Asymmetric Adapters : 2

Adpt#NameStateModeSelectErrorsPathsActive
0fscsi1NORMALACTIVE12380470508072
1fscsi3NORMALACTIVE12739346308072

The output from the above command is crucial. Pay particular attention to the number of "Paths" and "Active" which are in this case 80 and 72. Ideally, they should be the same numbers as before SVC was migrated. If the Paths number is not the same, then you might have some zoning issues. For example, if the Paths number before SVC migration is 80 and after migration is 64, then there are 16 Paths missing.

Review the output of the following command in this case:

=> pcmpath query device

DEV#: 4 DEVICE NAME: hdisk10 TYPE: 2145 ALGORITHM: Load Balance

SERIAL: 657507680180864345000000000000333

=======================================================================

Path#Adapter/Path NameStateModeSelectErrors
0fscsi1/path0OPENNORMAL26858970
1*fscsi1/path1OPENNORMAL70
2fscsi3/path2OPENNORMAL28530070
3*fscsi3/path3OPENNORMAL70

Review this command before and after the migration. Here, I have provided only one entry, but you will have many. You might find that before migration, every hdisk had four paths as indicated above, two for fscis1 and two for fscsi3. Now, review the output of this command after the SVC migration. You might find that some hdisks have two paths instead of four. This would account for the missing paths. Pass the output to the SAN team and let them know that some paths are missing. You can pass them the output before and after of this command and they can find and fix the problem. In this case, the problem is with zoning and the SAN team can fix it.

The reason why we have 80 paths and only 72 are active according to the output above is that because fscsi1 has two hdisks that are not assigned to a volume group (VG) and each one of them has four paths and a total of 8 paths. These disks are not active means that they have not been assigned to a VG and are not being used. These hdisks will show in the pcmpath query device command as CLOSE NORMAL for STATE and MODE as opposed to OPEN NORMAL, which means that they have been assigned to a VG and are being used.

I also would like to check the number of enabled paths before and after and make sure that I have the same count. You can also browse quickly through the output from before and after the lspath command.

lspath | grep Enabled | wc -l (get a count of the enabled paths. Should be same as before)

I also like to browse all the paths that are not enabled from before or after. You may have to deal with some problems here, but not likely, if everything else looked good.

lspath | grep -v Enabled (look for all the paths that are not enabled)


Script output

You can use this script to get a report for many AIX servers with their SDDPCM and host attachment script releases along with their CCIN. It includes more useful info. You need to run it on a jump server (meaning an AIX server) from which you can use Secure Shell (SSH) to all AIX servers from where you want to get the report without having to enter the password. You also have to be the root user to run it. Refer to the script and its output.

The output of the script is in the comma separated value (CSV) format which means that you can copy it and paste it in a Microsoft Excel sheet. I usually like to highlight the label for all the fields. Here are the labels:

AIX SERVER, POWER, S/N, MODEL TYPE, AIX RELEASE, IOSLEVEL, SDDPCM, HAS, SAN DISK HBA, Z9, CURRENT FIRMWARE, LATEST FIRMWARE, CCIN, and FC.

See Table 1 to find the details.

Most of the labels in this table, such as "AIX RELEASE" are self explanatory. "IOSLEVEL" will be reported if the server is a VIOS instance. HAS stands for host attachment script. The label, "SAN DISK HBA", refers to fcs derived from the pcmpath query adapter command. You have to fill in the FC field and the LATEST FIRMWARE field as well. I have created a field in the Excel sheet right after the label "SDDPCM" and labeled it as "UPDATE TO". If the current SDDPCM release is fine, then I enter OK in this field. If it needs to be updated, then I would enter the new release to which it needs to be updated and then I would highlight the SDDPCM release and the "UPDATE TO" release to indicate that update or upgrade is needed for this server. I would also highlight the release for "LATEST FIRMWARE" if it needs to be updated as well.


Resources

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into AIX and Unix on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=AIX and UNIX
ArticleID=973966
ArticleTitle=IBM AIX SAN Volume Controller update and migration
publish-date=06192014