The SVC and Storwize V7000 offers a command line interface that you access via SSH. You start your favorite SSH client (such as PuTTY or Mindterm) and then logon as adminor as your own user-id. Right now to do this you need to generate a private/public key pair, although with release 6.3 (which will be available November 2011), you will be able to logon via SSH using just a user-id and password.
Having logged on there are three categories of commands you can issue:
svcinfo: Informational commands that let you examine your configuration. svctask: Task commands that let you change your configuration. satask: Service commands that are only used in specific circumstances.
There are several CLI usability features that I routinely find users are not aware of, so I thought I would share some of them here:
1) Listing all possible commands
If you cannot remember a command, here is a simple trick to list them all. Issue one of the following commands:
svcinfo -h or svcinfo -?
svctask -h or svctask -?
You can also type either svcinfo or svctask and then hit the tab key twice to get a full listing. With svctask you will need to type y to list them all, as per the example shown below:
IBM_2076:STG_V7000:admin>svctask (HIT TAB twice!)
Display all 139 possibilities? (y or n) y
2) Getting help on a particular command
Having found the command you want, issue that command with either -? or -h to get help information. For instance:
svctask mkvdisk -?
svctask mkvdisk -h
You will be shown the same help information that you can find in the Infocenter, including examples of syntax.
3) Drop the svctask and svcinfo prefixes
In release 6.2 of the SVC and Storwize V7000 firmware, the requirement to prefix a command with svcinfo or svctask has been removed. However I tend to keep using them because I write a lot of example commands for clients and I cannot be sure which version of firmware they are running.
4) Use the shell
When we SSH to the SVC or Storwize V7000 we are connecting to a Linux operating system using a special restricted shell. Some of the common Unix commands don't work (such as ls or grep or awk), but any commands that are provided by the shell itself, will work, such as while, if, read, pipe and echo.
We can use this to construct some really clever commands.
For instance creating volume copies is very popular, but the default copy rate is rather slow (50, which equals 2 MBps). It is not unusual for end users to speed up the background copy and then forget to slow it down when they are finished. So I wrote two commands to help me out. Firstly I run a command to display the copy rate of every volume. Ideally I should see 50 alongside each volume. However I often find that some volumes are set to higher numbers, such as the maximum value of 100 (which is 64 MBps).
svcinfo lsvdisk -nohdr |while read id name IO_group_id;do svcinfo lsvdisk $id |while read id value;do if [[ $id == "sync_rate" ]];then echo $value" "$name;fi;done;done
Lets break down this command. The structure looks like this:
We start with svcinfo lsvdisk -nohdr. This gives us a list of every VDisk in column format with no header information.
We pipe the output of that lsvdisk command to while read. This reads the output one line at a time and lets us work with that output. We read the first three columns of output and label the data in the first column id, the second column name and the third column IO_Group. I find we need to label at least three columns. We could read extra columns if we wanted to, but all I want is the VDisk id and name.
For every line of data we issue an lsvdisk command against each listed VDisk using the VDisk id. This output is not in column format so we need to do something different here.
We now examine the output of the lsvdisk command for each VDisk by piping the output to while read. Since each line contains a descriptor and a value, we label them id and value. We use if to look for a line that starts with sync_rate.
When we find the sync_rate for a VDisk we print the value of the sync_rate and the VDisk name. We are done for this VDisk.
We now examine the next VDisk and again look for the sync_rate for that VDisk.
Once we have examined every VDisk, we are done.
I then run the following command which sets the copy rate for every volume to the default value of 50 (2 MBps):
svcinfo lsvdisk -nohdr |while read id name IO_group_id;do svctask chvdisk -syncrate 50 $id;done
Clearly you could edit this second command to change the copy rate to any value between 0 and 100. In each case you just paste the entire command in, and hit Enter.
Lets break down this command. The structure looks like this:
We start with svcinfo lsvdisk -nohdr. This gives us a list of every VDisk in column format with no header information.
We pipe the output of that lsvdisk command to while read. This reads the output one line at a time and lets us work with that output. We read the data ion the first three columns of output and label the data in the first column id, the second column name and the third column IO_Group. I find we need to label at least three columns. We could read extra columns if we wanted to, but all I want is the VDisk id.
For every line of data we read, we do the following command: svctask chvdisk -syncrate 50 $id. Since we labelled the first column of output from the lsvdisk command as id, and that column contains VDisk IDs, we are going to issue this command against every VDisk that got listed.
Once we have run the chvdisk command against every VDisk listed, we aredone.
There are lots of possible clever combinations and I will list a few more in upcoming posts.
I have also been getting lots of requests to write a post about updating drive firmware, so expect something on that very soon.
IBM recently released a new version of firmware for the SAN Volume Controller and Storwize V7000. This is known as release 6.3 and continues the tradition of two major updates per year, each adding significant new functions.
So the 6.3 release notes for both Storwize V7000and SVC listed the following new feature:
Support for 4096 host WWPNs
Since I blithely listed this feature in a recent post I have received lots of emails asking exactly what it means, so I thought I had better explain.
The IBM SVC and Storwize V7000 have always had very clearly published maximum capabilities such as the ones listed here for Storwize V7000 release 6.3 and here for SVC release 6.3.
Most of these numbers are very high and few customers actually approach these maximums. The main issue I am seeing for some of our larger AIX customers is this one:
Total Fibre Channel ports (WWPNs) per I/O group: 512
The reason this can become an issue is the combination of NPIV and AIX Live Partition Mobility. NPIV allows one physical HBA to be shared among multiple operating system instances, each one believing it has exclusive access to the HBA with each one allocated its own unique WWPNs. Suddenly a single HBA which used to present just one WWPN through the SAN to the SVC, can now present vast numbers of them. In addition AIX Live Partition Mobility (which lets you move AIX operating systems between LPARs on the fly) needs additional pre-configured WWPNs defined on the target LPAR to support the move. This further increases the quantity of WWPNs that need to be defined to the SVC (one easy way to spot NPIV generated WWPNs is they normally start with the letter C).
So the bottom line is that IBM needs to make this limit bigger and SVC and Storwize V7000 6.3 code contains the necessary architectural changes to allow this. The first phase is to start potentially supporting up to 2048 WWPNs per I/O group although clearly based on the initial version of the release notes, the long-term plan is to support 4096.
But there is a problem and it has nothing to do with the SVC or Storwize V7000. The problem is that there are certain SAN configurations which may have issues with these large numbers of WWPNs (mainly around older SAN switches not having the CPU power for the switch fibre channel name-server and login-server to handle vast numbers of WWPNs coming out of one HBA).
So what should you do if you need to push the limits?
Contact your IBM Pre-Sales support and ask for a SCORE request to be opened (also known as an RPQ). You will need to detail your current SAN configuration (especially switch models and firmware levels) so that SVC development can ensure you won't overwhelm your switches. It will also allow our development team to learn how many clients our there need this support. All approvals will include a requirement to upgrade to release 6.3, so you should include this in your planning.
Any questions? Feel free to leave a comment or send me a tweet or an email.
The Storwize V7000 and SVC have a command line interface that you access via SSH. Every-time you logon, whether it is to transfer a file (using a tool like pscp), issue a single shot command from a script (using a tool like plink) or logon to issue commands interactively (using a tool like PuTTY), you clearly need to authenticate yourself. Since June 2003, the way you did this was to use a public/private key pair, where the SVC or Storwize V7000 had the public key and the SSH client (such as PuTTY) authenticated using the private key (the PPK file).
However with release 6.3 of the SVC and Storwize V7000 firmware, the use of key files is now optional. A user can now authenticate purely by using a password. This includes using your domain ID. So if you defined LDAP to your machine, as I documented here, you could now SSH direct to your Storwize V7000 or SVC, use your Domain user id and password and not go through the key file setup task. Nice!
The choice to continue to authenticate just with an SSH key remains available. If a user has both a password and a configured key file, then either method will work (you only need to use one - not both). Existing scripts will be unaffected by this change, so nothing gets broken because of this.
I think this is a very positive change and one I openly welcome. Combined with LDAP, this really makes user account setup an easy and simple task.
Lets imagine a new rack server or a new blade server has been added to your Fibre Channel SAN. The first job for the SAN administrator is to zone it to the storage it requires access to. The task normally runs something like this:
Identify the WWPNs for the new server HBA. We can do this using Qlogic SAN Surfer or Emulex HBAnywhere, or by looking at the WWPNs reported by the Fibre Channel switch or by using datapath query wwpn (with SDD and SDDDSM) or by using the xiv_fc_admin -P command with the XIV HAK. There are lots of different ways, you get the idea.
On fabric 1 create a new alias for the server HBA port cabled to that fabric.
For each storage device that the server needs access to on fabric 1 (or possibly just switch 1), create a new zone and include the new server alias and the alias for every relevant storage port on that device. Repeat if you have other storage devices (so two XIVs means two new zones).
Put the new zone (or zones) into the active zoneset (or a clone of it) and activate it.
Repeat on fabric 2 (after waiting a decent interval to ensure no mistakes were made in fabric 1... well I hope you wait.... you do don't you?).
The main trap here is that when creating a zone, you need to ensure you select all of the correct storage aliases for your selected storage device. For instance we could have a simple layout like this:
Fabric 1 contains our new server (in this example an IBM x3850) and three XIV ports:
This means when creating the zone I need to identify and select four separate aliases. What I could do instead is create an alias with all my XIV target ports in it. Now I only have two aliases to select in that fabric:
This means when creating the zone I need to identify and select three separate aliases. What I could do instead is create an alias with both my Storwize V7000 WWPNs in it. Now I only have two aliases to select in that fabric:
This method of amalgamating multiple storage port aliases works fine for devices like DS8000, SVC, Storwize V7000 and XIV. I use this method all the time to simplify zoning and I find it reduces both mistakes and the time required to complete zoning tasks.
The only exceptions are:
Don't do it for DS3000, DS4000, DS5000 or DCS3700 as the controllers on these devices do not like to see each other through the switch.
Don't combine ports from different storage devices, so if you have two XIVs in a fabric create one alias for the target ports of each XIV (although you could combine ports from different SVC I/O groups within the same SVC cluster into one alias). You should still use individual aliases for ports being used for migration or replication purposes.
Don't use the WWNN to create an alias. Always create multi-WWPN aliases so you have granular control of which ports go into the alias. If you use the WWNN from an XIV you will also implicitly include any ports that are being used for replication or migration and thus zone them to the host, which makes no sense.
I would love to hear any techniques you have to make your (and my) life easier.
SDDPCM (Subsystem Device Driver Path Control Module) is the multi-pathing plug-in for AIX 5.3 and 6.1. Customers who use IBM SVC, DS6000, DS8000 and/or ESS800 use this package to allow the operating system to handle multiple paths from OS to storage. The good news is that this plug-in is supplied free of charge. The bad news is that it is not included with AIX fixpacks. What this means is that while you may be dilligent with keeping AIX up to date, you may miss SDDPCM in the process.
There are two good reasons to keep SDDPCM in mind when planning updates:
1) Planning an upgrade from AIX 5.3 to 6.1
Before the AIX OS is upgraded, SDDPCM must be uninstalled and then reinstalled after the upgrade. There are cases when the host attachment script must also be uninstalled and reinstalled. This is explained in the SDD Users Guide found here:
If you have already upgraded from AIX 5.3 to 6.1 but you are still using the AIX 5.3 version of SDDPCM, you may need help from IBM before you can upgrade your SDDPCM to the AIX 6.1 version. This will come in the form of some special scripts. 2) General SDDPCM maintenance
As I noted in my previous blog entry, there are quite a few SDDPCM flashes out there right now. You need to check these out and ensure you are not exposed to the issues that are corrected by later versions of SDDPCM. Check out the flashes listed here (or read my previous blog entry): http://www-01.ibm.com/support/search.wss?rs=540&tc=ST52G7&dc=D600&dtm
What about SDD (Subsystem Device Driver) for AIX? Prior to AIX 5.2 FP5, AIX did not offer native multi-pathing (MPIO). This meant that each hardware vendor had to offer their own third-party software to handle multiple paths. To achieve this with the ESS (Shark), IBM released a product called DPO (Data Path Optimiser). This product became SDD and was made available for a wide variety of operating systems.
When AIX offered MPIO, IBM then also offered a vendor plug-in (Path Contol Module) for native AIX MPIO which IBM called SDDPCM. This means you have two choices with AIX: SDD or SDDPCM. If your considering which is best, SDDPCM is my preference. This is because it is native to the operating system and also better supports the possibility of co-existence of multiple PCMs. Note that migrating from SDD to SDDPCM is not supported by the VIOS at this time, so if your running VIOS you will need to stay put for now.
IBM SAN Volume Controller (SVC) has offered Fibre Channel Storage Virtualization since June 2003. Two SVC nodes communicate with each other via fibre channel to form a high availability I/O group. They then communicate with the storage that they virtualize via Fibre Channel and with the hosts they serve that virtual storage to, via Fibre Channel. When IBM added real-time (metro mirror) and near real-time (global mirror) replication it was also done using Fibre Channel, with each SVC cluster communicating to the other by connecting using fibre channel protocol transported over dark fibre with or without a WDM or via FCIP (Fibre Channel over IP) routers.
Each Fibre Channel port on an SVC node can be a SCSI initiator to backend storage, a SCSI target to hosts and all the time communicate to its peer nodes using those same ports. With every generation of SVC node, these ports got faster and faster, going from 2 Gbps to 4 Gbps to 8 Gbps. In SVC firmware V5.1 IBM added iSCSI capability to the SVC using the two 1 Gbps ethernet ports in each node. This allowed each node to also be an iSCSI target to LAN attached hosts.
When the Storwize V7000 came out in Oct 2010 it offered all of this capability, plus offered two fundamental changes to the design.
Firstly the two controllers in a Storwize V7000 can communicate with each other across an internal bus, eliminating the need to zone them together (or even attach the Storwize V7000 to Fibre Channel fabrics).
The other more obvious difference is that a Storwize V7000 comes with its own disks, which it communicates with via multi-lane 6 Gbps SAS.
When IBM added 10 Gbps Converged Enhanced Ethernet adapters to the SVC and to the Storwize V7000, these adapters operated as iSCSI Targets, allowing clients to access their volumes via a high-speed iSCSI network. In V6.4 code IBM allowed these adapters to also be used for FCoE (Fibre Channel over Ethernet). These are also effectively SCSI targets ports allowing hosts that use CEE adapters to connect to the SVC or V7000 over a converged network.
If you have a look at the Configuration limits page for SVC and Storwize V7000 version 6.4 (the Storwize V7000 one is here), you will see this interesting comment:
"Partnerships between systems, for Metro Mirror or Global Mirror replication, do not require Fibre Channel SAN connectivity and can be supported using only FCoE if desired"
So does this mean we can stop using FCIP routers to achieve near real-time replication between SVC clusters or Storwize V7000s? The short answer is most likely not. Lets look at why...
The whole reason Fibre Channel became the standard method to interconnect Enterprise Storage to Enterprise hosts is simple: Packet loss is prevented by buffer credit flow control. Frames are not allowed to enter a Fibre Channel network unless there are buffers in the system to hold them. Frames are normally only dropped if there is no destination to accept them. Fibre channel is a highly reliable, scalable and mature architecture. When we extend Fibre Channel over a WAN we do not want to lose this reliable nature, so we use FCIP routers like Brocade 7800s, that continues to ensure frames are reliably delivered in order, from one end point to another.
Converged enhanced ethernet allows Fibre Channel to be transported inside enhanced ethernet frames. The one fundamental that CEE brings to the table is the same principle that a frame should not enter the network without a buffer to hold it. Extending FCoE over distance has the same challenge: the moment you start moving those frames over a WAN connection you need to ensure frames are not lost due to congestion. How do we do this? The same way we did with Fibre Channel: we use Dark Fibre, we use WDMs or we use routers. The same issues and requirements exist.
For more information on FCoE over distance check out this fantastic Q&A from Cisco:
It is ironic that only days after I wrote that 497 is the IT number of the beast, I learn that Linux has another unfortunate number: 208.
The reason for this is a defect in the internal Linux kernel used in recent firmware levels of SVC, Storwize V7000 and Storwize V7000 Unified nodes. This defect will cause each node to reboot after 208 days of uptime. This issue exists in unfixed versions of the 6.2 and 6.3 level of firmware, so a large number of users are going to need to take some action on this (except those who are still on a 4.x, 5.x, 6.0 or 6.1 release). If you have done a code update after June 2011, then you are probably affected. This means that if you are an IBM client you need to read this alert now and determine how far you are into that 208 day period. If you are an IBMer or an IBM Business Partner, you need to make sure your clients are aware of this issue, though hopefully they have signed up for IBM My Notifications and have already been notified by e-mail.
In short what needs to happen is that you must:
Determine your current firmware level.
Check the table in the alert to determine if you are affected at all, and if so, how far you are potentially into the 208 day period.
Prior to the 208 day period finishing, either reboot your nodes (one at a time, with a decent interval between them) or install a fixed level of software (as detailed in the alert).
To give you an example of the process, my lab machine is on software version 18.104.22.168 which you can see in the screen capture below. So when I check the table in the alert, I see that version 22.214.171.124 was made available on January 24, 2012, which means the 208 day period cannot possibly end before August 19, 2012.
Earliest possible date that a system running this release could hit the 208 day reboot.
SAN Volume Controller and Storwize V7000 Version 6.3
30 November 2011
25 June 2012
24 January 2012
19 August 2012
Regardless, I need to know the uptime of my nodes, so I download the Software Upgrade Test Utility (in case you have an older copy, we need at least version 7.9) and run it using the Upgrade Wizard (NOTE! We are NOT updating anything here, just checking):
I Launch the Upgrade Wizard, use it to upload the tool and follow the prompts to run it, so that I get to see the output of that tool. The output in this example shows the uptime of each node is 56 days, so I have a maximum of 152 days remaining before I have to take any action. At this point I select Cancel. You can run this tool as often as you like to keep checking uptime.
Note if you are on 6.1 or 6.2 code you may see a timeout error when running the tool, especially for the first time. If you do see an error, please follow the instructions in the section titled "When running the the upgrade test utility v7.5 or later on Storwize V7000 v6.1 or v6.2" at the Test Utility download site.
As per the Alert:
If you are running a 6.0 or 6.1 level of firmware, you are not affected.
If you are running a 6.2 level of firmware, the fix level is v126.96.36.199 which is available here for Storwize V7000 and here for SVC.
If you are running a 6.3 level of firmware, the fix level is v188.8.131.52 which is available here for Storwize V7000 and here for SVC.
If you are using a Storwize V7000 Unified, the fix level is v184.108.40.206 which is available here.
You should keep checking the alert to find out any new details as they come to hand. If you are curious about Linux and 208 day bugs, try this Google search.
*** Updated April 4, 2012 with links to fix levels ***
If you have any questions or need help, please reach out to your IBM support team or leave me a comment or a tweet.
*** April 10: The IBM Web Alert has been updated with new information on what to do if your uptime has actually gone past 208 days without a reboot. In short you still need to take action. Please read the updated alert and follow the instructions given there. ***
I always laugh when people say to me: I wouldn't know what to blog about!
When you work in pre-sales support, you constantly get asked questions and each one of them could be the subject of a new blog post. Right now the most common question I am getting is:
I am implementing VMware Site Recovery Manager (SRM). One of the components I need are vendor specific Site Recovery Agents (SRA). I have searched IBM's website but cannot find them. Where are they?
So the short answer is: you get them from the VMware SRM download site. However before downloading, there is a key task that absolutely needs to be performed:
Visit the VMware vCenter Site Recovery Manager Storage Partner Compatibility Matrix. This site will confirm what products are supported by each version of SRM. You can find it here, but clearly you need to check back regularly to ensure you have the latest information.
Now find your storage device in the matrix and confirm what firmware levels are supported. This is really important. For example, the Feb 27, 2012 edition of the matrix tells me that the Storwize V7000 is supported for SRM version 5.0, but only when running Storwize V7000 firmware version 6.1 or 6.2. This is significant because if you upgrade to version 6.3 you are not supported. In fact that combination doesn't actually work yet, as detailed here. Clearly something you need to be aware of when planning firmware updates.
So where are the SRAs? On each of the pages below use the Show Details button to see what version SRAs are being shipped with that SRM (although sometimes the pages take a few days between an SRA being added and the page being updated):
There are a few more questions I routinely get asked:
Does IBM actually have an SRA download site?
The answer is yes, but it is an FTP site only for SRAs written by IBM. It is principally a repository for older SRAs and beta SRAs but you can also find the current SRAs on it. You can find the site here. Note however that it is NOT the official source. For that you need to use the VMware site.
What about the SRA for LSI/Engenio based products like the DS4800?
These used to also be found on the LSI site, but since LSI sold Engenio to NetApp, it is no longer available from the LSI or NetApp websites. You need to download the current version from the VMware sites listed above. There is a version for SRM 5 on the VMware download site.
What about nSeries SRAs?
If you need an nSeries SRA, again you should go to the VMware download pages. There are separate SRAs listed and available for IBM nSeries (as opposed to an SRA for NetApp branded filers).
What about an SRA for XIV with SRM version 5?
The answer: The SRA for XIV with SRM 5 (and 5.0.1) is now available from VMware. If you have access to download SRM, you will be able to download SRA version 2.1.0. It is the same SRA for both XIV Generation2 and Gen3.
What about an SRA for Storwize V7000 and SVC version 6.3 code?
The answer: It is coming. We are working to make it available as soon as possible. I will update this post as soon as I have a date for you (we are talking weeks, not months).
*** Update March 23, 2012 - Added details on SRM 5.0.1 ***
I am getting this question on a very regular basis:
"We have just upgraded to ESXi 5.0 but we cannot find the VAAI driver on the IBM Website"
The answer? There is no vendor supplied driver because no driver is needed. ESXi 5.0 uses a SCSI T10 compliant set of commands that all vendors need to support for VAAI to work.
But of course in the tradition of all answered questions, it leads to another question:
"Once I have upgraded to ESXi 5.0 how can I tell if VAAI is really working?"
The good news is that it is very easy to spot if ESXi 5.0 has detected a VAAI capable LUN. The moment a new LUN is detected by ESXi 5.0 it tries out an Atomic Test and Set command. If that works, you will see that Hardware Acceleration shows as Supported in vCenter. In the screen capture below I have three datastores, two from XIV and one from Storwize V7000, all presented to an ESXi 5.0 server. I dragged the Hardware Acceleration column over from the right hand side to help with the screen capture (in case your vCenter looks different), but you can see the Hardware Acceleration column shows each DataStore as Supported (and did so the moment the volume was detected).
Of course having seen the Hardware Acceleration Supported message only proves that Atomic Test and Set works. To confirm if XCopy (Hardware Accelerated Move) is working, on SVC or Storwize V7000 we can use the Performance monitoring panel. In the example below I first performed a storage vMotion, moving a virtual machine between two Datastores located on the same Storwize V7000 (running 220.127.116.11 firmware). I then performed a clone of the same virtual machine, where the source was on one datastore and the target was placed on another (but both located on the same Storwize V7000). What you can clearly see is that both operations (storage vMotion and cloning) generated no volume traffic, only MDisk traffic. This means that the ESXi server is doing none of the work and the storage is doing all of the work.
With the 6.3 release of the Storwize V7000 and SVC code (which I blogged about here), there are so many new features and functions that I have plenty more to blog about!
The first new feature I blogged about was LDAP support, but an existing feature that has been enhanced is the performance monitor (brought in with release 6.2). When this first came out I put a video on You Tube showing what metrics could be displayed in that release. This is a sped up image with no voiceover:
Now with release 6.3 IBM has added separate graphs for reads and writes plus the ability to display IOPS or MBPS, plus the ability to display graphs of read and write latency. Nice! I got so excited I made another You Tube video, this one with narration. So now you can compare the new to the old:
Once your SVC or Storwize V7000 is upgraded to version 6.3 you can start using LDAP for authentication. This means that when you logon, you authenticate with your domain user-id and password rather than a locally created user-id and password.
So why is this important?
It saves you having to configure every user on every SVC or Storwize V7000. If you have multiple machines this makes it far more efficient to set up authentication.
It means that when commands are executed on the SVC or Storwize V7000, the audit log will show the domain username that issued that command, rather than a local username, or worse just superuser (i.e. who mapped that volume? The superuser did.... who? )
It gives you central control over access. If someone leaves the company you just need to remove access at the domain controller, meaning there won't be orphan user-ids left on your Storage equipment.
So as an exercise I added my lab Storwize V7000 to our domain to show how it is done. This example also applies to an SVC so don't be confused if I only refer to Storwize V7000 from now on.
The first task is to negotiate with your Domain administrator to get a new group setup on the domain. In this example I use a group called IBM_Storage_Admins which lets me use this group for various storage devices (such as an XIV or a SAN Switch).
To create this group we need to logon to the Domain Controller and configure Active Directory. An easy way to do this from the AD controller is to go to Start → Run and type dsa.msc and hit OK. The Active Directory Users and Computers Management Console should open.
Select the groups icon to create a new group.
Enter your group name, in my case: IBM_Storage_Admins and hit OK.
Now right select relevant users who need access to the storage and add them to the IBM_Storage_Admins group. In this example I have selected Anthony (which uses anthonyv as a username).
In this example we are adding anthony into the IBM_Storage_Admins group:
Now it is time to configure the Storwize V7000 so start the Web GUI and logon as Superuser.
Firstly we go to Settings → Directory Services:
We choose the button to Configure Remote Authentication:
We choose LDAP and hit next.
We choose Microsoft Active Directory with no Transport layer Security. We then expand the Advanced Settings. My lab domain is ad.mel.stg.ibm so I use the Administrator ID on the Domain Controller to authenticate access. You could use any user that has authority to query the LDAP directory. We then hit Next.
We then add the domain controller which in this example is 10.1.60.50 and the base domain name chopped into pieces (so ad.mel.stg.ibm becomes dc=ad,dc=mel,dc=stg,dc=ibm ) and hit Finish.
Provided the command completes successfully we have defined the domain controller to the Storwize V7000. Now we need to add a group. Go to Access → Users.
Select the option to add a New User Group.
In this example we want to add a group for users allowed full admin access to the Storwize V7000. This matches the group we created on the Domain Controller. So we call the group IBM_Storage_Admins and we use the Security Administrator role (which is the most powerful role) and tick the box to enable LDAP for this group.
Now to test, I logon to the Storwize V7000 using the domain user-id anthonyv with that users domain password. Remember this user is not defined on the Storwize V7000 itself and that if it all goes wrong, we can still logon as Superuser.
Now I create a volume and delete it. Then I check the audit log from Access → Audit log.
Sure enough, we see exactly who did that command.
This is a great outcome for security,auditing and for easy access administration.
If you have issues, from the Settings → Directory Services menu, use the Global Actions dropdown on the right hand side to Test LDAP Connections and Authentication or re-configure LDAP.
If you already have existing users (what we call Local users), configuring remote authentication using LDAP does not disable or invalidate those local user-ids. This means you can either logon with a local user-id or logon with a Domain user-id. This is handy if the domain controller fails but can confuse you if your local user name and your domain user name are the same name (for example both anthonyv). The Storwize V7000 will look you up in the local user name list first. I suggest removing all local users (except superuser) as this will reduce confusion but still leave you a backdoor in case remote authentication stops working.
If you see any mistakes or have suggestions to improve the way I described this, please let me know.
The latest release of SVC and Storwize V7000 firmware is now available for download. The major new features that are added with this release are:
Global Mirror with Change Volumes
Native LDAP Authentication
Extended distance split clusters (for SVC)
Support for 4096 host WWPNs
These are some great new features. The ability to use Global Mirror with Change Volumes means clients can now mirror across far smaller pipes, while the increase in host WWPNs is very welcome news for NPIV installations that are suffering from WWPN sprawl.
If you plan to upgrade, firstly grab the new Upgrade Test Utility from here. The links to the Storwize V7000 and SVC versions are both on that page. Remember you can run this test as many times as you want whenever you want, to check the health of your device for upgrade. When you run the upgrade test utility on a Storwize V7000 you may get a message that your disks have down-level firmware. The process to update them is documented here.
If you're using a Storwize V7000 you can grab the 18.104.22.168 code from here. If you're using an SVC you can grab the 22.214.171.124 code from here. I am sending you to the compatibility matrix page because you should always check that your from level is ok for your to level.
To run the upgrade go to Configuration (the spanner icon) → Advanced → Upgrade Software →Launch Upgrade Wizard
I have not shown all the panels you will see because it is very much a follow-your-nose task, but in essence, first we feed it the Upgrade Test Utility file and run that test.
If you get warnings you may need to act on these. If you are unsure what to do to resolve a warning message, place a service call.
Once the test passes or you're happy you understand the warnings, we now point it at the code package and wait for it to copy across and keep hitting Next.
The application of the code shuts down and reboots each controller, with a 30 minute gap in between. You will transition from this (both nodes down-level, node1 being upgraded):
To this (node1 upgraded, node2 still online but waiting for 30 minutes):
When node2 starts the upgrade the GUI will failover to node1 and be upgraded to the new version. You will notice the difference immediately, it has a different look and feel. Please don't be tempted to play with the new functions until both controllers are upgraded! Wait until you see this (note a slight change, the GUI flow is now Settings (the spanner icon) → General → Upgrade Software:
Now your complete it is time to start checking out what is new... but that's a whole different blog post!
IBM have offered Enterprise Storage Virtualization since June 2003 with the IBM SAN Volume Controller (SVC). October 2010 saw IBM releasing the Storwize V7000, taking the SVC code and packaging it into a midrange disk product. So now you have four possible choices:
Use SVC to virtualize your storage.
Use Storwize V7000 to provide internal SAS drives plus virtualize your storage.
Use Storwize V7000 as a midrange disk product.
Use Storwize V7000 virtualized behind SVC.
The great thing is that all four choices are valid and all four choices work just fine. But for customers already using SVC, or considering SVC, the question then becomes, should I virtualize a Storwize V7000 behind an SVC? Does this makes sense?
The short answer: YES!
We have a great many customers happily doing this, so I thought I would share some common questions I get around configuration. Firstly there is an InfoCenter page on this which you will find here. Secondly there is a debate about whether we should create individual volume/arrays on the Storwize V7000 or just create a single pool on the Storwize V7000 (which equates to striping on striping). More bench marking is being done to see if one method is truly better than the other, so until then I recommend the method described below. If you have already done stripe on stripe, don't go changing anything until I update this post.
How many ports should I use for Zoning?
The Storwize V7000 has 8 Fibre Channel ports, 4 from each node canister. You need to zone at least two ports from each node canister to your SVC cluster. This is no different to how you would zone a DS5100 or an EMC VNX.
How will the SVC detect the Storwize V7000?
On the SVC you will see two storage controllers, one for each node canister. This is quite normal. The reason for this is that each node canister reports its own WWNN. This is not a problem and will not affect volume failover if one node canister goes offline.
In the example below the SVC has detected two new controllers. The confusing factor is that both report as 2145s, but they are a Storwize V7000. Rename them to reflect what they really are (something like StorwizeV7000_1_Node1 and StorwizeV7000_1_Node2).
How should I define the SVC on the Storwize V7000?
You need to create a new host on the Storwize V7000 and call it something like SVC_1. if the SVC WWPNs don't appear in the WWPN dropdown, you will need to manually add them as shown below:
You can get the SVC WWPNs from your existing zoning or by doing an svcinfo lsnodeagainst each SVC node or display them in the SVC GUI as shown below:
What size Storwize V7000 volumes should I create?
My recommendation is to do the following on the Storwize V7000
Create arrays of preferably 8 disks in size. The ideal number will depend on how many disks you have. On my machine I have 22 disks, so I create three arrays each with seven disks (and one hot spare):
Create one pool for each array:
Create one volume out of each pool (using all space in the pool).
Define the SVC to the Storwize V7000 as a host (as described above) and map all volumes to the SVC.
On the SVC detect all the Storwize V7000 LUNs as MDisks and create one pool.
Now you should have a pool on the SVC that you can use to create volumes to present to your hosts. They will be striped by default, which is exactly what you want.
Hopefully all of this makes sense. Questions and comments very welcome.
There was a time when 32 bits was considered a lot. A hell of a lot.
With 32 bits, you can create a hexadecimal number as big as 0xFFFFFFFE (presuming we reserve one bit). In decimal that's 4,294,967,295. Hey... imagine a bank account balance that big? If you use 32 bits to count out 512 byte sectors on a disk, you could have a disk that's 4,294,967,295 times 512... or 2,199,023,255,040 bytes! That's sounds huge, right?
Well... actually...no... that's 2 TiB, which most people would refer to as 2 Terabytes. Mmm.. Suddenly I am less impressed (still wouldn't mind that as a bank account though).
Now there are plenty of running Systems that still cannot work with a disk that is larger than 2 TiB. One of the more common is ESX. I am presuming this limitation is going to disappear, so Storage susbsystems need to be ready to create volumes that are larger than 2 TiB.
The good news is that with the May 2011 announcements, IBM is removing the last 2 TiB sizing limitations from its current storage products. There appears to have been some confusion in the past, so I thought I would go through and be clear where each product is at:
Firmware version 07.35.41.00 added support to create volumes larger than 2 TB. The maximum volume size is limited only by the size of the largest array you can create. This capability has been available for some time and hopefully you are already on a much higher release.
DS4000 and DS5000
Firmware version 07.10.22.00 added support to create volumes larger than 2 TB. The maximum volume size is limited only by the size of the largest array you can create. This capability has been available for some time and hopefully you are already on a much higher release.
DS8700 and DS8800
The DS8700 and DS8800 will support the creation of volumes larger than 2 TB once a code release in the 6.1 family has been installed. With this release you will be able to create a volume up to 16 TiB in size. The announcement letter for this capability is here.
The volume size on an XIV is limited only by the soft limit of the pool you are creating the volume in. This allows the possibility of a 161 TB volume.
SVC and Storwize V7000
These two products have two separate concepts:
Volumes (or VDisks) that hosts can see.
Managed disk (or MDisks) that are presented by external storage devices to be virtualized. Within this there are two further categories: - Internal MDisks created using the Storwize V7000 SAS disks. - External MDisks created by mapping volumes from external storage (such as from a DS4800).
SVC and Storwize V7000 Volumes (VDisks).
Prior to release 5.1 of the SVC firmware, the largest volume or VDisk that you could create using an SVC was 2 TiB in size. With the 5.1 release this was raised to 256 TiB, as announced here. When the Storwize V7000 was announced (with the 6.1 release) it also inherited the ability to create 256 TiB volumes.
Because the Storwize V7000 has its own internal disks, it can create RAID arrays. Each RAID array becomes one Mdisk. This means the largest MDisk we can create is limited only by the size of the largest disk (currently 2 TB), times the size of the largest array (16 disks). This means we can make arrays of over 18 TiB in size (using a 12 disk RAID6 array with 2 TB disks). Thus internally the Storwize V7000 supports giant MDisks. We can also present these giant MDisks to an SVC running 6.1 code and the SVC will be able to work with them.
SVC and Storwize V7000 External Managed Disks.
When presenting a volume to the SVC or Storwize V7000 to be virtualized into a pool (a managed disk group) we need to ensure two things are confirmed. Firstly you need to be on firmware version 6.2 as confirmed here for SVC and here for Storwize V7000. Secondly that the controller presenting the volume has to be approved to present a volume greater than 2 TiB. From an architectural point of view, MDisks can be up to 1 PB in size as confirmed here, where it says:
Capacity for an individual external managed disk
Note: External managed disks larger than 2 TB are only supported for certain types of storage systems. Refer to the supported hardware matrix for further details.
I recommend you go to the supported hardware matrix and confirm if your controller is approved. The links for Storwize V7000 6.2 are here and for SVC here. As of this writing, the list has still not been updated, but I am reliably informed it will include the DS3000, DS4000, DS5000, DS8700 and DS8800. It will not initially include XIV, which will come later. Please also note the following:
Support for giant MDisks (greater than 2 TiB) is firmware controlled. If the controller (e.g. a DS5300) presenting a giant MDisk is not on the supported list for your SVC/Storwize V7000 firmware version, then only the first 2 TiB of that MDisk will be used.
If your already presenting a giant MDisk (and using just the first 2 TiB), then just upgrading your SVC/Storwize V7000 firmware won't make the extra space useable. You will need to remove the MDisk from the pool, then do an MDisk discovery and then add the MDisk back to the pool. All of this can of course be done without disruption, using the basic data migration features we have supported since 2003.
What to do in the meantime?
If your currently using an SVC or external MDisks with a Storwize V7000, then you need to work within the 2 TiB MDisk limit (except for Storwize V7000 behind SVC). The recommendation is a single volume per Array for performance reasons (so the disk heads don't have to keep jumping all over the disk to support consecutive extents on different parts of the disk). This can require careful planning. For instance using 7+P RAID5 Arrays of 450 GB drives makes an array that is over 3 TB. What to do in this example?
Divide it in half? (by creating two 1.5TB volumes)
Waste space? (a whole 1 TB)
Use smaller arrays? (a 4+P array of 450GB disks is 1.8 TB)
The answer is that where possible, create single volume arrays using 4+P or larger. If the disk size precludes that, then create multiple volumes per array and preferably split these volumes across different pools (MDisk groups).
Anything else to consider?
Well first up, will your Operating System support giant volumes? Googling produces so much old material that it becomes hard to nail down exact limits. For Microsoft, read this article here. For AIX check out this link. For ESX, check out this link.
Second of course is the consideration of size. File systems that utilize the space of giant volumes could potentially lead to giant timing issues. How long will it take to backup, defragment, index or restore a giant file system based on a giant volume (the restore part in particular)? Outside the scientific, video or geo-physics departments, are giant volumes becoming popular? Are they being held back by practical realities or plain fear? Would love to hear your experiences in the real world.
And a big thank you to Dennis Skinner, Chris Canto and Alexis Giral for their help with this post.