I am facing problem with my storage DS3512 i have configure storage with IBM blade Center H (88524TG) and host server is IBM blade server HX5 (787264G) OS windows 2008 DATA Center x64 everything seems to be working fine but my logical drive changed its preferred path intermittently i have try to update HBA device driver and MPIO driver but still its changing path and also i saw some error on event log "Controller cache not enabled or was internally disabled" or "Logical Drive not on preferred path due to ADT/RDAC failover"
can anyone help me to find out root causes of this issue ?
logs atteched herewith for your reference.
Noman Ajaz Ahmed Siddiqui
This topic has been locked.
23 replies Latest Post - 2012-11-07T20:30:11Z by SystemAdmin
Pinned topic DS3512 prefered path changing intermittently issue .
Answered question This question has been answered.
Unanswered question This question has not been answered yet.
- 802 KB
Updated on 2012-11-07T20:30:11Z at 2012-11-07T20:30:11Z by SystemAdmin
andrew.scott 270003R6AC1 Post
Re: DS3512 prefered path changing intermittently issue .2011-04-13T11:29:57Z in response to andrew.scottIts new configuration.
its not esx4.1 hosts. its windows 2008 Data center x64 with hyper-visor on blade server HX5 and its scalable partition.
And i have attached my hardware details.
pavel.alexei 2700002XMC334 PostsACCEPTED ANSWER
Re: DS3512 prefered path changing intermittently issue .2011-04-14T08:30:08Z in response to SystemAdminAs far i understood you have 4 HX5 blades.
Judging Storage profile 2 blades works independently and didn't access storage (OTHMKT-BLADE1 and OTHMKT-BLADE2) and from rest 2 you created a two node complex. I am right?
Did you zone Fiber Switches?
And what is the purpose of OSLUN1,OSLUN2,OSLUN3?
dbilyk 27000285KB1 PostACCEPTED ANSWER
Re: DS3512 prefered path changing intermittently issue .2011-05-06T20:27:20Z in response to SystemAdminHas any progress been made with this issue? I have been using our DS3512 for the past four months without issue and now suddenly, the preferred controller keeps switching from one to the other. Things have just worked up until now, so much so that I did not suspect anything wrong with the DS3512 until our ESXi VMs suddenly started to bog on heavy I/O access. When I checked the DS3512, that is when I saw the warning that the current controller was not the preferred one. Then, 15 minutes later, things swapped; it now preferred A and was connected to B whereas previously it preferred B and was connected to A. I do not know what is causing this switch back and forth but I suspect it is the cause of our VMs bogging down so heavily.
Despite operating the DS3512 for four months, I am still a relative n00b. Until now, things just worked. I should have suspected something going together so easily was just hiding an issue that would someday materialize. At least we are still in pre-production.
If anyone has any suggestions, I would love to hear them.
skyron 2700028U8P84 PostsACCEPTED ANSWER
Re: DS3512 prefered path changing intermittently issue .2011-05-07T04:52:12Z in response to dbilykThis issue mostly comes up due to either incorrect zoning, or having old fiber card drivers (qlogic/emulex) on the server. If you update them to the latest, especially for ESX/ESXi, it resolves the issue in most cases.
kashif.adeel 0600003BP33 Posts
Re: DS3512 prefered path changing intermittently issue .2012-10-19T03:27:12Z in response to kashif.adeelHi all,
To possibly help a few people I though I would not some core configuration steps, as most people have said the majority of the time "path thrashing" is due to incorrect cabling, zoning or host type / multi-pathing setup issues. Each setup has different specific requirements so my comments below are broad and just a quick summary....
1. Storage Host Port configuration
Unless you are using SAS connections, you must have a switch fabric (redundant preferred) between the hosts and the storage.
Cabling for Fibre channel and iSCSI from storage(generally):
Controller A Port 1 -- Switch 1
Controller A Port 2 -- Switch 2
Controller B Port 1 -- Switch 1
Controller B Port 2 -- Switch 2
Cabling from Host:
Host HBA 1 -- Switch 1
Host HBA 2 -- Switch 2
Note: To rule out issues; if you are configuring multipathing to dual controller, ALL hosts should be configured for multipathing with visibility to both controllers. Especially if they are accessing the same LUN's. Generally each HBA in the host should have visibility to BOTH controllers in the storage.
Create Alia's for your Storage Ports and your Host ports.
For DS3500 setup "single target, single initiator" zoning. So, create a zone for each host port to each storage port, then put all the zones in a config and enable it.(NOTE: This is different for other arrays E.G: V7000)
The main thing is that each host can see 4 paths. once you have setup the zoning and the storage is running
Register hosts in storage (depending on connection type and card type)
Register 2 x HBA's for fibre to the host
Register 2 x Hardware iSCSI HBA's to the host
Register 1 x Sofware iSCSI Initiator to the host (you only have 1 x software initiator however pathing can still be through multiple NIC's for redundancy).
NOTE: If you IQN or WWN for your cards are not showing up under "Auto Discovered" then there is something wrong, don't bother trying to enter them manually.
Re-Scan from your hosts to help the HBA's register to the storage....see if they are now register under "Auto Discovered", if not you still have problems.
Make sure you use the correct host type when registering the host.
For VMware - select VMWARE (or for old firmware LNXCLVMWARE)
For Microsoft Servers running IBM DSM Software with Microsoft MPIO select Microsoft XXXXXX
For Microsoft Servers running Microsoft MPIO with Veritas Dynamic Multi-pathing select Microsoft XXXXX (Supporting DMP)
When Clustering select the cluster host type option.
Once you have the hosts register you can MAP a LUN, once the LUN is mapped you can check via the HBA BIOS to see if both HBA's can see the target device and LUN.
You want to make sure all HBA's can see the Array.
For Software iSCSI, you want to make sure that you can ping ALL host ports of the Array.
Once you have your mappings right and your LUN/s mapped you need to setup the multipathing on the Host
For VMware, multipathing is enabled by default, however you should check your pathing policy and the QTy of paths to be sure the zoning is correct
For Microsoft, firstly make sure you have the latest HBA firmware and StorPort drives, then you will need to install Microsoft MPIO framework (windows 2008) and then the IBM DSM modules. The IBM DSM modules are now packaged separately to Storage Manager software so make sure you install it. Just installing IBM storage manager will NOT install the multipathing software.
Once the multipathing is install and the LUN is visible you can run command lines with the IBM DSM to confirm pathing visibility utilising the "SMcli" commands...Google SMcli...
You should see multiple paths for all luns.
Another item to check is pathing issues due to fabric issues, so check your port error levels on your FC switches....
Re: DS3512 prefered path changing intermittently issue .2011-11-24T17:06:40Z in response to SystemAdminHi Sorry for digging up an old post, but I'm having the same issue. DS3500 with Windows 2008 Cluster. Logical Drive not on preferred path due to ADT/RDAC failover errors and the drives would go offline.
What was the fix for it?? thanks.
Re: DS3512 prefered path changing intermittently issue .2012-10-20T02:25:52Z in response to SystemAdminim also experiencing this behavior after firmware upgrade 7.83. message like "Controller cache not enabled or was internally disabled" or "Logical Drive not on preferred path due to ADT/RDAC failover".
Re: DS3512 prefered path changing intermittently issue .2012-10-22T13:18:21Z in response to BerniebgbbWe have one ibm bladecenter h with 8 blade server (4 hs22v & 4 hx5). the chassis have 2 brocade fibre switch connected to one ibm ds3512 with 2 controller and 2 exp3512 enclosures. We use this hardware to support a vmware infrastructure cluster version 5.0.U1
Recently we upgrade the ds3512 FW from version 7.77.34 to 7.83.22 and the brocade fabric os from 6.3.1 to 6.4.2a. Since then we are experiencing this kind of behavior also i suspect that this kind of errors are produced when there are a lot of IOps on the luns
here is one of the support files.
Novikov_Alexander 100000HTD51392 PostsACCEPTED ANSWER
Re: DS3512 prefered path changing intermittently issue .2012-10-22T17:56:55Z in response to SystemAdminDear Jonathan,
read section Appendix B. Using a storage subsystem with a VMware ESX Server configuration and fix your configuration:
Installation and Host Support Guide v10.8 - IBM System Storage DS Storage Manager
And lets check your blades and switches. Post output of command supportshow from both FC switches, AMM Service Data, and DSA logs from all blades (use DSA Preboot Edition and post resulted files *.xml.gz).
jfrench 270001FJXY9 PostsACCEPTED ANSWER
Re: DS3512 prefered path changing intermittently issue .2012-10-23T01:11:48Z in response to Novikov_AlexanderHello,
I have been having this issue with a DS3524 since May. I have had several calls with IBM open and my last call had 11 techs on it without any resolution. My setup is using iSCSI with a BladeCenter H and 9 blades.
Can anyone post a specify fix that worked.
In our case replacing the Switches/Cables in the blade center did not resolve issue and no IBM tech seems to have any idea on cause or fix.
Re: DS3512 prefered path changing intermittently issue .2012-10-23T12:34:17Z in response to jfrenchjfrench,
There are so many specifics with respect to a iSCSI solution with Blades with redundant pathing setup there is no simple answer.
Can you tell us more about your environment and how it is setup?
1. Storage Setup? Logs
2. Switch setup? logs
3. Host setup? Logs
4. Solution setup? what are you trying to achieve?
I have found that IBM tech's typically look at what they are responsible for (E.G: Server, Storage, fabric....) and not across the complete environment, and when / if they do it is through many diffident people. However my experience has been that most of the time I get a quick resolution for "known issues", but when it comes to user configurations issues that are nothing to do with hardware or firmware it is a different story....
Hopefully the forum can help if you provide more info.
Re: DS3512 prefered path changing intermittently issue .2012-10-23T12:26:59Z in response to SystemAdminRebelinux,
In the past when I have had "not on preferred path" issues specifically under load, it has been due to a partially faulty Fibre cable. It may not be relevant in your case but still worth checking.
First capture the support logs from your FC switches as they are, then clear (reset) the port error counts on the external FC ports. In theory there should be no problems with the internal ports as they are through the back-plane of the chassis....
Keep any eye on the error counts, don't expect zero as it is not un-usually to see errors. However compare the error counts across ports and see if any specific ports stands out for high errors. Example: If you are seeing tens of thousands of errors on a specific port then there are issues ;-).
Don't divert all our energy to this, just worth reviewing.
Have you had a chance to note down some details on the issue?
1. What LUN's are effected?
2. What date's and time's?
3. Under what load?
4. What overlaps with issue? (Batch processing period, Backup period, Login Storm period?, "VMware Event", "Host Event" (E.G: Reboot)).
If there is a pattern, have you had a chance to isolate any infrastructure components? (E.G: Hosts...)
Can you let us know when you did the firmware update on the storage? and if you have made any other changes to the storage or VMware environment at this time?
As mentioned, send through the data on the chassis and the switches...
jfrench 270001FJXY9 PostsACCEPTED ANSWER
Re: DS3512 prefered path changing intermittently issue .2012-10-23T14:36:19Z in response to BerniebgbbBernie,
Thanks for the response but at this point I was just looking to see other people's resolutions for this issue. I am currently dealing with IBM but no resolution has been found yet.
Re: DS3512 prefered path changing intermittently issue .2012-11-07T20:23:22Z in response to BerniebgbbI can confirm that disabling VAAI DataMover.HardwareAcceleratedMove and DataMover.HardwareAcceleratedInit resolve my issues of intermittent controller path changes. No more errors seen since 2 weeks. Maybe firmware bug?
Re: DS3512 prefered path changing intermittently issue .2012-11-07T20:30:11Z in response to SystemAdminalso forgot to mention possible data losses. Also we fix this disabling VAAI DataMover.HardwareAcceleratedMove and DataMover.HardwareAcceleratedInit