Topic
  • 10 replies
  • Latest Post - ‏2013-01-18T18:06:09Z by richiet
richiet
richiet
11 Posts

Pinned topic DS3524 contoroller issue

‏2013-01-18T08:49:48Z |
Hi All,

We seem to have a controller issue, we have tried putting it back online put with no success, I have attached the logs. If anyone has any ideas we would be most grateful.

Many Thanks

Richard

Attachments

Updated on 2013-01-18T18:06:09Z at 2013-01-18T18:06:09Z by richiet
  • SystemAdmin
    SystemAdmin
    4779 Posts

    Re: DS3524 contoroller issue

    ‏2013-01-18T10:28:54Z  
    There is no problems with controllers itself. Recovery guru message "says" that hosts can't access Logical Volume "2" using preffered controller A and switched to controller B
    Logical Drive name: 2
    Logical Drive status: Optimal
    Preferred owner: Controller in slot A
    Current owner: Controller in slot B
    But controller A is fine.
    Controller in Enclosure 99, Slot A
    Status: Online
    But all Fiber ports on it are down
    Host interface: Fibre
    Host Interface Card(HIC):
    Channel: 3
    Data rate control: Auto
    Link status: Down
    Host interface: Fibre
    Host Interface Card(HIC):
    Channel: 4
    Data rate control: Auto
    Link status: Down
    Host interface: Fibre
    Host Interface Card(HIC):
    Channel: 5
    Data rate control: Auto
    Link status: Down
    Host interface: Fibre
    Host Interface Card(HIC):
    Channel: 6
    Data rate control: Auto
    Link status: Down
    How hosts are connected? What king of FC switches do you use?
    Verify Fiber connectivity path from hosts to Storage Controller A.
  • richiet
    richiet
    11 Posts

    Re: DS3524 contoroller issue

    ‏2013-01-18T10:51:05Z  
    There is no problems with controllers itself. Recovery guru message "says" that hosts can't access Logical Volume "2" using preffered controller A and switched to controller B
    Logical Drive name: 2
    Logical Drive status: Optimal
    Preferred owner: Controller in slot A
    Current owner: Controller in slot B
    But controller A is fine.
    Controller in Enclosure 99, Slot A
    Status: Online
    But all Fiber ports on it are down
    Host interface: Fibre
    Host Interface Card(HIC):
    Channel: 3
    Data rate control: Auto
    Link status: Down
    Host interface: Fibre
    Host Interface Card(HIC):
    Channel: 4
    Data rate control: Auto
    Link status: Down
    Host interface: Fibre
    Host Interface Card(HIC):
    Channel: 5
    Data rate control: Auto
    Link status: Down
    Host interface: Fibre
    Host Interface Card(HIC):
    Channel: 6
    Data rate control: Auto
    Link status: Down
    How hosts are connected? What king of FC switches do you use?
    Verify Fiber connectivity path from hosts to Storage Controller A.
    There are 4 hosts, directly connected by fibre to the storage with no switches. One to each fibre port on controller A and B. The hosts are running vmware.
  • richiet
    richiet
    11 Posts

    Re: DS3524 contoroller issue

    ‏2013-01-18T10:53:28Z  
    • richiet
    • ‏2013-01-18T10:51:05Z
    There are 4 hosts, directly connected by fibre to the storage with no switches. One to each fibre port on controller A and B. The hosts are running vmware.
    Also the fibre indictor light are on and green, the same indications as on contorller B.
  • SystemAdmin
    SystemAdmin
    4779 Posts

    Re: DS3524 contoroller issue

    ‏2013-01-18T11:31:46Z  
    • richiet
    • ‏2013-01-18T10:53:28Z
    Also the fibre indictor light are on and green, the same indications as on contorller B.
    First of all, this configuration is not supported by Vmware. According IBM documents and Vmware compatible lists, DS3500 storage in FC connectivity is supported only using switches. It doesn't mean that it will not work, just only if you'll try to request official support from ether IBM or Vmware, you'll get response that this configuration is not supported. Practically you may have problems with this configuration in case when some path from server to storage fails (i.e. one HBA or SFP dies or some cable problems). Switching to the secondary path may last too long and Vmware can lost access to datastore. Volumes may intermittently switch to non proffered controller. In case of SAN switched environment each host HBA (FC controller) are connected cross to both storage controllers (A and B) simultaneous.
    But in you current situation you have to verify why FC ports on Controller A are down. Most strange is that you say that all ports light green. Can you verify just now Controller A FC ports status in Storage Manager? May be just when you collected logs were problems?
    1. Check SFP. Does the same SFP are in both Controllers? DS3500 supports only one type of SFP, i don't remember manufactured. If you'll search this forum, you may found a discussion related to problems with FC connectivity.
    2. If storage FC daughter card was installed later, not on factory, you may need to reflash storage firmware.
    http://www-947.ibm.com/support/entry/portal/docdisplay?lndocid=MIGR-5091899
    3. Reseat FC daughter card in controller A.
  • SystemAdmin
    SystemAdmin
    4779 Posts

    Re: DS3524 contoroller issue

    ‏2013-01-18T11:34:50Z  
    First of all, this configuration is not supported by Vmware. According IBM documents and Vmware compatible lists, DS3500 storage in FC connectivity is supported only using switches. It doesn't mean that it will not work, just only if you'll try to request official support from ether IBM or Vmware, you'll get response that this configuration is not supported. Practically you may have problems with this configuration in case when some path from server to storage fails (i.e. one HBA or SFP dies or some cable problems). Switching to the secondary path may last too long and Vmware can lost access to datastore. Volumes may intermittently switch to non proffered controller. In case of SAN switched environment each host HBA (FC controller) are connected cross to both storage controllers (A and B) simultaneous.
    But in you current situation you have to verify why FC ports on Controller A are down. Most strange is that you say that all ports light green. Can you verify just now Controller A FC ports status in Storage Manager? May be just when you collected logs were problems?
    1. Check SFP. Does the same SFP are in both Controllers? DS3500 supports only one type of SFP, i don't remember manufactured. If you'll search this forum, you may found a discussion related to problems with FC connectivity.
    2. If storage FC daughter card was installed later, not on factory, you may need to reflash storage firmware.
    http://www-947.ibm.com/support/entry/portal/docdisplay?lndocid=MIGR-5091899
    3. Reseat FC daughter card in controller A.
    JDSU SFP
    https://www.ibm.com/developerworks/forums/thread.jspa?messageID=14900235
  • richiet
    richiet
    11 Posts

    Re: DS3524 contoroller issue

    ‏2013-01-18T12:01:45Z  
    JDSU SFP
    https://www.ibm.com/developerworks/forums/thread.jspa?messageID=14900235
    Hi,

    Thanks for the info, the fibre lights are defiantly lit, please see the attached file which is saying that the controller A cannot be contacted.

    All the SFP's are the same, I unfortunatly do not know whether the daughter card came shipped from ibm with the array.

    Kind regards

    Richard
  • SystemAdmin
    SystemAdmin
    4779 Posts

    Re: DS3524 contoroller issue

    ‏2013-01-18T15:53:00Z  
    • richiet
    • ‏2013-01-18T12:01:45Z
    Hi,

    Thanks for the info, the fibre lights are defiantly lit, please see the attached file which is saying that the controller A cannot be contacted.

    All the SFP's are the same, I unfortunatly do not know whether the daughter card came shipped from ibm with the array.

    Kind regards

    Richard
    bad news for you.
    I didn't observer this message

    Controller in Enclosure 99, Slot A
    Status: Online
    Host Interface(s): Unable to retrieve latest data; using last known state.

    If device is under warranty, i would suggest to ask for a support.
    If'll prefer to "play" by yourself, i would try to reseat Controller A and it FC daughter card. And reflash to 7.77.83 FW version.
  • richiet
    richiet
    11 Posts

    Re: DS3524 contoroller issue

    ‏2013-01-18T16:56:15Z  
    bad news for you.
    I didn't observer this message

    Controller in Enclosure 99, Slot A
    Status: Online
    Host Interface(s): Unable to retrieve latest data; using last known state.

    If device is under warranty, i would suggest to ask for a support.
    If'll prefer to "play" by yourself, i would try to reseat Controller A and it FC daughter card. And reflash to 7.77.83 FW version.
    Hi Thankyou for you persevering, we have tried reseating the controller card with no joy, in fact putting the controller back online, hit the I/O hard and a few virtual servers locked.

    Think it may need a new controller card? :(
  • SystemAdmin
    SystemAdmin
    4779 Posts

    Re: DS3524 contoroller issue

    ‏2013-01-18T17:46:18Z  
    • richiet
    • ‏2013-01-18T16:56:15Z
    Hi Thankyou for you persevering, we have tried reseating the controller card with no joy, in fact putting the controller back online, hit the I/O hard and a few virtual servers locked.

    Think it may need a new controller card? :(
    I saw you set offline/online controller. But did you physically pull out controller from storage, then eject FC daughter card from it and then reassemble them again? I mean this procedure. Sometime just an reassembling could help.
    If wont help, try to reflash device. But not with latest 7.84 fw version, try latest in 7.77 release which i mentioned.
  • richiet
    richiet
    11 Posts

    Re: DS3524 contoroller issue

    ‏2013-01-18T18:06:09Z  
    I saw you set offline/online controller. But did you physically pull out controller from storage, then eject FC daughter card from it and then reassemble them again? I mean this procedure. Sometime just an reassembling could help.
    If wont help, try to reflash device. But not with latest 7.84 fw version, try latest in 7.77 release which i mentioned.
    Thanks, the controller has been reseated after the support logs were gathered so it doesn't show. But it has had no effect on the current fault.

    Not too sure I want to flash it, in its current state, I only ever really do firmware when no faults are on the equipment.

    we have sourced a replacement controller, but the fibre board has proved more of a problem.

    fingers crossed a controller change sorts it.