Topic
IC4NOTICE: developerWorks Community will be offline May 29-30, 2015 while we upgrade to the latest version of IBM Connections. For more information, read our upgrade FAQ.
4 replies Latest Post - ‏2013-03-29T15:47:46Z by SystemAdmin
SystemAdmin
SystemAdmin
3234 Posts
ACCEPTED ANSWER

Pinned topic Unable to communicate with the controller

‏2012-11-06T17:48:02Z |
Dears,

After a power outage of one of our Bladecenter S Chassis, all the HS22 blade servers were failed to connect to the SAS drives. The SCM could connect to the SAS switch but not to the RAID subsystem.

After some diagnosis, I found that everytime after a restart of the SAS switch, I can issure some CLI commands for about 5 minutes. After that, no CLI command can be proceeded and all responded with 'Unable to communicate with the controller. Please try again.'

During that 5 minutes period, I can be able to use ftx.exe to transfer uimage_H-2.4.20.12 to the RAID controller but nothing happened after issuing 'firmwareupgrade -target controller -nopreverify'. I have tried to use ftx.exe to transfer S0rc124011 but it took more than 5 minutes to upload and hence the process was failed in the middle.

Compare the swversion output with another working Bladecenter S, I found 2 differences in that the "SES version" is empty (instead of 0107) and 'BBU FirmwareRev' is 'UNAVAILABLE' (instead of 58.0). I believe the problem may be related to the corruption of the firmwares.

Would appreciate for any suggestions.

Followings are some of the CLI commands I have issued after the switch was restarted:
<CLI> list controller
_______________________________________________________________________
Ctlr# Controller Status Ports LUNs
_____________ ________________ ____________________ _________ _________
0 Ctlr0 STARTING 1 --
1 Ctlr1 SERVICE 1 --
_____________ ________________ ____________________ _________ _________


<CLI> firmwareupgrade -getstatus
Ctlr 0 Firmware Upgradation Status is FW_UPGRADE_INITIALIZING
Ctlr 1 Firmware Upgradation Status is FW_UPGRADE_NO_PEER

Drive Upgrade Cumulative Status is IDLE

<CLI> swversion
Software version : H-2.1.2.4
UBoot version : H-1.1.4.6
OS version : H-2.4.20.12
SES version :
BMC version : S0BT10A 0121 02/08/2010
FPGA version : 01.07
CPLD version : S0CP00A C00A 01/01/2000
SAS switch version : S0SW01D R107 12/17/2009
BBU FirmwareRev : UNAVAILABLE
Package Build No : 1.2.3.006

<CLI> detail controller -ctlr 0
Controller Information :

UltraSlice Version : ALC3300
Software version : H-2.1.2.4
Uboot Version : H-1.1.4.6
OS version : H-2.4.20.12
SES version :
BMC version : S0BT10A 0121 02/08/2010
FPGA version : 01.07
CPLD version : S0CP00A C00A 01/01/2000
SAS switch version : S0SW01D R107 12/17/2009
Chassis Serial Number : 9988776
WWN : 5005076b074db87f

Chassis Machine Type/Model : 88861NA
SAS RAID Controller Module Part Number :
SAS RAID Controller Module FRU Part Number :
SAS RAID Controller Module Serial Number :
Machine Signature : 8886-0000-0000-C07E
SAS RAID Controller Module Location : IO Bay 3
Manufacturer ID : IBM
SAS RAID Controller Module Hardware Revision ID : 7
SAS RAID Controller Module Hardware Product ID : 00a5
MAC Address (upto 8) : 00:1A:64:9E:1A:FB
Target port WWN : 5005076b074db820
Target port protocol : SAS
Target port speed : 3Gbps

Current Status : STARTING
System Hardware Configuration Mode : Single
BBU information is not available.

Associated Volumes currently serviced by this controller :

Error : list associated volumes command failed
Reason: Controller is busy

<CLI> shutdown -ctlr 0 -state servicemode
Error : shutdown command failed
Reason: Request to change controller state failed
Updated on 2013-03-29T15:47:46Z at 2013-03-29T15:47:46Z by SystemAdmin
  • SystemAdmin
    SystemAdmin
    3234 Posts
    ACCEPTED ANSWER

    Re: Unable to communicate with the controller

    ‏2012-11-21T13:01:01Z  in response to SystemAdmin
    Hello Brio,

    Analysis/diagnosis may take time. I suggest you to log a hardware call with your country technical support.
  • SystemAdmin
    SystemAdmin
    3234 Posts
    ACCEPTED ANSWER

    Re: Unable to communicate with the controller

    ‏2013-03-29T07:13:16Z  in response to SystemAdmin
    I have the same issue

    The first controller says:

    MontaVista(R) Linux(R) Professional Edition 3.1

    <CLI> list controller
    Current Machine Local Time: 03/29/2013 06:58:18 AM
    _______________________________________________________________________
    Ctlr# Controller Status Ports LUNs
    _____________ ________________ ____________________ _________ _________
    0 Ctlr0 SURVIVOR 1 20
    1 Ctlr1 FAILED - --
    _____________ ________________ ____________________ _________ _________

    <CLI>

    <CLI> firmwareupgrade -getstatus
    Current Machine Local Time: 03/29/2013 07:01:11 AM
    Ctlr 0 Firmware Upgradation Status is FW_UPGRADE_IDLE
    Ctlr 1 Firmware Upgradation Status is FW_UPGRADE_NO_PEER

    Drive Upgrade Cumulative Status is IDLE

    <CLI>
    First (BAY3) SAS and RAID modules works OK
    The second (BAY4) SAS module works OK, but RAID module subsystem FAILED.
    The interesting part is that i have ping to second RAID module , and I CAN to login, but when I try to place some command module return:
    "Unable to communicate with the controller. Please try again."
    The controller can not be put in service/recovery mode.
    When controller is rebooted, it can communicate for a few seconds, but after that fail again.
    I saw that some software moduel restart netwok configuration.
    I try to flash firmware from linux, windows, SCM without success.
    I wonder, what this FW_UPGRADE_NO_PEER mean?
    NEED HELP
    (or pinout for small usb RS232 ports on the back) :D

    Best Regards
    • SystemAdmin
      SystemAdmin
      3234 Posts
      ACCEPTED ANSWER

      Re: Unable to communicate with the controller

      ‏2013-03-29T14:05:37Z  in response to SystemAdmin
      Have you reseated the controllers? It seems you have a bad controller, possibly from a corrupt flash, but as mentioned before I would contact support and open a hardware ticket if reseating the controller makes no difference. Also, I wouldn't bother using SCM. Just use the CLI to work with the controllers. Much more stable and you won't have to fight SCM and the controller issue at the same time.
  • SystemAdmin
    SystemAdmin
    3234 Posts
    ACCEPTED ANSWER

    Re: Unable to communicate with the controller

    ‏2013-03-29T15:47:46Z  in response to SystemAdmin
    Try to reseat also BBU - battery and also media tray. There can be some troubles with backup battery connectivity.