Topic
  • 12 replies
  • Latest Post - ‏2014-07-26T08:23:26Z by pwuestefeld
pwuestefeld
pwuestefeld
18 Posts

Pinned topic PowerKVM IPMI sessions lost

‏2014-07-16T20:39:23Z |

Hi all,

we noticed an issue with IPMI on our S822L system with firmware FW810.01 (SV810_058).

The system was running fine for a while (~ three weeks) and lost then suddenly the IPMI connection. When trying to connect again we saw these messages:

psesles01:~ # ipmitool -I lanplus -H my.powerkvm.address -P mypw sol activate
Error: Unable to establish IPMI v2 / RMCP+ session
Error: No response activating SOL payload

However, the system itself ran fine and the IP address was pingable:

psesles01:~ # ping 10.10.113.138
PING 10.10.113.138 (10.10.113.138) 56(84) bytes of data.
64 bytes from 10.10.113.138: icmp_seq=1 ttl=63 time=5.13 ms
64 bytes from 10.10.113.138: icmp_seq=2 ttl=63 time=0.274 ms

This was also true with the next function we tried to use:

psesles01:~ # ipmitool -I lanplus -H my.powerkvm.address -P mypw power status
Error: Unable to establish IPMI v2 / RMCP+ session
Unable to get Chassis Power Status

psesles01:~ # ping 10.10.113.138
PING 10.10.113.138 (10.10.113.138) 56(84) bytes of data.
64 bytes from 10.10.113.138: icmp_seq=1 ttl=63 time=0.316 ms
64 bytes from 10.10.113.138: icmp_seq=2 ttl=63 time=0.276 ms

We then shut down PowerKVM, switched off the machine and resetted the FSP via ASMI. Starting the machine worked, but after a short while (~ 3h) the very same situation occurred again. Up to now we had to reboot the machine twice since after equally short times of operation the IP MI connection failed again.

Are there any methods known to restart IPMI without taking down all of the VMs and the KVM host? A bit more stability of IPMI connections would also do no harm :-)

Thanks for any information that may help in this situation -

Peter

  • bofferdn
    bofferdn
    2 Posts

    Re: PowerKVM IPMI sessions lost

    ‏2014-07-16T21:32:38Z  

    Peter,

     

    Sorry to see you have hit some IPMI instability; the most effective way to get to the root of this problem is to wait until the problem hits, initiate a service processor dump from the ASMI web-based menu system, offload it to a USB key, and transfer it to a personal device.  Once the dump has been saved, please open up a call with IBM support, which will guide you through the process of submitting the dump to IBM. 

     

    To initiate a service processor dump, you can follow IBM's infocenter instructions here:

    http://www-01.ibm.com/support/knowledgecenter/POWER8/p8hby/spdump.htm?cp=POWER8%2F1-5-2-2-7-3

     

    To transfer the dump to a USB key, follow the instructions here, again in infocenter:

    http://www-01.ibm.com/support/knowledgecenter/POWER8/p8hby/usbenabledservice.htm?cp=POWER8%2F1-5-2-2-7-15

     

    Good luck!

    Nick

  • pwuestefeld
    pwuestefeld
    18 Posts

    Re: PowerKVM IPMI sessions lost

    ‏2014-07-16T23:53:00Z  

    Thanks for the info - will have an eye for it if I'm back home. At the moment it's a bit hard to do since I'm in Austin and the machine is in Germany :-)

    Greetings,

    Peter

  • PowerLinuxTeam
    PowerLinuxTeam
    9 Posts

    Re: PowerKVM IPMI sessions lost

    ‏2014-07-17T11:51:35Z  

    Peter, another off-line suggestion..

    "When I've seen this happen with IPMI I have simply done the following...e.g."

     

    ipmitool -I lanplus -H my.powerkvm.address -P mypw sol deactivate
    
    -and then-
    ipmitool -I lanplus -H my.powerkvm.address -P mypw sol activate
    
    That still begs the question why things are happening..    still pursuing that...  Nick's suggestion above is the best bet in the meantime..
  • ThinkOpenly
    ThinkOpenly
    42 Posts

    Re: PowerKVM IPMI sessions lost

    ‏2014-07-17T15:41:04Z  

    Peter, another off-line suggestion..

    "When I've seen this happen with IPMI I have simply done the following...e.g."

     

    <pre dir="ltr">ipmitool -I lanplus -H my.powerkvm.address -P mypw sol deactivate </pre>
    -and then-
    <pre dir="ltr">ipmitool -I lanplus -H my.powerkvm.address -P mypw sol activate </pre>
    That still begs the question why things are happening..    still pursuing that...  Nick's suggestion above is the best bet in the meantime..

    In my experience, when one is "unable to establish IPMI" session, doing "sol deactivate" will also fail.

    I do think, however, that just restarting the FSP might be sufficient, but there is no need to bring down the entire system to do that.  It can be done in isolation, leaving KVM and the guests running. [ed. the system must be powered off.]

    Updated on 2014-07-21T15:24:54Z at 2014-07-21T15:24:54Z by ThinkOpenly
  • jscheel
    jscheel
    67 Posts

    Re: PowerKVM IPMI sessions lost

    ‏2014-07-17T16:24:04Z  

    Thanks for the info - will have an eye for it if I'm back home. At the moment it's a bit hard to do since I'm in Austin and the machine is in Germany :-)

    Greetings,

    Peter

    Peter, as a customer, you should open a defect for this problem with support.  They can also help you collect data on the problem.

  • pwuestefeld
    pwuestefeld
    18 Posts

    Re: PowerKVM IPMI sessions lost

    ‏2014-07-17T17:07:51Z  

    Peter, another off-line suggestion..

    "When I've seen this happen with IPMI I have simply done the following...e.g."

     

    <pre dir="ltr">ipmitool -I lanplus -H my.powerkvm.address -P mypw sol deactivate </pre>
    -and then-
    <pre dir="ltr">ipmitool -I lanplus -H my.powerkvm.address -P mypw sol activate </pre>
    That still begs the question why things are happening..    still pursuing that...  Nick's suggestion above is the best bet in the meantime..

    We also experienced that - it just can't connect to IPMI and thus a "deactivate" doesn't get through, let alone an "activate".

    Should that show up the next time I prefer to go with Nick's suggestion since I'd like to have that behaviour analyzed and probably solved. Just resetting the FSP doesn't achieve that, although it might help in some situations. At least as a fallback it can be used.

  • walesaf
    walesaf
    2 Posts

    Re: PowerKVM IPMI sessions lost

    ‏2014-07-18T05:58:23Z  

    Peter,

    1) In your case ipmitool running on x86 ? or Power ? As few critical fixes on power related to ipmi sessions are yet to make into the build.

    2) You are facing this issue only after every few hours ? or even in any shorter durations ?

    While we get into the root of the issue, you can reboot the FSP alone without needing to bring the guests of KVM host down.

  • pwuestefeld
    pwuestefeld
    18 Posts

    Re: PowerKVM IPMI sessions lost

    ‏2014-07-21T14:35:38Z  

    In my experience, when one is "unable to establish IPMI" session, doing "sol deactivate" will also fail.

    I do think, however, that just restarting the FSP might be sufficient, but there is no need to bring down the entire system to do that.  It can be done in isolation, leaving KVM and the guests running. [ed. the system must be powered off.]

    No, that won't work - the FSP doesn't let you reset it while the machine is up and running. You need to shutdown all guests and the host (switching off the machine) and after that a reset of the FSP is possible.

  • pwuestefeld
    pwuestefeld
    18 Posts

    Re: PowerKVM IPMI sessions lost

    ‏2014-07-21T14:41:15Z  
    • walesaf
    • ‏2014-07-18T05:58:23Z

    Peter,

    1) In your case ipmitool running on x86 ? or Power ? As few critical fixes on power related to ipmi sessions are yet to make into the build.

    2) You are facing this issue only after every few hours ? or even in any shorter durations ?

    While we get into the root of the issue, you can reboot the FSP alone without needing to bring the guests of KVM host down.

    We are running on Power (a PowerKVM machine, IBM mod. S822L).

    The issue did show up in no special time order - firt the machine was up a few weeks before the firt occurence of this. Afterwards, the symptom came up three-four times, just a few hours apart. Now it ran around a week before we faced the issue again.

    We now took a Service Processor Dump as Nick suggested above. Please bear in mind that a USB stick of 8GB can already be an issue, showing just ~800MB size and 400MB of that free. An offload to such a stick will fail. We used a 4GB stick with FAT32 formatting and were just after that successful with offloading.

    The dump is now on its way into IBM ....

  • ThinkOpenly
    ThinkOpenly
    42 Posts

    Re: PowerKVM IPMI sessions lost

    ‏2014-07-21T15:23:28Z  

    No, that won't work - the FSP doesn't let you reset it while the machine is up and running. You need to shutdown all guests and the host (switching off the machine) and after that a reset of the FSP is possible.

    I stand corrected. It does indeed require the system powered off. Apologies for the confusion.

  • tomjose
    tomjose
    1 Post

    Re: PowerKVM IPMI sessions lost

    ‏2014-07-23T10:13:59Z  

    We are running on Power (a PowerKVM machine, IBM mod. S822L).

    The issue did show up in no special time order - firt the machine was up a few weeks before the firt occurence of this. Afterwards, the symptom came up three-four times, just a few hours apart. Now it ran around a week before we faced the issue again.

    We now took a Service Processor Dump as Nick suggested above. Please bear in mind that a USB stick of 8GB can already be an issue, showing just ~800MB size and 400MB of that free. An offload to such a stick will fail. We used a 4GB stick with FAT32 formatting and were just after that successful with offloading.

    The dump is now on its way into IBM ....

    Hello Peter,

    From your last post i understand you are using ipmitool on Power Architecture. I would be interested to know the version of the ipmitool you are using. Run the ipmitool with -V option to know the ipmitool version(eg: ipmitool -V)

    There are some specific bugs with ipmitool on Power Architecture like the one mentioned below. This impacts IPMI SOL behaviour.

    http://sourceforge.net/p/ipmitool/bugs/304/

    This fix is available in ipmitool version 1.8.14 onwards. In case you are running an older version of ipmitool, please feel free to upgrade to the latest version and let us know if you are hitting the same issue.

    Regards,

    Tom

     

    Updated on 2014-07-23T10:18:46Z at 2014-07-23T10:18:46Z by tomjose
  • pwuestefeld
    pwuestefeld
    18 Posts

    Re: PowerKVM IPMI sessions lost

    ‏2014-07-26T08:23:26Z  
    • tomjose
    • ‏2014-07-23T10:13:59Z

    Hello Peter,

    From your last post i understand you are using ipmitool on Power Architecture. I would be interested to know the version of the ipmitool you are using. Run the ipmitool with -V option to know the ipmitool version(eg: ipmitool -V)

    There are some specific bugs with ipmitool on Power Architecture like the one mentioned below. This impacts IPMI SOL behaviour.

    http://sourceforge.net/p/ipmitool/bugs/304/

    This fix is available in ipmitool version 1.8.14 onwards. In case you are running an older version of ipmitool, please feel free to upgrade to the latest version and let us know if you are hitting the same issue.

    Regards,

    Tom

     

    Hi Tom,

     

    seems you are the one who found out whats going on here - no problem with PowerKVM but with ipmitool connecting.

    We used a SLES11SP3 ppc64 LPAR with ipmitool version 1.8.11 to connect to the PowerKVM host. Apparently that version does not contain the fixes you mentioned.  Also, RHEL 6.5 is on the very same version. At the moment we are reluctant to upgrade the installations as that would leave the supported software path for both. Instead we are using a Ubuntu installation on x86 as fallback ... not a compelling solution.

    Thanks for your help!

    Greetz,

    Peter