Topic
  • 8 replies
  • Latest Post - ‏2013-04-05T20:29:27Z by SystemAdmin
SystemAdmin
SystemAdmin
6772 Posts

Pinned topic Unable to log in into DataPower - appliance completely locked

‏2013-04-04T12:33:32Z |
We have a DataPower XI52 running firmware 5.0.0.5.

A couple of weeks ago we found a bug in that firmware, where by importing a domain from another DataPower appliance, the XI52 would reload. After creating a PMR with IBM, I was told to log in as 'admin' and run the command 'all-domains disabled'. After several attempts I was able to log in as 'admin' using the 'Serial over LAN' access as described here http://pic.dhe.ibm.com/infocenter/wsdatap/v5r0m0/index.jsp?topic=%2Fcom.ibm.dp.xi.doc%2Fadministratorsguide.xi50170.htm&path%3D4_2_0_4_2_1_1, since that DataPower appliance is at a remote Data Center location.

While I was on vacation last week, the problem reoccurred (probably someone tried to import one of those objects again), but this time I have not been able to log in.
What happens is that, when connecting to the DataPower using 'Serial over LAN', I receive the expected 'login:' (and I enter 'admin') and 'Password:' (and I enter the correct password) prompts, but then nothing happens (I do know that the password I am entering is correct; if I type random sequence of characters as the password, I receive the 'login:' prompt right away).
Also using the 'Serial over LAN' access I am able to power cycle the DataPower remotely, but that didn't help either, since when the DataPower comes back up all the domains are still enabled and the reload cycle keeps repeating.

I created PMR 60285,999,000 with IBM, but unfortunately they haven't been able to help me regain control of that DataPower, and honestly I don't know what else to do.

Any help, suggestion is welcome.

Thanks in advance,
Franco Venturi
Updated on 2013-04-05T20:29:27Z at 2013-04-05T20:29:27Z by SystemAdmin
  • SystemAdmin
    SystemAdmin
    6772 Posts

    Re: Unable to log in into DataPower - appliance completely locked

    ‏2013-04-04T14:50:36Z  
    > fventuri wrote:
    > We have a DataPower XI52 running firmware 5.0.0.5.
    >
    > A couple of weeks ago we found a bug in that firmware, where by importing a domain from another DataPower appliance, the XI52 would reload. After creating a PMR with IBM, I was told to log in as 'admin' and run the command 'all-domains disabled'. After several attempts I was able to log in as 'admin' using the 'Serial over LAN' access as described here http://pic.dhe.ibm.com/infocenter/wsdatap/v5r0m0/index.jsp?topic=%2Fcom.ibm.dp.xi.doc%2Fadministratorsguide.xi50170.htm&path%3D4_2_0_4_2_1_1, since that DataPower appliance is at a remote Data Center location.
    >
    > While I was on vacation last week, the problem reoccurred (probably someone tried to import one of those objects again), but this time I have not been able to log in.
    > What happens is that, when connecting to the DataPower using 'Serial over LAN', I receive the expected 'login:' (and I enter 'admin') and 'Password:' (and I enter the correct password) prompts, but then nothing happens (I do know that the password I am entering is correct; if I type random sequence of characters as the password, I receive the 'login:' prompt right away).
    > Also using the 'Serial over LAN' access I am able to power cycle the DataPower remotely, but that didn't help either, since when the DataPower comes back up all the domains are still enabled and the reload cycle keeps repeating.
    >
    > I created PMR 60285,999,000 with IBM, but unfortunately they haven't been able to help me regain control of that DataPower, and honestly I don't know what else to do.
    >
    Franco,
    Do you have backup or privileged account setup for the appliance? If yes than you can reset the admin password. However if you don't have backup account setup than I am afraid you have to use exiting or new PMR and you may have to ship the appliance to IBM.

    Regards,
    Kumar
  • msiebler
    msiebler
    141 Posts

    Re: Unable to log in into DataPower - appliance completely locked

    ‏2013-04-04T15:38:02Z  
    > fventuri wrote:
    > We have a DataPower XI52 running firmware 5.0.0.5.
    >
    > A couple of weeks ago we found a bug in that firmware, where by importing a domain from another DataPower appliance, the XI52 would reload. After creating a PMR with IBM, I was told to log in as 'admin' and run the command 'all-domains disabled'. After several attempts I was able to log in as 'admin' using the 'Serial over LAN' access as described here http://pic.dhe.ibm.com/infocenter/wsdatap/v5r0m0/index.jsp?topic=%2Fcom.ibm.dp.xi.doc%2Fadministratorsguide.xi50170.htm&path%3D4_2_0_4_2_1_1, since that DataPower appliance is at a remote Data Center location.
    >
    > While I was on vacation last week, the problem reoccurred (probably someone tried to import one of those objects again), but this time I have not been able to log in.
    > What happens is that, when connecting to the DataPower using 'Serial over LAN', I receive the expected 'login:' (and I enter 'admin') and 'Password:' (and I enter the correct password) prompts, but then nothing happens (I do know that the password I am entering is correct; if I type random sequence of characters as the password, I receive the 'login:' prompt right away).
    > Also using the 'Serial over LAN' access I am able to power cycle the DataPower remotely, but that didn't help either, since when the DataPower comes back up all the domains are still enabled and the reload cycle keeps repeating.
    >
    > I created PMR 60285,999,000 with IBM, but unfortunately they haven't been able to help me regain control of that DataPower, and honestly I don't know what else to do.
    >
    Franco,
    Do you have backup or privileged account setup for the appliance? If yes than you can reset the admin password. However if you don't have backup account setup than I am afraid you have to use exiting or new PMR and you may have to ship the appliance to IBM.

    Regards,
    Kumar
    How long did you wait at the serial while the device was hung? If it is crashing repeatedly on startup; it should eventually go into the failsafe mode; where you could recover it.
  • SystemAdmin
    SystemAdmin
    6772 Posts

    Re: Unable to log in into DataPower - appliance completely locked

    ‏2013-04-04T17:27:27Z  
    • msiebler
    • ‏2013-04-04T15:38:02Z
    How long did you wait at the serial while the device was hung? If it is crashing repeatedly on startup; it should eventually go into the failsafe mode; where you could recover it.
    @Kumar - The problem is not with the 'admin' account or its password - the problem is that the DataPower reloads/reboots before the login process continues and I never receive the 'xi52# ' prompt from which I can type further commands to disable all the domains

    @msiebler: after I posted my initial request here, IBM gave me the instructions to enter 'failsafe' mode (just type the command 'failsafe' at the 'DPOS> ' prompt after power cycling the appliance - you'll have to enter ESC at some point during the boot sequence when it tells you). Unfortunately the 'failsafe' command prompts then for a password, and this password doesn't seem to be the regular 'admin' password, since no matter what I type there, the DataPower goes back rebooting in normal mode.

    Franco
  • SystemAdmin
    SystemAdmin
    6772 Posts

    Re: Unable to log in into DataPower - appliance completely locked

    ‏2013-04-04T18:07:41Z  
    @Kumar - The problem is not with the 'admin' account or its password - the problem is that the DataPower reloads/reboots before the login process continues and I never receive the 'xi52# ' prompt from which I can type further commands to disable all the domains

    @msiebler: after I posted my initial request here, IBM gave me the instructions to enter 'failsafe' mode (just type the command 'failsafe' at the 'DPOS> ' prompt after power cycling the appliance - you'll have to enter ESC at some point during the boot sequence when it tells you). Unfortunately the 'failsafe' command prompts then for a password, and this password doesn't seem to be the regular 'admin' password, since no matter what I type there, the DataPower goes back rebooting in normal mode.

    Franco
    Sorry I have miss understood the question. I am wondering what will happen if you use default admin password and there is chance that entire configs will be wipe out. Do you have the secure/ backup of entire appliance including certs?

    Regards,
    Kumar
  • SystemAdmin
    SystemAdmin
    6772 Posts

    Re: Unable to log in into DataPower - appliance completely locked

    ‏2013-04-04T18:47:47Z  
    Sorry I have miss understood the question. I am wondering what will happen if you use default admin password and there is chance that entire configs will be wipe out. Do you have the secure/ backup of entire appliance including certs?

    Regards,
    Kumar
    @Kumar: I do have a complete secure backup of the appliance from the night before this problem occurred (I think it was a week or so ago). IBM should be calling me shortly with the next steps, so I'll let you know how it goes.
  • SystemAdmin
    SystemAdmin
    6772 Posts

    Re: Unable to log in into DataPower - appliance completely locked

    ‏2013-04-05T12:35:44Z  
    It turns out that the password for the DPOS 'failsafe' mode is an internal IBM password, not the 'admin' password. Anyhow with the help of IBM I was finally able to log in in 'failsafe' mode, wipe out the configuration, reboot, restore the basic networking for the WebGUI and the CLI, and validate that I can log back in normally.
    Unfortunately this approach seems to be destructive because I lost the configuration (fortunately I am able to restore from the last good secure backup) and also all the logs, which means that I may never know what/who caused this problem in first place.

    Thanks everyone here for their help; I really appreciated it.

    Franco
  • msiebler
    msiebler
    141 Posts

    Re: Unable to log in into DataPower - appliance completely locked

    ‏2013-04-05T16:55:43Z  
    It turns out that the password for the DPOS 'failsafe' mode is an internal IBM password, not the 'admin' password. Anyhow with the help of IBM I was finally able to log in in 'failsafe' mode, wipe out the configuration, reboot, restore the basic networking for the WebGUI and the CLI, and validate that I can log back in normally.
    Unfortunately this approach seems to be destructive because I lost the configuration (fortunately I am able to restore from the last good secure backup) and also all the logs, which means that I may never know what/who caused this problem in first place.

    Thanks everyone here for their help; I really appreciated it.

    Franco
    I'm glad you are back up. If you have not done so already; please send in a copy of the bad configuration to the PMR so we can try to recreate it here in our labs. You had mentioned previously; 'by importing a domain from another DataPower appliance,' the box crashed; it would be good if we could get that from you.
    Do you have a hunch what the configuration item was?
  • SystemAdmin
    SystemAdmin
    6772 Posts

    Re: Unable to log in into DataPower - appliance completely locked

    ‏2013-04-05T20:29:27Z  
    • msiebler
    • ‏2013-04-05T16:55:43Z
    I'm glad you are back up. If you have not done so already; please send in a copy of the bad configuration to the PMR so we can try to recreate it here in our labs. You had mentioned previously; 'by importing a domain from another DataPower appliance,' the box crashed; it would be good if we could get that from you.
    Do you have a hunch what the configuration item was?
    msiebler,
    unfortunately I spoke too soon.

    I tried a couple of different secure restores, one from 3/27, and another from 3/20 (the oldest I have) and in both cases I can login as 'admin' with the default 'admin' password, it asks me to change the password, I do that, and then after 3 or 4 minutes (with no messages), I get another 'login:' prompt, and this login again wants the 'admin' password, and it asks me again to change the password - this is all through the 'Serial over LAN' access, because I can't connect via the WebGUI or the ssh CLI.
    Also sometimes the DataPower doesn't respond at all through the 'Serial over LAN' access, which makes me think it is reloading again.

    This is all very frustrating and I am ready to call it a (wasted) week; I'll let you know how it goes on Monday.

    Franco