I mentioned a few days ago that I assembled a server from spare parts but was having trouble with the networking. Well I got the networking problem sorted out (just a weird IBM firewall issue) but ran into an even trickier problem trying to start IBM WebSphere Application Server (WAS) v6.
Here's what happened. I logged in as root and tried to startup WAS with the simple command:
But when I ran this command I got the following error message:
ADMU3011E: Server launched but failed initialization. Server log files should contain failure information.
I looked at the logfiles stored in
SystemOut.logcontained the following exception:
SECJ0336E: Authentication failed for user uid=12345, c=us, ou=bluepages, o=ibm.com because of the following exception: javax.naming.AuthenticationException: [LDAP: error code 49 - Invalid Credentials]
The LDAP syntax and the bluepages reference made it clear that the
startServerscript was attempting to authenticate someone against IBM's corporate directory (which is cleverly called "Bluepages") and failing. I looked at the security configuration file (
WASROOT/profiles/AppSrv01/config/cells/localhostNode01Cell/security.xml) and discovered that indeed WAS global security was enabled and configured to use LDAP authentication against Bluepages. The configured username was that of Ritchie Schacher, my colleague who owned the server before giving it to me. I looked at the timestamp on
security.xmland saw that it hadn't been updated since February (7 months ago) which made everything clear - we at IBM have to update our intranet id and password every six months, so the password stored with Ritchie's username must have been out of date and therefore failing. So it was clear that I either had to change security to use my IBM intranet id / password or temporarily disable WAS global security.
This is where things got really tricky. You see, the typical way of configuring WAS is via the web-based WAS Admin Console. The problem was that the WAS Admin Console needs a running WAS server, but my WAS server couldn't start because of a configuration problem - a classic catch-22! Luckily, the web-based admin console isn't the only way to configure WAS. WAS also provides an interactive command-line admin client called
wsadmin. By default,
wsadminexpects to connect to a running server, but I discovered that you could use it in "offline mode" by starting it with the command:
wsadmin -conntype NONE
Once I was in the
wsadmininteractive command line, I was able to turn off WAS global security with the following commands:
I attempted to start the server once again with
wsadmin> $AdminConfig save
./startServer.sh server1and this time it started successfully. Once the server started, I was able to go into the web-based admin console, change Ritchie's LDAP information to my info, re-enable WAS global security, and I was good to go.
So net, if you find yourself in a situation where a configuration problem is preventing you from starting your appserver, login as root, use the
wsadmintool in disconnected mode to fix the problem and then use web-based admin console to make more sophisticated changes. If things are really screwed up, you can always blow away your profile, or if worse comes to worse, re-install WAS. But usually with a little study, you can do most things via
- WAS InfoCenter: Wsadmin tool reference
- WAS InfoCenter: Configuring security with scripting (to fix the problem I was having)
Ah, the joys of a system administration. :-)
Update: Bobby Woolf linked to this entry and added some additional comments on what the wsadmin tool really is and explained why this solution doesn't present a security risk.
contact me: firstname.lastname@example.org