IBM developerWorks: Wikis - Tivoli Identity Manager

1. *DB2 Primary Server fails.
Approximately 2 minutes after DB2 primary goes down, failover occurs
From db2diag.log:

FUNCTION: DB2 UDB, High Availability Disaster Recovery, hdrSDoTakeover, probe:47007
MESSAGE : Info: Standby has completed takeover (now primary).

2. *WebSphere Timeout messages are logged
from WAS stdout

00000011 TimeoutManage I   WTRN0124I: When the timeout occurred the 
thread with which the transaction is, or was most recently, associated was Thread.
The stack trace of this thread when the timeout occurred was:
        java.net.PlainSocketImpl.socketConnect(Native Method)
        java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
        java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
        java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
        java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
        java.net.Socket.connect(Socket.java:529)
        com.ibm.db2.jcc.t4.y.run(y.java:34)
        java.security.AccessController.doPrivileged(Native Method)

3. *WebSphere hung thread logged
from WAS stdout

00000027 ThreadMonitor W   WSVR0605W: Thread "SIBJMSRAThreadPool : 13" 
(00000072) has been active for 713504 milliseconds and may be hung.  
There is/are 1 thread(s) in total in the server that may be hung.
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
        at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
        at java.net.Socket.connect(Socket.java:529)
        at com.ibm.db2.jcc.t4.y.run(y.java:34)
        at java.security.AccessController.doPrivileged(Native Method)

4. *WebSphere data source reconnection messages
from WAS stdout

00000062 ConnectionEve A   J2CA0056I: 
The Connection Manager received a fatal connection error from the 
Resource Adapter for resource enroleDataSource. 
The exception is: com.ibm.db2.jcc.b.ClientRerouteException: jcc t4 2027 11212 3.50.152 
A connection failed but has been re-established. 
The host name or IP address is <standby.DB2.server> and the service name or port number is <HA designated port>.

5 *DB2 Indoubt Transaction Lock message
from DB2diag.log

FUNCTION: DB2 UDB, lock manager, sqlpldl, probe:1280
MESSAGE : ADM1838W  An application is waiting for a lock held by an indoubt transaction.  
This will cause the application to wait indefinitely. 
Use the LIST INDOUBT TRANSACTIONS command to investigate and resolve the indoubt transactions.

6. *SIB Messaging Engine Shutdown
from WAS stdout

0000007d SibMessage    E   CWSIS1504E: The messaging engine is stopping due to an exception from the data source
00000030 SibMessage    I   itim_bus:App_Cluster.000-itim_bus CWSIT0029I: 
The connection for messaging engine App_Cluster.000-itim_bus in bus itim_bus to messaging engine 
App_Cluster.001-itim_bus stopped.
00000019 HAGroupImpl   I   HMGR0130I: The local member of group 
IBM_hc=App_Cluster,WSAF_SIB_BUS=itim_bus, 
WSAF_SIB_MESSAGING_ENGINE=App_Cluster.000 tim_bus,type=WSAF_SIB has indicated that is it not alive. 
The JVM will be terminated.

7. *WAS XA Commit Error Messages/Emergency Server Shut down
from WAS stdout

0000007d RegisteredRes W   WTRN0048W: An attempt by the transaction manager to call commit on a 
transactional resource has resulted in an XAER_RMFAIL error. 
The messaging engine, ME_UUID=E674F947547B7381, INC_UUID=3AA0EB51A762DF60, 
has lost the lock on the data store.
00000019 SystemOut     O java.lang.RuntimeException: emergencyShutdown called:
        at com.ibm.ws.runtime.component.ServerImpl.emergencyShutdown(ServerImpl.java:633)
        at com.ibm.ws.hamanager.runtime.RuntimeProviderImpl.panicJVM(RuntimeProviderImpl.java:92)
        at com.ibm.ws.hamanager.coordinator.impl.JVMControllerImpl.panicJVM(JVMControllerImpl.java:56)
        at com.ibm.ws.hamanager.impl.HAGroupImpl.doIsAlive(HAGroupImpl.java:866)
        at com.ibm.ws.hamanager.impl.HAGroupImpl$HAGroupUserCallback.doCallback(HAGroupImpl.java:1364)
        at com.ibm.ws.hamanager.impl.Worker.run(Worker.java:64)
        at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1563)

8. *WebSphere Server auto restart
from WAS stdout

0000001c RecoveryDirec I   CWRLS0012I: All persistent services have been directed to perform 
recovery processing for this WebSphere server 
(omegaCell01\c23sun1Node01\c23sun1_AppCM).
0000001d SibMessage    I   itim_bus:App_Cluster.000-itim_bus CWSID0016I: 
Messaging engine App_Cluster.000-itim_bus is in state Starting.
0000001e RecoveryManag A   WTRN0028I: Transaction service recovering 3 transactions. 
These are transactions that will be committed.
0000001e RecoveryManag A   WTRN0134I: Recovering 3 XA resource manager(s) from the transaction partner logs
0000001e XARecoveryDat A   WTRN0151I: Preparing to call xa recover on XAResource: ITIM Data Source
0000000a WsServerImpl  A   WSVR0001I: Server c23sun1_AppCM open for e-business

9. *Transaction Recovery Begins
from WAS stdout

0000001e XARecoveryDat A   WTRN0146I: Obtained 2 xid(s) from xa recover on XAResource: 
ITIM Data Source of which 2 will be processed by this server
0000001e XARecoveryDat A   WTRN0149I: Recovered xid 2 
from XAResource: ITIM Data Source - xid has associated transaction (tid=3) with logged state 
Status.STATUS_COMMITTING
0000001e XARecoveryDat A   WTRN0148I: Recovered xid 2 from XAResource: 
ITIM Data Source - xid has no associated transaction and will be rolled back
0000001e XARecoveryDat A   WTRN0150I: Response from rolling back recovered 
xid 2 from XAResource: ITIM Data Source - XA_OK

10. *WAS exclusive locks obtained on data source
From WAS stdout

00000043 SibMessage    I   itim_bus:App_Cluster.000-itim_bus CWSIS1538I: 
The messaging engine, ME_UUID=E674F947547B7381, INC_UUID=36314AB8BB87ED11,
 is attempting to obtain an exclusive lock on the data store.
00000044 SibMessage    I   itim_bus:App_Cluster.000-itim_bus CWSIS1545I: 
A single previous owner was found in the messaging engine's data store, 
ME_UUID=E674F947547B7381, INC_UUID=3AA0EB51A762DF60
00000043 SibMessage    I   itim_bus:App_Cluster.000-itim_bus CWSIS1537I: 
The messaging engine, ME_UUID=E674F947547B7381, INC_UUID=36314AB8BB87ED11, 
has acquired an exclusive lock on the data store.
0000001d SibMessage    W   itim_bus:App_Cluster.000-itim_bus CWSII0269W: 
The runtime accessed the user repository for bus itim_bus 
to populate missing unique name data when loading the authorization model for the bus.
0000001d SibMessage    I   itim_bus:App_Cluster.000-itim_bus CWSIP0212I: 
Messaging engine App_Cluster.000-itim_bus on bus itim_bus
is starting to reconcile the WCCM destination and link configuration.
0000001d SibMessage    I   itim_bus:App_Cluster.000-itim_bus CWSIP0213I: 
Messaging engine App_Cluster.000-itim_bus on bus itim_bus 
has finished reconciling the WCCM destination and link configuration.
0000001d SibMessage    I   itim_bus:App_Cluster.000-itim_bus CWSJZ0021I: 
WS-ReliableMessaging is enabled for messaging engine 
App_Cluster.000-itim_bus on bus itim_bus.

11. *Success Transaction recovery/ Commit
from WAS system.out

0000001e XARecoveryDat A   WTRN0149I: Recovered xid 2 from XAResource: ITIM Data Source - 
xid has associated transaction (tid=3) with logged state Status.STATUS_COMMITTING
0000001e XARecoveryDat A   WTRN0151I: Preparing to call xa recover on XAResource: 
SIBus:itim_bus:App_Cluster.000-itim_bus
0000001e XARecoveryDat A   WTRN0146I: Obtained 3 xid(s) from xa recover on XAResource: 
SIBus:itim_bus:App_Cluster.000-itim_bus of which 3 will be processed by this server
0000001e XARecoveryDat A   WTRN0149I: Recovered xid from XAResource: SIBus:itim_bus:
App_Cluster.000-itim_bus - 
xid has associated transaction (tid=1) with logged state Status.STATUS_COMMITTING
0000001e XARecoveryDat A   WTRN0149I: Recovered xid from XAResource: SIBus:itim_bus:
App_Cluster.000-itim_bus - 
xid has associated transaction (tid=2) with logged state Status.STATUS_COMMITTING
0000001e XARecoveryDat A   WTRN0149I: Recovered xid from XAResource: SIBus:itim_bus:
App_Cluster.000-itim_bus - 
xid has associated transaction (tid=3) with logged state Status.STATUS_COMMITTING
0000001e TransactionIm A   WTRN0136I: Processing recovered transaction (tid=1) withStatus.
STATUS_COMMITTING
0000001e TransactionIm A   WTRN0137I: Recovered transaction (tid=1) committing xid 2 with 
XAResource: ITIM Data Source
0000001e TransactionIm A   WTRN0141I: Recovered transaction (tid=1) commit of xid 2 with 
XAResource:ITIM Data Source resulted in XAER_NOTA
0000001e TransactionIm A   WTRN0137I: Recovered transaction (tid=1) committing xid 1 with XAResource: 
SIBus:itim_bus:App_Cluster.000-itim_bus
0000001e TransactionIm A   WTRN0140I: Recovered transaction (tid=1) committed xid 1 
successfully with XAResource: SIBus:itim_bus:App_Cluster.000-itim_bus
0000001e TransactionIm A   WTRN0136I: Processing recovered transaction 1 
(tid=2) with Status.STATUS_COMMITTING
0000001e RecoveryManag A   WTRN0133I: Transaction recovery processing for this server is complete

12. *Messaging Engine Stops
from WAS stdout

0000005e SibMessage    I   itim_bus:App_Cluster.000-itim_bus 
CWSIT0029I: The connection for messaging engine App_Cluster.000-itim_bus in bus itim_bus to 
messaging engine Msg_Cluster.000-itim_bus stopped.
00000032 SibMessage    W   CWSIV0770W: A consumer for remote destination itim_wf_shared on bus 
itim_bus for endpoint activation com.ibm.ws.sib.ra.inbound.impl.
SibRaColocatingEndpointActivation@2bb6fd5c 
<active=true> <connections=E674F947547B7381=com.ibm.ws.sib.ra.inbound.impl.
SibRaMessagingEngineConnection@58be33e>  
<xaRecoveryAlias=itim_jms> 
<destination=queue://itim_wf_shared?busName=itim_bus> 
Failed with exception com.ibm.websphere.sib.exception.SIResourceException:
CWSIP0471E: Unable to communicate with the messaging engine Msg_Cluster.000-itim_bus 
where the destination itim_wf_shared is Localised..

13 *Messaging Engine Restart/ View All Requests
from WAS stdout

0000005b SibMessage    I   CWSIT0028I: The connection for messaging engine App_Cluster.000-itim_bus in bus 
itim_bus to messaging engine Msg_Cluster.000-itim_bus started.
00000077 SibMessage    I   CWSIV0777I: A connection to messaging engine App_Cluster.000-itim_bus for destination 
itim_policy_simulation on bus itim_bus has been successfully created.
00000079 SibMessage    I   CWSIV0777I: A connection to messaging engine App_Cluster.000-itim_bus for destination 
itim_policy on bus itim_bus has been successfully created.
0000007a SibMessage    I   CWSIV0777I: A connection to messaging engine App_Cluster.000-itim_bus for destination 
itim_wf_shared on bus itim_bus has been successfully created.
00000039 servlet       I com.ibm.ws.webcontainer.servlet.ServletWrapper init SRVE0242I: /itim/console 
/jsp/requests/request_details.jsp Initialization successful.