My dear debuggers,
Today, another user came into the practice. It is urgent as he cannot access his Process Center and Process Admin Console after an installation of BPM v8.5.7. It was a new installation, so the user expected that everything works well.
Unfortunately, he is getting the following exception when trying to enter Process Admin Console:
[10/02/17 16:00:30:447 GMT] 00000149 ServletWrappe I com.ibm.ws.webcontainer.servlet.ServletWrapper init SRVE0242I:
[/ProcessAdmin/com.lombardisoftware.processadmin.ProcessAdmin/ProcessAdmin.jsp]: Initialization successful.
[10/02/17 16:00:30:867 GMT] 00000149 ServletWrappe E com.ibm.ws.webcontainer.servlet.ServletWrapper service SRVE0014E: Uncaught service() exception root cause /ProcessAdmin/com.lombardisoftware.processadmin.ProcessAdmin/ProcessAdmin.jsp: com.ibm.websphere.servlet.error.ServletErrorReport: com.lombardisoftware.client.delegate.BusinessDelegateException:
Context: PCCell/clusters/AppTarget, name: ejb/Security: First component in name Security not found.
Caused by: javax.naming.NameNotFoundException: Context: PCCell/clusters/AppTarget, name: ejb/Security: First component in name Security not found. [Root exception is org.omg.CosNaming.NamingContextPackage.NotFound: IDL:omg.org/CosNaming/NamingContext/NotFound:1.0]
Caused by: org.omg.CosNaming.NamingContextPackage.NotFound: IDL:omg.org/CosNaming/NamingContext/NotFound:1.0
Since the issue emerged after fresh installation of BPM, usually we are reviewing at installation related logs like BPMConfig logs and logs from DmgrProfile. However, as a first step we can start to look at the AppTarget SystemOut.log first. Hereby, also the time from starting the Server is interesting.
The following can be seen there:
At first the Repository starts:
[10/02/17 16:00:25:494 GMT] 00000001 RepoComponent I BPM Repository started
Then the Composition unit of Business-Level Application (BLA) "IBM_BPM_Teamworks_AppTarget" is starting (this one is the Business Level Application (BLA) of the Teamworks application):
[10/02/17 16:00:25:653 GMT] 00000093 CompositionUn A WSVR0190I: Starting composition unit
WebSphere:cuname=IBM_BPM_Teamworks_AppTarget in BLA
Right afterwards exceptions occur stating that Binaries were not added:
[10/02/17 16:00:25:683 GMT] 00000093 wtp E org.eclipse.jst.j2ee.commonarchivecore.internal.strategy.DirectoryLoadStrategyImpl collectFilesFromBinaries Failed to extract binaries to temporary location:
Binaries files were not added!!!
This load strategy [org.eclipse.jst.j2ee.commonarchivecore.internal.strategy.DirectoryArchiveLoadStrategyImpl@23d541d3 ] Container [com.ibm.etools.commonarchive.impl.EJBJarFileImpl@9863cde9 (URI: EmbeddedECMStartupBean.jar, lastModified: 0, size: 0, directoryEntry: <unset>, originalURI: EmbeddedECMStartupBean.jar) (types: null) ]
Such exceptions can be seen for several binaries like bpmasset.war, bpm.coachflow.war, bpmrepo-services.war and bpmwebservices.war.
Then the Application "IBM_BPM_Teamworks_AppTarget" and its binaries are starting:
[10/02/17 16:00:25:901 GMT] 00000093 ApplicationMg A WSVR0200I: Starting application: IBM_BPM_Teamworks_AppTarget
[10/02/17 16:00:25:902 GMT] 00000093 ApplicationMg A WSVR0203I: Application: IBM_BPM_Teamworks_AppTarget Application build level: BPM8570-20160301-140232
However, it fails at teamworks.war (again with Binaries not added exception):
[10/02/17 16:00:26:611 GMT] 00000093 wtp E org.eclipse.jst.j2ee.commonarchivecore.internal.strategy.DirectoryLoadStrategyImpl collectFilesFromBinaries Failed to extract binaries to temporary location:
Binaries files were not added!
This load strategy [org.eclipse.jst.j2ee.commonarchivecore.internal.strategy.DirectoryArchiveLoadStrategyImpl@b7e8e4ae ] Container [com.ibm.etools.commonarchive.impl.WARFileImpl@de82ae1d (URI: teamworks.war, lastModified: 1396457130000, size: 37, directoryEntry: <unset>, originalURI: teamworks.war) (types: null) ]
On loading the web module "Lombardi Software Teamworks Web Interface" an NullPointer occurs and the application is stopping:
[10/02/17 16:00:26:749 GMT] 00000093 webapp I com.ibm.ws.webcontainer.webapp.WebGroupImpl WebGroup SRVE0169I: Loading Web Module: Lombardi Software Teamworks Web Interface.
[10/02/17 16:00:26:801 GMT] 00000093 webapp W com.ibm.ws.webcontainer.webapp.WebAppImpl notifyStop
InitializationCollaborator threw an exception stopping. Will continue destroying java.lang.NullPointerException
[10/02/17 16:00:26:851 GMT] 00000093 webapp E com.ibm.ws.webcontainer.webapp.WebAppImpl destroy SRVE0297E: Web Application IBM_BPM_Teamworks_AppTarget#teamworks.war destroy encountered errors: java.lang.NullPointerException
[10/02/17 16:00:26:853 GMT] 00000093 webapp E com.ibm.ws.webcontainer.webapp.WebGroupImpl WebGroup SRVE0015E: Failure to initialize Web application Lombardi Software Teamworks Web Interface
[10/02/17 16:00:26:883 GMT] 00000093 DeployedAppli W WSVR0206E: Module, teamworks.war, of application,
IBM_BPM_Teamworks_AppTarget.ear/deployments/IBM_BPM_Teamworks_AppTarget, failed to start
[10/02/17 16:00:26:889 GMT] 00000093 ApplicationMg W WSVR0101W: An error occurred starting, IBM_BPM_Teamworks_AppTarget
[10/02/17 16:00:26:890 GMT] 00000093 ApplicationMg A WSVR0217I: Stopping application: IBM_BPM_Teamworks_AppTarget
[10/02/17 16:00:27:372 GMT] 00000093 ApplicationMg A WSVR0220I: Application stopped: IBM_BPM_Teamworks_AppTarget
[10/02/17 16:00:27:595 GMT] 00000093 CompositionUn E WSVR0194E: Composition unit WebSphere:cuname=IBM_BPM_Teamworks_AppTarget in BLA
WebSphere:blaname=IBM_BPM_Teamworks_AppTarget failed to start.
Similar issues could be seen for further applications like "IBM_BPM_WebAPI_AppTarget".
The most important part in this is the one at the beginning where it states that the Binaries were not added:
** org.eclipse.jst.j2ee.commonarchivecore.internal.strategy.DirectoryLoadStrategyImpl collectFilesFromBinaries Failed to extract binaries to temporary location:
Binaries files were not added! **
These exceptions relate to applications deployed on WAS which are important to run specific components of BPM (e.g. teamworks.war). This should be the root cause of the problem: It seems that binaries are missing which are essential for these BPM applications.
I know that applications deployed on WAS should be available under Install_Root\profiles\NodeProfile\installedApps. This should be the case for these BPM related applications. The installedApps directory contains the applications after they have been synchronized to the nodes.
So, I followed up comparing the following two directories:
Interesting! The number of applications in installedApps directory are less than the number under the config directory. Well, that means the BPM installation brings another stack of applications than it was ultimatively installed on the node (applications are missing). How can we fix that?
In a discussion with the user, I explained him:
At first, we should try a manual Node synchronization since it might be a simple synchronization issue. Using the WAS Admin Console, we click on
System Administration > Nodes > Synchronize. Command-line will also work to synchronize the nodes. See the following technote for more details:
We tried that way, but it did not helped. Ok, then let’s try the following steps:
* Stop the NodeAgent
* Backup your file system
* Rename the installedApps directory like:
* Backup and delete \config\..\applications directory on Node level like:
mv Install_Root\profiles\NodeProfile\config\cells\Cell_Name\applications \tmp\applications
ATTENTION: Do not do that under the \config\..\applications directory, just work on the NodeProfile directory!
* Sync the nodes
(e.g. by using command-line: Install_Root\profiles\NodeProfile\bin> syncNode.bat host port -user -password )
* Review and make sure both the config\..\applications directory and installedApps directory are populated and having the same content.
* Start the node and the cluster member.
For more details, see the post in the following dWAnswer topic: https://developer.ibm.com/answers/answers/251156/view.html
With that procedure I could solve the problem finally.
And if this does not help, take two of these and call me in the morning.
Your Dr. Debug