If you’re deploying a large EAR file which includes a lot of EJBs, it is possible for the EJB deployment process to fail when you’re installing or updating an enterprise application. If this happens you’ll see in the admin console:
ADMA5018I: The EJBDeploy program
is running on file <temp EAR>.
JVMDUMP006I Processing dump event “systhrow”, detail “java/lang/OutOfMemoryError” – please wait.
ADMA5008E: The EJBDeploy program failed on file <temp EAR>. Exception com.ibm.etools.ejbdeploy.EJBDeploymentException: Error executing EJBDeploy
What you need to know is that the EJB deployment step of enterprise application installation is handled by a specific component called ejbdeploy which uses it’s own seperate JVM instance. Therefore, to fix this error you need to increase the heap size for the ejbdeploy process. On Windows you need to edit ejbdeploy.bat and on UNIX, ejbdeploy.sh. Both can be found in: <INSTALL_DIR>/WebSphere/AppServer/deploytool/itp.
Inside this file, look for $JAVA_CMD, and then find the –Xms (minimum heap size) and –Xmx (maximum heap size) parameters (which both default to 256m). I was able to overcome my issue by increasing to 512m.