Troubleshooting Liberty JVM servers and Java web applications
If you have a problem with a Java™ web application, you can use the diagnostics that are provided by CICS® and Liberty to determine the cause of the problem.
Avoiding problems
CICS uses the values of the region APPLID and the
JVMSERVER resource name to create unique zFS file and directory names. Some of the acceptable
characters have special meanings in the UNIX System Services
shell. For example, the dollar sign ($) means the start of an environment variable name. Some of
these characters can cause an Exception
in the Equinox OSGi framework and prevent
the JVM server from starting. Avoid using non-alphanumeric characters in the region APPLID and JVM
server name. If you do use these characters, you might need to use the backslash (\) as an escape
character in the UNIX System Services shell. For example, if
you called your JVM server MY$JVMS and wanted to read the JVM system out file:
cat CICSPRD.MY\$JVMS.D20140319.T124122.dfhjvmout
Unable to start Liberty JVM server
- If you are unable to start a Liberty JVM server, check that your setup is correct; see Configuring a Liberty JVM server for web applications for more information. Use the messages in the CICS system log and the Liberty messages.log file that is located below WLP_OUTPUT_DIR to determine what might be causing the problem.
- Check that the -Dfile.encoding JVM property in the JVM profile specifies either ISO-8859-1 or UTF-8. These are the two code pages that are supported by Liberty. If you set any other value, the JVM server fails to start.
Unable to authenticate a user when trying to access a protected web application in a CICS Liberty JVM server
ICH420I PROGRAM DFHSIP FROM LIBRARY hlq.SDFHAUTH CAUSED THE ENVIRONMENT TO BECOME UNCONTROLLED BPXP014I ENVIRONMENT MUST BE CONTROLLED FOR DAEMON (BPX.DAEMON) PROCESSING.The Liberty messages.log contains the message:
CWWKS1100A: Authentication did not succeed for user ID user. An invalid user ID or password was specified.
Web application is not available after it is deployed to the dropins directory
If you receive a CWWK0221E error message in dfhjvmerr, check that you set the right values for the host name and port number in the JVM profile and server.xml. The port might be in use by another process and port sharing disabled. The host name might not be resolvable by the client.
CICS CPU use increased after a Liberty JVM server is enabled
Liberty can be configured to regularly check for updates to both configuration and installed applications using the <config> and <applicationMonitor> elements in server.xml. If the configuration polling rate or application monitor interval is set too frequently it can cause excessive use of CPU and I/O.
For <config> you can reduce the frequency using the monitorInterval attribute. Do not set the updateTrigger attribute to disabled because CICS requires Liberty to pick up configuration changes within a few seconds.
For <applicationMonitor> you can reduce the frequency using the pollingRate attribute, or change updateTrigger attribute to "mbean", or disable it.
For more information see Controlling dynamic updates.
Application not available
You copy a WAR file into the dropins directory but your application is not available. Check the Liberty messages.log file for error messages. If you receive the CWWKZ0013E error message, you already have a web application running in the Liberty JVM server with the same name. To fix this problem, change the name of the web application and deploy to the dropins directory.
Web application returns Context Root Not Found
You enabled your Liberty JVM server and deployed your Web application, and the JVM server reports it is enabled, but when you are accessing your application, you receive Context Root Not Found. Accessing the Web application a short time later results in success. This is a known timing window in which the server reports it is enabled while applications are still starting in the background. You are more likely to experience this condition in a multi-region environment that uses Sysplex Distributor or port sharing. You are also likely to experience this condition if you use automation to access the application triggered from the enabled status. If you are using Sysplex Distributor or port sharing, TCP/IP automation can be used to silence a port and then resume the port once the web application is available. Workarounds might involve the addition of a pause in automation scripts, or the application writing a flag to a known location when it is available.
Web application is not requesting authentication
You configured security, but the web application is not requesting authentication.
- Although you can configure CICS security for web applications, the web application uses security only if it includes a security restraint in the WAR file. Check that a security restraint was defined by the application developer in the web.xml file in the dynamic web project.
- Check that the server.xml file contains the correct security information. Any configuration errors are reported in dfhjvmerr and might provide some useful information. If you are using CICS security, check that the feature cicsts:security-1.0 is specified in server.xml. If CICS security is switched off, check that you specified a basic user registry to authenticate application users.
- Check that server.xml is configured either for <safAuthorization> to take advantage of EJBRoles, or for a local role mapping in an <application-bnd> element. The <application-bnd> element is found with in the <application> element in server.xml or installedApps.xml. The default security-role added by CICS for a local role mapping is cicsAllAuthenticated.
Web application is returning an HTTP 403 error code
The web application is returning an HTTP 403 error code in the web browser because either your user ID is revoked or you are not authorized to run the application transaction.
- Check the CICS message log for the error message ICH408I to see what type of authorization failure occurred. To fix the problem, make sure that the user ID has a valid password and is authorized to run the transaction.
- If no ICH408I message is found check the messages.log file.
- For the following
message:
CWWKS3005E: A configuration exception has occurred. No UserRegistry implementation service is available. Ensure that you have a user registry configured.
You must ensure that you have configured a SAF registry in server.xml. For more information, see Manually tailoring server.xml. - For the following message, when distributed identity is in
use:
CWWKS9104A: Authorization failed for user alidist:defaultRealm while invoking LdapTests on /basic. The user is not granted access to any of the required roles: [testing].
If server.xml is configured for <safAuthorization> or includes thecicsts:distributedIdentity-1.0
feature, then ensure the appropriate EJBRoles for the RACMAPped user ID have been defined. For more information, see JEE application role security . If server.xml is not configured for <safAuthorization> and does not include thecicsts:distributedIdentity-1.0
feature, then ensure that the appropriate distributed user ID is defined to have access to the appropriate role in an <application-bnd> element. For more information, see Authorizing users to run applications in a Liberty JVM server .
- For the following
message:
- If the application is retuning an exception for the class com.ibm.ws.webcontainer.util.Base64Decode, check dfhjvmerr for error messages. If you see configuration error messages, for example CWWKS4106E or CWWKS4000E, the server is trying to access configuration files that were created in a different encoding. This type of configuration error can occur when you change the file.encoding value and restart the JVM server. To fix the problem, you can either revert to the previous encoding and restart the JVM server, or delete the configuration files. The JVM server re-creates the files in the correct file encoding when it starts.
Web application is returning an HTTP 500 error code
The web application is returning an HTTP 500 error in the web browser. If you receive an HTTP 500 error, a configuration error occurred.
- Check the CICS message log for DFHSJ messages, which might give you more information about the specific cause of the error.
- If you are using a URIMAP to run application requests on a specific transaction, make sure that the URIMAP specifies the correct transaction ID.
- Make sure that the SCHEME and USAGE attributes are set correctly. The SCHEME must match the application request, either HTTP or HTTPS. The USAGE attribute must be set to JVMSERVER.
Web application is returning an HTTP 503 error code
The web application is returning an HTTP 503 error in the web browser. If you receive an HTTP 503 error, the application is not available.
- Check the CICS message log for DFHSJ messages for additional information.
- Make sure that the TRANSACTION and URIMAP resources for the application are enabled. If these resources are packaged as part of the application in a CICS bundle, check the status of the BUNDLE resource.
- The request might have been purged before it completed. The error messages in the log describe why the request was purged.
Unable to access your web application using distributed identity mapping
FFDC1015I: An FFDC Incident has been created: "com.ibm.ws.security.saf.SAFException:
CWWKS2905E: SAF service IRRSIA00_CREATE did not succeed because user null was not found in the SAF registry. SAF return code 0x00000008. RACF return code 0x00000008. RACF reason code 0x00000010.
FFDC1015I: An FFDC Incident has been created: "javax.security.auth.login.CredentialException: could not create SAF credential for <distid> DistIdCheck the CICS message log for the error message ICH408I to see what type of authorization failure occurred. If it is ICH408I USER(<userid>) GROUP(TSOUSER ) NAME(<name>) DISTRIBUTED IDENTITY IS NOT DEFINED: 776 cn= <distid> DistId,ou=users,dc=domain,dc=com LdapRegistry you need to create the appropriate RACMAP for the distributed identity being used to access the application. The RACMAP QUERY command is useful for debugging. For example:
RACMAP QUERY USERDIDFILTER(NAME('ou=users,dc=domain,dc=com')) REGISTRY(NAME('LdapRegistry'))
The web application is returning exceptions
The web application is returning exceptions in the web browser; for example, the application is retuning an exception for the class com.ibm.ws.webcontainer.util.Base64Decode.
- Check the dfhjvmerr for error messages.
- If you see configuration error messages, for example CWWKS4106E or CWWKS4000E, the server is trying to access configuration files that were created in a different encoding. This type of configuration error can occur when you change the file.encoding value and restart the JVM server. To fix the problem, you can either revert to the previous encoding and restart the JVM server, or delete the configuration files. The JVM server re-creates the files in the correct file encoding when it starts.
Error message CWWKB0109E in messages.log
If Liberty fails to shut down cleanly, the next Liberty JVM server with WLP_ZOS_PLATFORM=TRUE will write the error message CWWKB0109E to the messages.log file. You do not need to fix this error and it can be ignored.
Error message WTRN0078E
An attempt by the transaction manager to call start on a transactional resource has resulted in an error. The error code was XAER_PROTO. If you experience this error, the most likely scenario is that you have the default JTA integration in operation on your Liberty server, and your application uses a bean method declared as REQUIRES_NEW. For example: @Transactional(value = TxType.REQUIRES_NEW) void yourMethod{} The use of REQUIRES_NEW inside an XA transaction is not supported by CICS. You must alter the application before it will run.
Using the productInfo script to verify integrity of Liberty
You can verify the integrity of the Liberty installation after you install CICS or applying service, by using the productInfo script.
- Change directory to the CICS USSHOME directory.
- As productInfo uses Java, you must ensure that Java is included in your PATH. Alternatively, set the
JAVA_HOME environment variable to the value of JAVA_HOME
in your JVM profile, for example:
export JAVA_HOME=/usr/lpp/java/J7.0_64
- Run the productInfo script, supplying the validate option wlp/bin/productInfo validate. No errors should be reported. For more information about the Liberty productInfo script, see Verifying the integrity of Liberty profile installation.
Using the wlpenv
script to run Liberty commands
You might be asked by IBM® service to run one or more of the Liberty supplied commands, such as productInfo or server dump. To run these commands, you can use the wlpenv script as a wrapper to set the required environment. The script is created and updated every time that you enable a Liberty JVM server after the JVM profile has been successfully parsed. Because the script is unique for each JVM server in each CICS region, it is created in the WORK_DIR/APPLID/JVMSERVER as specified by default in the JVM profile and is called wlpenv. APPLID is the value of the CICS region APPLID and JVMSERVER is the name of the JVMSERVER resource.
./wlpenv productInfo version
./wlpenv server dump --archive=package_file_name.dump.pax --include=heap
For
the server dump command, you do not supply the server name because it is set by
the wlpenv script to the value set the last time the JVM server was enabled.For more information about the Liberty commands, see Liberty profile: productInfo command in WebSphere Application Server for z/OS and Generating a Liberty server dump from the command line
Troubleshooting invoking a Java EE application
- EXEC CICS LINK command fails with RESP = PGMIDERR, RESP2 = 1
-
- Check the application to determine whether the correct artifacts have been generated.
- Check that annotation processing is enabled on the source project.
- Check if an @CICSProgram has been added to a Java method and that it compiles correctly.
- Export the application and check for generated code in the
com.ibm.cics.server.invocation.proxy
package. For example, on a workstation, open the WAR or EAR file using an archive manager, or on z/OS®, use thejar -tf
command, to examine the contents of the WAR or EAR file. If code has not been generated, check you have the latest version of the CICS Explorer®, CICS build toolkit, or the annotation processor.
- Review the CICS message log for messages similar to:
- DFHSJ1204: A linkable service has been registered for class examples.TSQ.ClassOne method anotherMethod with program name LINKJCIN in JVMSERVER LINKJVM
- DFHPG0101: Resource definition for LINKJCIN has been added.
- Ensure you have a Liberty JVM server in the enabled state.
- Ensure you have the
cicsts:link-1.0
feature configured in yourserver.xml
. If it is configured, you will see message J2CA7001I: Resource adapter com.ibm.cics.wlp.program.link.connectorinstalled in messages.log. - If you are deploying your application using a CICS bundle, ensure the bundle is installed and enabled.
- Ensure the application is installed in Liberty. If it is, in the
messages.log
you will get a message including the name of the user's application. For example: CWWKZ0001I: Application com.ibm.cics.test.javalink started.
- Check the application to determine whether the correct artifacts have been generated.
- EXEC CICS LINK command fails with RESP = PGMIDERR, RESP = 27
- This indicates that CICS tried to invoke a Java EE application in Liberty but a timeout occurred before the application was successful. The most common cause for this issue is that there was no thread available in the JVM server. To resolve this, increase the JVM server thread limit or increase the value of WLP_LINK_TIMEOUT to allow the tasks to wait longer to acquire a thread. For more information, see WLP_LINK_TIMEOUT in JVM server options and Managing the thread limit of JVM servers.
- JCICS API call throws a CICSRuntimeException
-
com.ibm.cics.server.CicsRuntimeException: DTCTSQ_READNEXT: No JCICS context is associated with the current thread.
The most likely cause of this exception is that you created a JCICS object on one thread and tried to call its instance methods from a different thread. Change your application to construct the JCICS object on the same thread that calls its methods.
Patterns that lead to inadvertently using an object on a different thread include:- Constructing a JCICS object in constructor of a
java.lang.Runnable
orjava.util.concurrent.Callable
. Construct the object in therun()
method instead. - Assigned JCICS objects to static variables. Use instance variables instead.
- Passing a JCICS object as a parameter to a method that is executed by another thread. The thread should construct JCICS object itself.
- Constructing a JCICS object in constructor of a
- Transaction abends AJ05 when using invoking a Java EE application
- The following exceptions will be logged to the dfhvjmerr
file:
Using JTA with Link to Liberty is only supported with CICS JTA integration disabled. Configure this by using <cicsts_jta integration="false"/> in server.xml.com.ibm.cics.server.InvalidRequestException: CICS INVREQ Condition(RESP=INVREQ, RESP2=200) java.lang.RuntimeException: javax.transaction.RollbackException: XAResource start association error:XAER_PROTO
Error message CWWKC2262E The server is unable to process the 4.0 version and the http://xmlns.jcp.org/xml/ns/javaee namespace
providedRuntime("org.springframework.boot:spring-boot-starter-tomcat")
, while in
Maven you have used the scope 'provided', for
example:<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-tomcat</artifactId>
<scope>provided</scope>
</dependency>