Troubleshooting Liberty JVM servers and Java web applications

Avoiding problems

CICS uses the values of the region APPLID and the JVMSERVER resource name to create unique zFS file and directory names. Some of the acceptable characters have special meanings in the UNIX System Services shell. For example, the dollar sign ($) means the start of an environment variable name. Some of these characters can cause an Exception in the Equinox OSGi framework and prevent the JVM server from starting. Avoid using non-alphanumeric characters in the region APPLID and JVM server name. If you do use these characters, you might need to use the backslash (\) as an escape character in the UNIX System Services shell. For example, if you called your JVM server MY$JVMS and wanted to read the JVM system out file:

cat CICSPRD.MY\$JVMS.D20140319.T124122.dfhjvmout

Unable to start Liberty JVM server

1. If you are unable to start a Liberty JVM server, check that your setup is correct; see Configuring a Liberty JVM server for more information. Use the messages in the CICS system log and the Liberty messages.log file that is located after WLP_OUTPUT_DIR to determine what might be causing the problem.
2. Check that the -Dfile.encoding JVM property in the JVM profile specifies either ISO-8859-1 or UTF-8. These are the two code pages that are supported by Liberty. If you set any other value, the JVM server fails to start.

Local Liberty JVM server cannot connect to the remote Liberty JVM server for JCICSX when SSL is enabled (SRVE0777E)

Application Error
SRVE0777E: Exception thrown by application class 'com.ibm.cics.jcicsx.http.CICSContextProviderImpl.initialise:112'
com.ibm.cics.jcicsx.http.JCICSXException: Failed to retrieve server info
at com.ibm.cics.jcicsx.http.CICSContextProviderImpl.initialise(CICSContextProviderImpl.java:112)
at com.ibm.cics.jcicsx.http.CICSContextProviderImpl.getCICSContext(CICSContextProviderImpl.java:85)
at com.ibm.cics.harness.TaskProducer.produceTask(TaskProducer.java:26)
at com.ibm.cics.harness.HarnessServletFilter.doFilter(HarnessServletFilter.java:36)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:201)
at [internal classes]
Caused by: javax.ws.rs.ProcessingException: java.io.IOException: IOException invoking https://remotejcicsxserver.com:portNum/jcicsxServer/info: HTTPS hostname wrong: should be <remotejcicsxserver.com>
at org.apache.cxf.jaxrs.client.AbstractClient.checkClientException(AbstractClient.java:643)
at [internal classes]
at com.sun.proxy.$Proxy34.getInfo
at com.ibm.cics.jcicsx.http.CICSContextProviderImpl.initialise(CICSContextProviderImpl.java:101)
... 5 more
Caused by: java.io.IOException: IOException invoking https://remotejcicsxserver.com:portNum/jcicsxServer/info: HTTPS hostname wrong: should be <remotejcicsxserver.com>
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(NativeConstructorAccessorImpl.java:-2)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.mapException(HTTPConduit.java:1447)
... 8 more
Caused by: java.io.IOException: HTTPS hostname wrong: should be <remotejcicsxserver.com>
at sun.net.www.protocol.https.HttpsClient.checkURLSpoofing(HttpsClient.java:649)
at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:573)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1564)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:347)
at org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream$2.run(URLConnectionHTTPConduit.java:427)
... 8 more

Unable to authenticate a user when trying to access a protected web application in a CICS Liberty JVM server

ICH420I PROGRAM DFHSIP FROM LIBRARY hlq.SDFHAUTH CAUSED THE 
ENVIRONMENT TO BECOME UNCONTROLLED
BPXP014I ENVIRONMENT MUST BE CONTROLLED FOR DAEMON (BPX.DAEMON) PROCESSING.

CWWKS1100A: Authentication did not succeed for user ID user. 
An invalid user ID or password was specified.

The CICS Liberty JVM server security implementation uses the Liberty angel process to perform authorized security checks. If Liberty is unable to connect to the angel process, it fails over to using UNIX System Services security, which requires all members in the STEPLIB and DFHRPL concatenations to be program controlled.

Unable to authenticate a user with user ID and password, cannot access APPL-ID when trying to access a protected web application in a CICS Liberty JVM server

com.ibm.ws.security.saf.SAFServiceResult E CWWKS2909E: 
A SAF authentication or authorization attempt was rejected because the server 
is not authorized to access the following SAF resource: 
APPL-ID APPL-ID. Internal error code 0x03008108.

Web application is not available after it is deployed to the dropins directory

If you receive a CWWK0221E error message in dfhjvmerr, check that you set the right values for the host name and port number in the JVM profile and server.xml. The port might be in use by another process and port sharing disabled. The host name might not be resolvable by the client.

CICS CPU use increased after a Liberty JVM server is enabled

Liberty can be configured to regularly check for updates to both configuration and installed applications using the <config> and <applicationMonitor> elements in server.xml. If the configuration polling rate or application monitor interval is set too frequently it can cause excessive use of CPU and I/O.

For <config> you can reduce the frequency using the monitorInterval attribute. Do not set the updateTrigger attribute to disabled because CICS requires Liberty to pick up configuration changes within a few seconds.

For <applicationMonitor> you can reduce the frequency using the pollingRate attribute, change the updateTrigger attribute to mbean, or disable it.

For more information, see Controlling dynamic updates.

Application not available

You copy a WAR file into the dropins directory but your application is not available. Check the Liberty messages.log file for error messages. If you receive the CWWKZ0013E error message, you already have a web application running in the Liberty JVM server with the same name. To fix this problem, change the name of the web application and deploy to the dropins directory.

Web application returns Context Root Not Found

You enabled your Liberty JVM server and deployed your web application. The JVM server reports it is enabled, but when you are accessing your application, you receive Context Root Not Found. Accessing the web application a short time later results in success. This is a known timing window in which the server reports it is enabled while applications are still starting in the background. You are more likely to experience this condition in a multi-region environment that uses Sysplex Distributor or port sharing. You are also likely to experience this condition if you use automation to access the application triggered from the enabled status. If you are using Sysplex Distributor or port sharing, TCP/IP automation can be used to silence a port and then resume the port once the web application is available. Workarounds might involve the addition of a pause in automation scripts, or the application writing a flag to a known location when it is available.

Web application is not requesting authentication

You configured security, but the web application is not requesting authentication.

1. Although you can configure CICS security for web applications, the web application uses security only if it includes a security restraint in the WAR file. Check that a security restraint was defined by the application developer in the web.xml file in the Dynamic Web Project.
2. Check that the server.xml file contains the correct security information. Any configuration errors are reported in dfhjvmerr and might provide some useful information. If you are using CICS security, check that the feature cicsts:security-1.0 is specified in server.xml. If CICS security is switched off, check that you specified a basic user registry to authenticate application users.
3. Check that server.xml is configured either for <safAuthorization> to take advantage of EJBRoles, or for a local role mapping in an <application-bnd> element. The <application-bnd> element is found with in the <application> element in server.xml or installedApps.xml. The default security-role added by CICS for a local role mapping is cicsAllAuthenticated.

Web application is returning an HTTP 403 error code

The web application is returning an HTTP 403 error code in the web browser because either your user ID is revoked or you are not authorized to run the application transaction.

1. Check the CICS message log for the error message ICH408I to see what type of authorization failure occurred. To fix the problem, make sure that the user ID has a valid password and is authorized to run the transaction.
2. If no ICH408I message is found check the messages.log file.
   • For the following message:

     CWWKS3005E: A configuration exception has occurred. 
     No UserRegistry implementation service is available.  
     Ensure that you have a user registry configured.

     You must ensure that you have configured a SAF registry in server.xml. For more information, see Manually tailoring server.xml.
   • For the following message, when distributed identity is in use:

     CWWKS9104A: Authorization failed for user alidist:defaultRealm 
     while invoking LdapTests on /basic. 
     The user is not granted access to any of the required roles: [testing].

     If server.xml is configured for <safAuthorization> or includes the cicsts:distributedIdentity-1.0 feature, then ensure the appropriate EJBRoles for the RACMAPped user ID have been defined. For more information, see Authorization using SAF role mapping. If server.xml is not configured for <safAuthorization> and does not include the cicsts:distributedIdentity-1.0 feature, then ensure that the appropriate distributed user ID is defined to have access to the appropriate role in an <application-bnd> element. For more information, see Authorizing users to run applications in a Liberty JVM server.
3. If the application is retuning an exception for the class com.ibm.ws.webcontainer.util.Base64Decode, check dfhjvmerr for error messages. If you see configuration error messages, for example CWWKS4106E or CWWKS4000E, the server is trying to access configuration files that were created in a different encoding. This type of configuration error can occur when you change the file.encoding value and restart the JVM server. To fix the problem, you can either revert to the previous encoding and restart the JVM server, or delete the configuration files. The JVM server re-creates the files in the correct file encoding when it starts.

Web application is returning an HTTP 500 error code

The web application is returning an HTTP 500 error in the web browser. If you receive an HTTP 500 error, a configuration error occurred.

1. Check the CICS message log for DFHSJ messages, which might give you more information about the specific cause of the error.
2. If you are using a URIMAP to run application requests on a specific transaction, make sure that the URIMAP specifies the correct transaction ID.
3. Make sure that the SCHEME and USAGE attributes are set correctly. The SCHEME must match the application request, either HTTP or HTTPS. The USAGE attribute must be set to JVMSERVER.

Web application is returning an HTTP 503 error code

The web application is returning an HTTP 503 error in the web browser. If you receive an HTTP 503 error, the application is not available.

1. Check the CICS message log for DFHSJ messages for additional information.
2. Make sure that the TRANSACTION and URIMAP resources for the application are enabled. If these resources are packaged as part of the application in a CICS bundle, check the status of the BUNDLE resource.
3. The request might have been purged before it completed. The error messages in the log describe why the request was purged.

Unable to access your web application by using distributed identity mapping

FFDC1015I: An FFDC Incident has been created: "com.ibm.ws.security.saf.SAFException: 

CWWKS2905E: SAF service IRRSIA00_CREATE did not succeed because 
user null was not found in the SAF registry. 
SAF return code 0x00000008. RACF return code 0x00000008. RACF reason code 0x00000010.

FFDC1015I: An FFDC Incident has been created: 
"javax.security.auth.login.CredentialException: could not create SAF credential 
for <distid> DistId

RACMAP QUERY USERDIDFILTER(NAME('ou=users,dc=domain,dc=com')) REGISTRY(NAME('LdapRegistry'))

The web application is returning exceptions

The web application is returning exceptions in the web browser; for example, the application is retuning an exception for the class com.ibm.ws.webcontainer.util.Base64Decode.

1. Check dfhjvmerr for error messages.
2. If you see configuration error messages, for example CWWKS4106E or CWWKS4000E, the server is trying to access configuration files that were created in a different encoding. This type of configuration error can occur when you change the file.encoding value and restart the JVM server. To fix the problem, you can either revert to the previous encoding and restart the JVM server, or delete the configuration files. The JVM server re-creates the files in the correct file encoding when it starts.

Error message WTRN0078E An attempt by the transaction manager to call start on a transactional resource has resulted in an error.

The error code was XAER_PROTO. If you experience this error, the most likely scenario is that you have the default JTA integration in operation on your Liberty server, and your application uses a bean method declared as REQUIRES_NEW. For example, the use of REQUIRES_NEW inside an XA transaction is not supported by CICS: @Transactional(value = TxType.REQUIRES_NEW) void yourMethod{} You must alter the application before it will run.

Error message DFHSJ1004 in MSGUSER, but no corresponding STDERR exception

A symptom of running out of zFS file system space could be a DFHSJ1004 with no corresponding STDERR exception. The message is sent because of the lack of space, but there is no exception in STDERR because there is no space to write a message to the files.

You can plan and monitor the size of your file system using the techniques detailed in Managing file systems in z/OS UNIX System Services Planning.

Using the productInfo script to verify integrity of Liberty

You can verify the integrity of the Liberty installation after you install CICS or applying service, by using the productInfo script.

1. Change directory to the CICS USSHOME directory.
2. As productInfo uses Java, you must ensure that Java is included in your PATH. Alternatively, set the JAVA_HOME environment variable to the value of JAVA_HOME in your JVM profile, for example:

   export JAVA_HOME=/usr/lpp/java/J8.0_64 

3. Run the productInfo script, supplying the validate option wlp/bin/productInfo validate. No errors should be reported. For more information about the Liberty productInfo script, see Verifying the integrity of Liberty profile installation.

Using the wlpenv script to run Liberty commands

You might be asked by IBM® service to run one or more of the Liberty supplied commands, such as productInfo or server dump. To run these commands, you can use the wlpenv script as a wrapper to set the required environment. The script is created and updated every time that you enable a Liberty JVM server after the JVM profile has been successfully parsed. Because the script is unique for each JVM server in each CICS region, it is created in the WORK_DIR/APPLID/JVMSERVER as specified by default in the JVM profile and is called wlpenv. APPLID is the value of the CICS region APPLID and JVMSERVER is the name of the JVMSERVER resource.

./wlpenv productInfo version

./wlpenv server dump --archive=package_file_name.dump.pax --include=heap

For more information about Liberty commands, see productInfo command and Generating a Liberty server dump from the command line.

Troubleshooting invoking an Enterprise Java application

EXEC CICS LINK command fails with RESP = PGMIDERR, RESP2 = 1

1. Check the application to determine whether the correct artifacts have been generated.
   1. Check that annotation processing is enabled on the source project.

      

   2. Check if an @CICSProgram has been added to a Java method and that it compiles correctly.
   3. If your project contains a web.xml, check the version of the servlet specification it specifies. It must be at least version 2.5.
   4. Export the application and check for generated code in the com.ibm.cics.server.invocation.proxy package. For example, on a workstation, open the WAR or EAR file using an archive manager, or on z/OS® use the jar -tf command, to examine the contents of the WAR or EAR file. If code has not been generated, check you have the latest version of the CICS Explorer®, CICS build toolkit, or the annotation processor.
2. Review the CICS message log for messages similar to:
   • DFHSJ1204: A linkable service has been registered for class examples.TSQ.ClassOne method anotherMethod with program name LINKJCIN in JVMSERVER LINKJVM
   • DFHPG0101: Resource definition for LINKJCIN has been added.
   If these messages don't appear then:
   1. Ensure you have a Liberty JVM server in the enabled state.
   2. Ensure you have the cicsts:link-1.0 feature configured in your server.xml. If it is configured you will see message J2CA7001I: Resource adapter com.ibm.cics.wlp.program.link.connectorinstalled in messages.log.
   3. If you are deploying your application using a CICS bundle, ensure the bundle is installed and enabled.
   4. Ensure the application is installed in Liberty, if it is, in the messages.log you will get a message including the name of the user's application. For example: CWWKZ0001I: Application com.ibm.cics.test.javalink started.

EXEC CICS LINK command fails with RESP = PGMIDERR, RESP = 27

This indicates that CICS tried to invoke an Enterprise Java application in Liberty but a timeout occurred before the application was successful. The most common cause for this issue is that there was no thread available in the JVM server. To resolve this, increase the JVM server thread limit or increase the value of WLP_LINK_TIMEOUT to allow the tasks to wait longer to acquire a thread. For more information see WLP_LINK_TIMEOUT in Symbols used in the JVM profile and Managing the thread limit of JVM servers.

JCICS API call throws a CICSRuntimeException

com.ibm.cics.server.CicsRuntimeException: 
DTCTSQ_READNEXT: No JCICS context is associated with the current thread. 

The most likely cause of this exception is that you created a JCICS object on one thread and tried to call its instance methods from a different thread. Change your application to construct the JCICS object on the same thread that calls its methods.

Patterns that lead to inadvertently using an object on a different thread include:

• Constructing a JCICS object in constructor of a java.lang.Runnable or java.util.concurrent.Callable. Construct the object in the run() method instead.
• Assigned JCICS objects to static variables. Use instance variables instead.
• Passing a JCICS object as a parameter to a method that is executed by another thread. The thread should construct JCICS object itself.

Transaction abends AJ05 when using invoking an Enterprise Java application

The following exceptions will be logged to the dfhvjmerr file:

com.ibm.cics.server.InvalidRequestException: CICS INVREQ Condition(RESP=INVREQ, RESP2=200)
java.lang.RuntimeException: 
javax.transaction.RollbackException: 
XAResource start association error:XAER_PROTO

Using JTA with Link to Liberty is only supported with CICS JTA integration disabled. Configure this by using <cicsts_jta integration="false"/> in server.xml.

Java stack overflows

The Java error message java.lang.StackOverflowError: operating system stack overflow is typically seen when the thread exceeds the initial stack size for operating system threads. The size is set by the JVM option -Xmso in the JVM profile. This value might need to be increased if Java Platform Debugger Architecture (JPDA) is enabled.

Unexpected ICH408I messages in log

These are standard audit messages. For more information, see Classes that control auditing for z/OS UNIX System Services in z/OS Security Server RACF Auditor's Guide.

IPCOBJ is defined only for auditing z/OS UNIX security events, it is not used for authorization checking.

Liberty Bundlepart hits timeout

The application installed by bundlepart <symbolic-name> was not started 
after 30000 milliseconds. Either a problem exists with the application, or the system is busy. 
This timeout can be controlled by the System Property 'com.ibm.cics.jvmserver.wlp.bundlepart.timeout=n' 
where n is the value of milliseconds to wait.

<config updateTrigger="polled" monitorInterval="10s" />

CICS Explorer cannot export a bundle or find a project with error Unable to find a built {project_type} with symbolic name {symbolic_name}.

You might see the following validation errors when adding a project to your CICS bundle:

The bundle cannot be exported because CICS Explorer cannot find a built {project_type} with symbolic name {symbolic_name}. 
To resolve this, either install the IBM CICS SDK for Java EE, Jakarta EE and Liberty to build the project or import the built {file_extension} project directly.

Unable to find a built OSGi Application Project with symbolic name {symbolic_name}.
To resolve this, either install LDT to find and validate projects in your workspace or add the built EBA project into the root of the CICS bundle project.

Unable to find a {project_type} with symbolic name {symbolic_name}.
To resolve this, either import the {project_type} into your workspace or add the built {file_extension} project into the root of the CICS bundle project.

{symbolic_name} cannot be exported. Install the Liberty Developer Tools (LDT) from Eclipse Marketplace.

The WAR, EAR or EBA cannot be exported. Install the IBM CICS SDK for Java EE, Jakarta EE and Liberty.

EBA application fails to install with a CWWKZ0005E or CWWKZ0021E messages

If Liberty fails to install an EBA, it produces either the CWWKZ0005E or CWWKZ0021E messages. This might be caused when the wab-1.0 feature is not installed. Ensure that the wab-1.0 feature is correctly installed.

Due to the stabilization of Liberty's OSGi support, WABs are not compatible with Java EE 8 features. The wab-1.0 feature can automatically be uninstalled if Java EE 8 features are also installed in the same Liberty server, causing any EBAs to be removed from the server with the above message. The JVM profile property com.ibm.cics.jvmserver.wlp.wab can be used to control whether the wab-1.0 feature is added to server.xml.

Error message CWWKC2262E The server is unable to process the 4.0 version and the http://xmlns.jcp.org/xml/ns/javaee namespace

<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-tomcat</artifactId>
<scope>provided</scope>
</dependency>