Troubleshooting tips for Liberty

Check that your JDKs are at a supported level

If you experience problems that are not readily explained, check that the JDKs you are using are compliant with Java Version 7 or later, and are at a current service level. See Minimum supported Java levels.

Troubleshooting security

This section describes some common security problems and solutions you can choose.

SESN0008E: A user authenticated as anonymous has attempted to access a session owned by user:LdapRegistry/cn=steven,o=myCompany,c=US.

This error happens when an unauthenticated user tries to access a session that is created by an authenticated user. Session security that is enabled by default prevents unauthorized access of the sessions. Only the user who created a session can access it. For more information, see session security.

This error can happen when you use a JSP (login.jsp for example) for your form-login and the SSO token sent by the browser is expired. Because the SSO token is expired, the user is prompted to log in again using the login.jsp page that is configured for the form-login. The jsp page, by default, tries to get a session. This session was originally created by the user whose token is expired. Because the token is expired and the user is not authenticated, no credentials are established when you access this session, which results in this error.

To avoid this error, restart the browser that starts a new session, or configure the login.jsp file to not create the session by default. If you choose to update the login.jsp file, set <%@ page session="false" %>.

CWWKS9104A: Authorization failed for user {0} while invoking {1} on {2}. The user is not granted access to any of the required roles: {3}.

This error occurs when you do not have authorization to the roles required by the application. Make sure that the user or the group it belongs to is mapped to at least one of the roles that are mentioned in the error message. A user-to-role mapping that is defined in the ibm-application-bnd.xmi/xml file takes precedence over a mapping that is defined in the server.xml file. Check both resources to ensure that the correct mapping is defined. See Configuring authorization for applications in Liberty.

CWWKS9104A: Authorization failed for user {0}.

This error can occur if you specify both an application and webApplication for the same context root. If a conflict happens the latest configuration that is defined is ignored and causes an unexpected error, such as CWWKS9104A.

CWWKZ0013E: It is not possible to start two applications called {0} followed by unexpected security behavior and error messages such as CWWKS9104A.

This error occurs when you specify your application in both the server.xml by using the application element and in the dropins folder. As a result, the application is attempted to be installed twice and the security configuration in the server.xml file might or might not take effect. To fix this problem, you must remove your application from the dropins folder and copy it to another directory. If you must leave it in the dropins folder, you must disable the application monitoring by using the following code in your server.xml file:

<applicationMonitor dropinsEnabled="false"/>

An unauthenticated user was able to access my servlet or JSP.

A user with a principal of UNAUTHENTICATED (or the unauthenticated SAF user on z/OS®) is created when authentication fails or when your servlet or JSP is unprotected. An unauthenticated user can access your servlet or JSP if you do not define any security constraints or if you map the EVERYONE special subject to the role required by your application. Review the user-to-role mappings in the ibm-application-bnd.xmi/xml and server.xml files. Take one of the following options:

• If your servlet or JSP is unprotected, add security constraints to the deployment descriptor of your application. See Authentication.
• If you do not want any unauthenticated user to access your application, remove the EVERYONE special subject from the mapping for that role. See Configuring authorization for applications in Liberty.
• If a certain user cannot be authorized to your servlet or JSP, review who is mapped to that role by examining the ibm-application-bnd.xmi/xml and server.xml files. You can map a role to a user, group, or special subject. If your role is mapped to the EVERYONE special subject, any user is granted access. If your role is mapped to the ALL_AUTHENTICATED special subject, any authenticated user is granted access. Remove these special subjects if you want to further limit who can access your servlet or JSP. Finally, check what group the user belongs to. Although the user might not have explicit access, the group might be mapped to the role. In this case, remove the user from the authorized group or remove the group from role mapping. See Configuring authorization for applications in Liberty.

Single sign-on (SSO) does not work.

If SSO does not work, make sure that the different Liberty servers that use the same LTPA keys, password, and ssoCookieName attribute of webAppSecurity element have the same Universal Time (UTC) and share the user registry. Also, if the token expires or if the cookie is sent from a web browser after you change the user registry in a manner that is inconsistent, such as modifying the realm or removing the user that the cookie represents, the SSO fails, and the user is prompted to enter the credential information again. See Customizing SSO configuration using LTPA cookies in Liberty.

Debugging security public APIs.

WSSecurityHelper and RegistryHelper are loaded even if security is not enabled, for example, if a security feature - appSecurity-1.0, appSecurity or zosSecurity-1.0 - is not specified. If security is not enabled, then WSSecurityHelper.isServerSecurityEnabled() and WSSecurityHelper.isGlobalSecurityEnabled() methods both return false, and RegistryHelper.getUserRegistry() method returns null.

Other security public API classes might not be loaded if security is not enabled. If you try to access these classes and call a method on one of these classes, you might get a java.lang.NoClassDefFoundError exception.

To avoid getting java.lang.NoClassDefFoundError exceptions, you must first test to see whether security is enabled by calling the WSSecurityHelper.isServerSecurityEnabled() or WSSecurityHelper.isGlobalSecurityEnabled() class, and then call other security public API classes only when security is enabled. See Security public APIs for examples of this coding technique.

Note: This behavior is different from the WebSphere Application Server traditional. In WebSphere Application Server traditional, all classes are always loaded regardless of whether security is enabled or not.

Cannot authenticate users with Unicode characters

To authenticate users whose names contain Unicode characters, you must set the character encoding type that is used by the Liberty server to UTF-8 by adding the following jvm option to the server start command:

-Dclient.encoding.override=UTF-8

You must also specify the charset and pageEncoding in your login page. Here is an example for specifying these parameters on a login JSP page:

<%@page contentType="text/html; charset=UTF-8" pageEncoding="UTF-8"%>

java.lang.annotation.AnnotationFormatError: java.lang.IllegalArgumentException:Wrong type at constant pool index at sun.reflect.annotation.AnnotationParser.parseAnnotations(AnnotationParser.java:87)

This error can occur when an OpenID Connect or OAuth provider uses a Database Store for client registration with some JDK 7 versions.

To avoid this problem, upgrade to JDK version 7.1.

Troubleshooting LDAP

This section describes some common LDAP problems and solutions you can choose.

FFDC1015I: An FFDC Incident has been created: javax.naming.ServiceUnavailableException: myldapserver.mycompany.com:636; socket closed com.ibm.ws.security.registry.ldap.internal.LdapRegistry 298

This message in messages.log indicates that the configured LDAP server cannot be reached. Check your LDAP server to verify that it is running and that its IP address can be accessed from the Liberty server.

The javax.naming.CommunicationException: simple bind failed: myldapserver.mycompany.com:636 [Root exception is javax.net.ssl.SSLHandshakeException: com.ibm.jsse2.util.g: PKIX path building failed: java.security.cert.CertPathBuilderException: unable to find valid certification path to requested target]

If you enable SSL on your LDAP server without copying the signer of the LDAP server into the truststore that is referenced in the LDAPSSLSettings element, a connection with the LDAP server fails with an SSL handshake error. Make sure that you copy the signer of the LDAP server into your truststore.

The javax.naming.CommunicationException: myldapserver.mycompany.com:389 [Root exception is java.net.BindException: Address already in use: connect]

This message might appear in the FFDC logs and indicates that the usable sockets on the local server are exhausted, which can result in a failure when you connect to your LDAP server. Make sure that the socket is not used and try again.

CWWKS1100A: Authentication did not succeed for user ID xxxxx. An invalid user ID or password was specified

This FFDC exception might happen on the server even though the user who is mentioned in the message is a valid user on the LDAP server under heavy load. With the LDAP configuration, you can add the reuseConnection=false property or disable it by using the developer tools. To fix the problem, set this property to the default value of true.

Login, authentication, or authorization is slow for LDAP or a federated repository.

Several reasons can exist for a slow login. Review the following list for some common causes.

• If the LDAP configuration for referrals is set to follow, then any referrals to other LDAP instances are followed on LDAP requests. This situation can result in contacting one or more additional LDAP servers, depending on your LDAP configuration, and thus, additional time is spent. For example, the LDAP1 server could follow a referral to the LDAP2 server, which could follow a referral to the LDAP3 server. If you do not need to acquire more information from other LDAP servers, set referrals to ignore. Additionally, one LDAP server could be experiencing a problem and error that might not display in the JVM logs. Examples include a TCP read timeout or a DNS issue when the LDAP server talks to a referred LDAP server. To diagnose these situations, you can capture packets to see how many calls are being made and if any delays or errors exist due to an LDAP server that is following a referral.
• A firewall or other software closes connections to LDAP. By default, a pool of LDAP connections is maintained to improve performance. If a cached connection is closed remotely, a new connection is made and put back in the context pool. This process can cause a delay and can cause errors to be logged in the JVM logs. To avoid this situation, set the context pool timeout to less than the remote connection closure time. For example, if a firewall closes the connection at 10 minutes, you could set the connection pool timeout for 9 minutes. Instead of encountering a failed connection and creating a new connection, the expiration is checked on a connection from the pool and a new connection is created, skipping the failure step. Example errors include the following one:

  java.net.SocketException: Connection reset

  For more information on setting the context pool timeout, see the contextPool element on the ldapRegistry element.
• With federated repositories, all repositories and registries are checked to ensure that a unique user is in the realm. If a repository or registry is responding slowly, every call is slow even if the user is not in the slow registry. Ensure that all participating base entries are responding promptly.

Occasional connection exceptions accessing the LDAP server. For example, java.net.SocketException: Connection reset

A firewall or other software closes connections to LDAP. By default, a pool of LDAP connections is maintained to improve performance. If a cached connection is closed remotely, a new connection is made and put back in the context pool. This process can cause a delay and can cause errors to be logged in the JVM logs. To avoid this situation, set the context pool timeout to less than the remote connection closure time. For example, if a firewall closes the connection at 10 minutes, the connection pool timeout could be set for 9 minutes. Instead of encountering a failed connection and creating a new connection, the expiration is checked on a connection from the pool and a new connection is created, skipping the failure step.

Troubleshooting SSL

This section describes some common SSL problems and solutions you can choose.

CWWKS9105E: Could not determine the SSL port for automatic redirection.

This error occurs when you try to access an application that redirects to an SSL port and the SSL port is not available. The port might not be available because of a missing SSL configuration or some error in the SSL configuration definition. Check the SSL configuration in the server.xml file to make sure that it exists and is correct. See Securing communications with Liberty.

FFDC1015I: An FFDC Incident has been created: java.lang.IllegalArgumentException: Unknown, incomplete configuration: missing id field com.ibm.ws.config.internal.cm.ManagedServiceFactoryTracker aSyncReadNupdate. Exception thrown while trying to read configuration and update ManagedServiceFactory. Exception = java.lang.IllegalArgumentException: Unknown, incomplete configuration: missing id field at ffdc_12.04.18_16.09.42.0.log

This error occurs when a keystore element exists in the configuration without an ID field. If you use a minimal SSL configuration, set the ID field to defaultKeyStore.

You might get an exception if you use an LDAP user registry with sslEnabled and a sslRef value is not specified.

For example, a configuration has sslEnabled set to true but there is not a sslRef value, as shown in the following example:

<ltldapRegistry id="ldap" realm="SampleLdapIDSRealm" 
host="ccwin12.austin.ibm.com" port="636" ignoreCase="true"
baseDN="o=ibm,c=us"
bindDN="cn=root"
bindPassword="rootpwd"
ldapType="IBM Tivoli Directory Server"
idsFilters="ibm_dir_server"
sslEnabled="true"
searchTimeout="8m" />

You must enter a sslRef value. If you are using a minimal SSL configuration that is similar to the following configuration, the sslRef field must be set to defaultSSLConfig.

If you are using the PKCS12 keystore, your minimal configuration would be similar to the following one.

<ltkeyStore id="defaultKeyStore" location="key.p12"
password="mypassword" />

If you are using the JKS keystore, your minimal configuration would be similar to the following one.

<ltkeyStore id="defaultKeyStore" location="key.jks"
password="mypassword" />

If a custom SSL configuration is configured, the name of that configuration must be placed in the sslRef field.

If you use a JDK from the WebSphere Application Server, you might see the following error if SSL is enabled on your Liberty Server.

java.net.SocketException: java.lang.ClassNotFoundException: Cannot find the specified class com.ibm.websphere.ssl.protocol.SSLSocketFactory 
      at javax.net.ssl.DefaultSSLSocketFactory.a(SSLSocketFactory.java:11) 
      at javax.net.ssl.DefaultSSLSocketFactory.createSocket(SSLSocketFactory.java:6) 
      at com.ibm.net.ssl.www2.protocol.https.c.afterConnect(c.java:161) 
      at com.ibm.net.ssl.www2.protocol.https.d.connect(d.java:36) 
      at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1184) 
      at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:390) 
      at com.ibm.net.ssl.www2.protocol.https.b.getResponseCode(b.java:75) 
      at com.ibm.ws.jmx.connector.client.rest.internal.RESTMBeanServerConnection.loadJMXServerInfo(RESTMBeanServerConnection.java:142) 
      at com.ibm.ws.jmx.connector.client.rest.internal.RESTMBeanServerConnection.<init>(RESTMBeanServerConnection.java:114) 
      at com.ibm.ws.jmx.connector.client.rest.internal.Connector.connect(Connector.java:315) 
      at com.ibm.ws.jmx.connector.client.rest.internal.Connector.connect(Connector.java:103) 

This error occurs because the WebSphere Application Server SSL socket factories are not supported by Liberty. You can get past this problem by taking the following steps:

• Create a text file, such as my.java.security with the following two empty entries

  ssl.SocketFactory.provider=
  ssl.ServerSocketFactory.provider=

• Create a jvm.options file for your Liberty server
• Add the following property to your jvm.options file, that includes the full path to your text file that you just created

  -Djava.security.properties=fullPathTo/my.java.security 

• If you want to make this more reusable, you can put the my.java.security file in your server directory, and then you are able to use a relative path like this:

  -Djava.security.properties=./my.java.security 

For more information, see Customizing the Liberty environment.

Troubleshooting CORBA/IIOP

This section describes some common CORBA problems and solutions you can choose.

If you use a JDK from the WebSphere Application Server, you might see the following error if your application uses CORBA/IIOP communications.

15:21:58.096 com.ibm.rmi.pi.InterceptorManager runPreInit:178 Init Process ORBRas [default]  java.lang.ClassNotFoundException:
com.ibm.ISecurityLocalObjectBaseL13Impl.CSIClientRI
        at com.ibm.CORBA.iiop.UtilDelegateImpl.loadClass(UtilDelegateImpl.java:685)
        at javax.rmi.CORBA.Util.loadClass(Util.java:246)
        at com.ibm.rmi.pi.InterceptorManager.runPreInit(InterceptorManager.java:172)
        at com.ibm.rmi.corba.ORB.initializeInterceptors(ORB.java:664)
        at com.ibm.CORBA.iiop.ORB.initializeInterceptors(ORB.java:1084)
        at com.ibm.rmi.corba.ORB.orbParameters(ORB.java:1359)
        at com.ibm.rmi.corba.ORB.set_parameters(ORB.java:1271)
        at com.ibm.CORBA.iiop.ORB.set_parameters(ORB.java:1694)
        at org.omg.CORBA.ORB.init(ORB.java:371) 
        ...

This error occurs because the WebSphere Application Server Object Request Broker (ORB) interceptors are not supported by Liberty. You can resolve this problem by editing the orb.properties file from the JDK to not use these interceptors. This file is found in the WebSphere <JAVA_HOME>/jre/lib directory, though it might be overwritten with a copy in the user's <USER_HOME> directory. The following example shows the lines in the orb.properties file that must be commented out:

# WS Interceptors
#org.omg.PortableInterceptor.ORBInitializerClass.com.ibm.ws.Transaction.JTS.TxInterceptorInitializer
#org.omg.PortableInterceptor.ORBInitializerClass.com.ibm.ejs.ras.RasContextSupport
#org.omg.PortableInterceptor.ORBInitializerClass.com.ibm.ISecurityLocalObjectBaseL13Impl.ClientRIWrapper
#org.omg.PortableInterceptor.ORBInitializerClass.com.ibm.ws.activity.remote.cos.ActivityServiceClientInterceptor
#org.omg.PortableInterceptor.ORBInitializerClass.com.ibm.ISecurityLocalObjectBaseL13Impl.CSIClientRI 
#org.omg.PortableInterceptor.ORBInitializerClass.com.ibm.debug.olt.ivbtrjrt.OLT_RI
#org.omg.PortableInterceptor.ORBInitializerClass.com.ibm.ws.wlm.client.WLMClientInitializer

# WS ORB & Plugins properties
#com.ibm.ws.orb.transport.ConnectionInterceptorName=com.ibm.ISecurityLocalObjectBaseL13Impl.SecurityConnectionInterceptor

Troubleshooting logging and tracing

This section describes some common problems with logging and tracing.

The java.util.logging -- Java logging programming interface.

Liberty does not support using the logging.properties file to configure java.util.logging. Use Java code, for example in a deployed application or user feature to create and add java.util.logging handlers, filters, or formatters.

Since the Liberty server manages the java.util.logging logger levels in accordance with the traceSpecification attribute of the logging configuration element, avoid using the Logger.setLevel method.

Applying fix packs and interim fixes to an archive install

If you installed your Liberty runtime environment from an archive file, rather than by using Installation Manager, you must take special measures when you apply service updates. For more information, see Applying a fix pack to a Liberty Java archive installation and Applying an interim fix to a Liberty archive installation.

Troubleshooting performance

This section describes some common performance problems and solutions you can choose.

High CPU usage by your application monitor.

This error can occur if your application monitor has many files under the apps/ directory and is polling too frequently.

To avoid this problem there are a number of things you can change.

1. Increase the value of the pollingRate attribute in the applicationMonitor element.
2. Update the server.xml to include an applicationMonitor element with an updateTrigger that is not polled.

   server.xml                                                                      
   <applicationMonitor updateTrigger="mbean" /> 
   

3. Reduce the number of files under the apps/ directory.

For more information about the applicationMonitor element, see Controlling dynamic updates.

Troubleshooting collectives

This section describes a problem with collectives and the solution you must apply.

java.lang.IllegalArgumentException: CWWKX0219E: The MBean 'WebSphere:feature=collectiveController,type=CollectiveRepository,name=CollectiveRepository' does not have an operation by the name 'retrieveMemberRootCertificate'

A server with feature collectiveMember-1.0, a member, cannot register with a server with feature collectiveController-1.0, a controller, that is at an earlier level of Liberty. For example, a member at this beta level of Liberty cannot register itself with a controller at V8.5.5.2 Liberty.

With default logging, this problem is reported as a First Failure Data Capture (FFDC) in the FFDC logs of the member.

To fix the problem, you must move the controller to the same or higher level of Liberty as the member.

Troubleshooting SAML

This section describes a problem with SAML and the solution you must apply.

java.lang.ArrayIndexOutOfBoundsException: Array index out of range: 0

This exception can occur when you attempt multiple logins through an unsolicited Service Provider (SP) initiated request without removing the Identity Provider token (IdP).

To avoid this, add <httpSession invalidateOnUnauthorizedSessionRequestException="true" /> in the relevant unsolicited server.xml file.

java.lang.IllegalStateException: CWWKS0010E: While getting the caller principal, the caller subject was found to have more than one principal of type WSPrincipal. Only one WSPrincipal can exist in the subject. The names of the WSPrincipals are: {0}

This exception can occur if a SAML user has previously logged directly into an on-premises user registry. To avoid this problem, a SAML user must not directly login to an on-premises user registry.

Troubleshooting REST API Discovery

If the IBM REST API Discovery Explorer Try it out selection is invoked remotely and fails with a response code equal to 0, ensure that the fully qualified hostname or the IP address is returned to the IBM REST API Discovery Explorer in the Curl and Request URL associated with the GET, PUT, POST, or DELETE operation. If the fully qualified hostname or IP address is not in the Curl and Request URL, complete the following actions:

1. Add the variable element in server.xml and specify the fully qualified hostname. Here is the example of adding the variable element that specifies the fully qualified hostname in the server.xml file:

    <httpEndpoint host="*" httpPort="9080" httpsPort="9443" id="defaultHttpEndpoint"/>          
   <variable name="defaultHostName" value="developer.rtp.raleigh.ibm.com"/>

2. Specify the full computer name for your operating system as the fully qualified host name.
3. Verify that the fully qualified hostname is returned in the Curl and Request URL associated with the GET, PUT, POST, or DELETE operations in the IBM REST API Discovery Explorer. For more information, see Setting the default host name of a Liberty server and read about configuring your network in the IBM InfoSphere® Information Server, Version 11.3.1 product documentation.

Troubleshooting applications not starting in an embedded Liberty server

Ensure that the Java process that starts the embedded Liberty server was started with the -javaagent JVM argument that pointed to the libertyInstallDir/bin/tools/ws-javaagent.jar. If the -javaagent JVM argument is not used the server runtime starts, but applications fail to start with no obvious exceptions.