IBM Support

MustGather: Documentation for diagnosing SSL handshake failures on Liberty Application Server for z/OS

Troubleshooting


Problem

For problems occurring on Liberty Application Server for z/OS with inbound or outbound SSL, the following documentation can be collected (MustGather) before contacting support to expedite problem resolution.

Diagnosing The Problem

The initial documentation consists of:
1. RACF output
2. Complete Job output for Liberty Application Server and messages.log
3. Server dump .zip file of the Liberty Application Server


1. RACF output

a) Confirmation that Liberty Application Server region userid is permitted to FACILTY class IRR.DIGTCERT.LIST and IRR.DIGTCERT.LISTRING

RACF commands:
RLIST FACILITY IRR.DIGTCERT.LIST ALL
RLIST FACILITY IRR.DIGTCERT.LISTRING ALL

If RDATALIB class is being used instead of IRR.DIGTCERT.* class

RLIST RDATALIB LIBERTY_USERID.LibertyKeyring.LST ALL


Change the name of LIBERTY_USERID to reflect the address space that Liberty is running in.
Change the name LibertyKeyring to reflect the name of the keyring used in the server.xml

b) A listing of all certificates on the Liberty Application Server region userid keyring

RACF commands:


RACDCERT LISTRING(LibertyKeyring) ID(LIBERTY_USERID)

Change the name of LIBERTY_USERID to reflect the address space that Liberty is running in.
Change the name LibertyKeyring to reflect the name of the keyring used in the server.xml

If the keyring name is unknown at the time of problem occurrence, the LibertyKeyring in the SAF command can be substituted with an asterisk (*) to get a list of all keyrings for that address space userid.

For example:
RACDCERT LISTRING(*) ID(LIBERTY_USERID)


c) Details of the signer certificates (certificates with USAGE CERTAUTH) on the keyring

RACF command:
RACDCERT CERTAUTH LIST(LABEL('CERTAUTH_LABEL'))

The CERTAUTH_LABEL is obtained from the RACF LISTRING output in #1b.

This information is needed to:
- Confirm the certificate has a status of TRUST
- Confirm certificates are not expired
- The private key type (NONE, NON-ICSF, or ICSF)
- Confirm the certificate chain for all root signer and intermediate signing certificates that use issuer and subject DN information.


d) Details of Personal certificates (certificates with USAGE PERSONAL) on the keyring

RACF commands:
RACDCERT LIST ID(LIBERTY_USERID)

This information is needed to confirm:
- The certificate has a status of TRUST
- Certificates are not expired
- The private key type (NONE, NON-ICSF, or ICSF)
- The certificate chain for all root signer and intermediate signing certificates that use issuer and subject DN information.


2. If Liberty for z/OS was started from JCL, then send the complete Job output of Liberty Application Server region.

Job output, which includes (JESMSGLG, JESJCL, JESYSMSG, SYSOUT, and SYSPRINT) for the Liberty Application Server region.

The output can be saved to a data set easily by typing "XDC" next to the address space in SDSF.

The JESMGSGLG and JESYSMSG are of important as this output indicates the userid the address space is running under.
SYSPRINT is needed as it contains the error or exception related to the SSL failure.

Note: If the logs are large, terse the data set


3. A server dump .zip file containing the messages.log, trace.log, and any FFDC logs. 

Locate the messages.log
The messages.log file is located in:
WLP_USER_DIR/servers/defaultServer/logs

The top of the file has information containing the:

- wlp.install.dir
- server.config.dir
- java.home

for which the serverdump command can be constructed.

For example:
********************************************************************************
product = WebSphere Application Server 25.0.0.6 (wlp-1.0.102.202505301044)
wlp.install.dir = /WebSphere/Liberty/wlp/
server.config.dir = /WebSphere/Liberty/servers/defaultServer/
java.home = /WebSphere/Liberty/wlp/java/J21.0_64
java.version = 21.0.7
java.runtime = IBM Semeru Runtime Certified Edition for z/OS (21.0.7+6)
os = z/OS (29.00; s390x) (en)
process = 83953207@BOSS0181
********************************************************************************

From an OMVS shell or UNIX System Services shell issue the following 4 commands:

export WLP_USER_DIR=/WebSphere/Liberty
export JAVA_HOME=/WebSphere/Liberty/wlp/java/J21.0_64
cd /WebSphere/Liberty/wlp/bin

server dump defaultServer

An example output looks like:
-----------------
/WebSphere/Liberty/wlp/bin:>server dump defaultServer
Dumping server defaultServer.
Server defaultServer is not running.
Server defaultServerdump complete in /WebSphere/Liberty/servers/defaultServer/defaultServer.dump-15.02.24_10.45.52.zip.
------------------

If the Liberty for z/OS server was started from JCL, the same server dump can be obtained by using a dynamic MVS console command.

F JOBNAME,DUMP

An example MVS output looks like:
---------------
SY1 F BBGZSRV1,DUMP
SY1 +CWWKB0004I: Server defaultServer dump complete in
/WebSphere/Liberty/wlp/usr/servers/defaultServer/defaultServer.dump-16.08.26_14.08.38.zip.
SY1 +CWWKB0005I: COMMAND RESPONSES COMPLETED SUCCESSFULLY FROM Server
Dump Command Handler.
SY1 +CWWKB0002I: MODIFY COMMAND DUMP COMPLETED SUCCESSFULLY.
---------------------

Send the following items to support when opening the case:

- RACF output from section 1a-d
- Complete Job output for Liberty Application Server region (section 2)
- defaultServer.dump compressed file

Resolving The Problem

For most SSL handshake issues, the documentation in items 1 - 3 is sufficient for diagnosing and resolving the problem.
However, there are cases where more documentation, such as WebSphere SSL trace or a Java JSSE trace, might be needed to further narrow down the problem.

Items 4, 5, and/or 6 may also be requested from the support team depending on the the type of problem encountered.

4. Liberty SSL trace

In the server.xml add the SSL traceSpecification to the logging tag:

<logging traceSpecification="*=info:SSL=all:SSLChannel=all" />
 

Alternatively the Liberty trace can also be enabled dynamically to limit trace output by issuing the MVS console command:

F JOBNAME,LOGGING='SSL=all'
F JOBNAME,LOGGING='SSLChannel=all'

Where JOBNAME is the Liberty Application Server region job name.

The trace can be reset  back to what the server had when it started (ie. *=info)

F JOBNAME,LOGGING=RESET

An example output looks like:
----------------

SY1 F BBGZSRV,LOGGING='SSL=all'
SY1 +CWWKB0005I: COMMAND RESPONSES COMPLETED SUCCESSFULLY FROM Logging Command Handler.
SY1 +CWWKB0002I: MODIFY COMMAND LOGGING='SSL=all' COMPLETED SUCCESSFULLY.

SY1 F BBGZSRV,LOGGING=RESET
SY1 +CWWKB0005I: COMMAND RESPONSES COMPLETED SUCCESSFULLY FROM Logging Command Handler.
SY1 +CWWKB0002I: MODIFY COMMAND LOGGING=RESET COMPLETED SUCCESSFULLY.
-----------------
Note: Not all stack products support the MVS console commands for enabling and disabling trace.


5. Java JSSE trace set in jvm.options

Create an ebcdic file called jvm.options in WLP_USER_DIR/servers/defaultServer

Add the line:
-Djavax.net.debug=all

Note: the Java JSSE trace requires a restart of the server to enable the trace, and a restart of the server to disable the trace.


6. Java auth.debug trace set in jvm.options

The following trace may be required to diagnose issues in which the server needs to have the unrestricted policy jars installed (i.e. local.policy.jar and US_export_policy.jar), or may be needed for issues accessing certificates with keys in hardware crypto.

Create an ebcdic file called jvm.options in WLP_USER_DIR/servers/defaultServer

Add the line:
-Djava.security.auth.debug=all

Note: the Java auth.debug trace requires a restart of the server to enable the trace, and a restart of the server to disable the trace.
7. Reference Links for other issues with SSL setup on z/OS

 

[{"Product":{"code":"SS7K4U","label":"WebSphere Application Server for z\/OS"},"Business Unit":{"code":"BU048","label":"IBM Software"},"Component":"Security","Platform":[{"code":"PF035","label":"z\/OS"}],"Version":"8.5","Edition":"","Line of Business":{"code":"LOB77","label":"Automation Platform"}}]

Document Information

Modified date:
18 September 2025

UID

swg22014098