IBM Support

Websphere Application Server with SSL enabled - request fails with the following message in the http_plugin.log: Failed in r_gsk_secure-Soc_init: Peer not recognized or badly formatted message received.(gsk rc = 410)

Troubleshooting


Problem

HTTP request of Websphere application fails when SSL is enabled on the Websphere Application Server with error: Failed in r_gsk_secure-Soc_init: Peer not recognized or badly formatted message received.(gsk rc = 410)

Symptom

Failed in r_gsk_secure-Soc_init: Peer not recognized or badly formatted message received.(gsk rc = 410)

Cause

The version(s) of SSL/TLS that the Websphere Application Server is trying to negotiate is not supported on the IBM i operating system.

Environment

Websphere Application Server configured to use SSL

Diagnosing The Problem

Enable the JVM custom property javax.net.debug. Here are the instructions:


Follow these steps to set up the tracing and recreate the issue.
  • In the Administrative Console set the javax.net.debug system property using one of the following options, depending on where the SSL issue is occurring:
  • For tracing an Application server, select the following: Servers > Server Types > WebSphere Application Servers > server_name > Expand Java and Process Management (under Server Infrastructure) - >Process definition > Java Virtual Machine > Custom properties > New...
  • For tracing a Deployment Manager, select the following: System Administration > Deployment manager > Expand Java and Process Management (under Server Infrastructure) >Process definition > Java Virtual Machine > Custom properties > New...
  • For tracing a Nodeagent, select the following: System Administration > Node agents > (pick a nodeagent) > Expand Java and Process Management (under Server Infrastructure) >Process definition > Java Virtual Machine > Custom properties > New...
Note: If you were not told which JVM to trace, or for some reason you are not sure which of the JVMs need this kind of tracing... set it on all of them.

Type the following:
Name:
javax.net.debug 
Value:
true

Note:
Support may request this value be set to ssl:handshake to limit the volume of trace output.

Click Apply, and Save your changes to the master configuration.

Expand Troubleshooting > Logs and trace > server_name.

Select Diagnostic Trace. Set the Maximum Number of Historical Files to 20.

Click Apply, then select Change log detail levels.

Set the trace specification string to:
*=info : SSL=all


Click OK, then OK.

Select JVM Logs. Ensure under System.out under Installed Application Output that the
Show application print statements
box is checked.

Click OK, and Save to the master configuration.

Stop the server(s) and backup/clear the logs directory for the server(s) you are tracing and the FFDC directory as well.

The trace.log will show similar to the following:

[11/10/16 11:12:48:636 NZDT] 00000067 O UOW= source=SystemOut org=IBM
prod=WebSphere component=Application Server thread=[WebContainer : 0]
WebContainer : 0, READ: TLSv1.1 Alert, length = 2
[11/10/16 11:12:48:636 NZDT] 00000017 O UOW= source=SystemOut org=IBM
prod=WebSphere component=Application Server thread=[Finalizer thread]
Finalizer thread, called closeInternal(true)
[11/10/16 11:12:48:637 NZDT] 00000067 O UOW= source=SystemOut org=IBM
prod=WebSphere component=Application Server thread=[WebContainer : 0]
WebContainer : 0, RECV TLSv1 ALERT: fatal, protocol_version

The http_plugin.log shows the following:

ERROR: lib_stream: openStream: Failed in r_gsk_secure_soc_init: Peer not
recognized or badly formatted message received.(gsk rc = 410)

In this example, the Websphere Application Server was trying to negotiate TLSV1 and the IBM i operating system was set to only support TLSV1.1 and TLSV1.2. The operating system is configured by the system value QSSLPCL. A value of *OPSYS means support whatever that version of OS supports. They had it set to the following:

*TLSV1.2
*TLSV1.1

This was on a V7R1 system so in this case *SSLV3 and *TLSV1 were excluded.

See the following links in regards to the SSLV3 security vulernabilities:

http://www-01.ibm.com/support/docview.wss?uid=swg21687173

http://www.ibm.com/support/docview.wss?uid=nas8N1020431

http://www.ibm.com/support/docview.wss?uid=nas8N1020451


Resolving The Problem

Add the requested SSL/TLS version to the system value QSSLPCL or change the list of supported versions on Websphere Application Server to include a SSL/TLS version on the IBM i OS.

Websphere Application Server can be changed by doing the following (example is WAS 8.5):

  • In the Integrated Solutions Console, expand Security on the left. Now click on SSL certificates and key management.
  • Click on SSL Configurations on the right.
  • Choose a configuration
  • Click on Quality of protection (QoP) settings
  • Choose one from the Protocol drop down box
  • Click Apply/Save.

[{"Product":{"code":"SWG60","label":"IBM i"},"Business Unit":{"code":"BU054","label":"Systems w\/TPS"},"Component":"WebSphere Application Server","Platform":[{"code":"PF012","label":"IBM i"}],"Version":"Version Independent","Edition":"WAS v9.0;WAS v8.5.5;WAS v8.0;WAS v7.0","Line of Business":{"code":"LOB08","label":"Cognitive Systems"}}]

Document Information

Modified date:
18 December 2019

UID

nas8N1021674