IBM Support

Understanding TLS Connection Closure Behaviour and Common TLS Errors in IBM DataPower.

How To


Summary

The following information explains the behavior introduced by RFC 8446 for mitigating truncation attacks, its impact on TLS connection closures, and how it affects IBM DataPower's handling of unexpected EOFs. It also provides details on two common TLS errors observed in DataPower and how to address or suppress them appropriately.

Objective

To provide details of the TLS behavior on connection closure notifications and offer guidance on addressing specific TLS errors within IBM DataPower.

Environment

  • IBM DataPower Gateway (all firmware versions using TLS)
  • TLS 1.0, 1.1, 1.2, 1.3

Steps

TLS Connection Closure Behavior:
  1. In TLS versions prior to 1.3, the protocol required that implementations send a close_notify alert and discard any pending writes when receiving a close_notify from the peer.
  2. This behavior could result in truncation vulnerabilities on the read side.
  3. RFC 8446 Section 6.1 mitigates against this potential for truncation vulnerability by introducing a different behaviour for closure alerts.
  4. The change introduced in closure alert by RFC 8446 is that close_notify while mandatory, is only required on the write side and does not impact the read side.
  5. DataPower's TLS implementation interprets RFC 8446 as applying to all TLS versions uniformly. Since the check for close_notify is performed in the application layer, version specific information is unavailable and thus obviating selective control of this behaviour due to the way the TLS library enforces

Error: "TLS library error: SSL routines::unexpected EOF while reading"
  1. This is not a defect. It occurs when a TLS peer closes the TCP connection without first sending a TLS close_notify alert.
  2. As a security measure to prevent truncation attacks, DataPower enforces this check by default.
  3. This enforcement can be controlled by setting and resetting "Require closure notification" property in the relevant TLS profile.
  4. When this property is set to "on" mitigation against truncation attacks is engaged and peers are required to send close_notify.
  5. When this property set to off, behaviour reverts to pre-RFC 8446 and a TCP EoF will be interpreted as close_notify. This restores legacy behaviour where unexpected EOFs were silently ignored.Refer to IBM documentation for more details on this setting- https://www.ibm.com/docs/en/datapower-gateway/10.6.x?topic=commands-require-closure-notification


Error: "Fatal alert 46 received from TLS server instead of ServerHello"
  1. This occurs when a TLS server sends a fatal alert (alert code 46: internal_error) instead of a proper ServerHello during the handshake.
  2. By default, DataPower uses a built-in TLS configuration if no custom TLS Server Profile is attached.
  3. If the handshake does not complete as expected, this error is logged.
  4. If there are no actual connection failures or application-level issues, the error can generally be ignored.
  5. To avoid such warnings and gain better control:

Additional Information

IBM DataPower Gateway has unexpected system instability issues triggered using TLS with early releases of the 10.6.0.x fixpacks
 https://www.ibm.com/support/pages/node/7235707
DT436845 Runtime latency when TLS connection is closed
 https://www.ibm.com/mysupport/s/defect/aCIKe00000007uVOAQ/dt436845?language=en_US
DT444257 IBM DataPower Gateway might exhibit high CPU utilization when connections are in CLOSE_WAIT state
https://www.ibm.com/mysupport/s/defect/aCIgJ0000002mF3WAI/dt444257?language=en_US

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB77","label":"Automation Platform"},"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SS9H2Y","label":"IBM DataPower Gateway"},"ARM Category":[{"code":"a8m50000000CdoNAAS","label":"DataPower-\u003ESecurity (SE)-\u003ETLS"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"10.5.0;10.5.4;10.6.0;10.6.1"}]

Document Information

Modified date:
01 August 2025

UID

ibm17234831