Troubleshooting
Problem
The Ethernet connection restrictions for the SAN Volume Controller and the Storwize Storage Systems, limit the number of concurrent SSH sessions to 15 new connections per sec with a burst limit of 3 and a maximum of 32 active SSH connections. Attempts to open multiple concurrent SSH sessions will fail after the user log-on limit has been reached. An existing active connection must first be closed before a new connection to the cluster will be accepted. This document outlines the key connection restrictions for each firmware version of the SAN Volume Controller and the Storwize Storage Systems.
Cause
The ethernet connection restrictions on the SAN Volume Controller and the Storwize Storage Systems, are necessary to manage the cluster configuration node system resources. These restrictions protect the cluster and the I/O service against processing large amounts of ethernet related work which could risk consuming excessive resources.
SSH - Active Connection Limit
SVC restricts the maximum of concurrent SSH session to 32. There is also a limit of 3 concurrent unauthenticated SSH connection attempts to the cluster. Subsequent connections will be dropped until authentication of the existing connections succeeds, fails, or times out. Systems may need to be running v8.2.1.x or later to avoid Out-of-Memory conditions when approaching the 32 active connection maximum.
Note that connections used by the SAN Volume Controller Console (GUI) CIMOM application and host automation such as HACMP-XD count towards these limits, so the number of interactive SSH sessions you can open to the cluster concurrently may be less than 32.
Once the connection limit has been reached then subsequent connection attempts will fail, with the new SSH session closing immediately after user authentication.
SSH - New Connection Rate Limit
SVC will allow 15 new SSH connections per second.
SSH - Active Connection Limit
SVC restricts the maximum of concurrent SSH session to 32. There is also a limit of 3 concurrent unauthenticated SSH connection attempts to the cluster. Subsequent connections will be dropped until authentication of the existing connections succeeds, fails, or times out. Systems may need to be running v8.2.1.x or later to avoid Out-of-Memory conditions when approaching the 32 active connection maximum.
Note that connections used by the SAN Volume Controller Console (GUI) CIMOM application and host automation such as HACMP-XD count towards these limits, so the number of interactive SSH sessions you can open to the cluster concurrently may be less than 32.
Once the connection limit has been reached then subsequent connection attempts will fail, with the new SSH session closing immediately after user authentication.
SSH - New Connection Rate Limit
SVC will allow 15 new SSH connections per second.
If more than 15 connection attempts are made in a second then only the first ten attempts will be accepted, subsequent attempts will fail before the user authentication stage.
Any failure to get a connection results in an explicit reject packet from the SVC. This enables the remote system, which is attempting to connect, to regain control immediately and implement a retry algorithm, rather than waiting for the TCP layer to retry (which may take up to 3 minutes).
SSH - Authentication Limit
There is a limit of 3 authentication attempts per connection. After a third authentication failure the connection will be closed.
SSH - KeepAlive
SVC makes use of "ClientAlive" timeouts in the SSH2 protocol, requiring the SSH client to respond to the server after a period of inactivity to keep the connection alive. Unresponsive connections are dropped after approximately 90 seconds. This allows the connection slots held by stale SSH connections to be released much quicker than using the TCP layer keepalive alone.
Ping - Rate Limit
Ping response is limited to 140 pings/minute (average)
Any failure to get a connection results in an explicit reject packet from the SVC. This enables the remote system, which is attempting to connect, to regain control immediately and implement a retry algorithm, rather than waiting for the TCP layer to retry (which may take up to 3 minutes).
SSH - Authentication Limit
There is a limit of 3 authentication attempts per connection. After a third authentication failure the connection will be closed.
SSH - KeepAlive
SVC makes use of "ClientAlive" timeouts in the SSH2 protocol, requiring the SSH client to respond to the server after a period of inactivity to keep the connection alive. Unresponsive connections are dropped after approximately 90 seconds. This allows the connection slots held by stale SSH connections to be released much quicker than using the TCP layer keepalive alone.
Ping - Rate Limit
Ping response is limited to 140 pings/minute (average)
Resolving The Problem
It is recommended to close SSH connections when they are no longer required, in order to avoid unknowingly reaching the connection limit. The 'exit' command should be used to terminate an interactive SSH session.
Related Information
[{"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"STPVGU","label":"SAN Volume Controller"},"Component":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"5.1;6.1;6.2;6.3;6.4;7.1;7.2;7.3;7.4;7.5;7.6;7.7;7.8;8.1;8.2","Edition":"","Line of Business":{"code":"LOB26","label":"Storage"}},{"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"STLM6B","label":"IBM Storwize V3500 (2071)"},"Component":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"6.4;7.1;7.2;7.3;7.4;7.5;7.6;7.7;7.8","Edition":"","Line of Business":{"code":"LOB26","label":"Storage"}},{"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"STLM5A","label":"IBM Storwize V3700 (2072)"},"Component":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"6.4;7.1;7.2;7.3;7.4;7.5;7.6;7.7;7.8","Edition":"","Line of Business":{"code":"LOB26","label":"Storage"}},{"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"STHGUJ","label":"IBM Storwize V5000"},"Component":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"7.1;7.2;7.3;7.4;7.5;7.6;7.7;7.8;8.1;8.2","Edition":"","Line of Business":{"code":"LOB26","label":"Storage"}},{"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"ST3FR7","label":"IBM Storwize V7000"},"Component":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"6.1;6.2;6.3;6.4;7.1;7.2;7.3;7.4;7.5;7.6;7.7;7.8;8.1;8.2","Edition":"","Line of Business":{"code":"LOB26","label":"Storage"}}]
Was this topic helpful?
Document Information
Modified date:
28 March 2023
UID
ssg1S1002896