Application client is unable or slow to establish the new conversation with the Server.
If any application is experiencing this problem, we can find the following clues:
1. "iptrace" show SYN packet coming into the server but SYN/ACK will not going back to client.
2. "netstat -an" show a lot of connections in SYN_RCVD state.
3. "netstat -s" show increasing "discarded due to listener's queue full" counter.
4. "netstat -ano" show "(qlen + q0len) > 1.5 * qlimit" for the port in LISTEN state.
For example, listen queue limit for telnet application (port 23) is 1000.
tcp 0 0 *.23 *.* LISTEN
so_options: (ACCEPTCONN|REUSEADDR)
q0len:1510 qlen:0 qlimit:1000 so_state: (PRIV)
The listen queue is full, so the tcp stack discards SYN packet for new connection from remote client and increase "discarded due to listener's queue full" counter.
About the q0len, qlen, qlimit
qlimit is MIN (backlog value in listen() Subroutine, somaxconn set by no Command).
qlen is the number of completed connections (completed three-way handshake but waiting for accept()).
q0len is the number of partial connections waiting for third segment of the three-way handshake (i.e. ACK) from client.
Once (qlen+q0len) becomes higher than 1.5 times qlimit for the socket, tcp stack discards SYN packet for new connection from remote client and increase "discarded due to listener's queue full" counter.
About the listen() Subroutine
Purpose:
Listens for socket connections and limits the backlog of incoming connections.
Library:
Standard C Library (libc.a)
Syntax:
#include <sys/socket.h>
int listen ( Socket, Backlog)
int Socket, Backlog;
Description:
The listen subroutine performs the following activities:
(1). Identifies the socket that receives the connections.
(2). Marks the socket as accepting connections.
(3). Limits the number of outstanding connection requests in the system queue.
The outstanding connection request queue length limit is specified by the parameter backlog per listen call. A no parameter - somaxconn - defines the maximum queue length limit allowed on the system, so the effective queue length limit will be either backlog or somaxconn, whichever is smaller.
About the no parameter - somaxconn
Purpose:
Specifies the maximum listen backlog.
Values:
Default: 1024 connections
Range: 0 to MAXSHORT
Type: Connect
Diagnosis:
N/A
Tuning
Increase this parameter on busy Web servers to handle peak connection rates.
qlimit is MIN (backlog value in listen(), somaxconn set by no option)
Both of the settings need be checked:
somaxconn is global parameter for all the applications in this system. somaxconn can be tuned by no command.
backlog is set in the application code by listen() Subroutine.
| SUPPORT |
|
If additional assistance is required after completing all of the instructions provided in this document, please follow the step-by-step instructions below to contact IBM to open a case for software under warranty or with an active and valid support contract. The technical support specialist assigned to your case will confirm that you have completed these steps.
1. Document and/or take screen shots of all symptoms, errors, and/or messages that might have occurred.
2. Capture any logs or data relevant to the situation.
3. Contact IBM to open a case:
-For electronic support, please visit the IBM Support Community:
https://www.ibm.com/mysupport
-If you require telephone support, please visit the web page:
https://www.ibm.com/planetwide/
4. Provide a good description of your issue, and reference this Technote, and any issues you had with the instructions.
5. Collect the system snap and upload all of the details and data for your case.
| To collect a complete snap of your system information: |
|
5.1) Remove previously gathered data
# snap -r
5.2) Copy related files from #1 and #2 to the snap data directory
# mkdir -p /tmp/ibmsupt/testcase
# cp <logs, screenshots, etc> /tmp/ibmsupt/testcase
5.3) Run the snap command with one of the following options to collect all info.
* If you have already engaged with a support engineer, use the flags specified by your support team.
# snap -aZc (Omits system dump data)
OR
# snap -ac (Use if system dump data is needed)
5.4) Rename the testcase to include your case number to ensure it is properly attached to your case
# mv /tmp/ibmsupt/snap.pax.Z /tmp/ibmsupt/yourcase#[.optional_description].snap.pax.Z
5.5) Upload the file by one of the following options (a, b, or c)
a) Attach to your case
https://www.ibm.com/mysupport/s/my-cases
b) Upload to the Enhanced Customer Data Repository(ECuRep)
https://www.secure.ecurep.ibm.com/app/upload_sf
c) Upload to the Blue Diamond FTP server (Blue Diamond Customers Only)
https://msciportal.im-ies.ibm.com
* Note: For information about blue diamond upload see:
http://www.ibm.com/support/docview.wss?uid=nas8N1020947
|
|
[{"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SWG10","label":"AIX"},"Component":"","Platform":[{"code":"PF002","label":"AIX"}],"Version":"All Versions","Edition":"","Line of Business":{"code":"LOB08","label":"Cognitive Systems"}}]