IBM Support

Server goes down with thread creation failure errors while ulimits look good

Troubleshooting


Problem

The IBM Spectrum Protect server goes down with errors like below. All indicate that the server meet a resource limit like maximum number of allowed process, or maximum open files. But the ulimits are good.

Symptom

The following type of errors may appear in the activity log before the server crash. Note that there may be other errors as well depending on what the server was doing when the resource contention happens:
ANR9999D_0422427600 pkBeginNamedThread(pkthread.c:898) Thread<333>: Thread creation failed; rc=11.

ANR8223W TCP/IP driver is unable to accept a new session with client at address a.b.c.d due to an error in creating a new thread

ANR0169E An unexpected error has occurred and the IBM Spectrum Protect server is stopping.
ANR0162W Supplemental database diagnostic information: -1 :57049:-1225 ([IBM][CLI Driver] SQL1225N The request failed because an operating system process, thread, or swap space limit was reached. SQLSTATE=57049).

ANR0108E sdcntr.c(10756): could not start a new transaction.
ANR0108E tmtxn.c(438): could not start a new transaction.

ANR9999D_0645605689 tbOpenX(tbtbl.c:5430) Thread<268>: Failure participating on transaction.

In the dsmserv.err file this may be seen at the time of the crash:
ANR7824S Server operation terminated.
ANR7823S Internal error SETUPHANGMONITOR detected.

In the Linux system messages file, this error is reported around the crash:
kernel: [ 278.416369] cgroup: fork rejected by pids controller in /system.slice/tsminst1.service

[{"Product":{"code":"SSEQVQ","label":"IBM Spectrum Protect"},"Business Unit":{"code":"BU048","label":"IBM Software"},"Component":"Server","Platform":[{"code":"PF016","label":"Linux"}],"Version":"7.1;8.1","Edition":"","Line of Business":{"code":"LOB69","label":"Storage TPS"}}]

Log InLog in to view more of this document

This document has the abstract of a technical article that is available to authorized users once you have logged on. Please use Log in button above to access the full document. After log in, if you do not have the right authorization for this document, there will be instructions on what to do next.

Document Information

Modified date:
17 June 2018

UID

swg22013457