APAR status
Closed as program error.
Error description
Informix's JSON database service can suffer an outage with the symptom that a client cannot connect to the database. Analysis shows that all sqlexec threads are in "mutex wait session" status except for one, which holds the "session" mutex for thousands of seconds. The server permitts no new connections. $ onstat -g wmx IBM Informix Dynamic Server Version 12.10.FC2 -- On-Line -- Up 5 days 19:39:13 -- 18821440 Kbytes Mutexes with waiters: mid addr name holder lkcnt waiter waittime 17 440ca838 session 1687439 0 1687441 6527 1687442 6527 14 6527 1687443 6527 1687444 6527 12 6158 1652248 5875 1652249 5875 1666664 3015 1666665 3015 1669536 2373 1669534 2373 1669538 2373 1669537 2373 1669535 2373 1600443 590 1687432 590 1687434 590 1687411 590 1687306 590 1687316 590 1687312 590 1687422 590 1687329 590 1687413 590 1687300 590 1687320 590 Holder 1687439 corresponds to a sqlexec thread doing this query against sysmaster:sysdbopen: ... Stored procedure stack : context proc-counter opcode name ---------------------------------------------------------------- -- 0x000000008e712058 0x6522b2a8+0x0010 EXPR o_170488:dbs_check_user_limit 0x0000000080b4ec58 0x93752b90+0x0010 EXPR o_170488:sysdbopen Current SQL statement (5) in procedure o_170488:dbs_check_user_limit proc-counter 0x0x6522b2a8 opcode EXPR (select (nvl (count *), 0) from sysmaster:sysopendb where (= odb_dbname, (<unknown op>: 136 ))) Last parsed SQL statement : DATABASE o_170488 ... OS tools (pstack) were used to get the running thread's stack multiple times, which showed sometimes #-1 in lock_held () sometimes #0 in mt_lock () sometimes #1 in mt_nxtscb () ---- this part of the stack down never changed ---- #2 in propendb () #3 in pstread () #4 in pst_rsread () #5 in rsread () #6 in fmread () #7 in readseq_single () #8 in gettupl () #9 in scan_next () #10 in next_row () #11 in get_first_row_from_producer () #12 in group_open () #13 in prepselect () #14 in subqprep () #15 in exsubq () #16 in ev_cb () #17 in new_eval () #18 in ip_evalexpr () #19 in runproc () #20 in udrlm_spl_execute () #21 in udrlm_exec_routine () #22 in udr_execute () #23 in exroutine () #24 in geval () #25 in ev_geval () #26 in new_eval () #27 in ip_evalexpr () #28 in runproc () #29 in udrlm_spl_execute () #30 in udrlm_exec_routine () #31 in udr_execute () #32 in exec_sysdbproc () #33 in sqopendb () #34 in excommand () #35 in sq_execute () #36 in sqmain () #37 in spawn_thread () #38 in th_init_initgls () #39 in startup () #40 in ?? () which indicates a problem traversing the mcb's session list (ses_scblist).
Local fix
Bounce the database server.
Problem summary
**************************************************************** * USERS AFFECTED: * * IDS 11.50 users running queries on sysmaster:sysopendb with * * an unusually volatile session list. * **************************************************************** * PROBLEM DESCRIPTION: * * See Error Description * **************************************************************** * RECOMMENDATION: * * Update to latest version of IBM Informix Server 11.50.xC10 * ****************************************************************
Problem conclusion
Problem Fixed In IBM Informix Server 11.50.xC10
Temporary fix
Comments
APAR Information
APAR number
IT15326
Reported component name
IBM IDS EXPRESS
Reported component ID
5724L2305
Reported release
B15
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
2016-05-17
Closed date
2017-06-15
Last modified date
2017-06-15
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
IBM IDS EXPRESS
Fixed component ID
5724L2305
Applicable component levels
RB15 PSN
UP
RB15 PSY
UP
[{"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSGU8G","label":"Informix Servers"},"Component":"","ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"B15","Edition":"","Line of Business":{"code":"","label":""}}]
Document Information
Modified date:
15 June 2017