I'm very new to Informix IDS (we're running 11.50FC3 on AIX boxes). Had a problem this morning with one of the 15 Informix instances we run on our prod server. Below is an extract from the message log. This instance had been up and running for 120 days, and I had been checking the dbspaces and buffers via the web OpenAdmin Tool application when the error occurred. Does any one able to offer an idea as to why this error happened, and what I can do to prevent it happening again???
Thanks in advanced!
09:35:52 stack trace for pid 692658 written to /dbdump/af.a8528fe8
09:35:52 Assert Failed: No Exception Handler
09:35:52 IBM Informix Dynamic Server Version 11.50.FC3
09:35:52 Who: Session(237517, informix@::ffff:10.103.245.81, 3292, 700000045b980c8)
Thread(7840874, sqlexec, 700000034f12bb8, 5)
File: mtex.c Line: 481
09:35:52 Results: Exception Caught. Type: MT_EX_OS, Context: ill
09:35:52 Action: Please notify IBM Informix Technical Support.
09:35:52 See Also: /dbdump/af.a8528fe8, shmem.a8528fe8.0
09:38:44 Checkpoint Completed: duration was 0 seconds.
09:38:44 Sat Aug 21 - loguniq 238289, logpos 0x100d018, timestamp: 0x674e8be0 Interval: 129284
09:38:44 Maximum server connections 201
09:38:44 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked 0, Plog used 1539, Llog used 2259
09:38:46 mtex.c, line 481, thread 7840874, proc id 692658, No Exception Handler.
09:38:46 Fatal error in ADM VP at mt.c:13418
09:38:46 Unexpected virtual processor termination, pid = 692658, exit = 0x100
09:38:46 PANIC: Attempting to bring system down
09:55:11 IBM Informix Dynamic Server Started.
09:55:11 WARNING: If you intend to use J/Foundation or GLS for Unicode feature(GLU) with this Server instance, please make sure that your SHMBASE value specifies in onconfig is 0x700000010000000 or above. Otherwise you will have problems while attaching or dynamimically adding virtual shared memory segments. Please refer to Server machine notes for more information.
09:55:12 Segment locked: addr=700000000000000, size=595034112
09:55:14 Segment locked: addr=700000030000000, size=1300234240
09:55:15 VP pid=774192 priority fixed at 60, former = 90
This topic has been locked.
8 replies Latest Post - 2010-08-27T11:38:08Z by SystemAdmin
Pinned topic Assert Failed: No Exception Handler - Instance crashed
Answered question This question has been answered.
Unanswered question This question has not been answered yet.
Updated on 2010-08-27T11:38:08Z at 2010-08-27T11:38:08Z by SystemAdmin
Re: Assert Failed: No Exception Handler - Instance crashed2010-08-24T11:54:11Z in response to SystemAdminHi,
You're using version of IDS 11.50.FC3. It's a bit outdated. Current version is 11.50.FC7, which means that four fix packs were released already. Each of them contains some set of defects fixed. Therefore it's very probable that you've hit some known issue.
You need to check the AF file (/dbdump/af.a8528fe8) for a stack trace. Try to match it with any of existing APARs.
Re: Assert Failed: No Exception Handler - Instance crashed2010-08-24T12:04:44Z in response to SystemAdminThanks for the info, but sorry it it seems a stupid question (I'm very new to Informix!), but what are APAR's?
I've also uploaded the stack trace file for you to have a look at, and for you to point me at anything in there I should be using to do the APAR match.
Re: Assert Failed: No Exception Handler - Instance crashed2010-08-24T13:13:17Z in response to SystemAdminAPAR is the term of IBM, used for products defect, published to be available for customer. You can see them on IBM Support Portal.
I've checked your APAR. It seems to be incomplete (probably because the engine went down too quick). But still it has stack backtrace:
base: 0x0700000047979000 len: 135168 pc: 0x000000010006c08c tos: 0x0700000047997bb0 state: running vp: 5 0x000000010006c2c0 (oninit)afstack 0x0000000100064174 (oninit)mt_ex_throw_sig 0x000000010005083c (oninit)afsig_handler 0x0000000000000000 (*nosymtab*)0x0 0x000000010097cc18 (oninit)join_next 0x000000010097f020 (oninit)sort_open 0x00000001007cb474 (oninit)prepselect 0x0000000100440d20 (oninit)open_cursor 0x000000010044fec4 (oninit)sql_open 0x000000010044ff54 (oninit)sq_open 0x000000010029d85c (oninit)sqmain 0x00000001002d6834 (oninit)listen_verify 0x00000001002d4fd8 (oninit)spawn_thread 0x0000000100c7cba4 (oninit)startup
So if we search through APARs with kewords "join_next" & "sort_open":
Unfortunately the result set doesn't show any APAR with exact stack match.
At this point I would suggest you to contact IBM Informix Support and consider upgrading of your instance to a newer version.
Re: Assert Failed: No Exception Handler - Instance crashed2010-08-25T15:35:15Z in response to SystemAdminHi,
I've found the following IBM website on the STACKSIZE parameter (we currently have it set to 128 - we have a 64-bit AIX server where the IDS instance runs): http://www-01.ibm.com/support/docview.wss?uid=swg21423191
What are your thoughts on this parameter, and what affect increasing it 256 might have?
Re: Assert Failed: No Exception Handler - Instance crashed2010-08-27T06:04:34Z in response to SystemAdminHi,
the STACKSIZE parameter defines the stack size that engine initially allocates for thread execution. Although in theory IDS should check the possibility of stack overrun and and automatically expands the stack, it is recommended to set STACKSIZE to not less than 64K on 64bit platforms.
As for your question, doubling the STACKSIZE value will lead to increased memory consumption of course.
I suggest you to monitor the stack usage via 'onstat -g sts'. If most of the threads run with stack >128K it may worth to increase the STACKSIZE value, otherwise 128K should be enough.
If your intention is related to current topic, I must say that personally I don't think that this Assertion Failure is somehow related to insufficient stack space. Therefore I doubt that this would solve current issue.
Re: Assert Failed: No Exception Handler - Instance crashed2010-08-27T10:42:41Z in response to SystemAdminI've run that onstat -g sts command a few times, and most of the threads are only using 13k on the Current bytes and ~ 33k on the Max bytes. So on that basis, it's probably not of any benefit to increase the STACKSIZE to 256 then?
As for trying to find out what caused the assertion failure, I'm fresh out of ideas??? Are there any tools worth using to monitor things on the instance - have Server Studio/Sentinel Server. Would you recommend using OpenAdmin Tool too for performance monitoring?
Re: Assert Failed: No Exception Handler - Instance crashed2010-08-27T10:53:03Z in response to SystemAdminYep, it seems that there is not much sense in increasing the STACKSIZE.
As I already said, the assertion failure seems to be caused by a code defect. Since the search through existing APARs wasn't successful, the best thing you can do is to contact IBM Informix support for investigation and/or upgrade your instance.
Unfortunately I don't have much experience with 3d party monitoring tools - onstat is all I need so far. But I'm constantly hearing good feedbacks on OAT and it's developing quite fast.
Re: Assert Failed: No Exception Handler - Instance crashed2010-08-27T11:38:08Z in response to SystemAdminThanks for your help.
Waiting on a license issue with IBM, as can't raise a SR until then unfortunately.
I've installed OAT, so perhaps I'll have a go with it, seems if it can help with diagnosing the issue!