Topic
8 replies Latest Post - ‏2010-08-27T11:38:08Z by SystemAdmin
SystemAdmin
SystemAdmin
228 Posts
ACCEPTED ANSWER

Pinned topic Assert Failed: No Exception Handler - Instance crashed

‏2010-08-24T11:44:50Z |
Hi,

I'm very new to Informix IDS (we're running 11.50FC3 on AIX boxes). Had a problem this morning with one of the 15 Informix instances we run on our prod server. Below is an extract from the message log. This instance had been up and running for 120 days, and I had been checking the dbspaces and buffers via the web OpenAdmin Tool application when the error occurred. Does any one able to offer an idea as to why this error happened, and what I can do to prevent it happening again???

Thanks in advanced!

09:35:52 stack trace for pid 692658 written to /dbdump/af.a8528fe8
09:35:52 Assert Failed: No Exception Handler
09:35:52 IBM Informix Dynamic Server Version 11.50.FC3
09:35:52 Who: Session(237517, informix@::ffff:10.103.245.81, 3292, 700000045b980c8)
Thread(7840874, sqlexec, 700000034f12bb8, 5)
File: mtex.c Line: 481
09:35:52 Results: Exception Caught. Type: MT_EX_OS, Context: ill
09:35:52 Action: Please notify IBM Informix Technical Support.
09:35:52 See Also: /dbdump/af.a8528fe8, shmem.a8528fe8.0
09:38:44 Checkpoint Completed: duration was 0 seconds.
09:38:44 Sat Aug 21 - loguniq 238289, logpos 0x100d018, timestamp: 0x674e8be0 Interval: 129284

09:38:44 Maximum server connections 201
09:38:44 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked 0, Plog used 1539, Llog used 2259

09:38:46 mtex.c, line 481, thread 7840874, proc id 692658, No Exception Handler.
09:38:46 Fatal error in ADM VP at mt.c:13418
09:38:46 Unexpected virtual processor termination, pid = 692658, exit = 0x100

09:38:46 PANIC: Attempting to bring system down
09:55:11 IBM Informix Dynamic Server Started.
09:55:11 WARNING: If you intend to use J/Foundation or GLS for Unicode feature(GLU) with this Server instance, please make sure that your SHMBASE value specifies in onconfig is 0x700000010000000 or above. Otherwise you will have problems while attaching or dynamimically adding virtual shared memory segments. Please refer to Server machine notes for more information.

09:55:12 Segment locked: addr=700000000000000, size=595034112
09:55:14 Segment locked: addr=700000030000000, size=1300234240
09:55:15 VP pid=774192 priority fixed at 60, former = 90
Updated on 2010-08-27T11:38:08Z at 2010-08-27T11:38:08Z by SystemAdmin
  • SystemAdmin
    SystemAdmin
    228 Posts
    ACCEPTED ANSWER

    Re: Assert Failed: No Exception Handler - Instance crashed

    ‏2010-08-24T11:54:11Z  in response to SystemAdmin
    Hi,

    You're using version of IDS 11.50.FC3. It's a bit outdated. Current version is 11.50.FC7, which means that four fix packs were released already. Each of them contains some set of defects fixed. Therefore it's very probable that you've hit some known issue.

    You need to check the AF file (/dbdump/af.a8528fe8) for a stack trace. Try to match it with any of existing APARs.
    • SystemAdmin
      SystemAdmin
      228 Posts
      ACCEPTED ANSWER

      Re: Assert Failed: No Exception Handler - Instance crashed

      ‏2010-08-24T12:04:44Z  in response to SystemAdmin
      Thanks for the info, but sorry it it seems a stupid question (I'm very new to Informix!), but what are APAR's?

      I've also uploaded the stack trace file for you to have a look at, and for you to point me at anything in there I should be using to do the APAR match.

      Ta
      • SystemAdmin
        SystemAdmin
        228 Posts
        ACCEPTED ANSWER

        Re: Assert Failed: No Exception Handler - Instance crashed

        ‏2010-08-24T13:13:17Z  in response to SystemAdmin
        APAR is the term of IBM, used for products defect, published to be available for customer. You can see them on IBM Support Portal.

        I've checked your APAR. It seems to be incomplete (probably because the engine went down too quick). But still it has stack backtrace:

        
        base: 0x0700000047979000 len:   135168 pc: 0x000000010006c08c tos: 0x0700000047997bb0 state: running vp: 5   0x000000010006c2c0 (oninit)afstack 0x0000000100064174 (oninit)mt_ex_throw_sig 0x000000010005083c (oninit)afsig_handler 0x0000000000000000 (*nosymtab*)0x0 0x000000010097cc18 (oninit)join_next 0x000000010097f020 (oninit)sort_open 0x00000001007cb474 (oninit)prepselect 0x0000000100440d20 (oninit)open_cursor 0x000000010044fec4 (oninit)sql_open 0x000000010044ff54 (oninit)sq_open 0x000000010029d85c (oninit)sqmain 0x00000001002d6834 (oninit)listen_verify 0x00000001002d4fd8 (oninit)spawn_thread 0x0000000100c7cba4 (oninit)startup
        


        So if we search through APARs with kewords "join_next" & "sort_open":

        http://www.ibm.com/search/csass/search?q=sort_open%20join_next&filter=%2Bibmcontenttypeid:CT748%20%2Blanguage:en%20%2Bcollection:dblue&prod=P080940C30734F09~0::11.5~0&sn=spe&sortby=&o=0

        Unfortunately the result set doesn't show any APAR with exact stack match.

        At this point I would suggest you to contact IBM Informix Support and consider upgrading of your instance to a newer version.
        • SystemAdmin
          SystemAdmin
          228 Posts
          ACCEPTED ANSWER

          Re: Assert Failed: No Exception Handler - Instance crashed

          ‏2010-08-25T15:35:15Z  in response to SystemAdmin
          Hi,

          I've found the following IBM website on the STACKSIZE parameter (we currently have it set to 128 - we have a 64-bit AIX server where the IDS instance runs): http://www-01.ibm.com/support/docview.wss?uid=swg21423191

          What are your thoughts on this parameter, and what affect increasing it 256 might have?

          Thanks
          • SystemAdmin
            SystemAdmin
            228 Posts
            ACCEPTED ANSWER

            Re: Assert Failed: No Exception Handler - Instance crashed

            ‏2010-08-27T06:04:34Z  in response to SystemAdmin
            Hi,

            the STACKSIZE parameter defines the stack size that engine initially allocates for thread execution. Although in theory IDS should check the possibility of stack overrun and and automatically expands the stack, it is recommended to set STACKSIZE to not less than 64K on 64bit platforms.

            As for your question, doubling the STACKSIZE value will lead to increased memory consumption of course.

            I suggest you to monitor the stack usage via 'onstat -g sts'. If most of the threads run with stack >128K it may worth to increase the STACKSIZE value, otherwise 128K should be enough.

            If your intention is related to current topic, I must say that personally I don't think that this Assertion Failure is somehow related to insufficient stack space. Therefore I doubt that this would solve current issue.
            • SystemAdmin
              SystemAdmin
              228 Posts
              ACCEPTED ANSWER

              Re: Assert Failed: No Exception Handler - Instance crashed

              ‏2010-08-27T10:42:41Z  in response to SystemAdmin
              I've run that onstat -g sts command a few times, and most of the threads are only using 13k on the Current bytes and ~ 33k on the Max bytes. So on that basis, it's probably not of any benefit to increase the STACKSIZE to 256 then?

              As for trying to find out what caused the assertion failure, I'm fresh out of ideas??? Are there any tools worth using to monitor things on the instance - have Server Studio/Sentinel Server. Would you recommend using OpenAdmin Tool too for performance monitoring?
              • SystemAdmin
                SystemAdmin
                228 Posts
                ACCEPTED ANSWER

                Re: Assert Failed: No Exception Handler - Instance crashed

                ‏2010-08-27T10:53:03Z  in response to SystemAdmin
                Yep, it seems that there is not much sense in increasing the STACKSIZE.

                As I already said, the assertion failure seems to be caused by a code defect. Since the search through existing APARs wasn't successful, the best thing you can do is to contact IBM Informix support for investigation and/or upgrade your instance.

                Unfortunately I don't have much experience with 3d party monitoring tools - onstat is all I need so far. But I'm constantly hearing good feedbacks on OAT and it's developing quite fast.
                • SystemAdmin
                  SystemAdmin
                  228 Posts
                  ACCEPTED ANSWER

                  Re: Assert Failed: No Exception Handler - Instance crashed

                  ‏2010-08-27T11:38:08Z  in response to SystemAdmin
                  Thanks for your help.

                  Waiting on a license issue with IBM, as can't raise a SR until then unfortunately.

                  I've installed OAT, so perhaps I'll have a go with it, seems if it can help with diagnosing the issue!