Topic
IC4NOTICE: developerWorks Community will be offline May 29-30, 2015 while we upgrade to the latest version of IBM Connections. For more information, read our upgrade FAQ.
9 replies Latest Post - ‏2013-01-04T18:25:48Z by SystemAdmin
SystemAdmin
SystemAdmin
849 Posts
ACCEPTED ANSWER

Pinned topic Postprocessor stalled

‏2012-12-28T23:40:55Z |
Today I queued over 5K single file container scan requests via DmhLoad. The requests were placed in the queue with no problem. I then started the post processor and it went painfully through the scan requests, but eventually stalled with no progress with some 1500 container scan requests still pending. No indication of any exceptions anywhere. I heard comments from the customer that may me suspect they may have also seen something similar.

Is there any way to enable some tracing to determine what is happening? What logLevel would catch this ?
Updated on 2013-01-04T18:25:48Z at 2013-01-04T18:25:48Z by SystemAdmin
  • SystemAdmin
    SystemAdmin
    849 Posts
    ACCEPTED ANSWER

    Re: Postprocessor stalled

    ‏2012-12-31T17:49:25Z  in response to SystemAdmin
    Tony,

    For me, the count of container scans that cause the issue is variable but it always happens some time soon after the javaw.exe process exceeds 1GB in memory use. It looks like there is a memory leak.

    Dave
  • SystemAdmin
    SystemAdmin
    849 Posts
    ACCEPTED ANSWER

    Re: Postprocessor stalled

    ‏2012-12-31T18:26:08Z  in response to SystemAdmin
    When memory available to java process gets tight things slow down to a crawl (garbage collection is forced frequently) and it may finally fail or break through. Memory leak or simply memory usage might be at play, distributed postprocessing is done in memory.

    You may want to change the JVM to a lot (if you are on 64bit or max on 32) and watch the memory usage.

    Yet... 1500 files is not that much unless you have some larger archives in there (zip, war, ear).
    • SystemAdmin
      SystemAdmin
      849 Posts
      ACCEPTED ANSWER

      Re: Postprocessor stalled

      ‏2012-12-31T18:44:51Z  in response to SystemAdmin
      Leshek,

      This is scanning mainframe assets on Windows Server 2008... The problem was encountered scanning a single container (Windows folder) for individual COBOL members. We're running 64 bit - how do we control the JVM size for this process? For the WAS JVM we're set at 1GB min, 4GB max.

      Dave
      • SystemAdmin
        SystemAdmin
        849 Posts
        ACCEPTED ANSWER

        Re: Postprocessor stalled

        ‏2012-12-31T21:26:26Z  in response to SystemAdmin
        Windows folder (directory) with just COBOL? No distributed assets?

        By itself it should not even kick off the distributed postprocessing... unless it is part of application with distributed assets, or something else is left over to be processed on the queue.
        • SystemAdmin
          SystemAdmin
          849 Posts
          ACCEPTED ANSWER

          Re: Postprocessor stalled

          ‏2012-12-31T21:30:54Z  in response to SystemAdmin
          Maybe my terminology is wrong... I start postprocessor (Database-->Start postprocessor) and it eventually hangs on 'Analyze container' step.

          Dave
          • SystemAdmin
            SystemAdmin
            849 Posts
            ACCEPTED ANSWER

            Re: Postprocessor stalled

            ‏2013-01-01T00:20:47Z  in response to SystemAdmin
            Dave, the analyze container should go fast... look over the logs in RAA log directory and if you see nothing striking maybe it is time to open a PMR. zip-up the contain and have our service recreate the problem... but... you are running 6.0.0.11 right?
    • SystemAdmin
      SystemAdmin
      849 Posts
      ACCEPTED ANSWER

      Re: Postprocessor stalled

      ‏2012-12-31T22:50:05Z  in response to SystemAdmin
      Leshek,

      when a JVM gets into continuous CG trashing, CPU goes through the roof. Not the case here. No sign of any significant activity going on. I did let a deadlock wait overnight. This is not a JVM size problem.

      It looks to me that a post is lost. Question. Is it possible that repeated requests to refresh analysis queue display may interfere with the container scanner communication?
      • SystemAdmin
        SystemAdmin
        849 Posts
        ACCEPTED ANSWER

        Re: Postprocessor stalled

        ‏2013-01-01T00:23:28Z  in response to SystemAdmin
        Tony, I would not expect requests to refresh analysis queue display to interfere with the process. Are the logs clean of errors? Can you recreate at will?
      • SystemAdmin
        SystemAdmin
        849 Posts
        ACCEPTED ANSWER

        Re: Postprocessor stalled

        ‏2013-01-04T18:25:48Z  in response to SystemAdmin
        Leshek,

        Yes - we're running 6.0.0.11... As in Tony's post, it just stops and consumes no cpu once the problem is encountered. I will recreate with a 1GB JVM and open a PMR with supporting data. I also plan to test with 1.5GB.

        Dave