Topic
  • 4 replies
  • Latest Post - ‏2013-08-19T16:46:39Z by ThomasBien
Professor
Professor
9 Posts

Pinned topic Parsing 100MB file in WMB v8 fails for >1 file

‏2013-08-14T16:45:05Z |

I have a series of connected flows that call each other to parse and process XML messages from an input file. Message size is 10KB and 100MB file has 10,000 such XML messages.

I have set the number of additional flow instances to 10 for my File parsing flow and 50 flow instances for other flows downstream. Running WMB v8.0.0.2 on RHEL 6 32bit. JVM heap is set to 2GB (also tried with 1GB)

WHAT WORKS:

- When I run the flow (FileINput node) on a single file in the input directory it works fine (any file size - from 10K to 100MB)

- When I run the flow having more then one input file in the input directory it works - I tried 17 files of 10MB and 1000 files of 100KB.

PROBLEM: when I have two or more files of 100MB size at the same time in my input directory the processing fails and I do not see any error messages anywhere. The files disappear from the input directory, but they do not show up in the mqsibackout or other working directories.

Any clues as to how to make it possible to put 10 files of 100MB in the input and have broker handle it? (short of reducing the number of FLow instances to 1 as it will limit the speed of processing for small files).

Or shall I have special flow instance (unique name) for large file with only 1 concurrent copy and then have 10 instances of the same flow with a different name for smaller files?

Where would I look for trace or error information? The files /var/mqsi/components/MB8BROKER/stderr and stdout do not have any useful info, neither does the Admin Log shown in MQ Explorer.
 

Updated on 2013-08-14T17:08:54Z at 2013-08-14T17:08:54Z by Professor
  • KY0J_Simbu_Selvarasu
    40 Posts

    Re: Parsing 100MB file in WMB v8 fails for >1 file

    ‏2013-08-15T16:55:38Z  

    Try running User Trace. It will give you much more information.

  • Professor
    Professor
    9 Posts

    Re: Parsing 100MB file in WMB v8 fails for >1 file

    ‏2013-08-15T17:47:17Z  

    Try running User Trace. It will give you much more information.

    when I enable the service trace for the execution group to NORMAL, the speed drops by 100x and it never gets to where I need it to be to diagnose the problem. Any other way to see trace without impacting the speed?

  • KY0J_Simbu_Selvarasu
    40 Posts

    Re: Parsing 100MB file in WMB v8 fails for >1 file

    ‏2013-08-16T04:08:34Z  

    How multiple file nodes share access to files in the same directory

    http://publib.boulder.ibm.com/infocenter/wmbhelp/v8r0m0/topic/com.ibm.etools.mft.doc/ac55400_.htm

     

    put trace node inbetween the nodes( each terminal) and log (Root,Environment,LocalEnvironemnt,ExceptionList). but this also impact the performance.

    As you are diagnose the problem, please don't look for the speed.

  • ThomasBien
    ThomasBien
    10 Posts

    Re: Parsing 100MB file in WMB v8 fails for >1 file

    ‏2013-08-19T16:46:39Z  

    I see you are using 32bit Linux.
    A maximum JVM size of 2GB is rather large for a 32bit process.
    Depending on how you are handling these 100MB files, you could be exhausting the available resources for the EG.
    Have you confirmed whether the EG is abending?

    You may want to revisit the settings on the FileInput node for what it is configured to do in the event of a failure.

    Simbu also recommended collecting a user level trace. That is a wise recommendation.
    Note that the user level trace is different from a service level trace, and has a much smaller overhead.