I have a series of connected flows that call each other to parse and process XML messages from an input file. Message size is 10KB and 100MB file has 10,000 such XML messages.
I have set the number of additional flow instances to 10 for my File parsing flow and 50 flow instances for other flows downstream. Running WMB v18.104.22.168 on RHEL 6 32bit. JVM heap is set to 2GB (also tried with 1GB)
- When I run the flow (FileINput node) on a single file in the input directory it works fine (any file size - from 10K to 100MB)
- When I run the flow having more then one input file in the input directory it works - I tried 17 files of 10MB and 1000 files of 100KB.
PROBLEM: when I have two or more files of 100MB size at the same time in my input directory the processing fails and I do not see any error messages anywhere. The files disappear from the input directory, but they do not show up in the mqsibackout or other working directories.
Any clues as to how to make it possible to put 10 files of 100MB in the input and have broker handle it? (short of reducing the number of FLow instances to 1 as it will limit the speed of processing for small files).
Or shall I have special flow instance (unique name) for large file with only 1 concurrent copy and then have 10 instances of the same flow with a different name for smaller files?
Where would I look for trace or error information? The files /var/mqsi/components/MB8BROKER/stderr and stdout do not have any useful info, neither does the Admin Log shown in MQ Explorer.