I am trying to select data from a database table and map the resultset to a DFDL FixedRecord file or a DFDL CSV file. The data in the SQLServer tables are from 5 Mb to 130 Mb and the resulting DFDL file is about the same size. It seems like the Graphical Mapping node get the 'java.lang.OutOfMemoryError' when the select resultset is somewhere between 15 to 20 Mb of data. This is the case when the Execution Group (EG) has the following configuration:
JVM Heap Size = 512 Mb
JVM Stack Size = Default
JVMVerbose = None
There is only one messageflow processing in the EG, but there are 7 other messageflows deployed and running in the same application in the EG. This is the only application deployed in the EG.
The SQLServer JDBC 4.0 type 4 driver is used when connecting with integrated security to the SQLServer database. The connection URI property 'selectMethod=cursor' is used to get server-side cursor processing. It seems to have some effect on the processing in the mapping node.
The WMB version is 220.127.116.11, x64.
WMQ version is 18.104.22.168.
OS is Windows 2008 R2 and the processor are AMD64.
Total Memory is 6 Gb on the server, and the total amount of memory used is below 3 Gb in the above JVM configuration.
I have tried to set the JVM Heap Size to 1,5 Gb and the JVM Stack Size to 10 Mb, but still got the 'java.lang.OutOfMemoryError' when trying to select data from a table containing 60 Mb of data without getting close to allocating the servers total memory. It is nothing special in my mapping I think, just a 'For Each' with a couple of 'fn:Replece'. But there are two fn:count mappings on the resultset to give the total number of records in the file and a couple of fn:Sum mappings. No Custom Java or ESQL.
I am aware of the fact that the WMB explode the data into a Message Tree, but I think the Mapping Node should cope with this amount of data. You could point out that this could have been done in a different way, but there are 'newbies' to WMB that are going to maintain the flows. And the mapping are a bit easier to understand than the other options in this case.
I need some help on what to do and if this should have been doable in the Graphical Mapping Node. If you need more information about this case, I'll be happy to provide further information. My next move would be something like a 'Flow-loop' reading just portions of the database every time in the mapping node, and controlling the looping and propagation to the 'FileOutput' node in a Compute node after the Mapping Node. The looping mechanism will be by putting and reading MQ messages from the same queue.
Pinned topic v22.214.171.124 'java.lang.OutOfMemoryError' errors in Graphical Mapping Node
Answered question This question has been answered.
Unanswered question This question has not been answered yet.
Updated on 2012-11-29T16:57:24Z at 2012-11-29T16:57:24Z by SystemAdmin
SystemAdmin 110000D4XK4179 Posts
Re: v126.96.36.199 'java.lang.OutOfMemoryError' errors in Graphical Mapping Node2012-11-29T16:57:24ZThis is the accepted answer. This is the accepted answer.I'm sorry but I forgot to mention that the the 60 Mb SQLServer table has 135381 rows. The one of about 18 Mb of data has 37587 rows, and this mapping is reporting error 'java.lang.OutOfMemoryError'.
Below is some extracts of the heap dump files.
0SECTION MEMINFO subcomponent dump routine
1STHEAPFREE Bytes of Heap Space Free: 3B428
1STHEAPALLOC Bytes of Heap Space Allocated: 10000000
1STGCHTYPE GC History
3STHSTTYPE 11:40:20:132613020 GMT j9mm.100 - J9AllocateObject() returning NULL! 48 bytes requested for object of class 000000001CA3F450 from memory space 'Flat' id=0000000000575F88
3STHSTTYPE 11:40:20:132354568 GMT j9mm.83 - Forcing J9AllocateObject() to fail due to excessive GC
3STHSTTYPE 11:40:20:132338642 GMT j9mm.134 - Allocation failure end: newspace=0/0 oldspace=242728/268435456 loa=0/0
3STHSTTYPE 11:40:20:132333333 GMT j9mm.139 - Reference count end: weak=147 soft=211 phantom=1 threshold=0 maxThreshold=32
3STHSTTYPE 11:40:20:132327186 GMT j9mm.82 - Excessive GC raised!
3STHSTTYPE 11:40:20:132299804 GMT j9mm.91 - GlobalGC end: workstackoverflow=0 overflowcount=0 weakrefs=147 soft=211 threshold=0 phantom=1 finalizers=558 newspace=0/0 oldspace=244776/268435456 loa=0/0
3STHSTTYPE 11:40:20:132115674 GMT j9mm.90 - GlobalGC collect complete
3STHSTTYPE 11:40:20:132078792 GMT j9mm.94 - Class unloading end: classloadersunloaded=23 classesunloaded=23
3STHSTTYPE 11:40:20:124809443 GMT j9mm.60 - Class unloading start
3STHSTTYPE 11:40:20:124787929 GMT j9mm.137 - Compact end: bytesmoved=260405760
3STHSTTYPE 11:40:18:657384744 GMT j9mm.136 - Compact start: reason=low free space (less than 4%)
3STHSTTYPE 11:40:18:657380832 GMT j9mm.57 - Sweep end
3STHSTTYPE 11:40:18:654921207 GMT j9mm.56 - Sweep start
3STHSTTYPE 11:40:18:654917295 GMT j9mm.55 - Mark end
3STHSTTYPE 11:40:18:010270187 GMT j9mm.54 - Mark start