Topic
  • 1 reply
  • Latest Post - ‏2013-08-20T07:38:10Z by Katherine Sanders
d.coli
d.coli
2 Posts

Pinned topic memory leak or retained variables in large job?

‏2013-08-19T16:27:57Z |

I'm having trouble with Cast Iron orchestration that I'm attempting to run in Studio. When I leave the default heap size alone (512MB) the job freezes at around 39K records. When I double the heap to 1024MB (the max) it freezes at 80K records. So it's pretty obvious the job is retaining something every time the loop runs. But what types of looped activities retain data between cycles?

I include an image of my orchestration here to show how simple this job seems to be. Only one javascript function is being used to translate a value (in the Read XML activity). It's a custom function I wrote that uses only local vars (assuming there could be some sort of shared scope between functions - I doubt it.) Here it is.

The Update activity is also an Insert (this is Cast Iron 6.4).

Is there anything about one of these steps that retains variables from step to step? I'm reusing the same loop variable, which is just a segment ("item") of the translated XML file.

my orchestration

 

 

Here's the xsd for my membership loop value:



<?
xml version
=
"1.0" 
encoding
=
"UTF-8"
?>
<!-- edited with XMLSpy v2012 (x64) (http://www.altova.com) by Colin MacAllister (Columbia Business School) -->
<!--W3C Schema generated by XMLSpy v2012 (x64) (http://www.altova.com)-->
<xs:schema 
xmlns:xs
=
"http://www.w3.org/2001/XMLSchema"
>  
<xs:element 
name
=
"rss"
> 
   
<xs:complexType> 
     
<xs:sequence> 
       
<xs:element 
name
=
"channel" 
minOccurs
=
"0" 
maxOccurs
=
"unbounded"
> 
         
<xs:complexType> 
           
<xs:sequence> 
             
<xs:element 
name
=
"title"
> 
               
<xs:simpleType> 
                 
<xs:restriction 
base
=
"xs:string"
/> 
               
</xs:simpleType> 
             
</xs:element> 
             
<xs:element 
name
=
"link"
> 
               
<xs:simpleType> 
                 
<xs:restriction 
base
=
"xs:anyURI"
/> 
               
</xs:simpleType> 
             
</xs:element> 
             
<xs:element 
name
=
"description" 
type
=
"xs:string"
/> 
             
<xs:element 
name
=
"pubDate"
> 
               
<xs:simpleType> 
                 
<xs:restriction 
base
=
"xs:string"
/> 
               
</xs:simpleType> 
             
</xs:element> 
             
<xs:element 
name
=
"generator"
> 
               
<xs:simpleType> 
                 
<xs:restriction 
base
=
"xs:anyURI"
/> 
               
</xs:simpleType> 
             
</xs:element> 
             
<xs:element 
name
=
"language"
> 
               
<xs:simpleType> 
                 
<xs:restriction 
base
=
"xs:string"
> 
                   
<xs:enumeration 
value
=
"en"
/> 
                 
</xs:restriction> 
               
</xs:simpleType> 
             
</xs:element> 
             
<xs:element 
name
=
"item" 
minOccurs
=
"0" 
maxOccurs
=
"unbounded"
> 
               
<xs:complexType> 
                 
<xs:sequence> 
                   
<xs:element 
name
=
"cgId" 
type
=
"xs:unsignedInt"
/> 
                   
<xs:element 
name
=
"firstName"
> 
                     
<xs:simpleType> 
                       
<xs:restriction 
base
=
"xs:string"
/> 
                     
</xs:simpleType> 
                   
</xs:element> 
                   
<xs:element 
name
=
"lastName"
> 
                     
<xs:simpleType> 
                       
<xs:restriction 
base
=
"xs:string"
/> 
                     
</xs:simpleType> 
                   
</xs:element> 
                   
<xs:element 
name
=
"email"
> 
                     
<xs:simpleType> 
                       
<xs:restriction 
base
=
"xs:string"
/> 
                     
</xs:simpleType> 
                   
</xs:element> 
                   
<xs:element 
name
=
"accountType"
> 
                     
<xs:simpleType> 
                       
<xs:restriction 
base
=
"xs:string"
/> 
                     
</xs:simpleType> 
                   
</xs:element> 
                   
<xs:element 
name
=
"member" 
type
=
"xs:unsignedInt"
/> 
                   
<xs:element 
name
=
"officer" 
type
=
"xs:unsignedInt"
/> 
                   
<xs:element 
name
=
"officerPosition"
> 
                     
<xs:simpleType> 
                       
<xs:restriction 
base
=
"xs:string"
/> 
                     
</xs:simpleType> 
                   
</xs:element> 
                   
<xs:element 
name
=
"activeOfficer" 
type
=
"xs:unsignedInt"
/> 
                   
<xs:element 
name
=
"membershipEndDate" 
type
=
"xs:string"
/> 
                 
</xs:sequence> 
               
</xs:complexType> 
             
</xs:element> 
           
</xs:sequence> 
         
</xs:complexType> 
       
</xs:element> 
     
</xs:sequence> 
     
<xs:attribute 
name
=
"version" 
use
=
"required"
> 
       
<xs:simpleType> 
         
<xs:restriction 
base
=
"xs:decimal"
/> 
       
</xs:simpleType> 
     
</xs:attribute> 
   
</xs:complexType>  
</xs:element>
</xs:schema>

 

  • Katherine Sanders
    Katherine Sanders
    47 Posts

    Re: memory leak or retained variables in large job?

    ‏2013-08-20T07:38:10Z  

    Which activity is running when the out of memory occurs?  Also why do you have the second for loop?  Cast Iron can update records in batches, you usually don't have to iterate through the records.

    I wouldn't recommend testing with large amounts of data in Studio.  You could temporarily limit your query to test that your orchestration works functionally for a smaller set of records.  Then remove the limit, publish the project to the appliance, and do the full load test there.  The appliance will store large variables on disk to stop it running out of memory (ensure persistence is turned on in the orchestration properties, click on the green circle at the start to check).