Topic
2 replies Latest Post - ‏2013-10-21T17:16:08Z by Stan
AdamOw
AdamOw
1 Post
ACCEPTED ANSWER

Pinned topic Running several FileSource operators

‏2013-10-18T18:53:59Z |

Hi,

I used similar script to generate over 100 FileSource operators (./etc/logfiles.csv contains log file names with full path and correcponding operator name, split by coma):

composite LogNameGen {

graph
<%
my @Unions;
my $file = './etc/logfiles.csv';
open(my $data, '<', $file);
 
while (my $line = <$data>) {
chomp $line;
my @fields = split "," , $line;
%>
        stream<rstring outTuple, rstring filename> <%=$fields[1]%> = FileSource()
        {
            param
                file: "<%=$fields[0]%>" ;
                format : line ;
                hotFile : true ;
            output
                 <%=$fields[1]%>: filename = "<%=$fields[1]%>" ;
        }        
<%
       push @Unions, $fields[1];

        }
        my $LogUnion = join (";", @Unions);
%>

stream<rstring outTuple, rstring filename> Logs
        = Union(
    <%
        print $LogUnion;

    %>
    ) {}
}

I'm doing rather unimpressive things after that (custom operator + sink).

I ran this for 20ish FileSource and it was executing fine, I tried it for over 100 and ran into OutOfMemory exception that crashes hc service and forces quit on Streams Studio at the same time. At first I thought that this might be studio related problem, so I increased the maximum xmx param in eclipse config file from 256mb to 8 gb. Didn't help. Increasing 'open file' param in 'ulimits' from 1024 to 4096 didn't help. I just stared at output from 'top' during task startup, but memory utilization stays at ~6gb, and the machine has 64gb total memory. Maybe I should do some more 'advanced' method for memory monitoring, but it appears everything else works ok.

In hc.trace I'm getting a lot of error messages like:

"[16391] ERROR :::Core.Utils M[Thread.cpp:create:140]  - Attempt #10 of 10 calling 'pthread_create(&myTid, myAttrs, threadStart, (void*)p.get())' failed with error code 11 (which translates to 'Resource temporarily unavailable') --> abandoning with an exception"

 

Anyone has ideas how to solve this?

 

 

  • Stan
    Stan
    76 Posts
    ACCEPTED ANSWER

    Re: Running several FileSource operators

    ‏2013-10-21T14:20:32Z  in response to AdamOw

    Typically 'error code 11' indicates a ulimit resouce limitation.  Your seeting of Open files = 4096 is 4% of the recommended valiue.  Try increasing this to 100000 and also increase (if needed - it often is) Max user Processes.  The recommeneded values are documented here:  http://pic.dhe.ibm.com/infocenter/streams/v3r1/topic/com.ibm.swg.im.infosphere.streams.install-admin.doc/doc/ibminfospherestreams-install-operating-system-settings.html

    • open files value = 100000
    • max user processes value = pending signals value

    If the Out of Memory issue contineus please supply the stack trace so we can see what module if encountering the OOM.

     

       

    • Stan
      Stan
      76 Posts
      ACCEPTED ANSWER

      Re: Running several FileSource operators

      ‏2013-10-21T17:16:08Z  in response to Stan

      One important thing - be sure you are setting and checking your soft limits - these are the ones applied to processes that will cause failures if the exceeded.   The command to do this (not in the  current Info Center) is 'uliimit -aS'

      And a senior member of my team pointed out that the values in my previous post are the values Streams Dev use when testing so saying they are documented recommendations is  incorrect.  They values do work well and have helped ohters resolve resource limit failures.  They are good values to use to avoid resource limitation failures.