Topic
4 replies Latest Post - ‏2014-11-13T12:54:16Z by Jeff Rusk (IBM)
SystemAdmin
SystemAdmin
184 Posts
ACCEPTED ANSWER

Pinned topic LogFile protocol Bug

‏2013-01-22T09:30:58Z |
Hi,

While working with UDSM and Log File as source I came across the bug for which I found information in fourm, here is the link - https://qmmunity.q1labs.com/node/2304#comment-form .
I downloaded the RPM and installed it on console (upgrade mode - rpm -Uv) and deployed full configuration, followed by tomcat service restart.
Now error is gone but whenever I am trying to pull data by using same log source via Log File and FTP protocol, log source is not fetching data and also shows no error.
Please help me to get rid of this issue.

Thanks
Sunil -------Posted BY Sunil Nishankar
  • MaheshPatharkar
    MaheshPatharkar
    2 Posts
    ACCEPTED ANSWER

    Re: LogFile protocol Bug

    ‏2014-11-12T03:57:37Z  in response to SystemAdmin

    Hey Sunil,

    did your problem solved, please do let me know what solution you used. Also could you please let me know the bug id , could not access the link you mentioned.

    I am also experiencing the same problem.. UDSM+log file protocol to read a CSV file over FTP. DSM is reading a file but not completely, it just read 2-3 % of the lines randomly.

     

    Thanks,

    Mahesh

    • Jeff Rusk (IBM)
      Jeff Rusk (IBM)
      2 Posts
      ACCEPTED ANSWER

      Re: LogFile protocol Bug

      ‏2014-11-12T14:05:51Z  in response to MaheshPatharkar

      Hi Mahesh.

      That original post is an old post from the Qmmunity site, which was originally from QRadar's Q1 Labs days, and no longer functional.  See decomission notice at https://qmmunity.q1labs.com/

      I suspect your issue is likely different than that original post though.  There are all kinds of reasons why a given configuration using log file protocol source may not give you what you expected, but be behaving exactly as configured (file permissions, regular expression chosen, etc - not that your issue relates to those, but there is not enough information here to troubleshoot).

      Sounds like very odd behaviour that Log File protocol source would not do in normal conditions though.  Can you tell us more about your csv file?  For one thing, what kind of line terminators does it have?  Do you have any sample lines you could paste along with your LSX?

      Updated on 2014-11-12T14:09:14Z at 2014-11-12T14:09:14Z by Jeff Rusk (IBM)
      • MaheshPatharkar
        MaheshPatharkar
        2 Posts
        ACCEPTED ANSWER

        Re: LogFile protocol Bug

        ‏2014-11-13T01:37:54Z  in response to Jeff Rusk (IBM)

        Thanks for the reply Jeff.

         

        I have file having 10k rows, its a csv file. following are the sample entries from this file,

         

        LogSourceIdentifier,key,AAA,Data1,INDIA JALAN AMPANG,Data2,MUM
        LogSourceIdentifier,key,BBBB,Data1,USA,Data2,NY
        LogSourceIdentifier,key,CCCCCC,Data1,CANADA,Data2,ONT
        LogSourceIdentifier,key,DDD,Data1,UK,Data2,LON
        LogSourceIdentifier,key,ABCD,Data1,AUS,Data2,MEL

         

        I used, UDSM and Log file option (kept on FTP server) , selecting encoding as UTF8, I was able to read the file.

        But when I select EPS throttle as 100, QRadar only reads 100 rows and then stops. I changed that value to 500, again observed the same thing, it only read 500 rows and stopped.

        Ideally it should read with the EPS speed, lets say 100 rows per second and next 100 rows in 2nd second and on..till the end of the file.

        Why this is happening? have you came across such scenario?

         

        Note: I am not using LSX, I have created custom properties to read values from this logs. as customizing is not the scope of our service.

         

        Thanks,

        Mahesh

        • Jeff Rusk (IBM)
          Jeff Rusk (IBM)
          2 Posts
          ACCEPTED ANSWER

          Re: LogFile protocol Bug

          ‏2014-11-13T12:54:16Z  in response to MaheshPatharkar

          That is definitely not how the throttle should be behaving.  If EPS throttle is set fo 100 it should do 100 events in that one second (assuming the parsing can do it that fast) from that file and then move on to the next 100 for the next second.  I've heard no other reports of the throttle function in the Log File protocol source behaving in that manner.  You may need to open a Support ticket about this.  Support may need to put the Log File protocol source into debug logging mode as well (i.e. the com.q1labs.semsources.sources.remote package) to further identify what is going on here.

          Just to confirm, these files are not being actively written to while you are pulling data from them, correct? 

          Updated on 2014-11-13T12:56:10Z at 2014-11-13T12:56:10Z by Jeff Rusk (IBM)