Topic
  • 4 replies
  • Latest Post - ‏2014-11-13T12:54:16Z by Jeff Rusk (IBM)
SystemAdmin
SystemAdmin
184 Posts

Pinned topic LogFile protocol Bug

‏2013-01-22T09:30:58Z |
Hi,

While working with UDSM and Log File as source I came across the bug for which I found information in fourm, here is the link - https://qmmunity.q1labs.com/node/2304#comment-form .
I downloaded the RPM and installed it on console (upgrade mode - rpm -Uv) and deployed full configuration, followed by tomcat service restart.
Now error is gone but whenever I am trying to pull data by using same log source via Log File and FTP protocol, log source is not fetching data and also shows no error.
Please help me to get rid of this issue.

Thanks
Sunil -------Posted BY Sunil Nishankar
  • MaheshPatharkar
    MaheshPatharkar
    2 Posts

    Re: LogFile protocol Bug

    ‏2014-11-12T03:57:37Z  

    Hey Sunil,

    did your problem solved, please do let me know what solution you used. Also could you please let me know the bug id , could not access the link you mentioned.

    I am also experiencing the same problem.. UDSM+log file protocol to read a CSV file over FTP. DSM is reading a file but not completely, it just read 2-3 % of the lines randomly.

     

    Thanks,

    Mahesh

  • Jeff Rusk (IBM)
    Jeff Rusk (IBM)
    2 Posts

    Re: LogFile protocol Bug

    ‏2014-11-12T14:05:51Z  

    Hey Sunil,

    did your problem solved, please do let me know what solution you used. Also could you please let me know the bug id , could not access the link you mentioned.

    I am also experiencing the same problem.. UDSM+log file protocol to read a CSV file over FTP. DSM is reading a file but not completely, it just read 2-3 % of the lines randomly.

     

    Thanks,

    Mahesh

    Hi Mahesh.

    That original post is an old post from the Qmmunity site, which was originally from QRadar's Q1 Labs days, and no longer functional.  See decomission notice at https://qmmunity.q1labs.com/

    I suspect your issue is likely different than that original post though.  There are all kinds of reasons why a given configuration using log file protocol source may not give you what you expected, but be behaving exactly as configured (file permissions, regular expression chosen, etc - not that your issue relates to those, but there is not enough information here to troubleshoot).

    Sounds like very odd behaviour that Log File protocol source would not do in normal conditions though.  Can you tell us more about your csv file?  For one thing, what kind of line terminators does it have?  Do you have any sample lines you could paste along with your LSX?

    Updated on 2014-11-12T14:09:14Z at 2014-11-12T14:09:14Z by Jeff Rusk (IBM)
  • MaheshPatharkar
    MaheshPatharkar
    2 Posts

    Re: LogFile protocol Bug

    ‏2014-11-13T01:37:54Z  

    Hi Mahesh.

    That original post is an old post from the Qmmunity site, which was originally from QRadar's Q1 Labs days, and no longer functional.  See decomission notice at https://qmmunity.q1labs.com/

    I suspect your issue is likely different than that original post though.  There are all kinds of reasons why a given configuration using log file protocol source may not give you what you expected, but be behaving exactly as configured (file permissions, regular expression chosen, etc - not that your issue relates to those, but there is not enough information here to troubleshoot).

    Sounds like very odd behaviour that Log File protocol source would not do in normal conditions though.  Can you tell us more about your csv file?  For one thing, what kind of line terminators does it have?  Do you have any sample lines you could paste along with your LSX?

    Thanks for the reply Jeff.

     

    I have file having 10k rows, its a csv file. following are the sample entries from this file,

     

    LogSourceIdentifier,key,AAA,Data1,INDIA JALAN AMPANG,Data2,MUM
    LogSourceIdentifier,key,BBBB,Data1,USA,Data2,NY
    LogSourceIdentifier,key,CCCCCC,Data1,CANADA,Data2,ONT
    LogSourceIdentifier,key,DDD,Data1,UK,Data2,LON
    LogSourceIdentifier,key,ABCD,Data1,AUS,Data2,MEL

     

    I used, UDSM and Log file option (kept on FTP server) , selecting encoding as UTF8, I was able to read the file.

    But when I select EPS throttle as 100, QRadar only reads 100 rows and then stops. I changed that value to 500, again observed the same thing, it only read 500 rows and stopped.

    Ideally it should read with the EPS speed, lets say 100 rows per second and next 100 rows in 2nd second and on..till the end of the file.

    Why this is happening? have you came across such scenario?

     

    Note: I am not using LSX, I have created custom properties to read values from this logs. as customizing is not the scope of our service.

     

    Thanks,

    Mahesh

  • Jeff Rusk (IBM)
    Jeff Rusk (IBM)
    2 Posts

    Re: LogFile protocol Bug

    ‏2014-11-13T12:54:16Z  

    Thanks for the reply Jeff.

     

    I have file having 10k rows, its a csv file. following are the sample entries from this file,

     

    LogSourceIdentifier,key,AAA,Data1,INDIA JALAN AMPANG,Data2,MUM
    LogSourceIdentifier,key,BBBB,Data1,USA,Data2,NY
    LogSourceIdentifier,key,CCCCCC,Data1,CANADA,Data2,ONT
    LogSourceIdentifier,key,DDD,Data1,UK,Data2,LON
    LogSourceIdentifier,key,ABCD,Data1,AUS,Data2,MEL

     

    I used, UDSM and Log file option (kept on FTP server) , selecting encoding as UTF8, I was able to read the file.

    But when I select EPS throttle as 100, QRadar only reads 100 rows and then stops. I changed that value to 500, again observed the same thing, it only read 500 rows and stopped.

    Ideally it should read with the EPS speed, lets say 100 rows per second and next 100 rows in 2nd second and on..till the end of the file.

    Why this is happening? have you came across such scenario?

     

    Note: I am not using LSX, I have created custom properties to read values from this logs. as customizing is not the scope of our service.

     

    Thanks,

    Mahesh

    That is definitely not how the throttle should be behaving.  If EPS throttle is set fo 100 it should do 100 events in that one second (assuming the parsing can do it that fast) from that file and then move on to the next 100 for the next second.  I've heard no other reports of the throttle function in the Log File protocol source behaving in that manner.  You may need to open a Support ticket about this.  Support may need to put the Log File protocol source into debug logging mode as well (i.e. the com.q1labs.semsources.sources.remote package) to further identify what is going on here.

    Just to confirm, these files are not being actively written to while you are pulling data from them, correct? 

    Updated on 2014-11-13T12:56:10Z at 2014-11-13T12:56:10Z by Jeff Rusk (IBM)