IC SunsetThe developerWorks Connections platform will be sunset on December 31, 2019. On January 1, 2020, this forum will no longer be available. More details available on our FAQ.
Topic
  • 12 replies
  • Latest Post - ‏2016-01-13T14:56:04Z by JonathanPechtaIBM
Muhammad_Hamza
Muhammad_Hamza
6 Posts

Pinned topic QRadar 'Duplicate Logs'

‏2015-12-29T16:27:48Z | events

Hello,

I am using QRadar Log Manager 7.2.6

I sometimes get Duplicate Logs. Can anyone help me in fixing the issue?

 

  • Aaron_Breen(IBM)
    Aaron_Breen(IBM)
    96 Posts

    Re: QRadar 'Duplicate Logs'

    ‏2015-12-29T17:15:08Z  

    Do you mean events or flows? You stated you have log manager which would imply you have now flows but then you state you have removed duplicate flows from qflow.

  • Muhammad_Hamza
    Muhammad_Hamza
    6 Posts

    Re: QRadar 'Duplicate Logs'

    ‏2015-12-29T19:19:37Z  

    Do you mean events or flows? You stated you have log manager which would imply you have now flows but then you state you have removed duplicate flows from qflow.

    Thanks for the response,

    I meant events.

     

  • Nikodim
    Nikodim
    40 Posts

    Re: QRadar 'Duplicate Logs'

    ‏2015-12-30T07:58:47Z  

    Can you post a screenshot with this issue?

  • Muhammad_Hamza
    Muhammad_Hamza
    6 Posts

    Re: QRadar 'Duplicate Logs'

    ‏2015-12-30T12:21:55Z  
    • Nikodim
    • ‏2015-12-30T07:58:47Z

    Can you post a screenshot with this issue?


    Here it is

  • Nikodim
    Nikodim
    40 Posts

    Re: QRadar 'Duplicate Logs'

    ‏2015-12-30T12:25:26Z  


    Here it is

    You can start investigation with capturing traffic with tcdump or other similar software and check whether your Log Source sending non-duplicated data.

  • Aaron_Breen(IBM)
    Aaron_Breen(IBM)
    96 Posts

    Re: QRadar 'Duplicate Logs'

    ‏2015-12-31T12:37:24Z  
    • Nikodim
    • ‏2015-12-30T12:25:26Z

    You can start investigation with capturing traffic with tcdump or other similar software and check whether your Log Source sending non-duplicated data.

    Is Coalescing turned on or off for this log source? If off and the log source sends 2 events where the normalized indexed values are the same (IP, username etc) then there will be 2. If on it should combine them with an event count of 2

    Also, look at the raw payloads, are there any differences? Lastly, as suggested by Nikodim, check the sending server to see if there are 2 events being generated then sent to Qradar

  • Muhammad_Hamza
    Muhammad_Hamza
    6 Posts

    Re: QRadar 'Duplicate Logs'

    ‏2016-01-01T13:39:07Z  

    Is Coalescing turned on or off for this log source? If off and the log source sends 2 events where the normalized indexed values are the same (IP, username etc) then there will be 2. If on it should combine them with an event count of 2

    Also, look at the raw payloads, are there any differences? Lastly, as suggested by Nikodim, check the sending server to see if there are 2 events being generated then sent to Qradar

    Thanks for the response,

    Coalescing is turned on
    for this log source, and it is showing two uncombined events, I tried tcpdump command to capture the events but it doesn't capture any logs whereas I am getting logs in QRadar Log Activity, I checked their payloads / Raw Events,  still all the values for those duplicate events are same. Could it be the issue in Log Source sending logs?

    Regards

  • Aaron_Breen(IBM)
    Aaron_Breen(IBM)
    96 Posts

    Re: QRadar 'Duplicate Logs'

    ‏2016-01-03T17:00:57Z  

    Thanks for the response,

    Coalescing is turned on
    for this log source, and it is showing two uncombined events, I tried tcpdump command to capture the events but it doesn't capture any logs whereas I am getting logs in QRadar Log Activity, I checked their payloads / Raw Events,  still all the values for those duplicate events are same. Could it be the issue in Log Source sending logs?

    Regards

    Make sure you are performing the tcpdump from the proper QRADAR machine. If you have a console and collectors, you need to ssh into the console first then ssh to the collector receiving the syslog. Also make sure you are performing a tcpdump using the IP address of the machine sending the logs.

  • Muhammad_Hamza
    Muhammad_Hamza
    6 Posts

    Re: QRadar 'Duplicate Logs'

    ‏2016-01-04T19:00:25Z  

    Make sure you are performing the tcpdump from the proper QRADAR machine. If you have a console and collectors, you need to ssh into the console first then ssh to the collector receiving the syslog. Also make sure you are performing a tcpdump using the IP address of the machine sending the logs.

    Thanks Aaron for the help

    I have investigated the logs using tcpdump command and verified that the Log Source is sending a single event/log to QRadar and QRadar Log Manager is showing the duplicate values in Log Activity and generated reports. And it randomly duplicates an event and not some particular event, also not all events are duplicated.
    Also we are using Custom Event Properties for the log source.

    What can be the cause of this issue?

    Regards

  • JonathanPechtaIBM
    JonathanPechtaIBM
    197 Posts

    Re: QRadar 'Duplicate Logs'

    ‏2016-01-06T15:52:54Z  

    Thanks Aaron for the help

    I have investigated the logs using tcpdump command and verified that the Log Source is sending a single event/log to QRadar and QRadar Log Manager is showing the duplicate values in Log Activity and generated reports. And it randomly duplicates an event and not some particular event, also not all events are duplicated.
    Also we are using Custom Event Properties for the log source.

    What can be the cause of this issue?

    Regards

    @Muhammad_Hamza,
    @Nikodim

     

    This is an issue we are looking in to currently and there is an APAR pending. This event duplication issue is very rare based on our testing and can occur on QRadar systems at 7.2.5 Patch 4 IF02 and above. We were able to replicate this issue and get a duplicate event on a Universal DSM after letting events soak long enough as it seems to be very infrequent, however, it can occur also on normal log sources (non-Universal DSMs). The event that gets generated occurs after licensing in ECS, meaning that the duplicate event does not impact your licensed EPS rate in any way. ​

     

    When the APAR is opened, I'll post a link here for reference so you can subscribe to this issue.
     

  • Muhammad_Hamza
    Muhammad_Hamza
    6 Posts

    Re: QRadar 'Duplicate Logs'

    ‏2016-01-11T14:52:50Z  

    @Muhammad_Hamza,
    @Nikodim

     

    This is an issue we are looking in to currently and there is an APAR pending. This event duplication issue is very rare based on our testing and can occur on QRadar systems at 7.2.5 Patch 4 IF02 and above. We were able to replicate this issue and get a duplicate event on a Universal DSM after letting events soak long enough as it seems to be very infrequent, however, it can occur also on normal log sources (non-Universal DSMs). The event that gets generated occurs after licensing in ECS, meaning that the duplicate event does not impact your licensed EPS rate in any way. ​

     

    When the APAR is opened, I'll post a link here for reference so you can subscribe to this issue.
     

    Thank you for your response.

    Till IBM fixes this issue, is there any way around to fix this, for example if we put a check on uniqueness of any attribute, that we are extracting from payload. For example my payload has a unique ID attribute. 

    Can that be used?

    Regards

  • JonathanPechtaIBM
    JonathanPechtaIBM
    197 Posts

    Re: QRadar 'Duplicate Logs'

    ‏2016-01-13T14:56:04Z  

    There is an APAR open for this issue here: APAR IV80076.

     

    There might be a method of using an advanced search to pick out values that have the duplicate log source times that can simplify this. However, I know when we replicated this issue in our lab, we ran searches that were grouped by multiple unique custom property values and also grouped by the log source. When the search was run you should only get a count of 1 for each event. If you get any Multiple (2) values, then these are your duplicate events.

     

    If would suggest that you subscribe to this APAR so you get an alert for when the issue is resolved.