Topic
  • 3 replies
  • Latest Post - ‏2014-01-22T20:16:22Z by Mr10001
Mr10001
Mr10001
20 Posts

Pinned topic Custom Log Source DSM Not Parsing Fields

‏2014-01-21T20:02:00Z | dsm lsx qradar regex udsm

After hours of searching and failed trouble shooting attempts, I've relented and decided to ask for directions... =)

I'm attempting to send a log from an unsupported device to QRadar and have it parsed correctly.  Unfortunately, the fields do not appear to be parsing correctly and the events are still coming in as unknown.

Here are the steps I've taken thus far...

1. Setup the Log Source using the Universal DSM.

- Turned on Coalescing Events, and Store Event Payload

- Set the Log Source Extension to the correct LSX

- Set Extension Use Condition to "Parsing Override"

2. Had the log(s) sent to QRadar and validated that they are coming in under the Log Source setup in step one.

3. Created and uploaded my LSX file without errors or issues.

- Set Use Condition to "Parsing Override"

4. Mapped one of the Events, using the "Map Event" button.

- I was under the impression that when I am in the pop-up box for the Map Event portion, I should see a value in the Log Source Event ID field, which is equal to the EventName that I parsed out with the associated LSX (Log Source Extension).  However, my value is blank/null/empty.  My Log Source Type is "GenericDSM", My Log Source Event Category is "GenericDSM", my Original QID is "11750924"... which is what I Mapped the Event to.

5. Sent additional events to QRadar in the hopes they would now be parsed correctly.

- The events are not parsed e.g. The following matcher fields are not showing up as parsed in the event (EventName, SourceIP, UserName, HostName, DeviceTime).

6. I've tested each of my REGEX's within the Extract Property area and all work great.

Here's a copy of my LSX with dummy information (which includes an example of an event)...

<?xml version="1.0" encoding="UTF-8" ?> 
<!-- 
Author:Matthew 
Device Type:UDSM
Device Version:?
Protocol:Syslog
 
Example Event:
<142>2014-01-20 07:01:58,157 EXAMPLE_LOGIN_AUDIT  - eventname="LOGIN", environment="XXXXX", appid="XXXX", area="XXXXX", username="USERNAME", sourceip="X.X.X.X", httpreferer="https://xxx.xxx.com/", 
 
Fields:
 
1. Date/Time = DateTime-
2. Event Type e.g. LOGIN = EventName-TEST
3. User Name = UserName-TEST
4. Source IP = SourceIp-TEST
5. Host = HostName-TEST
  --> 
<device-extension xmlns="event_parsing/device_extension">
<!--  Do not remove the "allEventNames" value  --> 
<pattern id="allEventNames" xmlns="">
<![CDATA[(.*)]]> 
</pattern>
<!--  Everything below this line can be modified --> 
 
<pattern id="EventName-TEST" xmlns="">
<![CDATA[eventname\=\"(LOGIN)\"]]> 
</pattern>
  
<pattern id="SourceIp-TEST" xmlns="">
<![CDATA[sourceip="(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})"]]> 
</pattern>
  
<pattern id="UserName-TEST" xmlns="">
<![CDATA[username="(.*?)",]]> 
</pattern>
  
<pattern id="HostName-TEST" xmlns="">
<![CDATA[httpreferer="http\w?://(.*?)/]]> 
  </pattern>
  
<pattern id="DateTime-TEST" xmlns="">
<![CDATA[(\d{4}-\d{1,2}-\d{1,2}\s\d{1,2}:\d{1,2}:\d{1,2}),]]> 
  </pattern>
  
<match-group order="1" description="TEST Successful Authentication" xmlns="">
  <matcher field="EventName" order="1" pattern-id="EventName-TEST" capture-group="1" /> 
  <matcher field="SourceIp" order="1" pattern-id="SourceIp-TEST" capture-group="1" /> 
  <matcher field="UserName" order="1" pattern-id="UserName-TEST" capture-group="1" /> 
  <matcher field="HostName" order="1" pattern-id="HostName-TEST" capture-group="1" /> 
  <matcher field="DeviceTime" order="1" pattern-id="DateTime-TEST" capture-group="1" /> 
  <event-match-multiple pattern-id="allEventNames" capture-group-index="1" device-event-category="Unknown" send-identity="OverrideAndAlwaysSend" /> 
  </match-group>
  </device-extension>

If anyone has any ideas on this one, they would be greatly appreciated.

  • Mr10001
    Mr10001
    20 Posts
    ACCEPTED ANSWER

    Re: Custom Log Source DSM Not Parsing Fields

    ‏2014-01-22T16:05:30Z  

    - Coalescing Events should be to test

    At a quick glance I do not see anything wrong with the Regex and or extension. Try the coalescing off and restart tomcat with a service tomcat restart

    If you still have an issue, make a small change to the XML or save as a new name and upload again BUT tail -f /var/log/qradar.error at the same time to see if there are any errors related to XML

    All else, tail the qradar.error and restart tomcat & ecs

    service tomcat restart

    service ecs restart

    Aaron,

    Thank you for the quick response on this!

    After trying what you suggested I still came up empty handed, my guess is I had a formatting issue with my LSX.  I say that because after receiving some validation from you on my LSX, I took it upon myself to rebuild my LSX from the template and it appears to have partially worked.  I'm getting the user ID mapped, the source IP mapped, and my host name mapped.

    However, I'm now seeing the event name is "unknown", the low level category is "Unknown", and the Event Description is "Unknown Generic Event", even after I've mapped the event to a QID (11750924) and received the confirmation page after mapping (which I did not receive before).

    Is it possible to have these values set to something other than unknown?  Is my low level category being set because my LSX has device-event-category="unknown"?  I was under the impression that after mapping the event to a QID it would change the category and event name.

     

    Thanks for any help on this

  • Mr10001
    Mr10001
    20 Posts
    ACCEPTED ANSWER

    Re: Custom Log Source DSM Not Parsing Fields

    ‏2014-01-22T20:16:22Z  
    • Mr10001
    • ‏2014-01-22T16:05:30Z

    Aaron,

    Thank you for the quick response on this!

    After trying what you suggested I still came up empty handed, my guess is I had a formatting issue with my LSX.  I say that because after receiving some validation from you on my LSX, I took it upon myself to rebuild my LSX from the template and it appears to have partially worked.  I'm getting the user ID mapped, the source IP mapped, and my host name mapped.

    However, I'm now seeing the event name is "unknown", the low level category is "Unknown", and the Event Description is "Unknown Generic Event", even after I've mapped the event to a QID (11750924) and received the confirmation page after mapping (which I did not receive before).

    Is it possible to have these values set to something other than unknown?  Is my low level category being set because my LSX has device-event-category="unknown"?  I was under the impression that after mapping the event to a QID it would change the category and event name.

     

    Thanks for any help on this

    Good news, I changed my QID Map and that did the trick.  Apparently the one I was using originally had sparse details.

  • Aaron_Breen(IBM)
    Aaron_Breen(IBM)
    112 Posts

    Re: Custom Log Source DSM Not Parsing Fields

    ‏2014-01-21T20:45:00Z  

    - Coalescing Events should be to test

    At a quick glance I do not see anything wrong with the Regex and or extension. Try the coalescing off and restart tomcat with a service tomcat restart

    If you still have an issue, make a small change to the XML or save as a new name and upload again BUT tail -f /var/log/qradar.error at the same time to see if there are any errors related to XML

    All else, tail the qradar.error and restart tomcat & ecs

    service tomcat restart

    service ecs restart

  • Mr10001
    Mr10001
    20 Posts

    Re: Custom Log Source DSM Not Parsing Fields

    ‏2014-01-22T16:05:30Z  

    - Coalescing Events should be to test

    At a quick glance I do not see anything wrong with the Regex and or extension. Try the coalescing off and restart tomcat with a service tomcat restart

    If you still have an issue, make a small change to the XML or save as a new name and upload again BUT tail -f /var/log/qradar.error at the same time to see if there are any errors related to XML

    All else, tail the qradar.error and restart tomcat & ecs

    service tomcat restart

    service ecs restart

    Aaron,

    Thank you for the quick response on this!

    After trying what you suggested I still came up empty handed, my guess is I had a formatting issue with my LSX.  I say that because after receiving some validation from you on my LSX, I took it upon myself to rebuild my LSX from the template and it appears to have partially worked.  I'm getting the user ID mapped, the source IP mapped, and my host name mapped.

    However, I'm now seeing the event name is "unknown", the low level category is "Unknown", and the Event Description is "Unknown Generic Event", even after I've mapped the event to a QID (11750924) and received the confirmation page after mapping (which I did not receive before).

    Is it possible to have these values set to something other than unknown?  Is my low level category being set because my LSX has device-event-category="unknown"?  I was under the impression that after mapping the event to a QID it would change the category and event name.

     

    Thanks for any help on this

  • Mr10001
    Mr10001
    20 Posts

    Re: Custom Log Source DSM Not Parsing Fields

    ‏2014-01-22T20:16:22Z  
    • Mr10001
    • ‏2014-01-22T16:05:30Z

    Aaron,

    Thank you for the quick response on this!

    After trying what you suggested I still came up empty handed, my guess is I had a formatting issue with my LSX.  I say that because after receiving some validation from you on my LSX, I took it upon myself to rebuild my LSX from the template and it appears to have partially worked.  I'm getting the user ID mapped, the source IP mapped, and my host name mapped.

    However, I'm now seeing the event name is "unknown", the low level category is "Unknown", and the Event Description is "Unknown Generic Event", even after I've mapped the event to a QID (11750924) and received the confirmation page after mapping (which I did not receive before).

    Is it possible to have these values set to something other than unknown?  Is my low level category being set because my LSX has device-event-category="unknown"?  I was under the impression that after mapping the event to a QID it would change the category and event name.

     

    Thanks for any help on this

    Good news, I changed my QID Map and that did the trick.  Apparently the one I was using originally had sparse details.