Topic
  • 9 replies
  • Latest Post - ‏2015-09-15T19:48:05Z by JonathanPechtaIBM
JonathanPechtaIBM
JonathanPechtaIBM
28 Posts

Pinned topic Event: QRadar Open Mic #10 - Let's talk about Log Source Extensions (LSXs)

‏2015-08-11T19:58:21Z | discussion how-to lsxs mic open qradar

Event Announcement

Join members of the IBM Security QRadar Team on August 24th for the next QRadar Open Mic webcast where we talk about Log Source Extensions. During this session, we will be discussing how log source extensions work, how to write extensions, tips, and taking your questions. This session is not only an opportunity to learn about log source extensions, but also to provide feedback directly to developers, support, and product management about your experiences. During this session we will not be troubleshooting specific customer tickets or regex issues, but teaching and talking about extensions. 

 

 

Preliminary list of topics we plan to discuss:

  1. About log source extensions
  2. When to use log source extensions (Custom Event Properties vs using an Extension)
  3. Looking at an example payload
  4. LSX structure, limitations, & requirements
  5. Identity
  6. Creating QID Map Entries (qidmap_cli.sh)
  7. Mapping events
  8. Discussion & Questions

 

Note: This topic list may change depending on the volume of questions and how the discussion goes. Since this is a live presentation, we may divert from the slides to discuss topics from the online chat.

 

 

The videos are posted to YouTube playlist at the following URL: https://www.youtube.com/playlist?list=PLFip581NcL2XgKVHE3ayAS9JqI9u6dwAV.

 

Individual videos

Let's talk about Log Source Extensions (LSXs)

  1. Video 1: Introduction & about
  2. Video 2: Getting source data
  3. Video 3: LSX structure
  4. Video 4: Identity
  5. Video 5: Limitations and tips
  6. Video 6: Uploading and mapping events
  7. Video 7: Q&A / discussion (Coming soon)

 

Attachments

  • Serenity
    Serenity
    3 Posts

    Re: Event: QRadar Open Mic #10 - Let's talk about Log Source Extensions (LSXs)

    ‏2015-08-14T18:05:17Z  

    Why operational changes are made, without any customer (or, apparently, even support team notification?  (The intentional, and significant, reduction of the number of supported capture groups, which broke numerous of our LSX's, with no visibility to why they suddenly would not work.)

    Why does the upload engine not properly vet the LSX when it exceeds the newly reduced capture group limit?

    Why does nothing else in the system generate alerts for any pre-existing LSXs, that exceed the newly capture group limit?  And/or, why is that not vetted as part of each upgrade script, to look for this problem, and alert the customer to this issue?

    How are you proposing that we test our RegEx?   We previously utilized the Custom Properties dialog, which is presumably JRE, considering the Custom Properties expressions are supposed to be codified in JRE.  I've now been told by QR support that the RegEx engine utilized for LSX was also recently replaced (maybe at the same time that the capture group limit was drastically reduced), and that is not (now) compatible with JRE.

    Why are the docs not kept current?  I had to open a ticket to find out whether it supported IPv6 variables, and what their names were.  The only docs that I have, and can find, are years old.

    Why is this component such a step-child?  The sales people have always pitched this as the solution that makes QR so flexible, yet in almost all cases where I've ever called technical support, they quickly retreat to the "it's not supported" line, and start telling me that I'll have to contract for professional services.  That might be understandable if the product was WELL documented, and FULLY documented, but as noted in a previous question, the docs don't appear to ever be updated, and also don't even explain all of the existing parameters.  There are many variable names used in the LSX construct that we are simply left to GUESS about the underlying logic by which they actually function.  The match-groups being a perfect example of this, where there are no examples of using "more than one", and when I tried to get clarification, the "professional services" response was all that I received.  I actually got one of these working, but purely by trial and error.

  • Adam_Frank
    Adam_Frank
    10 Posts

    Re: Event: QRadar Open Mic #10 - Let's talk about Log Source Extensions (LSXs)

    ‏2015-08-14T18:48:22Z  
    • Serenity
    • ‏2015-08-14T18:05:17Z

    Why operational changes are made, without any customer (or, apparently, even support team notification?  (The intentional, and significant, reduction of the number of supported capture groups, which broke numerous of our LSX's, with no visibility to why they suddenly would not work.)

    Why does the upload engine not properly vet the LSX when it exceeds the newly reduced capture group limit?

    Why does nothing else in the system generate alerts for any pre-existing LSXs, that exceed the newly capture group limit?  And/or, why is that not vetted as part of each upgrade script, to look for this problem, and alert the customer to this issue?

    How are you proposing that we test our RegEx?   We previously utilized the Custom Properties dialog, which is presumably JRE, considering the Custom Properties expressions are supposed to be codified in JRE.  I've now been told by QR support that the RegEx engine utilized for LSX was also recently replaced (maybe at the same time that the capture group limit was drastically reduced), and that is not (now) compatible with JRE.

    Why are the docs not kept current?  I had to open a ticket to find out whether it supported IPv6 variables, and what their names were.  The only docs that I have, and can find, are years old.

    Why is this component such a step-child?  The sales people have always pitched this as the solution that makes QR so flexible, yet in almost all cases where I've ever called technical support, they quickly retreat to the "it's not supported" line, and start telling me that I'll have to contract for professional services.  That might be understandable if the product was WELL documented, and FULLY documented, but as noted in a previous question, the docs don't appear to ever be updated, and also don't even explain all of the existing parameters.  There are many variable names used in the LSX construct that we are simply left to GUESS about the underlying logic by which they actually function.  The match-groups being a perfect example of this, where there are no examples of using "more than one", and when I tried to get clarification, the "professional services" response was all that I received.  I actually got one of these working, but purely by trial and error.

    I cannot speak much to the capture group limit, however to address some of the questions.

    For Testing Regex's I've always used the Custom Properties Dialog, it uses a Javascript engine vs JRE so there are some minor differences that I've seen when attempting to use look-ahead and look-behind assertions. but it works in 99% of the cases.

    The regex engine in LSX has not changed that I know of, with 7.2.5 I have not seen any changes with the LSX's I work on. I will check with dev to confirm this and update this post if necessary.

    For the Docs you can find the most up to date ones here: http://www-01.ibm.com/support/knowledgecenter/SS42VS_7.2.5/com.ibm.dsm.doc/c_LogSourceGuide_ExtDocs_about.html%23c_logsourceguide_extdocs_about?lang=en

    It's good to hear that using multiple match-groups works, I have written 100's of LSX's over the years and it was something I had never tried.

     

  • Serenity
    Serenity
    3 Posts

    Re: Event: QRadar Open Mic #10 - Let's talk about Log Source Extensions (LSXs)

    ‏2015-08-17T20:50:31Z  

    I cannot speak much to the capture group limit, however to address some of the questions.

    For Testing Regex's I've always used the Custom Properties Dialog, it uses a Javascript engine vs JRE so there are some minor differences that I've seen when attempting to use look-ahead and look-behind assertions. but it works in 99% of the cases.

    The regex engine in LSX has not changed that I know of, with 7.2.5 I have not seen any changes with the LSX's I work on. I will check with dev to confirm this and update this post if necessary.

    For the Docs you can find the most up to date ones here: http://www-01.ibm.com/support/knowledgecenter/SS42VS_7.2.5/com.ibm.dsm.doc/c_LogSourceGuide_ExtDocs_about.html%23c_logsourceguide_extdocs_about?lang=en

    It's good to hear that using multiple match-groups works, I have written 100's of LSX's over the years and it was something I had never tried.

     

    >>>I cannot speak much to the capture group limit, however to address some of the questions.<<<

    Some change that was made in the last year or so, apparently to resolve some performance issue.  That's understandable, but not documenting the change (not even the tech support guys knew about it), or even (to this day) generating an error for LSX uploads that exceed the limit, are the real sins.

    >>>For Testing Regex's I've always used the Custom Properties Dialog, it uses a Javascript engine vs JRE so there are some minor differences that I've seen when attempting to use look-ahead and look-behind assertions. but it works in 99% of the cases.<<<

    I guess the question here is are those minor differences well documented, in current LSX documentation?  When an LSX doesn't work right, you have no debugging capability.  Minor differences that result in parsing failures, result in failures that then result in painful trial-and-error exercises.  Again, the failure to document the inconsistencies result in unwarranted pain and suffering by those of us attempting to develop the LSX on the end-user side.  Coding complex variants of RegEx expressions can already be classified as a "black art", IMHO, and not having visibility to some syntactical discrepancy that will fail, blindly, is unforgivable.  (I speak from experience.  I've toyed around with different parts of a RegEx for hours, trying to figure out how to change it, before stumbling upon some innocuous modification that suddenly made it work, even though ALL of the variations worked fine in the Custom Properties dialog.)

    >>>The regex engine in LSX has not changed that I know of, with 7.2.5 I have not seen any changes with the LSX's I work on. I will check with dev to confirm this and update this post if necessary.<<<

    Sorry, I was incomplete in what I said.  It has changed in some previous release, during the last year (apparently at the same time that the capture group "upper limit" was reduced).  Again, that was communicated to me by support, from what he was told by development; all adjunct to the discovery that my LSX failures were the result of changes made by development.  (And that discovery only occurred because, in my second attempt to get someone to help me, "one brave support tech" was willing to spin up an older version of QR, so that he could verify my claim that my broken LSXs had previously all worked fine.  And the "bug" from that finding, resulted in the development response… "oh yeah, BTW, we significantly reduced the capture group limit.")

    >>>For the Docs you can find the most up to date ones here: http://www-01.ibm.com/support/knowledgecenter/SS42VS_7.2.5/com.ibm.dsm.doc/c_LogSourceGuide_ExtDocs_about.html%23c_logsourceguide_extdocs_about?lang=en <<<

    FWLIW, I actually find that online format to be far more difficult to peruse than is the "flat" PDF version, but also, related to my original concern, even this version makes no mention of the IPv6-related variables, nor does it delve into the more complex capabilities that would seem to exist with the whole custom DSM environment.  And shortcoming like that, make we wonder "what other things" might be missing, beyond what I'm already aware of.  What is frustrating for me is that, obviously, someone went to a lot of trouble to codify some (seemingly) elaborate functionality in this whole LSX construct, but then, because someone is either unwilling to properly document all of its varied features, or has deemed those to be only within the purview of professional services, all of that coding effort is mostly wasted, as 99% of us don't have any idea how some of those more elegant, enhanced features work (except thru guesswork, and trial-end-error testing).

     

    You asked for questions.  You seem to be doing this presentation to encourage a wider-scale use of the "magic of LSX", but if nobody at IBM/QR is willing to address how poorly this has been documented and supported, then my prediction is just a lot more "ill will", especially when those customers call, and the first words out of the support team's mouths are "we don't support LSX, you'll need to engage prof. svcs. if you want help with it.".

    All, again, FWLIW.  No need to reply.  I do appreciate your attempt to offer some help.

  • EricLauzon
    EricLauzon
    17 Posts

    Re: Event: QRadar Open Mic #10 - Let's talk about Log Source Extensions (LSXs)

    ‏2015-08-20T12:40:13Z  

    A few question for the OpenMic.

    LSX are really a usefull feature, but i have a concern when trying to deploy to many different LSX.

    If a DSM does not exist for a device class and you want to bind a device to a LSX you have to use Universal DSM.

    From there you bind your LSX and then map proper event and you are done.

    But what happens when you have alot of different class of LSX all bounded to UDSM.

    Wouldn't it be possible to have the possibility to register a User Defined DSM where device type would be bound to a LSX.

    The question behind it is that if you define custom event properties for UDSM couldn't it theorically impact all UDSM?

    Now i guess that the work arround would be to define custom properties for event name but then this could become a tidious task if your LSX/UDSM combo

    has 100 event type and +- 10 common properties, then you would end up having 1000 custom properties for that particular use case.

    While if UDSM/LSX would be registered as DSM then i guess it would be possible to reduce that to 10 custom properties?

     

  • Andrew Wagg
    Andrew Wagg
    3 Posts

    Re: Event: QRadar Open Mic #10 - Let's talk about Log Source Extensions (LSXs)

    ‏2015-08-21T12:55:27Z  

    One question we have it how do you deal with a system sending events via syslog but using a non-standard syslog header?

    Basically they thought it was a good idea to put an extra field between the timestamp and hostname, rather than in the payload where it belongs. for example:

    <14>Aug 17 14:33:17 025696: HOSTNAME: PAYLOAD

  • KrishnanSarkar
    KrishnanSarkar
    1 Post

    Re: Event: QRadar Open Mic #10 - Let's talk about Log Source Extensions (LSXs)

    ‏2015-08-24T15:07:39Z  

    Hello

    I have just two questions.

    1. Is there a way to map events to qids pre-emptively via the terminal?

    And,

    2. Is there any available documentation for other similar back-end commands?

    Thank you!

  • JonathanPechtaIBM
    JonathanPechtaIBM
    28 Posts

    Re: Event: QRadar Open Mic #10 - Let's talk about Log Source Extensions (LSXs)

    ‏2015-08-26T18:41:32Z  

    One question we have it how do you deal with a system sending events via syslog but using a non-standard syslog header?

    Basically they thought it was a good idea to put an extra field between the timestamp and hostname, rather than in the payload where it belongs. for example:

    <14>Aug 17 14:33:17 025696: HOSTNAME: PAYLOAD

    Hey awagg,

     

    Sorry for the delay. As mentioned in the open mic, I wanted to make sure I discussed this in detail. There are two potential options here for resolving this issue.

     

    Option 1

    I think that you might be able to use the Syslog Redirect protocol with your Universal DSM. In a nutshell, what the Syslog Redirect protocol does is take a value (such an IP or hostname) from the Syslog payload and injects this value in to the packet header. The packet header is the highest priority for QRadar when determining the event source. In this case, you are going to want to look at the event payload to see if the hostname or IP is represent there that you can use. You have a slightly different issue than we normally see because there is a non-standard value in your header. This solution should work just fine for you and option 2 is likely not required.

     

    Below is a generic payload I created to represent what your event might look like. This is an anti-virus payload, so yours might look very different than my example. In this case, we would want to try to replace 14>Aug 17 14:33:17 025696: myhostname.example.com header with a new header that contains the value Local:10.10.10.10, which is the true IP of the device that sent the event.

     

    My example event

    <14>Aug 17 14:33:17 025696: myhostname.example.com Driver Remote Code Execution attack blocked: SYSTEM,Local: 10.10.10.10,Local: 000000000000,Remote: ,Remote: 11.11.11.11,Remote: 000000000000,Inbound,TCP,Intrusion ID: 0,Begin: 2013-09-28 16:02:40,End: 2013-09-28 16:02:40,Occurrences: 1,Application: SYSTEM,Location: Default,User: ,Domain: ,Local Port 445,Remote Port 11111,CIDS Signature ID: 21111,CIDS Signature string: SMB Srv.sys Driver Remote Code Execution,CIDS Signature

    Note: There might be multiple IPs or hostnames in an event payload. You are going to want to select the best option to represent the device that sent the event.

     

     

    How to configure your log source

     

    IMPORTANT: For the Log Source Identifier field, you will notice that I list "Source IP" generically when adding the log source. This is because with a non-standard header I'm not sure what QRadar thinks the event source IP or hostname really is. You should put the Packet IP address of the system sending the event in the Log Source Identifier field. Then assign your protocol TCP/UDP and upload/apply your LSX. Your LSX would then need to be adjusted to identify the proper values from the event payload since there is now going to be a new syslog header. Your appliance must also be configured to sent to port 517 TCP or 517 UDP.

     

    Payload value that represents the actual IP of your appliance/server/application: Local: 10.10.10.10
    Log Source Identifier Regex field in your log source would be: Local:\s(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})

     


    What Syslog Redirect is going to give you as a new event:
    <22>May 23 10:56:31 10.10.10.10 <14>Aug 17 14:33:17 025696: myhostname.example.com Driver Remote Code Execution attack blocked: SYSTEM,Local: 10.10.10.10,Local: 000000000000,Remote: ,Remote: 11.11.11.11,Remote: 000000000000,Inbound,TCP,Intrusion ID: 0,Begin: 2013-09-28 16:02:40,End: 2013-09-28 16:02:40,Occurrences: 1,Application: SYSTEM,Location: Default,User: ,Domain: ,Local Port 445,Remote Port 11111,CIDS Signature ID: 21111,CIDS Signature string: SMB Srv.sys Driver Remote Code Execution,CIDS Signature SubID: 73920,Intrusion URL: ,Intrusion Payload URL:

     

    Notice in bold where the event above now has a new syslog header created by QRadar. Syslog redirect injects a new packet header before processing the event. Your LSX would then need to be crafted to identify the proper values from the event payload.

     

    These are the values you can put in your LSX as discussed during the open mic:

    • EventName
    • EventCategory
    • SourceIp
    • SourcePort
    • SourceIpPreNAT
    • SourceIpPostNAT
    • SourceMAC
    • SourcePortPreNAT
    • SourcePortPostNAT
    • DestinationIp
    • DestinationPort
    • DestinationIpPreNAT
    • DestinationIpPostNAT
    • DestinationPortPreNAT
    • DestinationPortPostNAT
    • DestinationMAC
    • DeviceTime
    • Protocol
    • UserName
    • HostName
    • GroupName
    • NetBIOSName
    • ExtraIdentityData
    • SourceIpv6
    • DestinationIpv6

    Any other fields else that is important data from the payload that you are going to want to capture is going to be a custom event property in QRadar.

     

     

    Option 2

    The other option is to send these payloads to an intermediate server that is running syslog-ng, then use syslog redirect. You can configure an instance of Syslog-ng to listen and route these packets to QRadar. When you forward the Syslog event, you can configure the Syslog-ng server inject it's own header.

     

    This would mean that this is what gets forwarded to QRadar: [Syslog-ng header 1][Syslog non-compliant header2][event payload]
     
    However, the IP in the "redirected" header will have the Syslog-ng servers address (unless the config is specalized). So, then you apply syslog redirect to the event and get this, which is messy, but it also works:


    <22>May 23 10:56:31 10.10.10.10 [Syslog-ng header 1][Syslog non-compliant header2][event payload]

     

    I do not think that option 2 is required, but it is a use case that someone else might hit, so I wanted to outline both potential solutions.

     

    Hope this helps.

     

  • JonathanPechtaIBM
    JonathanPechtaIBM
    28 Posts

    Re: Event: QRadar Open Mic #10 - Let's talk about Log Source Extensions (LSXs)

    ‏2015-08-26T20:34:16Z  

    Hello

    I have just two questions.

    1. Is there a way to map events to qids pre-emptively via the terminal?

    And,

    2. Is there any available documentation for other similar back-end commands?

    Thank you!

    KrishnanSarkar,

     

     

    Q1. Is there a way to map events to qids pre-emptively via the terminal?

    A1: No. Unfortunately, there is no method of mapping events without using the QRadar user interface. You can create your custom QIDs and import them to QRadar before you create your LSX, however, there needs to be events in the QRadar user interface to map these events. This might be something that we could add as a possible API endpoint. You would need to open an RFE to make a feature request around being able to

     

    Q2. Is there any available documentation for other similar back-end commands?

    A2: The documentation covers all supported command-line parameters. There is an "unofficial log source extension guide" that shows how to use a script called logrun.pl. This script allows you to take payloads from events and replay them through QRadar. This can be helpful to generate events for LSX testing without having to wait for events to stream in from the event source. If you have the full event with payloads in a file, you can use logrun to replay events through QRadar. This will generate duplicate events, but might save time without having to wait or log in to the event source to trip a specific event you are working on parsing. Logrun is in /opt/qradar/bin and it has a help interface through typing --help. The unoffical guide is discussed in this post: https://www.ibm.com/developerworks/community/forums/html/topic?id=77777777-0000-0000-0000-000014970193 . This is the only instance I can think of a script that is not in the official documentation that might be helpful.

     

     

  • JonathanPechtaIBM
    JonathanPechtaIBM
    28 Posts

    Re: Event: QRadar Open Mic #10 - Let's talk about Log Source Extensions (LSXs)

    ‏2015-09-15T19:48:05Z  

    Almost all of the videos are posted to YouTube now at the following URL: https://www.youtube.com/playlist?list=PLFip581NcL2XgKVHE3ayAS9JqI9u6dwAV.

     

    Individual videos

    Let's talk about Log Source Extensions (LSXs)

    1. Video 1: Introduction & about
    2. Video 2: Getting source data
    3. Video 3: LSX structure
    4. Video 4: Identity
    5. Video 5: Limitations and tips
    6. Video 6: Uploading and mapping events
    7. Video 7: Q&A / discussion (Coming soon)