Topic
IC4NOTICE: developerWorks Community will be offline May 29-30, 2015 while we upgrade to the latest version of IBM Connections. For more information, read our upgrade FAQ.
4 replies Latest Post - ‏2014-02-05T14:49:41Z by Jason_Keirstead (IBM)
JasonMelbourne
JasonMelbourne
2 Posts
ACCEPTED ANSWER

Pinned topic Combining data from flows and events

‏2013-08-12T17:02:46Z |

Curious the best way to automate this investigation.

1.) Rule fires on outbound traffic over 1GB from watching the flows.   
 
Standard Flow 2013-08-12 11:06:47 10.151.208.58 9422 54.230.123.206 80
Source Payload: 4339363098 bytes
 
This flow sees an internal IP (10.151.208.58) hit an external IP (54.230.123.206) and transfer 4GB,  the issue is that internal IP is really a proxy.   So the next step is then to manually look for the proxy event that will tell us the original source.
 
2.) Bluecoat|src=10.10.76.45|srcport=55444|dst=54.230.123.206|dstport=80|username=-|devicetime=[12/08/2013:16:13:16 GMT]|s-action=TCP_HIT|sc-status=200|cs-method=GET|time-taken=13|sc-bytes=8605|cs-bytes=8605|cs-uri-scheme=http|cs-host=d16s8pqtk4uodx.cloudfront.net|cs-uri-path=/foxnews-prod/load.js|cs-uri-query=-|cs-uri-extension=js|cs-auth-group=-|s-hierarchy=-|rs(Content-Type)=application/x-javascript|cs(User-Agent)=Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; InfoPath.2; MS-RTC EA 2; MS-RTC LM 8; .NET4.0C; .NET4.0E)|cs(Referer)=http://www.foxnews.com/us/2013/08/11/|sc-filter-result=OBSERVED|filter-category="Content Servers"|cs-uri=http://d16s8pqtk4uodx.cloudfront.net/foxnews-prod/load.js|s-ip=10.151.208.58
 
 
Now we know the original source 10.10.76.45.... 
 
I can do this externally via ArielClient, reference sets, and scripting, but it seems like there should be a way within Qradar.   I was thinking maybe something like have the blue coat populate a reference set of maps or sets..
 
SourceIP, SourcePort, Destination IP, Destination Port, URL 
 
Then have the Large File rule fire like normally, but pull the match from the referenceset of sets into the event as well? Or add volume to the end of it? Either would be perfect.  Im still fuzzy on the exact purpose of a reference map of maps or reference map of sets.   It seems like its just a different way to structure data but the only option to use them is still just... "is X contained in this data".     It seems like the rules would need to be able to write or read specific fields in order for those to be really useful?

 

Thanks,

Jason

 
 
 
 
  • dwight s (IBM)
    dwight s (IBM)
    4 Posts
    ACCEPTED ANSWER

    Re: Combining data from flows and events

    ‏2013-08-12T18:53:25Z  in response to JasonMelbourne

    Hi Jason, ... 

    AdamF and I were chatting about this (he may respond on here too), and currently there's no way to trigger something like this in QRadar. What you'd need, would be a method that when a rule fires, it runs a query of some sort, either an ariel search or reference set search (assuming your proxy logs were creating a reference map of ip->destinations).  Timing would be an issue, though, as most likely you would have multiple internal addresses hitting the same external site, a lot of the time. 

    We've talked about ways to "annotate" events, where we could append/include additional details to events as they come into the system, that sounds like it might be something you could use too, but that's still just in the discussion phase. 

    something like this perhaps?
    a) populate ip addresses to URL destinations in reference set, with a short TTL to keep it to a smaller size
    b) on your "Large Transfer" rule , have it fire messages into syslog, that show up in the qradar.log file
    c) write script that monitors qradar.log for CRE responses
    d) in your script, query postgres reference set table for that ip address and what URL's they are hitting?

    would that get what you need?   Admittedly, there's not really anything you could do with that data, unless your large file response was to also create an offense, then have your script generate a new event back into QRadar that could be parsed (LSX) and add to the offense? I'm leaving out a few steps, but would that provide what you're looking for?

    dwight

    • JasonMelbourne
      JasonMelbourne
      2 Posts
      ACCEPTED ANSWER

      Re: Combining data from flows and events

      ‏2013-08-12T21:31:41Z  in response to dwight s (IBM)

      Yes, I built something similar to get around qradar's lack of the ability to use large lists of CIDR ranges, it pulls unique permitted non 25/53 traffic via psql query of a RS im populating via rule, checks against millions of ips on various lists, then uses the arielclient search to pull event info about the worst violators, and logs the output to a file being watched by tail2syslog.   [The keys here were only writing unique IPs to the RS and never checking an IP against a watchlist twice until that list has been updated. whitelisting my blacklist lookups as it were]

      Setting it up to stitch together the events & flows for this use case as well but wanted to make sure I wasnt missing some way to do it via reference sets of maps.

      Great answer and Id love to hear more discussion on this, there a probably a million different applications for this functionality!

      I would think the simplest to implement though would be just to allow the match in a reference set/of maps to be annotated to the event/offense.   That would instantly make reference sets of sets and maps extremely useful while adding minimal overhead.   

      Currently the reference set rules work like... 

      if grep "xyz"  ReferenceSetA >/dev/null

      then...

      Instead just have it do

      if grep "xyz"  ReferenceSetA  > $TMP

      then...

      And add the action

      "Annotate Reference set match to,, [the event|offense|email]"   which writes that $TMP file to the event

      I like where your going with the query as well, but this solution would have nearly no impact to performance and add huge functionality.   Can we make it a enhancement request?

      Thanks!

      Jason

       

       

       

      • dwight s (IBM)
        dwight s (IBM)
        4 Posts
        ACCEPTED ANSWER

        Re: Combining data from flows and events

        ‏2013-08-13T14:31:36Z  in response to JasonMelbourne

        hmm.   Regarding enhancement requests, I believe users create them directly via the support portal.  Look for a link for "request for enhancement" where you create new tickets.

        Back to our example, I wonder if you could use the option to tag events with building blocks or rules.  For example, you could have a rule populating a reference set (or map)  based on the criteria you require, such a source ip address to URL map based on proxy logs, for URL's you don't trust?  whatever filter you want, or just all urls, perhaps, though that's bound to go quickly, so you may want to limit the list of URL's your looking for. (xforce list, perhaps?)

        Then, when you find a flow that matches your large file transfer rule, and part of that rule is to also check that source ip address against the reference map set of ips & bad urls.  If found, then the rule fires, generates email or offence, and tags the event that it matches your rule.  The order of operations would be an issue here, i think, because if the "large transfer" occurred before we had the ip address->url mapping in the reference set, it would not fire.  You could use 2 rules, 1 to tag the large transfer, then a second rule, that does the reference set/map lookup, then increases the magnitude/severity of the offence?   Would that be comparable what you're looking for?

        dwight

  • Jason_Keirstead (IBM)
    3 Posts
    ACCEPTED ANSWER

    Re: Combining data from flows and events

    ‏2014-02-05T14:49:41Z  in response to JasonMelbourne

    Hi Jason - There is a way to solve this use case for you in QRadar 7.2 MR1 and later. What you can do is bind a custom right click action to your Flow viewer that will take the IP address, port, and time information in, run an event search for the BlueCoat event using those parameters, and return the resulting IP to yourself... in this way, when you see these proxy logs, finding out the true IP is only a right click away.

    Take a look at the "Enhancing the right-click menu for event and flow viewers" in the 7.2.1 documentation (http://www-01.ibm.com/support/docview.wss?uid=swg27040321) for how to do this. If you need any help, feel free to send me an email and I will help where I can.