Topic
  • 8 replies
  • Latest Post - ‏2016-02-25T05:56:08Z by damian-sirius
SystemAdmin
SystemAdmin
117 Posts

Pinned topic Using Flow and Event Log Hashing

‏2009-07-30T15:07:50Z |
Users who are required to meet compliancy standards have asked about the "Flow / Event Log Hashing" option in the Admin Console / System settings tab. From the online help, these are explained as follows:

  • Flow Log Hashing - Enables or disables the ability for QRadar to store a hash file for every stored flow log file. The default is No.
  • Event Log Hashing Enables or disables the ability for QRadar to store a hash file for every stored event log file. The default is No.
Hashing Algorithm

You can use a hashing algorithm for database storage and encryption. You can use one of the following hashing algorithms:
  • Message-Digest Hash Algorithm - Transforms digital signatures into shorter values called Message-Digests (MD).
  • Secure Hash Algorithm (SHA) Hash Algorithm - Standard algorithm that creates a larger (60 bit) MD.

Specify the log hashing algorithm you want to use for your deployment. The options are:
  • MD2 - Algorithm defined by RFC 1319.
  • MD5 - Algorithm defined by RFC 1321.
  • SHA-1 - Default. Algorithm defined by Secure Hash Standard (SHS), NIST FIPS 180-1.
  • SHA-256 - Algorithm defined by the draft Federal Information Processing Standard 180-2, SHS. SHA-256 is a 255-bit hash algorithm intended for 128 bits of security against security attacks.
  • SHA-384 - Algorithm defined by the draft Federal Information Processing Standard 180-2, SHS. SHA-384 is a bit hash algorithm is provided by truncating the SHA-512 output.
  • SHA-512 - Algorithm defined by the draft Federal Information Processing Standard 180-2, SHS. SHA-512 is a bit hash algorithm intended to provide 256 bits of security.


When this option is enabled, any system that is writing out event and flow data will create hash files in the /store/ariel/(events or flows)/md/(y/m/d/hour)/ directory structure. For Example:

root@csd6 11# pwd
/store/ariel/flows/md/2009/7/30/11
root@csd6 11# ls -ltr | tail -5
-rw-r--r-- 1 root root 20 Jul 30 11:58 flows~57_0~dc415f99847a4424~a09308b32e2e1300.SHA-1
-rw-r--r-- 1 root root 20 Jul 30 11:59 payload_flows~58_0~8fdcc87aad3a44c2~968a71af814d4689.SHA-1
-rw-r--r-- 1 root root 20 Jul 30 11:59 flows~58_0~8fdcc87aad3a44c2~968a71af814d4689.SHA-1
-rw-r--r-- 1 root root 20 Jul 30 12:00 payload_flows~59_0~2f51eb8edb2b444c~877bc9f883bc09ef.SHA-1
-rw-r--r-- 1 root root 20 Jul 30 12:00 flows~59_0~2f51eb8edb2b444c~877bc9f883bc09ef.SHA-1
root@csd6 11#

These files in themselves are not text readable, but can be used to verify that your files have not been tampered with, since originally created. To verify that your files have not been tampered with, you will need to login to the system with the data storage for events or flows and run the verification utility to check the files - the event and flow viewer interfaces do not currently check for tampered files.

root@csd6 11# /opt/qradar/bin/check_ariel_integrity.sh -h
/usr/java/j2sdk/bin/java -Dapplication.baseURL=file:/opt/qradar/conf/ -Djava.awt.headless=true -server -Dapp_id=check_ariel_integrity com.q1labs.ariel.io.SecureFileWriter
data base name is required
Configured data bases:
events
flows
hprofs
usage: Usage:
-a,--algo hashing algorithm (MD2, MD5, SHA-1, SHA-256, SHA-384,
SHA-512), optional, normally specified in arielConfig.xml, by default
SHA-1
-d,--duration time duration to look files for in minutes, for example
-d 5
-n,--name data base name, for example -n flows
-t,--endtime end time, for example -t "2009/07/30 12:06", optional,
by default current system time
-h,--help print this message
-r,--hashroot overrides hash root directory, optional, normally
specified in arielConfig.xml

root@csd6 11# /opt/qradar/bin/check_ariel_integrity.sh -n flows -d 3
/usr/java/j2sdk/bin/java -Dapplication.baseURL=file:/opt/qradar/conf/ -Djava.awt.headless=true -server -Dapp_id=check_ariel_integrity com.q1labs.ariel.io.SecureFileWriter -n flows -d 3
Verifying files for data base flows in /store/ariel/flows/records using hashes from /store/ariel/flows/md
Hash algorithm:SHA-1
Start time:2009/07/30 11:40
End time:2009/07/30 11:43
Verifying /store/ariel/flows/records/2009/7/30/11/flows~41_0~3e8fdf46853b41e1~961e30f724a8e093:OK
Verifying /store/ariel/flows/payloads/2009/7/30/11/payload_flows~41_0~3e8fdf46853b41e1~961e30f724a8e093:OK
Verifying /store/ariel/flows/records/2009/7/30/11/flows~40_0~4fab23c0dccb4912~8c9af0029c0a7a1c:OK
Verifying /store/ariel/flows/payloads/2009/7/30/11/payload_flows~40_0~4fab23c0dccb4912~8c9af0029c0a7a1c:OK
root@csd6 11# pwd
/store/ariel/flows/md/2009/7/30/11
root@csd6 11# ls -ltr | tail -5
-rw-r--r-- 1 root root 20 Jul 30 11:58 flows~57_0~dc415f99847a4424~a09308b32e2e1300.SHA-1
-rw-r--r-- 1 root root 20 Jul 30 11:59 payload_flows~58_0~8fdcc87aad3a44c2~968a71af814d4689.SHA-1
-rw-r--r-- 1 root root 20 Jul 30 11:59 flows~58_0~8fdcc87aad3a44c2~968a71af814d4689.SHA-1
-rw-r--r-- 1 root root 20 Jul 30 12:00 payload_flows~59_0~2f51eb8edb2b444c~877bc9f883bc09ef.SHA-1
-rw-r--r-- 1 root root 20 Jul 30 12:00 flows~59_0~2f51eb8edb2b444c~877bc9f883bc09ef.SHA-1
root@csd6 11#


and that will verify that the flow and event log files have not been tampered with. The hash files are generated in memory before the files are written to disk, so there is no chance that files could be tampered with prior to generating the hash files.

Impact
Depending on the encryption type you choose, the performance hit averages between 4-8% of total load. On systems that are not running at capacity (up to around 4500EPS on each system calculating the hash values), this impact will be negligible. Note also, that you may not have any of these "hash" files on the console itself, if there is no local event or flow storage, and that the files will actually exist on the systems with your event and flow data storage.
dwight s.

Q1 Labs customer support-------Posted BY dwight (q1)
  • SystemAdmin
    SystemAdmin
    117 Posts

    Re: Flow-Event-Hashing

    ‏2010-04-07T18:31:00Z  
    Taking off from the notes in "using-flow-and-event-log-hashing", I am viewing this as something I would use to verify that files are not changed, so I am not looking at encrypting data. I have three questions:
    1) This mentions multiple hash algorithms. Is any of these preferred when I am dealing with PCI certification?
    2) This appears to have been written in 2008 and states that nothing "currently" checks for tampered files. At that time it was a manual process. Is that still true? Is there a reasonable way, or is a reasonable way planned, to actively monitor for tampering with files?
    3) Under "impact" it mentions "encryption", and adding 4-8% load. Is that impact related to calculating the hash (ie it has nothing to do with encrypting the data)?

    Also, in general, I would appreciate any comments on security measures people are taking related to QRadar when it is employed as part of a PCI solution.
    Posted By RPorritt
  • SystemAdmin
    SystemAdmin
    117 Posts

    Re: event hashing

    ‏2010-04-16T18:30:40Z  
    Hi Ron, a few comments:

    1) I'm not familiar enough with hashing algorithms and PCI requirements to know if one is going to be any better than another. I'd suggest you choose the one your most familiar with.

    2) Verification of tampering is still a manual process, and currently, we do not have any plans to make this an automated process.

    3) The performance impact would be during the write process of data. This overhead would be caused by the processing of data to create the hashes, as written to disk. The reading and verification would have their own distinct impact, depending on when you did the check and how much data you were verifying.

    dwight.


    Posted By dwight (q1)
  • SystemAdmin
    SystemAdmin
    117 Posts

    Re: Event Hashing Error

    ‏2011-08-04T20:28:01Z  
    Running "/opt/qradar/bin/check_ariel_integrity.sh -n events -d 1" on v7.0.0.202206 results in...

    Verifying /store/ariel/events/records/2011/8/4/15/events~3_0~24891a501c224674~a42b2c4a421a1e7a - Digest file /store/ariel/events/md/2011/8/4/15/events~3_0~24891a501c224674~a42b2c4a421a1e7a.SHA-1 does not exist or corrupted

    It looks like the hash files are being written to "/store/ariel/events/md/8/4/15/" (missing the year from the path) which is why the script can't find them.

    Posted By Josh McCune
  • SystemAdmin
    SystemAdmin
    117 Posts

    Re: storage encryption

    ‏2011-11-02T07:46:59Z  
    Is there a way of encrypting stored events? Since hashing is only covering integrity and not confidentiality. I am looking on how to ensure the confidentiality of the stored events.
    Posted By Dennis Jensen
  • SystemAdmin
    SystemAdmin
    117 Posts

    Re: Dennis, There is no

    ‏2011-11-02T16:31:16Z  
    Dennis,

    There is no currently supported method to encrypt the events on disk. The Disk IO and CPU would take a hit performing encryption, especially at higher flow and EPS rates. I can look for an FR and you to it (I noticed you logged a support ticket on this also)
    Posted By Aaron.Breen
  • SystemAdmin
    SystemAdmin
    117 Posts

    Re: Event Hashing Error

    ‏2011-11-30T11:31:00Z  
    Hi,

    I need to use the check_ariel_integrity.sh but I can see that it still fails to run.

    Are there any plans to fix this script, or is there a work around to do it?

    Sorry, looks like it's fixed in an update
    Posted By Hendrik Johns
  • SystemAdmin
    SystemAdmin
    117 Posts

    Re: check_ariel_integrity.sh script

    ‏2011-11-30T14:36:10Z  
    Hi Hendrik ..

    Indeed, the check_ariel_integrity.sh script was fixed in 7.0 maintenance release 2 (build 214487)

    dwight
    Posted By dwight (q1)
  • damian-sirius
    damian-sirius
    1 Post

    Re: Using Flow and Event Log Hashing

    ‏2016-02-25T05:56:08Z  

    In that case where hashing of events and flows are enabled and you end up migrating to a newer appliance.  Will restoring of the configuration onto the new appliance able to read the hashed events/flows?