Topic
  • 2 replies
  • Latest Post - ‏2014-03-05T22:10:40Z by TuringElite
TuringElite
TuringElite
2 Posts

Pinned topic RE: Log source extensions (LSXs) that generate a large number of asset updates

‏2014-03-03T15:57:03Z |

I have a few questions about the recently posted notice: http://www-01.ibm.com/support/docview.wss?uid=swg21666016

Isn't the intended function of send-identity="OverrideAndAlwaysSend" to update asset profiles? How can I tell if the volume of updates is too much?

How long is too long for loading the asset tab? (I wouldn't describe it as fast.) How can you tell how many updates to an asset there have been?

I don't see any warnings for "TX Sentry" in /var/log/qradar.log . About every 30 seconds "Processing TX" and "Saving TX" show about 0.3MB. What is an acceptable volume?

  • Jonathan.Pechta (IBM)
    Jonathan.Pechta (IBM)
    11 Posts
    ACCEPTED ANSWER

    Re: RE: Log source extensions (LSXs) that generate a large number of asset updates

    ‏2014-03-04T01:06:27Z  

    TuringElite,

     

    Yes, OverrideAndAlwaysSend is used to ensure that the event includes a flag to force identity, but there are situations where identity is helpful and some where it is not.  The article on send-identity="OverrideAndAlwaysSend" was written was to provide information on how the identity field can impact your asset profile when using log source extensions. You might not be seeing any issues, but we wanted to make customers aware of potential problems. Depending on the log source extension and the type of event information that is being parsed, it can create issues for some administrators when identity is always sent with every event. The purpose of the article is to raise awareness about always sending identity in every event generated by the Log Soure Extension (LSX). It is actually the first of a few articles about assets and identity.

     

    A large number of LSX examples that were published where the "OverrideAndAlwaysSend" value was set as the default. In QRadar 7.1, having an LSX that sends identity on every event was not an issue due to how we only correlated asset information by the IPV4 address. In QRadar 7.2, the asset model was updated and expanded to treat assets more like an entity with unique asset ID values. Asset IDs can be updated by more information that just an IP address. In QRadar 7.2 assets can support multiple IP addresses, multiple MAC Addresses, and multiple hostnames (DNS or NetBIOS/WINS). The purpose of this is to make the system more flexible than just looking at an IP address to determine if an event in the system is for a new asset or if the event is for an existing asset, which should be updated with new information.

     

    As mentioned, there is going to be a follow up article on how to determine if you have assets with large amounts of identity data. This article will describe how to determine the number of MAC address or IP addresses belong to a single asset ID in the system and help administrators spot potential issues.

     

    I didn't cover all of your questions, but did want you to know that there is more info pending. That information should cover a majority of your questions.

     

    If you want to read some more on assets, there is also these two articles which are already available.

     

  • Jonathan.Pechta (IBM)
    Jonathan.Pechta (IBM)
    11 Posts

    Re: RE: Log source extensions (LSXs) that generate a large number of asset updates

    ‏2014-03-04T01:06:27Z  

    TuringElite,

     

    Yes, OverrideAndAlwaysSend is used to ensure that the event includes a flag to force identity, but there are situations where identity is helpful and some where it is not.  The article on send-identity="OverrideAndAlwaysSend" was written was to provide information on how the identity field can impact your asset profile when using log source extensions. You might not be seeing any issues, but we wanted to make customers aware of potential problems. Depending on the log source extension and the type of event information that is being parsed, it can create issues for some administrators when identity is always sent with every event. The purpose of the article is to raise awareness about always sending identity in every event generated by the Log Soure Extension (LSX). It is actually the first of a few articles about assets and identity.

     

    A large number of LSX examples that were published where the "OverrideAndAlwaysSend" value was set as the default. In QRadar 7.1, having an LSX that sends identity on every event was not an issue due to how we only correlated asset information by the IPV4 address. In QRadar 7.2, the asset model was updated and expanded to treat assets more like an entity with unique asset ID values. Asset IDs can be updated by more information that just an IP address. In QRadar 7.2 assets can support multiple IP addresses, multiple MAC Addresses, and multiple hostnames (DNS or NetBIOS/WINS). The purpose of this is to make the system more flexible than just looking at an IP address to determine if an event in the system is for a new asset or if the event is for an existing asset, which should be updated with new information.

     

    As mentioned, there is going to be a follow up article on how to determine if you have assets with large amounts of identity data. This article will describe how to determine the number of MAC address or IP addresses belong to a single asset ID in the system and help administrators spot potential issues.

     

    I didn't cover all of your questions, but did want you to know that there is more info pending. That information should cover a majority of your questions.

     

    If you want to read some more on assets, there is also these two articles which are already available.

     

  • TuringElite
    TuringElite
    2 Posts

    Re: RE: Log source extensions (LSXs) that generate a large number of asset updates

    ‏2014-03-05T22:10:40Z  

    TuringElite,

     

    Yes, OverrideAndAlwaysSend is used to ensure that the event includes a flag to force identity, but there are situations where identity is helpful and some where it is not.  The article on send-identity="OverrideAndAlwaysSend" was written was to provide information on how the identity field can impact your asset profile when using log source extensions. You might not be seeing any issues, but we wanted to make customers aware of potential problems. Depending on the log source extension and the type of event information that is being parsed, it can create issues for some administrators when identity is always sent with every event. The purpose of the article is to raise awareness about always sending identity in every event generated by the Log Soure Extension (LSX). It is actually the first of a few articles about assets and identity.

     

    A large number of LSX examples that were published where the "OverrideAndAlwaysSend" value was set as the default. In QRadar 7.1, having an LSX that sends identity on every event was not an issue due to how we only correlated asset information by the IPV4 address. In QRadar 7.2, the asset model was updated and expanded to treat assets more like an entity with unique asset ID values. Asset IDs can be updated by more information that just an IP address. In QRadar 7.2 assets can support multiple IP addresses, multiple MAC Addresses, and multiple hostnames (DNS or NetBIOS/WINS). The purpose of this is to make the system more flexible than just looking at an IP address to determine if an event in the system is for a new asset or if the event is for an existing asset, which should be updated with new information.

     

    As mentioned, there is going to be a follow up article on how to determine if you have assets with large amounts of identity data. This article will describe how to determine the number of MAC address or IP addresses belong to a single asset ID in the system and help administrators spot potential issues.

     

    I didn't cover all of your questions, but did want you to know that there is more info pending. That information should cover a majority of your questions.

     

    If you want to read some more on assets, there is also these two articles which are already available.

     

    More...

    Thanks. I'll keep an eye out of the upcoming articles.

Feed for this topic