Topic
  • 4 replies
  • Latest Post - ‏2016-03-18T09:30:51Z by aa.stepanov
J_e_K
J_e_K
3 Posts

Pinned topic Rule response Add/Delete Reference Data

‏2015-11-18T12:50:30Z | refrancedata rule

Hy,

Stuck a bit with References. Does anybody know a way to dynamically add and remove data to and from a reference set of maps?

Briefly, situation looks like that: we have two parameters which stats we should monitor. For such case, a "ReferenceMapOfSets" suits me fine. We can always use an action in Rule Response "Add to Reference Data" to add new data to the set. What about removing expired data? Is the any way to dynamically do it with a rule? Cleaning data while time pass is impossible for us.

Thank you.

  • J_e_K
    J_e_K
    3 Posts
    ACCEPTED ANSWER

    Re: Rule response Add/Delete Reference Data

    ‏2015-12-11T07:51:33Z  

    Hi,

    It's sad to admit that, but except command line scripting nothing suit me.

    Let me describe my solution for Adding and Deleting data from Reference Sets basing on events but not the Time stamps.

    First of all I created two similar Reference sets, let's say RefActive and RefDel. First one for collecting and saving ACTIVE data, second one for data needed to be deleted. Both of tam must be of the same structure and should save data independent of the time passed.

    Next step is to create rules that will add data to the Reference Sets basing on our needs.

    And the third step is to run .sh scrip that basing on ReferenceDataUtil.sh will synchronize data once a period of time.

    My scrip looks like that:

    /opt/qradar/bin/ReferenceDataUtil.sh list RefDel displayContents | tail -n +6 | sed 's/Key1=\(.*\)  Data=\(.*\)/\2 \1/' > /tmp/TEMP.txt

    # Extracting and parsing data from our Reference Set to proper structure for deleting. In my case I had Reference MapOfSets, so two field are used (Key, Value). Case to case structure may differ a bit.

    filename="/tmp/TEMP.txt"

    while read -r line

     do

            name=$line

            /opt/qradar/bin/ReferenceDataUtil.sh delete RefActive $name

    done < "$filename"

    # Deleting line by line data from our main Reference Set

    /opt/qradar/bin/ReferenceDataUtil.sh purge RefDel

    # Cleaning our support Reference Set.

     

    That's all. Can't say that's best way, but it's the only one that I could create that solved my problem.

    Thank you.

     

  • sree_ibm
    sree_ibm
    16 Posts

    Re: Rule response Add/Delete Reference Data

    ‏2015-11-27T17:33:48Z  

    Hi, J_e_K,

    You are on the right track.  By default the data is stored forever.

    To have and use any Data Collection dynamically, you will have to consider 2 aspects

     1. When does the expiry happen? ie from the initial data entry ( first _ seen) or is the expiry time applied from the last seen time.

     2. How long you want to store the data and if so,

    These is done by  defining  -timeoutType=FIRST_SEEN or -timeoutType=LAST_SEEN along with the timeToLive=' '

    The create command syntax is :
     create <name> <collectionType> <elementType> [-timeoutType=[FIRST_SEEN | LAST_SEEN]] [-timeToLive='']

    [-timeoutType=[FIRST_SEEN | LAST_SEEN]] specifies whether the timeToLive is from the time the element was inserted (FIRST_SEEN) or last seen (LAST_SEEN)
     [-timeToLive=''] specifies a PGInterval string which is the amount of time reference data collection elements remain in the collection.  For example, '5 minutes', '1 month'.

    Default is lives forever.

    For example:

     ./ReferenceDataUtil.sh create RF_TST MAP ALN -timeoutType=FIRST_SEEN -timeToLive='5 mins'
    Arg: create
    Arg: RF_TST
    Arg: MAP
    Arg: ALN
    Arg: -timeoutType=FIRST_SEEN
    Arg: -timeToLive=5 mins
    Successfully created Reference Data Collection.  ReferenceDataCacheMap  Id:22 Name:RF_TST CollectionType:MAP ElementType:ALN CreatedTime:2015-11-27 11:58:17 TimeoutType:FIRST_SEEN Timeout Interval:[0 years 0 mons 0 days 0 hours 5 mins 0.00 secs] CurrentCount:0 Key1Label:null ValueLabel:null
     

    In this case when You use the set the values of the collection will be deleted 5 mins from the time the value is seen, ie deleted 5 mins after. LAST _SEEN as the option would mean the last seen time would decide if the value is deleted.

    Note: If the value is seen again,  after the value is deleted, then it would get readded into the collection if you are using a Rule Response.

    You may also find this thread in the forum useful:

    https://www.ibm.com/developerworks/community/forums/html/topic?id=06614154-30e8-412f-89c2-d802546291b7&ps=25

    Regards,

    Sree


     

    For example:

  • J_e_K
    J_e_K
    3 Posts

    Re: Rule response Add/Delete Reference Data

    ‏2015-11-30T07:52:57Z  
    • sree_ibm
    • ‏2015-11-27T17:33:48Z

    Hi, J_e_K,

    You are on the right track.  By default the data is stored forever.

    To have and use any Data Collection dynamically, you will have to consider 2 aspects

     1. When does the expiry happen? ie from the initial data entry ( first _ seen) or is the expiry time applied from the last seen time.

     2. How long you want to store the data and if so,

    These is done by  defining  -timeoutType=FIRST_SEEN or -timeoutType=LAST_SEEN along with the timeToLive=' '

    The create command syntax is :
     create <name> <collectionType> <elementType> [-timeoutType=[FIRST_SEEN | LAST_SEEN]] [-timeToLive='']

    [-timeoutType=[FIRST_SEEN | LAST_SEEN]] specifies whether the timeToLive is from the time the element was inserted (FIRST_SEEN) or last seen (LAST_SEEN)
     [-timeToLive=''] specifies a PGInterval string which is the amount of time reference data collection elements remain in the collection.  For example, '5 minutes', '1 month'.

    Default is lives forever.

    For example:

     ./ReferenceDataUtil.sh create RF_TST MAP ALN -timeoutType=FIRST_SEEN -timeToLive='5 mins'
    Arg: create
    Arg: RF_TST
    Arg: MAP
    Arg: ALN
    Arg: -timeoutType=FIRST_SEEN
    Arg: -timeToLive=5 mins
    Successfully created Reference Data Collection.  ReferenceDataCacheMap  Id:22 Name:RF_TST CollectionType:MAP ElementType:ALN CreatedTime:2015-11-27 11:58:17 TimeoutType:FIRST_SEEN Timeout Interval:[0 years 0 mons 0 days 0 hours 5 mins 0.00 secs] CurrentCount:0 Key1Label:null ValueLabel:null
     

    In this case when You use the set the values of the collection will be deleted 5 mins from the time the value is seen, ie deleted 5 mins after. LAST _SEEN as the option would mean the last seen time would decide if the value is deleted.

    Note: If the value is seen again,  after the value is deleted, then it would get readded into the collection if you are using a Rule Response.

    You may also find this thread in the forum useful:

    https://www.ibm.com/developerworks/community/forums/html/topic?id=06614154-30e8-412f-89c2-d802546291b7&ps=25

    Regards,

    Sree


     

    For example:

    Thank you for advice, but as I wrote before, the entrance to ReferenceSet (RS) should be deleted basing on event but not on the time criteria.  To explain little better let me describe the situation.
    Let's say, we have an HR application, where we introduce all possible states of our users accounts. The user status is defined by "ACTIVE" and "LOCKED'. In the moment when user status changes, we receive an event that fully describes
    the status for the account.
    In case of "ACTIVE" status we introduce the respective username to RS.
    Until the event, "LOCKED" appears all logons in business applications are going to be verified with the RS.

    When the event "Locked" appears, we should remove the entrance from RS.

    As you can see in my case, we cannot use any time stamp, because the user can stay in the RS as 1 day so 10 years. Therefore, I'm interested in the way to ADD and DELETE data in ReferenceSet.

    Thank you.

  • J_e_K
    J_e_K
    3 Posts

    Re: Rule response Add/Delete Reference Data

    ‏2015-12-11T07:51:33Z  

    Hi,

    It's sad to admit that, but except command line scripting nothing suit me.

    Let me describe my solution for Adding and Deleting data from Reference Sets basing on events but not the Time stamps.

    First of all I created two similar Reference sets, let's say RefActive and RefDel. First one for collecting and saving ACTIVE data, second one for data needed to be deleted. Both of tam must be of the same structure and should save data independent of the time passed.

    Next step is to create rules that will add data to the Reference Sets basing on our needs.

    And the third step is to run .sh scrip that basing on ReferenceDataUtil.sh will synchronize data once a period of time.

    My scrip looks like that:

    /opt/qradar/bin/ReferenceDataUtil.sh list RefDel displayContents | tail -n +6 | sed 's/Key1=\(.*\)  Data=\(.*\)/\2 \1/' > /tmp/TEMP.txt

    # Extracting and parsing data from our Reference Set to proper structure for deleting. In my case I had Reference MapOfSets, so two field are used (Key, Value). Case to case structure may differ a bit.

    filename="/tmp/TEMP.txt"

    while read -r line

     do

            name=$line

            /opt/qradar/bin/ReferenceDataUtil.sh delete RefActive $name

    done < "$filename"

    # Deleting line by line data from our main Reference Set

    /opt/qradar/bin/ReferenceDataUtil.sh purge RefDel

    # Cleaning our support Reference Set.

     

    That's all. Can't say that's best way, but it's the only one that I could create that solved my problem.

    Thank you.

     

  • aa.stepanov
    aa.stepanov
    1 Post

    Re: Rule response Add/Delete Reference Data

    ‏2016-03-18T09:30:51Z  

    Hi, J_e_K,

    If you just need to check whether user account is active, you can create custom property UserStatus (for example) and write user account and UserStatus into Reference Table or Reference Map..

    How it works:

    Let's say, you have Key1 = 'John', Key2 = 'UserStatus', Data = 'ACTIVE' in your reftable. So you can change this entry on rule response to Key1 = 'John', Key2 = 'UserStatus', Data = 'LOCKED' when John has been fired, and then use it in your other rules.