Expiring TADDM Data

6 likes Updated 5/23/16, 1:26 PM by doormat18Tags:
Update:
- taddm_archive.jy updated on 12/18/2015. Major changes:
  • Only supported on v7.3, use older versions if needed
  • Threading added for performance

  • Ability to skip systems that failed discovery recently so avoid deletion of systems on remediation list

  • archive.ps1 added to support PowerShell on Windows
  • skip_zos argument added for customers running zDLA in delta mode
  • Added flag to print lastStoredTime
  • Added flag to use specific epoch instead of Age (in days)
  • Updated views for 7.3 to include new views for db calls
  • Updated to use upgraded Jython v2.5.3 in TADDM v7.3
- taddm_archive_722.jy updated on 1/3/2014. Major changes:
  • Using direct DB queries to build list of GUIDs for performance improvement
  • Added more parameters to allow for archive best practices
  • Tested on TADDM 7.2.2+ and 7.3.0.0
  • Added wrapper shell script and changed wrapper scripts to adhere to best practices
  • Minor changes on September 2014:
    • Performance improvement for querying dormant BindAddress components
Please ensure you download the correct version!

Overview

In most deployments, TADDM is considered to be a list of Active Systems. To maintain this integrity, it is necessary to implement a set of processes to ensure the data remains solid. Part of that process is to implement a discovery remediation process to resolve discovery problems and the other part is to remove dormant components.
Dormant Components in TADDM are machines that have not been discovered in a particular period of time, for example 90 days. Since TADDM collects and maintains all of the data it has discovered indefinitely, it is necessary to expire data in order to reduce the size and complexity of the database.
 
The script taddm_archive.jy (new) or taddm_archive_722.jy (old) can do this for you and will be discussed below – along with command line methods.


 
 
These scripts are minimally tested and provided as is as an example for how this can be done. It is highly recommended that it be tested in your environment prior to relying on it in production.

Best Practices

  1. Determine a time-frame. It should be a multiple of your discovery period. For example, if you discover your whole environment once a month, then expiring things after 90 days should mean that anything old was attempted at least 3 times and is most likely gone.
  2. Attempt a Rediscovery – Use the '-s' option of taddm_archive.jy to export the systems older than your period (90 days in this example) to attempt a rediscovery:
 

bash# cd $COLLATION_HOME/custom

bash# ./taddm_archive.jy -A 90 -C ComputerSystem -s OlderThan90

bash# ../bin/loadscope.jy -s “OldMachinesScope” load ./OlderThan90.txt

 

Now you should be able to go to your discovery server and run a discovery against this scope.
 
  1. If necessary, archive the Remaining Machines – If you are concerned about losing any data, archive the remaining old machines when you delete them.
  2. Delete the machines – Run taddm_archive.jy regularly (weekly) to handle your dormant machines. To archive and remove the dormant data, do the following

     
 

bash# cd $COLLATION_HOME/custom

bash# ./taddm_archive.jy -A 30 -C ComputerSystem -S -E export -D

Detailed Usage and Description

Finding Old Systems

Objects in the TADDM database maintain a 'LastModifiedTime' which indicates the last time the object was discovered or any components of the object were changed. This attribute can be used to determine what objects have not been modified in a certain amount of time.


api.sh


LastModifiedTime is in Milliseconds from January 1, 1970 (epoch). For comparison purposes, you can use

 
 

bash# now=$(date +%s) # The current time in seconds

bash# echo $now 1273777585

bash# 30_days_ago=$((now-30*24*3600)) # 30 days ago in seconds

bash# echo $30_days_ago 1271185585

All of the above numbers are in seconds versus Milliseconds. You need to multiply the above by 1000 for use in the following query

An example using the api.sh script to find all ComputerSystems that have not been modified in 30 days would be:
 
 

bash# $COLLATION_HOME/bin/sdk/api.sh -u user -p password find "select * from ComputerSystem where lastModifiedTime<'1271185585000'"

 

 
You could also use AppServer or some other ClassName in the above Query rather than ComputerSystem
 
taddm_archive.jy
 
To find out the objects in a class that are older than a specific age in days, you would do:


 
 

bash# cd $COLLATION_HOME/custom

bash# ./taddm_archive.jy -A 30 -C ComputerSystem

This will return a list of GUIDs, DisplayNames that have not been modified in 30 days.

Archiving Old Systems
 
It may be useful to save the systems that are old in XML format prior to deleting them. To do that, you would do the following:
 
api.sh
 
Using the lastModifiedTime from above with the api.sh script to find all ComputerSystems that have not been modified in 30 days would be:


 
 
bash# $COLLATION_HOME/bin/sdk/api.sh -u user -p password find --depth 3 "select * from ComputerSystem where lastModifiedTime<'1271185585000'" >> export.xml

You could also use AppServer or some other ClassName in the above Query rather than ComputerSystem

taddm_archive.jy
 
Using the taddm_archive.jy script, you have two archive options:
 
-S – This saves each GUID at Depth 3 in an XML file in the current directory

-E – Gives you the option to specify the directory where you want this saved.
 
For example:
 
 

bash# cd $COLLATION_HOME/custom

bash# ./taddm_archive.jy -A 30 -C ComputerSystem -S -E export

This will save all ComputerSystems not modified in 30 days as <GUID>.XML in ./export

Creating a Scope with Old Systems for Rediscovery
 
taddm_archive.jy
 
Using the taddm_archive.jy script, you can:
-s ScopeName – This attempts to get the IP for all objects to be deleted and creates a file named ScopeName.txt that can be loaded with loadscope.jy
-E – Gives you the option to specify the directory where you want this saved.
 
For example:
 
 

bash# cd $COLLATION_HOME/custom

bash# ./taddm_archive.jy -A 30 -C ComputerSystem -s Olderthan30 -E scope

This will save all ComputerSystems not modified in 30 days into a file Olderthan30.txt in ./scope that can be loaded as a scope for re-discovery using dist/bin/loadscope.jy

Note that if disableDNSLookups is set to true (default is false) in collation.properties on the TADDM server where you are executing the script, then the -s scope generation will not work. 

Deleting Old Systems
 
In order to delete the systems that are old you would do the following:
 
api.sh
 
Using the xml file that was saved in the export, you would need to parse all of the GUIDs out and then delete them one by one using the following command:

 

 
bash# $COLLATION_HOME/bin/sdk/api.sh -u user -p password delete GUID

It is beyond the scope of this document to describe how to parse the xml and retrieve the GUIDs

taddm_archive.jy
 
The taddm_archive.jy script can delete the old objects using the following syntax:

 

 

 
bash# cd $COLLATION_HOME/custom bash# ./taddm_archive.jy -A 30 -C ComputerSystem -D -S -E export



 

It is not necessary to Save (-S -E export) the data prior to deleting it, However it is recommended if you are NOT doing regular database backups.


 
This will save all ComputerSystems not modified in 30 days as <GUID>.XML in ./export and then also delete them from the system.
 
taddm_archive.jy Installation and Usage
 
Download and extract the zip file to $COLLATION_HOME/custom. You may need to create the directory 'custom'. $COLLATION_HOME is by default /opt/IBM/taddm/dist.

For non-Windows platforms, make the scripts executable by issuing the following command.

chmod a+rx archive.sh taddm_archive.jy

Note that if disableDNSLookups is set to true (default is false) in collation.properties on the TADDM server where you are executing the script, then the -s scope generation will not work.

For Windows, you need to execute the taddm_archive.bat wrapper script for jython to work properly.

 
 

bash$ ./taddm_archive.jy --help

 usage: script_name [options]

   or: script_name [options] -H displayName|-A days [ -S -D -C -E dir]



   Return Model Objects that have not been modifed in AGE Days.  Optionally

   Save and Delete these objects

    Options:

    -u userid           User required to login to TADDM Server

                        Defaults to 'administrator'

    -p password         Password for TADDM Server user

                        Defaults to 'collation'

    -h                  print this message



    Arguments:

    -A age             Use Age in DAYS as time to consider something old for

                       ClassName (defaults to ComputerSystem)

                       An age of '0' will return all objects in the class.

    -S                 Save Objects that are 'Old' in XML format

    -s scope_name      Create a scope named 'scope_name' of old machines

    -H                 displayName -- Ignore age and use this specific displayname

    -C                 Classname to examine -- Defaults to ComputerSystem -- a good

                       alternative would be AppServer

    -L num             Limit results to a maximum of num

    -D                 Delete the objects that are considered OLD

    -E dir             Export Directory (must exist) defaults to current directory


    -c                 Limit to Classname only and do not include the sub-classes --

                       For example, query ComputerSystem instances but not

                       WindowsComputerSystem instances

    --l2orphans        Use all L2Interface instances that are orphans irrespective of

                       age. -C option must be 'L2Interface'. -A option must be 0.

    --chk_sups         Check the superiors table to ensure that the instances are

                       not part of a naming rule

    --skip_zos         Skip all zOS components, recommended if running z/OS DLA in

                       delta mode

                       not part of a naming rule


    -q                 Display the SQL query used

    --print_laststored Display LastStored in addition to GUID and DisplayName

    --epoch=<time>     Use specific epoch formated time instead of -A age option


    --skip_failed=<weeks>      

                       Skip deletion of components if there is a failed session sensor

                       matching the contexIp in the last <weeks> week(s).

    -t num             Number of threads to use for deletion, ignored if -D is not used,

                       Defaults to single thread

 

 

 

 

Automating Data Expiration

It is an important best practice to run data expiration on a regular basis. The following section describes some tips and provides some tools to get you started in automating data expiration.

In the zip file you will find three tools to assist you in automating data expiration.

  • classes.txt - This file lists all of the CI classes to expire. The classes listed by default are best practice. You may need to add or remove classes to fit the requirements for your environment. Use # to add comments or comment out classes for specific runs.
  • archive.ps1/archive.sh - This script executes taddm_archive.jy for all of the CI classes in classes.txt according to best practices. It automatically creates sub-directories under %COLLATION_HOME%\custom for the classes and timestamp of the execution.

    The default AGE variable is 182 and the default LIMIT variable is 1000000.

    At the end of execution, e-mail notification is present (Notification.ps1 is called for Windows and mailx for Unix) to send an e-mail notification that the job finished. The script may need tweaked to get e-mail notification to work.

    This script should not be considered final and will most likely require some tweaking for your requirements and environment. This is a good starting point for automating data expiration.
  • archive.properties - Edit this file to set properties for archive.sh/ps1. Make sure that you set archive.skipzos=true if you are running zDLA in delta mode.
  • Notification.ps1 - WINDOWS ONLY. This is a powershell script to send e-mail notifications via SMTP. Edit the script to set the default $from variable and $smtpServer variable for your environment.

For notification to work on Windows, you may need to install PowerShell if you are running on Windows 2003. Also, make sure that you run 'Set-ExecutionPolicy Unrestricted' for both the 32-bit and 64-bit powershell at the command prompt. This only needs done once.

Now simply use the Windows task scheduler or cron to schedule execution of archive.ps1 or archive.sh. It should be run as often as a full discovery is run.

When scheduling via the Windows task scheduler, you need to run the powershell command and the file argument.

 

Known Issues

  • The log tracing for the script might not work. Check tomcat.log in some TADDM distributions.
  • Processing BindAddress may hang the process or take too long to delete. Refer to the Dormant Data cleanup in TADDM blog entry section titled "TABLES TO USE DATABASE SQL TO CLEANUP" for a discussion of how to handle the initial clean up of the BindAddress table.
  • zDLA deltas: The archive process assumes that all CI components are being discovered on a regular basis. If you are running the zDLA in delta mode, then it is not guaranteed that all CI components are being updated regularly and some components will appear dormant when they actually are not. For this reason, ensure that you use the skip_zos feature to skip all zOS components.
  • It is possible that the -S option, if used, can cause OutOfMemoryException to occur. If this is the case then remove that option and ensure that you are doing proper database backups.
  • Manually created components: If you have manually created components in TADDM that do not get updated regularly, those will eventually get cleaned up by this script. Keep that in mind.