Big Data Intelligence with data marts

Guardium® Big Data Intelligence uses data marts to export data to a central storage location.

To deploy and use Guardium Big Data Intelligence:

  1. Define the central storage. Use the GuardAPI create_datasource or the GUI (Creating a datasource definition). Use Big Data Intelligence as the application type.
  2. Define your data export profile by using the GuardAPI Big Data Intelligence function enable_big_data_interface. The profile defines the datasource, the target host, the units that export data, the data mart extraction profile (made up of one or more data marts), and the export schedule. There are four predefined, non-modifiable profiles. To create your own data export profile, copy and modify the predefined profiles. You must use the data marts specified in the predefined profiles in order to access your data in the Big Data Intelligence domains of the Query-Report Builder.

    The profile is applied to the managed units you specify in the command enable_big_data_interface; all the managed units of the central manager, and to the central manager itself, a managed unit group, or specific managed units.

  3. After you run enable_big_data_interface you'll see Big Data Intelligence domains in the Query-Report Builder. When you define a query, Guardium needs to connect to the GBDI to validate and save the query. This can take up to a minute. Watch the lower left corner of your GUI; you'll see text waiting for server until it connects. If Guardium cannot connect to the GBDI, it responds: Unable to establish connection..., and the query is not saved.
  4. Use the standard Guardium predefined reports and Investigation Dashboards (Quick Search) to analyze your data. You can also create reports using the Big Data Intelligence domains.

You can enable additional managed units for data extraction at a later date with the command grdapi local_enable_big_data_interface profile_name="<profile name>". This is useful if a collector was offline or not in the MU group when the interface was enabled; or, for advanced users, if you want to include data from a collector that requires a different profile, or is in another group. For example, two managed units of a central manager run only VA, and the other managed units are tracking other data. You would create a second profile that is a subset of the main profile, and run it only on the specified units. However, the local profile does not have a target for the data; add it using the command: datamart_update_copy_file_info.

Data handling guidelines

  • Data retention on the collector can be reduced to 1 day since the Big Data Intelligence server saves data over a long time period.
  • Data backup can be handled on the Big Data Intelligence server.
  • Configuration backup should be handled on the Guardium system.
  • Archives should be handled according to your regulation requirements. The Big Data Intelligence server keeps data longer than collectors and aggregators, and can be used for archiving.

The Guardium capabilities to read Big Data Intelligence directly using the Guardium Query-Report Builder, or enterprise search, require Guardium Big Data Intelligence Version 3.3. If you already have a data mart extraction to an earlier GBDI version, disable the extractions that are running, and then re-enable by using the following API: enable_big_data_interface

Summary of data marts in each profile

Basic summary
Export:Access Log, Export:Session Log, Export:Session Log Ended, Export:Exception Log, Export:Full SQL, Export:Outliers List - enhanced, Export:Outliers Summary by hour - enhanced, Export:Group Members, Export:Export Extraction Log, Export:Policy Violations, Export:Buff Usage Monitor
Comprehensive summary
Export:Access Log, Export:Session Log, Export:Session Log Ended, Export:Exception Log, Export:Full SQL, Export:Outliers List - enhanced, Export:Outliers Summary by hour - enhanced, Export:Group Members, Export:Export Extraction Log, Export:Policy Violations, Export:Buff Usage Monitor, Export:VA Results, Export:STAP Status, Export:Discovered Instances, Export:Databases Discovered, Export:Classifier Results, Export:Installed Patches, Export:System Info
Basic Details
Export:Access Log - Detailed, Export:Session Log, Export:Session Log Ended, Export:Exception Log, Export:Full SQL, Export:Outliers List - enhanced, Export:Outliers Summary by hour - enhanced, Export:Group Members, Export:Export Extraction Log,Export:Policy Violations - Detailed, Export:Buff Usage Monitor
Comprehensive details
Export:Access Log - Detailed, Export:Session Log, Export:Session Log Ended, Export:Exception Log, Export:Full SQL, Export:Outliers List - enhanced, Export:Outliers Summary by hour - enhanced, Export:Group Members, Export:Export Extraction Log, Export:Policy Violations - Detailed, Export:Buff Usage Monitor, Export:VA Results, Export:STAP Status, Export:Discovered Instances, Export:Databases Discovered, Export:Classifier Results, Export:Installed Patches, Export:System Info