Use InfoSphere Guardium Universal Feed to create a customized data activity monitoring solution, Part 1: Create a feed for a database source

New databases and new applications are continually being created and adopted to meet specific organizational needs. The requirement for data protection and auditing capabilities is required by mandate and is more critical than ever. The InfoSphere® Guardium® data protection solution is extensible to enable the integration of a wide variety of new databases and sources into its platform, thereby providing a consistent enterprise-wide monitoring solution. This article provides an example of how to use the Universal Feed feature in InfoSphere Guardium to integrate logs and security events from a database source into an external system, ensuring separation of duties.

Share:

Joe DiPietro (Joe_DiPietro@us.ibm.com), IBM InfoSphere Data Governance Center of Excellence Leader, IBM

Author photo of Joe DiPietroJoe DiPietro is IBM InfoSphere Data Governance Center of Excellence Leader. Joe has over 25+ years experience in security and network design and implementation. Prior to IBM and Guardium, he worked at security pioneer Check Point Software for 8+ years. Previously, DiPietro was corporate systems engineer for SynOptics Communications and a member of the company's World Wide Technical Counsel (WWTC). Joe holds a Masters degree in Computer Science, a Master of Arts, and a Bachelors degree in Mechanical Engineering.



25 October 2012

Also available in Chinese Portuguese

Introduction

InfoSphere Guardium provides a comprehensive data activity monitoring and protection solution, and includes support for a wide range of databases, file shares, and other systems such as Hadoop and Microsoft Sharepoint. In most cases, the solution relies on lightweight software probes (called S-TAPs) to monitor transactions, including those of privileged users. The monitored activity is sent to the InfoSphere Guardium appliance and stored in its internal database. The information can be used for audit reporting, real time alerts, and much more.

The key for security and event logs is to build these into the database or application transaction and store them externally to these systems to ensure separation of duties. With InfoSphere Guardium, event data is sent to a secure appliance, known as a collector, and stored in an internal database there for reporting and alerting. With the Universal Feed capability included with InfoSphere Guardium, you can integrate related audit data (perhaps for databases or other activity not supported by Guardium) into your current Data Activity Monitoring (DAM) environment by sending it to the collector and storing it in the internal tables there.

The Universal Feed has the following options for supporting different types of activity monitoring.

  • The first one, described in this article, is targeted for activity that can easily integrate into the existing internal InfoSphere Guardium tables. This would typically mean some kind of database source, since InfoSphere Guardium specializes in support for database activity. For more information about the entities and attributes within the InfoSphere Guardium system, see the product help appendix on that subject. The sample code included with this article provides a simulated database log which will be sent to the Guardium appliance.
  • The other option, described in Part 2 of this article series, enables you to integrate any arbitrary data source activity by enabling you to create your own table structure in the Guardium database to store the messages sent over by your feed. In that article, you'll learn how to create a feed for an SSH log.

You get the following benefits when using the Universal Feed to store the audit data off of the actual device that is monitored.

  • Audit and log information cannot be erased to cover nefarious breaches to the device.
  • Separation of duties can be maintained to ensure correct audit information is captured.
  • Privileged users don't have access to the audit logs if they decided to tamper or alter this information.

The Universal Feed agent will send information to the Guardium appliance, as shown in Figure 1.

Figure 1. Universal feed overview
UF sends messages to Guardium appliance collector - one way.

The red dot represents the Universal Feed agent that is running on the host and which will be used to send audit information to the Guardium appliance. This article and the included sample code should help you develop your own agent.

Table 1 highlights what the developer of a universal feed agent is responsible for providing.

Table 1. Universal feed responsibilities
Universal Feed developerInfoSphere Guardium
Identifying appropriate audit information. Providing APIs and sample code to help Universal Feed developers.
Use Guardium message protocol to package the information to send to the Guardium appliance.Receiving the information which enables this data to be used by the Guardium infrastructure (reporting, alerting, report distribution, and so on).

Agent requirements

The first aspect in coding the Universal Feed agent is to understand the requirements for the agent. These requirements are as follows.

  • To test and run your feed, you need an InfoSphere Guardium appliance at a minimum level of V9.0.
  • The agent must use the Guardium message format to send and receive information to be processed into the Guardium internal tables. The message format shown in Listing 1 is for a database-type Universal Feed agent as indicated by the "0" in the vendor field. A number of 10001 and above represents a non-database implementation and will be described in part 2 of this article series.
    Listing 1. Guardium message format for a database source
    struct sqlguard_msg {
    unsigned char msg_type;	    // Must be 'G'
        unsigned char pad;	    // Must be 0
        unsigned short data_len;// Length of  data in the "data" fields, network order
        uint32_t mark;		    //
        uint32_t timestamp;	    // Time in UNIX format (retval of time() syscall)
        uint32_t protocol_version;  // Must be 7
        uint32_t vendor;	    // Must be 0
        char identification[40];    // Must be all 0
        char data[MAX_DATA_LEN];    // Put the serialized protobuf message here
    };

    There are two directions flowing for these messages.
    • From the agent to the collector.
    • From the collector to the agent.
  • It must adhere to the Guardium communication protocol based on TCP, and it must send a Guardium handshake message that does the following.
    • This handshake message allows for the collector to register the name of the UFA.
    • Turns the agent green in the GUI so that you know it is operational.
  • It must send a Guardium Ping message every 30 to 60 seconds. Figure 2 shows the Guardium monitor report that has received a Ping message.
    Figure 2. S-TAP status monitor is green indicating all is operational
    The monitor includes things as stap host name and a status column. the whole row is green.
  • It must read everything the Guardium appliance sends to the agent, as shown in Figure 3.
    Figure 3. Agent is responsible for processing messages from the appliance
    simple graphic showing an arrow going from the appliance to the agent on the host.

    For example, after the handshake message, the Guardium appliance will send the current audit policy on the appliance to the UFA. The agent can then process this information to identify relevant details on how to configure the UFA behavior. Figure 4 shows a diagram of the message flow.

    Figure 4. Universal feed message processing overview
    shows flow of tcp socket open, to appliance, handshake to appliance, policy rules from appliance, session start to appliance, client request to appliance, etc. more details in text..

The following describes the message flow in more detail.

  1. The agent opens a TCP socket with the GA on port 16016, which is the default port that Guardium listens on.
  2. After port 16016 is opened, the agent sends the handshake message which tells the appliance what type of agent is communicating (using the msg_type field).
  3. More sophisticated agents can handle policy rules sent by the appliance. The agent can then adjust its behavior based on the policy rules.
  4. Once the agent has information that it wants to send to the appliance, it will send a session start message. This tells the appliance that there is new audit data to be stored, and it will timestamp this information.
  5. The agent can then send more audit information that will be associated with this particular session. These messages should be sent using the Client Request message format.
  6. Optionally, you can also send information that the server replied to based on the client request. This information is sent in the server reply message.
  7. Occasionally (every 60 seconds or less), the agent should send a ping message to let the appliance know that the agent is healthy. If the appliance does not receive a ping message, then it can send an alert through syslog, SMTP, and so on, to an operational staff to investigate the reason why the agent is not operational.
  8. After all of the information has been sent for this specific session, then you should close the session by sending a session end message. This will tell the Guardium appliance that all the audit data for this particular session has been received and to close the session record.

Listing 2 shows the messages sent by the agent, as described previously for the handshake, ping, session start, client request, and session end messages.

Listing 2. Sample message exchange between the Universal Feed agent and the Guardium collector
*** 1. SEND ***
type: HANDSHAKE
handshake {
    timestamp {
        unix_time: 1331583609
    }
    client_identifier: "NewUniversalFeedLogger"
    current_master: "NewUniversalFeedCollector"
    current_master_ip: 4027124234
}
*** 2. SEND ***
type: PING
ping {
    timestamp {
        unix_time: 1331583609
    }
    client_identifier: "NewUniversalFeedLogger"
    current_master: "NewUniversalFeedCollector"
    current_master_ip: 4027124234
}
                
*** 3. SEND ***
type: SESSION_START
session_start {
    session_id: 56
    session_locator {
        client_ip: 4027124234
        client_port: 1053
        server_ip: 4161341962
        server_port: 1030
    }
    timestamp {
        unix_time: 1331583631
    }
    accessor {
        db_user: "JoeD"
        server_type: "DB2"
        server_os: "ServerOperatingSystem"
        client_os: "ClientOSValue"
        client_hostname: "ClientHostName-Joe\'s Laptop"
        server_hostname: "ServerDBHostName"
        comm_protocol: "CommProtocol"
        db_protocol: "DB Protocol"
        db_protocol_version: "DBProtocolVersion"
        os_user: "OSUSer-JoeDiPietro"
        source_program: "SourceProgramNewApplication"
        client_mac: "clientMac"
        server_description: "ServerDescription"
        service_name: "ServiceName"
        language: DB2
        type: CONSTRUCT
    }
    process_id: "ProcessID String"
    terminal_id: "Terminal ID String"
    db_name: "DatabaseName"
    app_user_name: "AppUserName"
}
                
***4. SEND ***
type: CLIENT_REQUEST
client_request {
    session_id: 1
    data {
        language: DB2
        type: CONSTRUCT
        construct {
         sentences {
          verb: "Select"
           objects {
            name: "PatientTable"
          }
        }
        full_sql: "SELECT * FROM PatientTable"
    }
    timestamp {
      unix_time: 1331583634
    }
    session_locator {
       client_ip: 16777343
       client_port: 1030
       server_ip: 16843009
       server_port: 1030
    }
  }
}

*** 5. SEND ***
type: SESSION_END
session_end {
    session_id: 48013
    timestamp {
        unix_time: 1331583634
    }
    session_locator {
      client_port: 1053
      server_ip: 16843009
      server_port: 1030
  }
}

Components of the universal feed sample program

This sample code included with this article can easily be used to build security and audit events into custom applications or databases. This will help enhance the security of these applications because these events can be stored off of the application/database servers where privilege users and potential hackers cannot access these logs.

Sample audit file

In the sample application, the audit file shown in Listing 3 is sent to the Guardium appliance.

Listing 3. Sample audit file (sampleAuditLog.txt)
03/06/2012 16:24|48009|10.10.9.240|10.10.9.248|DB2|IBMUSER|SELECT * FROM CREDITCARD1
03/06/2012 16:26|48009|10.10.9.240|10.10.9.248|DB2|IBMUSER|SELECT * FROM CREDITCARD2
03/06/2012 16:27|48009|10.10.9.240|10.10.9.248|DB2|IBMUSER|SELECT * FROM CREDITCARD3
03/06/2012 16:28|48009|10.10.9.240|10.10.9.248|DB2|IBMUSER|SELECT * FROM CREDITCARD4
03/06/2012 16:29|48009|10.10.9.240|10.10.9.248|DB2|IBMUSER|SELECT * FROM CREDITCARD5

The records in the audit log are formatted as follows:
Timestamp | Session ID|Client IP|Server IP|Database type|Database user|SQL statement text

Application code

The application code can be found in the following files:

  • com/guardium/proto/datasource/test/Type1/FilePollingSample1.java
  • com/guardium/proto/datasource/test/Type1/TestUniversalFeed.java

The com/guardium/proto/datasource/test/Type1/FilePollingSample1.java class does the following.

  1. Reads a flat file (sampleAuditLog.txt).
  2. Parses these events.
  3. Formats the information into specific Guardium messages.
  4. Sends these messages over a TCP socket to the Guardium appliance.

The Guardium appliance will store this information into Guardium internal tables for reporting.

The com/guardium/proto/datasource/test/Type1/TestUniversalFeed.java is a simple program that sends the Guardium message types to the appliance.

Infrastructure and utility code

Utility and infrastructure code can be found in the following classes:

  • com/guardium/proto/datasource/DatasourceMessageUtil.java
  • com/guardium/proto/datasource/Socket.java
  • com/guardium/proto/datasource/Wrapper.java

com/guardium/proto/datasource/DatasourceMessageUtil.java is used to assist in building Guardium Messages to send to the appliance from the Universal Feed agent. You can use your other Java classes to call the methods in this class to easily build your Guardium messages.

com/guardium/proto/datasource/Socket.java is used to open a TCP Socket to the Guardium Appliance from the Universal Feed Agent

com/guardium/proto/datasource/Wrapper.java is used to wrap the data into the proper message format.

Batch files

The following files are also included in the attached download.

  • compileJava.bat is an example of how to compile the code in a Windows environment.
  • runFilePollingSample1.bat is a sample of how to start the program.

Running the sample program

To run the sample program, enter the command shown in Listing 4 (or see runFilePollingSample1.bat).

Listing 4. Sample program
java -jar FilePollingSample1.jar -host 10.70.147.72 -infile sampleAuditLog.txt -outfile
eventFileOut.txt -sendFileOnly true -loopAmount 1

Replace the host IP with the IP address of your InfoSphere Guardium appliance. The program reads the SampleAudit.txt file and sends the information to the Guardium appliance. Figure 5 is a sample report that shows the results on the Guardium appliance.

Figure 5. Report shows that messages successfully sent to Guardium
The report shows the contents of the audit log in a Guardium report.

Conclusion

New databases and new applications are continually being created and adopted to meet specific organizational needs. However, the requirement for data protection and auditing capabilities is consistent across the board by mandate. The InfoSphere Guardium data protection solution is extensible to enable the integration of a wide variety of new databases and sources into its platform, thereby providing a consistent, enterprise-wide solution. The sample code attached to this article is also well-documented and should help you get on your way to create a Universal Feed agent for a database server source.


Download

DescriptionNameSize
Sample program and utility files for this articleUnivFeedSamplePart1.zip172KB

Resources

Learn

Get products and technologies

  • Build your next development project with IBM trial software, available for download directly from developerWorks.
  • Evaluate IBM products in the way that suits you best: Download a product trial, try a product online, use a product in a cloud environment, or spend a few hours in the SOA Sandbox learning how to implement Service Oriented Architecture efficiently.

Discuss

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into Information management on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Information Management, Security
ArticleID=841537
ArticleTitle=Use InfoSphere Guardium Universal Feed to create a customized data activity monitoring solution, Part 1: Create a feed for a database source
publish-date=10252012