Figure 1 shows the sample scenario architecture you'll set up in this tutorial:
Figure 1. The tutorial's architecture
Assume there are multiple client applications generating log files and saving them in the solidDB cache databases. IBM solidDB is a fast in-memory cache database that is optimized for high performance. Since it keeps all data in memory and does not write any data to disk, except transaction logs for persistence, it's fast but also reliable.
A Java tool will then pick the log files up and transfer them into WebSphere MQ. WMB will analyze and transfer the files to the DB2 back-end database.
To simulate application logging clients, this tutorial uses a Java tool that generates and loads XML files into the cache databases. The tool — Java Load Generator Tool — is included with this tutorial (see Downloads). By running the tool without command line options, it prints out usage information. This tutorial uses this tool to load messages into solidDB.
To ship log files into the back-end database, this tutorial uses IBM WebSphere Message Broker. It offers transactional and persistent message queues and routing mechanisms to transfer messages from one location to another. Additionally, you can use WebSphere Message Broker to analyze and transform the messages. Figure 2 shows the message flows introduced in this tutorial:
Figure 2. WebSphere message flows
Messages are read from two queues, holding different message formats. Queue 1 (Q1) will hold messages consisting of one element named Record. Every message will either be routed to another queue (Q3) or to the back-end database.
Queue 2 (Q2) will hold File messages that consist of one or more Record elements. Each File message will be split into single Record messages and a summary message per Record. Record messages will then be routed to the back-end database, while Summary messages will be stored in another queue (Q4).
WMB can interact with IBM DB2 to store messages transactional. That way, no message can be lost due to (network) failures. IBM DB2 for Linux, UNIX, and Windows will finally store all the messages. It uses pureXML to natively store XML files in a dedicated data structure. That way, queries can be performed efficiently while preserving the original structure. DB2 also supports compression of XML files, which will save a lot of storage space.