• No replies
3 Posts

Pinned topic Log Analysis Benchmark Application Accelerator added to Streams Exchange

‏2012-02-24T22:19:48Z |
 Anyone want to analyze log messages in real time?   With massive scalability?  Millions of messages per second?
This Application Accelerator has driver code to simulate flooding Streams with log messages with varying formats. It also has a set of 9 complex event processing functional and performance tests implemented.  The tests are run 4 times - once using only 4 cores of an octicore server, once using all eight cores, a third time using 2 servers and a fourth time using 4 servers.  This shows how simple it is to scale up - simply redeploy the application!  Performance results are also shared for the tests.  Your performance will likely exceed those documented, since the cluster only used 1Gb Etnernet, and the CPU utilization was low.  For one test, we did re-run it using 10Gb network, and nearly tripled performance - 7 millions messages per second on the 4 node cluster.
This opens up great new possibilities for people, since log analysis can not only be done like this example using native SPL code, but you could extend the analysis with things like SPSS or PMML mining and analytic modules.  Or advanced text analytics to quickly be alerted to anomalous behavior.
Thanks to the team that created these applications!  I hope you enjoy working with them and extending them.
You'll find them as part of the collection called "Streams Applications (SPL)".

Updated on 2012-03-07T21:34:43Z at 2012-03-07T21:34:43Z by Roger_Rea