Serverless log file analysis with web scale using IBM SQL Query

5 min read

Serverless log file analysis with web scale using IBM SQL Query

The applications we build and the systems we run generate a massive amount of logging data ranging from alerting, monitoring, availability and performance. Analyzing logs, or log analytics, is a widespread practice to capture insights from mobile data, IoT devices, servers, websites and other sources. Most companies need programmatic access to this massive amount of log data for compliance reasons, to provide audit evidence, to execute historical trend analyses, as well as get real-time insights from data streams such as click-stream data, IoT messages and raw analytics events.

IBM Cloud SQL Query is an interactive query service that can be used to directly analyze logs stored in IBM Cloud Object Storage. With SQL Query, you can build and run data pipelines and analyze your log messages seamlessly, taking full advantage of cloud elasticity. SQL Query uses standard SQL with Apache Spark SQL and is a serverless solution, so you don’t have to create schemas or do any extra setup, just simply create a storage bucket, add your log files, click the one-button set up for the SQL Query service and start analyzing your logs instantly. If your data is stored in Cloud Object Storage as CSV, JSON, or Apache Parquet, you don’t have to worry about converting your data to a single format before applying log analytics because SQL Query can read and analyze this data using only a standard SQL JOINs clause.

To help you understand how to use SQL Query and Cloud Object Storage to upload data and make query log files, we invite you to read “How to Query and Analyze Call Logs with IBM Cloud SQL Query.”

You’ll learn how to:

  1. Set up Cloud Object storage and SQL Query,

  2. Upload call logs to cloud Object Storage using IBM Aspera high-speed data transfer, and

  3. Run SQL queries over multiple log files to gain insights into calls spanning a couple of years.

If you’re interested in building out a fully functioning logging pipeline, review “Big Data Log Analysis with Streaming Analytics.”

You’ll learn how to:

  • Generate Application log events and send them to Message Hub

  • Intercepting and analyzing the event with Streaming Analytics

  • Appending the logs to a CSV file located in Cloud Object Storage

  • Issuing a SQL statement using SQL Query

  • Executing that statement on log files in Cloud Object Storage and storing the result set for further analysis

Interested in jumping right into IBM SQL Query? This service is absolutely free during our Public beta.

Start your cost-free trial today!

Or, you can dive deeper into our Cloud Object Storage offering.

Be the first to hear about news, product updates, and innovation from IBM Cloud