Serverless log file analysis with web scale using IBM SQL Query

Share this post:

The applications we build and the systems we run generate a massive amount of logging data ranging from alerting, monitoring, availability and performance. Analyzing logs, or log analytics, is a widespread practice to capture insights from mobile data, IoT devices, servers, websites and other sources. Most companies need programmatic access to this massive amount of log data for compliance reasons, to provide audit evidence, to execute historical trend analyses, as well as get real-time insights from data streams such as click-stream data, IoT messages and raw analytics events.

IBM Cloud SQL Query is an interactive query service that can be used to directly analyze logs stored in IBM Cloud Object Storage. With SQL Query, you can build and run data pipelines and analyze your log messages seamlessly, taking full advantage of cloud elasticity. SQL Query uses standard SQL with Apache Spark SQL and is a serverless solution, so you don’t have to create schemas or do any extra setup, just simply create a storage bucket, add your log files, click the one-button set up for the SQL Query service and start analyzing your logs instantly. If your data is stored in Cloud Object Storage as CSV, JSON, or Apache Parquet, you don’t have to worry about converting your data to a single format before applying log analytics because SQL Query can read and analyze this data using only a standard SQL JOINs clause.

To help you understand how to use SQL Query and Cloud Object Storage to upload data and make query log files, we invite you to read “How to Query and Analyze Call Logs with IBM Cloud SQL Query.”

You’ll learn how to:

  1. Set up Cloud Object storage and SQL Query,
  2. Upload call logs to cloud Object Storage using IBM Aspera high-speed data transfer, and
  3. Run SQL queries over multiple log files to gain insights into calls spanning a couple of years.

If you’re interested in building out a fully functioning logging pipeline, review “Big Data Log Analysis with Streaming Analytics.”

You’ll learn how to:

  • Generate Application log events and send them to Message Hub
  • Intercepting and analyzing the event with Streaming Analytics
  • Appending the logs to a CSV file located in Cloud Object Storage
  • Issuing a SQL statement using SQL Query
  • Executing that statement on log files in Cloud Object Storage and storing the result set for further analysis

Interested in jumping right into IBM SQL Query? This service is absolutely free during our Public beta.

Or, you can dive deeper into our Cloud Object Storage offering.

Software Engineer and Evangelist - IBM Cloud Databases

More How-tos stories
May 3, 2019

Kubernetes Tutorials: 5 Ways to Get You Building Fast

Ready to start working with Kubernetes? Want to build your Kubernetes skills? The five tutorials in this post will teach you everything you need to know about how to manage your containerized apps with Kubernetes.

Continue reading

May 2, 2019

Video – What is a DDoS Attack?

Ryan Sumner, Chief Networking Architect, gives an overview of DDoS attacks and just how the attacker's botnet can affect the target application and its users.

Continue reading

April 30, 2019

Introducing IBM Analytics Engine v1.2 and Announcing the Deprecation of IBM Analytics Engine v1.0

We are excited to inform you about the new version of IBM Analytics Engine v1.2 that will be available starting May 15, 2019. Along with this release, Analytics Engine v1.0 will be retired.

Continue reading