How-tos

Serverless log file analysis with web scale using IBM SQL Query

Share this post:

The applications we build and the systems we run generate a massive amount of logging data ranging from alerting, monitoring, availability and performance. Analyzing logs, or log analytics, is a widespread practice to capture insights from mobile data, IoT devices, servers, websites and other sources. Most companies need programmatic access to this massive amount of log data for compliance reasons, to provide audit evidence, to execute historical trend analyses, as well as get real-time insights from data streams such as click-stream data, IoT messages and raw analytics events.

IBM Cloud SQL Query is an interactive query service that can be used to directly analyze logs stored in IBM Cloud Object Storage. With SQL Query, you can build and run data pipelines and analyze your log messages seamlessly, taking full advantage of cloud elasticity. SQL Query uses standard SQL with Apache Spark SQL and is a serverless solution, so you don’t have to create schemas or do any extra setup, just simply create a storage bucket, add your log files, click the one-button set up for the SQL Query service and start analyzing your logs instantly. If your data is stored in Cloud Object Storage as CSV, JSON, or Apache Parquet, you don’t have to worry about converting your data to a single format before applying log analytics because SQL Query can read and analyze this data using only a standard SQL JOINs clause.

To help you understand how to use SQL Query and Cloud Object Storage to upload data and make query log files, we invite you to read “How to Query and Analyze Call Logs with IBM Cloud SQL Query.”

You’ll learn how to:

  1. Set up Cloud Object storage and SQL Query,
  2. Upload call logs to cloud Object Storage using IBM Aspera high-speed data transfer, and
  3. Run SQL queries over multiple log files to gain insights into calls spanning a couple of years.

If you’re interested in building out a fully functioning logging pipeline, review “Big Data Log Analysis with Streaming Analytics.”

You’ll learn how to:

  • Generate Application log events and send them to Message Hub
  • Intercepting and analyzing the event with Streaming Analytics
  • Appending the logs to a CSV file located in Cloud Object Storage
  • Issuing a SQL statement using SQL Query
  • Executing that statement on log files in Cloud Object Storage and storing the result set for further analysis

Interested in jumping right into IBM SQL Query? This service is absolutely free during our Public beta.

Or, you can dive deeper into our Cloud Object Storage offering.

Developer Advocate - IBM Cloud

More How-tos stories
December 13, 2018

Java Microservices with MicroProfile – API Documentation

To benefit from the reuse and consistency microservice APIs offer, other developers will need guidance to use your APIs correctly. With annotations defined in the MicroProfile OpenAPI specification from Java EE, it's easy to generate clear documentation.

Continue reading

December 13, 2018

Tutorial: Deploying the Jenkins Helm Community Chart on IBM Cloud Kubernetes Service

The IBM Hybrid Cloud Team has authored a tutorial that will guide you through the steps required to set up and install a Jenkins server and deploy a sample Node and React application on IBM Cloud Kubernetes Service.

Continue reading

December 12, 2018

Deploying to IBM Cloud Private 3.1 with IBM Cloud Developer Tools CLI

IBM Cloud Developer Tools CLI version 2.1.12 adds deployment support for IBM Cloud Private 3.1.

Continue reading