Serverless log file analysis with web scale using IBM SQL Query

Share this post:

The applications we build and the systems we run generate a massive amount of logging data ranging from alerting, monitoring, availability and performance. Analyzing logs, or log analytics, is a widespread practice to capture insights from mobile data, IoT devices, servers, websites and other sources. Most companies need programmatic access to this massive amount of log data for compliance reasons, to provide audit evidence, to execute historical trend analyses, as well as get real-time insights from data streams such as click-stream data, IoT messages and raw analytics events.

IBM Cloud SQL Query is an interactive query service that can be used to directly analyze logs stored in IBM Cloud Object Storage. With SQL Query, you can build and run data pipelines and analyze your log messages seamlessly, taking full advantage of cloud elasticity. SQL Query uses standard SQL with Apache Spark SQL and is a serverless solution, so you don’t have to create schemas or do any extra setup, just simply create a storage bucket, add your log files, click the one-button set up for the SQL Query service and start analyzing your logs instantly. If your data is stored in Cloud Object Storage as CSV, JSON, or Apache Parquet, you don’t have to worry about converting your data to a single format before applying log analytics because SQL Query can read and analyze this data using only a standard SQL JOINs clause.

To help you understand how to use SQL Query and Cloud Object Storage to upload data and make query log files, we invite you to read “How to Query and Analyze Call Logs with IBM Cloud SQL Query.”

You’ll learn how to:

  1. Set up Cloud Object storage and SQL Query,
  2. Upload call logs to cloud Object Storage using IBM Aspera high-speed data transfer, and
  3. Run SQL queries over multiple log files to gain insights into calls spanning a couple of years.

If you’re interested in building out a fully functioning logging pipeline, review “Big Data Log Analysis with Streaming Analytics.”

You’ll learn how to:

  • Generate Application log events and send them to Message Hub
  • Intercepting and analyzing the event with Streaming Analytics
  • Appending the logs to a CSV file located in Cloud Object Storage
  • Issuing a SQL statement using SQL Query
  • Executing that statement on log files in Cloud Object Storage and storing the result set for further analysis

Interested in jumping right into IBM SQL Query? This service is absolutely free during our Public beta.

Or, you can dive deeper into our Cloud Object Storage offering.

Developer Advocate - ICD, SQL Query

More How-tos stories
September 24, 2018

Migrating Java Microservices from Spring Boot to MicroProfile

Spring or MicroProfile, which one should you pick for more interaction between backend microservices? Which one is more easily adaptable to different user interaction patterns?  And most of all, which one allows you to write less code? This series explores the migrating of our reference Java microservices app from Spring Boot to MicroProfile.

Continue reading

September 20, 2018

Deploying to IBM Cloud Private with IBM Cloud Developer Tools CLI

IBM Cloud Developer Tools CLI version 2.1.4 adds deployment support for IBM Cloud Private This version of IBM Cloud Private uses a more secure Helm for Kubernetes deployments and simplifies the cluster configuration for the client compared to prior IBM Cloud Private releases.

Continue reading

September 19, 2018

Tutorial: Apply End-to-End Security to Cloud Applications

A new tutorial will show you how to use IBM Cloud services to secure your cloud application. Capture and review security-related events, encrypt storage, integrate authentication, and more.

Continue reading