My IBM Log in Subscribe

Search LogDNA Records from the Command Line

9 August 2019

2 min read

Author

Henrik Loeser

Technical Offering Manager / Developer Advocate

Search audit events in LogDNA using Python

Earlier this year, IBM and LogDNA announced an integrated offering—Activity Tracker with LogDNA. It allows you to manage and search activity events in LogDNA instances on IBM Cloud. There are IAM, account management, and all kinds of service instance events that can be tracked. Viewing the events is typically done in the LogDNA UI. I, however, want to perform searches on the command line and integrate it with IBM Cloud Functions. 

In this article, I discuss the small tool that I wrote to search the activity logs and export them.

 

LogDNA interfaces

After provisioning the LogDNA instance as part of the IBM Cloud observability portal, you can launch its UI. There, it is possible to define views, perform searches, and manage instance settings. An important setting is the configuration of log archiving (i.e., the automatic backup of activity data to cloud object storage—compliance and more).

LogDNA allows you to ingest (link resides outside ibm.com) and export (link resides outside ibm.com) events through APIs. The required API keys can be obtained via UI in the settings for organization.

Search and export via API

The export API requires to and from parameters to limit the time range and supports event filters (e.g., by hosts, event level, and, most importantly, by query string). Authentication is performed by API service key.

With that information, I had everything I needed to write a small Python script—logdna-search—that serves as a foundation for a more capable command line tool or package for some serverless activity analytics. The script requires a configuration file with the instance region and the service key. The number of hours for the time range and a query string are optional. The following searches for log events from the past 24 hours, where the initiator of that security-related event has a name starting with “hloeser“:

searchLogDNA.py logConfigEU.json 24 'initiator.name:hloeser'

The above returns events as JSON lines (link resides outside ibm.com); each event is a JSON record on its own.

More fun with jq

If you are familiar with jq (link resides outside ibm.com), a command line JSON processor, you appreciate its filtering and formatting capabilities. Because of the nature of the activity events and LogDNA logs, the JSON-based events are embedded into JSON records with additional metadata, making it harder to read. With the help of jq, it is easy to filter down:

searchLogDNA.py logConfigEU.json 24 'initiator.name:hloeser' | jq -r '._line'

To extract just the initiator part of the events, apply another filter:

searchLogDNA.py logConfigEU.json 24 'initiator.name:hloeser' | jq -r '._line' | jq -r '.initiator'

Try it out

Want to test it on your own? The Python script is available on GitHub in the repo logdna-search. Information on Activity Tracker with LogDNA on IBM Cloud is in the documentation.

If you have feedback, suggestions, or questions about this post, please reach out to me on Twitter (@data_henrik) (link resides outside ibm.com) or LinkedIn (link resides outside ibm.com).

Related solutions

Related solutions

IBM Z Developer Tools

Discover how software developers can create, deploy and run apps using IBM Z developer tools.

Explore Cloud Pak for Applications
DevOps Solutions

Use DevOps software and tools to build, deploy, and manage cloud-native apps across multiple devices and environments.

Explore devops solutions
Enterprise Application Development Services

Cloud application development means building once, iterating rapidly, and deploying anywhere.

Application development services
Take the next step

Unlock the potential of DevOps to build, test and deploy secure cloud-native apps with continuous integration and delivery.

Explore DevOps solutions Learn about hybrid cloud for AI