March 17, 2020 By Charles Comiskey
Michael Shi
2 min read

Clients today are increasingly accessing archived data for ad hoc insights over extended periods of time, forensic investigation, and archive for regulatory and business compliance.

The challenge: Running cloud operations at volume may result in large archives of log and activity event data.

The solution: Archiving with hourly frequency.

We are excited to introduce an improved archiving feature for IBM Log Analysis with LogDNA and IBM Cloud Activity Tracker with LogDNA that breaks up archives into an hourly cadence. For most customers, the benefits will be seamless. If you have a custom workflow based on your archived logs, it may have to be updated to accommodate the change.

What’s changing

Starting April 6, 2020, we will slowly transition our customers to a new archive file format that breaks up archives into an hourly cadence instead of a daily cadence.

This feature will allow you to receive archived logs faster, and you’ll be able to unzip just a portion of your logs instead of an entire day’s worth. Additionally, the new folder format better supports Hive partitions.

As we transition you to the new format, you might observe a small overlap between the older daily format archives and the new archives. Expect to see hourly archives in your provider within 24 hours of the logs being received by us.

The new archive directory format for all providers will look like this:

year=YYYY/month=MM/day=DD/<accountID>.<YYYY>-<MM>-<DD>.<HH>00.json.gz (where HH is hours in 24 format)

The current archive directory format looks like this:

/<accountID>.<YYYY>-<MM>-<DD>.<clusterId>.json.gz

The current archive directory format for S3 looks like this:

YYYY/MM/<accountID>.<YYYY>-<MM>-<DD>.<clusterId>.json.gz

The content of the archive files will remain the same as before.

What you need to do

For most customers, no re-configuration is needed. We will gradually be rolling out these changes starting on April 6, 2020, to accounts with archiving enabled.

If you have systems that consume your archived files (like a data warehouse), you’ll have to update those systems to expect the new archiving directory structure. Nothing in the Export API or other LogDNA app experiences will change.

You can learn more about archiving in our archiving guide.

Learn more

IBM Log Analysis with LogDNA and IBM Cloud Activity Tracker with LogDNA are two IBM Cloud services delivered in partnership with LogDNA to deliver full Log Management Service capabilities. Both services are hosted on IBM Cloud in many world-wide regions, allowing you to co-locate your logging and cloud activity events.

Setup is simple and allows you to quickly gain insights into your workloads. Click on the Observability link to gain visibility into the performance and health of your workloads today.

Realize additional value from your data

Archiving your data to IBM Cloud Object Storage opens your data in new exciting ways to realize additional value from your operational logs. Learn more about two of them here:

More from Cloud

Enhance your data security posture with a no-code approach to application-level encryption

4 min read - Data is the lifeblood of every organization. As your organization’s data footprint expands across the clouds and between your own business lines to drive value, it is essential to secure data at all stages of the cloud adoption and throughout the data lifecycle. While there are different mechanisms available to encrypt data throughout its lifecycle (in transit, at rest and in use), application-level encryption (ALE) provides an additional layer of protection by encrypting data at its source. ALE can enhance…

Attention new clients: exciting financial incentives for VMware Cloud Foundation on IBM Cloud

4 min read - New client specials: Get up to 50% off when you commit to a 1- or 3-year term contract on new VCF-as-a-Service offerings, plus an additional value of up to USD 200K in credits through 30 June 2025 when you migrate your VMware workloads to IBM Cloud®.1 Low starting prices: On-demand VCF-as-a-Service deployments begin under USD 200 per month.2 The IBM Cloud benefit: See the potential for a 201%3 return on investment (ROI) over 3 years with reduced downtime, cost and…

The history of the central processing unit (CPU)

10 min read - The central processing unit (CPU) is the computer’s brain. It handles the assignment and processing of tasks, in addition to functions that make a computer run. There’s no way to overstate the importance of the CPU to computing. Virtually all computer systems contain, at the least, some type of basic CPU. Regardless of whether they’re used in personal computers (PCs), laptops, tablets, smartphones or even in supercomputers whose output is so strong it must be measured in floating-point operations per…

IBM Newsletters

Get our newsletters and topic updates that deliver the latest thought leadership and insights on emerging trends.
Subscribe now More newsletters