Clients today are increasingly accessing archived data for ad hoc insights over extended periods of time, forensic investigation, and archive for regulatory and business compliance.
The challenge: Running cloud operations at volume may result in large archives of log and activity event data.
The solution: Archiving with hourly frequency.
We are excited to introduce an improved archiving feature for IBM Log Analysis with LogDNA and IBM Cloud Activity Tracker with LogDNA that breaks up archives into an hourly cadence. For most customers, the benefits will be seamless. If you have a custom workflow based on your archived logs, it may have to be updated to accommodate the change.
What’s changing
Starting April 6, 2020, we will slowly transition our customers to a new archive file format that breaks up archives into an hourly cadence instead of a daily cadence.
This feature will allow you to receive archived logs faster, and you’ll be able to unzip just a portion of your logs instead of an entire day’s worth. Additionally, the new folder format better supports Hive partitions.
As we transition you to the new format, you might observe a small overlap between the older daily format archives and the new archives. Expect to see hourly archives in your provider within 24 hours of the logs being received by us.
The new archive directory format for all providers will look like this:
year=YYYY/month=MM/day=DD/<accountID>.<YYYY>-<MM>-<DD>.<HH>00.json.gz (where HH is hours in 24 format)
The current archive directory format looks like this:
/<accountID>.<YYYY>-<MM>-<DD>.<clusterId>.json.gz
The current archive directory format for S3 looks like this:
YYYY/MM/<accountID>.<YYYY>-<MM>-<DD>.<clusterId>.json.gz
The content of the archive files will remain the same as before.
What you need to do
For most customers, no re-configuration is needed. We will gradually be rolling out these changes starting on April 6, 2020, to accounts with archiving enabled.
If you have systems that consume your archived files (like a data warehouse), you’ll have to update those systems to expect the new archiving directory structure. Nothing in the Export API or other LogDNA app experiences will change.
You can learn more about archiving in our archiving guide.
Learn more
IBM Log Analysis with LogDNA and IBM Cloud Activity Tracker with LogDNA are two IBM Cloud services delivered in partnership with LogDNA to deliver full Log Management Service capabilities. Both services are hosted on IBM Cloud in many world-wide regions, allowing you to co-locate your logging and cloud activity events.
Setup is simple and allows you to quickly gain insights into your workloads. Click on the Observability link to gain visibility into the performance and health of your workloads today.
Realize additional value from your data
Archiving your data to IBM Cloud Object Storage opens your data in new exciting ways to realize additional value from your operational logs. Learn more about two of them here: