January 22, 2021 By Damir Krstanovic 4 min read

With PromQL, the Prometheus query language available to all IBM Cloud Monitoring users, we will be performing a metric format migration and increasing metrics limits.

Read on to understand the impact on your Prometheus metrics monitoring.

PromQL GA availability on IBM Cloud Monitoring

Since September 2020, PromQL functionality has been available on IBM Cloud Monitoring with Sysdig. We are now excited to announce that the PromQL functionality graduates to GA, becoming available to all existing and new monitoring instances.

While the graphical Query UI is the most simple and straightforward approach to query metrics, users can also leverage PromQL, the Prometheus query language. You can use PromQL within both Dashboards and Alerts to create advanced data queries and perform arithmetic operations on all metrics — custom metrics such as Prometheus and statsd, but also on out-of-the-box Sysdig agent infrastructure and container metrics:

Prometheus metrics native ingest and migration

Users ingesting Prometheus metrics will be affected by this migration. If you are only using infrastructure metrics and statsd, or you created your account after September 2020, you are not impacted.

PromQL requires Prometheus metrics native ingestion. Until now, Prometheus metrics were ingested as calculated metrics — a format that allowed simpler aggregations based on statsd-like monitoring. With the adoption of PromQL, and to align with the OpenMetrics standard and the Prometheus monitoring ecosystem, we started to ingest Prometheus metrics in native format (raw) in parallel.

This migration will automatically replace Dashboards and Alerts that were created using the old Prometheus metrics format (metrics prefixed with the promlegacy. prefix) with the native metric format (and remove the prefix). Agents will stop sending metrics in calculated format.

During the automated migration, alerts will be disabled. This process takes less than a minute, so your alerting service will not be affected.

What do I need to do?

You should upgrade your agent to the latest version. This is a good practice and will allow you to benefit from increased metrics limits. If you are running agents older than Sysdig agent version 10.0 Prometheus, metrics might stop working since Prometheus native ingestion is not supported.

If you are using the .avg or .count metrics generated by the calculated format from Histogram Prometheus metric types in Dashboards or Alerts, you will have to manually update them. Check the migration details for more information on this.

Increasing metrics limits

Together with this change, the limit of the number of custom metrics time series that agents can send to the IBM Cloud Monitoring service will be updated. The new default limits will be as follows:

  • Prometheus metrics: 17k
  • statsd metrics: 1k
  • JMX metrics: 1k
  • AppCheck metrics: 1k

Agents older than 10.8.0 will not exceed 10k time series, being Prometheus metrics limits 7K time series.

You can manually reconfigure your agent to customize any of these limits — from disabling the metrics that you don’t need, to going to a maximum of 20k time series. The aggregate of all custom metrics should not exceed 20k time series.

You can check your current agent versions, time series usage, and configured limits on the Sysdig agent health and status dashboard:

Further improvements and exciting features to improve Prometheus monitoring compatibility, Kubernetes monitoring and IBM Cloud service monitoring are coming to the IBM Cloud Monitoring service really soon, so stay tuned for updates.

Migration details and roll-out

Time aggregation changes

Calculated metrics format performs some aggregations within the agent for Prometheus counter type metrics. When users query these metrics using the Query UI, the rate is already calculated at ingestion time.

With the new Prometheus native format (where metrics are ingested raw), the following time aggregations will be automatically replaced in user’s dashboards and alerts — based on the type of the metric —when calculated metrics (with promlegacy. prefix) are found:

Prometheus Histogram support (manual action required)

We are deprecating the special handling we did with calculated format on Prometheus Histogram metrics type, where for one Prometheus Histogram metric, calculated metrics automatically generated two — .avg and .count metrics.

Now these generated metrics disappear, and a manual update is required. You should replace queries on Histograms with the right PromQL function for you. Typically, those would be avg, count, rate or histogram_quantile. You can do this any time before or after the migration, but after the migration, the dashboard panels and alerts with this metric type will stop working.

Migration roll-out timelines

The migration will across all IBM Cloud Monitoring regions according to the following schedule:

  • AU-SYD: January 25, 2021
  • EU-DE: January 25, 2021
  • JP-TOK: January 26, 2021
  • EU-GB: January 27, 2021
  • US-EAST: January 27, 2021
  • US-SOUTH: January 26, 2021

Getting help

We’d love your feedback and an opportunity to help in event you have any questions, so feel free to email us directly.

Learn more about IBM Cloud Monitoring with Sysdig.

More from Analytics

A new era in BI: Overcoming low adoption to make smart decisions accessible for all

5 min read - Organizations today are both empowered and overwhelmed by data. This paradox lies at the heart of modern business strategy: while there's an unprecedented amount of data available, unlocking actionable insights requires more than access to numbers. The push to enhance productivity, use resources wisely, and boost sustainability through data-driven decision-making is stronger than ever. Yet, the low adoption rates of business intelligence (BI) tools present a significant hurdle. According to Gartner, although the number of employees that use analytics and…

VeloxCon 2024: Innovation in data management

3 min read - VeloxCon 2024, the premier developer conference that is dedicated to the Velox open-source project, brought together industry leaders, engineers, and enthusiasts to explore the latest advancements and collaborative efforts shaping the future of data management. Hosted by IBM® in partnership with Meta, VeloxCon showcased the latest innovation in Velox including project roadmap, Prestissimo (Presto-on-Velox), Gluten (Spark-on-Velox), hardware acceleration, and much more. An overview of Velox Velox is a unified execution engine that is built and open-sourced by Meta, aimed at…

How the Recording Academy uses IBM watsonx to enhance the fan experience at the GRAMMYs®

3 min read - Through the GRAMMYs®, the Recording Academy® seeks to recognize excellence in the recording arts and sciences and ensure that music remains an indelible part of our culture. When the world’s top recording stars cross the red carpet at the 66th Annual GRAMMY Awards, IBM will be there once again. This year, the business challenge facing the GRAMMYs paralleled those of other iconic cultural sports and entertainment events: in today’s highly fragmented media landscape, creating cultural impact means driving captivating content…

IBM Newsletters

Get our newsletters and topic updates that deliver the latest thought leadership and insights on emerging trends.
Subscribe now More newsletters