This post describes the use of the Knative Quarkus Bench to explore cold start times of serverless functions running on the IBM Cloud Code Engine.

Our previous blog posts have introduced running and tuning the performance of serverless functions on IBM Cloud Code Engine using the IBM Cloud port of the Knative Quarkus Benchmark.

A primary advantage of serverless functions is automatic and nearly transparent scalability of underlying computational resources, including scale-to-zero, which fully releases unused resources. The ability to only deploy and use resources when needed helps reduce total consumed resources, thereby reducing cost. It can also be considered an environmentally friendly approach by reducing energy consumption. However, intuitively, one would expect a serverless function that has not been used recently and has had its resources scaled to zero to take longer to respond to a request than one that has been recently used and thus has its resources currently available.

Base experiment

The following pseudocode demonstrates an experiment our team used to understand the actual impact scale-to-zero has on the response time for serverless functions in IBM Cloud Code Engine. (Details on deploying and deleting serverless applications and accessing the benchmark with curl can be seen in our first blog post.)

Experiment pseudocode:

For pauseTime in 0, 15, 30, 45, 60, 120, 180, 240, 300:
    Deploy sleep benchmark on IBM Cloud Code Engine
    Repeat five times:
        Pause pauseTime seconds, then access benchmark with curl command
    Delete sleep benchmark from IBM Cloud Code Engine

Running this experiment, we learned that without tuning, a warmed up serverless function that has been accessed within the past 60 seconds will respond on average in 0.22 seconds, and a cold serverless function that has not been accessed for over 60 seconds will respond on average in 17.2 seconds. This does seem reasonable, since in one case, the pod is running and available to reply to requests, and in the other case, a pod and other networking services must be deployed. There are certainly many use cases in which the advantage of resource and cost savings offered by scale-to-zero overcome the disadvantage of a potentially slower response time for cold requests.

Sample command to start Knative Quarkus Bench on IBM Cloud Code Engine for this experiment:

ibmcloud code-engine application create --name sleep --image

Sample command used to access and measure sleep bench response time:

$ URL=$(ibmcloud ce app list | grep sleep | tr -s ' ' | cut -d ' ' -f 3)
$ /usr/bin/time curl -s -w "\n" -H 'Content-Type:application/json' -d '"0"' -X POST ${URL}/sleep

Experimental verification of tuning

Next, we experimented with tuning to support use cases where slow cold requests are not acceptable. The min-scale option can be specified when creating applications in IBM Cloud Code Engine. The default value for this option is zero, which permits the number of pods to scale down to zero, thus enabling scale-to-zero.

Theoretically, if min-scale is set to 1, there will always be at least a single pod that can ideally promptly service requests even if there has been no activity for longer than 60 seconds. We verified this behavior by modifying the previous experiment to use --min-scale 1 when creating the serverless function application and then measured the response time after varying pause times, as before. We observed that regardless of pause time, the average response time was 0.22 seconds. Hence, setting --min-scale to at least 1 will significantly improve cold request performance.

Figure 1: Default vs. tuned average serverless function response time after varying periods of inactivity


This post has demonstrated how to use Knative Quarkus Bench to determine the potential performance differences between cold and warm requests to serverless functions. Furthermore, we demonstrated how to use the --min-scale option to avoid the potential performance impacts of scale-to-zero.

We encourage those currently running serverless functions on IBM Cloud Code Engine to consider if their workload could benefit from using the --min-scale option to improve cold-request performance. If you have not already tried out serverless function applications, check out the step-by-step instructions in our first blog post to deploy and access a serverless benchmark application.


More from Cloud

IBM Cloud inactive identities: Ideas for automated processing

4 min read - Regular cleanup is part of all account administration and security best practices, not just for cloud environments. In our blog post on identifying inactive identities, we looked at the APIs offered by IBM Cloud Identity and Access Management (IAM) and how to utilize them to obtain details on IAM identities and API keys. Some readers provided feedback and asked on how to proceed and act on identified inactive identities. In response, we are going lay out possible steps to take.…

IBM Cloud VMware as a Service introduces multitenant as a new, cost-efficient consumption model

4 min read - Businesses often struggle with ongoing operational needs like monitoring, patching and maintenance of their VMware infrastructure or the added concerns over capacity management. At the same time, cost efficiency and control are very important. Not all workloads have identical needs and different business applications have variable requirements. For example, production applications and regulated workloads may require strong isolation, but development/testing, training environments, disaster recovery sites or other applications may have lower availability requirements or they can be ephemeral in nature,…

IBM accelerates enterprise AI for clients with new capabilities on IBM Z

5 min read - Today, we are excited to unveil a new suite of AI offerings for IBM Z that are designed to help clients improve business outcomes by speeding the implementation of enterprise AI on IBM Z across a wide variety of use cases and industries. We are bringing artificial intelligence (AI) to emerging use cases that our clients (like Swiss insurance provider La Mobilière) have begun exploring, such as enhancing the accuracy of insurance policy recommendations, increasing the accuracy and timeliness of…

IBM NS1 Connect: How IBM is delivering network connectivity with premium DNS offerings

4 min read - For most enterprises, how their users access applications and data is an essential part of doing business, and how they service those application and data responses has a direct correlation to revenue generation.    According to We Are Social’s Digital 2023 Global Overview Report, there are 5.19 billion people around the world using the internet in 2023. There’s an imperative need for businesses to trust their networks to deliver meaningful content to address customer needs.  So how responsive is the…