Monitoring Google Cloud Run

To monitor applications that are running on Google Cloud Run (GCP), you need to set up the GCP agent and then install Instana Google Cloud Run in-process collectors.

The support for Google Cloud Run rests on the following two pillars:

  • An Instana host agent that monitors the Google Cloud Platform (GCP) project and its resources.

    The remainder of the documents refers to the Instana host agent setup to monitor a GCP project as the GCP agent.

    Setting up an Instana host agent to monitor a GCP project provides metrics for the Cloud Run service revisions that the project contains.

  • Instana in-process collectors that are integrated into the container image that you use on Cloud Run.

    Adding an Instana in-process collector to the Docker image adds visibility into the individual instances that Cloud Run starts, provides runtime specific metrics (such as JVM metrics, Node.js metrics, .Net metrics), and also provides tracing of calls.

Instana currently supports only the fully managed platform. Cloud Run on Anthos or GKE is not supported.

Setting up the GCP agent

Before you install a Google Cloud Run in-process collector, ensure that the Instana host agent has been installed on the host machine where GCP exists to monitor the GCP project. A dedicated machine needs to be used to monitor GCP resources. If you want to monitor multiple GCP projects, you need one agent per project.

The Instana Google Cloud integration uses service accounts to connect to the GCP admin API from the GCP agent. So you need to create a service account and provide its credentials to the GCP agent so that it can make API calls on your behalf.

Note: If you have already set it up for another GCP service (such as Google Cloud Storage, Google Cloud SQL), you can skip creating a new service account, but you might need to add extra permissions for Cloud Run.

  1. Go to Google Cloud credentials page for the Google Cloud project that you want to set up the Instana integration.

  2. Press Create credentials and then select Service account key.

    SelectServiceAccount

  3. In the Service account list, select New service account.

  4. Give the service account a unique name.

  5. Add the role Cloud Run Viewer to the account. Alternatively, you can also create a custom role, which has at least the following permissions:

    • resourcemanager.projects.get
    • run.revisions.list
    • run.services.list
    • monitoring.timeSeries.list
  6. Select JSON as the key type, and press Create.

  7. Take note where this file is saved, as it is needed to complete the integration.

  8. In the host agent configuration file <instana-agent-dir>/etc/instana/configuration.yaml file, add an field credentials_path to the com.instana.plugin.gcp section (for all GCP resources) or to the com.instana.plugin.gcp.run section (credentials just for monitoring Cloud Run). The value for this field is the path of the credentials file. (See section Optional configuration for an example.)

To monitor a GCP project, the following services need to be enabled:

Optional configuration

Poll rate

You can configure the rate, at which the GCP agent polls the metric API in seconds by adding the following configuration to the Configuration file of the GCP Agent:

com.instana.plugin.gcp.run:
  enabled: true
  poll_rate: 60  # How often Google's metrics API will be polled
  credentials_path: '/opt/instana/credentials/cred-gcp.json'  # Path to Service Account credentials
  include_tags:
    ...
  exclude_tags:
    ...

Filtering

Multiple tags can be defined, separated by a comma. Tags need to be provided as a key-value pair separated by the : character. To make configuration easier, it is possible to define which tags you want to include in (or exclude from) the discovery process. In case a tag is present in both lists (include and exclude), the exclude list has higher priority. If there is no need for service filtering, the attributes include_tags and exclude_tags needs to be omitted.

To include or exclude Cloud Run service revisions by tags, use the following configuration:

com.instana.plugin.gcp.run:
  include_tags: # Comma separated list of tags in key:value format (e.g. env:dev,env:test)
  exclude_tags: # Comma separated list of tags in key:value format (e.g. env:dev,env:test)

Installing Instana Google Cloud Run in-process collectors

To collect tracing data, runtime-specific metrics, and extra data about the Cloud Run service revision instances, follow the steps:

Automatic installation: Using Instana Cloud native buildpack for Google Cloud Run

The Instana Cloud Native buildpack for Google Cloud Run automatically adds the in-process collectors for your Cloud Run applications that run on:

  • Java®
  • Node.js
  • .NET Core

Using the Instana Cloud Native buildpack for Google Cloud Run is as simple as running the following commands from your application's folder:

# Replace <download_key> with a valid download key; we need to be logged in containers.instana.io because that is where the buildpack image lives
docker login containers.instana.io -u _ -p <download_key>

# Replace <image-name> with your actual image name
pack build <image-name> --buildpack from=builder --buildpack containers.instana.io/instana/release/google/buildpack --builder gcr.io/buildpacks/builder

The --buildpack from=builder option of the pack command is undocumented](https://github.com/buildpacks/docs/issues/258), and it allows you to consider in the build process all the buildpacks that is shipped by the builder even when there is a buildpack explicitly mentions, namely the --buildpack containers.instana.io/instana/release/google/buildpack.

When you use the Instana Cloud Native buildpack for Google Cloud Run, the only extra step that is required to configure the Instana collectors is adding the two environment variables to the service revision as described in the Configure Your Cloud Run Service section.

Manual installation

This section explains how to add the Instana in-process collectors for Cloud Run when you use the Instana Cloud Native buildpack for Google Cloud Run is not possible, or when the Cloud Run application is written in Go.

Java

The Java Cloud Run in-process collector is delivered over the containers.instana.io/instana/release/google/cloud-run/jvm Docker image. You can use in multi-step Docker builds to streamline and speed up the building of Docker images for Java applications that run on Google Cloud Run as follows.

Add the following lines to your Dockerfile anywhere after the last FROM clause:

COPY --from=containers.instana.io/instana/release/google/cloud-run/jvm /instana /instana
ENV JAVA_TOOL_OPTIONS="-javaagent:/instana/instana-standalone-collector.jar"

When you use the Java Cloud Run in-process collector, the only extra step that is required to configure the Instana collectors beyond building the image is adding the two environment variables to the service revision as described in the Configure Your Cloud Run Service section.

Notes:

  • The Docker daemon that you use for the build process needs to be logged in to containers.instana.io.

  • Auto-profile of Java processes running in Google Cloud Run is not supported.

Node.js

The Node.js Cloud Run in-process collector is delivered over the icr.io/instana/google-cloud-run-nodejs Docker image. You can use in multi-step Docker builds to streamline and speed up the building of Docker images for Node.js applications that run on Cloud Run as follows. If your application is developed by using CommonJS, follow the steps in the CommonJS section. If your application is developed in ECMAScript modules, follow the steps in the ECMAScript modules section.

CommonJS

Add the following lines to your Dockerfile anywhere after the last FROM clause:

COPY --from=icr.io/instana/google-cloud-run-nodejs:latest /instana /instana
RUN /instana/setup.sh
ENV NODE_OPTIONS="--require /instana/node_modules/@instana/google-cloud-run"
ECMAScript modules

Add the following lines to your Dockerfile anywhere after the last FROM clause:

  • For Node.js 18.19.0 and later, add the following commands:

    COPY --from=icr.io/instana/google-cloud-run-nodejs:latest /instana /instana
    RUN /instana/setup.sh
    ENV NODE_OPTIONS="--import=/instana/node_modules/@instana/google-cloud-run/esm-register.mjs"
    
  • For Node.js versions earlier to 18.19.0, add the following commands:

     COPY --from=icr.io/instana/google-cloud-run-nodejs:latest /instana /instana
     RUN /instana/setup.sh
     ENV NODE_OPTIONS="--experimental-loader=/instana/node_modules/@instana/google-cloud-run/esm-loader.mjs"
    

Build the container and push it to the image repository of your choice.

Finally, follow the configuration steps that are described in the section Configure Your Cloud Run Service.

Notes:

  • If you are using Multi-stage builds (that is, if you have multiple FROM clauses in your Dockerfile), you need to add the following lines the last FROM clause of your Dockerfile.
  • Auto-profile of Node.js processes running in Cloud Run is not supported.
Versioning

icr.io/instana/google-cloud-run-nodejs:latest uses the latest available image version, no matter what the current major version is. For example, if a new major version v4 is released and currently your image version is v3, the image will be automatically updated to v4 on the next rebuild if you use the tag ":latest".

If you don't want the image to be automatically updated to the latest available image version on the next rebuild, use a specific major version of the image like icr.io/instana/google-cloud-run-nodejs:2 (version 2) and manually update to the next major version.

Build dependencies and native add-ons

If you are using a rather minimal base image for your Node.js containers, which does not have the required packages to build native Node.js add-ons, you might see a warning like this in your Docker build:

gyp ERR! ...some error message from trying to rebuild native add-ons, for example:
gyp ERR! stack Error: Could not find any Python installation to use
...
Warning: Rebuilding native add-ons for @instana/google-cloud-run failed. Monitoring the Cloud Run service revision instance will work nonetheless, but you will miss some Node.js metrics (GC metrics, event loop metrics, ...). See https://www.instana.com/docs/ecosystem/google-cloud-run/#build-dependencies-and-native-add-ons for details.

As the log message implies, monitoring your Node.js Cloud Run service revision instance with Instana works. But certain metrics are missing. Instana recommends installing the missing operating system packages, at least temporarily, during the build. They do not need to be included in the final image. Which packages to install and how to install them depends on the distribution you are using.

See the following example Dockerfile snippet for Linux® Alpine:

COPY --from=icr.io/instana/google-cloud-run-nodejs:latest /instana /instana
RUN apk add --no-cache --virtual .gyp \
        build-base \
        python \
    && /instana/setup.sh \
    && apk del .gyp python
ENV NODE_OPTIONS="--require /instana/node_modules/@instana/google-cloud-run"

See the following example for Debian based distributions (including Ubuntu):

COPY --from=icr.io/instana/google-cloud-run-nodejs:latest /instana /instana
RUN apt-get update \
    && apt-get install build-essential \
    && /instana/setup.sh \
    && apt-get remove -y build-essential
ENV NODE_OPTIONS="--require /instana/node_modules/@instana/google-cloud-run"

For distributions that use yum (CentOS, RHEL, ...), the command to install the required packages would be:

RUN yum groupinstall "Development Tools"
Getting The Node.js Layer From containers.instana.io

If you also use Instana's Cloud Run support with other runtimes (like Java), for which the image layer is only available on the authenticated registry containers.instana.io, and you want to keep things consistent, you can also get the Node.js Instana Cloud Run image layer from containers.instana.io rather than the public IBM container registry, as an alternative to the approach outlined above.

Since the containers.instana.io registry requires authentication, you need to run the following command before you use it in your Docker build:

docker login -u _ -p <your-agent-key> containers.instana.io

The placeholder <your-agent-key> needs to be replaced with your actual Instana agent key. This needs to be done on the machine where you build your Docker image.

Add the following lines to your Dockerfile anywhere after the last FROM clause:

COPY --from=containers.instana.io/instana/release/google/cloud-run/nodejs:latest /instana /instana
RUN /instana/setup.sh
ENV NODE_OPTIONS="--require /instana/node_modules/@instana/google-cloud-run"

Go

The Go Fargate collector is included in github.com/instana/go-sensor starting from v1.22.0 and does not require extra configuration apart from setting up the environment variables that are described in Configure Your Cloud Run Service section. The tracer detects that the app is running on Google Cloud Run and activates the collector automatically.

.Net Core

Instana's support for .Net Core applications that runs in Cloud Run requires the following steps:

  1. Include the Instana Nuget packages to your application.
  2. Set up the environment variables that are described in Configure Your Cloud Run Service section.

And you are good to go, all the rest is entirely automated.

Adding the NuGet packages to your container image

To modify the Dockerfile to add Instana's NuGet packages at build time of your Docker images, see the following example:

# If running on Alpine or muslc-based images
RUN dotnet add Instana.Tracing.Core.Rewriter.Alpine
# Glibc-based images like RHEL and Debian derivatives
RUN dotnet add Instana.Tracing.Core.Rewriter.Linux

# set CoreCLR tracing environment variables
ENV CORECLR_ENABLE_PROFILING=1
ENV CORECLR_PROFILER={cf0d821e-299b-5307-a3d8-b283c03916dd}
ENV CORECLR_PROFILER_PATH=/app/instana_tracing/CoreProfiler.so
ENV DOTNET_STARTUP_HOOKS=/app/Instana.Tracing.Core.dll

Important:

  • Two different packages based on the flavor of lib that your containers use are offered:

    • muslc, used by Alpine-based containers, need the Instana.Tracing.Core.Rewriter.Alpine package.

    • glibc, used by pretty much all non-Alpine distributions, needs Instana.Tracing.Core.Rewriter.Linux.

  • Replace the /app/ with the actual path to the root of your application.

  • Remember to set up the environment variables that are described in Configure Your Cloud Run Service section.

Python

For Google Cloud Run support in Python applications, follow these steps:

  1. Install the instana PIP package

    pip install instana
    
  2. Import the Instana Collector for Google Cloud Run by importing it into your application code

    import instana
    

    Important: Remember to set up the environment variables that are described in Configure Your Cloud Run Service section.

Configuring your Cloud Run service

Notes:

  • These instructions are common to the setup of all Instana in-process collectors on Cloud Run.

  • Before you follow the steps that are described in this section, make sure that your container images are built according to the instructions specific to the runtime that your application uses.

All Instana in-process collectors on Cloud Run need a common set of configuration values to be able to connect to and authenticate with the Instana backend:

  1. Create a revision of the Cloud Run service.

    Creating a Cloud Run service revision

  2. In the service revision configuration page, go to Advanced Settings > Variables.

    Edit the container definition

  3. Add the following two environment variables to the container specification:

Viewing metrics

To view the metrics, complete the following steps:

  1. In the sidebar of the Instana UI, select Infrastructure.
  2. Click a specific monitored host.

Then, you can see a host dashboard with all the collected metrics and monitored processes.

Configuration data

The following static configuration data is collected for each service revision:

  • region,
  • service name,
  • configuration name,
  • revision name,
  • project ID,
  • numeric project ID (project number),
  • creation time of the revision,
  • revision labels

For Cloud Run services with multiple revisions, the GCP agent collects data for the most recent revision.

Performance metrics

Metric Description
Request Count Number of requests that reach the revision. Excludes requests that are not reaching your container instances (for example unauthorized requests or when maximum number of instances is reached).
Request Latency 99th Percentile 99th percentile for the distribution of request latency in milliseconds (for requests that reach the revision).
Request Latency 95thh Percentile 95th percentile for the distribution of request latency in milliseconds (for requests that reach the revision).
Request Latency 50th Percentile 50th percentile for the distribution of request latency in milliseconds (for requests that reach the revision).
Billable Instance Time Billable time aggregated from all container instances of the revision. For a given container instance, billable time occurs when the container instance is starting or at least one request is being processed. Billable time is rounded up to the nearest 100 milliseconds.
Container Memory Allocation 99th Percentile 99h percentile of the container memory allocation of the revision in Gigabytes-seconds.
Container Memory Allocation 95th Percentile 95h percentile of the container memory allocation of the revision in Gigabytes-seconds.
Container Memory Allocation 50th Percentile 50h percentile of the container memory allocation of the revision in Gigabytes-seconds.

Granularity for all metrics is 60 seconds.