Installing the host agent on Linux

You can install the Instana host agent on Linux in an online or offline (air-gapped) environment.

Introduction

In an online environment (where the host can communicate with the Instana hosted agent repositories), you can have the installation steps completed automatically by using the one-liner installation technique on supported Linux distributions.

Alternatively, in an online or offline (air-gapped) environment, you can download the agent package manually and then either install the package manually or create a repository and instruct your package manager to install the package for you. You can also install and run the agent from a tarball archive, such as to run the Instana agent as a non-root user.

In situations where it is undesirable or impossible to make the agent communicate with the Instana hosted agent repositories, you can use a local mirror instead. You can configure the Instana host agent to use the agent repository as the mirror, configure a different mirror, or can change the agent repository location.

Procedure: Start your installation here

Before installing the host agent, consider the following topics:

  • The Instana host agent has deep access into the observed system where it is installed. For security considerations when the agent is running, see Host agent security considerations.
  • Different versions of the Instana host agent appear in the Instana UI and in the documentation. To collect the various versions, see Host agent versioning.

Use the following procedure to install the host agent on Linux. The section for each step guides you onward to any subsequent or alternative step so that you don't need to return to this section.

Note

The main sections that follow are to install the host agent on Linux by using either an automated (one-liner) script or (DEB, RPM) package. If you want to install and run the agent from a tarball archive, such as to run the Instana agent as a non-root user, see one of the following sections:

Steps:

  1. Check that you have a supported operating system. Before installing the agent, ensure that you have a supported Linux operating system and distribution.

  2. Check that you have a supported platform architecture. Before installing the agent, ensure that you have a supported platform architecture.

  3. Choose how you want to install the agent. Choose either (1) to use the automated one-liner installation technique or (2) to download the agent package then install using a package manager or manually.

  4. Install the agent. Depending on how you chose to install the agent, complete the steps needed:

After you install the agent on Linux, you can perform a variety of activities on the agent including to check the agent's status and start or update the agent.

Checking that you have a supported operating system

Check that you have one of the following Linux operating systems and distributions that are supported for the Instana host agent:

  • Ubuntu Linux 14.04 (trusty)
  • Ubuntu Linux 16.04 (xenial)
  • Ubuntu Linux 18.04 (beaver)
  • Ubuntu Linux 20.04 (focal)
  • Ubuntu Linux 22.04 (jammy)
  • CentOS 6 [1]
  • CentOS 7
  • CentOS 8
  • CentOS 9
  • Debian 9 (stretch)
  • Debian 10 (buster)
  • Debian 11 (bullseye)
  • SUSE Linux Enterprise Server (SLES) 12 [2]
  • SUSE Linux Enterprise Server (SLES) 15
  • Red Hat Enterprise Linux (RHEL) 6 [3]
  • Red Hat Enterprise Linux (RHEL) 7 [3:1]
  • Red Hat Enterprise Linux (RHEL) 8 [4]
  • Red Hat Enterprise Linux (RHEL) 9
  • Oracle Enterprise Linux 7
  • Oracle Enterprise Linux 8
  • Oracle Enterprise Linux 9
  • Amazon Linux 1
  • Amazon Linux 2
  • Alma Linux 8
  • Alma Linux 9
  • z/Linux, Linux on Z (s390x, 64 bit) [5]
  • z/OS (OS/390) - (s390x) - Supported WebSphere and Liberty tracing
  • Kylin Linux Advanced Server v10 (x86_64 & aarch64) [6]

What to do next: Check that you have a supported platform architecture

Checking that you have a supported platform architecture

Check that you have a Linux operating system and distribution on the following supported platform architectures:

Name Bitness Vendor
x86 32 bit Intel/AMD
x64, amd64 64 bit Intel/AMD
arm 32 bit ARM
arm64, aarch64 64 bit ARM
ppc 32 bit IBM Power
ppc 64 bit IBM Power
ppc64le 64 bit IBM Power, little endian
s390x 64 bit IBM z/Architecture

What to do next: Choose how you want to install the agent

Choosing how you want to install the agent

Choose one of the following installation options and click the associated link to continue:

  • In an online environment (where the host can communicate with the Instana hosted agent repositories), you can have the installation steps completed automatically by using the one-liner installation technique on supported Linux distributions. To continue, see Installing the agent by the automated (one-liner) script.

  • Alternatively, in an online or offline (air-gapped) environment, you can download the agent package manually and then either install the package manually or create a repository and instruct your package manager to install the package for you. You can also install and run the agent from a tarball archive, such as to run the Instana agent as a non-root user. To continue, see Installing the agent by package manager or manually (DEB, RPM).

  • Install and run the agent from a tarball archive. To install and run the agent from a tarball archive, download and extract the archive, and then start the agent. To continue, see Installing and running the agent from a tarball archive.

  • Install and run the agent as a non-root user.

    To run the Instana agent as a non-root user, you install the agent from a tarball archive, but only some sensors are supported. To continue, see Installing and running the agent as a non-root user.

Installing the agent by the automated (one-liner) script

The agent one-liner is a powerful script that performs automated installations of the host agent on Linux-based operating systems.

Checking that you have a supported Linux distribution

The one-liner supports the following Linux distributions:

  • Debian derivatives (apt package manager)
  • Red Hat derivatives (yum package manager)
  • SUSE derivatives (zypp package manager)

Installing the agent by using the one-liner

You need root user privileges to install the Instana host agent.

To use the one-liner to install the host agent, complete the following steps:

  1. Sign in to the Instana UI, and then select an option to display the agent catalog; for example, on the home page, click Deploy Agent located at the top right corner.

    If you are starting a new trial instance of Instana, the agent catalog is displayed with a prompt to select a host agent to install.

  2. Click the tile Linux - Automatic Installation (One-liner)

  3. Select the packaging option to install a dynamic or static agent, and the runtime options that you want to use. If you change options, then the script parameters are updated.

    The script is pre-filled with your agent key and host agent endpoint.

    For example, the UI provides the following script to install the agent with the Dynamic packaging option, and with the Azul Zulu 1.8 and Interactive runtime options:

    curl -o setup_agent.sh https://setup.instana.io/agent && chmod 700 ./setup_agent.sh && sudo ./setup_agent.sh -a <your_agent_key> -d <your_agent_key> -t dynamic -e <host-agent-endpoint>
    
  4. (Optional) Select the option to install and start the host agent as a service (only supported for systemd-based systems). This option adds the -s parameter to the script.

  5. Copy and then run the script to install the host agent.

    Before running the script, you can add other parameters for more options. For a description of available parameters, see One-liner script parameters.

Result: You have installed the host agent.

  • If you selected the option to install and start the host agent as a service, the agent starts automatically.
  • The public systems package repository of Instana is added to your system by creating a repository file. This file adds Instana authenticated repositories to the machine's installation sources.

What to do next:

Check the status of the agent in the Instana UI or on the host. For more information, see Checking the status of the host agent.

If the host agent is not started, start the host agent by running one of the following commands:

  • If your operating system uses systemd as the init system (and you have not already started the agent as a service), run the systemctl start instana-agent.service command.
  • If your operating system uses SysVinit as the init system, run the service instana-agent start command.

For more information about starting the agent and enabling it as a service, see Starting the agent.

If you have problems installing the agent, see Potential problems during installation.

You can perform other installation-related activities on the agent to customize the agent configuration, update the agent, uninstall the agent, and manage the agent in the Instana UI.

One-liner script parameters

The one-liner script accepts the following parameters:

Parameter Description
-a = (required) The agent key.
-d = (optional) The Instana download key. If you are using a self-hosted (on-premises) Instana deployment, the key is provided for you by Instana.
-e = (required) The host agent endpoint.
-m = (optional) Sets the Instana agent mode, apm (default), infra, or aws.
-t = (optional) The agent flavor, dynamic (default) or static.
-j = (optional) Selects Eclipse OpenJ9 11 as the bundled Java runtime.
-y = (optional) Noninteractive prompt. Specify it if the setup is being performed without an interactive shell.
-s = (optional) Start the instana-agent service, and enable it to start at boot time. This option works only for systems that is running systemd.
-g = (optional, needed if -b is set) The specification of the remote URL for the Git-based configuration management capability of the host agent.
-b = (optional, needed if -g is set) The name of the remote branch to track for the Git-based configuration management capability of the host agent.
-u = (optional, needed if -p is set) The username for basic authentication if you are using HTTP-based remotes for the Git-based configuration management capability of the host agent.
-p = (optional) The password for basic authentication if you are using HTTP-based remotes for the Git-based configuration management capability of the host agent.

Potential problems during installation

Debian-based

With Debian-based derivatives, you might experience the following output when you issue the installation. In this case, install the package apt-transport-https because Apt doesn't currently pull fonts from HTTPS servers.

  Setting up Instana APT repository
  Importing Instana GPG key
  Updating apt metadata ...
  E: The method driver /usr/lib/apt/methods/https could not be found.
  APT repository metadata update failed

SUSE and SLES

If the SUSE Linux Enterprise Server 12 gives you the following output, you need to update your openssl to properly receive artifacts through modern HTTPS connections:

  Setting up Instana agent for GNU/Linux
  Setting up Instana zypper repository
  Updating zypper metadata ...
  Installing Instana agent ...
  Error building the cache:
  [instana-agent|https://_@packages.instana.io/agent/generic/x86_64] Valid metadata not found at specified URL
  Some of the repositories have not been refreshed because of an error.
  No provider of 'instana-agent-static' found.
  Instana agent package install failed

CentOS

On CentOS 6 based systems, outdated libcurl versions can cause this log output:

Setting up Instana agent for GNU/Linux
Setting up Instana YUM repository
Updating YUM metadata ...
YUM repository metadata update failed

RHEL

Enabled FIPS (Federal Information Processing Standard) or SELinux can lead to the following error during one-liner installation:

RHEL 8 rpm package installer is compliant with FIPS standards, while RHEL 6 and 7 ELS are not.

Error unpacking rpm package instana-agent-dynamic-20210630-0948.x86_64
Error: Transaction failed

Manual installation of instana-agent packages fails in the following setup:

sudo rpm -ivh instana-agent-dynamic-20210713-1352.x86_64.rpm
...
error: unpacking of archive failed on file /etc/init.d/instana-agent;60eda3b3: cpio: Digest mismatch
error: instana-agent-dynamic-20210713-1352.x86_64: install failed

A possible workaround for manual installation is to skip verification with the following command:

sudo rpm -ivh --nodigest --nofiledigest instana-agent-dynamic-20210713-1352.x86_64.rpm

Installing the agent by a package manager or manually (DEB, RPM)

You can download an agent package for DEB or RPM from the Instana UI. You can then instruct your package manager to install the package for you or install the package manually.

In situations where it is undesirable or impossible to make the agent communicate with the Instana hosted agent repositories, you can use a local mirror instead. You can configure the Instana host agent to use the agent repository as the mirror, configure a different mirror, or can change the agent repository location.

Downloading the agent package

If you are on a self-hosted backend, the backend version must be 275 to download an agent package.

To download an Instana agent package, complete the following steps:

  1. Sign in to the Instana UI, and then select an option to display the agent catalog; for example, on the home page, click Deploy Agent.

    If you are starting a new trial instance of Instana, the agent catalog is displayed with a prompt to select a host agent to install.

  2. Click the tile Linux - Packages(DEB, RPM).

  3. To configure the agent, select one of the options in the following areas:

    • Packaging: Click Dynamic or Static.
    • Runtime: Click Azul Zulu 1.8 or Eclipse OpenJ9 11.
    • Mode: Click RPM or DEB.
  4. From the Platform architecture menu, select your platform architecture.

  5. Click the Download icon to download the agent package.

In an offline (air-gapped) environment that is not connected to the internet, you can download the agent package for DEB or RPM directly from the Artifactory. Make sure that you download the latest package.

Installing the agent package

Instruct your package manager to install the agent package for your Linux distribution (or install the agent package manually):

Debian derivatives (apt package manager)

To prepare Apt with resources for the Instana agent packages, create the following files:

# /etc/apt/sources.list.d/instana-agent.list
deb [arch=amd64] https://packages.instana.io/agent generic main
# /etc/apt/auth.conf.d/instana-packages.conf
machine packages.instana.io
login _
password <your_agent_key>

Alternatively, you might hardcode the authentication in the URL in /etc/apt/sources.list.d/instana-agent.list, but it causes a mild security risk and apt issues warnings that are shown whenever you install or update Instana packages.

To add the Instana GPG key to APT, run the following command:

wget -qO - https://packages.instana.io/Instana.gpg | sudo apt-key add -

To continue, after you refresh the package manager's sources, prepare the agent for installation.

RPM-based distributions (yum package manager)

Copy the following code snippet into your yum sources (/etc/yum.repos.d/Instana-Agent.repo):

[instana-agent]
name=Instana
baseurl=https://_:<your_agent_key>@packages.instana.io/agent/generic/x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.instana.io/Instana.gpg
priority=5
sslverify=1

Replace <your_agent_key> with your agent key.

If you want to enable GPG package verification, some distributions require that you import the key into the package manager's key ring beforehand.

To import the key, run the following command:

rpm --import https://packages.instana.io/Instana.gpg

To continue, after you refresh the package manager's sources, prepare the agent for installation.

openSUSE/SLES (zypp package manager)

For openSUSE and SUSE Linux Enterprise Server, the path is /etc/zypp/yum.repos.d/Instana-Agent.repo, but the contents can remain the same as RPM-based distribution.

To continue, after you refresh the package manager's sources, prepare the agent for installation.

Preparing the agent

After you refresh the package manager's sources, configure the host agent endpoint and the agent key.

If the following environment variables are set before you run the installation, the agent is automatically configured.

export INSTANA_AGENT_KEY=$agentkey
export INSTANA_AGENT_HOST=$endpoint
export INSTANA_AGENT_PORT=$endpoint_port

The endpoint, port, and agent key values are shown on Agent deployment screens in the Instana UI; for example, on a Linux - Automatic Installation (One-liner) screen, the deployment code includes ... -a aGeNTKEY0vaLuO0Eu1ABc ... -e ingress-green-saas.instana.io:443.

Summary of options for setting environment variables:

  • Git-based configuration management: Placing this information in a file is a good approach.
  • Systemd: A drop-in is the easiest approach to overwriting environment specifics for services.
  • SysVinit: Place the files in either /etc/sysconfig (Red Hat derivatives) or /etc/default (Debian derivatives).

For more information about the options for environment variables, see Setting and overriding environment variables.

To continue, install the agent on your chosen architecture.

Installing the agent

After you refresh the package manager's sources, run the command to install the agent on your chosen architecture:

Debian derivatives

  • Static agent: apt-get install instana-agent-static
  • Dynamic agent: apt-get install instana-agent-dynamic

Red Hat derivatives

  • Static agent: yum install instana-agent-static
  • Dynamic agent: yum install instana-agent-dynamic

Eclipse OpenJ9 11 bundled packages

Install the agent with one of the following package names: instana-agent-static-j9 or instana-agent-dynamic-j9.

Result: You have installed the host agent.

What to do next:

Check the status of the agent in the Instana UI or on the host. For more information, see Checking the status of the host agent.

If you have problems installing the agent, see Potential problems during installation.

You can perform other installation-related activities on the agent to customize the agent configuration, update the agent, uninstall the agent, and manage the agent in the Instana UI.

Potential problems during installation

You might meet the following problems during installation:

  • Debian-based Derivatives might present a connection error message. This bug is found in GnuTLS-depending programs like Curl, git, or apt:

    GnuTLS recv error (-9): A TLS packet with unexpected length was received
    
  • With Debian-based Derivatives, you might experience the following output when you issue the installation. In this case, install the package apt-transport-https because Apt doesn't currently pull fonts from HTTPS servers.

    E: The method driver /usr/lib/apt/methods/https could not be found.` `APT repository metadata update failed
    
  • In case SUSE Linux Enterprise Server 12 gives you the following output, you need to update your GnuTLS, OpenSSL, and NSS libraries and its dependent Programs (for example Curl). This update is important to properly receive artifacts through modern HTTPS connections:

    Error building the cache:
    [instana-agent|https://_@packages.instana.io/agent/generic/x86_64] Valid metadata not found at specified URL
    Some of the repositories have not been refreshed because of an error.
    No provider of 'instana-agent-static' found.
    Instana agent package install failed
    
  • If you try to enable the instana-agent service by using systemd, you can see the following error message in SUSE Enterprise Linux:

    systemctl enable instana-agent
    
    Synchronizing state of instana-agent.service with SysV service script with /usr/lib/systemd/systemd-sysv-install.
    Executing: /usr/lib/systemd/systemd-sysv-install enable instana-agent
    ln -sf ../instana-agent /etc/init.d/rc2.d/S50instana-agent
    ln: failed to create symbolic link '/etc/init.d/rc2.d/S50instana-agent': No such file or directory
    

    To solve the error, delete the file /etc/init.d/instana-agent, and then run the systemctl enable instana-agent command.

    Then, you can see that the symlink is created correctly:

    systemctl enable instana-agent
    
    Created symlink /etc/systemd/system/multi-user.target.wants/instana-agent.service → /lib/systemd/system/instana-agent.service.`
    
  • On CentOS 6 based systems, outdated libcurl versions can cause this log output:

    Updating YUM metadata ...
    YUM repository metadata update failed
    

To solve the issue, update the libcurl versions.

Performing post-installation activities

After installing the agent by automated (one-liner) script, package manager, or manually (DEB, RPM), you can perform installation-related activities on the agent such as to customize the agent configuration or update the agent.

You can also use the Instana UI to manage the agent. You can get a quick overview of all agents, their important properties, and general health. Each host agent also has its own Agent Management View, which displays real-time information about important metrics and allows you to configure the agent in various ways.

Customizing the agent configuration

After you install the host agent, you can customize the configuration of the agent as you need. For example, you can change the TCP port and host name of the Instana backend/service that your host agent connects to, configure the agent to report to multiple backends, or configure use of a proxy for indirect communication from the host agent to the backend.

For more information about customizing the agent configuration, see Configuring the host agent.

Starting the agent

To start the Instana agent, use one of the following commands:

  • systemctl start instana-agent.service, if your operating system uses systemd as init system.

  • service instana-agent start, if your operating system uses SysVinit as init system.

When you install the agent, it is enabled to start automatically at system boot. You can turn off this behavior by setting the environment variable INSTANA_AGENT_AUTOSTART=false when installing the agent.

Updating the agent

You can use the package manager of your operating system to upgrade an agent package on your machine. To update and manage the packages and patches of your system, run the following command:

yum update // apt-get upgrade

Then the instana-agent-static or instana-agent-dynamic packages are upgraded.

Uninstalling the agent

To uninstall the Instana agent, the first step is to get the name of the package. Then you can run a command with the package name.

Uninstall on Debian Derivatives

Get the name of the package that is installed on your system by running the following command:

apt list --installed | grep instana-agent

The output holds the exact package name to use for the uninstall command. In the following example, the package name is instana-agent-static-j9/generic:

instana-agent-static-j9/generic,now 20221024-1455 amd64 [installed]

Uninstall the agent by running the following command:

sudo apt-get purge <package_name>

Where <package_name> is the package name that you get.

Uninstall on Red Hat Derivatives

Get the name of the package that is installed on your system by running the following command:

yum list installed | grep instana-agent

The output holds the exact package name to use for the uninstall command. In the following example, the package name is instana-agent-static-j9.x86_64:

instana-agent-static-j9.x86_64       20221024-1455                 @instana-agent

Uninstall the agent by running the following command:

sudo yum remove <package_name>

Where <package_name> is the package name that you get.

Some files that are created by the Instana agent are not part of the package on rollout. The process of clean-up the installation of the Instana agent requires an extra manual step to remove the installation directory. The clean-up is required for SUSE and Red Hat derivatives because Yum doesn't support clean-up.

Run the following command to remove the installation directory:

rm -rf /opt/instana/agent

Installing and running the agent from a tarball archive

To install and run the agent from a tarball archive, download and extract the archive, and then start the agent.

The tarball archive is pre-filled with your agent key and host agent endpoint.

The Instana host agent must be installed in a system-wide accessible location and needs to be run as the root user. If you run the host agent with any other user, the functions of the host agent are limited. This is because some of the performance metrics are supported only by the root. The monitoring Docker containers can be done only by the root on the Docker host machine. in addition, only either the root or startup users can attach and monitor Java virtual machines (JVMs). If you can't run the host agent as root, ensure that the agent user is listed in sudoers with a valid shell.

The user that is running the agent needs to be able to write in the agent directory and all its subdirectories. The agent downloads the required sensors in accordance with the auto detection and creates log files in its data subdirectory. Make sure that you have about 100 MB of free disk space.

Prerequisites

A Java Development Kit (JDK) needs to be available for the agent. You can see the list of supported JDKs as follows. You have two options:

  • Place or link the JDK into <instana-agent-install-dir>/jvm (so that <instana-agent-install-dir>/jvm/bin/java exists).

  • The customizable way is to export an environment variable that is called JAVA_HOME to point to that JDK (this environment variable can be also set via instana-agent-install-dir>/bin/setenv).

The following JVMs are supported for running the agent:

The JVM must be executable for all users on the system. You are recommended to use the latest available patch release of the Java distribution of your choice. The agent requires a JDK that supports TLSv1.3 (available in all current JDK 8 and JDK 11 builds). Depending on your OS distribution, the packages that are provided by the OS distributor might not contain strong encryption support due to export control. If you use such a package, you might meet the errors like "java.lang.RuntimeException: Could not generate DH key pair."

To reduce the resources used by the agent when detect file system changes, you need to install the inotify-tools package on your Linux distribution.

Agent communication

The agent downloads updates and sensors from the following host:

DNS Name: artifact-public.instana.io Destination Port: tcp/80 and tcp/443

Currently, the Instana Service is provided in two different regions. Your individual instance is located geographically closest to most of your agents and users. Agents are preconfigured upon download, but some installation methods that require Instana Backend configuration. Consult the Agent Management section inside the product or your technical contact at Instana to learn about the region where your instance is located.

If you are using the self-hosted (on-premises) Instana, use the endpoints that are described at Self-Hosted Classic Edition (Docker).

If you are using Instana SaaS, use the endpoints that are described at host agent endpoints.

Downloading the agent

To download the agent, complete the following steps:

  1. Sign in to the Instana UI, and then select an option to display the agent catalog; for example, on the home page, click Deploy Agent.

    If you are starting a new trial instance of Instana, the agent catalog is displayed with a prompt to select a host agent to install.

  2. Click the tile Linux - Archive (tar.gz)

  3. Select the packaging option to install a dynamic or static agent.

  4. In the dropdown menu, select the platform architecture (operating system).

    The archive file is automatically preconfigured with your Instana account settings so that you only need to extract and start the agent.

  5. Click the download link, and then extract the agent archive.

Download the agent with wget

To download the agent with wget, run the following commands:

wget --save-cookies {{agent_folder_name}}/instana-cookies.txt --post-data 'email={{instana_username}}&password={{instana_password}}' https://{{instana_tenant_unit}}-{{instana_tenant}}.instana.io/auth/signIn
wget --content-disposition --load-cookies {{agent_folder_name}}/instana-cookies.txt --post-data 'type=linux64' https://instana.io/ump/{{instana_tenant}}/{{instana_tenant_unit}}/agent/download -O {{opt_folder}}/{{name_of_agent_archive.tar.gz}}

Starting the agent

To start the agent, run the following command:

INSTANA_AGENT_FOLDER/bin/start

Stopping the agent

To stop the agent, run the following command:

INSTANA_AGENT_FOLDER/bin/stop

Status of the agent

To see the status of the agent, run the following command:

INSTANA_AGENT_FOLDER/bin/status

Installing and running the Instana agent as a non-root user

This feature is experimental. Only the .tar file installation and selected sensors are supported now.

To run the Instana agent as a non-root user, you need to extract the Instana agent as a .tar file.

.tar file installation (RHEL 9)

To install the Instana agent, complete the following steps:

  1. Create a normal user for the Instana agent:

    sudo adduser instana
    
  2. Install Java.

  3. Download the Instana agent .tar file from the agent download page, and then extract the .tar files. For extraction, use GNU tar. After you extract the .tar files, the instana-agent-* directory is displayed.

    tar -xvf <instana-agent-*>.gz
    
  4. Change the owner of the instana-agent directory:

    chown -R instana:instana /instana-agent
    
  5. Go to the INSTANA_AGENT directory and start the Instana agent:

    #To start agent
    ./bin/start
    
  6. Optional: Check the status of the agent:

    ./bin/status
    

Supported features

The Instana agent that is run as a non-root user supports the following features:

Java attachment

For JVM-based applications, you need to run target JVM application on the same user account where the Instana agent is running so that the attachment works.

Host sensor

The host sensor requires the sudo entry permission in the sudoers file to access readlink command (/proc directory) that is mentioned in the Troubleshooting section.

ACE sensor

If the ACE process is started with the user that starts the Instana agent, everything works well including auto-discovery.

If the ACE process is started with a different user (not the one that starts the Instana agent), then environment variables return empty. In this case, the ACE sensor supports remote monitoring and local monitoring with rest API. But auto-discovery doesn't work.

Troubleshooting a non-root agent

The following issues might be encountered when the Instana agent is started:

  • Issue: Access denied for /var/lib/instana

    java.nio.file.AccessDeniedException: /var/lib/instana
    

    Solution: Create a directory with root privilege and change ownership to an Instana user

    sudo mkdir /var/lib/instana
    sudo chown instana: /var/lib/instana
    sudo chmod 700 /var/lib/instana
    
  • Issue: Memory calculcator error

    Skipping the memory calculator: cannot retrieve the maximum amount of memory to be used by the agent from /sys/fs/cgroup/memory/memory.limit
    The memory calculator has not been used, and neither the 'JAVA_MAX_MEM' nor 'JAVA_OPTS' environment variables specify a maximum heap setting
    

    Solution: This error must not impact the agent itself. The memory calculator is a small binary that is used for systemd or containers to determine how much Java heap to assign to the agent, based on limits set. You can fix it by entering the JAVA_OPTS environment variable.

  • Issue: Password prompting issue that is caused by the agent trying to read the symlink of the process, which is owned by the root.

    Solution:

    • Add following entry in the sudoers file:

      instana ALL=(root)    NOPASSWD:/usr/bin/readlink
      

      This entry allows the agent to access the readlink command for root process.

    • Optional: Debug the issue by enabling sudo logs:

      Defaults logfile=/var/log/sudo.log
      

      The following snippet displays a log where the agent script tries to read the readlink command:

      Nov 12 06:03:19 : instana : command not allowed ; TTY=pts/2 ;
          PWD=/home/tester/instana-agent ; USER=root ; COMMAND=/usr/bin/readlink
          /proc/21025/exe
      Nov 12 06:03:19 : instana : command not allowed ; TTY=pts/2 ;
          PWD=/home/tester/instana-agent ; USER=root ; COMMAND=/usr/bin/readlink
          /proc/21025/cwd
      Nov 12 06:03:19 : instana : command not allowed ; TTY=pts/2 ;
          PWD=/home/tester/instana-agent ; USER=root ; COMMAND=/usr/bin/readlink
          /proc/12152/exe
      Nov 12 06:03:19 : instana : command not allowed ; TTY=pts/2 ;
          PWD=/home/tester/instana-agent ; USER=root ; COMMAND=/usr/bin/readlink
          /proc/12152/cwd
      

Reference

Reference information about environment variables, configuration files, and temporary files used when installing the host agent.

Setting and overriding environment variables

When the agent is installed as a package and is started by SystemV init or systemd, a place is needed to set specific Instana environment variables. Also, it might be necessary to override specific environment variables. For example, for LXC container support, the agent needs to have the LXC command-line utils on its path.

Check the following sections about setting and overriding environment variables:

SystemV Init

The agent sources the corresponding shell scripts in the /etc/default/ directory (Debian and derivatives) or /etc/sysconfig/ directory (Red Hat and derivatives). If the service name is not changed, the SysVinit script sources /etc/default/instana-agent file or /etc/sysconfig/instana-agent file.

For example, in Debian the Instana variables and PATH can be updated by using /etc/default/instana-agent (when this file is found readable):

INSTANA_AGENT_KEY=$agentkey
INSTANA_AGENT_HOST=$endpoint
INSTANA_AGENT_PORT=$endpoint_port
PATH=/usr/local/bin:${PATH}

The endpoint, port, and agent key values are shown on Agent deployment screens in the Instana UI; for example, on a Linux - Automatic Installation (One-liner) screen, the deployment code includes ... -a aGeNTKEY0vaLuO0Eu1ABc ... -e ingress-green-saas.instana.io:443.

Systemd

For systemd, you can find the available options, drop-in units, EnvironmentFile directives, and global environment settings. Drop-in units are the most recommended approach.

In the following options, make sure to run sudo systemctl daemon-reload followed by sudo systemctl restart instana-agent.service to reload the agent with the variable changes.

Drop-in units

By using drop-in units, you can override the specific settings without changing the original systemd unit file. Drop-ins must be placed in /etc/systemd/system/<unit>.d/ with a name <name>.conf.

For example, to set the Instana and PATH environment variables, create the /etc/systemd/system/instana-agent.service.d/10-environment.conf file with the contents:

[Service]
Environment=INSTANA_AGENT_KEY=$agentkey
Environment=INSTANA_AGENT_HOST=$endpoint
Environment=INSTANA_AGENT_PORT=$endpoint_port
Environment=PATH=/usr/local/bin:${PATH}
EnvironmentFile directives

This solution uses the drop-in units except the environment directives that is read from a separate file. For example, create a /etc/instana/environment.conf file, with the following contents:

INSTANA_AGENT_KEY=$agentkey
INSTANA_AGENT_HOST=$endpoint
INSTANA_AGENT_PORT=$endpoint_port
PATH=/usr/local/bin:${PATH}

Then, in the systemd (drop-in) unit file, use the following configuration:

[Service]
EnvironmentFile=/etc/instana/environment.conf
Global environment settings

By default, systemd reads global configuration from various paths such as /etc/systemd/system.conf or /etc/systemd/system.conf.d/10-default-env.conf. By placing DefaultEnvironment configuration in these files, they become available to all systemd units. For more information, see systemd config.

For example, the following configuration creates three Instana environment variables when you set in the/etc/systemd/system.conf.d/10-default-env.conf file:

[Manager]
DefaultEnvironment="INSTANA_AGENT_KEY=<key>" "INSTANA_AGENT_HOST=<host>" "INSTANA_AGENT_PORT=<port>"

Configuration files

The following files are created once during the installation:

  • /opt/instana/agent/etc/mvn-settings.xml
  • /opt/instana/agent/etc/org.ops4j.pax.url.mvn.cfg
  • /opt/instana/agent/etc/instana/com.instana.agent.bootstrap.AgentBootstrap.cfg
  • /opt/instana/agent/etc/instana/com.instana.agent.main.config.Agent.cfg
  • /opt/instana/agent/etc/instana/com.instana.agent.main.config.UpdateManager.cfg
  • /opt/instana/agent/etc/instana/com.instana.agent.main.sender.Backend.cfg
  • /opt/instana/agent/etc/instana/configuration.yaml

These files are not overwritten when a user edits them or the package manager upgrades the agent package.

Temporary files

The Instana host agent installs some files that it needs in /tmp/.instana/ directory, if those files are deleted over the uptime of the host, it can cause issues during updates.

Tmpfiles.d

On system.d-based systems, the systemd-tmpfiles deamon can delete files that are needed by the host agent in the /tmp/.instana directory, which causes update failures that are hard to troubleshoot. Since version v181, DEB and RPM agent packages automatically install the following configuration script in /usr/lib/tmpfiles.d that adjusts the behavior to play nice with the Instana host agent:

# Instana files in /tmp cannot be deleted when the agent is running,
# or it may lead to issues on agent update
R! /tmp/.java_pid*
R! /tmp/.instana/*

This configuration allows tmpdfiles.d to delete the files that are needed by the Instana agent only after a host restart. The newly running Instana host agent re-creates the files on startup.

IMPORTANT: Red Hat Enterprise Linux and CentOS, in versions 6 and 7, ship default tmpdfiles.d policies that delete the entire /tmp directory every 10 days regardless of which policies a package like Instana's can specify. It means that a package like instana can define custom policies to control the behavior of the system or the package itself.

#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

# See tmpfiles.d(5) for details
# Clear tmp directories separately, to make them easier to override
v /tmp 1777 root root 10d
v /var/tmp 1777 root root 30d

...

Short of editing the default tmpfiles.d policies, no known workaround is available. The offending tmpfiles.d policy is located in /usr/lib/tmpfiles.d/tmp.conf.

IMPORTANT: Many Debian derivatives, including Ubuntu, ship default tmpdfiles.d policies that delete the entire /tmp directory regardless of which policies a package like Instana's can specify. Upon installation, the DEB, and RPM packages run a test to verify the behavior of the temporary files tmpfiles.d, running the command systemd-tmpfiles --remove. On systems with policies that are override the wanted behavior that is specified by the Instana agent packages, you might see the following warning:

WARNING: The tmpfiles.d policies of this system may cause the Instana agent to malfunction by deleting the /tmp/.instana directory; refer https://www.instana.com/docs/setup_and_manage/host_agent/on/linux#tmpfiles.d for more information.

Short of editing the default tmpfiles.d policies, no known workaround is available. The offending tmpfiles.d policy is located in /usr/lib/tmpfiles.d/tmp.conf.

Tmpwatch

Red Hat Enterprise Linux 6 and CentOS 6 include the tool tmpwatch, which is known to delete unnecessary files in the /tmp/.instana. However, the use of tmpwatch in these distributions can occasionally cause update problems with the host agent. For more information, see Temporary files.

# Deal with tmpwatch, found in RHEL 6 and CentOS 6.
# It is a cronjob, installed in /etc/cron.d/daily, with pretty much hardcoded
# configurations to wipe data the Instana agent needs.
readonly TMPWATCH_CRONJOB='/etc/cron.d/daily/tmpwatch'
if [ -f "${TMPWATCH_CRONJOB}" ]; then
  # Let's check if we already touched the file and, if not, add our excludes
  if ! grep "${TMPWATCH_CRONJOB}" Instana 2&>1 > /dev/null; then
    echo "tmpwatch discovered in '${TMPWATCH_CRONJOB}': adding the exclude flags for Instana"
    # Edit in place with `sed`
    sed -i '/^flags=/a flags="${flags} -x /tmp/instana -x /tmp/.javapid* # Added by Instana"' "${TMPWATCH_CRONJOB}"
  fi
fi

Troubleshooting agent deployment

If installing the agent is not successful at first, you can check log messages and troubleshooting tips. If this troubleshooting section does not answer the questions you have, contact the IBM Instana support team with information about your experience, so that we can help you and update our documentation accordingly.

For troubleshooting information that is general to all host agents, see Managing host agents / Troubleshooting.

Footnotes:


  1. Red Hat Enterprise Linux 6 and CentOS 6 include the tool tmpwatch, which is known to delete necessary files in the /tmp/.instana. However, the use of tmpwatch in these distributions can occasionally cause update problems with the host agent. For more information, see Temporary files. ↩︎

  2. SUSE Linux Enterprise Server (SLES) 11 is not supported due to outdated OpenSSL binaries. ↩︎

  3. RHEL ELS 6 and 7 rpm package installer does not support SHA-256 , resulting in non-compliance with FIPS standards. ↩︎ ↩︎

  4. RHEL 8 rpm package installer is compliant with FIPS standards. ↩︎

  5. Instana host agent supports the s390x platform with Red Hat, Suse, and Ubuntu versions that are previously listed and are also supported on IBM Z. ↩︎

  6. Support for Kylin Linux is limited to Mainland China. ↩︎