Git-based configuration management

Self-signed certificates

If you host the Git repository yourself, you can use a self-signed certificate. For the agent to connect securely to a Git repository, you must import the self-signed certificate into the Java truststore. For more information, see Self-signed certificates.

Authentication with .netrc file

Access to a Git repository can be authorized by using a .netrc file to provide the password or access token. .netrc is widely used and details about it can be found on the .netrc man page.

You can configure the .netrc file in the home directory of the user (for example, root) where the Instana host agent is running. See the following sample .netrc file for the github.com repository:

machine github.com
login oauth2
password <access_token>

Replace <access_token> with an access token.

Git repository structure

In the Git repository, you need to have same structure of directories as in <instana-agent-dir>/etc. For example, if you want to get data from the Git file named configuration-mysql.yaml, this file needs to be placed in the Git directory named instana (that is, the file instana/configuration-mysql.yaml). Otherwise, when the file is obtained from the Git repository, the file is placed in the wrong location on the server and is not used by the Instana host agent.

The following figure shows the example structure of a Git repository:

repository-root/
├── instana/
│   ├── configuration-mysql.yaml
│   └── com.instana.agent.main.config.UpdateManager.cfg
└── org.ops4j.pax.url.mvn.cfg

With systemd

By using systemd, the easiest way to specify the additional parameters is using a drop-in file. To view the path to the drop-in directory, run the following command:

systemctl status instana-agent

The files in this directory must have the file extension .conf, otherwise the files are ignored.

1. Create a new file in the /etc/systemd/system/instana-agent.service.d/ directory with the extension .conf, such as git-configuration-environments.conf.
2. Add the following content to the file:

   [Service]
   Environment=INSTANA_GIT_REMOTE_BRANCH=<branch>
   Environment=INSTANA_GIT_REMOTE_REPOSITORY=<repository_url>
   Environment=INSTANA_GIT_REMOTE_USERNAME=<username>
   Environment=INSTANA_GIT_REMOTE_PASSWORD=<access_token>
   ```   **Notes:**
    - Replace *&lt;branch&gt;* with the name of the remote branch that the Instana host agent connects to.
    - Replace *&lt;repository_url&gt;* with the URL of the remote Git repository; for example, `https://github.com/<your_account>/<your_repo>.git`.
    - Replace *&lt;username&gt;* with a username or access token (for basic authentication). If you are using a `.netrc` file or a public repository, you can remove the whole line `Environment=INSTANA_GIT_REMOTE_USERNAME=<username>`.
    - Replace *&lt;access_token&gt;* with an access token. If you are using an access token as username (basic authentication), a `.netrc` file, or a public repository, you can remove the whole line `Environment=INSTANA_GIT_REMOTE_PASSWORD=<access_token>`.
   
   

After you add or change a drop-in file, complete the following steps:

1. Reload the drop-in files, by running the systemctl daemon-reload command.
2. Restart the Instana host agent, by running the command systemctl restart instana-agent.

With environment variables

If you have set the following environment variables, then when a host agent is started it checks for the presence of the local Git repository in the <instana-agent-dir>/etc/ folder. If the local Git repository is not present, the host agent creates a local repository that is connected to the remote repository.

If the Instana agent is running directly on a host, the following environment variables need to be set on the host. If the host agent is running in a containerized environment, set the variables on the container-level.

If you want to use basic authentication with an access token instead of a username and password, set the access token in the parameter INSTANA_GIT_REMOTE_USERNAME. This ensures that basic authentication is configured correctly without a username.

With the one-liner command

Refer to the -g, -b, -u and -p options in the OneLiner documentation, which map to the INSTANA_GIT_REMOTE_REPOSITORY, INSTANA_GIT_REMOTE_BRANCH, INSTANA_GIT_REMOTE_USERNAME and INSTANA_GIT_REMOTE_PASSWORD environment variables for the environment method.

The following table shows the mapping between the environment variables and the parameters of the one-liner command:

With Docker images

The steps are fundamentally the same as the environment method, by adding the INSTANA_GIT_REMOTE_REPOSITORY, INSTANA_GIT_REMOTE_BRANCH, INSTANA_GIT_REMOTE_USERNAME and INSTANA_GIT_REMOTE_PASSWORD environment variables to the Docker container.

For example, the following extract gives the parameters that are needed to start the host agent in a container and the required Git parameters to enable Git-based configuration management:

# Please enter proper values for all --env arguments.
docker run \
   --detach \
   --name instana-agent \
   --volume /var/run:/var/run \
   --volume /run:/run \
   --volume /dev:/dev:ro \
   --volume /sys:/sys:ro \
   --volume /var/log:/var/log:ro \
   --privileged \
   --net=host \
   --pid=host \
   --env="INSTANA_AGENT_ENDPOINT=" \
   --env="INSTANA_AGENT_ENDPOINT_PORT=" \
   --env="INSTANA_AGENT_KEY=" \
   --env="INSTANA_DOWNLOAD_KEY=" \
   --env="INSTANA_AGENT_ZONE=" \
   --env="INSTANA_GIT_REMOTE_BRANCH=" \
   --env="INSTANA_GIT_REMOTE_PASSWORD=" \
   --env="INSTANA_GIT_REMOTE_REPOSITORY=" \
   --env="INSTANA_GIT_REMOTE_USERNAME=" \
   icr.io/instana/agent

With the agent management dashboard

The Git-based configuration management can be set up in the Configuration management section on the Agent management dashboard.

Important: You must add the .git suffix to the end of the INSTANA_GIT_REMOTE_REPOSITORY address, such as https://github.com/<your_account>/<your_repo>.git. In addition, to configure basic authentication for access to the Git repository, you can use a .netrc file. For more information, see Authentication with .netrc file.

With the API

The Git-based configuration management can be initialized and configured by using the API as described in the Host Agent section of the Web REST API documentation.

Manual setup

A manual setup consists of initializing a Git repository in the <instana-agent-dir>/etc/ folder and adding a remote named configuration.

The host agent pulls the local main branch by using the configured tracking branch from the configuration remote. There are multiple ways to set up the Git repository, and you can freely choose the one that fits best your setup.

For example, the following setup allows you to have the host agent pull configurations from the main branch of the configuration remote:

etc> git init
etc> git remote add -m main -t main configuration <git-repository-url>
etc> git fetch configuration
etc> git checkout -f -b main --track configuration/main

The following example allows you to pull the configurations from the my-configurations branch in the configuration remote:

etc> git init
etc> git remote add -m main -t my-configurations configuration <git-repository-url>
etc> git fetch configuration
etc> git checkout -f -b main --track configuration/my-configurations

Possible <git-repository-url> formats are described in the official documentation on Git URLs.

Triggering updates by using Git Hooks

Git Hooks are programs that you can place in the .git/hooks directory of the repository, and they trigger at different times in the lifecycle of your Git repository. Git offers many different hooks, and the post-update hook is the best suited for integration with the Git-based configuration management of host agents.

The GitOps-Demo repository provides a sample post-update Git hook that triggers the update of host agents whenever a branch in the repository is updated.

You see that the expected output of the script sent back to your Git client after you trigger a push:

$ gitops-test# git push origin HEAD:main
[detached HEAD 9632c09] origin
 1 file changed, 1 insertion(+), 1 deletion(-)
Enumerating objects: 7, done.
Counting objects: 100% (7/7), done.
Delta compression using up to 12 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (4/4), 376 bytes | 376.00 KiB/s, done.
Total 4 (delta 1), reused 0 (delta 0)
remote: GitOps update: Triggering the configuration update of agents matching the following query: 'entity.zone:"test"' ... OK
   699ada9..9632c09  HEAD -> main

Triggering updates by using GitHub Update Action

The Instana GitHub action for GitOps allows you to straightforwardly trigger the remote of the host agents by using Instana's Web API.

The GitOps-Demo repository shows how to set up a repository on GitHub to manage your host agent configurations and how to automatically trigger the updates by using the Instana GitHub action for GitOps.

Private GitHub repository

GitHub supports HTTPS authentication by using tokens, and this can be used by the host agent as described in this section.

To use a GitHub repository, you first must create a personal access token. Details on how to do that are documented on the Creating a personal access token GitHub documentation page. The token needs only read permissions to access the repository.

GitHub expects the token to be provided as HTTPS basic authentication username with an empty password. Therefore, the host agent can be configured to use HTTPS basic authentication for GitHub repositories by setting the following environment variables set:

INSTANA_GIT_REMOTE_REPOSITORY='https://github.com/<user/organization>/<repository>.git
INSTANA_GIT_REMOTE_BRANCH='<branch>'
INSTANA_GIT_REMOTE_USERNAME='<token>'
INSTANA_GIT_REMOTE_PASSWORD=''

Notice that the same principle of empty password and personal token as username works also with the other means of configuring the host agent for Git-based configuration management. For an overview of the possible configuration approaches, refer to the Setup section.

Controlling agent updates with Git

You can use multiple methods to control the version updates to the host agent that is deployed into your production environments. If you use static agents, you must update the host agent manually as you require. If you use dynamic agents, you can configure the updates to set update intervals, pinned versions, and so on. For more information, see Configuring updates of dynamic host agents. If no configuration is specified, updates are automatically applied to the dynamic agent every day. Additionally, if you need stricter control over the version updates to the host agent, you can use a combination of dynamic agent version pinning with Git-based configuration management. If you have a test environment, you can test the agent deployments against a production-like scenario before you apply the update into production.

For an example on how to implement the workflow in which agent version pinning is combined with Git-based configuration management, see the instana/gitops-demo repository.

This example repository uses GitHub as a repository hosting service, Mend Renovate for dependency update automation, and GitHub Actions for sending requests to the Instana backend. You can replace all of these services with the services of your choice. Make sure you meet the following requirements to implement a Git-based update flow:

• Git as source code repository format

• automation to send an HTTP request to the Instana backend on configuration changes

  You can use any automation of your choice here. However, make sure that the version in instana/com.instana.agent.bootstrap.AgentBootstrap.cfg is updated and you use different branches for different environments.

The repository contains the main branch that tracks the current state of the agent production deployments and the test branch that represents the agent test environment.

Mend Renovate is configured to watch the instana/agent-updates repository for new tags and compare the tags with the version in the instana/com.instana.agent.bootstrap.AgentBootstrap.cfg configuration file. When a new tag is detected in the instana/agent-updates repository, Mend Renovate creates a branch based on the test branch and raises a GitHub pull request to add the changes to the configuration file on the test branch. You can use a variety of options provided by Renovate, for example, to schedule updates only once a week, or to assign certain GitHub handles to the pull requests for manual approval. When a pull request is merged to the test branch, GitHub Actions runs a workflow to notify the configured Instana backend. Then, the Instana backend sends out an agent reboot command to all agents that match the particular zone and tag constraints of the test environment.

Additionally, a second workflow copies the current instana/com.instana.agent.bootstrap.AgentBootstrap.cfg file from the test branch to a new feature branch and opens a pull request against the main branch, which ensures the promotion of the agent version from the test environment to production.

After the changes are validated in a production-like environment, you can merge the changes to the main branch. When you merge, GitHub Actions runs a workflow to send an update request to the Instana Backend for all agents that are connected to production.

See the following sample configuration:

# This field specifies which version set of components should be used by the agent;
# its value must be a valid tag from https://github.com/instana/agent-updates/tags
# version = <tag>
# renovate: datasource=github-tags depName=instana/agent-updates versioning=loose
version = 2024.09.02.0634

# This field controls which set of components is used by the agent at runtime.
productName = ${env:INSTANA_PRODUCT_NAME}

origin = package dynamic

Whenever the Instana backend receives an agent configuration update, it restarts all matching agents. Depending on the number of agents and their distribution on hardware, this restart might cause spikes in resource consumption. If load must be delayed over time, you can extend the example with additional branches that represent batches of agents to update. You can also adjust the the repository automation to wait for only the manual approval once and to push updates to other branches automatically after a pre-defined delay to avoid increased manual management effort.