Migrating Instana from Single-Host to Multi-Host

Steps

Due to performance reasons, you might need to migrate your current single-host Instana backend to a multi-host installation in which the Clickhouse database is located in a dedicated server. This document covers only the Instana backend installed on Docker because the Kubernetes version is already scalable by definition.

Here are the steps that we took and which allowed us to have a working multi-host environment starting from a single-host installation:

1. Optional: Create a server on which you install Instana single-host OnPrem. Let's call it the main server. In a real scenario, this main server should exist.
2. Create a server for our Clickhouse database.
3. On this new server for Clickhouse, create the necessary directories for data, traces, and logs (exact name as in the main server) for Clickhouse and Zookeper databases.
4. Now let's install Docker on the new Clickhouse server.
5. Once Docker is installed, expose the Docker Daemon port by using these steps:
   1. Navigate to /lib/systemd/system in your terminal and open docker.service file: vi /lib/systemd/system/docker.service.
   2. Find the line, which starts with ExecStart and add -H=tcp://0.0.0.0:2376 to make it look like: ExecStart=/usr/bin/dockerd -H=fd:// -H=tcp://0.0.0.0:2376.
   3. Save the Modified File.
   4. Reload the docker daemon: systemctl daemon-reload.
   5. Restart the container: sudo service docker restart.
   6. Test if it is working by using this command: curl http://localhost:2376/images/json, if everything is fine returns a JSON.
   7. To test remotely, use the PC name or IP address of Docker Host.
6. Run instana init on the Clickhouse server by using type: dual instana-clickhouse.
7. Once is done, stop Instana on the Clickhouse server by using the command: instana stop.
8. Also, stop Instana on the *main server*.
9. Backup (create tar balls) traces (Clickhouse) and data (Zookeeper + /var/log/instana/) from the main server.

1. Copy the tar files for Clickhouse and Zookeeper from main server to the Clickhouse server.
2. Extract the tar files on the same location of the original files and check that permissions and owners are the same as the original files.
3. The settings.hcl file in Clickhouse is already updated because instana init was run in the step 6.
4. Now we need to edit the settings.hcl file on main server like this:
   • Type = "dual-instana".
   • Clickhouse_bind_address = "IP of ClickHouse Box".
5. Start Clickhouse on the Clickhouse server:
   • systemctl start docker (if it was stopped).
   • Run docker ps to confirm that Zookeper and Clickhouse containers come up healthy.
6. Start Instana on the main server:
   • systemctl start docker(if it was stopped).
   • Or run instana start and start all Instana containers with the OLD settings.hcl.
7. Run instana update on the main server:
   • Zookeeper container started.
   • Clickhouse container is not start as expected because the one on the dedicated server is used.
8. Finally, access the UI and confirm you have both data from before the migration and current.

Links

• Installing Instana backend on Docker
• How to enable docker remote API on docker host?