How To

Summary

To ensure business continuity, system administrators need to do maintenance in daily work, such as doing an upgrade when new features are added. In addition, they need a robust, tested disaster recovery plan when an unexpected outage occurs. Backup and disaster recovery are used to achieve this goal.

Steps

Introduction

Backup is the process of making an extra copy (or multiple copies) of data. You back up data to protect it. If you encounter an accidental deletion, database corruption, or a problem with a software upgrade, you might need to restore backup data

Disaster recovery, on the other hand, refers to the plan and processes for quickly reestablishing access to applications, data, and IT resources after an outage. This plan might involve switching over to a redundant set of servers and Storage Systems until your primary data center is functional again.

This blog mainly focuses on introducing the upgrade best practices of Instana backend (on-premises) during daily maintenance, including backup and upgrade steps.

Instana environment prerequisites

Ensure that you have a running on-premises Instana backend.

Upgrade process

Backup data and configurations
Download images
Import images
Upgrade Instana backend
If the upgrade fails, restore Instana.

Backup data and configurations

Before an upgrade, you need to back up data and configurations to avoid the important data going missing when if the upgrade fails. Follow these steps:

1. If the Instana backend is running on VM, you can take a snapshot of the VM to back up both the data and the configuration.

2. Back up data, which means copying files.

Note: You need to stop Instana by running the command instana stop before you copy the Instana data files.

See the following example procedure:

            instana stop
            tar -cf ./data_backup/data.tar /mnt/data/
            tar -cf ./data_backup/metric.tar /mnt/metric/
            tar -cf ./data_backup/traces.tar /mnt/traces/
            cp settings.hcl ./data_backup/
            instana start

For more information, refer to the configuration guide here.

3. Back up configurations by using a tool that can back up self-defined configurations such as events, dashboards, etc. For more information, refer to the guide here.

a. Install the tools by running the following commands:

            wget  https://nodejs.org/dist/v14.17.1/node-v14.17.1-linux-x64.tar.xz
            scp -r root@9.46.72.72:/root/images/instana-move  .
            ln -s /root/images/node-v14.17.1-linux-x64/bin/node /usr/bin/
            which node
            ln -s /root/images/node-v14.17.1-linux-x64/bin/npm /usr/bin/
            npm -v

b. Execute the tool to back up the configurations by running the following commands:

            /root/ServerBackup/instana-move
            npm install
            npm run build
            node . export all ~/ServerBackup

c. Use the tool to back up configurations by running the following commands: Note: "export" is used for backup, "import" is used for restore, "export" is used to export application configurations

            $ node . export application {writable local directory path for exported files}

import application configurations

            $ node . import application {writable local directory path for exported files}

delete application configurations exported

            $ node . delete application {local directory path of json app files to delete}

export website and website alert configurations

            $ node . export website {writable local directory path for exported files}

import website and website alert configurations

            $ node . import website { local directory path of json files to import}

delete website and website alert configurations exported

            $ node . delete website {local directory path of json files to delete}

Download images

Find a box that can connect to the Instana website to download images.

1. To download new package images, you need to confirm the exact upgrade version first.

Use the command yum --showduplicates list instana-console to get the target release version.

   # yum --showduplicates list instana-console
   Updating Subscription Management repositories.
   Red Hat Enterprise Linux 8 for x86_64 - BaseOS (RPMs)                                                                                               11 kB/s | 2.4 kB     00:00
   Red Hat Enterprise Linux 8 for x86_64 - AppStream (RPMs)                                                                                            15 kB/s | 2.8 kB     00:00
   Installed Packages
   instana-console.x86_64                                                                   227-1                                                                     @instana-product
   Available Packages
   instana-console.x86_64                                                                   170-0                                                                     instana-product
   instana-console.x86_64                                                                   170-1                                                                     instana-product
   instana-console.x86_64                                                                   170-2                                                                     instana-product
   ..............................................  
   instana-console.x86_64                                                                   225-4                                                                     instana-product
   instana-console.x86_64                                                                   227-0                                                                     instana-product
   instana-console.x86_64                                                                   227-1                                                                     instana-product
   instana-console.x86_64                                                                   227-2                                                                     instana-product
   instana-console.x86_64                                                                   227-3                                                                     instana-product
   instana-console.x86_64                                                                   227-4                                                                     instana-product
   instana-console.x86_64                                                                   227-5                                                                     instana-product
   instana-console.x86_64                                                                   229-0                                                                     instana-product
   instana-console.x86_64                                                                   229-1                                                                     instana-product
   instana-console.x86_64                                                                   229-2                                                                     instana-product
   instana-console.x86_64                                                                   229-3                                                                     instana-product
   instana-console.x86_64                                                                   231-0                                                                     instana-product
   instana-console.x86_64                                                                   231-1                                                                     instana-product
   instana-console.x86_64                                                                   233-0                                                                     instana-product

2. Update the Instana console to a specified version.

Confirm the specified version as "231-1". Pay attention to the small version "-1", it is needed to specify the version. And then upgrade the Instana console to this specified version with the following command:

   # yum update instana-console-231-1 # use update if you already have lower version Instana-console
   # yum install instana-console-231-1. # use install if Instana-console is already removed

3. Download images with the download key by using the command "instana images export -k DOWNLOAD_KEY". The downloaded version will be the same as the Instana console version.

    # instana images export -k $DOWNLOAD_KEY
    Images export ✓
    File is stored at /root/LubanInstana/APM225/export-231-1.tar

Import images

Copy images to the Instana backend machine, and then import images by using the command as follows:

    # instana images import -f ./export-231-1.tar
    Images import ⣷ {"status":"Loading layer","progressDetail":{"current":6144,"total":6144},"progress":"[==================================================\u003e]  6.144kB/6.144kB","idImages import ⣾ {"status":"Loading layer","progressDetail":{"current":6144,"total":6144},"progress":"[==================================================\u003e]  6.144kB/6.144kB","idImages import ⣽ {"status":"Loading layer","progressDetail":{"current":6144,"total":6144},"progress":"[==================================================\u003e]  6.144kB/6.144kB","idImages import ✓

Check the Instana version and Instana images version.

    # instana version

    Instana self-hosted: 227-1 3.227.192-0 (e70cb126bc21e38c1256072355d6eaab2af7ef4b)


    # instana images version

    containers.instana.io/instana/release/product/butler:3.227.192-0

    containers.instana.io/instana/release/product/groundskeeper:3.227.192-0

    containers.instana.io/instana/release/product/accountant:3.227.192-0

    containers.instana.io/instana/release/product/cashier-ingest:3.227.192-0

    containers.instana.io/instana/release/product/cashier-rollup:3.227.192-0

    containers.instana.io/instana/release/product/acceptor:3.227.192-0

    containers.instana.io/instana/release/product/eum-acceptor:3.227.192-0

    containers.instana.io/instana/release/product/eum-processor:3.227.192-0

    containers.instana.io/instana/release/product/eum-health-processor:3.227.192-0

    containers.instana.io/instana/release/product/appdata-health-processor:3.227.192-0

    containers.instana.io/instana/release/product/sli-evaluator:3.227.192-0

    containers.instana.io/instana/release/product/js-stack-trace-translator:3.227.192-0

    containers.instana.io/instana/release/product/appdata-writer:3.227.192-0

    containers.instana.io/instana/release/product/appdata-reader:3.227.192-0

    containers.instana.io/instana/release/product/appdata-health-aggregator:3.227.192-0

    containers.instana.io/instana/release/product/serverless-acceptor:3.227.192-0

    containers.instana.io/instana/release/product/tag-processor:3.227.192-0

    containers.instana.io/instana/release/product/tag-reader:3.227.192-0

    containers.instana.io/instana/release/product/ui-client:3.227.192-0

    containers.instana.io/instana/release/product/filler:3.227.192-0

    containers.instana.io/instana/release/product/processor:3.227.192-0

    containers.instana.io/instana/release/product/issue-tracker:3.227.192-0

    containers.instana.io/instana/release/product/appdata-processor:3.227.192-0

    containers.instana.io/instana/release/product/appdata-legacy-converter:3.227.192-0

    containers.instana.io/instana/release/product/ui-backend:3.227.192-0

    containers.instana.io/instana/release/product/zookeeper:3.7.1_v0.36.0

    containers.instana.io/instana/release/product/clickhouse:21.3.19.1_v0.35.0

    containers.instana.io/instana/release/product/aggregator:1.1.134-release.1_v0.36.0

    containers.instana.io/instana/release/product/nginx:1.18_v0.27.0

    containers.instana.io/instana/release/product/cockroachdb:21.1.7_v0.28.0

    containers.instana.io/instana/release/product/cassandra:4.0.3_v0.39.0

    containers.instana.io/instana/release/product/elasticsearch:7.16.3_v0.13.0

    containers.instana.io/instana/release/product/kafka:3.2.0_v0.49.0

Upgrade Instana backend

After images are imported, the upgrade can be done by using the command ./instana update -f. See the following example:

# instana update -f /root/instana/Instana_APM193/docker-installer/settings.hcl

Setup host environment ✓

Handle certificates ✓

Ensure images ✓

Clean docker containers ✓

Check data directories ✓

Create configurations ✓

Migrate data stores ✓

Run components ✓

Check components ✓

Setup environment urls ✓

Run nginx ✓

Welcome to the World of Automatic Infrastructure and Application Monitoring

Note: Sometimes the upgrade process might be very slow if a new version database is migrated during the upgrade, so you need to wait until the upgrade process ends.

Restoring Instana

If the upgrade fails, a restore is needed. This restore procedure has different use cases. It can be used as a regular restore from a snapshot in an emergency case or used to restore EC2 instances inside auto-scaling groups for an automated restore after termination.

If there is a snapshot for the Instana backend VM, directly restore the VM and start Instana to see whether the Instana backend is recovered.

If there is no snapshot, follow the steps:

1. Restore Instana backend configurations by using the command "instana restore -f ./settings.hcl". You need to restore Instana on an empty instance. The restore procedure consists of the following steps:

(1) Set up the instana-console repository.

(2) Install Docker.

(3) Install instana-console in the same version as on the original instance.

(4) Import the data backups to the mount paths of the file settings.hcl.

(5) Run the command instana restore -f.

2. Restore the backup configurations with the Instana-move tool, refer to the guide here.

Some useful commands for troubleshooting

Sometimes the upgrade can’t complete successfully. Here are some common problems and commands, which can help you debug and fix the problems.

1. The upgrade needs to be done in two neighboring versions.

For example, it’s not supported to upgrade 227 to 233. In this case, you can upgrade 227 to 229, upgrade 229 to 231, and then upgrade 231 to 233.

    # instana update -f ./settings.hcl
    Setup host environment ✓
    Handle certificates ✓

    A direct update from version 227 to 233 is not supported. The latest supported version is 229

2. If the upgrade fails due to database problems, try to run the following commands several times. The problem might be fixed.

   # instana repair
   # instana restore -f ./settings.hcl

3. To remove Instana completely for reinstallation if the upgrade fails, follow these steps:

(1) Remove Instana-console. (2) Delete Instana images by running the command instana images prune.

4. If the Instana images fail to be displayed or deleted, run the command as follows:

docker login https://containers.instana.io-u="_" -p="$DOWNLOAD_KEY"

After the Docker login is done, the problem might be fixed.

5. To check whether there are network problems with the instana server, run the following command:

    curl -X GET -k -H 'Authorization: apiToken $apiToken' -i ' https://luban-instana.luban.cdl.ibm.com/api/settings/rabc/groups/'

Reference

Upgrading Instana backend on Kubernetes

Backup and restore for Instana backend on docker

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB45","label":"Automation"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSE1JP5","label":"IBM Instana Observability"},"ARM Category":[{"code":"a8m3p000000UoUVAA0","label":"Backend On-Prem"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]

Tips

Instana backend upgrade best practices (on-premises)