Upgrading Instana backend on Kubernetes/OpenShift

To upgrade the operator-based Instana setup, follow these steps:

Prerequisites

Make sure that the following prerequisites are met before you upgrade Instana:

  • The cluster has adequate capacity. If the cluster is nearing its request capacity, add an extra node to avoid the pods stuck in Pending status.
  • The data store versions are compatible with the Instana version that you want to upgrade to. For data store versions, see Installing third-party data store operators.

Upgrade procedure

The upgrade policy provides that at most one release can be skipped, that is, it's possible to upgrade from release 247-x to release 249-x or release 251-x, but no further. If the policy does not allow you to do the required upgrade in one go, then you need to carry out more than one upgrade step.

  1. Install the target version of Instana kubectl plug-in. For more information, see Installing the Instana kubectl plug-in.

  2. Upgrade the operator by using either of the following methods:

    Notes:

    • Ensure that you apply or generate new YAML manifests. Don't update the image version in the existing YAML manifests directly. If you update the image version in the existing YAML manifests, then you might miss the CustomResourceDefinition (CRD) updates or other changes that can lead to unpredictable errors.

    • To upgrade to a specific version, you might require to do some extra actions. Refer to the Upgrade notes section. When you skip a release, ensure that you consider the upgrade notes (including upgrade notes for the skipped version and the target version).

  3. Verify the Instana backend upgrade by doing the following steps:

    • Check whether the Instana kubectl plug-in version is updated by running the kubectl instana -version command.
    • Check whether the Instana backend version is updated by running the kubectl get core -n instana-core and kubectl get units -n instana-units commands.

Upgrade notes

Upgrading to release 277

If you are upgrading from release 273 or earlier, follow the steps in Upgrading to release 275. No special steps are needed to upgrade from release 275 onwards.

Upgrading to release 275

Starting from Instana release 275, a new table is introduced to the ClickHouse logs database that uses the ClickHouse TTL mechanism to automatically move data to cold storage over time and then expire it as needed.

The ClickHouse datastore configuration needs to be updated for release-275 migrations to be applied.

Proceed with the following steps:

  1. Set your Instana backend into maintenance mode by updating the Core spec:
kubectl patch core <core_name> -n <core_namespace> --type='merge' -p='{"spec": {"operationMode": "maintenance"}}'

  1. Add the following configurations to your ClickHouseInstallation resource:

    • Define a new cold_disk in <disks>, create a new policy called logs_policy_v4 and reference the cold_disk in the new policy:

      config.d/storage.xml: |
   <clickhouse>
     <storage_configuration>
       <disks>
         <default/>
         <cold_disk>
           <path>/var/lib/clickhouse-cold/</path>
         </cold_disk>
       </disks>
       <policies>
         <logs_policy>
           <volumes>
             <data>
               <disk>default</disk>
             </data>
             <cold>
               <disk>cold_disk</disk>
             </cold>
           </volumes>
         </logs_policy>
         <logs_policy_v4>
           <volumes>
             <tier1>
               <disk>default</disk>
             </tier1>
             <tier2>
               <disk>cold_disk</disk>
             </tier2>
           </volumes>
         </logs_policy_v4>
       </policies>
     </storage_configuration>
  </clickhouse>

    • Define a new volumeClaimTemplate which will let ClickHouse create an additional PVC for the cold_disk:

      volumeClaimTemplates:
  ...
  ...
  - name: instana-clickhouse-data-cold-volume
    spec:
      accessModes:
         - ReadWriteOnce
      resources:
         requests:
         storage: 100Gi
      # storageClassName: <lower_or_cheaper_tier_storageclassname>

    • Finally, mount the new volume in the ClickHouse Pod by defining volumeMounts in the podTemplates:

      containers:
  - name: instana-clickhouse
    ...
    ...
    volumeMounts:
      - mountPath: /var/lib/clickhouse-cold/
        name: instana-clickhouse-data-cold-volume

      For the full ClickHouseInstallation resource YAML, see:

    • Before proceeding with the upgrade to release-275, make sure ClickHouse is showing a status of Completed after applying the new changes, by running the following command:

      % kubectl get clickhouseinstallation -n instana-clickhouse

NAME      CLUSTERS   HOSTS   STATUS      AGE
instana   1          2       Completed   4d1h

  2. Set your Instana backend back to normal mode by updating the Core spec:

kubectl patch core <core_name> -n <core_namespace> --type='merge' -p='{"spec": {"operationMode": "normal"}}'
  1. Upgrade to Instana 275 by following the Upgrade procedure.

Upgrading to release 273

The upgrade does not require any special steps.