Restoring application logging data

Restore your backup application logging data by exporting the backup and S3 bucket names to environment variables and running a command-line script.

Procedure

Stop the ibm-tnc-orchestration-controller-manager pod by running the following commands:
```
oc get deploy
oc edit deploy ibm-tnc-orchestration-controller-manager
```
Change the value of the replicas property to zero, as shown in the following sample YAML:
```
spec:
    progressDeadlineSeconds: 600
    replicas: 0
```
Save the deployment, then wait for the ibm-tnc-orchestration-controller-manager to shut down.
Delete the Fluentd daemonset by running this command:
```
oc delete daemonset cp4na-o-fluentd
```
Wait for the Fluentd pods to shut down.
Check that the Fluentd pods shut down correctly by running this command:
```
oc get pods | grep fluentd
```
The names of the Fluentd pods begin with cp4na-o-fluentd-, for example cp4na-o-fluentd-4gkcw. Verify that no Fluentd pods are running.

Delete the log indexes from OpenSearch by running these commands:

PASSWORD=$(oc get secret ibm-opensearch-ibm-elasticsearch-cred-secret -o go-template='{{.data.elastic|base64decode}}')
OPENSEARCH_API=$(oc get route cp4na-o-opensearch -o jsonpath='{.status.ingress[0].host}')
curl -ik -XDELETE -u elastic:$PASSWORD https://$OPENSEARCH_API/cp4na-logs

Export the S3 bucket name and the logging data backup name to local environment variables by running these commands:
```
export S3_PATH=<s3_bucket_name>
export SNAPSHOT_NAME=<backup_name>
```
<s3_bucket_name> is the name of the S3 bucket that contains the backup application logging data that you want to restore. Specify a name that does not include a trailing slash, for example, s3://my-s3-bucket/backup

<backup_name> is the name of the backup that you want to restore application logging data from.
You can retrieve the backup name by viewing the output of this command, which lists all backups:
```
oc get OrchestrationBackup
```
In the output, for application logging data backups:
- The value in the BACKUP TYPE column is Opensearch.
- The backup name is shown in the NAME column.
Restriction: In active-active configurations of IBM® Cloud Pak for Network Automation, orchestration data is replicated across clusters, but application logging data is not. As a result, each application logging data backup contains only the logging data from the cluster where you created the backup.
To restore application logging data for all clusters in your active-active configuration, you must restore on each cluster from the backup that was created on that cluster.

Create a bash script that contains the following text:

require_env() {
  if [[ -z "${!1}" ]]; then
    echo "Require Env.$1... Not Found"
    exit 1
  fi
  echo Require Env.$1... OK "(${!1})"
}

# As it appears in the `orchestration` CR, for example:
# export S3_PATH=s3://cp4na-backups2/backup_test3
require_env S3_PATH
require_env SNAPSHOT_NAME

if ! grep -q '^s3://' <<< $S3_PATH
then
    echo "ERROR: S3_PATH must be in format s3://bucket/basepath"
    exit 1
fi

PASSWORD=$(oc get secret ibm-opensearch-ibm-elasticsearch-cred-secret -o go-template='{{.data.elastic|base64decode}}')
OPENSEARCH_API=$(oc get route cp4na-o-opensearch -o jsonpath='{.status.ingress[0].host}')

S3_BUCKET=$(cut -d/ -f 3 <<< $S3_PATH)
S3_BASEPATH=$(cut -d/ -f 4- <<< $S3_PATH)

repo=$(cat <<EOF
{
    "type": "s3",
    "settings": {
        "bucket": "$S3_BUCKET",
        "base_path": "$S3_BASEPATH/opensearch"
    }
}
EOF
)

function kurl() {
    method=$1
    path=$2
    data=$3
    res=""

    echo ""
    echo "$method $path"
    if [ -n "$data" ]; then
        res=$(curl -sik -X$method -H "Content-Type: application/json" -u elastic:$PASSWORD "https://$OPENSEARCH_API$path" -d "$repo")
    else
        res=$(curl -sik -X$method $headers -u elastic:$PASSWORD "https://$OPENSEARCH_API$path")
    fi

    status=$(head -n1 <<< "$res" | awk '{print $2}')
    echo "$res"

    if [ "$status" -ne "200" ]; then
        echo "HTTP ERROR: $status"
        exit 1
    fi
}

kurl POST "/_snapshot/restore-repo" "$repo"
kurl GET "/_snapshot/restore-repo"
kurl GET "/_snapshot/restore-repo/$SNAPSHOT_NAME"
kurl POST "/_snapshot/restore-repo/$SNAPSHOT_NAME/_restore"

Restore your backup application logging data by running the script. For example, if the name of your script is restore_logging_data.sh, run this command:
```
sh restore_logging_data.sh
```

Verify that the script restored the backup data successfully. The following example shows the type of output that is displayed when the script is successful:

Require Env.S3_PATH... OK (s3://cp4na-backups2/backup_test6)
Require Env.SNAPSHOT_NAME... OK (cp4na-fullbackup-opensearch-1)

POST /_snapshot/restore-repo
HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
content-length: 21

{"acknowledged":true}

GET /_snapshot/restore-repo
HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
content-length: 107

{"restore-repo":{"type":"s3","settings":{"bucket":"cp4na-backups2","base_path":"backup_test6/opensearch"}}}

GET /_snapshot/restore-repo/cp4na-fullbackup-opensearch-1
HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
content-length: 459

{"snapshots":[{"snapshot":"cp4na-fullbackup-opensearch-1","uuid":"lTeVBfUGQh6MSVz-3Qp4ew","version_id":136227927,"version":"2.1.1","indices":["cp4na-logs"],"data_streams":[],"include_global_state":true,"state":"SUCCESS","start_time":"2023-05-05T07:09:16.376Z","start_time_in_millis":1683270556376,"end_time":"2023-05-05T07:09:24.009Z","end_time_in_millis":1683270564009,"duration_in_millis":7633,"failures":[],"shards":{"total":5,"failed":0,"successful":5}}]}

POST /_snapshot/restore-repo/cp4na-fullbackup-opensearch-1/_restore
HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
content-length: 17

{"accepted":true}

If you see similar output, the script completed successfully.

Restart the ibm-tnc-orchestration-controller-manager pod by running the following commands:
```
oc get deploy
oc edit deploy ibm-tnc-orchestration-controller-manager
```
Change the value of the replicas property to 1, as shown in the following sample YAML:
```
spec:
    progressDeadlineSeconds: 600
    replicas: 1
```
Save the deployment, then wait for the ibm-tnc-orchestration-controller-manager to restart and go into running state.
Verify that the status of your Cloud Pak instance is Ready by running this command:
```
oc get orchestration -n <cp4na_namespace>
```
<cp4na_namespace> is the namespace where the Cloud Pak is installed.
Verify that the Fluentd pods restarted successfully by running this command:
```
oc get pods | grep fluentd
```
If the cp4na-o-fluentd- pods are shown in a Running state, the pods restarted successfully.

Optional: Verify that the indexes that contain the backup application logging data are successfully restored by running these commands:

PASSWORD=$(oc get secret ibm-opensearch-ibm-elasticsearch-cred-secret -o go-template='{{.data.elastic|base64decode}}')
OPENSEARCH_API=$(oc get route cp4na-o-opensearch -o jsonpath='{.status.ingress[0].host}')
curl -ik -XGET -u elastic:$PASSWORD https://$OPENSEARCH_API/_cat/indices

The following example shows the type of output that is generated from these commands when the log indexes are successfully restored:

HTTP/1.1 200 OK
content-type: text/plain; charset=UTF-8
content-length: 68

green open cp4na-logs amRSXd6iQ1i7EieGkodPuw 5 1 2175 0 1.7mb 1.3mb

If you see similar output, your log indexes are successfully restored. If not, you might need to restore application logging data from a different backup object.