Log forwarding to AWS CloudWatch

When you submit the Data Gate on Cloud CloudFormation templates, the deployment application logs from the Red Hat® OpenShift® (ROSA) cluster are forwarded automatically to AWS CloudWatch. The information here provides a description of the enablement process, as well as instructions on how to disable log forwarding, re-enable it, and remove the log forwarding capability completely.

Automatic setup

When the deployment templates are submitted, application logs are written to standard out by each Data Gate on Cloud server pod. The ROSA Logging Operator then forwards the application logs from each pod and container to CloudWatch as soon as the pods are started.

In addition, the setup logs for both templates nested inside the first template, datagate.template and ami.template, are forwarded to CloudWatch as well. These logs are streamed in real time during the setup on the boot node. The streaming stops when the setup finishes and the boot nodes are terminated.

On AWS CloudWatch, the application logs for a particular namespace are grouped into a log group of its own, and the logs from each pod within that namespace are placed into a log stream of its own. On AWS CloudWatch, the log groups are named <rosa-cluster-name>-logs.<namespace name>, where:

<rosa-cluster-name>

Is the name of the ROSA cluster.

<namespace name>

Is the name of the namespace.
Likewise, the boot node logs for the first nested template (datagate.template), are grouped into another separate log group and are streamed into another log stream of its own. The log group name follows the pattern <rosa cluster name>-<namespace name>-installation-<bootnode instance id>, where:

<rosa-cluster-name>

Is the name of the ROSA cluster.

<namespace name>

Is the name of the namespace.

<boodnode instance id>

Is the name of the log stream.
The procedure for the second nested template (ami.template) is similar. The log group name follows the pattern <rosa cluster name>-installation-<bootnode instance id>, where:

<rosa-cluster-name>

Is the name of the ROSA cluster.

<boodnode instance id>

Is the name of the log stream.
To enable and set off this process, the following steps are executed:
1. The ROSA Logging Operator is installed.
2. An AWS IAM policy for CloudWatch access is created. It is called <rosa-cluster-name>-<region>-CloudWatchPolicy, where:
  
  <rosa-cluster-name>
  
  Is the name of the ROSA cluster.
  
  <region>
  
  Is the name of the AWS region that the ROSA cluster is deployed to, for example us-east-1.
3. An AWS IAM role <rosa-cluster-name>-<region>-CloudWatchRole is created. The AWS IAM role is created for each cluster that is deployed, and attached to the corresponding IAM policy <rosa-cluster-name>-<region>-CloudWatchPolicy. For the meaning of the variables in these names, see step 2.
4. An OpenShift secret called cw-logforwarding-creds is created in the OpenShift logging namespace. This secret references the <rosa-cluster-name>-<region>-CloudWatchRole IAM role.
5. ClusterLogging and ClusterLogForwarder instances are created as part of the ROSA Logging Operator. The ClusterLogForwarder instance also references the cw-logforwarding-creds secret and configures the log forwarding.
Note: The information in steps 1 through 5 was partly extracted from the following AWS web site: Logging using Amazon CloudWatch

Disabling log forwarding to CloudWatch

To disable the log forwarding process previously described:

Log in to your OpenShift system by using the oc login command.
Delete the ClusterLogForwarder instance from the openshift-logging namespace by running the following command:
```
oc delete ClusterLogForwarder/instance -n openshift-logging
```
Likewise, delete the ClusterLogging instance from the openshift-logging namespace by running the following command:
```
oc delete ClusterLogging/instance -n openshift-logging
```

Attention: If you want to re-enable log forwarding later on, do not delete any of the following objects or components:

ROSA Logging Operator
cw-logforwarding-creds
<rosa-cluster-name>-<region>-CloudWatchPolicy
<rosa-cluster-name>-<region>-CloudWatchRole

For the meaning of the variables in these names, see step 2.

Re-enabling log forwarding to CloudWatch

To re-enable the log forwarding process:

Log in to your OpenShift system by using the oc login command.
Provided that the objects and components listed in this note still exist, re-create the ClusterLogForwarder and ClusterLogging instances. For this purpose, complete the following steps to create two YAML files:
- oc create -f ClusterLogForwarder.yaml
- oc create -f ClusterLogging.yaml

Insert the following content in the ClusterLogForwarder.yaml file:

apiVersion: "logging.openshift.io/v1"
kind: ClusterLogForwarder
metadata:
  name: instance 
  namespace: openshift-logging 
spec:
  outputs:
   - name: data-gate-forwarded-logs
     type: cloudwatch 
     cloudwatch:
       groupBy: namespaceName 
       groupPrefix: <namespace name>-logs
       region: <aws region>
     secret:
        name: cw-logforwarding-creds
  pipelines:
    - name: rosa-forwarded-logs-pipeline 
      inputRefs: 
        - application
      outputRefs:
        - data-gate-forwarded-logs

Note: Do not forget to replace the following variables in the yaml file with your actual values:

<namespace name>: Is the name of the namespace.
<aws region>: Is the name of the AWS region that the ROSA cluster is deployed to, for example us-east-1.

Save the file.

Insert the following content in the ClusterLogging.yaml file:

apiVersion: logging.openshift.io/v1
kind: ClusterLogging
metadata:
  name: instance
  namespace: openshift-logging
spec:
  collection:
    logs:
       type: fluentd
  forwarder:
    fluentd: {}
  managementState: Managed

Save the file.

Removing log forwarding

If you remove all objects related to the log forwarding capability, the steps in Re-enabling log forwarding to CloudWatch do not work. In case you want to restore log forwarding after a complete removal, you must submit the CloudFormation templates for the Data Gate on Cloud deployment again. To fully remove the log forwarding:

Log in to your OpenShift system by using the oc login command.
Delete the ClusterLogForwarder instance from the openshift-logging namespace by running the following command:
```
oc delete ClusterLogForwarder/instance -n openshift-logging
```
Likewise, delete the ClusterLogging instance from the openshift-logging namespace by running the following command:
```
oc delete ClusterLogging/instance -n openshift-logging
```
Uninstall the ROSA Logging Operator in the openshift-logging namespace by following these steps:
1. Stop the logging subscription:
```
oc delete subscription cluster-logging -n openshift-logging
```
2. Find the correct version of the logging service:
```
oc get clusterserviceversion -n openshift-logging
```
  The version is output on the screen.
3. Delete the version of the logging service identified in the previous step. See the following example:
```
oc delete clusterserviceversion cluster-logging.v5.7.7 -n openshift-logging
```
  In this example, the version of the logging service is cluster-logging.v5.7.7.

Delete the OpenShift secret named cw-logforwarding-creds:

oc delete secret/cw-logforwarding-creds -n openshift-logging

From the IAM module of the AWS console, delete the IAM role named <rosa-cluster-name>-<region>-CloudWatchRole.
From the same module of the AWS console, also delete the IAM policy named <rosa-cluster-name>-<region>-CloudWatchPolicy.

For the meaning of the variables in these names, see step 2.