Setting up Instana to send alerts to WebSphere Automation

WebSphere Automation requires Instana for automatic memory leak detection and analysis on any registered managed servers. Without Instana, manual memory leak analysis can be manually invoked from the WebSphere Automation UI for any registered managed servers.

Before you begin

Confirm that you have:

An installation of Instana. For more information, see Installing and configuring self-hosted Instana backend (on-premises).
Access to the Instana administrative console with sufficient privileges to change the configuration.
Instana agents installed on the servers that you want to monitor. For more information, see Installing Instana agents or the Installing Instana Agents page in the Instana administrative console.
Servers that are registered with WebSphere Automation for security monitoring. For more information, see Setting up security monitoring.

Procedure

Create a custom payload.
To create a custom payload by using the Instana administrative console,
1. On the Team Settings page, under Events & Alerts, click Custom Payload.
2. Click Add Row.
3. In the Key field, type PID. Do not use any other value for the key.
4. Set the Value type to Dynamic.
5. In the Value field, click Select Tag and select Internal - Process ID.
6. Click Save.
To create a custom payload by using the Instana API,
1. Create an Instana API key with 'Category' permission: Events
2. Create a createCustomPayLoad.json file.
```
{
  "fields": [
    {
      "type": "dynamic",
      "key": "PID",
      "value": {
        "tagName": "process.id"
      }
    }
  ]
}
```
3. Create the custom payload. Replace <API_KEY> with the value of the API key from the first step.
```
curl -Lk --request PUT --url https://example.com//api/events/settings/custom-payload-configurations --header 'authorization: apiToken <API_KEY>' --header 'content-type: application/json' --data @createCustomPayLoad.json
```
For more information, see Create / Update Custom Payload Configuration in the Instana documentation .

Create an alert channel.

To create an alert channel by using the Instana administrative console,

On the Team Settings page, under Events & Alerts, click Alert Channels.
Click Add Alert Channel and select Generic Webhook.
On the Create Generic Webhook Alert Channel page, enter a name of your choice in the Name field.
```
WebSphere Automation Healing Webhook
```
Enter the Webhook URL for your WebSphere Automation installation.
```
https://www.example.com/websphereauto/health/webhooks/instana/memleak
```
You can get the Webhook URL for your WebSphere Automation installation with the following oc command:
```
oc get route cpd -o jsonpath=https://{.spec.host}/websphereauto/health/webhooks/instana/memleak
```
For more information, check with your cluster administrator.
Under Custom HTTP Request Headers, click Add Header.

Create the following two request headers:

Table 1. Request Headers
Key	Value
Content-Type	`application/json`
Authorization	`Bearer <TOKEN>` , where the TOKEN value is returned by `oc get secret -o jsonpath='{.data.token}' WebSphereHealth_CR_name-webhooks-apis-sa-generated-token \| base64 -d` Replace `WebSphereHealth_CR_name` with the name of the WebSphereHealth custom resource. The following example shows the command when the WebSphereHealth custom resource name is `wsa-health`: `oc get secret -o jsonpath='{.data.token}' wsa-health-webhooks-apis-sa-generated-token \| base64 -d` This value is also referred to as API token in Red Hat® OpenShift® documentation.

Create an alert.
To create an alert by using the Instana administrative console,
1. On the Team Settings page, under Events & Alerts, click Alerts.
2. On the Alerts page, click New Alert.
3. In the Name field, type a name for the alert.
```
WebSphere Automation Healing Memory Leak
```
4. In the Events menu, select Alert on Event(s).
5. In the Events section, click Add Event.
6. In the Add Events window, select Memory Leak. You can filter the list by using a keyword (for example, leak) to help find the entry in the list.
7. Click Add 1 Event.
8. In the Scope menu, select either All available entities or Selected Entities Only (Dynamic Focus Query). The latter option focuses the scope of the memory leak detection by using a filter that you select.
9. Under Alerting, click Add Alert Channel, select the alert channel that you created previously, and click Add 1 Channel.
10. Do not add a custom payload.
11. Click Create.
Optional: If your cluster is configured with a self-signed certificate, add the certificate to Instana's truststore, so that Instana can send alerts to WebSphere Automation.
The following instructions are for an on-premises, self-hosting instance of Instana.
1. On your Instana host, log in as the root user.
2. Make sure that the Java™ Software Development Kit (SDK) is installed.
3. Create a new directory /root/InstanaCerts.
4. Create a custom keystore file jssecacerts by copying the cacerts file from the Java SDK.
```
cp $JAVA_HOME/lib/security/cacerts /root/InstanaCerts/jssecacerts
```
5. Update the /root/settings.hcl file to point to the custom keystore.
```
[root@svt-instana-1 ~]# head -16 settings.hcl 
type                    = "single"
profile                 = "normal"
tenant                  = "svt"
unit                    = "was"
agent_key               = "xxx"
download_key            = "yyy"
sales_key               = "zzz"
host_name               = "my-instana.com
token_secret            = "secret"
clickhouse_bind_address = ""
custom_keystore         = "/root/InstanaCerts/jssecacerts"

cert {
  crt = "/root/cert/tls.crt"
  key = "/root/cert/tls.key"
}
```
6. Activate the new location.
```
instana update
```
7. Navigate to the InstanaCerts folder.
```
cd InstanaCerts
```
8. Create a shell script named updateCerts.sh with the following contents.
```
#!/bin/bash
pem=$1
cert_alias=$2
keytool -importcert -keystore /root/InstanaCerts/jssecacerts -keypass <KEY_PASSWORD> -storepass <STORE_PASSWORD> -file  $pem -alias $cert_alias
docker restart instana-groundskeeper
docker restart instana-issue-tracker
```
9. Change the permission of the shell script to 755.
```
chmod 755 updateCerts.sh
```
10. In the Red Hat OpenShift command window, create the new certificate file named mynew.crt.
  Ensure that the $WSA_INSTANCE_NAMESPACE environment variable is set to your WebSphere Automation instance namespace.
  
  If you configured a custom certificate by using the wsa-external-tls-secret secret, then run the following command:
```
oc get secret wsa-external-tls-secret -n $WSA_INSTANCE_NAMESPACE -o jsonpath='{.data.cert\.crt}' | base64 -d > mynew.crt
```
  Otherwise, run this command:
```
oc get secret wsa-default-external-tls-secret -n $WSA_INSTANCE_NAMESPACE -o jsonpath='{.data.cert\.crt}' | base64 -d > mynew.crt
```
11. Copy the mynew.crt file to the /root/InstanaCerts folder.
12. Update the keystore and restart the containers.
```
./updateCerts.sh mynew.crt <RHOCP_CLUSTER_NAME>
```
  Answer Yes when prompted to trust the certificate.