Security Troubleshooting Guide

Environment variables missing

In order to ensure Transformation Advisor security, certain environment variables are required. These variables are set automatically when Transformation Advisor is installed. If any of the values are missing, you will see this message from the UI server logs:

Authentication is set to enable, but some required environment variable(s) is missing. This may cause Readiness probe failure. Please uninstall Transformation Advisor, re-configure Transformation Advisor and try again

One or more of the following variables may not be set correctly:

TA_OCP_API_ENDPOINT=https://example.eu-de.containers.cloud.ibm.com:30999
TA_AUTH_ISSUER_ENDPOINT=https://example.eu-de.containers.cloud.ibm.com:30999
TA_AUTH_OIDC_CLIENT_ID=example5282946fac07867fbc937548cb35d3ebbace00
TA_AUTH_OIDC_CLIENT_SECRET=example5282946fac07867fbc937548cb35d3ebbace00
TA_API_KEY=B513VT1zlF56J9ZBjdambQQVM5_jePVN
TA_AES_IV=n6Ma2S4DiYVK3AlM
TA_AES_KEY=B513VT1zlF56J9ZBjdambQQVM5_jePVN
TA_DB_USER=admin
TA_AUTH_OIDC_CALLBACK_URI=http://example-ui-server/auth/callback
# the value of the private key shall be the base64 encoded pem format key
TA_PRIVATE_KEY
# the value of the public key shall be the base64 encoded cert
TA_PUBLIC_KEY

You can find this list of variables by running the following command:

oc exec -ti <server_pod_name> env | grep TA

To manually configure the environment variables, you need to update the data/transadv.yaml file that is provided by the installer:

1. Uninstall Transformation Advisor.

2. Re-configure data/transadv.yaml.

3. Install Transformation Advisor again.

For example, to create a test public key and private key:

# Run this only once and make sure that you save and don't lose the files:
openssl req -newkey rsa:2048 -new -nodes -keyout private.pem -out unsigned-public.pem -subj "/C=IE/ST=Cork/L=Cork/O=IBM/CN=www.ibm.com"
openssl x509 -req -days 730 -in unsigned-public.pem -signkey private.pem -out public.crt

# Example to base64 encode the key pairs:
base64 -w 0 ./private.pem > private-base64
base64 -w 0 ./public.crt > public-base64

The values of private-base64 and public-base64 are the values of TA_PRIVATE_KEY and TA_PUBLIC_KEY.

Some values are generated by the installer and are not configurable in the data/transadv.yaml file.

Authentication page inaccessible

If you reconfigure the data/transadv.yaml file while troubleshooting and then cannot get to the authentication page, check that the OAuth callback URL is set correctly.

In the following example, the client ID is: TA_AUTH_OIDC_CLIENT_ID=example5282946fac07867fbc937548cb35d3ebbace00

ocp:
  authIssuerEndpoint: "{{ cluster.authorizationEndpoint | regex_replace('/oauth/.*', '') }}"
  apiEndpoint: "{{ clusterUrl }}"
  secretName: "transformation-advisor-secret"
oidc:
  endpointPort: "{{ cluster.authorizationEndpoint | urlsplit('port') }}"
  clientId: ta-ui
  clientSecret: "{{ lookup('password', '/dev/null length=40 chars=hexdigits') }}"

Ensure that you have the Red Hat OpenShift Container Platform connected to your terminal or can initialize the environment and issue the following command:

oc edit oauthclient example5282946fac07867fbc937548cb35d3ebbace00

If the redirect URL does not exist or is wrongly configured, update it to the correct URL. For example:

redirectURLs:
- http://your-ta-ui-url/auth/callback

Disabling authentication

You can disable authentication for debugging or testing purposes by re-configuring the data/transadv.yaml file:

1. Uninstall Transformation Advisor.

2. Re-configure data/transadv.yaml to update authentication:

   authentication:
     disabled:
       liberty: true
       ui: true

3. Install Transformation Advisor again.

Key rotation

Red Hat Provided Certificates and Keys

The certificates (referred to as certs in the rest of this section) and keys are stored in the secrets:

# used by the couch db pod
db-internal-cert

# used by the Liberty server pod
liberty-internal-cert

# used by the UI pod
ui-internal-cert

To replace the cert and key pairs used by internal TLS by Transformation Advisor, complete the following steps:

1. Delete the secret used by the pod.

   Always back up the data before removing the Couch DB pod if Transformation Advisor is installed without a Persistent Volume (PV).

2. Delete the pod.

For example, to replace the UI pod’s cert and key:

# delete the secret
oc delete secret ui-internal-cert

# delete the pod
oc delete pod ui-pod-name -n ta

# always delete the liberty pod
oc delete pod liberty-pod-name -n ta

To get the pod’s name:

oc get pods -n ta

Verify the Key rotation

You can verify the change of the cert by logging in to the liberty pod and connecting to the UI pod:

oc exec -ti liberty-pod-name bash

# in the liberty pod
openssl s_client -connect ui-service-name:3443

Then compare the Server certificate section before and after the key rotation.

To get the service’s name:

oc get services -n ta

Customer-Provided Certificates and Keys

Follow the steps to replace customer-provided certs and keys:

1. Backup data

2. Uninstall Transformation Advisor.

3. Delete Transformation Advisor secret transformation-advisor-secret, if any.

4. Follow the Enable Bring Your Own Key (BYOK) section and use a new cert and key pair.

Reset Internal Credentials

All internal credentials are stored in the secret transformation-advisor-secret.

Follow the steps to reset internal passwords, key, and initial vector:

1. Back up your old secret:

   oc get secret transformation-advisor-secret -o yaml > backup.yaml

2. Delete the secret:

   oc delete secret transformation-advisor-secret

3. Re-create the credentials. Follow the Enable Bring Your Own Key (BYOK) section to create a new secret. For example:

   oc create secret generic transformation-advisor-secret \
   --from-literal=db_username='updated-plain-text-username' \
   --from-literal=secret='updated-text-password' \
   --from-file=ta_public_key=./public-base64 \
   --from-file=ta_private_key=./private-base64 \
   --from-literal=ta_aes_key=$TA_TEMP_KEY \
   --from-literal=ta_aes_iv=$TA_TEMP_IV \
   --from-file=key.p12=key.p12

4. Delete the CouchDB pod, UI pod, and then the Liberty pod:

   # delete the pod
   oc delete pod couchdb-pod-name -n ta
   oc delete pod ui-pod-name -n ta
   oc delete pod liberty-pod-name -n ta

Invalid Certificate Issue on OCP on IBM Cloud (ROKS)

If you already have a valid certificate installed on IBM Cloud, and the certificate is also valid on the OCP web console page on IBM Cloud, you can enable this certificate to be valid for Transformation Advisor by changing Transformation Advisor's routes from ta.apps to ta-apps before you install Transformation Advisor.

You can change the routes with the following steps:

1. In the Installed Operators, click IBM Transformation Advisor.

2. In the Installed Operators > Operator details page, click Create instance.

3. In the IBM Transformation Advisor > Create TransAdv page, go to YAML view.

4. In the YAML view editor, search for ta.apps and change it to ta-apps.

5. Click Create in the same page.

Egress Network Policy (ENP)

A Transformation Advisor provided Egress network policy is enabled by default as of Transformation Advisor 3.1.0, in keeping with Kubernetes security hardening principles to limit pod external communication. If the OpenShift SDB plugin is not installed, the Transformation Advisor egress network policy will be disabled.

The Transformation Advisor default ENP denies outbound communication from Transformation Advisor's installation namespace, with the exception of DNS names github.com and github.ibm.com.

If desired, the Transformation Advisor provided Egress network policy can be disabled. Disabling the Transformation Advisor provided network policy allows Transformation Advisor pods non-isolated (or unrestricted) outbound communication.

You can modify the ENP before or after the Transformation Advisor installation.

Modify Egress Network Policy before Installation

1. Install Transformation Advisor Operator.

2. Click Create Instance inside IBM Transformation Advisor operator detail page.

3. Go to YAML view. You should see something similar to the following code:

   networkPolicy:
     enabled: true
     egress:
       enabled: true
       default:
         - type: Allow
           to:
             dnsName: github.com
         - type: Allow
           to:
             dnsName: github.ibm.com
         - type: Allow
           to:
             cidrSelector: 10.254.0.0/16
         - type: Allow
           to:
             cidrSelector: 10.17.76.179/32
         - type: Allow
           to:
             dnsName: api.fuguo.ken.fyre.ibm.com

4. To disable ENP, change networkPolicy.egress.enabled to false.

5. To allow or deny DNS names, add similar entries:

   - type: Deny
     to:
       dnsName: this.dns.name.will.be.denied
   - type: Allow
     to:
       dnsName: this.dns.name.will.be.allowed

6. To allow or deny CIDR ranges, use the cidrSelector keyword with the CIDR value to replace dnsName and its value.

Modify Egress Network Policy after Installation

1. Navigate through Edit TransAdv and go to TransAdv details page in Installed Operators > Operator details.

2. Click the YAML tab.

3. Follow the same instructions to modify the ENP as described in the Modify Egress Network Policy before Installation section.

4. The ENP will take effect after the Transformation Advisor pods are re-created by the operator.

5. If the pods' status does not change after a while, you can manually delete the Transformation Advisor pods.

6. To check the ENP, use the following command:

   oc describe egressnetworkpolicy ta-default-egress-network-policy -n your-ta-name-space