Security Troubleshooting Guide

Environment variables missing

In order to ensure Application Modernization Accelerator security, certain environment variables are required. These variables are set automatically when Application Modernization Accelerator is installed. If any of the values are missing, you will see this message from the UI server logs:

Authentication is set to enable, but some required environment variable(s) is missing. This may cause Readiness probe failure. Please uninstall Application Modernization Accelerator, re-configure Application Modernization Accelerator and try again

One or more of the following variables may not be set correctly:

TA_OCP_API_ENDPOINT=https://example.eu-de.containers.cloud.ibm.com:30999
TA_AUTH_ISSUER_ENDPOINT=https://example.eu-de.containers.cloud.ibm.com:30999
TA_AUTH_OIDC_CLIENT_ID=example5282946fac07867fbc937548cb35d3ebbace00
TA_AUTH_OIDC_CLIENT_SECRET=example5282946fac07867fbc937548cb35d3ebbace00
TA_API_KEY=B513VT1zlF56J9ZBjdambQQVM5_jePVN
TA_AES_IV=n6Ma2S4DiYVK3AlM
TA_AES_KEY=B513VT1zlF56J9ZBjdambQQVM5_jePVN
TA_DB_USER=admin
TA_AUTH_OIDC_CALLBACK_URI=http://example-ui-server/auth/callback
# the value of the private key shall be the base64 encoded pem format key
TA_PRIVATE_KEY
# the value of the public key shall be the base64 encoded cert
TA_PUBLIC_KEY

You can find this list of variables by running the following command:

oc exec -ti <server_pod_name> env | grep TA

To manually configure the environment variables, you need to update the data/appmod.yaml file that is provided by the installer:

Uninstall Application Modernization Accelerator.
Re-configure data/appmod.yaml.
Install Application Modernization Accelerator again.

For example, to create a test public key and private key:

# Run this only once and make sure that you save and don't lose the files:
openssl req -newkey rsa:2048 -new -nodes -keyout private.pem -out unsigned-public.pem -subj "/C=IE/ST=Cork/L=Cork/O=IBM/CN=www.ibm.com"
openssl x509 -req -days 730 -in unsigned-public.pem -signkey private.pem -out public.crt

# Example to base64 encode the key pairs:
base64 -w 0 ./private.pem > private-base64
base64 -w 0 ./public.crt > public-base64

The values of private-base64 and public-base64 are the values of TA_PRIVATE_KEY and TA_PUBLIC_KEY.

Some values are generated by the installer and are not configurable in the data/appmod.yaml file.

Authentication page inaccessible

If you reconfigure the data/appmod.yaml file while troubleshooting and then cannot get to the authentication page, check that the OAuth callback URL is set correctly.

In the following example, the client ID is: TA_AUTH_OIDC_CLIENT_ID=example5282946fac07867fbc937548cb35d3ebbace00

ocp:
  authIssuerEndpoint: "{{ cluster.authorizationEndpoint | regex_replace('/oauth/.*', '') }}"
  apiEndpoint: "{{ clusterUrl }}"
  secretName: "app-mod-accelerator-secret"
oidc:
  endpointPort: "{{ cluster.authorizationEndpoint | urlsplit('port') }}"
  clientId: ama-ui
  clientSecret: "{{ lookup('password', '/dev/null length=40 chars=hexdigits') }}"

Ensure that you have the Red Hat OpenShift Container Platform connected to your terminal or can initialize the environment and issue the following command:

oc edit oauthclient example5282946fac07867fbc937548cb35d3ebbace00

If the redirect URL does not exist or is wrongly configured, update it to the correct URL. For example:

redirectURLs:
- http://your-ama-ui-url/auth/callback

Disabling authentication

You can disable authentication for debugging or testing purposes by re-configuring the data/appmod.yaml file:

Uninstall Application Modernization Accelerator.

Re-configure data/appmod.yaml to update authentication:

authentication:
  disabled:
    liberty: true
    ui: true

Install Application Modernization Accelerator again.

Key rotation

Red Hat Provided Certificates and Keys

The certificates (referred to as certs in the rest of this section) and keys are stored in the secrets:

# used by the couch db pod
db-internal-cert

# used by the Liberty server pod
liberty-internal-cert

# used by the UI pod
ui-internal-cert

To replace the cert and key pairs used by internal TLS by Application Modernization Accelerator, complete the following steps:

Delete the secret used by the pod.

Always back up the data before removing the Couch DB pod if Application Modernization Accelerator is installed without a Persistent Volume (PV).
Delete the pod.

For example, to replace the UI pod’s cert and key:

# delete the secret
oc delete secret ui-internal-cert

# delete the pod
oc delete pod ui-pod-name -n ama

# always delete the liberty pod
oc delete pod liberty-pod-name -n ama

To get the pod’s name:

oc get pods -n ama

Verify the Key rotation

You can verify the change of the cert by logging in to the liberty pod and connecting to the UI pod:

oc exec -ti liberty-pod-name bash

# in the liberty pod
openssl s_client -connect ui-service-name:3443

Then compare the Server certificate section before and after the key rotation.

To get the service’s name:

oc get services -n ama

Customer-Provided Certificates and Keys

Follow the steps to replace customer-provided certs and keys:

Backup data
Uninstall Application Modernization Accelerator.
Delete Application Modernization Accelerator secret app-mod-accelerator-secret, if any.
Follow the Enable Bring Your Own Key (BYOK) section and use a new cert and key pair.

Reset Internal Credentials

All internal credentials are stored in the secret app-mod-accelerator-secret.

Follow the steps to reset internal passwords, key, and initial vector:

Back up your old secret:

oc get secret app-mod-accelerator-secret -o yaml > backup.yaml

Delete the secret:

oc delete secret app-mod-accelerator-secret

Re-create the credentials. Follow the Enable Bring Your Own Key (BYOK) section to create a new secret. For example:

oc create secret generic app-mod-accelerator-secret \
--from-literal=db_username='updated-plain-text-username' \
--from-literal=secret='updated-text-password' \
--from-file=ta_public_key=./public-base64 \
--from-file=ta_private_key=./private-base64 \
--from-literal=ta_aes_key=$TA_TEMP_KEY \
--from-literal=ta_aes_iv=$TA_TEMP_IV \
--from-file=key.p12=key.p12

Delete the CouchDB pod, UI pod, and then the Liberty pod:

# delete the pod
oc delete pod couchdb-pod-name -n ama
oc delete pod ui-pod-name -n ama
oc delete pod liberty-pod-name -n ama

Invalid Certificate Issue on OCP on IBM Cloud (ROKS)

If you already have a valid certificate installed on IBM Cloud, and the certificate is also valid on the OCP web console page on IBM Cloud, you can enable this certificate to be valid for Application Modernization Accelerator by changing Application Modernization Accelerator's routes from ama.apps to ama-apps before you install Application Modernization Accelerator.

You can change the routes with the following steps:

In the Installed Operators, click IBM Application Modernization Accelerator.
In the Installed Operators > Operator details page, click Create instance.
In the IBM Application Modernization Accelerator > Create appmod page, go to YAML view.
In the YAML view editor, search for ama.apps and change it to ama-apps.
Click Create in the same page.

Egress Network Policy (ENP)

The Application Modernization Accelerator default ENP denies outbound communication from Application Modernization Accelerator's installation namespace, with the exception of DNS names github.com and github.ibm.com.

If desired, the Application Modernization Accelerator provided Egress network policy can be disabled. Disabling the Application Modernization Accelerator provided network policy allows Application Modernization Accelerator pods non-isolated (or unrestricted) outbound communication.

You can modify the ENP before or after the Application Modernization Accelerator installation.

Modify Egress Network Policy before Installation

Install Application Modernization Accelerator Operator.
Click Create Instance inside IBM Application Modernization Accelerator operator detail page.

Go to YAML view. You should see something similar to the following code:

networkPolicy:
  enabled: true
  egress:
    enabled: true
    default:
      - type: Allow
        to:
          dnsName: github.com
      - type: Allow
        to:
          dnsName: github.ibm.com
      - type: Allow
        to:
          cidrSelector: 10.254.0.0/16
      - type: Allow
        to:
          cidrSelector: 10.17.76.179/32
      - type: Allow
        to:
          dnsName: api.fuguo.ken.fyre.ibm.com

To disable ENP, change networkPolicy.egress.enabled to false.

To allow or deny DNS names, add similar entries:

- type: Deny
  to:
    dnsName: this.dns.name.will.be.denied
- type: Allow
  to:
    dnsName: this.dns.name.will.be.allowed

To allow or deny CIDR ranges, use the cidrSelector keyword with the CIDR value to replace dnsName and its value.

Modify Egress Network Policy after Installation

Navigate through Edit appmod and go to appmod details page in Installed Operators > Operator details.
Click the YAML tab.
Follow the same instructions to modify the ENP as described in the Modify Egress Network Policy before Installation section.
The ENP will take effect after the Application Modernization Accelerator pods are re-created by the operator.
If the pods' status does not change after a while, you can manually delete the Application Modernization Accelerator pods.

To check the ENP, use the following command:

oc describe egressnetworkpolicy ta-default-egress-network-policy -n your-ama-name-space