Troubleshooting: Installation

During installation or image mirroring, oc-mirror process fails

When the mirroring processes are running during the installation script processing, the registry-quay-app pod or container restarts. If this happens you can see high memory usage during mirroring for registry-quay-app, or a slow image push with poor mirroring performance. The process will be stopped with an out of memory killed (OOOMKilled) status.

Why does this happen? Storage performance is critical to correct functioning of system processes. If you experience any issues, consider if these are the root cause.

Storage cluster is under heavy demand.
Insufficient disk input/output operations per second (IOPS) or throughput.
Large image sets are being mirrored.
Tight container memory limits.

Workarounds: If this issue occurs, you can take one of the following actions.

Increase the container memory limits.
Run a backend storage refresh or rebalance if required.
Plan for storage that meets your input/output demands.
Continuous review usage and storage performance.

The network observability plugin remains in a creating state

During the installation process, the netobserv-plugin pod remains in the ContainerCreating state and doesn't reach the ready state.

This is expected behaviour for these reasons.

The network observability plugin netobserv-plugin, depends on configuration maps ConfigMaps and secrets that are created by the LokiStack.
During the installation, LokiStack must go through it processes that include, deploying its components, and reconciling and creating required configuration artefacts.
Until the required ConfigMaps and secrets are present, Kubernetes cannot complete the container initialization for the netobserv-plugin pod.

When LokiStack finishes provisioning and the required resources become available, the netobserv-plugin pod transitions to the Running state. The netobserv-plugin pod becomes ready without any manual intervention

Mirror fails for bootstrap.sh

Occasionally the mirroring step in the installation script hangs when you are running the installation script.

Rerunning the step usually clears the delay. You can try up to five times, and the installation script step should complete as planned.

Installation fails after vault setup

If the bootstrap.sh step fails after the Vault setup runs successfully, locate $VAULTSCRIPT_PATH comment in the bootstrap.sh file and rerun it.

Pipeline run fails

If the pipeline run fails, click on overflow menu (three dots) next to the failure, and click Rerun.

Another option is to rerun the bootstrap.sh step after completing the following step:

When the bootstrap.sh reports Bootstrap complete, go to the OCP cluster, select Pipelines>Pipelineruns, and follow the sovcloud-installer-run-* logs in the openshift-pipelines namespace.

Argo CD synchronization issue

ArgoCD sync and wait task fail. Three tasks in the pipeline use the argocd sync and wait template task. Sometimes the task fails.

This is an intermittent issue. As a workaround, wait approximately five minutes and then rerun the pipeline.

Vault CLI installation error

During the installation of the Vault CLI, an error messages related to unknown flag and versions, can be ignored. This is a known issue and does not impact any functionality.

The required CLIs install successfully. Check the log summary for these confirmation messages.

✓ ihc-vault is available: installed
✓ ihc-cli is available: installed

Email notifications are not sent after installation

If email notifications are not sent, after the installation finishes, there are a number of things you can check.

Emails are not sent from the email service: Restart email service pods

If you have checked the SMTP settings in the template.env file and they are correct, sometimes restarting the email service pods after installation fixes the issue. You can restart the email service pods by using Argo CD, or through the OpenShift console.

Login to the Argo CD console.
From the navigation menu, click Applications .
Search for email-service-app
Open the email-service-app-* application by clicking it.
Open the overflow menu for the email-service-app resource deployment. Click Restart.

Alternatively, you can restart the email service pods by using the OpenShift console.

Login to the OpenShift console.
From the navigation menu, click Workloads > Deployments.
Locate email-service-app.
Open the overflow menu⋮ for the email-service-app deployment. Click Restart.

Emails are not sent from the email service: Check SMTP settings

Login to the OpenShift console as the admin user.
From the top menu, click your UserID, and Copy login command.
If you are prompted, log in again.
Click the Display token link.
Copy the Log in with this token command.
Open two terminals. One terminal for port-forward to Vault. The second terminal for commands.
In the first terminal, paste the command you copied in the OpenShift console window.
Run the command to port-forward the vault service.
```
oc port-forward svc/vault 8200:8200 -n vault
```
In the second terminal, paste the command you copied in the OpenShift console window.

Run the command to set the vault token.

export VAULT_TOKEN=$(oc get secret vault-prod-init -n vault -o jsonpath="{.data.root_token}" | base64 -d)

Run this command to check the SMTP settings in secret/tm/mail.
```
vault kv get -tls-skip-verify secret/tm/mail
```

Update the settings if they are incorrect.

Example 1: enable authentication, update user, update password.

vault kv patch -tls-skip-verify secret/tm/mail auth_enable="true" user="new-user" password="new-password"

Example 2: use SMTPS instead of STARTTLS.

vault kv patch -tls-skip-verify secret/tm/mail port="465" ssl_enable="true" starttls_enable="false"

Example 3: use STARTTLS instead of SMTPS.

vault kv patch -tls-skip-verify secret/tm/mail port="587" starttls_enable="true" ssl_enable="false

Run this command to view the updated SMTP settings.
```
vault kv get -tls-skip-verify secret/tm/mail
```
Unset the VAULT_TOKEN variable.
```
unset VAULT_TOKEN
```
Login to the Argo CD console.
From the Navigation menu, click Applications.
Locate email-service-app, and click email-service-app-* to open the application.
Click the Refresh down arrow, select Hard Refresh.
Depending on the values you updated, you see either email-service-env-configmap-prod or email-service-mpconfig-secrets which can show as temporarily out-of-sync.
Wait for the auto-sync to complete.
If auto-sync does not start, click Sync. Then click Synchronize.
When the sync finishes, restart the pods by opening the overflow menu ⋮ for the email-service-app resource deployment, and clicking Restart.
Verify that email notifications are sent.

Connection error 403 - password expires

If you experience any 403 connection errors, check if any of your passwords have expired. For example, if the bare metal Redfish connection fails with a HTTP 403 connection error, check if the password has expired. Update with a new password and retry the operation.

Compliance center does not start

If the Sovereign compliance center does not start from the IBM Sovereign Core navigation menu, rerun the installation script.

Cannot discover the tenant plane

If you have run the installation and cannot set up a tenant or cannot discover the tenant worker plane, here are some possible reasons that you can check.

Check that you have run the discovery step in the installation procedure? For details, see Step 8 here, Installing IBM Sovereign Core.
Check that the configuration for the worker nodes that you set in the configuration file infra_cloud/vars.yaml are correct.

To find details about each of these suggestions, see Installing IBM Sovereign Core.

Cannot create tenant, or see catalog

If you have run the installation and cannot set up a tenant or cannot discover the tenant worker plane, here are some possible reasons that you can check.

Check that you have run the discovery step in the installation procedure? For details, see Step 7 here, 7.
Check that the configuration for the worker nodes that you set in the configuration file infra_cloud/vars.yaml are correct.

To find details about each of these suggestions, see Installing IBM Sovereign Core.