Troubleshooting for IT operation

The Operator install status stays as Pending and never completes.

Symptom

When you install the Sandbox 1.2 Operator, the installation status is shown as Pending and never completes.

Explanation

This might happen if sandbox instances created by using the a previous version of Sandbox Operator are found to be running. For example, if you have a running sandbox instance created by using Sandbox 1.1 or Sandbox 1.0, you might encounter this issue.

Resolution

You must delete all the sandbox instances that are created by using a previous version of Sandbox Operator. For more information, see Deleting a sandbox instance.

To save changes made to your sandbox instance, see Saving changes made to your sandbox instance.

The Sandbox instance is not created.

Symptom

After you create and apply a Custom Resource (CR) file for sandbox system, the deployment does not build parts of the sandbox instance, or the sandbox instance is created but then deleted.

Explanation

The information in the CR needs to follow the documented syntax. Some information in the CR might not be formatted correctly.

The validation of the custom information in the CR must be more open for customization and allow more acceptable input.

Resolution

Update the CR to ensure that all indentation, formatting, and syntax are followed correctly.

The z/OS system is not responding.

Symptom

A Sandbox system is deployed and running. However, the z/OS® system is not responding.

Explanation

This might be caused by multiple issues. For example, the zPDT® emulator is not starting, or an issue occurs during the IPL process of z/OS system.

Resolution

Check the log file for complete error messages. By default, the log file is at /zdt/zvolumes/log.txt within the container on the volume storage. The log file contains the following information.

• The zVolume copy/decompress/decrypt information
• The output from the zPDT emulator when the emulator is started.
• The console information when a system starts to IPL. The console information is available until the active console is removed from that session.

The zPDT emulator fails to start.

Symptom

The Sandbox system is running, but the zPDT emulator fails to start because the encrypted zVolumes are still encrypted.

Explanation

The Extended ADCD image has two zVolumes that are encrypted and must be decrypted before you use the image. To decrypt the volumes, you need to request a license with the tooling to execute the decryption from a license server. In a standard environment, a license server might have only one license for this usage. Therefore, if multiple deployments are created at the same time, failures might occur due to the limited licenses.

Resolution

Ensure that the container or pod can obtain a license from the license server. You can run the query_license command from the ibmsys1 user ID within the container to check whether the license server is responding to the license request. If the request fails, you need to ensure that the license server information is correct, especially the hostname or IP address that is required in the CR. Also, you can run /zdt/bin/decrypt_volumes.sh script to decrypt the volumes from the containers command prompt that is a terminal session within the pod. If the decryption still fails, you need to check whether a limited number of licenses available that can be used to decrypt volumes.

If the failure is caused by the limited licenses, take the following actions.

• Stagger deployments to prevent two deployments that are decrypting at once.
• If a deployment encountered the issue, delete the pod to resolve the issue. After you delete the pod (not the deployment), the deployment will notify that it needs to scale up the number of pods to 1 and create a new pod that will restart the process. Any copy, decompression, or decryption that is done by the previous pod is detected and not performed again. The new pod will attempt to obtain a license to decrypt again, and start zPDT emulator that needs to IPL z/OS when it is ready.

How to access a new application?

Symptom

A new application within z/OS is started, but you do not know how to access it.

Explanation

The Sandbox Operator currently creates services only when the CR is created or applied.

Resolution

To obtain the access to a new application within z/OS, you need to create a new service, or update an existing service with the required ports and selector. Also, you can update the existing service that is configured to the Sandbox system pod.

As almost all ports from the container are forwarded to the emulated z/OS environment, you need to expose the applications ports that are needed. The following two ports are not available to z/OS directly.

• Port 3270 is not forwarded to z/OS, and the 3270 port on the Linux® is backed by an emulated device by zPDT. This device is configured in z/OS as a non-SNA terminal. Typically, the device is used for a console, or it is used for logging in to TSO or other VTAM® applications.
• Port 8443 is reserved for future usage by Sandbox.

An example of a service YAML is shown as follows.

    apiVersion: v1
    kind: Service
    metadata:
      name: <Service name>
      namespace: <namespace of instance>
    spec:
      selector:
        app.kubernetes.io/instance: <Name of intance/pod>
        app.kubernetes.io/name: wazi-sandbox-system
      type: NodePort
      ports:
        - protocol: TCP
          port: <Port of zOS application>

Message: Manual IPL required.

Symptom

The deployment is running, but the /zdt/zvolumes/log.txt has a message of "Manual IPL required."

Explanation

This issue occurs because either the expected devmap file /zdt/zvolumes/log.txt does not exist or the IPL_ZOS environment variable was set to "manual".

Resolution

Check whether the devmap.txt file exists within /zdt/zvolumes. If the file does not exist, you can copy the file from /zdt/staging to the /zdt/zvolumes. Then, this issue can be resolved.

If the IPL_ZOS was set to "manual", this issue occurs because IPL_ZOS environment variable was not set to "default". After you change this variable, start the ipl-zOS service. Then, the zPDT emulator and IPL process of z/OS will be started.