Troubleshooting IBM Hyper Protect Virtual Servers

Refer to the following information for troubleshooting issues with IBM Hyper Protect Virtual Servers

  • For Secure Service Container partitions, use the Secure Service Container user interface to view the logs.
  • For components such as Secure Build containers, use the hpvs sb log command to retrieve the build logs, or hpvs sb status command to check the progress of the Secure Build.
  • For the command line tools provided by the product, add --debug to view the detailed log.
  • All output from the command line is recorded in $HOME/hpvs/logs/.
  • For the commands that require a yaml configuration file, check the formatting of the yaml file by comparing with the example yaml file in the $HOME/hpvs/config directory of each command.
  • To know more about the errors you might encounter, see Error messages in Hyper Protect Virtual Servers.
  • Use the mustgather script to collect debugging information when you want to open a support ticket. For more information, see Gathering Information for IBM Support.

Known issues with IBM Hyper Protect Virtual Servers 1.2.7.2, 1.2.7.1, 1.2.7, 1.2.6.1, or 1.2.6

ERROR: Failed to pull image

If you see this error when running the hpvs image pull command or during the deployment of virtual servers, it might be because there is insufficient space in the appliance_data quotagroup. You can check the available size of the appliance_data quotagroup by using the command hpvs quotagroup show --name appliance_data. Ensure that the available size is larger than the image size. Increase the available size of the appliance_data quotagroup by using the command hpvs quotagroup update --size

ERROR: Failed to SSH to the HPVS container when both external and private networks exists

If you see this error, it might be because the default gateway is set to the private network. When you are setting up the network configuration and to setup the default gateway to the external network, specify the external network name as the first entry in the lexicographic order. Otherwise, access the container by using SSH fails. For example:

external
a_encf100_network

internal
b_encf960_network

ERROR: HVS-VSUD003 Update virtual server failed due to wrong quotagroup configuration

If you see this error, it might be because an invalid parameter was specified in the quotagroup configuration. You can provide valid quotagroup configuration details and retry the command.

HPVS container hangs or secure shell (SSH) access fails

If a running container hangs or SSH access to the container fails, check if the non-passthrough quotagroup that is used for the root disk (RUNQ_ROOTDISK) has enough available size. The available size should be set to more than 5 GB. You can use the hpvs quotagroup update command to increase the available size.

The "hpvs vs **" command failed

If you see the following issue:

$ hpvs vs * * command is throwing error like
ERROR: HVS-<Error code> API Error: .......: no space left on device 500

Then, check the appliance_data quotagroup size.

$ sudo hpvs quotagroup show --name appliance_data
+-------------+----------------+
| PROPERTIES  | VALUES         |
+-------------+----------------+
| name        | appliance_data |
| filesystem  | btrfs          |
| passthrough | false          |
| pool_id     | lv_data_pool   |
| size        | 10GB           |
| available   | 0B             |
| containers  | None           |
+-------------+----------------+

If the available size is displayed as 0 GB, then increase the size of the appliance_data quotagroup by running the following command.

hpvs quotagroup update --name appliance_data --size 80GB

The data pool is not ready

If you see this in an error message: "The data pool is not ready", ensure that you add storage disks to the datapool by using the User Interface of the Secure Service Container.

standard_init_linux.go: : exec user process caused "exec format error"

You might notice one of the following errors when running the IBM Hyper Protect Virtual Servers CLI tool.

  • standard_init_linux.go:211: exec user process caused "exec format error", or
  • standard_init_linux.go:190: exec user process caused "exec format error"

The problem is most likely caused by the CLI commands for x86 architecture being executed on the Linux on IBM Z/LinuxONE (i.e., s390x architecture) management server, or the s390x architecture CLI commands being executed on the x86 Linux system.

To workaround the problem, ensure that you use the CLI tool with the correct tag for the management server.

  • The IBM Hyper Protect Virtual Servers CLI tagged with 1.2.x.s390x.s390x must be executed on the s390x architecture management server.
  • The IBM Hyper Protect Virtual Servers CLI tagged with 1.2.x.s390x must be executed on the x86 architecture management server.

gpg: Invalid option errors when generating the GPG key pair

You might encounter an error messages such as gpg: Invalid option "--pinentry-mode=loopback" or gpg: Invalide opiton "--generate-key" when generating the GPG key pair on the s390x Linux management server.

The problem is most likely caused by a different version of GnuPG tools that you have installed, such as gnupg 1.4.20-1ubuntu3.3, or gnupg2 2.1.11-6ubuntu2.1.

To resolve the problem, use --gen-key option instead of --generate-key in the command, or upgrade GnuPG to a later version such as 2.2.17.

GPG hangs or "Not enough random bytes available." error when generating the GPG key pair

You might experience GPG hangs, or encounter an error messages such as Not enough random bytes available. Please do some other work to give the OS a chance to collect more entropy!  (Need 188 more bytes) when generating the GPG key pair on the s390x Linux management server.

The problem is most likely caused by a missing utility haveged on the Linux management server. For more information, see Stackoverflow.

To resolve the problem, install the haveged utility with the following command:

apt-get install -y haveged

Secure Build failed to clone the Github repository if a passphrase is associated with the private key

You might encounter the following error message or a similar one when the Secure Build tries to build the source code from a Github repository by using the private key with a passphrase.

Could not read from remote repository.

The problem is most likely caused by a known limitation that the Secure Build requires the private key used to secure access to the source Github repository does not have a passphrase.

To workaround the problem, consider one of the following options.

  • Generate a new SSH key pair with the -N parameter and an empty passphrase as in the following command example. Note that both private key and public key are generated. The -m pem parameter is optional and ensures the private key is generated with a RSA PRIVATE KEY comment line.
ssh-keygen -t rsa -b 4096 -f /tmp/id_rsa -N "" -m pem
  • Overwrite the private key with an empty passphrase as in the following command example. Note that the public key is not changed.
openssl rsa -in id_rsa -out id_rsa

After you generate the new key pair or overwrite the private key, ensure that you update the github:key value with the new private key, and then run the hpvs sb update command to apply the changes. For more information, see Building the application by using the Secure Build, and hpvs sb update command.

Hyper Protect Virtual Server instance restarting continuously when running hpvs vs list command

You might notice that the Hyper Protect Virtual Server container has been in the restarting state continuously if you run the hpvs vs list command.

The problem is most likely caused by the excessive memory setting assigned to the Hyper Protect Virtual Server container. The memory size allocated to the Hyper Protect Virtual Server container cannot exceed the available memory resource on the Secure Service Container partition.

To workaround the problem, consider one of the following options

  • Ask the appliance or system administrator to allocate sufficient memory on the Secure Service Container partition before creating or updating the Hyper Protect Virtual Server container.
  • Change the memory setting of the Hyper Protect Virtual Server container to a valid value, and then use the hpvs vs update command to update the container.