powervc-dbcheck Utility

Edit online

Starting with PowerVC 2.3.1, the powervc-dbcheck utility is introduced to detect and analyze inconsistencies in the PowerVC database. This utility simplifies troubleshooting and maintenance by providing detailed reports on discrepancies, which allows administrators to take corrective actions proactively.

The powervc-dbcheck utility helps to detect discrepancies in the following areas:

The powervc-dbcheck utility uses a new database user with read-only access to cinder and nova databases. This new user profile is used to check the all the inconsistencies from the block_device_mapping and volume_attachment table in the database.

Volume attachments of virtual machines

The powervc-dbcheck utility helps to identify and recover volume inconsistencies for volume attach or detach operations. If any error occurs during the volume attach or detach operation, the volume might go to the unexpected or transient state that results intermittent service restart or database connection failure. In such cases, all the PowerVC database volume resources are not updated properly and causes the future volume operations to fail.

The powervc-dbcheck utility can be used to find inconsistencies in one or more virtual machines, volumes, or projects at a time.

The powervc-dbcheck utility has the following subcommands:

show subcommand
The show subcommand displays the detailed output path of the log file for the powervc-dbcheck utility execution progress. It also displays the content of the .json file that contains the powervc-dbcheck execution results.
Syntax
To display the inconsistencies for all the volumes that are attached to a specific virtual machine, use the following command:
powervc-dbcheck volume-attachment show --instance_id <instance_id> -j
To display the inconsistencies for a specific volume, use the following command:
powervc-dbcheck volume-attachment show --volume_id <volume_id> -j
To display the inconsistencies for a specific project, use the following command:
powervc-dbcheck volume-attachment show --project_id <project_id> -j
Options
Table 1. Options to identify inconsistency with volumes
Options Description
--instance_id <instance_id> Displays inconsistencies in the specified virtual machines. You can specify one or more instance_id separated by commas.
--volume_id <volume_id> Displays inconsistencies in the specified volumes. You can specify one or more volume_id separated by commas.
--project_id <project_id> Displays inconsistencies in the specified projects. You can specify one or more project_id separated by commas.
-j Specifies that the output must be exported in a .json file. The -j parameter is mandatory.
Response

You can see the following response key attributes in the .json file to check the response for different inconsistencies:

Table 2. Key attributes for the show subcommand
Key attributes Description
stale_bdms_with_no_volume_attachments Displays a list of volumes for which nova can see the volumes as attached to the specified virtual machine, but there is no volume attachment entry in the cinder. It is possible that there is no volume mapping on the storage controller.
stale_volume_attachments_with_no_bdms Displays a list of volumes that are in the in-use state, but are not mapped to any virtual machine. This issue occurs when the virtual machine is deleted without unmapping the volumes.
other_inconsistent_volumes Displays the list of volumes that remain in error_recovering, error_attaching, or error_detaching state for a specific virtual machine. For these volumes, the record entry in the block_device_mapping and volume_attachment record table is null.
instance_exists Specifies whether a virtual machine is deleted of not. If the virtual machine is deleted, the instance_exists value is false, otherwise true.
volume_attachment_summary Specifies the number of volumes that are inconsistent with a specific virtual machine.
stale_volume Displays information about the specified volume if the volume is in error_recovering, error_attaching, or error_detaching state.
volume-attachment recover subcommand

The volume-attachment recover subcommand recovers the inconsistent volumes.

Syntax
To recover volumes that are attached to a specific virtual machine, use the following command:
powervc-dbcheck volume-attachment recover --instance_id <instance_id>
To recover a specific volume, use the following command:
powervc-dbcheck volume-attachment recover --volume_id <volume_id>
Options
Table 3. Options to recover inconsistent volumes
Options Description
--instance_id <instance_id> Recovers all the volumes attached to the specified virtual machine.
Note: You can specify only one instance_id at a time.
--volume_id <volume_id> Recovers the specified volume.
Note: You can specify only one volume_id at a time.
Response

You can see the following response key attributes in the .json file to check the response for the recovery operation:

Table 4. Key attributes for recover subcommand
Key attributes Description
RecoverAllVolumesFlow Indicates the overall result of the recovery task. The valid values are SUCCESS or FAILURE.
task_results Displays the results of each task of the recovery operation.
pending_tasks Displays the list of pending tasks for the recovery operation.
progress Displays the percentage of tasks that gets completed. It displays 100 after all the tasks are complete.
final_volume_status Displays the status of the volume after the recovery operation is complete. The valid values are available, in-use, or error_recovering.
recovery_task_message If a volume is mapped in the storage backend, PowerVC does not act without user confirmation. In such cases, PowerVC moves the volume to error_recovering state. The recovery_task_message section in the response code specifies the steps to perform before the recovery operation is attempted again. Unmap the volume from the backend before you use the recovery operation on the volume.
Note: The /var/log/powervc-dbcheck/powervc-dbcheck.log file is the default log file of the recover subcommand.

Residual network ports

If an error occurs during the virtual machine delete operation or network detach operation, the network ports might be retained in the PowerVC database. You can identify these network ports by using the powervc-dbcheck command.
Syntax
To identify the residual ports, use the following command:
powervc-dbcheck port show --all | --instance_id <instance_id> | --project_id <project_id> | --network_id <network_id>
Options
Table 5. Options to identify the residual ports
Options Description
--instance_id <instance_id> Lists the residual ports that are associated with a specific virtual machine.
--project_id <project_id> Lists the residual ports for a specific project.
--network_id <network_id> Lists the residual ports for a specific network.
--all Lists all the residual ports.
Note: You can export the report of inconsistencies in network ports to a .json file by using the following command:
powervc-dbcheck port show --all --json
Response

The powervc-dbcheck port show command provides the results in tabular format with details about stale port ID, reason, instance ID etc. If we export the output in a .json file, the results are populated in the port_inconsistencies key attribute section.

The following reasons for inconsistencies with network ports are identified and displayed:

Table 6.
Reason Description
Deleted Instance The virtual machine that is associated with the network port is deleted but the network port remains in the database. Also, if the network port is not reserved and its IP is not locked, the network port is considered a residual port.
Orphaned Port The network port is linked to a virtual machine that no longer exists in the database. Also, if the network port is not reserved and its IP is not locked, the network port is considered a residual port.
Port not used by PowerVC The network ports that are not linked to any instance or host group, are not reserved, and whose IP is not locked, are considered residual ports.
Port Mapped to Multiple Hosts A network port that is mapped to more than one host is considered residual network port due to conflicting associations.
Note: If inconsistencies are found with the network ports, contact the IBM support team to resolve them.

Allocation mismatches between Nova and placement

If volumes are allocated in the placement database without any corresponding virtual machine or the database do not have any placement entry for the volumes that are attached to a virtual machine, the volumes become inconsistent. You can detect such allocation mismatches by using the powervc-dbcheck command.
Syntax
To identify the allocation mismatches of the volumes, use the following command:
powervc-dbcheck allocations show --all | --instance_id <INSTANCE_ID> | --project_id <PROJECT_ID>
Options
Table 7. Options to identify the inconsistencies with allocations
Options Description
--instance_id <INSTANCE_ID> Displays the allocation mismatches for a specific virtual machine.
--project_id <PROJECT_ID> Displays all the allocation mismatches for a specific project.
--all Displays all the allocation mismatches.
Note: You can export the report of allocation mismatches to a .json file by using the following command:
powervc-dbcheck allocations show --all --json
Response

The powervc-dbcheck allocations showcommand provides the results in tabular format with details about expected memory, actual memory, expected core and actual core of a virtual machine. If we export the output in a .json file, the results are populated in the allocations_inconsistencies key attribute section.

Notes:
  • If inconsistencies are found with the allocations, contact the IBM support team to resolve them.
  • When a virtual machine is deployed with custom shared processor pool (SPP), only memory allocations are created. However, no CPU allocations are created. Therefore, for virtual machines that are deployed with custom SPP, if there is a mismatch in the memory, the cores field shows the value as NA.