Troubleshooting

This topic provides a list of common operational scenarios that are encountered in Appliance Control Center for IBM Z® & IBM® LinuxONE (ACC) and outlines recommended mitigation steps for each scenario.

Table 1. Troubleshooting scenarios
Scenario Mitigation
Appliance password changed Reset the appliance password on HMC/SE user interface.
Appliances are installed manually Unlock appliances in ACC
More LPARs are needed by the owner Delete and re-create resource package
More resources are needed by the owner Delete and re-create resource package
Admin has lost the password Reinstall ACC
ACC needs to be moved to another machine Export configuration, install, and import configuration
ACC cannot reach the appliance Wait and run the action again
HMC configuration must be changed In ACC, remove all resource packages and insert new configuration
Non-DPM LPARs in unhandled state Perform actions on HMC
DPM partitions in unhandled state Perform actions on HMC
Appliance activation failure Check the lpar

Getting ACC Information

You can retrieve information about the ACC and its appliance using the following APIs.

Note: You must have the appropriate authentication token for each API.
Retrieve ACC Information
Use the ACC admin token (ADMIN_TOKEN) to get ACC details:
curl -K -X 'GET' \
  "https://$ACC_IP:$ACC_PORT/api/init" \
  -H "Authorization: Bearer $ADMIN_TOKEN"
Retrieve Appliance Information
Use the SSC appliance token (SSC_TOKEN) to get details about the appliance running ACC:
curl -k -X 'GET' \
  "https://$ACC_IP/api/com.ibm.zaci.system/appliance" \
  -H "accept: application/vnd.ibm.zaci.payload+json" \
  -H "zACI-API: com.ibm.zaci.system/1.0" \
  -H "Authorization: Bearer $APP_TOKEN"

API Endpoints

To download the OpenAPI specifications, run the following command:
curl -o cc-api.yaml -k -X 'GET' \
 "https://$ACC_IP:$ACC_PORT/api/docs/cc-api.yaml" \
 -H 'accept: application/json' \
 -H "Authorization: Bearer $TOKEN"

ACC has no access to HMC

To initialize ACC in stand-alone mode, the ACC administrator must use the init API call with the hmc_managed parameter set to false.

In stand-alone mode, ACC cannot communicate with the Hardware Management Console (HMC). As a result, certain actions—such as activating or deactivating appliances—must be performed manually via the HMC user interface.

Once these manual steps are completed, ACC can be used to perform other operations, including:
  • Updating appliances
  • Gathering logs
  • Monitoring appliance health

Appliances are installed manually

If appliances are installed by using HMC, then they must be added to ACC.
Note: ACC automatically syncs appliance information (such as LPAR details) with the HMC. Any manual entries that are made by the ACC-admin in the resource package will be overwritten during this sync.However, changes related to CPCs (for example, if a CPC enters a not-operating state) are not automatically synced. Only LPAR-related data is synchronized.
Adding an appliance in default mode
Appliance-owner logs in, checks the resource package and running appliances.
It triggers a synchronization between the HMC and ACC. During the process, the ACC retrieves LPAR information from the HMC.
The appliance owner logs in to ACC using login API.
Appliance owner checks the resource package that is assigned to the owner by using the resource package API.
The appliance owner checks the list of running appliances by using the resource quota API
ACC will automatically sync the appliances' information with the HMC. However, if something changes with the CPCs (for example, the CPC goes into `not-operating` mode), this will not be automatically synced. Only LPAR-related information is synced.

ACC syncs appliance data from HMC, overwriting manual entries.

Adding appliance in stand-alone mode
Login to HMC and check the status of the appliance by observing the Operating System Messages

Manual updates to appliance data must be reflected in ACC by deleting and re-creating the resource package

ACC needs to be moved to another machine

ACC cannot reach the appliance

Possible causes:
Network issues between ACC and the appliance.
Appliance is booting or not accepting connections.
Example errors:
HTTPSConnectionPool(host='10.10.10.10', port=8081): Max retries exceeded with url:
/api/com.ibm.zaci.system/api-tokens (Caused by ConnectTimeoutError(...))
Mitigation:
Verify network connectivity.
Check appliance status by using HMC (Operating System Messages).

HMC configuration must be changed

Before modifying the HMC configuration, all resource packages associated with the previous HMC must be deleted. Following this, no new ACC tasks should be initiated to ensure that no HMC-related operations are in progress.

Once the previous configuration has been cleared, the ACC administrator (ACC-admin) can send the new HMC configurations using the /hmcconfg API.
Note: If a new HMC is being introduced, IBM recommends performing a fresh installation of ACC to ensure optimal configuration and stability

Non-DPM LPARs in unhandled state

For ACC to manage an LPAR on a non-DPM (non-Dynamic Partition Manager) machine, the LPAR must be in either the operating or not-activated state as reported by the HMC. If the LPAR enters a not-operating state or any other unsupported state, ACC will not be able to manage it. This means that operations such as syncing, activation, deactivation, and upgrade will not function.

To fix this issue, perform the following:

  • Login to HMC and check the status of the LPAR.
  • Fix any errors that you might encounter.

Unable to retrieve CPC information

You will not be able to retrieve CPC information if the credentials are incorrect in 4 or if a synchronization error occurs. To resolve this issue:

  • Enter correct host address and credentials in 4.
  • Resync the CPC information by using the following API:
    curl -k -X POST \
        "https://$ACC_IP:$ACC_PORT/api/sync/cpcs" \
        -H "Authorization: Bearer $ADMIN_TOKEN"
  • Resync the LPAR information by using the following API:
    curl -k -X POST \
        "https://$ACC_IP:$ACC_PORT/api/sync/lpars" \
        -H "Authorization: Bearer $ADMIN_TOKEN"
    Note: In the case where you need to put the LPARs into the lock state in the resource packages, resync will set is_locked to true.
  • If the appliance mode (installer or appliance) remains unsynced in a running appliance, then unlock the appliance to update the appliance mode. For more information on unlocking the appliance, see Unlocking Appliances using API

Deleting resource package

To delete a resource, run the following command:
curl -k -X 'DELETE' \
  "https://$ACC_IP:$ACC_PORT/api/resource/pkgs/${RESOURCE_PKG}?owner=${ACC_OWNER1_USERID}" \
  -H 'accept: */*' \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -H 'Content-Type: application/json'
In Default mode, ACC prevents deletion of a resource package if there are currently running appliances associated with it. To override this behavior, the user can include the force query parameter with a value of true. This allows ACC to proceed with the deletion despite the presence of running appliances. For example, you can modify the URI to the following:
/api/resource/pkgs/${RESOURCE_PKG}?owner=${ACC_OWNER1_USERID}&force=true
Note: Use &force=true to explicitly request force deletion.

However, if an appliance within the resource package is actively running a task, ACC will block the deletion request even with the force parameter. In such cases, an exception is thrown indicating that a task with the specified task ID is in progress. The deletion can only proceed once the task is completed.

In standalone mode, ACC automatically checks for running appliances on the HMC. The force parameter is not required in this mode.

Background Tasks information

Some requests to ACC may initiate long-running background tasks. Whenever a long-running task is initiated by a user, ACC executes the task in the background and returns a task ID. This task ID can be used to retrieve information about the ongoing background task in ACC.

To monitor the status of such tasks, users can query the task API. For example, to check the status of a specific task, use the following command:

curl -k -X 'GET' \
   "https://$ACC_IP:$ACC_PORT/api/tasks/${task_id}/status" \
  -H 'accept: */*' \
  -H "Authorization: Bearer $OWNER_TOKEN"
In certain cases, tasks may linger and fail to complete due to unknown reasons. These tasks can be manually deleted using the following command:
curl -k -X 'DELETE' \
   "https://$ACC_IP:$ACC_PORT/api/tasks/${task_id}" \
  -H 'accept: */*' \
  -H "Authorization: Bearer $OWNER_TOKEN" 

Restarting ACC

If ACC becomes unresponsive, restart the appliance using either the SSC APIs or the provided Ansible playbook.

CAUTION:
Any unsaved data will be lost. If the appliance does not restart after invoking the API, the ACC administrator should verify the LPAR status on the HMC.
To restart ACC, do the following command:
  1. Navigate to the other_usecases_ansible directory.
  2. Run the playbook
    ansible-playbook 08_restart_acc.yaml

Restarting an Appliance Managed by ACC

If you need to restart an appliance managed by ACC, you can use either the PUT /appliance/restart API or the provided Ansible playbook.

CAUTION:
Any unsaved data will be lost. If the appliance does not restart after invoking the API, the ACC administrator should verify the LPAR status on the HMC.
Using API
Run the following command to restart the appliance using API:
curl -X 'PUT' \
"https://$ACC_IP:$ACC_PORT/api/appliance/restart" \
-H "Authorization: Bearer $OWNER_TOKEN" \
-H 'accept: application/json' \
-H 'Content-Type: application/json' \
-d '{
  "resource_pkg_name": "rp1",
  "lpar": "LP01"
}'

This command sends a restart signal to the appliance running on LPAR LP01 under resource package rp1.

Using the Ansible Playbook
  1. Navigate to the other_usecases_ansible directory.
  2. Run the following playbook:
    ansible-playbook 13_restart_appliances.yaml
Changing Appliance Password
Certain appliances, such as SSA, allow you to change your passwords by using REST APIs. In these scenarios, you must also use the unlock_quota API in ACC to update the appliance password in ACC. This step ensures that ACC can continue to communicate with the appliance by using REST APIs. For more information, see Unlocking the Appliance.
Note: When you reset the password by using the REST API, the updated password applies only to REST API access. It does not apply to the LPAR. The LPAR profile password (Administrator password) remains unchanged.
Impact of updating the password by using the unlock_quota API

After you update the appliance password by using the unlock_quota API, ACC cannot perform operations that require LPAR access. Examples of these operations include LPAR activation, appliance re-installation, and appliance updates.

ACC operations that do not require LPAR access, such as health monitoring, continue to work.

Mitigation

If the appliance owner wants to change the appliance password and continue using ACC seamlessly, the owner must reset the password directly in the LPAR profile by using the Reset Logon Settings option in the HMC or SE user profile. For more information, see Changing the logon settings for a Secure Service Container partition on a standard mode system.

Cannot upload images to ACC

Ensure that no firewall rule is blocking data uploads to the ACC. In sensitive environments, security policies may restrict file transfers, which can prevent you from uploading any image or fix to the ACC.