Long-running operations

The IBM Storage Scale native REST API feature supports long-running operations (LRO). An LRO is an internal task that represents business logic to process user-requested actions that take a significant amount of time to complete. For example, deleting NSDs is an LRO. When an LRO is submitted, the LRO manager handles its lifecycle, including accepting, running, and monitoring operations.

An LRO can be in one of the following states during its lifecycle:
  • Accepted: The LRO manager acknowledged receiving an LRO.
  • Initializing: The LRO manager started the operation, and it is preparing to run.
  • Running: The LRO is processing the IBM Storage Scale native REST API user-requested action.
  • Failed: The LRO encountered an error that forced it to stop execution.
  • Interrupted: The LRO was unexpectedly terminated.
  • Canceled: The LRO was stopped due to a cancellation request.
  • Done: The LRO successfully completed the user-requested action.
  • Stale: The LRO is unresponsive and might be deadlocked.
Note: The LRO status represents its lifecycle but does not necessarily indicate the outcome of the operation. If an LRO stops due to an error, its status is Failed. If an LRO completes but encounters noncritical errors that do not stop execution, its status is Done. The result field in the operation resource contains a response if the LRO completes successfully or an error if execution stops due to an error.

LRO behavior in REST API and CLI

When an LRO completes, REST APIs return HTTP 200 (OK). If the request has been accepted but the operation is still in progress, the API returns HTTP 202 (Accepted).

For some LIST-type LROs that are invoked by using the scalectl CLI with pagination enabled and JSON output requested (for example, scalectl fileset list fs1 --json), the CLI might internally issue multiple LIST requests to retrieve all pages of data. Returning individual LRO responses for each request is not practical in this scenario. Instead, the CLI aggregates the results from multiple requests and returns a list of the corresponding objects rather than individual LRO responses.

To receive an individual response wrapped in an LRO object for each LIST request when using the CLI, pagination must be disabled.

When scalectl processes long-running operations (LROs), it polls the returned operation name until the request is complete, if the request is not completed immediately. This behavior can have role-based access control (RBAC) implications. For an LRO, users might require permissions to query LROs in addition to the permissions to create the resource. These permissions are necessary for scalectl to retrieve the final output during the polling phase. If a user does not have the required RBAC permissions to query LROs, scalectl returns a PERMISSION_DENIED error. This error indicates that the user does not have the necessary permissions on the /scalemgmt/v3/operations/ resource. For more information about the role-based access control, see Role-based access control.

Managing long-running operations

Users can interact with LRO through the LRO manager, which provides the endpoints in both the REST API and CLI. These endpoints provide an interface to display the execution status and results of LROs. They also allow for the cleanup of historical LROs that are no longer needed. The following endpoints allows to interact with the LRO manager:
LIST
The LIST operation retrieves all the LROs managed by the LRO manager. For more information about the scalectl operation list command or operations API endpoint, see scalectl operations command or /scalemgmt/v3/operations: GET.
GET
The GET operation retrieves information about a single LRO that is managed by the LRO manager. For more information about the scalectl operation get command or operations API endpoint, see scalectl operations command or /scalemgmt/v3/operations/{id}: GET.
OUTPUT
The OUTPUT operation retrieves any console output that is generated by an LRO job. LRO might generate console output during their execution. For more information about the scalectl operation output command or operations API endpoint, see scalectl operations command or /scalemgmt/v3/operations/{id}/output: GET.
CANCEL
The CANCEL operation submits a request to the LRO manager to stop an LRO job. The job status changes to Canceled once it stops. For more information about the scalectl operation cancel command or operations API endpoint, see scalectl operations command or /scalemgmt/v3/operations/{id}:cancel: POST.
You can cancel the following LRO endpoints:
File system endpoints
  • filesystems: POST
File system disk endpoints
  • filesystems/{filesystem}/disks: POST
  • filesystems/{filesystem}/disks/{disk_name}: DELETE
  • filesystems/{filesystem}/disks:batchAdd: POST
  • filesystems/{filesystem}/disks:batchDelete: POST
Snapshot endpoints
  • filesystems/{filesystem}/filesets/snapshots:batchCreate: POST
  • filesystems/{filesystem}/filesets/snapshots:batchDelete: POST
  • filesystems/{filesystem}/filesets/{fileset}/snapshots: GET
  • filesystems/{filesystem}/filesets/{fileset}/snapshots: POST
  • filesystems/{filesystem}/filesets/{fileset}/snapshots/{snapshot_name}: DELETE
  • filesystems/{filesystem}/snapshots: GET
  • filesystems/{filesystem}/snapshots: POST
  • filesystems/{filesystem}/snapshots/{snapshot_name}: DELETE
  • filesystems/{filesystem}/snapshots:batchDelete: POST
DELETE
The DELETE operation removes a non-running job from the LRO manager. This action permanently deletes all LRO information and metadata. Deleting an active or running job is not allowed. A successful delete action returns an empty JSON object. If an error occurs, the JSON response contains error codes and messages. For more information about the scalectl operation delete command or operations API endpoint, see scalectl operations command or /scalemgmt/v3/operations/{id}: DELETE.