Failure uploading assets to a project

You might experience failures in your Cloud Pak for Data workloads when you upload assets to a project and the asset-file-api scale is set to three replicas.

Symptoms

Application failures can occur when an application thread is looking up an asset that was recently created. The following sample error messages indicate that an asset was not found:

[v2-get-asset-controller] {
  code: 404,
  error: 'Not Found',
  reason: 'File or directory could not be located. Target container/folder/file may have been deleted.',
  message: 'The requested resource could not be found.'

Causes

This problem is caused by a delay in syncing multiple copies in the storage layer. It is a potential limitation for any storage with multiple replicas. You are more likely to experience issues when network latency between two storage replica nodes is not optimal.

The problem occurs when replication latency is larger than the time from when an asset is created in one thread to when the asset is looked up by another thread in the application.

There is a retry mechanism in the Asset Files API. Applications fail when the replication latency between the copies is longer than that retry mechanism can handle. The frequency of this problem varies depending on the storage replication latency in the environment and how quickly the new asset is found.

Resolving the problem

You can consider the following options:

For small deployments, where high availability of asset-file-api is not critical, consider scaling the value of asset-file-api to a single replica.
For large deployments, ensure that your storage I/O performance is optimized. This problem is expected to be rare in well-configured Cloud Pak for Data clusters with optimized storage I/O performance. For more information, see Storage considerations.