I/O upload fails with "Could not connect to the endpoint URL."

I/O upload fails because it could not connect to an endpoint URL.

Description

When I/O upload is in progress for a large object, some times there could be an error showing such as "Could not connect to the endpoint URL". This error could be due to the underlying CES IP movement, which might take time.

This leaves sometimes an unclaimed disk space for the S3 user, which needs to be cleaned.

Error

See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 
upload failed: ./file_20G to s3://newbucket-ha-io/file_20G Could not connect to the endpoint 
URL: "https://10.11.77.133:6443/newbucket-ha-io/file_20G?uploadId=82ea114f-04c5-4141-ae90-cd8f2433633f&partNumber=1232" 

Workaround

In case of I/O abort, the S3 user account can clean the disk space used by the partially uploaded object by using the following command:

AWS_ACCESS_KEY_ID=<accesskey> AWS_SECRET_ACCESS_KEY=<secret-key> aws --endpoint https://10.11.77.133:6443 
--no-verify-ssl s3api abort-multipart-upload --bucket <bucket-name> 
--key <object-path> --upload-id <82ea114f-04c5-4141-ae90-cd8f2433633f>  

Note: System administrator can run a clean-up script on all the S3 users buckets, to check whether there are any left overs in the bucket directory where multipart-upload has failed. However, the recommendation is that the S3 user can clean up the failed uploads by using the S3 APIs.