Storage

Create a Presigned URL to Download an Object in Python 3.x

Share this post:

Using presigned URLs in Cloud Object Storage to share an object publicly for direct download

In one of my serverless (IBM Cloud Functions) proof of concepts, I had an idea of creating a pre-signed URL. Presigned URLs create a temporary link that can be used to share an object publicly. Since all my actions are written in Python 3.x, I decided to automate the presigned URL creation. Before migrating the Python 2.x code provided in the link here, I realised that there is one more additional step needed—the creation of HMAC (Access/Secret key) credentials.

Generate new credential with HMAC

  • Click Add to create.

GET operation written in Python 3.x

Here an example of a basic GET operation written in Python 3.x. Pass the required request elements before using this code.

import datetime
import hashlib
import hmac
import requests
from requests.utils import quote

access_key = '<ACCESS_KEY>'
secret_key = '<SECRET_KEY>'

# request elements
http_method = 'GET'
region = '' # us-south
bucket = ''
cos_endpoint = ''
host = cos_endpoint
endpoint = 'https://' + host
object_key = ''
expiration = 86400  # time in seconds


# hashing methods
def hash(key, msg):
    return hmac.new(key, msg.encode('utf-8'), hashlib.sha256).digest()


# region is a wildcard value that takes the place of the AWS region value
# as COS doen't use regions like AWS, this parameter can accept any string
def createSignatureKey(key, datestamp, region, service):
    keyDate = hash(('AWS4' + key).encode('utf-8'), datestamp)
    keyRegion = hash(keyDate, region)
    keyService = hash(keyRegion, service)
    keySigning = hash(keyService, 'aws4_request')
    return keySigning


# assemble the standardized request
time = datetime.datetime.utcnow()
timestamp = time.strftime('%Y%m%dT%H%M%SZ')
datestamp = time.strftime('%Y%m%d')

standardized_querystring = ('X-Amz-Algorithm=AWS4-HMAC-SHA256' +
                            '&X-Amz-Credential=' + access_key + '/' + datestamp + '/' + region + '/s3/aws4_request' +
                            '&X-Amz-Date=' + timestamp +
                            '&X-Amz-Expires=' + str(expiration) +
                            '&X-Amz-SignedHeaders=host')
standardized_querystring_url_encoded = quote(standardized_querystring, safe='&=')

standardized_resource = '/' + bucket + '/' + object_key
standardized_resource_url_encoded = quote(standardized_resource, safe='&')

payload_hash = 'UNSIGNED-PAYLOAD'
standardized_headers = 'host:' + host
signed_headers = 'host'

standardized_request = (http_method + '\n' +
                        standardized_resource + '\n' +
                        standardized_querystring_url_encoded + '\n' +
                        standardized_headers + '\n' +
                        '\n' +
                        signed_headers + '\n' +
                        payload_hash).encode('utf-8')

# assemble string-to-sign
hashing_algorithm = 'AWS4-HMAC-SHA256'
credential_scope = datestamp + '/' + region + '/' + 's3' + '/' + 'aws4_request'
sts = (hashing_algorithm + '\n' +
       timestamp + '\n' +
       credential_scope + '\n' +
       hashlib.sha256(standardized_request).hexdigest())

# generate the signature
signature_key = createSignatureKey(secret_key, datestamp, region, 's3')
signature = hmac.new(signature_key,
                     (sts).encode('utf-8'),
                     hashlib.sha256).hexdigest()

# create and send the request
# the 'requests' package autmatically adds the required 'host' header
request_url = (endpoint + '/' +
               bucket + '/' +
               object_key + '?' +
               standardized_querystring_url_encoded +
               '&X-Amz-Signature=' +
               signature)

print(('request_url: %s' % request_url))

print(('\nSending `%s` request to IBM COS -----------------------' % http_method))
print(('Request URL = ' + request_url))
request = requests.get(request_url)

print('\nResponse from IBM COS ---------------------------------')
print(('Response code: %d\n' % request.status_code))
print((request.text))

# this information can be helpful in troubleshooting, or to better
# understand what goes into signature creation
print('These are the values used to construct this request.')
print('Request details -----------------------------------------')
print(('http_method: %s' % http_method))
print(('host: %s' % host))
print(('region: %s' % region))
print(('endpoint: %s' % endpoint))
print(('bucket: %s' % bucket))
print(('object_key: %s' % object_key))
print(('timestamp: %s' % timestamp))
print(('datestamp: %s' % datestamp))

print('Standardized request details ----------------------------')
print(('standardized_resource: %s' % standardized_resource_url_encoded))
print(('standardized_querystring: %s' % standardized_querystring_url_encoded))
print(('standardized_headers: %s' % standardized_headers))
print(('signed_headers: %s' % signed_headers))
print(('payload_hash: %s' % payload_hash))
print(('standardized_request: %s' % standardized_request))

print('String-to-sign details ----------------------------------')
print(('credential_scope: %s' % credential_scope))
print(('string-to-sign: %s' % sts))
print(('signature_key: %s' % signature_key))
print(('signature: %s' % signature))

print('Because the signature key has non-ASCII characters, it is')
print('necessary to create a hexadecimal digest for the purposes')
print('of checking against this example.')


def hex_hash(key, msg):
    return hmac.new(b'key', msg.encode('utf-8'), hashlib.sha256).hexdigest()


def createHexSignatureKey(key, datestamp, region, service):
    keyDate = hex_hash(('AWS4' + key).encode('utf-8'), datestamp)
    keyRegion = hex_hash(keyDate, region)
    keyService = hex_hash(keyRegion, service)
    keySigning = hex_hash(keyService, 'aws4_request')
    return keySigning


signature_key_hex = createHexSignatureKey(secret_key, datestamp, region, 's3')

print(('signature_key (hexidecimal): %s' % signature_key_hex))

print('Header details ------------------------------------------')
print(('pre-signed URL: %s' % request_url))

For raw code, refer this GitHub Gist link.

Pre-signed URL in the console

Once you pass the required elements in the placeholders provided and execute the code, you should see your pre-signed URL in the console.

 

You can set the expiration time in seconds. To experience the proper use of pre-signed URL, try this solution tutorial.

 

Technical Offering Manager & Polyglot Programmer | IBM Cloud

More Storage stories
March 11, 2019

VIDEO – Block Storage vs. File Storage

File storage and block storage are both very traditional types of storage, and Amy Blea is going to take a closer look at the two options. We'll go through an overview of the structures, take a look at the different benefits offered, and go through a few different scenarios, explaining whether block or file storage would be most useful.

Continue reading

March 8, 2019

IBM Cloud Announces New Veeam Backup Support for Microsoft Office 365

By popular demand, IBM Cloud is excited to announce the general availability of Veeam Backup for Office 365. With this new Veeam capability, customers can protect their valuable Office 365 data and restore that data if it is corrupted or erased.  

Continue reading

February 25, 2019

IBM Cloud Object Storage Regional Service Now Available in Sydney

We’re delighted to announce the immediate availability of our new IBM Cloud Object Storage Regional service in Sydney, Australia.

Continue reading