Storage

Create a Presigned URL to Download an Object in Python 3.x

Share this post:

Using presigned URLs in Cloud Object Storage to share an object publicly for direct download

In one of my serverless (IBM Cloud Functions) proof of concepts, I had an idea of creating a pre-signed URL. Presigned URLs create a temporary link that can be used to share an object publicly. Since all my actions are written in Python 3.x, I decided to automate the presigned URL creation. Before migrating the Python 2.x code provided in the link here, I realised that there is one more additional step needed—the creation of HMAC (Access/Secret key) credentials.

Generate new credential with HMAC

  • Click Add to create.

GET operation written in Python 3.x

Here an example of a basic GET operation written in Python 3.x. Pass the required request elements before using this code.

import datetime
import hashlib
import hmac
import requests
from requests.utils import quote

access_key = '<ACCESS_KEY>'
secret_key = '<SECRET_KEY>'

# request elements
http_method = 'GET'
region = '' # us-south
bucket = ''
cos_endpoint = ''
host = cos_endpoint
endpoint = 'https://' + host
object_key = ''
expiration = 86400  # time in seconds


# hashing methods
def hash(key, msg):
    return hmac.new(key, msg.encode('utf-8'), hashlib.sha256).digest()


# region is a wildcard value that takes the place of the AWS region value
# as COS doen't use regions like AWS, this parameter can accept any string
def createSignatureKey(key, datestamp, region, service):
    keyDate = hash(('AWS4' + key).encode('utf-8'), datestamp)
    keyRegion = hash(keyDate, region)
    keyService = hash(keyRegion, service)
    keySigning = hash(keyService, 'aws4_request')
    return keySigning


# assemble the standardized request
time = datetime.datetime.utcnow()
timestamp = time.strftime('%Y%m%dT%H%M%SZ')
datestamp = time.strftime('%Y%m%d')

standardized_querystring = ('X-Amz-Algorithm=AWS4-HMAC-SHA256' +
                            '&X-Amz-Credential=' + access_key + '/' + datestamp + '/' + region + '/s3/aws4_request' +
                            '&X-Amz-Date=' + timestamp +
                            '&X-Amz-Expires=' + str(expiration) +
                            '&X-Amz-SignedHeaders=host')
standardized_querystring_url_encoded = quote(standardized_querystring, safe='&=')

standardized_resource = '/' + bucket + '/' + object_key
standardized_resource_url_encoded = quote(standardized_resource, safe='&')

payload_hash = 'UNSIGNED-PAYLOAD'
standardized_headers = 'host:' + host
signed_headers = 'host'

standardized_request = (http_method + '\n' +
                        standardized_resource + '\n' +
                        standardized_querystring_url_encoded + '\n' +
                        standardized_headers + '\n' +
                        '\n' +
                        signed_headers + '\n' +
                        payload_hash).encode('utf-8')

# assemble string-to-sign
hashing_algorithm = 'AWS4-HMAC-SHA256'
credential_scope = datestamp + '/' + region + '/' + 's3' + '/' + 'aws4_request'
sts = (hashing_algorithm + '\n' +
       timestamp + '\n' +
       credential_scope + '\n' +
       hashlib.sha256(standardized_request).hexdigest())

# generate the signature
signature_key = createSignatureKey(secret_key, datestamp, region, 's3')
signature = hmac.new(signature_key,
                     (sts).encode('utf-8'),
                     hashlib.sha256).hexdigest()

# create and send the request
# the 'requests' package autmatically adds the required 'host' header
request_url = (endpoint + '/' +
               bucket + '/' +
               object_key + '?' +
               standardized_querystring_url_encoded +
               '&X-Amz-Signature=' +
               signature)

print(('request_url: %s' % request_url))

print(('\nSending `%s` request to IBM COS -----------------------' % http_method))
print(('Request URL = ' + request_url))
request = requests.get(request_url)

print('\nResponse from IBM COS ---------------------------------')
print(('Response code: %d\n' % request.status_code))
print((request.text))

# this information can be helpful in troubleshooting, or to better
# understand what goes into signature creation
print('These are the values used to construct this request.')
print('Request details -----------------------------------------')
print(('http_method: %s' % http_method))
print(('host: %s' % host))
print(('region: %s' % region))
print(('endpoint: %s' % endpoint))
print(('bucket: %s' % bucket))
print(('object_key: %s' % object_key))
print(('timestamp: %s' % timestamp))
print(('datestamp: %s' % datestamp))

print('Standardized request details ----------------------------')
print(('standardized_resource: %s' % standardized_resource_url_encoded))
print(('standardized_querystring: %s' % standardized_querystring_url_encoded))
print(('standardized_headers: %s' % standardized_headers))
print(('signed_headers: %s' % signed_headers))
print(('payload_hash: %s' % payload_hash))
print(('standardized_request: %s' % standardized_request))

print('String-to-sign details ----------------------------------')
print(('credential_scope: %s' % credential_scope))
print(('string-to-sign: %s' % sts))
print(('signature_key: %s' % signature_key))
print(('signature: %s' % signature))

print('Because the signature key has non-ASCII characters, it is')
print('necessary to create a hexadecimal digest for the purposes')
print('of checking against this example.')


def hex_hash(key, msg):
    return hmac.new(b'key', msg.encode('utf-8'), hashlib.sha256).hexdigest()


def createHexSignatureKey(key, datestamp, region, service):
    keyDate = hex_hash(('AWS4' + key).encode('utf-8'), datestamp)
    keyRegion = hex_hash(keyDate, region)
    keyService = hex_hash(keyRegion, service)
    keySigning = hex_hash(keyService, 'aws4_request')
    return keySigning


signature_key_hex = createHexSignatureKey(secret_key, datestamp, region, 's3')

print(('signature_key (hexidecimal): %s' % signature_key_hex))

print('Header details ------------------------------------------')
print(('pre-signed URL: %s' % request_url))

For raw code, refer this GitHub Gist link.

Pre-signed URL in the console

Once you pass the required elements in the placeholders provided and execute the code, you should see your pre-signed URL in the console.

 

You can set the expiration time in seconds. To experience the proper use of pre-signed URL, try this solution tutorial.

 

Technical Offering Manager & Polyglot Programmer | IBM Cloud

More Storage stories
December 14, 2018

Introducing a New Cloud Backup Service: IBM Cloud Backup

Introducing a cloud backup service that makes recovering your data much easier. IBM Cloud Backup is a fully-automated solution that allows users to worry less about "What if?" and more on "What next?"

Continue reading

November 26, 2018

IBM Cloud Expands Availability of its Intel® Xeon® 2600 v3 Family Bare Metal Servers Closeout Pricing Promotion

IBM Cloud is expanding the data centers and inventory included in its current Intel® Xeon® 2600 v3 family bare metal servers close-out pricing promotion.

Continue reading