Storage

Create a Presigned URL to Download an Object in Python 3.x

Share this post:

Using presigned URLs in Cloud Object Storage to share an object publicly for direct download

In one of my serverless (IBM Cloud Functions) proof of concepts, I had an idea of creating a pre-signed URL. Presigned URLs create a temporary link that can be used to share an object publicly. Since all my actions are written in Python 3.x, I decided to automate the presigned URL creation. Before migrating the Python 2.x code provided in the link here, I realised that there is one more additional step needed—the creation of HMAC (Access/Secret key) credentials.

Generate new credential with HMAC

  • Click Add to create.

GET operation written in Python 3.x

Here an example of a basic GET operation written in Python 3.x. Pass the required request elements before using this code.

import datetime
import hashlib
import hmac
import requests
from requests.utils import quote

access_key = '<ACCESS_KEY>'
secret_key = '<SECRET_KEY>'

# request elements
http_method = 'GET'
region = '' # us-south
bucket = ''
cos_endpoint = ''
host = cos_endpoint
endpoint = 'https://' + host
object_key = ''
expiration = 86400  # time in seconds


# hashing methods
def hash(key, msg):
    return hmac.new(key, msg.encode('utf-8'), hashlib.sha256).digest()


# region is a wildcard value that takes the place of the AWS region value
# as COS doen't use regions like AWS, this parameter can accept any string
def createSignatureKey(key, datestamp, region, service):
    keyDate = hash(('AWS4' + key).encode('utf-8'), datestamp)
    keyRegion = hash(keyDate, region)
    keyService = hash(keyRegion, service)
    keySigning = hash(keyService, 'aws4_request')
    return keySigning


# assemble the standardized request
time = datetime.datetime.utcnow()
timestamp = time.strftime('{07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0}Y{07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0}m{07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0}dT{07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0}H{07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0}M{07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0}SZ')
datestamp = time.strftime('{07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0}Y{07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0}m{07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0}d')

standardized_querystring = ('X-Amz-Algorithm=AWS4-HMAC-SHA256' +
                            '&X-Amz-Credential=' + access_key + '/' + datestamp + '/' + region + '/s3/aws4_request' +
                            '&X-Amz-Date=' + timestamp +
                            '&X-Amz-Expires=' + str(expiration) +
                            '&X-Amz-SignedHeaders=host')
standardized_querystring_url_encoded = quote(standardized_querystring, safe='&=')

standardized_resource = '/' + bucket + '/' + object_key
standardized_resource_url_encoded = quote(standardized_resource, safe='&')

payload_hash = 'UNSIGNED-PAYLOAD'
standardized_headers = 'host:' + host
signed_headers = 'host'

standardized_request = (http_method + '\n' +
                        standardized_resource + '\n' +
                        standardized_querystring_url_encoded + '\n' +
                        standardized_headers + '\n' +
                        '\n' +
                        signed_headers + '\n' +
                        payload_hash).encode('utf-8')

# assemble string-to-sign
hashing_algorithm = 'AWS4-HMAC-SHA256'
credential_scope = datestamp + '/' + region + '/' + 's3' + '/' + 'aws4_request'
sts = (hashing_algorithm + '\n' +
       timestamp + '\n' +
       credential_scope + '\n' +
       hashlib.sha256(standardized_request).hexdigest())

# generate the signature
signature_key = createSignatureKey(secret_key, datestamp, region, 's3')
signature = hmac.new(signature_key,
                     (sts).encode('utf-8'),
                     hashlib.sha256).hexdigest()

# create and send the request
# the 'requests' package autmatically adds the required 'host' header
request_url = (endpoint + '/' +
               bucket + '/' +
               object_key + '?' +
               standardized_querystring_url_encoded +
               '&X-Amz-Signature=' +
               signature)

print(('request_url: {07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0}s' {07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0} request_url))

print(('\nSending `{07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0}s` request to IBM COS -----------------------' {07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0} http_method))
print(('Request URL = ' + request_url))
request = requests.get(request_url)

print('\nResponse from IBM COS ---------------------------------')
print(('Response code: {07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0}d\n' {07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0} request.status_code))
print((request.text))

# this information can be helpful in troubleshooting, or to better
# understand what goes into signature creation
print('These are the values used to construct this request.')
print('Request details -----------------------------------------')
print(('http_method: {07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0}s' {07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0} http_method))
print(('host: {07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0}s' {07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0} host))
print(('region: {07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0}s' {07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0} region))
print(('endpoint: {07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0}s' {07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0} endpoint))
print(('bucket: {07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0}s' {07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0} bucket))
print(('object_key: {07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0}s' {07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0} object_key))
print(('timestamp: {07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0}s' {07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0} timestamp))
print(('datestamp: {07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0}s' {07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0} datestamp))

print('Standardized request details ----------------------------')
print(('standardized_resource: {07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0}s' {07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0} standardized_resource_url_encoded))
print(('standardized_querystring: {07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0}s' {07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0} standardized_querystring_url_encoded))
print(('standardized_headers: {07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0}s' {07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0} standardized_headers))
print(('signed_headers: {07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0}s' {07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0} signed_headers))
print(('payload_hash: {07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0}s' {07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0} payload_hash))
print(('standardized_request: {07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0}s' {07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0} standardized_request))

print('String-to-sign details ----------------------------------')
print(('credential_scope: {07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0}s' {07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0} credential_scope))
print(('string-to-sign: {07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0}s' {07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0} sts))
print(('signature_key: {07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0}s' {07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0} signature_key))
print(('signature: {07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0}s' {07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0} signature))

print('Because the signature key has non-ASCII characters, it is')
print('necessary to create a hexadecimal digest for the purposes')
print('of checking against this example.')


def hex_hash(key, msg):
    return hmac.new(b'key', msg.encode('utf-8'), hashlib.sha256).hexdigest()


def createHexSignatureKey(key, datestamp, region, service):
    keyDate = hex_hash(('AWS4' + key).encode('utf-8'), datestamp)
    keyRegion = hex_hash(keyDate, region)
    keyService = hex_hash(keyRegion, service)
    keySigning = hex_hash(keyService, 'aws4_request')
    return keySigning


signature_key_hex = createHexSignatureKey(secret_key, datestamp, region, 's3')

print(('signature_key (hexidecimal): {07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0}s' {07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0} signature_key_hex))

print('Header details ------------------------------------------')
print(('pre-signed URL: {07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0}s' {07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0} request_url))

For raw code, refer this GitHub Gist link.

Pre-signed URL in the console

Once you pass the required elements in the placeholders provided and execute the code, you should see your pre-signed URL in the console.

 

You can set the expiration time in seconds. To experience the proper use of pre-signed URL, try this solution tutorial.

 

Technical Offering Manager & Polyglot Programmer | IBM Cloud

More Storage stories
May 1, 2019

Two Tutorials: Plan, Create, and Update Deployment Environments with Terraform

Multiple environments are pretty common in a project when building a solution. They support the different phases of the development cycle and the slight differences between the environments, like capacity, networking, credentials, and log verbosity. These two tutorials will show you how to manage the environments with Terraform.

Continue reading

April 29, 2019

Transforming Customer Experiences with AI Services (Part 1)

This is an experience from a recent customer engagement on transcribing customer conversations using IBM Watson AI services.

Continue reading

April 26, 2019

Analyze Logs and Monitor the Health of a Kubernetes Application with LogDNA and Sysdig

This post is an excerpt from a tutorial that shows how the IBM Log Analysis with LogDNA service can be used to configure and access logs of a Kubernetes application that is deployed on IBM Cloud.

Continue reading