How-tos

CI/CD Pipeline for OpenWhisk Functions Using Whisk Deploy

Share this post:

CI/CD Pipeline for OpenWhisk Functions Using Whisk Deploy

Teams typically develop OpenWhisk functions using IBM Cloud‘s user interface. They manually catalog function code in a source repository to prevent accidental loss of code. However, there could be a situation where the runtime code becomes out-of-sync with cataloged code. To prevent this situation, teams require a continuous integration/continuous deployment pipeline. In this article, I present a technique using an emerging technology called Whisk Deploy.

Whisk Deploy provides capabilities for function definition and definitions for binding functions to triggers using a simple YAML-based manifest file. It also provides a mechanism for the deployment of the functions in the cloud. I’m going to show you how to create a DevOps pipeline using the IBM Cloud and enable such a pipeline for continuous deployment of function code changes.

What we will do in this tutorial

  • Create an API key for accessing the IBM Cloud API from within the deployment automation scripts.
  • Setup a DevOps pipeline in IBM Cloud that uses GitHub for source control and Delivery Pipeline for build/deploy of artifacts using Whisk Deploy.
  • Create a deployment script that describes the steps to install Whisk Deploy and invoke it for deployment of the functions as described in its manifest file. This is needed because the default container image available in the Delivery Pipeline stage does not contain the executable for it.
  • Setup an IBM Message Hub instance and create a topic in it. Messages sent to this topic will fire a trigger that will invoke the OpenWhisk function.
  • Author a “Hello, World”-type OpenWhisk function that will use the NodeJS runtime. Two input parameters—a name and a pre-defined greeting—will be provided to the function. The function will combine the greeting with the provided name and return that as its response. The function will be triggered when a message is posted to a topic on the IBM Message Hub.
  • Source control this function’s code along with a Whisk Deploy manifest file that will contain the definition of the function, a trigger for IBM Message Hub, and a rule that will bind the function with its trigger.
  • Push changes in the source code repository to demonstrate deployment of the function and its trigger via an automated deployment pipeline.

This article assumes that you already have access to IBM Cloud, are familiar with its command-line, and have already installed it on your favorite shell. Additionally, it assumes you have an active account on Github.com. Furthermore, it is assumed that you know how OpenWhisk functions work and how to manually create them in the IBM Cloud.

1. Create a Platform API key for accessing IBM Cloud API

  • Navigate to https://console.bluemix.net/catalog/ and log into the IBM Cloud using your credentials.
  • Under the Manage drop-down menu, locate Security and click on the Platform API keys menu option.
  • On the Platform API keys page, click Create to create a new key.
  • In the presented dialog, enter “deployer” as the name of the key and “Deployment Automation API Key” as the description of the key. Click Create to create the key.
  • View and Download the key JSON file.
    IMPORTANT: Record this key’s value since it will be needed during creation of the CI/CD pipeline. The key’s value is also available under the apiKey property in the downloaded key JSON file.
  • Close the dialog to return to the landing page of Platform API Keys.
  • Confirm that you have an API key as indicated above.

2. Create a DevOps toolchain in the IBM Cloud

  • Click on the menu icon on the top left and select DevOps.
    DevOps menu option within IBM Cloud
  • Ensure that the correct Region and Cloud Foundry Org are selected.
  • Click the Create a Toolchain button on the top right.
    Create a toolchain button
  • Click on Build your own toolchain under Other Templates.
    Build your own toolchain template
  • Provide an appropriate name to the toolchain and ensure that the correct region and organization are still selected.Toolchain description
  • Click Create.

3. Add a source control tool to the toolchain

  • Typically the Create button will bring you to the visual editor of the toolchain. However, if it does not, follow the first three steps from Section 2 and select the toolchain created on the sixth step.
  • Click on the Add a Tool button to add a tool to the toolchain.
    Add a tool to the toolchain
  • The first tool present in the toolchain is typically a source code repository, such as GitHub or Gitlab, that catalogs the source code of the function. For the purposes of this article, I am using a GitHub-based source code repository and will add an integration to it from within the toolchain. In the search bar of the Add Tool Integration, search for “GitHub.”Search for GitHub to add to toolchain
  • Click on GitHub to select it.
    Add GitHub to toolchain
  • On the next page, click the Create Integration button to add GitHub to your toolchain.
    Configure GitHub integration

4. Configure source code repository

Once GitHub has been added to the toolchain, the final step of Section 3 will bring you back to the toolchain’s Overview page.
This page visualizes all the tools enabled within your toolchain as individual tiles (currently we only see GitHub as a tool enabled under the CODE tile but as we add more tools to our toolchain, we will see them here).

  • Click the Configure button on the GitHub tile to configure it.
    Configure GitHub Integration
  • Click the Authorize button to authorize IBM Cloud to access your source code repository on GitHub.
  • Enter your GitHub username and password and click Sign in.
    Sign in to GitHub
  • Review authorizations and click Authorize IBM-Cloud to allow IBM Cloud to access your GitHub repositories.
    Enable GitHub access to IBM Cloud
  • Completing the previous step will bring you to a page that allows you to provide information about your GitHub repository. For the purposes of this article, I am assuming that we are creating a new source code repository on GitHub (this is indicated by New under the Repository Type field). However, if your team is already using GitHub to catalog its OpenWhisk functions in another repository, IBM Cloud allows you to either clone or fork that repository (enable this by selecting the appropriate option under Repository Type and providing details such as repository URL, name of the repository, etc., about your source code repository). While it is not required, I am assuming that we will be using this new repository for issue management as well. As a result, I have checked the Enable GitHub Issues option. Additionally, I have also selected the option for IBM Cloud to Track deployment of code changes on this repository.
    GitHub repository details
  • Save the integration. If you also selected the box to enable GitHub for issues, you will notice a new tile under THINK has been added to the toolchain that points to GitHub for issues.

5. Add a deployment tool to the toolchain

  • Click on the Add a Tool button to add a tool to the toolchain.
  • In the search bar, type “Delivery Pipeline.”
  • From the results, select the Delivery Pipeline tool to add to the toolchain.
  • Provide a suitable name for the Pipeline name, leave the option to Show apps in the View app menu, and click on Create Integration to enable the integration.
  • Upon successful save, you will notice a third tool added under DELIVER to the toolchain, as shown below.

6. Add deployment stage to delivery pipeline

  • On the toolchain’s Overview page, click on the Delivery Pipeline tile.
  • On the Delivery Pipeline landing page, click on Add Stage to add the first stage to the pipeline.
  • Provide “Deploy” as the name of the stage.
  • On the Input tab, validate Git repository for Input type. Ensure that the Git repository references the correct name of the repository and the Git URL points to the right URL to the repository (normally all of these fields are automatically pre-populated). Deployment happens off of the “master” branch in this example. However, a branch other than “master” can be specified by selecting it from the list of branches.
  • Switch to the JOBS tab and click on the ADD JOB button.
  • Select Job Type as Build and Builder type as Shell Script.
  • Under Build Script, paste the following script:

[sourcecode language=”javascript”]
#!/bin/bash
echo ">>> Downloading Whisk Deploy…"
wget https://github.com/apache/incubator-openwhisk-wskdeploy/releases/download/latest/openwhisk_wskdeploy-latest-linux-amd64.tgz
echo ">>> Installing Whisk Deploy…"
mkdir openwhisk
tar -zxvf ./openwhisk_wskdeploy-latest-linux-amd64.tgz -C ./openwhisk
rm -rd ./openwhisk_wskdeploy-latest-linux-amd64.tgz
echo ">>> Checking installed version of Whisk Deploy…"
./openwhisk/wskdeploy version
echo ">>> Checking installed version of IBM Cloud CLI…"
bx –version
echo ">>> Logging into IBM Cloud…"
bx login –apikey $DEPLOYER_API_KEY -a api.ng.bluemix.net -o "$ORG" -s "$SPACE"
echo ">>> Replacing FEED_BASE In Manifest File…"
sed -i "s/@FEED_BASE@/$FEED_BASE/g" manifest.yaml
echo ">>> Contents Of Manifest File:"
cat manifest.yaml
echo ">>> Currently Deployed Packages:"
bx wsk package list
echo ">>> Currently Deployed Actions:"
bx wsk action list
echo ">>> Currently Deployed Triggers:"
bx wsk trigger list
echo ">>> Currently Deployed Rules:"
bx wsk rule list
echo ">>> Deploying Actions Using WhiskDeploy…"
./openwhisk/wskdeploy -p .
echo ">>> Successfully Deployed Actions Using WhiskDeploy."
[/sourcecode]

  • Set all other fields as following:
    • Working directory: empty
    • Build archive directory: empty
    • Enable test report: unchecked
    • Enable code coverage report: unchecked
    • Stop running this stage if this job fails: checked
  • Switch to the ENVIRONMENT PROPERTIES tab and click ADD PROPERTY to add the following properties:
Property Type Property Name Property Value
Secure Property DEPLOYER_API_KEY <Value of the API Key as noted in the fifth step of Section 1>
Text Property ORG <Name of the IBM Cloud Organization>
Text Property SPACE <Name of the IBM Cloud Space>
  • Save the stage.

7. Create IBM Message Hub

  • Click on Catalog and type “Message Hub” in the search filter bar. Click on the Message Hub tile to create a new instance of an IBM Message Hub.
  • Provide a suitable name for the message hub instance and select an IBM data center and appropriate resource group on the account.
  • Select an appropriate Pricing Plan and click Create.
  • Wait for message hub instance to be provisioned.
  • Once provisioned, the landing page of the message hub instance will open. The hub’s landing page allows management of the instance’s topics and bridges.

 

8. Create service credentials in IBM Message Hub

  • Click on Service credentials on the IBM Message Hub instance’s landing page.
  • Click on the New credential button under Service credentials.
  • Provide Name, Role, and Service ID (select Auto Generate from the drop-down list). Then, click Add to create new credentials.
  • Verify that service credentials were successfully created by clicking on View credentials.

9. Create a topic on IBM Message Hub instance

  • Click on Manage on the IBM Message Hub instance’s landing page.
  • Click on the + (plus) icon to create a new topic.
  • Give a name, number of partitions, and retention period to the new topic. Click Create topic.
  • Ensure that the new topic is successfully created.

10. Bind Message Hub coordinates and credentials within IBM Cloud Functions

  • The steps in this section involve use of the IBM Cloud command-line that you have already installed on your favorite terminal. If the command-line is not already installed, you can install it here.
  • Open up your favorite terminal and log into the IBM Cloud using the command-line options as indicated in the following document. Ensure that the correct organization and space is provided with login.
  • Install the IBM Cloud Functions plugin using the following document.
  • In the terminal window, run the following command to create a binding to the IBM Message Hub instance provisioned in Section 7: ibmcloud wsk package refresh
    ibmcloud wsk package refresh
    '_' refreshed successfully
    created bindings:
    updated bindings:
    Bluemix_Message Hub_MessageHubCredentials
    deleted bindings:
  • The previous step creates a new action that binds the credentials and other relevant details of the IBM Message Hub instance (created in Sections 7 and 8) for use by IBM Cloud Functions. This allows other actions and triggers to interact with the IBM Message Hub instance seamlessly without providing any of these connection and authentication/authorization parameters. Once bound, the new action is visible within the IBM Cloud Functions user interface as following:
  • Go back to the Deploy stage of the delivery pipeline (as created in Section 6) and add one more environment property to it:
Property Type Property Name Property Value
Text Property FEED_BASE Name of the binding action to the IBM Message Hub created above. For e.g., Bluemix_Message Hub _MessageHubCredentials

11. Add function script and WhiskDeploy manifest to source control

  • Clone the git repository created under the fifth step of Section 4 to your workstation.
  • Switch to a branch as specified in the fourth step of Section 6. Note that the assumption here is that it’s okay to directly push changes to this branch. However, software development best practices require all changes made to source code to be cataloged in a private branch and a pull request created to integrate those with a mainline branch, such as the master branch, after they have been reviewed for accuracy and content.
  • Under the src folder of the repository, add the following sample OpenWhisk function script: hello_function.js

[sourcecode language=”javascript”]
/**
*
* main() will be run when you invoke this action
*
* @param Cloud Functions actions accept a single parameter, which must be a JSON object.
*
* @return The output of this action, which must be a JSON object.
*
*/
‘use strict’;
function main(params) {
console.log(‘Hello function called with: ‘ + JSON.stringify(params));
var greeting = params.greeting;
var user = params.user;
var message = greeting + ", " + user;
return {
"Message": message,
"Status": "Success",
"Code": 200
}
}
[/sourcecode]

  • Under the root of the repository, add the following WhiskDeploy manifest file: manifest.yaml

[sourcecode language=”javascript”]
packages:
hello_package:
actions:
say_hello:
location: src/hello_function.js
runtime: nodejs:6
inputs:
greeting: "Hello"
user: "World"
triggers:
greeting_trigger:
feed: @FEED_BASE@/messageHubFeed
inputs:
isJSONData: string
topic: GreetAll
rules:
greeting_rule:
action: say_hello
trigger: greeting_trigger
[/sourcecode]

The manifest file describes a package called hello_package that contains three main sections: actions, triggers, and rules. The actions section describes a single action called say_hello that references the OpenWhisk function script created in the third step of this section. The runtime of the action is NodeJS and the action expects two input parameters called greeting and user. Default values are described for these parameters but those values can be overridden by supplying new values to the trigger that invokes the action. The triggers section describes a trigger called greeting_trigger that is initiated when a new message is posted to the IBM Message Hub instance created in previous sections. Notice that the feed property describes a token called @FEED_BASE@ that is replaced with the name of the binding action as created in the fourth step of Section 10. The token value is replaced at runtime within the build/deployment script as defined under the seventh step of Section 6. The trigger’s definition also describes that it is fired when a message is posted to the IBM Message Hub topic called GreetAll. You will recall that this topic added to the IBM Message Hub in Section 9. Finally, the rules section of the manifest brings together the action and its trigger by a rule called greeting_rule.

  • Add and push these changes to the remote git repository.
  • Confirm that the delivery pipeline now creating actions and triggers in the IBM Cloud Functions is using the code that is source controlled in your GitHub repository.
  • Now, every time a change is applied to the source code of the function and pushed into the described branch of the GitHub repository, the delivery pipeline will run and update the IBM Cloud Functions with new content.
  • Test the trigger by sending a POST request to the trigger’s URL. Look up the URL to the trigger by navigating to the Triggers section of the IBM Cloud Functions, selecting the greeting_trigger created by the delivery pipeline using WhiskDeploy, and selecting the Endpoint section.

Wrapping up

We now have a continuous integration/continuous deployment pipeline for our small OpenWhisk function. Anytime a change is made to this function and pushed in the branch that is being tracked by the delivery pipeline, the function will be redeployed in the IBM Cloud. This allows us to have a standard mechanism for pushing new changes to the function code. Finally, the use of Whisk-Deploy-based automation makes function deployment a consistent and well-orchestrated activity that all developers on the team will be able to consume. The team will no longer have to rely on a single developer’s abilities to create the functions and bind them to their triggers, nor will they have the fear of losing precious function code as it will always be cataloged within the source code repository.

Additional References

Learn more about OpenWhisk and Whisk Deploy by exploring these online resources:

Senior Software Engineer

More How-tos stories
September 20, 2018

Deploying to IBM Cloud Private 2.1.0.3 with IBM Cloud Developer Tools CLI

IBM Cloud Developer Tools CLI version 2.1.4 adds deployment support for IBM Cloud Private 2.1.0.3. This version of IBM Cloud Private uses a more secure Helm for Kubernetes deployments and simplifies the cluster configuration for the client compared to prior IBM Cloud Private releases.

Continue reading

September 19, 2018

Tutorial: Apply End-to-End Security to Cloud Applications

A new tutorial will show you how to use IBM Cloud services to secure your cloud application. Capture and review security-related events, encrypt storage, integrate authentication, and more.

Continue reading

September 18, 2018

How To Convert CSV Objects to Parquet in Cloud Object Storage

If you're looking to lower storage costs by compressing your data and get better query performance when querying the data in Cloud Object Storage, you may want to click to learn how to convert CSV objects to Parquet.

Continue reading