Using the IBM Cloud Provider to Provision Infrastructure

Using the IBM Cloud Provider to Provision Infrastructure

The IBM Cloud Provider is a Terraform plugin that lets you provision and orchestrate IBM Cloud resources. Terraform is a popular opensource infrastructure as code (IaC) solution supporting all the leading public cloud providers. Terraform templates describe the “to be” state of the infrastructure you want. The terraform engine takes a template and does the orchestration needed to transform the “as is” state to the “to be”.

The IBM Cloud Provider supports virtual machines, load balancers, IBM Containers, Watson services, Cloudant, and more. Using the provider you can create templates to provision the entire Bluemix service catalog. This article looks at how a fictitious services company uses Terraform and the IBM Cloud Provider to deliver client solutions.

Our Scenario

JK Services is a vertical solution provider with a focus on marketing automation solutions. Their flagship solution is ADA, the Ad Delivery Analytics application. ADA is a cloud native application that leverages IBM Watson to tailor online advertising and content delivery to website and app users. Put simply, ADA is an AI ad server.

For security and marketing reasons JK customers get their own instance of ADA when they purchase the SaaS application. Provisioning all the cloud infrastructure for even one ADA implementation was a manual, error-prone, and lengthy process. Using the IBM Cloud Provider, JK is able to provision and manage the infrastructure of each client in a repeatable and automated way. In addition, JK can be confident that each instance of the ADA application has the same infrastructure and software components.

ADA Infrastructure

ADA – the Ad Delivery Analytics app

The ADA application uses a number of cloud resources:

• A kubernetes cluster deployed using IBM Containers is responsible for delivering content based on a visitor’s profile and preferences.

• A click handler farm of virtual machines logs all events and actions the user takes on the delivered content (e.g. clicking on an ad)

• The Cloudant NoSQL database stores visitor profile and preference information

• IBM Watson Personality Insights and IBM’s BigInsigts Hadoop cluster injest the raw event logs and populate the visitor profile and preference database

• Object Storage does log management and stores ad creatives and other delivered content

ADA Templates

Using the IBM Cloud Provider, JK Services built a number of Terraform templates to automatically provision the ADA infrastructure. Let’s look at different parts of their template…

IBM Cloud Provider Credentials

Since Terraform supports more than 50 plugin providers you have to explicitly say which provider you are using and give the necessary credentials for provisioning cloud resources.

provider "ibm" {<br>
  bluemix_api_key = "${var.bluemix_api_key}"<br>
  softlayer_username = "${var.softlayer_username}"<br>
  softlayer_api_key = "${var.softlayer_api_key}"<br>
}

This snippet shows the three (3) needed credentials to provision IBM Cloud resources:

• An IBM Bluemix API key

• A softlayer username

• A softlayer API key

You can use environment variables like SL_API_KEY to specify these credentials or you can use Terraform variables. We use variables in this example. For example, the pattern "${var.NAME}” references the variable “NAME”. Here we have variables bluemix_api_key, softlayer_username, and softlayer_api_key defined.

variable "bluemix_api_key" {}<br>
variable "softlayer_username" {}<br>
variable "softlayer_api_key" {}

Variables can optionally have descriptions and default values. We’ll see examples of these later.

IBM Containers Based Kubernetes Cluster

Serving ads and content is the responsibility of an ad server. The ad server runs on a Kubernetes cluster implemented with IBM Containers and has to integrate with the profiles and preferences database.

Here’s the template definition for the ad server cluster:

resource "ibm_container_cluster" "ad_server_cluster" {<br>
  count        = "${var.create_ad_server}"<p></p>
<p>  name         = "ad-server-cluster-${random_id.name.hex}"<br>
  datacenter   = "${var.datacenter}"<br>
  org_guid     = "${data.ibm_org.org.id}"<br>
  space_guid   = "${data.ibm_space.space.id}"<br>
  account_guid = "${data.ibm_account.account.id}"<br>
  no_subnet    = true<br>
  subnet_id    = ["${var.subnet_id}"]</p>
<p>  workers = [<br>
    {<br>
      name   = "worker1"<br>
      action = "add"<br>
    },<br>
    {<br>
      name   = "worker2"<br>
      action = "add"<br>
    },<br>
    {<br>
      name   = "worker3"<br>
      action = "add"<br>
    },<br>
    {<br>
      name   = "worker4"<br>
      action = "add"<br>
    },<br>
    {<br>
      name   = "worker5"<br>
      action = "add"<br>
    },<br>
  ]</p>
<p>  machine_type    = "${var.machine_type}"<br>
  isolation       = "${var.isolation}"<br>
  public_vlan_id  = "${var.public_vlan_id}"<br>
  private_vlan_id = "${var.private_vlan_id}"<br>
}</p>

We can see that ad_server_cluster is specified as an ibm_container_cluster resource. Parts of the specification are hard-coded. For example, the workers array specifically says this is a 5-node cluster. Other parts of the spec use user-supplied values. For example, the datacenter where the cluster is deployed and the type of machine are specified using the variables datacenter and machine_type respectively.

variable "datacenter" {<br>
    description = "Datacenter location for the cluster"<br>
    default = "dal12"<br>
}<br>
variable "machine_type" {<br>
    description = "Cluster node machine type"<br>
    default = "u1c.2x4"<br>
}

Integrating with the Cloudant NoSQL Database

Cloudant database instances are available as a service in the IBM Bluemix catalog. The IBM Cloud Provider uses the service_resource to specify any and all Bluemix services. Describing a service just needs the name and billing information for the service:

resource "ibm_service_instance" "profiledb" {<br>
  name       = "profiledb-${random_id.name.hex}"<br>
  space_guid = "${data.ibm_space.space.id}"<br>
  service    = "cloudantNoSQLDB"<br>
  plan       = "Lite"<br>
  tags       = ["adserver"]<br>
}

Here we are using the free (“Lite”) plan for Cloudant. As stated above, specifying a Watson service like Personality Insights would be exactly the same. The only difference would be the name of the service and the plan being used.

NOTE: One of the things you might have noticed is that all resources have a name. Since JK Services is building out the same infrastructure for each customer they need to use unique names for all the resources. For example, the Cloudant database will have a name like “profiledb-A4By” because a variable random_id is used as a suffix on all resource names.

IBM Container clusters can use any IBM Service via a service binding. The service binding makes connection credentials available in the cluster’s environment. The IBM Cloud Provider resource for this is aptly called ibm_container_bind_service.

resource "ibm_container_bind_service" "profiledb_bind_service" {<br>
  cluster_name_id             = "${ibm_container_cluster.ad_server_cluster.name}"<br>
  service_instance_space_guid = "${data.ibm_space.space.id}"<br>
  service_instance_name_id    = "${ibm_service_instance.profiledb.id}"<br>
  namespace_id                = "default"<br>
  org_guid                    = "${data.ibm_org.org.id}"<br>
  space_guid                  = "${data.ibm_space.space.id}"<br>
  account_guid                = "${data.ibm_account.account.id}"<br>
}<

The profiledb_bind_service references the cluster’s name using the cluster resource directly (${ibm_container_cluster.ad_server_cluster.name}). The same is done for the Cloudant database service. You can reference any resource using Terraform’s ${TYPE.NAME.ATTRIBUTE} interpolation syntax.

Planning & Applying the Template

Terraform supports two separate actions with templates: plan and apply. “Plan” is a dry run. When you do a terraform plan you get back a report on the resources that needed to be created, updated, or deleted based on the template definitions and the current state of the infrastructure.

Here’s a snippet from a terraform plan report:

~/dev/ADA $ terraform plan<br>
Refreshing Terraform state in-memory prior to plan...<br>
The refreshed state will be used to calculate this plan, but will not be<br>
persisted to local or remote state storage.<p></p>
<p>data.ibm_org.org: Refreshing state...<br>
data.ibm_space.space: Refreshing state...<br>
data.ibm_account.account: Refreshing state...<br>
The Terraform execution plan has been generated and is shown below.<br>
Resources are shown in alphabetical order for quick scanning. Green resources<br>
will be created (or destroyed and then created if an existing resource<br>
exists), yellow resources are being changed in-place, and red resources<br>
will be destroyed. Cyan entries are data sources to be read.</p>
<p>Note: You didn't specify an "-out" parameter to save this plan, so when<br>
"apply" is called, Terraform can't guarantee this is what will execute.</p>
<p>+ ibm_compute_autoscale_group.sample-http-cluster<br>
    cooldown:                                                 "30"<br>
    health_check.%:                                           "1"<br>
    health_check.type:                                        "HTTP"<br>
    maximum_member_count:                                     "10"<br>
    minimum_member_count:                                     "1"<br>
    name:                                                     "${var.auto-scale-name}-${random_id.name.hex}"<br>
    port:                                                     "80"<br>
    regional_group:                                           "as-sgp-central-1"<br>
    termination_policy:                                       "CLOSEST_TO_NEXT_CHARGE"<br>
    virtual_guest_member_template.#:                          "1"<br>
    virtual_guest_member_template.0.block_storage_ids.#:      "&lt;computed&gt;"<br>
    virtual_guest_member_template.0.cores:                    "1"</p>
<p>    ...</p>

The terraform apply action is what takes the as-is state of the infrastructure and transforms it to the to-be state as described by the template. The apply action can take some time depending on the number and type of resources you are provisioning.

Summary

Using the IBM Cloud Provider JK Services is able to specify their infrastructure using simple text file templates and leverage the Terraform engine to manage their deployment in a repeatable and automated way. The IBM Cloud Provider supports the entire Bluemix services catalog, IBM Containers, virtual machines, and other storage, compute, and network resources.

Resources to Learn More

• Source to the JK Services ADA template

• IBM Cloud Schematics – IBM’s  “Infrastructure as Code as a Service” built on Terraform and leveraging the IBM Cloud Provider