Share this post:
IBM Cloud Private in a multicloud world
IBM Cloud Private runs on public clouds, including Amazon AWS. These best practices are based on our experience with multicloud deployments.
You may have interpreted “private cloud” as a computing model that runs inside a data center, behind a corporate firewall, or on-premise for reasons such as data locality and easier access to traditional applications. While those reasons are certainly valid, the value of IBM Cloud Private (ICP) goes beyond this definition of private cloud. ICP offers cloud-computing with greater control, predictable costs, tighter security, and flexible management capability dedicated to one organization. With the introduction of IBM Multicloud Manager, you decide where to run and deploy your cloud applications, including non-IBM cloud platforms like Amazon Web Services (AWS).
Kubernetes has become the de facto standard in managing container-based workloads. IBM Cloud Private builds on top of the Kubernetes orchestrator and provides a private cloud platform for developing and running a container workload solely for one organization. It is a reliable way of running containerized IBM middleware given the knowledge and verification test applied to both.
Software is software, capable of running anywhere supporting it (such as the virtualization layer). Of course, you can run IBM Private Cloud on-premise, but you can equally run it on other public clouds, such as Amazon’s AWS. To achieve the best outcome of running IBM Cloud Private on AWS, proper planning and operation best practices should be applied. In this blog series, we are going to share some of these practices based on proven experiences of working with clients.
Typical scenarios of running IBM Cloud Private on Amazon AWS
Below are four “why” scenarios for clients considering IBM Cloud Private running on AWS infrastructure:
- Multicloud strategy: Customers would like to leverage the strength and unique offerings from different cloud vendors, but also want to have a consistent operation and runtime environment that they can achieve portability without cloud platform lock-in. For example, running digital innovation on both IBM public cloud and AWS. IBM Multicloud Manager is the best way to develop a successful multicloud strategy.
- Cloud bursting: If you have private cloud environments running on-prem and would like to expand the cluster or private cloud to external infrastructure only in certain special condition or bursting workload.
- Disaster recovery: Since the same workload can be easily and quickly provisioned, external cloud providers can also be a great place to act as a disaster recovery data center. IBM Cloud Private on AWS fits this use case nicely.
- Existing AWS users with IBM Middleware workload: IBM middleware investments are further extended with an app modernization strategy. IBM’s middleware also benefits from associated data analytic services, such as IBM Cognos Analytics to gain a deeper understanding of your business and IBM SPSS statistical analysis to predict outcomes; you can continue deploying them in your AWS environment by leveraging ICP on AWS. This approach gives the best of both worlds.
We’ll cover these topics as part of the series; below are the ones we’ll cover in this installment:
- The “big picture” of running IBM Cloud Private on AWS
- Planning decisions:
- Define a private network for IBM Cloud Private
- Choose the right AWS compute resources
- Assure high availability
- Determine how many clusters are required
- Guidelines for network security and isolation
- Configuring load balancers
- Choose the storage strategy for IBM Cloud Private
- Configure DNS for worker-to-master node communication
- Scaling IBM Cloud Private clusters on AWS
- What’s next
The “big picture” of running IBM Cloud Private on AWS
The following diagram depicts one reference architecture of running IBM Cloud Private on AWS. We’ll then dive into the details of how we came up the solution in the Planning Decisions section.
IBM Cloud Private has four main classes of nodes: boot, master, worker, and proxy. You can optionally specify management, vulnerability advisor, and etc nodes in your cluster. For ICP architecture in detail, please reference this document.
A solid production deployment depends on carefully answering some of the key architectural decisions that cover aspects including high availability, security, automation, DevOps, monitoring, etc. I’ll go through some of the key decisions to help later build a foundation of the IBM Cloud Private (ICP) deployment on AWS.
Define a private network for IBM Cloud Private
Amazon Virtual Private Cloud (Amazon VPC) lets you provision a logically isolated section of the AWS Cloud where you can launch AWS resources in a virtual network that you define. You have complete control over your virtual networking environment, including selection of your own IP address range, creation of subnets, and configuration of route tables and network gateways. It is the foundation of establishing a private cloud environment on AWS.
IBM Cloud Private organizes and orchestrates workload in the concept of cluster. An ICP cluster is made of multiple components, including master, management, proxy, and worker nodes. All of these components are recommended to be in a single VPC, spreading over multiple availability zones (which will be described in detail in the following section).
IBM Cloud Private, based on Kubernetes, has its own networking layer. It is implemented as CNI plugins, specifically Calico implementation. VPC takes care of the host (VM) communication within a private network, while Calico handles the communication between ICP Pods (collection of containers making up the Kubernetes operation unit). VPC and IBM Cloud Private networking work together nicely to provide you a truly isolated private networking on top of a public cloud platform.
For IBM Cloud Private cluster VPC, we suggest configuring an internet gateway and putting all ICP nodes (EC2 instances) in private subnets, with its own NAT gateway. This approach allows us to put all IBM Cloud Private nodes behind firewall in a private network.
Choose the right compute resources for IBM Cloud Private
IBM Cloud Private runs on either bare metal or VM. AWS EC2 becomes the natural fit as IBM Cloud Private hosts. There are a couple of things you need to consider when planning your EC2 instances.
First, plan the size and number of your EC2 nodes. This takes into consideration the different ICP components that may pose different size requirements for the underneath EC2 instances. The following table can be used as a reference:
Worker> 3m4.xlarge (or larger)privateicp-default
||AWS EC2 Instance Type
||0 (optional, not required for cluster operation)
Worker node is where your application workload (cloud-native app, middleware components, or data) runs, so plan the size and type accordingly based on workload nature. Don’t forget to run performance benchmarks to confirm the appropriate sizing.
For resiliency, the master and proxy nodes should have AWS elastic network interfaces created and attached to them as the first network device. The private IP address is bound to the network interface, so when the interface is attached to a newly created instance, the IP address is preserved.
You also need to decide what type of AWS AMI (Amazon Machine Image) to use. Starting with the Host OS type, IBM Cloud Private supports various types of Linux distribution (Ubuntu, RHEL, and Suse). Choose the one approved for your company. Then, I suggest customizing the base OS AMI to pre-install Docker and pre-load ICP component images on the selected AMI. Doing so will shorten the ICP installation process, particularly when used in the automated installation process I will introduce later.
Assure high availability
High availability is a complex topic, but I’ll briefly introduce some important guidelines. Under the context of AWS, one simple rule to achieve HA is to spread key IBM Cloud Private components to multiple AWS availability zones (AZ) under a single AWS region.
Looking back at the reference architecture diagram above, ICP master, proxy, worker, and management nodes are deployed to two different AZs—this is fully supported by ICP. Access to these cross-AZ components is mainly handled by AWS load balancer or IBM Cloud Private proxies. I’ll discuss how to configure the load balancer later.
I typically do not recommend configuring an IBM Cloud Private cluster across regions. If you need to achieve a higher HA or plan for Disaster Recovery (DR) using multiple regions, you might consider a multicluster approach.
Determine how many clusters are required
You need to take into consideration the isolation and operation efficiency when deciding how many IBM Cloud Private clusters are needed. I suggest having at least two ICP clusters—one for non-production and one for production. Separating production clusters provides isolation needed for critical workload. Within each cluster, you should leverage Kubernetes namespaces to isolate the development, integration, or even QA workload.
To achieve higher isolation, you can consider separating the lower environment and production environments into their own AWS account.
Guidelines for network security and isolation
When deploying ICP to public cloud infrastructure like those on AWS, you need to pay extra attention to networking security and isolation.
Each cluster should have their own VPC, either under the same AWS account or on separate accounts. We suggest putting all ICP nodes in a private network, blocking direct access to the nodes from outside. The only access into your cluster is via either the load balancer or through AWS Bastion hosts for ssh and installation scripts.
We suggest configuring a set of AWS security groups to control the networking flow into or between the IBM Cloud Private components. We’ll discuss the detail in coming blogs.
Configuring load balancers
To achieve high availability (HA), ICP leverages multiple nodes for all components, and they are spread across multi-AZs as discussed earlier. To access the components and applications deployed on worker nodes, you’ll need to use load balancers.
We suggest having at least two load balancers to distribute traffic for IBM Cloud Private cluster management:
- Management traffic (including ICP console, API, and cli)
- Application Ingress resources (through ICP proxy node)
Both load balancers should be AWS Network load balancers for better performance and port management flexibility. The following are sample port configurations for the two load balancers:
- Network LoadBalancer for ICP console:
- Listen on 8443, forward to master nodes port 8443 (ICP Console)
- Listen on 8001, forward to master nodes port 8001 (Kubernetes API)
- Listen on 8500, forward to master nodes port 8500 (Image registry)
- Listen on 9443, forward to master nodes port 9443 (Auth service)
- Network LoadBalancer for ICP Ingress resources:
- Listen on port 80, forward to proxy nodes port 80 (http)
- Listen on port 443, forward to proxy nodes port 443 (https)
Additionally, Kubernetes AWS Cloud Provider integration can be used to dynamically create load balancers to forward traffic directly to worker nodes. This provides the best performance because each application has a dedicated load balancer and client traffic is not shared. The caveat is that you could potentially end up with many load balancers and elastic IPs, which would increase the overall cost.
Choose the storage strategy for IBM Cloud Private
There are different kinds of storage requirements in an IBM Cloud Private cluster. ICP relies on persistent storage to maintain its private (Docker) registry and audit logs. This is a cross-cluster, cross-AZ requirement, so I’d suggest using AWS elastic file system (EFS).
For application persistence, IBM Cloud Private standardizes on Kubernetes PV (Persistence Volume) approach. ICP supports AWS Elastic Block Store (EBS) volumes. EBS volumes can be made available to applications using a storage class, and dynamically requested by application users. The platform will handle dynamic provisioning and attaching the volume to the correct worker node running the application.
Configure DNS for worker-to-master node communication
For convenience, you can configure your IBM Cloud Private cluster with a private DNS Zone so that the worker nodes can talk to the master as well as discover other ICP components. In AWS, you can use Amazon Route53 to create the DNS zone.
Scaling IBM Cloud Private clusters on AWS
Scaling the IBM Cloud Private cluster in AWS is an evolving space. One solution we suggest is to leverage the AWS auto-scaling group along with some Lambda function to handle the actual scaling.
As part of the installation, you create an auto-scaling group that has the same configuration as the IBM Cloud Private worker nodes. Then develop a Lambda function that responds to auto-scaling events. Upon auto-scaling event received, the Lambda function creates Kubernetes jobs in ICP that add and remove worker nodes from the cluster using the ICP installer image. So far, this approach assumes manual trigger of scaling up and down the cluster.
What’s the recommended approach for installation? Automate, automate, and automate the installation! ICP has simplified Kubernetes and other component installations with Ansible-based installer. On AWS, we further enhanced the automation using Terraform along with Jenkins-based pipeline. The result is a fully automated installation approach that takes care of the cloud platform provisioning (such as VPC and EC2 instance provision) as well as the ICP product installation and basic post-installation configuration.
We will dive deeper into the installation in the next installment of this series.
Making these critical decisions ensures your successful journey of IBM Cloud Private on AWS. The IBM Cloud Architecture and Solution Engineering (CASE) team has been sharing their experience to provide you with a comprehensive guide. You can find out more about the practices of operating IBM Cloud Private and other architectures in the IBM Garage Method Architecture Center.
In coming installments, my colleagues Jeffrey Kwong, Hans Moen, and Hollis Chui will do deep dive on installation and operation with Terraform, networking, security and other infrastructure considerations, and DevOps. In the meantime:
Stay tuned for the next installment!