Secure microservice-to-microservice communication across Kubernetes clusters using a private network

Share this post:

Deploying microservices works great inside a single Kubernetes cluster. But what if your microservices need to communicate with other microservices that are deployed in a separate cluster? And how can you connect those microservices using an encrypted connection through a private network for added security? This how-to presents the steps for the IBM Cloud Container Service (ICCS).

What you’ll do

This how-to is divided into several sections:

• Setup: You’ll enable VLAN spanning, deploy two standard clusters, and enable Private Application Load Balancer.
• Deploy: You’ll deploy the sample Bluecompute application services across different clusters and validate the installation.
• Secure: You’ll enable TLS for the Bluecompute application service.

Before your start

First sign up for IBM Cloud account and then install the following CLIs:

• IBM Cloud CLI
• kubectl
• helm

Then clone the refarch-cloudnative-kubernetes repository and cd to cluster-to-cluster:

$ git clone https://github.com/ibm-cloud-architecture/refarch-cloudnative-kubernetes
$ cd refarch-cloudnative-kubernetes/cluster-to-cluster

Tip: If you’re new to Kubernetes or microservices, consider completing the Deploy a microservices app on IBM Cloud by using Kubernetes tutorial before continuing.

Topology of the microservices components

The application is a simple storefront shopping application that displays a catalog of antique computing devices, where users can search and buy products. For the purpose of this how-to, we’ve split the web services and orders services into separate clusters that will then communicate with each other over a private network.

(Click to expand)

The diagram above shows a modified version of our Microservices Reference Architecture Application, better known as bluecompute, which you can learn more about here. This application is deployed using a single Helm Chart, the bluecompute-ce chart, which contains multiple dependency charts representing all the different microservices. In this how-to blog, we are going to deploy the bluecompute-ce chart as follows:

• Deploy the orders chart, which contains ibmcase-orders-mysql as a dependency chart, into one cluster, which we are going to call orders-cluster.
• Deploy the bluecompute-ce, which does not contain neither orders nor its dependent ibmcase-mysql chart, into a separate cluster, which we are going to call web-cluster.
• The orders service in the orders-cluster will be exposed privately to the web service in the web-cluster via a Private Application Load Balancer (ALB).

To wrap up, we will be enabling TLS on the orders-cluster Private ALB to add an extra layer of security.

Setup and deploy the services in two clusters

In order for the clusters to talk to each other via Private Network, we need to turn on VLAN Spanning in your Infrastructure account. Open a browser window and use the following instructions to turn on VLAN spanning.

Deploy standard clusters

For this demo, we are going to create two standard clusters. Let’s call the first cluster orders-cluster and deploy it in the us-south-dal12 location; we will deploy the orders chart, which includes the orders and ibmcase-orders-mysql charts. The second cluster is called web-cluster and deploy it in us-south-dal10 location; we will deploy the bluecompute-ce chart, which includes the charts for the rest of the microservices, minus the orders chart.

To provision the two standard clusters, follow these instructions. Once the two clusters are provisioned, do the following:

• Open two terminal windows.
• In the first terminal window, download the Kubernetes cluster context for the orders-cluster, and then execute the helm init command.
• In the second terminal window, download the Kubernetes cluster context for the web-cluster, and then
  execute the helm init command.

Since we are going to use kubectl and helm on two clusters, the above instructions keep the context for each cluster in separate terminal windows, which prevents a constant back and forth between them.

Enable Private Application Load Balancer (ALB)

The Private Application Load Balancer (ALB) that comes with ICCS Standard clusters can be used as a custom Ingress Controller that exposes Kubernetes services over the private network only.

For this example, we only need to enable the Private ALB for the orders-cluster. This is so that the web application in the web-cluster can access the orders microservice in the orders-cluster over the private network. Follow the instructions below to enable the Private ALB:

• Open the orders-cluster terminal window.
• Use the instructions on this link to enable private load balancers.

Deploy the orders chart in the orders-cluster

Let’s deploy the orders chart in the orders-cluster and expose it to the web-cluster through the Private ALB. To do so, follow the instructions below:

• Go to the orders-cluster terminal window.
• Get the ALB ID for the Private ALB:

  $ bx cs albs --cluster <orders-cluster-name>
  
  OK
  ALB ID                                                Enabled   Status     Type      ALB IP
  private-cr708a92bffbde4569a04273eb64b3ff04-alb1       true      enabled    private   10.184.13.192
  

• Paste it in the loadBalancerID field inside the orders/values.yaml file and save it.
• Deploy the orders chart:

  $ helm install --name bluecompute orders
  

Note: For details on the contents of the orders chart, see the files in the orders folder. Particularly, check out orders/templates/ingress.yaml for the ingress resource.

Deploy the bluecompute-ce chart in the web-cluster

Now that we deployed orders chart in orders-cluster, we can deploy the bluecompute-ce chart in the web-cluster and have it communicate to orders service over the private network. But first, let’s cover Custom Endpoints a little bit, since it is the main Kubernetes feature we are leveraging. Then we will proceed to deploying the bluecompute-ce chart.

Quick review of Custom Endpoints

We want the web application to communicate with the orders service in the orders-cluster the same way it does it when the orders service is in the same cluster as the web application. In other words, we wanted the web application to reference the orders service using its kube-dns name of http://orders-service:8080 even if the orders service is in another cluster.

In order to accomplish cluster-to-cluster communication without changing any application code or editing the existing YAML files in the web chart, we created the following in the bluecompute-ce chart’s templates folder:

• A custom Service for the orders service. This service will point to a custom Endpoint for orders service in orders-cluster, explained below.
• For details on the custom endpoint, checkout the bluecompute-ce/templates/ordersService.yaml file.

A custom Endpoint for orders service contains the IP address of the Private ALB in the orders-cluster and port 80 for the Ingress. We are leveraging a Kubernetes feature called Headless services without selectors, which you can learn about here. Since we are not enabling Transport Layer Security (TLS) in the orders-cluster ingress yet (which will do in a later section), the endpoint will contain port 80 for the Ingress.

Note: For details on the custom endpoint, checkout the bluecompute-ce/templates/ordersEndpoint.yaml file.

Deploy the bluecompute-ce chart

To deploy the bluecompute-ce chart, follow the instructions below:

• Go to the web-cluster terminal window.
• Get the ALB IP address for the orders-cluster Private ALB:

  $ bx cs albs --cluster <orders-cluster-name>
  
  OK
  ALB ID                                                Enabled   Status     Type      ALB IP
  private-cr708a92bffbde4569a04273eb64b3ff04-alb1       true      enabled    private   10.184.13.192
  

• Paste it in the loadBalancerIp in the orders section inside the bluecompute-ce/values.yaml file and save it.
• Deploy the bluecompute-ce chart:

  $ helm install --name bluecompute bluecompute-ce
  

Validate the application

You can reference this link to validate the sample web application.

The main thing we want to validate is that, once you are logged in, you are able to buy items and see the orders listed in the Profile section. This means that the web app in the web-cluster is able to communicate with the orders service in the orders-cluster.

If you are able to do the above, congratulations! You have successfully deployed a microservice application across two clusters!

Delete the application

To delete the bluecompute application, follow these instructions:

• Go to the web-cluster terminal window and run the following command:

  $ helm delete bluecompute --purge
  

• Go to the orders-cluster terminal window and run the following command:

  $ helm delete bluecompute --purge
  

Note: The Helm releases in both clusters are called bluecompute.

Enable security with TLS

In some scenarios, just having private microservice communication between services across clusters is enough security-wise. But private and encrypted communication adds an extra layer of security. For that we can use Transport Layer Security (TLS) encryption.

To enable TLS, you need to do a few things, then you can proceed to deploy the applications as outlined in the sections above. In later sections you will see all the details to enable TLS, but here is an overview of the steps you will follow:

• Create a TLS certificate.
• Put the TLS certificate and TLS key in the orders/values.yaml file, which will create a secret.
• Enable TLS in the orders ingress resource via orders/values.yaml
• In the bluecompute-ce/values.yaml file, you need to update the orders protocol to https and the orders ports to 443 to fully enable TLS.

(Click to expand)

To enable TLS, the only additional Kubernetes resource that we are creating is a secret to hold the TLS certificate, which is used by the Ingress resource to enforce TLS.

Create a TLS certificate

The first thing to do to enable TLS is to create a certificate using OpenSSL, which you might need to install on your workstation. Here is the command to create a TLS certificate:

$ openssl req -newkey rsa:4096 -nodes -sha512 -x509 -days 3650 -nodes -out tls.crt -keyout tls.key

The above command will generate a key and ask you a series of questions, as shown below:

Generating a 4096 bit RSA private key
..................++
.........................................................................................................................................................................++
writing new private key to 'tls.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) []:US
State or Province Name (full name) []:Texas
Locality Name (eg, city) []:Austin
Organization Name (eg, company) []:IBM
Organizational Unit Name (eg, section) []:CASE
Common Name (eg, fully qualified host name) []:bluecompute-orders
Email Address []:you@email.com

Feel free to enter the values shown above, but the most important thing to enter is a value of bluecompute-orders for the Common Name field. This will allow the Private Ingress/ALB in orders-cluster to route requests for bluecompute-orders domain name to the orders service in the orders-cluster.

You may ask yourself whether you need to go to a domain registrar first and register this domain name, but since we are leveraging kube-dns and the Private Ingress/ALB domain routing features, we can avoid this step.

Put the TLS certificate and key into orders/values.yaml

Now that we have your TLS certificate, we need to put it in the orders/values.yaml as follows:

• Get the contents of the tls.crt file and then encode it in base64 format:

  $ cat tls.crt | base64
  

• Copy the output of the above command into the crt field in the tls section of orders/values.yaml file and save it.
• Get the contents of the tls.key file and then encode it in base64 format:

  $ cat tls.key | base64
  

• Copy the output of the above command into the key field in the tls section of orders/values.yaml file and save it.

The steps above are needed to create a Kubernetes Secret for the TLS Certificate and Key in orders/templates/secret.yaml, which the Ingress resource requires in orders/templates/ingress.yaml to enforce TLS and traffic rules for the bluecompute-orders domain name.

Enable TLS in orders chart

To enable TLS in the orders chart’s ingress controller, change the value of enabled field in the tls section to true in orders/values.yaml file and save it.

The above will enable TLS in the orders/templates/ingress.yaml resource, which will use the TLS secret created in the previous step to enforce TLS and traffic rules for the bluecompute-orders domain name.

Once the orders chart is deployed, TLS will be fully enabled.

Enable TLS for orders service in the bluecompute-ce chart

Now that TLS is enabled in the orders chart, we can tell the bluecompute-ce chart to start using https protocol when communicating with the bluecompute-orders service. To do so, you need to do the following in the bluecompute-ce/values.yaml:

• Change value both orders.port and orders.targetPort to 443. This will update the orders port values in both bluecompute-ce/templates/ordersService.yaml and bluecompute-ce/templates/ordersEndpoint.yaml
• Change value of web.services.orders.port to 443. This will update the port number that the web application uses to reference to the orders service, which you can checkout in the web application’s config map.
• Change value of web.services.orders.protocol to https. This will update the protocol that the web application uses to reference to the orders service, which you can checkout in the web application’s config map.

You have successfully enabled TLS between the web application in the web-cluster and the orders service in the orders-cluster. Now you can proceed to deploy both the orders and bluecompute-ce charts as instructed here.

If the orders and bluecompute-ce charts are still installed in their respective clusters, you can upgrade the chart releases as follows:

• Go to the orders-cluster terminal window and run the following command:

  $ helm upgrade bluecompute orders
  

• Go to the web-cluster terminal window and run the following command:

  $ helm upgrade bluecompute bluecompute-ce
  

The above commands will update the existing chart installations with the TLS options we just enabled.

Validate TLS in the application

After waiting a few minutes after the install/upgrade you can validate the basic application functionality as explained earlier. To validate that the web app is communicating with the orders microservice using TLS, do the following:

• Login to the bluecompute app using user as username and passw0rd for password.
• Click on any of the catalog items and place an order. You should see a confirmation that the order was placed successfully.
• Click on the Profile icon to retrieve the order you just placed. You should be able to see the item you ordered, the amount, and the date.

The above steps should have triggered some logs in the web pod that shows the networking request to the orders microservice. To see the logs, use the following commands in the web-cluster terminal window.

First, grab the pod name for the web app:

$ kubectl get pods | grep web
bluecompute-web-84bc5f67d9-m5dbr         1/1       Running             0          31s

Then copy the pod name above and get its logs as follows:

$ kubectl logs -f [pod_name]

You should get a stream of logs similar to the following:

The above images show the POST request that was made when making a purchase followed by the GET request that was made when retrieving the orders in the Profile section. Notice that each requests uses https://bluecompute-orders:443 as the URL for the orders microservice, showing that it’s using TLS as indicated by the https protocol and the 443 port.

Conclusion

While it’s ideal to have microservices running together in the same infrastructure, sometimes isolated infrastructure is required for each workload. This prompts the need for secure and private connections between workloads, which can be tedious to setup. Luckily, we can easily achieve this with IBM Cloud Container Service by leveraging open standards from Kubernetes.

If you are interested in learning the ins and outs of IBM Cloud Container Service networking, I recommend the series of blogs published by the Cloud Architecture and Solution Engineering (CASE) team How to deploy, manage, and secure your container-based workloads. You should also check out the IBM Cloud Garage Method site; it includes cloud-based architectural solutions, tutorials and proven development practices that are part of the IBM Cloud Garage services.