Running applications on a host name in IBM Cloud Kubernetes Service with NLB host names
Have you ever wanted to run an application on a host name in IBM Cloud Kubernetes Service, but you don’t want to worry about the DNS records? What about having health checks in a multizone environment to keep your service available? Maybe you’ve even wanted an SSL certificate that is automatically generated and renewed for that host name so that you don’t have to think about it.
In other words, if you’ve ever wanted to run a web server, an API gateway, an Ingress controller, a Kafka proxy, a service that has a binary protocol like an MQTT service or database, or essentially anything that runs on TCP (or UDP), you can now run it in IBM Cloud Kubernetes Service on a host name—and have SSL certs and health monitors for it too (health monitors only work with TCP).
How did it work until now?
Previously, you could deploy any application to your IBM Cloud Kubernetes Service cluster and expose it to the outside world on any port you like via the NLB. (You can read more about different deployment patterns in this post.) However, if you wanted to create a highly available service that has both a DNS–based health check for cross-zone failover within a region and a host name with an SSL certificate that is renewed automatically, you had two options:
Get a domain, configure DNS, configure health checks, and configure SSL certificates to automatically renew it with, for example, with Letsencrypt (or buy an SSL cert signed by a trusted CA).
Use the IBM Cloud Kubernetes Service ALB (Ingress controller) that has built-in multizone health checks and a service that automatically generates and renews an SSL cert for you via Letsencrypt.
What is changing?
With this announcement, we are introducing NLB host names as the DNS host name you can generate for each Kubernetes deployment exposed with the NLB (LoadBalancer service). We are happy to inform you that we moved services such as the SSL certificate, the DNS registration, and health checks to a lower level in the stack (to the NLB level) so you can benefit from it for any deployment that you expose via the NLB on IBM Cloud Kubernetes Service.
When you expose your deployment via an NLB, you will get an IP address that you can use to generate an IBM–provided host name. We take care of generating and maintaining the wildcard SSL cert for you, and you can configure your health checks against an HTTP(S) service or just a TCP port.
This means you can run the IBM Cloud Kubernetes Service ALB, an API gateway of your choice, an Istio ingress gateway, and an MQTT server in parallel in your IBM Cloud Kubernetes Service cluster. Each one will have its own:
Publicly available wildcard host name
Wildcard SSL certificate associated with the host name
Health checks that you can configure if you use multizone deployments. For example, you might have two NLB IP addresses in two zones; only the IPs where your service is functional will show up in the DNS responses for the NLB host name.
How does it work?
The NLB (layer 4 load balancer) is used to expose both the ALB and any other deployment you want to expose on TCP or UDP.
Each NLB has an IP address per zone (such as 126.96.36.199 [dal12] and 188.8.131.52 [dal13], etc.).
Each NLB host name has one or multiple IPs associated with it (see DNS response in diagram).
Each NLB host name has optional health monitors that periodically check the availability of the exposed IP address and service.
In case there is an issue with the service in a zone, the DNS response to the client will not contain the failing NLB IP address.
Please note in the diagram I have three zones, and all three have an ALB and the associated NLBs in front of them. In my example below, I will expose a custom deployment in [dal12] and [dal13] zones. This diagram reflects my setup.
How can I use it?
Make sure you update your CLI with the following to access the new commands:
The following new commands are introduced with this release:
nlb-dnss— List the registered NLB host names and IP addresses in a cluster.
nlb-dns-create— Create a DNS host name to register one NLB IP.
nlb-dns-add— Add an NLB IP to an existing host name that you created with ‘nlb-dns-create’.
nlb-dns-rm— Remove an NLB IP address from a host name. If you remove all IPs from a host name, the host name still exists but no IPs are associated with it.
nlb-dns-monitors— List the health check monitor settings for each NLB host name in a cluster.
nlb-dns-monitor-configure— Configure a health check monitor for an existing NLB host name in a cluster. To create and enable a new monitor, include the ‘–enable’ flag. To update an existing monitor, include only the flags for the settings that you want to change.
nlb-dns-monitor-get— View the settings for an existing health check monitor.
nlb-dns-monitor-status— List the health check status for the IPs behind NLB host names in a cluster.
nlb-dns-monitor-disable— Disable an existing health check monitor for a NLB host name in a cluster.
nlb-dns-monitor-enable— Enable a health check monitor for a NLB host name in a cluster. Note: You must first configure the monitor with ‘nlb-dns-monitor-configure’.
For the official documentation, please visit this link and check out the “Registering load balancer host name” section.
In the following example, I deploy an application (for demonstration purposes)—a basic web server in the
us-south region in two zones (
dal13). For this scenario, instead of exposing the application (web server) with the IBM Cloud Kubernetes Service ALB (the Ingress controller), I want to expose them in each zone with NLB 1.0. Why do I want to do that? Because I want my application to terminate the TCP connection directly from my clients; I do not want a proxy (the ALB) in between the client and my application. If you want to read further about the reasons why I’m choosing one solution over the other, jump over and read this blog post.
I used the following yaml if you want to look into the details.
The details of my cluster: Deployment and services
These are my worker nodes for the “nlb-hostnames-demo-cluster” via the IBM Cloud CLI:
Kubernetes sees them like this:
Here is my “echoserver” deployment (simple web server just for demonstration purposes):
Finally, here are my services—notice the external NLB IP addresses and port of the LoadBalancer (I used port 1884 to expose my service to the outside world):
Now that I have two LoadBalancer (NLB) IPs in two zones, let’s create a host name for them, add both IPs, and configure health monitors for DNS for them.
Create NLB host name
I will run the following command to request to create a new NLB host name for IP: 184.108.40.206 from zone [dal12]:
Great, I got a host name. Let me test it, checking if the host name and the NLB IP has populated to DNS:
Perfect, it is there. Let’s add the other NLB IP address 220.127.116.11 from zone [dal13] to this host name:
Now, I expect to have two NLB IPs from two zones to show up in DNS:
Now I have two NLB IPs under the same host name, but if the application itself or the complete zone fails behind one of them, my traffic can keep going to an NLB IP address that is unhealthy. Therefore, I want to set up health monitors which will check the availability of my service and only respond with healthy IPs from DNS.
Setting up health monitors
First, I will list the health monitors I have by running the following command:
I see that I already have a health monitor that is not configured yet. I will go ahead and configure a TCP health monitor for my service exposed on port 1884. The health monitor does an equivalent to “
$ nc -z -G 5 18.104.22.168 1884“, trying to open a TCP connection on port 1884 with a timeout of 5 seconds. If it fails, the health monitor will report that the service is down and remove that particular IP from DNS responses. These health checks are initiated from Cloudflare’s network PoPs all over the world.
The health monitor is configured as being set up. (You can safely ignore the HTTP related output, those are ignored in TCP health monitors.)
With the following command, I can query the health status of the individual NLB IP addresses from the perspective of the globally distributed health monitors:
I can also configure an HTTP or HTTPS based health check, as in the following example:
For testing purposes, I will now scale down the replicas of my application deployment to zero in zone [dal13], simulating an outage. (When there are no application pods running, the NLB is sending traffic to a black hole—there won’t be anything to respond there and connection requests will time out.)
Now, every connection request that goes to the zone [dal13] NLB IP address 22.214.171.124 will time out. This problem will be picked up by the health monitor and the failing NLB IP will get removed from the DNS response. After a minute or two, when I initiate the same DNS resolution request, I get only the [dal12] NLB IP back (126.96.36.199), therefore, my clients will not try to connect to the unhealthy [dal13] NLB IP.
Also important to mention that the host name I get has a wildcard entry. I can use this to have my unique application name at the beginning of the host name. Example:
my-app.nlb-hostnames-demo-c-30e6f46e0c4c7579550bcbc31122a383-0001.us-south.stg.containers.appdomain.cloud will also point to the IP addresses I registered via a CNAME in between:
Note: You cannot use dots (.) as a separator, use dash (-) instead. In other words:
Wildcard SSL certificate
Along the wildcard host name, I also get a wildcard SSL certificate automatically provisioned, signed by Letsencrypt and maintained (automatically renewed) by IBM Cloud. The SSL certificate is put in a form of secret into my cluster, I can find it with the following: