Enabling Istio on IBM Cloud Foundry Enterprise Environment
With the increased popularity of utilizing a cloud-native approach, microservice governance is becoming a more important and popular topic. Istio can provide behavioral insights and operational control over the service mesh as a whole, offering a complete solution to satisfy the diverse requirements of microservice applications.
When getting started with Istio, the best option is to read the official documents and learn about its features (you can also learn about the basics of Istio in our informational lightboarding video, “What is Istio?“). The example application Istio provides is called
Bookinfo. It is a well-known sample application on istio.io, and it is designed with four separate microservices so that users can easily inject Istio sidecar and apply Istio resources to manage it.
However, after you learn about the
Bookinfo application and start to adopt Istio for your own service, you may begin to feel it is totally a different story. For example, for a production-ready service, it may be composed of some microservices hosted on Kubernetes and some that are not. It is also very common that this service may consume external services, like calling a weather API, reading data from an existing external database, or calling Amazon S3 service.
Don’t be frustrated—we have been there and experienced the same things. We have explored Istio on IBM Cloud Foundry Enterprise Environment (CFEE), which is a very complex system. In this post, we will share our journey and experience with enabling Istio step-by-step on CFEE, and we hope it can help you with your own service.
IBM Cloud Foundry Enterprise Environment (CFEE) leverages IBM Cloud Kubernetes Service to host the Cloud Foundry PaaS environment in order to provide rapid deployment and management. There are 10+ Kubernetes services under the
cf namespace for Cloud Foundry core components. It also consumes three external services: IBM Token Service, IBM Cloud Object Storage (COS), and IBM Cloud Databases for PostgreSQL. CFEE also leverages IBM Cloud Kubernetes Service ALB (Application Load Balancer) to expose endpoints.
We’ve been working to enable Istio for Cloud Foundry core components, and we have summarized our best practices into the following six steps:
Identify traffic flow
Install Istio on IBM Cloud Kubernetes Service
Bypass Istio sidecar proxy and call external service directly
Inject Istio sidecar
Introduce external service into the mesh
Chain IBM Cloud Kubernetes Service ALB and Istio ingress gateway
Step 1: Identify traffic flow
Istio will block all inside-out traffic by default, and by doing this, services may fail because they may need to interact with services outside of the cluster. This is why services will sometimes be broken after we adopt Istio. So, before we get started to work on enabling services, we need to answer the following questions first:
Question 1: Are there services that will access external services?
Answer: Yes. CFEE need to access external services like as IBM Cloud Databases for PostgreSQL.
Question 2: Will all the services access each other through the Kubernetes service?
Answer: No. For example, CFEE service A will access pods directly by pod IP address after discovering service B.
Question 3: Are there services that need to exposed to have public access?
Answer: Yes. The CFEE API service needs to be exposed so users can target it.
Step 2: Install Istio on IBM Cloud Kubernetes Service
After identifying traffic flow, we can start working on enabling Istio on CFEE. Please refer to the steps in the following guides to install Istio on the IBM Cloud Kubernetes Service:
Step 3: Bypass Istio sidecar proxy for a specific range of IPs and call external service directly
If your answer to Question 1 in Step 1 is NO, please skip this step. If the answer is YES, we will first bypass the Istio sidecar proxy for a specific range of IPs so that all the services can call external services directly, allowing us to detect and locate issues more easily.
CFEE is deployed on the IBM Cloud Kubernetes Service, so we use the command below to bypass the Istio sidecar proxy for range of IPs “172.30.0.0/16\,172.20.0.0/16\,10.10.10.0/24”:
This command will use
helm template to render the YAML file, set the IP ranges, and apply it to the cluster. By running this command, only traffic in IP ranges “172.30.0.0/16\,172.20.0.0/16\,10.10.10.0/24” will be intercepted by the sidecar proxy, thereby excluding external IPs from being redirected to the sidecar proxy. This will give services direct access to external services.
Step 4: Inject Istio sidecar
From Question 2 in Step 1, we have identified all the services we decided to manage with Istio. Taking our
api service in CFEE as an example, we need to make sure the deployment and service definition satisfies the following requirements by Istio:
Named ports: Service ports must be named. The port names must be of the form <protocol>[-<suffix>] in order to take advantage of Istio’s routing features, such as name: http-api or name: http. Otherwise, traffic on the port will be treated as plain TCP traffic (unless the port explicitly uses Protocol: UDP to signify a UDP port).
Deployments with app and version labels: It is recommended that pods deployed using the Kubernetes Deployment have an explicit app label and version label in the deployment specification. The app label is to add contextual information in distributed tracing. The version label is to indicate the version of the app that the particular deployment corresponds to so that Istio can leverage them to perform A/B deployment, Canary Deployment, and etc. They are both also used to add contextual information in the metric telemetry collected by Istio.
Here are our modified
api service definition files:
With above service definition ready, we can start to inject the Istio sidecar. There are two ways to do it:
Manual injection actually modifies your deployment file and adds the sidecar container definition in your pod, while automatic sidecar injection leverages Kubernetes webhook admission controller and will add the sidecar container to your pod automatically without any modification to your deployment file. You can choose which approach to use based on your scenario. In this post, we’re using helm to deploy CFEE, and we used automatic sidecar injection.
Use the following steps to enable automatic sidecar injection for our
1. Label the
cf namespace with
istio-injection=enabled, which indicates enable proxy sidecar injection:
2. As injection happens at pod creation time, restart
api pod (see below) and wait until its status changes to
2/2 READY containers.
3. To make sure we have injected the proxy sidecar correctly, we can access the
api service and check if the traffic is intercepted by sidecar.
Access the API service endpoint:
Inspect the request details from pod logs for istio-proxy sidecar container:
Tip: If there are some other services in the labeled namespace that are not ready to be introduced into the mesh, we can add an annotation to prevent proxy sidecar automatic injection if the pod is restarted:
After all the microservices we planned have been injected with the proxy sidecar, we can run some tests to verify the traffic flow and service functions are working as expected.
Step 5: Introduce external service into the mesh
In Step 3, we bypassed external services and called them directly. Now it’s time to introduce them into the mesh. Istio enables access to external services by defining ServiceEntry configurations.
First, recover Istio default behavior for external services, which does not allow direct access to external services:
1. Update the sidecar-injector pod:
ServiceEntry for external services, as they may be called during pod recreation. Using our external database service as an example, it is hosted on
3. Re-inject the proxy sidecar by deleting the pod so that automatic injection will be triggered. Take the api service as an example:
4. Inspect pod logs for istio-proxy sidecar container with the following command:
After we introduce all external services into the mesh, we run another round of tests to verify that all services are working as expected. We can then leverage the fine-grained traffic management Istio provides with all these external services.
Step 6: Chain IBM Cloud Kubernetes Service ALB and Istio ingress gateway
CFEE leverages the IBM Cloud Kubernetes Service ALB (Application Load Balancer) to expose several services and utilizes IBM Cloud Kubernetes Service Ingress default domain, TLS termination, and various extensions. Istio also has an ingress gateway that operates at the edge of the mesh and receives incoming HTTP/TCP connections. Also, by configuring Istio Gateway and VirtualService resources, the user can get fine-grained traffic management with incoming traffic. In order to leverage the advantages of both of them, we choose to chain IBM Cloud Kubernetes Service ALB and Istio ingress gateway. There’s a blog, “Transitioning Your Service Mesh From IBM Cloud Kubernetes Service Ingress to Istio Ingress,” which has more details.
Again, we’ll use the
api service as an example. CFEE used to expose it with ingress. The original ingress resource will route traffic matched
api.istio-cfee.us-south.containers.appdomain.cloud to the api service.
Now we’ll start to chain IBM Cloud Kubernetes Service ALB and Istio ingress gateway.
1. Delete the original Ingress resource:
2. Inspect the Istio istio-ingressgateway service:
3. Create a new ingress resource which routes traffic matched
api.istio-cfee.us-south.containers.appdomain.cloudto the istio-ingressgateway service. We keep using the cluster default certificate stored in
istio-cfee secret for TLS termination so that traffic will be routed to istio-ingressgateway service at port 80:
4. Create a
Gateway which opens port 80 for incoming connections to
5. Create a
VirtualService to route matched traffic to the
Finally, we can access the
api service and make sure all configurations are working as expected.
After we go through the above steps, we will have Cloud Foundry core components that look like this:
In this post, we have tried to illustrate what we did in our IBM Cloud Foundry Enterprise Environment system to enable Istio. Enabling Istio is the very first step for microservice governance, and it lays the foundation if you want to have all the fancy and powerful features like fine-grained traffic management, telemetry, monitoring, etc. We hope these experiences and best practices are helpful and will facilitate your exploration of Istio.