Enable Istio for Your Service

5 min read

By: Gong Zhang and Yang Yang

Enabling Istio on IBM Cloud Foundry Enterprise Environment

With the increased popularity of utilizing a cloud-native approach, microservice governance is becoming a more important and popular topic. Istio can provide behavioral insights and operational control over the service mesh as a whole, offering a complete solution to satisfy the diverse requirements of microservice applications.

When getting started with Istio, the best option is to read the official documents and learn about its features (you can also learn about the basics of Istio in our informational lightboarding video, “What is Istio?“). The example application Istio provides is called Bookinfo. It is a well-known sample application on istio.io, and it is designed with four separate microservices so that users can easily inject Istio sidecar and apply Istio resources to manage it.

However, after you learn about the Bookinfo application and start to adopt Istio for your own service, you may begin to feel it is totally a different story. For example, for a production-ready service, it may be composed of some microservices hosted on Kubernetes and some that are not. It is also very common that this service may consume external services, like calling a weather API, reading data from an existing external database, or calling Amazon S3 service.

Don’t be frustrated—we have been there and experienced the same things. We have explored Istio on IBM Cloud Foundry Enterprise Environment (CFEE), which is a very complex system. In this post, we will share our journey and experience with enabling Istio step-by-step on CFEE, and we hope it can help you with your own service.

Background

IBM Cloud Foundry Enterprise Environment (CFEE) leverages IBM Cloud Kubernetes Service to host the Cloud Foundry PaaS environment in order to provide rapid deployment and management. There are 10+ Kubernetes services under the cf namespace for Cloud Foundry core components. It also consumes three external services: IBM Token Service, IBM Cloud Object Storage (COS), and IBM Cloud Databases for PostgreSQL. CFEE also leverages IBM Cloud Kubernetes Service ALB (Application Load Balancer) to expose endpoints.

We’ve been working to enable Istio for Cloud Foundry core components, and we have summarized our best practices into the following six steps:

  1. Identify traffic flow

  2. Install Istio on IBM Cloud Kubernetes Service

  3. Bypass Istio sidecar proxy and call external service directly

  4. Inject Istio sidecar

  5. Introduce external service into the mesh

  6. Chain IBM Cloud Kubernetes Service ALB and Istio ingress gateway

Step 1: Identify traffic flow

Istio will block all inside-out traffic by default, and by doing this, services may fail because they may need to interact with services outside of the cluster. This is why services will sometimes be broken after we adopt Istio. So, before we get started to work on enabling services, we need to answer the following questions first:

Question 1: Are there services that will access external services?

Answer: Yes. CFEE need to access external services like as IBM Cloud Databases for PostgreSQL.

Question 2: Will all the services access each other through the Kubernetes service?

Answer: No. For example, CFEE service A will access pods directly by pod IP address after discovering service B.

Question 3: Are there services that need to exposed to have public access?

Answer: Yes. The CFEE API service needs to be exposed so users can target it.

Step 2: Install Istio on IBM Cloud Kubernetes Service

After identifying traffic flow, we can start working on enabling Istio on CFEE. Please refer to the steps in the following guides to install Istio on the IBM Cloud Kubernetes Service:

Step 3: Bypass Istio sidecar proxy for a specific range of IPs and call external service directly

If your answer to Question 1 in Step 1 is NO, please skip this step. If the answer is YES, we will first bypass the Istio sidecar proxy for a specific range of IPs so that all the services can call external services directly, allowing us to detect and locate issues more easily.

CFEE is deployed on the IBM Cloud Kubernetes Service, so we use the command below to bypass the Istio sidecar proxy for range of IPs “172.30.0.0/16\,172.20.0.0/16\,10.10.10.0/24”:

$ helm template install/kubernetes/helm/istio <OTHER_FLAGS_YOU_NEED> \
--set global.proxy.includeIPRanges="172.30.0.0/16\,172.20.0.0/16\,10.10.10.0/24" \
-x templates/sidecar-injector-configmap.yaml | kubectl apply -f -

This command will use helm template to render the YAML file, set the IP ranges, and apply it to the cluster. By running this command, only traffic in IP ranges “172.30.0.0/16\,172.20.0.0/16\,10.10.10.0/24” will be intercepted by the sidecar proxy, thereby excluding external IPs from being redirected to the sidecar proxy. This will give services direct access to external services.

Step 4: Inject Istio sidecar

From Question 2 in Step 1, we have identified all the services we decided to manage with Istio. Taking our api service in CFEE as an example, we need to make sure the deployment and service definition satisfies the following requirements by Istio:

  • Named ports: Service ports must be named. The port names must be of the form <protocol>[-<suffix>] in order to take advantage of Istio’s routing features, such as name: http-api or name: http. Otherwise, traffic on the port will be treated as plain TCP traffic (unless the port explicitly uses Protocol: UDP to signify a UDP port).

  • Deployments with app and version labels: It is recommended that pods deployed using the Kubernetes Deployment have an explicit app label and version label in the deployment specification. The app label is to add contextual information in distributed tracing. The version label is to indicate the version of the app that the particular deployment corresponds to so that Istio can leverage them to perform A/B deployment, Canary Deployment, and etc. They are both also used to add contextual information in the metric telemetry collected by Istio.

Here are our modified api service definition files:

apiVersion: v1
kind: Service
metadata:
  labels:
    app: api
    version: v1
  name: api
  namespace: cf
spec:
  ports:
  - name: http-api
    port: 9022
    protocol: TCP
    targetPort: api
  - name: https-api-tls
    port: 9023
    protocol: TCP
    targetPort: api-tls
  - name: http-statsd
    port: 8125
    protocol: TCP
    targetPort: statsd
  selector:
    app: api
    version: v1
    skiff-role-name: api
  type: ClusterIP
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    app: api
    version: v1
    skiff-role-name: api
  name: api
  namespace: cf
spec:
  replicas: 1
  selector:
    matchLabels:
      app: api
      version: v1
      skiff-role-name: api
  template:
    metadata:
      labels:
        app: api
        version: v1
        skiff-role-name: api
      name: api

With above service definition ready, we can start to inject the Istio sidecar. There are two ways to do it:

Manual injection actually modifies your deployment file and adds the sidecar container definition in your pod, while automatic sidecar injection leverages Kubernetes webhook admission controller and will add the sidecar container to your pod automatically without any modification to your deployment file. You can choose which approach to use based on your scenario. In this post, we’re using helm to deploy CFEE, and we used automatic sidecar injection.

Use the following steps to enable automatic sidecar injection for our cf namespace:

1. Label the cf namespace with istio-injection=enabled, which indicates enable proxy sidecar injection:

$ kubectl label namespace cf istio-injection=enabled

2. As injection happens at pod creation time, restart api pod (see below) and wait until its status changes to 2/2 READY containers.

$ kubectl delete pod -l skiff-role-name=api
 
NAME                           READY     STATUS     RESTARTS   AGE
api-0                          2/2       Running     3         7h
api-1                          2/2       Running     3         7h

3. To make sure we have injected the proxy sidecar correctly, we can access the api service and check if the traffic is intercepted by sidecar.

Access the API service endpoint:

$ curl https://api.istio-cfee.us-south.containers.appdomain.cloud/info
{"name":"IBM Cloud Foundry Enterprise Environment","build":"1.21.0-rc.24","support":"http://ibm.biz/bluemix-supportinfo","version":2,"description":"Kubernetes based Cloud Foundry
","authorization_endpoint":"https://iam.bluemix.net/cloudfoundry/login/a96311ea205744e8b93623d2123e93c9-870e8ef3-695d-406e-b47e-ee36c2b0727d","token_endpoint":"https://uaa.istio-cfee
.us-south.containers.appdomain.cloud:2793","allow_debug":true}

Inspect the request details from pod logs for istio-proxy sidecar container:

$ kubectl logs api-0 -c istio-proxy -n cf --tail 200
[2018-09-13T16:04:37.987Z] "GET /info HTTP/1.1" 200 - 0 391 5 3 "9.42.130.109, 10.73.120.43, 172.30.140.156" "curl/7.47.0" "b3754deb-2369-97f9-84f5-006878521634" "api. [2018-09-13T16:04:37.987Z] "GET /info HTTP/1.1" 200 - 0 391 5 3 "9.42.130.109, 10.73.120.43, 172.30.140.156" "curl/7.47.0" "b3754deb-2369-97f9-84f5-006878521634" "api.istio-cfee.us-south.containers.appdomain.cloud" "127.0.0.1:9022".us-south.containers.appdomain.cloud" "127.0.0.1:9022"

Tip: If there are some other services in the labeled namespace that are not ready to be introduced into the mesh, we can add an annotation to prevent proxy sidecar automatic injection if the pod is restarted:

apiVersion: extensions/v1beta1
kind: StatefulSet
metadata:
  name: router
spec:
  template:
    metadata:
      annotations:
        sidecar.istio.io/inject: "false"

After all the microservices we planned have been injected with the proxy sidecar, we can run some tests to verify the traffic flow and service functions are working as expected.

Step 5: Introduce external service into the mesh

In Step 3, we bypassed external services and called them directly. Now it’s time to introduce them into the mesh. Istio enables access to external services by defining ServiceEntry configurations.

First, recover Istio default behavior for external services, which does not allow direct access to external services:

1. Update the sidecar-injector pod:

$ helm template install/kubernetes/helm/istio <OTHER_FLAGS_YOU_NEED> \
--set global.proxy.includeIPRanges="" -x templates/sidecar-injector-configmap.yaml | kubectl apply -f

2. Add ServiceEntry for external services, as they may be called during pod recreation. Using our external database service as an example, it is hosted on sl-us-south-1-portal.1.dblayer.com:45559:

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: external-db-ext
spec:
  hosts:
    sl-us-south-1-portal.1.dblayer.com
  ports:
  - number: 45559
    name: tcp
    protocol: TCP
    resolution: DNS

3. Re-inject the proxy sidecar by deleting the pod so that automatic injection will be triggered. Take the api service as an example:

$ kubectl delete pod -l skiff-role-name=api

4. Inspect pod logs for istio-proxy sidecar container with the following command:

$ kubectl logs api-0 -c istio-proxy -n cf --tail 200
[2018-09-13 09:32:14.369][39][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:494] add/update cluster outbound|45559||sl-us-south-1-portal.1.dblayer.com during init

After we introduce all external services into the mesh, we run another round of tests to verify that all services are working as expected. We can  then leverage the fine-grained traffic management Istio provides with all these external services.

Step 6: Chain IBM Cloud Kubernetes Service ALB and Istio ingress gateway

CFEE leverages the IBM Cloud Kubernetes Service ALB (Application Load Balancer) to expose several services and utilizes IBM Cloud Kubernetes Service Ingress default domain, TLS termination, and various extensions. Istio also has an ingress gateway that operates at the edge of the mesh and receives incoming HTTP/TCP connections. Also, by configuring Istio Gateway and VirtualService resources, the user can get fine-grained traffic management with incoming traffic. In order to leverage the advantages of both of them, we choose to chain IBM Cloud Kubernetes Service ALB and Istio ingress gateway. There’s a blog, “Transitioning Your Service Mesh From IBM Cloud Kubernetes Service Ingress to Istio Ingress,” which has more details.

Again, we’ll use the api service as an example. CFEE used to expose it with ingress. The original ingress resource will route traffic matched api.istio-cfee.us-south.containers.appdomain.cloud to the api service.

Now we’ll start to chain IBM Cloud Kubernetes Service ALB and Istio ingress gateway.

1. Delete the original Ingress resource:

$ kubectl delete ing api -n cf

2. Inspect the Istio istio-ingressgateway service:

$ kubectl get svc -n istio-system istio-ingressgateway
NAME                       TYPE           CLUSTER-IP     EXTERNAL-IP      PORT(S)                                                                                                                   AGE
istio-ingressgateway       LoadBalancer   172.21.22.7    <SOME_IP_ADDR>   80:31380/TCP,443:31390/TCP,31400:31400/TCP,15011:31381/TCP,8060:32726/TCP,853:30739/TCP,15030:31717/TCP,15031:31289/TCP   21d

3. Create a new ingress resource which routes traffic matched api.istio-cfee.us-south.containers.appdomain.cloudto the istio-ingressgateway service. We keep using the cluster default certificate stored in istio-cfee secret for TLS termination so that traffic will be routed to istio-ingressgateway service at port 80:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    ingress.bluemix.net/client-max-body-size: 1536m
  name: istio-api-ingress
  namespace: istio-system
spec:
  rules:
  - host: api.istio-cfee.us-south.containers.appdomain.cloud
    http:
      paths:
      - backend:
          serviceName: istio-ingressgateway
          servicePort: 80
  tls:
  - hosts:
    - api.istio-cfee.us-south.containers.appdomain.cloud
    secretName: istio-cfee

4. Create a Gateway which opens port 80 for incoming connections to *.istio-cfee.us-south.containers.appdomain.cloud:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: httpcf-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*.istio-cfee.us-south.containers.appdomain.cloud"

5. Create a VirtualService to route matched traffic to the api service:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: httpapi
spec:
  hosts:
  - "api.istio-cfee.us-south.containers.appdomain.cloud"
  gateways:
  - httpcf-gateway
  http:
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        port:
          number: 9022
        host: api.cf.svc.cluster.local

Finally, we can access the api service and make sure all configurations are working as expected.

After we go through the above steps, we will have Cloud Foundry core components that look like this:

Cloud Foundry core components

Conclusion

In this post, we have tried to illustrate what we did in our IBM Cloud Foundry Enterprise Environment system to enable Istio. Enabling Istio is the very first step for microservice governance, and it lays the foundation if you want to have all the fancy and powerful features like fine-grained traffic management, telemetry, monitoring, etc. We hope these experiences and best practices are helpful and will facilitate your exploration of Istio.

Additional resources

Be the first to hear about news, product updates, and innovation from IBM Cloud