Environment isolation troubleshooting

Troubleshooting for environment isolation network issues.

MustGather

MustGather for environment isolation is written with an example.

namespaces : devops and production

two isolated and dedicated nodes for namespace : devops

two isolated and dedicated nodes for namespace : production

two isolated and dedicated proxy nodes for namespace : devops

two isolated and dedicated proxy nodes for namespace : production

Configuration

Config.yaml

## Environment Isolation
# Example:[{namespace: production, hostgroup: prod}, {namespace:devops, hostgroup: dev}, {namespace: preproduction, hostgroup: preprod}]
isolated_namespaces: [{namespace: devops,  hostgroup: worker-dev}, {namespace: production,  hostgroup: worker-prod}]
isolated_proxies: [{namespace: devops,  hostgroup: proxy-dev, lb_address: x.x.x.x}, {namespace: production, hostgroup: proxy-prod, lb_address: y.y.y.y}]

Host file

hosts

[hostgroup-worker-dev]
172.16.206.190
172.16.207.105

[hostgroup-worker-prod]
172.16.208.37
172.16.208.194

[hostgroup-proxy-dev]
172.16.208.195
172.16.159.167

[hostgroup-proxy-prod]
172.16.208.197
172.16.209.180

Troubleshooting

Kube API service configuration

Pod mainfest file: 
/etc/cfc/pods/master.json

Parameter:
--enable-admission-plugins=PodNodeSelector,PodTolerationRestriction
--admission-control-config-file=/etc/cfc/conf/admission-control-config.yaml

      {
        "name": "apiserver",
        "image": "hyc-cloud-private-stable-docker-local.artifactory.swg-devops.com/ibmcom-amd64/hyperkube:v1.11.1-ee",
        "imagePullPolicy": "IfNotPresent",



          "--enable-admission-plugins=Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,PodSecurityPolicy,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,Priority,EventRateLimit,PodNodeSelector,PodTolerationRestriction",
          "--admission-control-config-file=/etc/cfc/conf/admission-control-config.yaml",

Admission control configuration

File: /etc/cfc/conf/admission-control-config.yaml

---
kind: AdmissionConfiguration
apiVersion: apiserver.k8s.io/v1alpha1
plugins:
  - name: EventRateLimit
    path: /etc/cfc/conf/eventconfig.yaml
  - name: PodNodeSelector
    path: /etc/cfc/conf/podnodeselector.yaml

Pod node selector policy

File: /etc/cfc/conf/podnodeselector.yaml

podNodeSelectorPluginConfig:
  clusterDefaultNodeSelector: ""
  production: "worker-prod=true"
  devops: "worker-dev=true"

Node labels

Nodes dedicated for namespace: "devops" are labeled as "worker-dev"

kubectl -n kube-system get nodes -l "worker-dev"

NAME             STATUS    ROLES        AGE       VERSION
172.16.206.190   Ready     worker-dev   1d        v1.11.1+icp-ee
172.16.207.105   Ready     worker-dev   1d        v1.11.1+icp-ee

Nodes dedicated for namespace: "production" are labeled as "worker-prod"

kubectl -n kube-system get nodes -l "worker-prod"

NAME             STATUS    ROLES         AGE       VERSION
172.16.208.194   Ready     worker-prod   1d        v1.11.1+icp-ee
172.16.208.37    Ready     worker-prod   1d        v1.11.1+icp-ee

Annotations in namespaces

Namespace: "devops" kubectl -n kube-system get ns devops -o yaml

apiVersion: v1
kind: Namespace
metadata:
  annotations:
    scheduler.alpha.kubernetes.io/defaultTolerations: '[{"key":"dedicated","operator":"Equal","value":"worker-dev","effect":"NoSchedule"}]'
    scheduler.alpha.kubernetes.io/node-selector: worker-dev=true
  creationTimestamp: 2018-09-12T08:21:29Z
  name: devops
  resourceVersion: "4857"
  selfLink: /api/v1/namespaces/devops
  uid: d9579db3-b664-11e8-a04b-00163e01af61
spec:
  finalizers:
  - kubernetes
status:
  phase: Active

Namespace: "production" kubectl -n kube-system get ns production -o yaml

apiVersion: v1
kind: Namespace
metadata:
  annotations:
    scheduler.alpha.kubernetes.io/defaultTolerations: '[{"key":"dedicated","operator":"Equal","value":"worker-prod","effect":"NoSchedule"}]'
    scheduler.alpha.kubernetes.io/node-selector: worker-prod=true
  creationTimestamp: 2018-09-12T08:21:27Z
  name: production
  resourceVersion: "4848"
  selfLink: /api/v1/namespaces/production
  uid: d7f29b43-b664-11e8-a04b-00163e01af61
spec:
  finalizers:
  - kubernetes
status:
  phase: Active

Isolated proxies

Isolated proxy nodes for namespace: devops

kubectl -n kube-system get nodes -l "proxy-dev"

NAME             STATUS    ROLES       AGE       VERSION
172.16.159.167   Ready     proxy-dev   1d        v1.11.1+icp-ee
172.16.208.195   Ready     proxy-dev   1d        v1.11.1+icp-ee

Isolated proxy nodes for namespace: production

kubectl -n kube-system get nodes -l "proxy-prod"

root@envisobase1:~# kc get nodes -l "proxy-prod"
NAME             STATUS    ROLES        AGE       VERSION
172.16.208.197   Ready     proxy-prod   1d        v1.11.1+icp-ee
172.16.209.180   Ready     proxy-prod   1d        v1.11.1+icp-ee

Ingress controllers

kubectl -n kube-system get ds | grep "nginx-ingress"

nginx-ingress-controller              1         1         1         1            1           proxy=true        1d
nginx-ingress-controller-proxy-dev    2         2         2         2            2           proxy-dev=true    1d
nginx-ingress-controller-proxy-prod   2         2         2         2            2           proxy-prod=true   1d

Ingress controller serving namespace devops

kubectl -n kube-system get ds nginx-ingress-controller-proxy-dev -o yaml | grep "watch"

        - --watch-namespace=devops

Ingress controller serving namespace production

kubectl -n kube-system get ds nginx-ingress-controller-proxy-prod -o yaml | grep "watch"

        - --watch-namespace=production