Defining Kubernetes Observer jobs

Create a service account in the Kubernetes environment and obtain its token

1. Create either clusterrole access or role access.

   For clusterole access

   Create a configuration file called `asm-k8s-observer.yaml` with the custom cluster role `asm:kubernetes-observer`.

   Use the following sample content:

   kind: ClusterRole
   apiVersion: rbac.authorization.k8s.io/v1
   metadata:
     annotations:
       rbac.authorization.kubernetes.io/autoupdate: "true"
     name: asm:kubernetes-observer
   rules:
   - apiGroups: [""]
     resources: [ "pods", "namespaces", "nodes", "services", "endpoints", "persistentvolumes", "persistentvolumeclaims"]
     verbs: ["get", "list"]
   - apiGroups: ["apps"]
     resources: ["replicasets", "deployments", "statefulsets", "daemonsets"]
     verbs: ["get", "list"]

   For role access

   Use the following sample content:

   kind: Role
   apiVersion: rbac.authorization.k8s.io/v1
   metadata:
     namespace: {namespace} <-- Specify which namespace to create the Role access
     name: asm:kubernetes-observer
   rules:
   - apiGroups: [""]
     resources: [ "pods", "services", "endpoints"]
     verbs: ["get", "list"]
   - apiGroups: ["apps"]
     resources: ["replicasets", "deployments", "statefulsets", "daemonsets"]
     verbs: ["get", "list"]  

   Observing Custom Resources

   To model custom resource definitions (CRDs) and custom resources (CRs) in the topology, identify the names of the CRDs that are present in the environment by running the `oc get crds` command. Then, update the job parameters to include the names of CRDs to be discovered. When a CRD is being included in the job, the respective CRs will also be discovered.

   Example (discovering ibm.com CRDs):

   oc get crds | grep ibm.com

   Sample output:

   asmformations.asm.ibm.com                                   2020-12-03T01:53:38Z
   asms.asm.ibm.com                                            2020-12-03T01:53:38Z
   cemformations.cem.ibm.com                                   2020-12-03T01:53:38Z
   cemserviceinstances.cem.ibm.com                             2020-12-03T01:53:38Z
   noiconnectionlayers.noi.ibm.com                             2020-12-03T01:47:19Z
   noiformations.noi.ibm.com                                   2020-12-03T01:47:19Z
   noihybrids.noi.ibm.com                                      2020-12-03T01:47:19Z
   nois.noi.ibm.com                                            2020-12-03T01:47:20Z

   Also, patch the existing ClusterRole in the target environment, to include the following resources:

   - apiGroups: ["apiextensions.k8s.io"]
     resources: ["customresourcedefinitions"]
     verbs: ["get", "list"]
   - apiGroups: ["noi.ibm.com"]
     resources: ["nois", "noihybrids", "noiformations", "noiconnectionlayers"]
     verbs: ["get", "list"]
   - apiGroups: ["asm.ibm.com"]
     resources: ["asms", "asmformations"]
     verbs: ["get", "list"]
   - apiGroups: ["cem.ibm.com"]
     resources: ["cemformations"]
     verbs: ["get", "list"]

   The `apiextensions.k8s.io` segment of the cluster role provides your permission required to access all the custom resource definitions resources.

   - apiGroups: ["apiextensions.k8s.io"]
     resources: ["customresourcedefinitions"]
     verbs: ["get", "list"]

   The `asm.ibm.com` segment provides access to `asms` and `asmformations` custom resources. The names of the resources listed are the plural for each of the resources. They are usually derived from the first word of the CRD.

   - apiGroups: ["asm.ibm.com"]
     resources: ["asms", "asmformations"]
     verbs: ["get", "list"]

   For a new (fresh) discovery

   Create a configuration file called asm-k8s-observer-v2.yaml in the target environment using the yaml resource, as provided in the sample content.

   Then run the following command:

   kubectl create -f asm-k8s-observer-v2.yaml

   Continue to the next step of creating a service account.

   For an existing discovery

   Create a configuration file called k8s-update-cr.yaml using the yaml resource provided in the sample content.

   Run the following command to patch the clusterole:

   kubectl patch clusterrole asm:kubernetes-observer --patch "$(cat k8s-update-cr.yaml)"

2. Create a service account in the target environment.

   oc create serviceaccount asm-k8s-account -n <namespace>

   Tip: Verify that the service account exists:

   kubectl get serviceaccount

   Important: When running the following command, ensure you know the namespace in which it was created:

   oc create serviceaccount asm-k8s-account

3. Bind the asm:kubernetes-observer clusterrole or role to the asm-k8s-account service account.
   Important: The service account and the role or clusterrole binding need to be created in the same namespace.

   Example command for clusterrole binding:

   oc create clusterrolebinding asm-k8s --clusterrole=asm:kubernetes-observer --serviceaccount={namespace}:asm-k8s-account

   Example command for role binding:

   oc create rolebinding asm-k8s --role=asm:kubernetes-observer --serviceaccount={namespace}:asm-k8s-account

   Example role binding output:

   rolebinding.rbac.authorization.k8s.io/asm-k8s created

4. Obtain the Kubernetes service account token.
   Complete the following substeps depending on your version of Kubernetes.

   Kubernetes on OpenShift

   1. Look for the mounted service account token secret from the secrets section:

      oc describe serviceaccount asm-k8s-account

      Example:

      oc describe serviceaccount asm-k8s-account
      Name: asm-k8s-account
      Namespace: default
      Labels: <none>
      Annotations: <none>
      Image pull secrets: asm-k8s-account-dockercfg-wwzx2
      Mountable secrets: asm-k8s-account-dockercfg-wwzx2
      Tokens: asm-k8s-account-token-tzl52
      Events: <none>
      

   2. Describe the asm-k8s-account-token-******* (which in this example is tzl52) to obtain the token's value:

      oc describe secret asm-k8s-account-token-tzl52

      Alternative: You can also obtain the token's value using the following command (applies to both OpenShift and GKE environments).

      oc describe serviceaccount asm-k8s-account | grep Tokens | awk '{ print $2} ' | xargs -I{} oc get secret {} --template='{{.data.token}}' | base64 -d

   Kubernetes on GKE

   1. Verify that the token has been created using the kubectl describe serviceaccount <service-account> command.
      Example of a missing token:

      oc kubectl describe serviceaccount asm-k8s-account
      Name:                asm-k8s-account
      Namespace:           default
      Labels:              <none>
      Annotations:         <none>
      Image pull secrets:  <none>
      Mountable secrets:   <none>
      Tokens:              <none>
      Events:              <none>

   2. If missing, create the token manually.
      Example:

      kubectl apply -f - <<EOF
      apiVersion: v1
      kind: Secret
      metadata:
        name: asm-k8s-account-secret
        annotations:
          kubernetes.io/service-account.name: asm-k8s-account
      type: kubernetes.io/service-account-token
      EOF
      secret/asm-k8s-account-secret created

   3. Verify that the token exists.
      Example of a token that is present:

      oc kubectl describe serviceaccount asm-k8s-account
      Name:                asm-k8s-account
      Namespace:           default
      Labels:              <none>
      Annotations:         <none>
      Image pull secrets:  <none>
      Mountable secrets:   <none>
      Tokens:              asm-k8s-account-secret
      Events:              <none>

   4. Describe the secret to obtain the token's value.
      Example:

      oc kubectl describe secret asm-k8s-account-secret
      Name:         asm-k8s-account-secret
      Namespace:    default
      Labels:       <none>
      Annotations:  kubernetes.io/service-account.name: asm-k8s-account
                    kubernetes.io/service-account.uid: d82e86c1-2258-4072-8a53-f517f776a149
      
      Type:  kubernetes.io/service-account-token
      
      Data
      ====
      ca.crt:     1509 bytes
      namespace:  7 bytes
      token:      eyJhbGciO

Define the SSL certificate

The Load job requires an SSL Certificate, and for it to be in a specific location.

5. Get the kubernetes master IP and its API port using:

   kubectl cluster-info

6. Define the SSL certificate for the Exact HTTPS certificate file name parameter.
   See Configuring observer job security for more information.

Edit the parameters in the configuration file

7. Open the kubernetes_observer_common.sh configuration file and edit the following parameters:

   Table 1. Kubernetes Observer Load job parameters

   Parameter	Action	Details
   Unique ID	Enter a unique name for the job. Note: The connection name cannot be the same as any of the following observer job names: 'local', 'weave_scope', 'load', 'listen', 'kubeconfig'.	Required
   Data center	Specify the name of the data center in which the Kubernetes instance is running. The data center name is treated as the provider of the data. Note: If you change the data center name, the observation is treated as a separate observation under the different provider, and a duplicate set of resources and groups will be created under the new provider.	Required
   Kubernetes Master IP address	Enter the Kubernetes Master IP address.	Required
   Kubernetes API port	Enter the Kubernetes API port number.	Required
   Kubernetes token	The service account token for Kubernetes.	Required. Must be encrypted.
   Trust all certificate by bypassing certificate verification	Enter true if you want to connect to Kubernetes without certificate	Required
   Exact HTTPS certificate file name	If the Trust all certificate parameter is set to false, then this parameter is required.	Required. For more information, see Configuring observer job security.
   SSl Validation	Choose whether SSL validation is true or false. If false, HTTPS without hostname validation is used.	Optional
   Terminated pods	Choose whether terminated pods should be hidden (true or false).	Optional. The default is false.
   API query timeout (ms)	Specify the Kubernetes REST API query timeout.	Optional. The default is 5000 ms (that is, 5 seconds)
   Correlate	If true, enables the Event Analytics correlation on the namespace groups.	Optional
   Names of the custom resource definitions	A number of custom resource definitions (CRDs) that can be queried from the Kubernetes API	Optional
   Role permission token	Determine whether the provided token is a role token. Default to false for cluster role permissions.	Optional
   Namespace	Specify the Kubernetes namespace.	Optional. If left empty, all namespaces are observed.

To start the job

8. To start the job, use the following command:

   $ASM_HOME/bin/kubernetes_observer_query_start.sh

Update the ClusterRole and Role configuration

1. Edit the `asm-k8s-observer.yaml` configuration file and add the following information:

   For clusterole access

   Add 'watch' as in the following example:

   kind: ClusterRole
   apiVersion: rbac.authorization.k8s.io/v1
   metadata:
     annotations:
       rbac.authorization.kubernetes.io/autoupdate: "true"
     name: asm:kubernetes-observer
   rules:
   - apiGroups: [""]
     resources: [ "pods", "namespaces", "nodes", "services", "endpoints", "persistentvolumes", "persistentvolumeclaims"]
     verbs: ["get", "list", "watch"]
   - apiGroups: ["apps"]
     resources: ["replicasets", "deployments", "statefulsets", "daemonsets"]
     verbs: ["get", "list", "watch"]

   For role access

   Add 'watch' as in the following example:

   kind: Role
   apiVersion: rbac.authorization.k8s.io/v1
   metadata:
     namespace: default
     name: asm:kubernetes-observer
   rules:
   - apiGroups: [""]
     resources: [ "pods", "services", "endpoints"]
     verbs: ["get", "list", "watch"]
   - apiGroups: ["apps"]
     resources: ["replicasets", "deployments", "statefulsets", "daemonsets"]
     verbs: ["get", "list", "watch"]

2. Run the following command:

   oc apply -f asm-k8s-observer.yaml

Edit the parameters in the configuration file

3. Open the kubernetes_observer_common.sh configuration file and edit the following parameters:

   Table 2. Kubernetes Observer Listen job parameters

   Parameter	Action	Details
   Unique ID	Enter a unique name for the job. Note: The connection name cannot be the same as any of the following observer job names: 'local', 'weave_scope', 'load', 'listen', 'kubeconfig'.	Required
   Data center	Specify the name of the data center in which the Kubernetes instance is running. The data center name is treated as the provider of the data. Note: If you change the data center name, the observation is treated as a separate observation under the different provider, and a duplicate set of resources and groups will be created under the new provider.	Required
   Kubernetes Master IP address	Enter the Kubernetes Master IP address.	Required
   Kubernetes API port	Enter the Kubernetes API port number.	Required
   Kubernetes token	The service account token for Kubernetes. Both cluster and role tokens are supported. If you select a role token, uou must also provide a namespace value.	Required. Use plain text.
   Trust all certificate by bypassing certificate verification	Enter true if you want to connect to Kubernetes without certificate.	Required
   Exact HTTPS certificate file name	If the Trust all certificate parameter is set to false, then this parameter is required.	Required. For more information, see Configuring observer job security.
   SSl Validation	Choose whether SSL validation is true or false. If false, HTTPS without hostname validation is used.	Optional
   Terminated pods	Choose whether terminated pods should be hidden (true or false).	Optional. The default is false.
   API query timeout (ms)	Specify the Kubernetes REST API query timeout.	Optional. The default is 5000 ms (that is, 5 seconds)
   Correlate	If true, enables the Event Analytics correlation on the namespace groups.	Optional
   Names of the custom resource definitions	A number of custom resource definitions (CRDs) that can be queried from the Kubernetes API	Optional
   Role permission token	Set to true if the provided token is a role token. Defaults to false for cluster role permissions.	Optional
   Namespace	Specify the Kubernetes namespace.	Optional. If left empty, all namespaces are observed.

To start the job

4. To start the job, use the following command:

   $ASM_HOME/bin/kubernetes_observer_query_start.sh

Deploy Weave Scope in your Kubernetes environment

1. Create Namespace 'weave' with 'ibm-privileged-psp'.

   kubectl create namespace weave 
   kubectl -n weave create rolebinding weave-clusterrole-rolebinding --clusterrole=ibm-privileged-clusterrole --group=system:serviceaccounts:weave
   

2. Install Weave Scope using the following command:

   kubectl apply -f "https://cloud.weave.works/k8s/scope.yaml?k8s-service-type=NodePort&k8s-version=$(kubectl version | base64 | tr -d '\n')"

   This will result in a port being opened that the Observer can use.
3. You can discover the NodePort using the following command:

   kubectl -n weave describe service weave-scope-app

4. Launch the Weave Scope UI using the following URL:

   https://<master ip>:<NodePort>

5. Create a new route using the OpenShift UI to expose the OCP Weave Scope service under the weave namespace. In the left panel of the OpenShift UI, click Networking > Routes.
6. Complete the required fields and then click Create.
   A new route has been created. Launch the Weave Scope UI using the location link.

Edit the parameters in the configuration file

7. Open the kubernetes_observer_common.sh configuration file and edit the following parameters:

   Table 3. Kubernetes Observer weave_scope job parameters

   Parameter	Action	Details
   Unique ID	Enter a unique name for the job. Note: The connection name cannot be the same as any of the following observer job names: 'local', 'weave_scope', 'load', 'listen', 'kubeconfig'.	Required
   Host	Enter the Weave Scope host name (or IP address) of the web socket to be observed.	Required
   Port	Enter the Weave Scope port number of the web socket to be observed.	Required
   Cluster Name	Enter the name of the cluster or data center to be observed.	Required
   Namespaces	Enter a list of namespaces to be observed.	Optional. If left empty, all namespaces will be observed.
   Resource types	Select the Weave Scope resource types to observe.	Optional.
   Resources to exclude	List resources to be excluded by ID, label, rank or namespace.	Optional. Containers named 'pod' are excluded by default.
   Refresh interval	The refresh interval in seconds between updates. The minimum allowed value is 60s. Default value is 120s.	Optional

To start the Weave Scope Listen job

8. To start the Kubernetes Observer Weave Scope Listen job, use the following command:

   $ASM_HOME/bin/kubernetes_observer_listen_start.sh

   The Listener job monitors its source for updates and runs until it is explicitly stopped, or until the Observer is stopped.