Configuring Kubernetes Observer jobs

Create a service account in the Kubernetes environment and obtain its token

1. Create either clusterrole access or role access.

   For clusterole access

   Create a configuration file called `asm-k8s-observer.yaml` with the custom cluster role `asm:kubernetes-observer`.

   Use the following sample content:

   kind: ClusterRole
   apiVersion: rbac.authorization.k8s.io/v1
   metadata:
     annotations:
       rbac.authorization.kubernetes.io/autoupdate: "true"
     name: asm:kubernetes-observer
   rules:
   - apiGroups: [""]
     resources: [ "pods", "namespaces", "nodes", "services", "endpoints", "persistentvolumes", "persistentvolumeclaims"]
     verbs: ["get", "list"]
   - apiGroups: ["apps"]
     resources: ["replicasets", "deployments", "statefulsets", "daemonsets"]
     verbs: ["get", "list"]

   For role access

   Use the following sample content:

   kind: Role
   apiVersion: rbac.authorization.k8s.io/v1
   metadata:
     namespace: {namespace} <-- Specify which namespace to create the Role access
     name: asm:kubernetes-observer
   rules:
   - apiGroups: [""]
     resources: [ "pods", "services", "endpoints"]
     verbs: ["get", "list"]
   - apiGroups: ["apps"]
     resources: ["replicasets", "deployments", "statefulsets", "daemonsets"]
     verbs: ["get", "list"]  

   Observing Custom Resources

   To model custom resource definitions (CRDs) and custom resources (CRs) in the topology, identify the names of the CRDs that are present in the environment by running the `oc get crds` command. Then, update the job parameters to include the names of CRDs to be discovered. When a CRD is being included in the job, the respective CRs will also be discovered.

   Example (discovering ibm.com CRDs):

   oc get crds | grep ibm.com

   Sample output:

   asmformations.asm.ibm.com                                   2020-12-03T01:53:38Z
   asms.asm.ibm.com                                            2020-12-03T01:53:38Z
   cemformations.cem.ibm.com                                   2020-12-03T01:53:38Z
   cemserviceinstances.cem.ibm.com                             2020-12-03T01:53:38Z
   noiconnectionlayers.noi.ibm.com                             2020-12-03T01:47:19Z
   noiformations.noi.ibm.com                                   2020-12-03T01:47:19Z
   noihybrids.noi.ibm.com                                      2020-12-03T01:47:19Z
   nois.noi.ibm.com                                            2020-12-03T01:47:20Z

   Also, patch the existing ClusterRole in the target environment, to include the following resources:

   - apiGroups: ["apiextensions.k8s.io"]
     resources: ["customresourcedefinitions"]
     verbs: ["get", "list"]
   - apiGroups: ["noi.ibm.com"]
     resources: ["nois", "noihybrids", "noiformations", "noiconnectionlayers"]
     verbs: ["get", "list"]
   - apiGroups: ["asm.ibm.com"]
     resources: ["asms", "asmformations"]
     verbs: ["get", "list"]
   - apiGroups: ["cem.ibm.com"]
     resources: ["cemformations"]
     verbs: ["get", "list"]

   The `apiextensions.k8s.io` segment of the cluster role provides your permission required to access all the custom resource definitions resources.

   - apiGroups: ["apiextensions.k8s.io"]
     resources: ["customresourcedefinitions"]
     verbs: ["get", "list"]

   The `asm.ibm.com` segment provides access to `asms` and `asmformations` custom resources. The names of the resources listed are the plural for each of the resources. They are usually derived from the first word of the CRD.

   - apiGroups: ["asm.ibm.com"]
     resources: ["asms", "asmformations"]
     verbs: ["get", "list"]

   For a new (fresh) discovery

   Create a configuration file called asm-k8s-observer-v2.yaml in the target environment using the yaml resource, as provided in the sample content.

   Then run the following command:

   kubectl create -f asm-k8s-observer-v2.yaml

   Continue to the next step of creating a service account.

   For an existing discovery

   Create a configuration file called k8s-update-cr.yaml using the yaml resource provided in the sample content.

   Run the following command to patch the clusterole:

   kubectl patch clusterrole asm:kubernetes-observer --patch "$(cat k8s-update-cr.yaml)"

2. Create a service account in the target environment.

   oc create serviceaccount asm-k8s-account -n <namespace>

   Tip: Verify that the service account exists:

   kubectl get serviceaccount

   Important: When running the following command, ensure you know the namespace in which it was created:

   oc create serviceaccount asm-k8s-account

3. Bind the asm:kubernetes-observer clusterrole or role to the asm-k8s-account service account.
   Important: The service account and the role or clusterrole binding need to be created in the same namespace.

   Example command for clusterrole binding:

   oc create clusterrolebinding asm-k8s --clusterrole=asm:kubernetes-observer --serviceaccount={namespace}:asm-k8s-account

   Example command for role binding:

   oc create rolebinding asm-k8s --role=asm:kubernetes-observer --serviceaccount={namespace}:asm-k8s-account

   Example role binding output:

   rolebinding.rbac.authorization.k8s.io/asm-k8s created

4. Obtain the Kubernetes service account token.
   Complete the following substeps depending on your version of Kubernetes.

   Kubernetes on OpenShift

   1. Look for the mounted service account token secret from the secrets section:

      oc describe serviceaccount asm-k8s-account

      Example:

      oc describe serviceaccount asm-k8s-account
      Name: asm-k8s-account
      Namespace: default
      Labels: <none>
      Annotations: <none>
      Image pull secrets: asm-k8s-account-dockercfg-wwzx2
      Mountable secrets: asm-k8s-account-dockercfg-wwzx2
      Tokens: asm-k8s-account-token-tzl52
      Events: <none>
      

   2. Describe the asm-k8s-account-token-******* (which in this example is tzl52) to obtain the token's value:

      oc describe secret asm-k8s-account-token-tzl52

      Alternative: You can also obtain the token's value using the following command (applies to both OpenShift and GKE environments).

      oc describe serviceaccount asm-k8s-account | grep Tokens | awk '{ print $2} ' | xargs -I{} oc get secret {} --template='{{.data.token}}' | base64 -d

   Kubernetes on GKE

   1. Verify that the token has been created using the kubectl describe serviceaccount <service-account> command.
      Example of a missing token:

      oc kubectl describe serviceaccount asm-k8s-account
      Name:                asm-k8s-account
      Namespace:           default
      Labels:              <none>
      Annotations:         <none>
      Image pull secrets:  <none>
      Mountable secrets:   <none>
      Tokens:              <none>
      Events:              <none>

   2. If missing, create the token manually.
      Example:

      kubectl apply -f - <<EOF
      apiVersion: v1
      kind: Secret
      metadata:
        name: asm-k8s-account-secret
        annotations:
          kubernetes.io/service-account.name: asm-k8s-account
      type: kubernetes.io/service-account-token
      EOF
      secret/asm-k8s-account-secret created

   3. Verify that the token exists.
      Example of a token that is present:

      oc kubectl describe serviceaccount asm-k8s-account
      Name:                asm-k8s-account
      Namespace:           default
      Labels:              <none>
      Annotations:         <none>
      Image pull secrets:  <none>
      Mountable secrets:   <none>
      Tokens:              asm-k8s-account-secret
      Events:              <none>

   4. Describe the secret to obtain the token's value.
      Example:

      oc kubectl describe secret asm-k8s-account-secret
      Name:         asm-k8s-account-secret
      Namespace:    default
      Labels:       <none>
      Annotations:  kubernetes.io/service-account.name: asm-k8s-account
                    kubernetes.io/service-account.uid: d82e86c1-2258-4072-8a53-f517f776a149
      
      Type:  kubernetes.io/service-account-token
      
      Data
      ====
      ca.crt:     1509 bytes
      namespace:  7 bytes
      token:      eyJhbGciO

Define the SSL certificate

The Load job requires an SSL Certificate, and for it to be in a specific location.

5. Get the kubernetes master IP and its API port using:

   kubectl cluster-info

6. Define the SSL certificate for the Exact HTTPS certificate file name parameter.
   See Configuring observer job security for more information.

Define the Load job

7. On the Observer jobs page, perform one of the following actions:

   To edit an existing job

   Open the List of options overflow menu next to the job and click View & edit.

   To create a new job

   Click Add a new job + and select the Kubernetes Observer tile.

8. Choose the Load job type from the drop-down, define or edit the following parameters, then click Save to save and run the job.

   Table 1. Kubernetes Observer Load job parameters

   Parameter	Action	Details
   Unique ID	Enter a unique name for the job. Note: The connection name cannot be the same as any of the following observer job names: 'local', 'weave_scope', 'load', 'listen', 'kubeconfig'.	Required
   Data center	Specify the name of the data center in which the Kubernetes instance is running. The data center name is treated as the provider of the data. Note: If you change the data center name, the observation is treated as a separate observation under the different provider, and a duplicate set of resources and groups will be created under the new provider.	Required
   Kubernetes Master IP address	Enter the Kubernetes Master IP address.	Required
   Kubernetes API port	Enter the Kubernetes API port number.	Required
   Kubernetes token	The service account token for Kubernetes.	Required. Use plain text.
   Trust all certificate by bypassing certificate verification	Enter true if you want to connect to Kubernetes without certificate.	Required
   Exact HTTPS certificate file name	If the Trust all certificate parameter is set to false, then this parameter is required.	Required if the Trust all parameter is set to false. Required. For more information, see Configuring observer job security. On-prem Create and store the certificate in the ASM_HOME/security directory. OCP Obtain the authentication certificate using OpenSSL and store it as a secret.
   SSl Validation	Choose whether SSL validation is true or false. If false, HTTPS without hostname validation is used.	Optional
   Terminated pods	Choose whether terminated pods should be hidden (true or false).	Optional. The default is false.
   API query timeout (ms)	Specify the Kubernetes REST API query timeout.	Optional. The default is 5000 ms (that is, 5 seconds)
   Correlate	If true, enables the Event Analytics correlation on the namespace groups.	Optional
   Names of the custom resource definitions	A number of custom resource definitions (CRDs) that can be queried from the Kubernetes API	Optional
   Role permission token	Determine whether the provided token is a role token. Default to false for cluster role permissions.	Optional
   Namespace	Specify the Kubernetes namespace.	Optional. If left empty, all namespaces are observed.
   Access scope	Enter text to provide a scope for the resources. Access scope can help map alerts to resources when resources in different scopes share the same parameters, such as matchTokens.	Optional. Tip: You can define access scope for locations, project names, namespaces, etc.
   Generate debug support file	Set the optional Generate debug support file parameter to 'True' in order to capture the output of the next scheduled job run as a file. This file will be stored with an observer's log files and can be used to debug observer issues, for example at the request of your designated Support team, or while using a test environment. For one-off jobs (that is, Load jobs), this parameter reverts to 'False' after the next completed run. To examine the output produced, you can load the generated debug file using the File Observer. The file is saved to the following locations: On-prem $ASM_HOME/logs/<obs>-observer/ On OCP /var/log/itsm/<obs>-observer	Optional
   Observer job description	Enter additional information to describe the job.	Optional
   Job schedule	Specify when the job should run, and whether it should run at regular intervals. By default the job runs immediately, and only once. Optionally you can specify a future date and time for the job to run, and then set it to run at regular intervals after that.	Optional. Transient (one-off) jobs only. If you set a job schedule, the run intervals must be at least 90 seconds apart, and if you set them at less than 15 minutes, a warning is displayed, as the frequency can impact system performance.

Update the ClusterRole and Role configuration

1. Edit the `asm-k8s-observer.yaml` configuration file and add the following information:

   For clusterole access

   Add 'watch' as in the following example:

   kind: ClusterRole
   apiVersion: rbac.authorization.k8s.io/v1
   metadata:
     annotations:
       rbac.authorization.kubernetes.io/autoupdate: "true"
     name: asm:kubernetes-observer
   rules:
   - apiGroups: [""]
     resources: [ "pods", "namespaces", "nodes", "services", "endpoints", "persistentvolumes", "persistentvolumeclaims"]
     verbs: ["get", "list", "watch"]
   - apiGroups: ["apps"]
     resources: ["replicasets", "deployments", "statefulsets", "daemonsets"]
     verbs: ["get", "list", "watch"]

   For role access

   Add 'watch' as in the following example:

   kind: Role
   apiVersion: rbac.authorization.k8s.io/v1
   metadata:
     namespace: default
     name: asm:kubernetes-observer
   rules:
   - apiGroups: [""]
     resources: [ "pods", "services", "endpoints"]
     verbs: ["get", "list", "watch"]
   - apiGroups: ["apps"]
     resources: ["replicasets", "deployments", "statefulsets", "daemonsets"]
     verbs: ["get", "list", "watch"]

2. Run the following command:

   oc apply -f asm-k8s-observer.yaml

Define the Listen job

3. On the Observer jobs page, perform one of the following actions:

   To edit an existing job

   Open the List of options overflow menu next to the job and click View & edit.

   To create a new job

   Click Add a new job + and select the Kubernetes Observer tile.

4. Choose the Listen job type from the drop-down, define or edit the following parameters, then click Save to save and run the job.

   Table 2. Kubernetes Observer Listen job parameters

   Parameter	Action	Details
   Unique ID	Enter a unique name for the job. Note: The connection name cannot be the same as any of the following observer job names: 'local', 'weave_scope', 'load', 'listen', 'kubeconfig'.	Required
   Data center	Specify the name of the data center in which the Kubernetes instance is running. The data center name is treated as the provider of the data. Note: If you change the data center name, the observation is treated as a separate observation under the different provider, and a duplicate set of resources and groups will be created under the new provider.	Required
   Kubernetes Master IP address	Enter the Kubernetes Master IP address.	Required
   Kubernetes API port	Enter the Kubernetes API port number.	Required
   Kubernetes token	The service account token for Kubernetes. Both cluster and role tokens are supported. If you select a role token, uou must also provide a namespace value.	Required. Use plain text.
   Trust all certificate by bypassing certificate verification	Enter true if you want to connect to Kubernetes without certificate.	Required
   Exact HTTPS certificate file name	If the Trust all certificate parameter is set to false, then this parameter is required.	Required if the Trust all parameter is set to false. Required. For more information, see Configuring observer job security. On-prem Create and store the certificate in the ASM_HOME/security directory. OCP Obtain the authentication certificate using OpenSSL and store it as a secret.
   SSl Validation	Choose whether SSL validation is true or false. If false, HTTPS without hostname validation is used.	Optional
   Terminated pods	Choose whether terminated pods should be hidden (true or false).	Optional. The default is false.
   API query timeout (ms)	Specify the Kubernetes REST API query timeout.	Optional. The default is 5000 ms (that is, 5 seconds)
   Correlate	If true, enables the Event Analytics correlation on the namespace groups.	Optional
   Names of the custom resource definitions	A number of custom resource definitions (CRDs) that can be queried from the Kubernetes API	Optional
   Role permission token	Set to true if the provided token is a role token. Defaults to false for cluster role permissions.	Optional
   Namespace	Specify the Kubernetes namespace.	Optional. If left empty, all namespaces are observed.
   Access scope	Enter text to provide a scope for the resources. Access scope can help map alerts to resources when resources in different scopes share the same parameters, such as matchTokens.	Optional. Tip: You can define access scope for locations, project names, namespaces, etc.
   Generate debug support file	Set the optional Generate debug support file parameter to 'True' in order to capture the output of the next scheduled job run as a file. This file will be stored with an observer's log files and can be used to debug observer issues, for example at the request of your designated Support team, or while using a test environment. For one-off jobs (that is, Load jobs), this parameter reverts to 'False' after the next completed run. To examine the output produced, you can load the generated debug file using the File Observer. The file is saved to the following locations: On-prem $ASM_HOME/logs/<obs>-observer/ On OCP /var/log/itsm/<obs>-observer	Optional
   Observer job description	Enter additional information to describe the job.	Optional
   Job schedule	Specify when the job should run, and whether it should run at regular intervals. By default the job runs immediately, and only once. Optionally you can specify a future date and time for the job to run, and then set it to run at regular intervals after that.	Optional. Transient (one-off) jobs only. If you set a job schedule, the run intervals must be at least 90 seconds apart, and if you set them at less than 15 minutes, a warning is displayed, as the frequency can impact system performance.

Deploy Weave Scope in your Kubernetes environment

1. Create Namespace 'weave' with 'ibm-privileged-psp'.

   kubectl create namespace weave 
   kubectl -n weave create rolebinding weave-clusterrole-rolebinding --clusterrole=ibm-privileged-clusterrole --group=system:serviceaccounts:weave
   

2. Install Weave Scope using the following command:

   kubectl apply -f "https://cloud.weave.works/k8s/scope.yaml?k8s-service-type=NodePort&k8s-version=$(kubectl version | base64 | tr -d '\n')"

   This will result in a port being opened that the Observer can use.
3. You can discover the NodePort using the following command:

   kubectl -n weave describe service weave-scope-app

4. Launch the Weave Scope UI using the following URL:

   https://<master ip>:<NodePort>

5. Create a new route using the OpenShift UI to expose the OCP Weave Scope service under the weave namespace. In the left panel of the OpenShift UI, click Networking > Routes.
6. Complete the required fields and then click Create.
   A new route has been created. Launch the Weave Scope UI using the location link.

Define the Weave Scope job

7. On the Observer jobs page, perform one of the following actions:

   To edit an existing job

   Open the List of options overflow menu next to the job and click View & edit.

   To create a new job

   Click Add a new job + and select the Kubernetes Observer tile.

8. Choose the Weave Scope job type from the drop-down, define or edit the following parameters, then click Save to save and run the job.

   Table 3. Kubernetes Observer Weave Scope job parameters

   Parameter	Action	Details
   Unique ID	Enter a unique name for the job. Note: The connection name cannot be the same as any of the following observer job names: 'local', 'weave_scope', 'load', 'listen', 'kubeconfig'.	Required
   Host	Enter the Weave Scope host name (or IP address) of the web socket to be observed.	Required
   Port	Enter the Weave Scope port number of the web socket to be observed.	Required
   Cluster Name	Enter the name of the cluster or data center to be observed.	Required
   Namespaces	Enter a list of namespaces to be observed.	Optional. If left empty, all namespaces will be observed.
   Resource types	Select the Weave Scope resource types to observe.	Optional.
   Resources to exclude	List resources to be excluded by ID, label, rank or namespace.	Optional. Containers named 'pod' are excluded by default.
   Refresh interval	The refresh interval in seconds between updates. The minimum allowed value is 60s. Default value is 120s.	Optional
   Access scope	Enter text to provide a scope for the resources. Access scope can help map alerts to resources when resources in different scopes share the same parameters, such as matchTokens.	Optional. Tip: You can define access scope for locations, project names, namespaces, etc.
   Generate debug support file	Set the optional Generate debug support file parameter to 'True' in order to capture the output of the next scheduled job run as a file. This file will be stored with an observer's log files and can be used to debug observer issues, for example at the request of your designated Support team, or while using a test environment. For one-off jobs (that is, Load jobs), this parameter reverts to 'False' after the next completed run. To examine the output produced, you can load the generated debug file using the File Observer. The file is saved to the following locations: On-prem $ASM_HOME/logs/<obs>-observer/ On OCP /var/log/itsm/<obs>-observer	Optional
   Observer job description	Enter additional information to describe the job.	Optional
   Job schedule	Specify when the job should run, and whether it should run at regular intervals. By default the job runs immediately, and only once. Optionally you can specify a future date and time for the job to run, and then set it to run at regular intervals after that.	Optional. Transient (one-off) jobs only. If you set a job schedule, the run intervals must be at least 90 seconds apart, and if you set them at less than 15 minutes, a warning is displayed, as the frequency can impact system performance.

Verify your kubeconfig.yml file

1. Ensure your kubeconfig.yml file is in the correct format.
   The kubeconfig must meet the following criteria:

   • Single cluster
   • User type of 'token'
   • Base64 encoded JSON
   • The token and certificate must not be omitted or redacted
   1. View the kubeconfig.yml file:

      oc config view --raw -o json

   2. If it meets all criteria, encode it using the following command:

      Meets all criteria

      Encode the kubeconfig.yml file using the following command:

      oc config view --raw -o json | base64 -w 0

      Next, define the job.

      Does not meet all criteria

      As described in steps two to seven: Create a cluster-role or role, service account, cluster-role/role binding; from the service account, get the secret, and from the secret get the token; then generate a new kubeconfig.yml file, before defining the job.

Create a service account in the Kubernetes environment and obtain its token

2. Complete steps one to four in the Kubernetes Load job procedure.

Generate a new kubeconfig.yml file

3. Get the current context details:

   oc config current-context

4. Get the current cluster, cluster ca (certificate-authority-data), and cluster server details:

   oc config view --raw -o json

5. Create a kubeconfig.yml file using the obtained data.
   Sample kubeconfig.yml file:

   apiVersion: v1
   kind: Config
   current-context: <CURRENT_CONTEXT>
   contexts:
   - name: <CURRENT_CONTEXT>
    context:
      cluster: <CURRENT_CONTEXT>
      user: <USER>
      namespace: <NAMESPACE>
   clusters:
   - name: <CURRENT_CONTEXT>
    cluster:
      certificate-authority-data: <CLUSTER_CA>
      server: <CLUSTER_SERVER>
   users:
   - name: <USER>
    user:
      token: <TOKEN_VALUE>

   Where USER is the user-defined value, and NAMESPACE is the namespace of the service account.
6. Test the kubeconfig validity and access:

   oc --kubeconfig <config>.yml get all

7. Run the following command to convert the file to base64 encoded json:

   oc --kubeconfig <config>.yml config view --raw -o json | base64 -w 0

Define the Kubeconfig job

8. On the Observer jobs page, perform one of the following actions:

   To edit an existing job

   Open the List of options overflow menu next to the job and click View & edit.

   To create a new job

   Click Add a new job + and select the Kubernetes Observer tile.

9. Choose the Kubeconfig job type from the drop-down, define or edit the following parameters, then click Save to save and run the job.

   Table 4. Kubernetes Observer Kubeconfig job parameters

   Parameter	Action	Details
   Unique ID	Enter a unique name for the job. Note: The connection name cannot be the same as any of the following observer job names: 'local', 'weave_scope', 'load', 'listen', 'kubeconfig'.	Required
   Data center	Specify the name of the data center in which the Kubernetes instance is running. The data center name is treated as the provider of the data. Note: If you change the data center name, the observation is treated as a separate observation under the different provider, and a duplicate set of resources and groups will be created under the new provider.	Required
   kubeconfig	Provide the contents of the kubeconfig in json format. The kubeconfig needs to be a json with a single cluster and its user needs to be of token type.	Required. Use plain text.
   Terminated pods	Choose whether terminated pods should be hidden (true or false).	Optional. The default is false.
   API query timeout (ms)	Specify the Kubernetes REST API query timeout	Optional. The default is 5000 ms (that is, 5 seconds)
   Correlate	If 'true', enables the Event Analytics correlation on the namespace groups	Optional
   Names of the custom resource definitions	A number of custom resource definitions (CRDs) that can be queried from the Kubernetes API	Optional
   Role permission token	Determine whether the provided token is a role token. Default to false for cluster role permissions.	Optional
   Namespace	Specify the Kubernetes namespace.	Optional. Specify the Kubernetes namespace to observe, or * to observe all. Leaving this field blank will observe the namespace in which IBM Cloud Pak for AIOps is installed. When using a role permission token, specify the single namespace where the role was created.
   Access scope	Enter text to provide a scope for the resources. Access scope can help map alerts to resources when resources in different scopes share the same parameters, such as matchTokens.	Optional. Tip: You can define access scope for locations, project names, namespaces, etc.
   Generate debug support file	Set the optional Generate debug support file parameter to 'True' in order to capture the output of the next scheduled job run as a file. This file will be stored with an observer's log files and can be used to debug observer issues, for example at the request of your designated Support team, or while using a test environment. For one-off jobs (that is, Load jobs), this parameter reverts to 'False' after the next completed run. To examine the output produced, you can load the generated debug file using the File Observer. The file is saved to the following locations: On-prem $ASM_HOME/logs/<obs>-observer/ On OCP /var/log/itsm/<obs>-observer	Optional
   Observer job description	Enter additional information to describe the job.	Optional
   Job schedule	Specify when the job should run, and whether it should run at regular intervals. By default the job runs immediately, and only once. Optionally you can specify a future date and time for the job to run, and then set it to run at regular intervals after that.	Optional. Transient (one-off) jobs only. If you set a job schedule, the run intervals must be at least 90 seconds apart, and if you set them at less than 15 minutes, a warning is displayed, as the frequency can impact system performance.