Configuring Kubernetes Observer jobs

Using this observer, you can configure jobs that discover the structure of your Kubernetes clusters, including pods, worker nodes and containers.

Before you begin

Important: The Kubernetes Observer has been tested against OpenShift version 4.12.53 with Kubernetes 1.25.8, and GKE version 1.27.8-gke.1067004.

Enabling access to the URL routes

To access the URL routes for the topology Swagger documentation, see the Enabling access to URL routes topic.

About this task

The observer reads topology data from Kubernetes through its REST APIs, or Weave Scope.

You can run the following jobs:

Load

A transient (one-off) job that loads all requested topology data from a Kubernetes environment.

By default, Load jobs are one-off, transient jobs that do a full upload of all requested topology data as soon as they are triggered.

You can also run these jobs (again) manually from the Observer UI, or schedule them to run at set times when configuring them.

Tip: When observing different Kubernetes clusters, make sure that the name of the data center is unique across all of the jobs.

Listen

A long-running job that monitors the Kubernetes REST API for updated resources, and loads them in the topology service.

Important: A Load job must be run before a Listen job can run.

Local

Note: The Kubernetes Observer Local job has been deprecated as of IBM Cloud Pak for AIOps 3.4.0. Use the Load job instead to performs local observations of the Kubernetes REST API for available resources.

Weave_scope Listen

Note: The Kubernetes Observer Weave Scope listen job has been deprecated as of IBM Cloud Pak for AIOps 4.5.0.

The Weave Scope listen job is a stand-alone job that listens to the Weave Scope agent and continues to stream topology and state data to IBM Cloud Pak® for AIOps.

It provides visibility of your Kubernetes services, pods, containers, deployments, stateful sets, Cron Jobs and processes for a specified namespace.

It is a long-running job that monitors its source for updates and runs until it is explicitly stopped, or until the Observer is stopped.

Kubeconfig Load

A stand-alone load job that accepts kubeconfig data in JSON format to perform a one-off discovery.

Kubernetes Observer Load job prerequisites

Before defining a Kubernetes Observer Load job, you must create a service account in the Kubernetes environment and obtain its token. The following example uses the default namespace.

  1. Create either cluster role access or role access.

    For clusterole access, create a configuration file called asm-k8s-observer.yaml with the custom cluster role asm:kubernetes-observer.

    Use the following sample content:

    kind: ClusterRole
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      annotations:
        rbac.authorization.kubernetes.io/autoupdate: "true"
      name: asm:kubernetes-observer
    rules:
    - apiGroups: [""]
      resources: [ "pods", "namespaces", "nodes", "services", "endpoints", "persistentvolumes", "persistentvolumeclaims"]
      verbs: ["get", "list"]    
    - apiGroups: ["apps"]
      resources: ["replicasets", "deployments", "statefulsets", "daemonsets"]
      verbs: ["get", "list"]
    

    For role access:

    Use the following sample content:

    kind: Role
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      namespace: {namespace} <-- Specify which namespace to create the Role access
      name: asm:kubernetes-observer
    rules:
    - apiGroups: [""]
      resources: [ "pods", "services", "endpoints"]
      verbs: ["get", "list"]
    - apiGroups: ["apps"]
      resources: ["replicasets", "deployments", "statefulsets", "daemonsets"]
      verbs: ["get", "list"]     
    

    Observing Custom Resources:

    To model custom resource definitions (CRDs) and custom resources (CRs) in the topology, identify the names of the CRDs that are present in the environment by running the oc get crds command. Then, update the job parameters to include the names of CRDs to be discovered. When a CRD is being included in the job, the respective CRs will also be discovered.

    Example (discovering ibm.com CRDs):

    oc get crds | grep ibm.com
    

    Sample output:

    asmformations.asm.ibm.com                                   2020-12-03T01:53:38Z
    asms.asm.ibm.com                                            2020-12-03T01:53:38Z
    cemformations.cem.ibm.com                                   2020-12-03T01:53:38Z
    cemserviceinstances.cem.ibm.com                             2020-12-03T01:53:38Z
    noiconnectionlayers.noi.ibm.com                             2020-12-03T01:47:19Z
    noiformations.noi.ibm.com                                   2020-12-03T01:47:19Z
    noihybrids.noi.ibm.com                                      2020-12-03T01:47:19Z
    nois.noi.ibm.com                                            2020-12-03T01:47:20Z
    

    Also, change the existing ClusterRole in the target environment, to include the following resources:

    - apiGroups: ["apiextensions.k8s.io"]
      resources: ["customresourcedefinitions"]
      verbs: ["get", "list"]
    - apiGroups: ["noi.ibm.com"]
      resources: ["nois", "noihybrids", "noiformations", "noiconnectionlayers"]
      verbs: ["get", "list"]
    - apiGroups: ["asm.ibm.com"]
      resources: ["asms", "asmformations"]
      verbs: ["get", "list"]
    - apiGroups: ["cem.ibm.com"]
      resources: ["cemformations"]
      verbs: ["get", "list"]
    

    The apiextensions.k8s.io segment of the cluster role provides your permission required to access all the custom resource definitions resources.

    - apiGroups: ["apiextensions.k8s.io"]
      resources: ["customresourcedefinitions"]
      verbs: ["get", "list"]
    

    The asm.ibm.com segment provides access to asms and asmformations custom resources. The names of the resources listed are the plural for each of the resources. They are usually derived from the first word of the CRD.

    - apiGroups: ["asm.ibm.com"]
      resources: ["asms", "asmformations"]
      verbs: ["get", "list"]
    

    For a fresh discovery:

    Create a configuration file called asm-k8s-observer.yaml in the target environment using the yaml resource provided in the sample content, then run the following command before moving to the next step:

    oc create -f asm-k8s-observer.yaml
    

    For an existing discovery:

    Create a configuration file called asm-k8s-observer.yaml using the yaml resource provided in the sample, then run the following command to patch the clusterole:

    oc patch clusterrole asm:kubernetes-observer --patch "$(cat asm-k8s-observer.yaml )"
    
  2. Create the service account in the target environment.

    oc create serviceaccount asm-k8s-account -n <namespace>
    

    To verify that the service account exists:

    oc get serviceaccount
    

    Important: When running the following command, ensure you know the namespace in which it was created:

    oc create serviceaccount asm-k8s-account
    
  3. Bind the asm:kubernetes-observer ClusterRole or role to the asm-k8s-account service account.

    Important: The service account and the cluster role binding need to be created in the same namespace.

    Example command for clusterole binding:

    oc create clusterrolebinding asm-k8s --clusterrole=asm:kubernetes-observer --serviceaccount={namespace}:asm-k8s-account
    

    Example command for role binding:

    oc create rolebinding asm-k8s --role=asm:kubernetes-observer --serviceaccount={namespace}:asm-k8s-account
    

    Example role binding output:

    rolebinding.rbac.authorization.k8s.io/asm-k8s created
    
  4. Obtain the Kubernetes service account token. Complete the following substeps depending on your version of Kubernetes.

    For Kubernetes on OpenShift:

    1. Look for the mounted service account token secret from the secrets section:

      oc describe serviceaccount asm-k8s-account
      

      Example:

      oc describe serviceaccount asm-k8s-account
      Name: asm-k8s-account
      Namespace: default
      Labels: <none>
      Annotations: <none>
      Image pull secrets: asm-k8s-account-dockercfg-wwzx2
      Mountable secrets: asm-k8s-account-dockercfg-wwzx2
      Tokens: asm-k8s-account-token-tzl52
      Events: <none>      
      
    2. Describe the asm-k8s-account-token-******* (which in this example is tzl52) to obtain the token's value:

      oc describe secret asm-k8s-account-token-tzl52
      

      Alternatively, use the following method to obtain the token's value (applicable to both OpenShift and GKE environments):

      oc describe serviceaccount asm-k8s-account | grep Tokens | awk '{ print $2} ' | xargs -I{} oc get secret {} --template='{{.data.token}}' | base64 -d
      

      For Kubernetes load jobs, ensure you have the Kubernetes service details to hand, such as the Kubernetes host IP and SSL Certificate details.

    For Kubernetes on GKE:

    1. Verify that the token has been created using the kubectl describe serviceaccount <service-account> command.

      Example of a missing token:

      oc kubectl describe serviceaccount asm-k8s-account
      Name:                asm-k8s-account
      Namespace:           default
      Labels:              <none>
      Annotations:         <none>
      Image pull secrets:  <none>
      Mountable secrets:   <none>
      Tokens:              <none>
      Events:              <none>
      
    2. If missing, create the token manually.

      Example:

      kubectl apply -f - <<EOF
      apiVersion: v1
      kind: Secret
      metadata:
         name: asm-k8s-account-secret
         annotations:
            kubernetes.io/service-account.name: asm-k8s-account
      type: kubernetes.io/service-account-token
      EOF
      secret/asm-k8s-account-secret created
      
    3. Verify that the token exists.

      Example of a token that is present:

      oc kubectl describe serviceaccount asm-k8s-account
      Name:                asm-k8s-account
      Namespace:           default
      Labels:              <none>
      Annotations:         <none>
      Image pull secrets:  <none>
      Mountable secrets:   <none>
      Tokens:              asm-k8s-account-secret
      Events:              <none>
      
    4. Describe the secret to obtain the token's value.

      Example:

      oc kubectl describe secret asm-k8s-account-secret
      Name:         asm-k8s-account-secret
      Namespace:    default
      Labels:       <none>
      Annotations:  kubernetes.io/service-account.name: asm-k8s-account
                    kubernetes.io/service-account.uid: d82e86c1-2258-4072-8a53-f517f776a149
      
      Type:  kubernetes.io/service-account-token
      
      Data
      ====
      ca.crt:     1509 bytes
      namespace:  7 bytes
      token:      eyJhbGciO
      

Kubernetes Observer Listen job prerequisites

Important: Before defining a Kubernetes Observer Listen job, you must have configured and run a Load job.

Before defining a Listen job, update the ClusterRole or Role configuration.

  1. Edit the asm-k8s-observer.yaml configuration file and add the following information:

    For clusterole access, add watch as in the following example.

    kind: ClusterRole
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      annotations:
        rbac.authorization.kubernetes.io/autoupdate: "true"
      name: asm:kubernetes-observer
    rules:
    - apiGroups: [""]
      resources: [ "pods", "namespaces", "nodes", "services", "endpoints", "persistentvolumes", "persistentvolumeclaims"]
      verbs: ["get", "list", "watch"]
    - apiGroups: ["apps"]
      resources: ["replicasets", "deployments", "statefulsets", "daemonsets"]
      verbs: ["get", "list", "watch"]
    

    For role access, add watch as in the following example.

    kind: Role
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      namespace: default
      name: asm:kubernetes-observer
    rules:
    - apiGroups: [""]
    resources: [ "pods", "services", "endpoints"]
    verbs: ["get", "list", "watch"]
    - apiGroups: ["apps"]
    resources: ["replicasets", "deployments", "statefulsets", "daemonsets"]
    verbs: ["get", "list", "watch"]
    
  2. Run the following command:

    oc apply -f asm-k8s-observer.yaml
    

Kubernetes Observer Weave Scope listen job prerequisites

For Weave Scope listen jobs, do a Weave Scope installation, and then configure a job using the Weave Scope IP and port parameters. You must install Weave Scope and then use the Weave Scope Master IP and Node port parameters. For more information on Weave Scope, see the following location: https://www.weave.works/docs/scope/latest/introducing/

  1. Create namespace 'weave' with 'ibm-privileged-psp'.

oc create namespace weave oc -n weave create rolebinding weave-clusterrole-rolebinding --clusterrole=ibm-privileged-clusterrole --group=system:serviceaccounts:weave


2. Install Weave Scope using the following command:

  ```
oc apply -f "https://cloud.weave.works/k8s/scope.yaml?k8s-service-type=NodePort&k8s-version=$(kubectl version | base64 | tr -d '\n')"
This will result in a port being opened that the observer can use.
  1. You can discover the NodePort using the following command:

oc -n weave describe service weave-scope-app


4. Launch the Weave Scope UI using the following URL:

  ```
https://<master ip>:<NodePort>
  1. Create a new route using the Red Hat OpenShift Container Platform console to expose the Red Hat OpenShift Container Platform Weave Scope service under the weave namespace. In the left panel of the Red Hat OpenShift Container Platform console, click Networking > Routes.

  2. Click Create Route to create a new route.

  3. Complete the required fields and then click Create.

  4. A new route has been created. Launch the Weave Scope UI using the location link.

Kubernetes Observer kubeconfig load job prerequisites

For a kubeconfig load job, ensure your kubeconfig.yml file is in the correct format. If it is not, generate a new file.

Important: The kubeconfig should meet the following criteria:

  • Single cluster
  • User type of 'token'
  • Base64 encoded JSON

View the kubeconfig.yml file:

oc config view --raw -o json

Important: Make sure the token and certificate are not omitted or redacted.

If it meets all required conditions, encode it using the following command:

oc config view --raw -o json | base64 -w 0

If it does not meet all conditions, generate a new kubeconfig.yml file using the following steps:

  1. Create a cluster-role or role, service account, cluster-role/role binding. From the service account, get the secret, and from the secret get the token. Refer to "Kubernetes Observer Load job prerequisites" for detailed steps.

  2. Get the current context details:

    oc config current-context
    
  3. Get the current cluster, cluster ca (certificate-authority-data), and cluster server details:

    oc config view --raw -o json
    
  4. Create a kubeconfig.yml using the obtained data. Sample kubeconfig.yml file:

    apiVersion: v1
    kind: Config
    current-context: <CURRENT_CONTEXT>
    contexts:
    - name: <CURRENT_CONTEXT>
      context:
        cluster: <CURRENT_CONTEXT>
        user: <USER>
        namespace: <NAMESPACE>
    clusters:
    - name: <CURRENT_CONTEXT>
      cluster:
        certificate-authority-data: <CLUSTER_CA>
        server: <CLUSTER_SERVER>
    users:
    - name: <USER>
      user:
        token: <TOKEN_VALUE>
    

    Where USER is the user-defined value, and NAMESPACE is the namespace of the service account.

  5. Test the kubeconfig validity and access:

    oc --kubeconfig <config>.yml get all
    
  6. Run the following command to convert the file to base64 encoded json:

    oc --kubeconfig <config>.yml config view --raw -o json | base64 -w 0
    

Procedure

Define or edit the following parameters, then click Run job to save and run the job.

Encryption requirement: See the Configuring observer jobs security topic for more information.

Load job parameters

Parameter Action Details
Unique ID Enter a unique name for the job. Note: The name cannot be the same as any of the following observer job names: 'local', 'listen', 'weave_scope', 'load', 'kubeconfig'. Required
data_center Specify the name of the data center in which the Kubernetes instance is running. The data center name is treated as the provider of the data. Note: If you change the data center name, the observation is treated as a separate observation under the different provider, and a duplicate set of resources and groups will be created under the new provider. Required
Kubernetes Master IP address Enter the Kubernetes Master IP address Required. Get the Kubernetes master IP and its API port using the 'oc cluster-info' command.
Kubernetes API port Enter the Kubernetes API port number Required
Kubernetes token The service account token for Kubernetes Required
Trust all certificate by bypassing certificate verification Enter true if you want to connect to Kubernetes without certificate Required
Exact HTTPS certificate file name Enter the exact name of the SSL/HTTPS certificate Optional. If 'Trust all certificate' is set to false, then this parameter is Required.
SSl Validation Choose whether SSL validation is true or false. If false, HTTPS without hostname validation is used. Optional
Terminated pods Choose whether terminated pods should be hidden (true or false) Optional. The default is false.
API query timeout (ms) Specify the Kubernetes REST API query timeout Optional. The default is 5000 ms (that is, 5 seconds)
Correlate If 'true', enables the Event Analytics correlation on the namespace groups Optional
Names of the custom resource definitions Specify the custom resource definitions (CRDs) queryable from the Kubernetes API Optional
Role permission token Determine whether the provided token is a role token. Defaults to false for cluster role permissions. Optional
Namespace Specify the Kubernetes namespace Optional. Specify the Kubernetes namespace to observe, leave blank or enter * to observe all.
Access scope Enter text to provide a scope for the resources. Access scope can help map alerts to resources when resources in different scopes share the same parameters, such as matchTokens. Optional. Tip: You can define access scope for locations, project names, namespaces, etc.
Generate debug support file Set this parameter to 'True' in order to capture the output of the next scheduled job run as a file. This file will be stored with an observer's log files and can be used to debug observer issues, for example at the request of your designated support team, or while using a test environment. For one-off jobs (that is, Load jobs), this parameter reverts to 'False' after the next completed run. To examine the output produced, you can load the generated debug file using the File Observer. Optional
Observer job description Enter additional information to describe the job Optional
Job schedule Specify when the job runs Optional. Load jobs only.

Listen job parameters

Parameter Action Details
Unique ID Enter a unique name for the job. Note: The name cannot be the same as any of the following observer job names: 'local', 'listen', 'weave_scope', 'load', 'kubeconfig'. Required
data_center Specify the name of the data center in which the Kubernetes instance is running. The data center name is treated as the provider of the data. Note: If you change the data center name, the observation is treated as a separate observation under the different provider, and a duplicate set of resources and groups will be created under the new provider. Required
Kubernetes Master IP address Enter the Kubernetes Master IP address Required. Get the Kubernetes master IP and its API port using the 'oc cluster-info' command.
Kubernetes API port Enter the Kubernetes API port number Required
Kubernetes token The service account token for Kubernetes Required
Trust all certificate by bypassing certificate verification Enter true if you want to connect to Kubernetes without certificate Required
Exact HTTPS certificate file name Enter the exact name of the SSL/HTTPS certificate Optional. If 'Trust all certificate' is set to false, then this parameter is Required.
SSl Validation Choose whether SSL validation is true or false. If false, HTTPS without hostname validation is used. Optional
Terminated pods Choose whether terminated pods should be hidden (true or false) Optional. The default is false.
API query timeout (ms) Specify the Kubernetes REST API query timeout Optional. The default is 5000 ms (that is, 5 seconds)
Correlate If 'true', enables the Event Analytics correlation on the namespace groups Optional
Names of the custom resource definitions A number of custom resource definitions (CRDs) that can be queried from the Kubernetes API Optional
Role permission token Determine whether the provided token is a role token. Default to false for cluster role permissions. Optional
Namespace Specify the Kubernetes namespace Optional. Specify the Kubernetes namespace to observe, leave blank or enter * to observe all.
Access scope Enter text to provide a scope for the resources. Access scope can help map alerts to resources when resources in different scopes share the same parameters, such as matchTokens. Optional. Tip: You can define access scope for locations, project names, namespaces, etc.
Generate debug support file Set this parameter to 'True' in order to capture the output of the next scheduled job run as a file. This file will be stored with an observer's log files and can be used to debug observer issues, for example at the request of your designated support team, or while using a test environment. For one-off jobs (that is, Load jobs), this parameter reverts to 'False' after the next completed run. To examine the output produced, you can load the generated debug file using the File Observer. Optional
Observer job description Enter additional information to describe the job Optional
Job schedule Specify when the job runs Optional. Load jobs only.

Weave_scope job parameters

Parameter Action Details
Unique ID Enter a unique name for the job. Note: The name cannot be the same as any of the following observer job names: 'local', 'weave_scope', 'load', 'kubeconfig'. Required
Host Enter the Weave Scope hostname (or IP address) of the web socket to be observed Required
Port Enter the Weave Scope port number of the web socket to be observed Required. The Port can be obtained using the 'oc -n weave describe service weave-scope-app' command.
Cluster Name Enter the name of the cluster or data center to be observed Required
Namespaces Enter a list of namespaces to be observed Optional. If left empty, all namespaces will be observed.
Resource types Select the Weave Scope resource types to observe Optional.
Resources to exclude List resources to be excluded by ID, label, rank or namespace Optional. Containers named 'pod' are excluded by default.
Refresh interval Specify the refresh interval in seconds between updates. The minimum allowed value is 60s, and the default value is 120s. Optional
Access scope Enter text to provide a scope for the resources. Access scope can help map alerts to resources when resources in different scopes share the same parameters, such as matchTokens. Optional. Tip: You can define access scope for locations, project names, namespaces, etc.
Generate debug support file Set this parameter to 'True' in order to capture the output of the next scheduled job run as a file. This file will be stored with an observer's log files and can be used to debug observer issues, for example at the request of your designated support team, or while using a test environment. For one-off jobs (that is, Load jobs), this parameter reverts to 'False' after the next completed run. To examine the output produced, you can load the generated debug file using the File Observer. Optional
Observer job description Enter additional information to describe the job Optional
Job schedule Specify when the job runs Optional. Load jobs only.

Kubeconfig load job parameters

Parameter Action Details
Unique ID Enter a unique name for the job. Note: The name cannot be the same as any of the following observer job names: 'local', 'weave_scope', 'load', 'kubeconfig'. Required
data_center Specify the name of the data center in which the Kubernetes instance is running. The data center name is treated as the provider of the data. Note: If you change the data center name, the observation is treated as a separate observation under the different provider, and a duplicate set of resources and groups will be created under the new provider. Required
kubeconfig Provide the contents of the kubeconfig in json format. The kubeconfig needs to be a json with a single cluster and its user needs to be of token type. Required. The json needs to be base64 encoded.
Terminated pods Choose whether terminated pods should be hidden (true or false) Optional. The default is false.
API query timeout (ms) Specify the Kubernetes REST API query timeout Optional. The default is 5000 ms (that is, 5 seconds)
Correlate If 'true', enables the Event Analytics correlation on the namespace groups Optional
Names of the custom resource definitions A number of custom resource definitions (CRDs) that can be queried from the Kubernetes API Optional
Role permission token Determine whether the provided token is a role token. Default to false for cluster role permissions. Optional
Namespace Specify the Kubernetes namespace Optional. Specify the Kubernetes namespace to observe, or * to observe all. Leaving this field blank will observe the namespace in which IBM Cloud Pak for AIOps is installed. When using a role permission token, specify the single namespace where the role was created.
Access scope Enter text to provide a scope for the resources. Access scope can help map alerts to resources when resources in different scopes share the same parameters, such as matchTokens. Optional. Tip: You can define access scope for locations, project names, namespaces, etc.
Generate debug support file Set this parameter to 'True' in order to capture the output of the next scheduled job run as a file. This file will be stored with an observer's log files and can be used to debug observer issues, for example at the request of your designated support team, or while using a test environment. For one-off jobs (that is, Load jobs), this parameter reverts to 'False' after the next completed run. To examine the output produced, you can load the generated debug file using the File Observer. Optional
Observer job description Enter additional information to describe the job Optional
Job schedule Specify when the job runs Optional. Load jobs only.