Verifying completeness of license usage data and troubleshooting

Learn how to check whether License Service is properly deployed to your cluster and whether it collects the complete license usage data from your cluster. View the steps for troubleshooting most common problems.

Common problems

Validating License Service deployment

You can check whether the License Service is successfully deployed on the cluster with the available tools.

For example, you can log in to the cluster and run the following command:

kubectl get pods --all-namespaces | grep ibm-licensing | grep -v operator

The following response is a confirmation of successful deployment:

 1/1     Running

Enabling additional information in logs for troubleshooting purposes

By default, the License Service instance pod logs contain only the basic information about the service. You can enable additional information in logs for troubleshooting purposes by updating the OperandConfig. Add the logLevel parameter to the ibm-licensing operator section, under spec.

Note: You can access the OperandConfig by using the OpenShift Container Platform console or by using the command-line interface (CLI).

  1. Access the OperandConfig.

    • In OpenShift Container Platform console, go to Installed Operators > Operand Deployment Lifecycle Manager > OperandConfig tab. Edit the common-service OperandConfig instance that the IBM Cloud Pak foundational services operator already created for you. Click the overflow menu icon, and then click Edit OperandConfig.

    • Use the following command:

      oc edit OperandConfig common-service -n <your-License-Service-namespace>

  2. Add the following parameter to the ibm-licensing operator section, under spec.

      name: ibm-licensing-operator
  spec:
    IBMLicensing:
      logLevel: <option>

    The available logLevel options:

    • DEBUG - This option enables all debug information in logs.
    • VERBOSE - This option extends the logs with information about license calculation and API calls.

Checking completeness of license usage data

License Service runs health checks every day. The service scans the cluster, checks the validity of your pods and the underlying metadata and whether the data is properly imported and processed. If any information about your product, or its bundled product that is crucial for license measurements is missing, you get a warning in the logs.

Use a logging service, for example ELK stack or other third-party solution to monitor the logs. If you use ELK stack, you can filter the dashboard by the following tags to see the License Service warnings: MissingPodAnnotation or DataImportProblem.

Note: It is a good practice to check the logs regularly, and whenever you notice that a product is not being reported.

Interpreting the warnings

License Service generates two types of warnings.

Missing pod annotations

The warnings from the service inform you when a pod is not properly annotated. Improper annotations result in the license usage of a product not being measured. The annotations include the product metadata, such as the product ID and metric, and are implemented by the IBM team that develops the product to enable license usage measurements.

Example - Missing pod annotations warning

Each warning contains a timestamp, information about a problematic pod, and a list of missing annotations. The following example shows a warning in the logs:

2020-Feb-11 12:38:59.597 [scheduling-1] WARN  [MissingPodAnnotation]
Pod kube-system/iam-onboarding-mktmv has invalid annotation:
productChargedContainers,productMetric

Resolving the warnings

To resolve this warning, identify the software which is deployed on the pod that is listed in the warning. Contact IBM Support for the identified software and provide them with the text of the warning. The product team needs to improve or add annotations to the problematic product so that it can be reporter by License Service. License Service support cannot resolve this issue.

Problems with importing or processing data

The warnings from the service inform you when the data might not be imported or processed by License Service. More information is listed in the logs and depends on the nature of the problem.

Example - Problems with importing or processing data

2020-03-04 10:52:55.889 [scheduling-1] ERROR [DataImportProblem] Error while importing and processing data

Viewing information about the pods from which license usage cannot be collected

You can check the list of pods from which License Service might not collect license usage in the unrecognized_apps.csv file that is retrieved with an audit snapshot. For more information, see Retrieving an audit snapshot.

License Service instances are crashing with CrashLoopBackOff status

If your License Service pods are crashing and you see multiple instances of License Service with the CrashLoopBackOff status in your OpenShift console, you might have License Service deployed to more than one namespace. As a result, two License Service operators are running in two namespaces and the service crashes. The ibm-licensing-operator should only be deployed in the foundational-services namespace, however, if you deployed License Service more than once, the older version of License Service might be deployed to kube-system namespace.

Complete the following steps to fix the problem:

  1. To check whether the ibm-licensing-operator is deployed to kube-system namespace, run the following command:

    • Linux®: kubectl get pod -n kube-system | grep ibm-licensing-operator
    • Windows: kubectl get pod -n kube-system | findstr ibm-licensing-operator

  2. If the response contains information about the running pod, uninstall License Service from kube-system namespace. To uninstall License Service in OpenShift console, complete the following steps:

    1. Log in to the OpenShift console.

    2. Go to Operators > Installed Operators.

    3. Switch to the kube-system project.

    4. Find the IBM Licensing Operator, click the overflow menu icon and select Uninstall Operator.

Duplicated products when retrieving license usage with /products API

If different bundled products report license usage under the same product but with a different version, you might get duplicated products when you run the /products API.

This is a known issue that is resolved with the 3.7.2 release of IBM Cloud Pak foundational services. Upgrade to the latest version of IBM Cloud Pak foundational services to fix this issue.

Note: By default the /products query returns the data for the last 30 days. Because of that, after you upgrade to the newest version of IBM Cloud Pak foundational services you might still get duplicated products for the days prior to the upgrade. You can mitigate this issue by filtering the data and setting the startDate and endDate parameters to view the usage for the specific date.

License Service API is unavailable with 503 Service Unavailable error

If you use OpenShift Container Platform 4.13.4 or earlier 4.13.x version, you must upgrade to OpenShift Container Platform version 4.13.5 or later. For more information, see Red Hat Product Errata Opens in a new tab.

You might get 503 Service Unavailable error when you make a License Service API call when you use the custom ingress certificate. The custom ingress certificate is not acceptable for License Service. To fix this issue, complete the following actions:

  1. Generate the correct certificate for Fully Qualified Domain Name (FQDN) of License Service. To learn how to check the License Service URL, see Obtaining the License Service URL.

  2. Configure your custom certificate.

Multiple IBM Licensing instances

Licensing Service as a singleton doesn't support running multiple instances in the same environment simultaneously. In case if multiple IBM Licensing instances are present, then only the first-formed object (judged by creation timestamp) will be used to create a Licensing Service instance.

When performing the upgrade on the environment, if there are multiply instances, they will be preserved along with active Licensing Service Services, but Custom Resource will be annotated INACTIVE. It is recommended to remove CR to avoid incoherent behavior of the Licensing operator.

To display currently running instances, run the following command:

oc get IBMLicensing

The sample output:

NAME        POD PHASE
instance2   Running
instance3
instance4

Only the instance2 in Running Phase which means instance2 is the only that is ACTIVE.

To resolve this issue, delete the INACTIVE instances by using following command:

oc delete IBMLicensing <instance name>

Where, instance name is the name of INACTIVE CR. In case of this example, the instance name would be instance3 and instance4.