Troubleshooting License Service

Learn how to troubleshoot most common issues with License Service.

Validating License Service deployment

You can check whether License Service is successfully deployed on the cluster by using any available tools. For example, you can log in to the cluster and run the following command:

kubectl get pods --all-namespaces | grep ibm-licensing | grep -v operator

The following response is a confirmation of successful deployment:

 1/1     Running

Enabling additional information in logs for troubleshooting purposes

By default, the License Service instance pod logs contain only the basic information. You can enable additional information in the logs for troubleshooting purposes by updating the IBMLicensing instance.

Use the following command to update the IBMLicensing instance with a logLevel parameter that sets the format of the pod logs:

kubectl patch IBMLicensing <LS_instance_name> --type merge -p '{"spec":{"logLevel":"<LOG_LEVEL>"}}'

Where:

  • <LS_instance_name> is the name of your License Service instance. By default, the License Service instance name is instance.

  • <Log_options> is the value of the logLevel parameter that determines what is the scope of your pod logs. The following logLevel options are available:

    • DEBUG - This option enables all debug information in logs.
    • VERBOSE - This option extends the logs with information about license calculations and API calls.

    Note: If you no longer need additional information in the logs, you can set the value of the logLevel parameter to INFO, which sets the logs to display the default, basic information about License Service. Alternatively, you can remove the logLevel parameter from the IBMLicensing instance.

Checking completeness of license usage data

License Service runs health checks every day. The service scans the cluster, checks the validity of your pods and the underlying metadata and whether the data is properly imported and processed. If any information about your product, or its bundled product that is crucial for license measurements is missing, you get a warning in the logs.

Use a logging service, for example ELK stack or other third-party solution to monitor the logs. If you use ELK stack, you can filter the dashboard by the following tags to see the License Service warnings: MissingPodAnnotation or DataImportProblem.

Note: It is a good practice to check the logs regularly, and whenever you notice that a product is not being reported.

Interpreting the warnings

License Service generates two types of warnings.

Missing pod annotations

The warnings from the service inform you when a pod is not properly annotated. Improper annotations result in the license usage of a product not being measured. The annotations include the product metadata, such as the product ID and metric, and are implemented by the IBM team that develops the product to enable license usage measurements.

Example - Missing pod annotations warning

Each warning contains a timestamp, information about a problematic pod, and a list of missing annotations. The following example shows a warning in the logs:

2020-Feb-11 12:38:59.597 [scheduling-1] WARN  [MissingPodAnnotation]
Pod kube-system/iam-onboarding-mktmv has invalid annotation:
productChargedContainers,productMetric

Resolving the warnings

To resolve this warning, identify the software which is deployed on the pod that is listed in the warning. Contact IBM Support for the identified software and provide them with the text of the warning. The product team needs to improve or add annotations to the problematic product so that it can be reporter by License Service. License Service support cannot resolve this issue.

Problems with importing or processing data

The warnings from the service inform you when the data might not be imported or processed by License Service. More information is listed in the logs and depends on the nature of the problem.

Example - Problems with importing or processing data

2020-03-04 10:52:55.889 [scheduling-1] ERROR [DataImportProblem] Error while importing and processing data

Viewing information about the pods from which license usage cannot be collected

You can check the list of pods from which License Service might not collect license usage in the unrecognized_apps.csv file that is retrieved with an audit snapshot. For more information, see Retrieving an audit snapshot.

License Service pods are crashing and License Service cannot run

If your License Service pods are crashing, and you see multiple instances of License Service with the CrashLoopBackOff status in your OpenShift console, you might have License Service deployed to more than one namespace. As a result, two License Service operators are running in two namespaces and the service crashes. The ibm-licensing-operator should only be deployed in one namespace. However, if you deployed License Service more than once, the older version of License Service might be deployed to, for example, kube-system namespace.

Complete the following steps to fix the problem:

  1. To check where the ibm-licensing-operator is deployed, run the following command:
  • Linux: kubectl get pod -A | grep ibm-licensing-operator
  • Windows: kubectl get pod -A | findstr ibm-licensing-operator
  1. If in the response you get more than one running pod, uninstall License Service from any extra namespaces. Only one instance of License Service should be installed in the cluster.

Duplicated products when retrieving license usage with /products API

If different bundled products report license usage under the same product but with a different version, you might get duplicated products when you run the /products API.

Note: By default the /products query returns the data for the last 30 days. Because of that, after you upgrade to the newest version of License Service you might still get duplicated products for the days prior to the upgrade. You can mitigate this issue by filtering the data and setting the startDate and endDate parameters to view the usage for the specific date.

License Service API is unavailable with 503 Service Unavailable error

If you use OpenShift Container Platform 4.13.4 or earlier 4.13.x version, you must upgrade to OpenShift Container Platform version 4.13.5 or later. For more information, see Red Hat Product Errata Opens in a new tab.

You might get a 503 Service Unavailable error when you make a License Service API call when you use the custom ingress certificate. The custom ingress certificate is not acceptable for License Service. To fix this issue, complete the following actions:

  1. Generate the correct certificate for Fully Qualified Domain Name (FQDN) of License Service. Get the License Service URL. To learn how to check the License Service URL, see Obtaining the License Service URL.

  2. Configure your custom certificate.

Multiple License Service instances

You cannot run multiple instances of License Service in the same cluster. When multiple instances of License Service are present on a cluster, only the first-formed object (judged by creation timestamp) is used to create a License Service instance.

During the upgrade, if there are multiply instances of License Service, all instances are preserved. However, only one instance is annotated as ACTIVE. The custom resources of the remaining instance of License Service are annotated as INACTIVE. It is recommended to remove these CRs to avoid the incoherent behavior of the License Service operator.

To display all running instances of License Service, run the following command:

oc get IBMLicensing

The sample output:

NAME        POD PHASE
instance2   Running
instance3
instance4

Only instance2 is in the Running phase which means that instance2 is the only instance that is ACTIVE. instance3 and instance4 are annotated as INACTIVE.

To resolve the issue, delete the INACTIVE instances by using the following command:

oc delete IBMLicensing <instance name>

Where <instance name> is the name of the INACTIVE CR.

In this example, you need to delete instance3 and instance4 by running the following commands:

oc delete IBMLicensing instance3
oc delete IBMLicensing instance4

Out of memory error in License Service

Symptoms

The following out of memory error message is displayed in the License Service logs:

[scheduling-1] ERROR Unexpected error occurred in scheduled task java.lang.OutOfMemoryError: Java heap space at...

Cause

The out of memory error occurs because of the high memory consumption of License Service to process the large sets of objects from the cluster. The excessive memory usage reduces the available heap space.

Resolving the problem

To resolve the issue that is related to out of memory error, add the JAVA_TOOL_OPTIONS parameter to the IBMLicensing custom resource to set the initial and maximum memory allocation for the pool of the JAVA application:

  1. Edit the IBMLicensing custom resource:

    • For OpenShift console, go to Administration > CustomResourceDefinitions > IBMLicensing > Instances > instance > YAML.

    • For Kubernetes cluster, run the following command:

      kubectl edit IBMLicensing instance -n ibm-licensing

      Note: The recommended namespace for License Service installation is ibm-licensing. The commands contain this default namespace. However, you can install License Service in any other custom namespace. If you installed License Service in a custom namespace, replace ibm-licensing with the name of the custom namespace.

  2. Update the IBMLicensing custom resource to add the JAVA_TOOL_OPTIONS parameter.

    spec:
  resources:
    limits:
      cpu: '4'
      memory: 6Gi 
    requests:
      cpu: '2'
      memory: 4Gi
  envVariable:
    JAVA_TOOL_OPTIONS: '-Xms<enter the value of initial RAM allocation>m -Xmx<enter the value of maximum RAM allocation>m'

    Make sure that the value of memory in limits and requests parameters are higher than the values of Xms and Xmx in the JAVA_TOOL_OPTIONS parameter.

    Note: You can update the values of memory allocation in bytes for Xms and Xmx such as '-Xms2048m -Xmx4096m' based on your requirements for License Service.