Troubleshooting License Service

Learn how to troubleshoot License Service problems.

Validating License Service deployment

You can check whether License Service was successfully deployed on the cluster by using any available tools. For example, you can log in to the cluster and run the following command:

kubectl get pods --all-namespaces | grep ibm-licensing | grep -v operator

The following response is a confirmation of successful deployment:

 1/1     Running

Enabling additional information in logs for troubleshooting purposes

By default, the License Service instance pod logs contain only the basic information about the service. You can enable additional information in the logs for troubleshooting purposes by updating the IBMLicensing instance.

  1. Use the following command to update the IBMLicensing instance:

    kubectl edit IBMLicensing instance -n <namespace>

    Where <namespace> is the namespace where the License Service is deployed.

  2. Add the following parameters to the spec section.

    spec:
  IBMLicensing:
    logLevel: <option>

    The available logLevel options:

  3. DEBUG - This option enables all debug information in logs.

  4. VERBOSE - This option extends the logs with information about license calculation and API calls.

Checking completeness of license usage data

License Service runs health checks every day. The service scans the cluster, checks the validity of your pods and the underlying metadata and whether the data is properly imported and processed. If any information about your product, or its bundled product that is crucial for license measurements is missing, a warning appears in the logs.

Use a logging service, for example ELK stack or other third-party solution to monitor the logs. If you use ELK stack, you can filter the dashboard by the following tags to see the License Service warnings: MissingPodAnnotation or DataImportProblem.

Note: It is a good practice to check the logs regularly, and whenever you notice that a product is not being reported.

Interpreting the warnings

License Service generates two types of warnings.

Missing pod annotations

The warnings from the service inform you when a pod is not properly annotated. Improper annotations result in the license usage of a product not being measured. The annotations include the product metadata, such as the product ID and metric, and are implemented by the IBM team that develops the product to enable license usage measurements.

Example - Missing pod annotations warning

Each warning contains a timestamp, information about a problematic pod, and a list of missing annotations. The following example shows a warning in the logs:

2020-Feb-11 12:38:59.597 [scheduling-1] WARN  [MissingPodAnnotation]
Pod kube-system/iam-onboarding-mktmv has invalid annotation:
productChargedContainers,productMetric

Resolving the warnings

To resolve this warning, identify the software which is deployed on the pod that is listed in the warning. Contact IBM Support for the identified software and provide them with the text of the warning. The product team needs to improve or add annotations to the problematic product so that it can be reporter by License Service. License Service support cannot resolve this issue.

Problems with importing or processing data

The warnings from the service inform you when the data might not be imported or processed by License Service. More information is listed in the logs and depends on the nature of the problem.

Example - Problems with importing or processing data

2020-03-04 10:52:55.889 [scheduling-1] ERROR [DataImportProblem] Error while importing and processing data

License Service pods are crashing and License Service cannot run

If your License Service pods are crashing, and you see multiple instances of License Service with the CrashLoopBackOff status in your OpenShift console, you might have License Service deployed to more than one namespace. As a result, two License Service operators are running in two namespaces and the service crashes. The ibm-licensing-operator should only be deployed in the foundational-services namespace, however, if you deployed License Service more than once, the older version of License Service might be deployed to kube-system namespace.

Complete the following steps to fix the problem:

  1. To check whether the ibm-licensing-operator is deployed to kube-system namespace, run the following command:

  2. Linux: kubectl get pod -n kube-system | grep ibm-licensing-operator

  3. Windows: kubectl get pod -n kube-system | findstr ibm-licensing-operator

  4. If the response contains information about the running pod, uninstall License Service from kube-system namespace.

License Service API is unavailable with 503 Service Unavailable error

You might get a 503 Service Unavailable error when you make a License Service API call when you use the custom ingress certificate. The custom ingress certificate is not acceptable for License Service. To fix this issue, complete the following actions:

  1. Generate the correct certificate for Fully Qualified Domain Name (FQDN) of License Service. Get the License Service URL. For example, to check the License Service URL, go to the OpenShift console, go to Networking > Routes. Find the ibm-licensing-service-instance route. The License Service URL is listed as Location.

  2. Configure your custom certificate.

Multiple IBM Licensing instances

Licensing Service as a singleton doesn't support running multiple instances in the same environment simultaneously. In the case if multiple IBM Licensing instances are present, then only the first-formed object (judged by creation timestamp) will be used to create a Licensing Service instance.

When performing the upgrade on the environment, if there are multiply instances, they are preserved along with active Licensing Services, but Custom Resource will be annotated INACTIVE. It is recommended to remove CR to avoid the incoherent behavior of the Licensing operator.

To display currently running instances, run the following command:

oc get IBMLicensing

The sample output:

NAME        POD PHASE
instance2   Running
instance3
instance4

Only the instance2 in Running Phase which means instance2 is the only that is ACTIVE.

To resolve this issue, delete the INACTIVE instances by using following command:

oc delete IBMLicensing <instance name>

Where, instance name is the name of INACTIVE CR. In case of this example, the instance name would be instance3 and instance4.

Out of memory error in License Service

The following out of memory error message is displayed in the License Service logs:

[scheduling-1] ERROR Unexpected error occurred in scheduled task java.lang.OutOfMemoryError: Java heap space at...

Cause

The out of memory error occurs because of the higher memory consumption of the License Service to process the large sets of objects from the cluster. The excessive memory usage reduces the available heap space.

Resolving the problem

To resolve the out of memory error, add the JAVA_TOOL_OPTIONS parameter to the IBMLicensing custom resource to set the initial and maximum memory allocation for the pool of a JAVA application:

  1. Edit the IBMLicensing custom resource:

    • For OpenShift console, go to Administration > CustomResourceDefinitions > IBMLicensing > Instances > instance > YAML.

    • For Kubernetes cluster, run the following command:

      kubectl edit IBMLicensing instance -n ibm-licensing

      Note: The ibm-licensing namespace is the default namespace of the License Service. If you installed the License Service in a custom namespace, replace ibm-licensing with the namespace where you deployed the License Service.

  2. Update the IBMLicensing custom resource to add the JAVA_TOOL_OPTIONS parameter.

    spec:
  resources:
    limits:
      cpu: '4'
      memory: 6Gi 
    requests:
      cpu: '2'
      memory: 4Gi
  envVariable:
    JAVA_TOOL_OPTIONS: '-Xms<enter the value of initial RAM allocation>m -Xmx<enter the value of maximum RAM allocation>m'

    Make sure that the value of memory in limits and requests parameters are higher than the values of Xms and Xmx in the JAVA_TOOL_OPTIONS parameter.

    Note: You can update the values of memory allocation in bytes for Xms and Xmx such as '-Xms2048m -Xmx4096m' based on your requirements for the License Service.