Verifying completeness of license usage data and troubleshooting
Learn how to check whether License Service is properly deployed to your cluster and whether it collects the complete license usage data from your cluster. View the steps for troubleshooting most common problems.
- Validating License Service deployment
- Enabling additional information in logs for troubleshooting purposes
- Checking completeness of license usage data
- Interpreting the warnings
- Viewing information about the pods from which license usage cannot be collected
- License Service instances are crashing with
- Duplicated products when retrieving license usage with
- License Service API is unavailable with 503 Service Unavailable error
- Multiple IBM Licensing instances
Validating License Service deployment
License Service is automatically deployed with your product. You can check whether License Service was successfully deployed on the cluster by using any available tools. For example, you can log in to the cluster and run the following command:
kubectl get pods --all-namespaces | grep ibm-licensing | grep -v operator
The following response is a confirmation of successful deployment:
Enabling additional information in logs for troubleshooting purposes
By default, the License Service instance pod logs contain only the basic information about the service. You can enable additional information in logs for troubleshooting purposes by updating the
OperandConfig. Add the
logLevel parameter to the
ibm-licensing operator section, under
Note: You can access the
OperandConfig by using the OpenShift Container Platform console or by using the command-line interface (CLI).
In OpenShift Container Platform console, go to Installed Operators > Operand Deployment Lifecycle Manager > OperandConfig tab. Edit the
OperandConfiginstance that the IBM Cloud Pak foundational services operator already created for you. Click the overflow menu icon, and then click Edit OperandConfig.
Use the following command:
oc edit OperandConfig common-service -n <enter the namespace of the License Service>
Add the following parameter to the
ibm-licensing operatorsection, under
name: ibm-licensing-operator spec: IBMLicensing: logLevel: <option>
DEBUG- This option enables all debug information in logs.
VERBOSE- This option extends the logs with information about license calculation and API calls.
Checking completeness of license usage data
License Service runs health checks every day. The service scans the cluster, checks the validity of your pods and the underlying metadata and whether the data is properly imported and processed. If any information about your product, or its bundled product that is crucial for license measurements is missing, a warning appears in the logs.
Use a logging service, for example ELK stack or other third-party solution to monitor the logs. If you use ELK stack, you can filter the dashboard by the following tags to see the License Service warnings:
Note: It is a good practice to check the logs regularly, and whenever you notice that a product is not being reported.
Interpreting the warnings
License Service generates two types of warnings.
Missing pod annotations
The warnings from the service inform you when a pod is not properly annotated. Improper annotations result in the license usage of a product not being measured. The annotations include the product metadata, such as the product ID and metric, and are implemented by the IBM team that develops the product to enable license usage measurements.
Example - Missing pod annotations warning
Each warning contains a timestamp, information about a problematic pod, and a list of missing annotations. The following example shows a warning in the logs:
2020-Feb-11 12:38:59.597 [scheduling-1] WARN [MissingPodAnnotation] Pod kube-system/iam-onboarding-mktmv has invalid annotation: productChargedContainers,productMetric
Resolving the warnings
To resolve this warning, identify the software which is deployed on the pod that is listed in the warning. Contact IBM Support for the identified software and provide them with the text of the warning. The product team needs to improve or add annotations to the problematic product so that it can be reporter by License Service. License Service support cannot resolve this issue.
Problems with importing or processing data
The warnings from the service inform you when the data might not be imported or processed by License Service. More information is listed in the logs and depends on the nature of the problem.
Example - Problems with importing or processing data
2020-03-04 10:52:55.889 [scheduling-1] ERROR [DataImportProblem] Error while importing and processing data
Viewing information about the pods from which license usage cannot be collected
You can check the list of pods from which License Service might not collect license usage in the
unrecognized_apps.csv file that is retrieved with an audit snapshot. For more information, see Retrieving an audit snapshot.
License Service instances are crashing with
If your License Service pods are crashing and you see multiple instances of License Service with the
CrashLoopBackOff status in your OpenShift console, you might have License Service deployed to more than one namespace. As a result,
two License Service operators are running in two namespaces and the service crashes. The
ibm-licensing-operator should only be deployed in the
<foundational-services> namespace, however, if you deployed License
Service more than once, the older version of License Service might be deployed to
Complete the following steps to fix the problem:
To check whether the
ibm-licensing-operatoris deployed to
kube-systemnamespace, run the following command:
kubectl get pod -n kube-system | grep ibm-licensing-operator
kubectl get pod -n kube-system | findstr ibm-licensing-operator
If the response contains information about the running pod, uninstall License Service from
kube-systemnamespace. To uninstall License Service in OpenShift console, complete the following steps:
Log in to the OpenShift console.
Go to Operators > Installed Operators.
Switch to the
Find the IBM Licensing Operator, click the overflow menu icon and select Uninstall Operator.
Duplicated products when retrieving license usage with
If different bundled products report license usage under the same product but with a different version, you might get duplicated products when you run the
This is a known issue that is resolved with the 3.7.2 release of IBM Cloud Pak foundational services. Upgrade to the latest version of IBM Cloud Pak foundational services to fix this issue.
Note: By default the
/products query returns the data for the last 30 days. Because of that, after you upgrade to the newest version of IBM Cloud Pak foundational services you might still get duplicated products for
the days prior to the upgrade. You can mitigate this issue by filtering the data and setting the
endDate parameters to view the usage for the specific date.
License Service API is unavailable with 503 Service Unavailable error
If you use OpenShift Container Platform 4.13.4 or earlier 4.13.x version, you must upgrade to OpenShift Container Platform version 4.13.5 or later. For more information, see Red Hat Product Errata .
You might get 503 Service Unavailable error when you make a License Service API call when you use the custom ingress certificate. The custom ingress certificate is not acceptable for License Service. To fix this issue, complete the following actions:
Generate the correct certificate for Fully Qualified Domain Name (FQDN) of License Service. To learn how to check the License Service URL, see Obtaining the License Service URL.
Configure your custom certificate.
Multiple IBM Licensing instances
Licensing Service as a singelton doesn't support running multiple instances in the same environment simultaneously. In case if multiple IBM Licensing instances are present, then only the first-formed object (judged by creation timestamp) will be used to create a Licensing Service instance.
When performing the upgrade on the environment, if there are multiply instances, they will be preserved along with active Licensing Service Services, but Custom Resource will be annotated
INACTIVE. It is recommended to remove CR to
avoid incoherent behavior of the Licensing operator.
To display currently running instances, run the following command:
oc get IBMLicensing
The sample output:
NAME POD PHASE instance2 Running instance3 instance4
Running Phase which means
instance2 is the only that is
To resolve this issue, delete the
INACTIVE instances by using following command:
oc delete IBMLicensing <instance name>
Where, instance name is the name of
INACTIVE CR. In case of this example, the instance name would be