Verifying completeness of license usage data and troubleshooting
Learn how to check whether License Service is properly deployed to your cluster and whether it collects the complete license usage data from your cluster. View the steps for troubleshooting most common problems.
- Validating License Service deployment
- Enabling additional information in logs for troubleshooting purposes
- Checking completeness of license usage data
- Interpreting the warnings
- Viewing information about the pods from which license usage cannot be collected
Common problems
- License Service instances are crashing with
CrashLoopBackOff
status - Duplicated products when retrieving license usage with
/products
API - License Service API is unavailable with 503 Service Unavailable error
- Multiple IBM Licensing instances
Validating License Service deployment
You can check whether the License Service is successfully deployed on the cluster with the available tools.
For example, you can log in to the cluster and run the following command:
kubectl get pods --all-namespaces | grep ibm-licensing | grep -v operator
The following response is a confirmation of successful deployment:
1/1 Running
Enabling additional information in logs for troubleshooting purposes
By default, the License Service instance pod logs contain only the basic information about the service. You can enable additional information in logs for troubleshooting purposes by updating the OperandConfig
. Add the logLevel
parameter to the ibm-licensing operator
section, under spec
.
Note: You can access the OperandConfig
by using the OpenShift Container Platform console or by using the command-line interface (CLI).
-
Access the
OperandConfig
.-
In OpenShift Container Platform console, go to Installed Operators > Operand Deployment Lifecycle Manager > OperandConfig tab. Edit the
common-service
OperandConfig
instance that the IBM Cloud Pak foundational services operator already created for you. Click the overflow menu icon, and then click Edit OperandConfig. -
Use the following command:
oc edit OperandConfig common-service -n <your-License-Service-namespace>
-
-
Add the following parameter to the
ibm-licensing operator
section, underspec
.name: ibm-licensing-operator spec: IBMLicensing: logLevel: <option>
The available
logLevel
options:DEBUG
- This option enables all debug information in logs.VERBOSE
- This option extends the logs with information about license calculation and API calls.
Checking completeness of license usage data
License Service runs health checks every day. The service scans the cluster, checks the validity of your pods and the underlying metadata and whether the data is properly imported and processed. If any information about your product, or its bundled product that is crucial for license measurements is missing, you get a warning in the logs.
Use a logging service, for example ELK stack or other third-party solution to monitor the logs. If you use ELK stack, you can filter the dashboard by the following tags to see the License Service warnings: MissingPodAnnotation
or DataImportProblem
.
Note: It is a good practice to check the logs regularly, and whenever you notice that a product is not being reported.
Interpreting the warnings
License Service generates two types of warnings.
Missing pod annotations
The warnings from the service inform you when a pod is not properly annotated. Improper annotations result in the license usage of a product not being measured. The annotations include the product metadata, such as the product ID and metric, and are implemented by the IBM team that develops the product to enable license usage measurements.
Example - Missing pod annotations warning
Each warning contains a timestamp, information about a problematic pod, and a list of missing annotations. The following example shows a warning in the logs:
2020-Feb-11 12:38:59.597 [scheduling-1] WARN [MissingPodAnnotation]
Pod kube-system/iam-onboarding-mktmv has invalid annotation:
productChargedContainers,productMetric
Resolving the warnings
To resolve this warning, identify the software which is deployed on the pod that is listed in the warning. Contact IBM Support for the identified software and provide them with the text of the warning. The product team needs to improve or add annotations to the problematic product so that it can be reporter by License Service. License Service support cannot resolve this issue.
Problems with importing or processing data
The warnings from the service inform you when the data might not be imported or processed by License Service. More information is listed in the logs and depends on the nature of the problem.
Example - Problems with importing or processing data
2020-03-04 10:52:55.889 [scheduling-1] ERROR [DataImportProblem] Error while importing and processing data
Viewing information about the pods from which license usage cannot be collected
You can check the list of pods from which License Service might not collect license usage in the unrecognized_apps.csv
file that is retrieved with an audit snapshot. For more information, see Retrieving an audit snapshot.
License Service instances are crashing with CrashLoopBackOff
status
If your License Service pods are crashing and you see multiple instances of License Service with the CrashLoopBackOff
status in your OpenShift console, you might have License Service deployed to more than one namespace. As a result,
two License Service operators are running in two namespaces and the service crashes. The ibm-licensing-operator
should only be deployed in the foundational-services
namespace, however, if you deployed License Service
more than once, the older version of License Service might be deployed to kube-system
namespace.
Complete the following steps to fix the problem:
-
To check whether the
ibm-licensing-operator
is deployed tokube-system
namespace, run the following command:- Linux®:
kubectl get pod -n kube-system | grep ibm-licensing-operator
- Windows:
kubectl get pod -n kube-system | findstr ibm-licensing-operator
- Linux®:
-
If the response contains information about the running pod, uninstall License Service from
kube-system
namespace. To uninstall License Service in OpenShift console, complete the following steps:-
Log in to the OpenShift console.
-
Go to Operators > Installed Operators.
-
Switch to the
kube-system
project. -
Find the IBM Licensing Operator, click the overflow menu icon and select Uninstall Operator.
-
Duplicated products when retrieving license usage with /products
API
If different bundled products report license usage under the same product but with a different version, you might get duplicated products when you run the /products
API.
Note: By default the /products
query returns the data for the last 30 days. Because of that, after you upgrade to the newest version of IBM Cloud Pak foundational services you might still get duplicated products for
the days prior to the upgrade. You can mitigate this issue by filtering the data and setting the startDate
and endDate
parameters to view the usage for the specific date.
License Service API is unavailable with 503 Service Unavailable error
If you use OpenShift Container Platform 4.13.4 or earlier 4.13.x version, you must upgrade to OpenShift Container Platform version 4.13.5 or later. For more information, see Red Hat Product Errata .
You might get 503 Service Unavailable error when you make a License Service API call when you use the custom ingress certificate. The custom ingress certificate is not acceptable for License Service. To fix this issue, complete the following actions:
-
Generate the correct certificate for Fully Qualified Domain Name (FQDN) of License Service. To learn how to check the License Service URL, see Obtaining the License Service URL.
Multiple IBM Licensing instances
Licensing Service as a singleton doesn't support running multiple instances in the same environment simultaneously. In case if multiple IBM Licensing instances are present, then only the first-formed object (judged by creation timestamp) will be used to create a Licensing Service instance.
When performing the upgrade on the environment, if there are multiply instances, they will be preserved along with active Licensing Service Services, but Custom Resource will be annotated INACTIVE
. It is recommended to remove CR to
avoid incoherent behavior of the Licensing operator.
To display currently running instances, run the following command:
oc get IBMLicensing
The sample output:
NAME POD PHASE
instance2 Running
instance3
instance4
Only the instance2
in Running
Phase which means instance2
is the only that is ACTIVE
.
To resolve this issue, delete the INACTIVE
instances by using following command:
oc delete IBMLicensing <instance name>
Where, instance name is the name of INACTIVE
CR. In case of this example, the instance name would be instance3
and instance4
.