IT34454: APPLICATION INVENTORY FOR KUBERNETES CLIENT STOPS WITH "ERROR MESSAGE: NOT FOUND"

APAR status

• Closed as program error.

Error description

• The inventory for a Kubernetes client using IBM Spectrum
  Protect Plus will display in the virgo log :
  
  
  [<timestamp>]
  INFO .. Starting job for policy Application Server
          Inventory. id -> <JobID>.
          IBM Spectrum Protect Plus version 10.1.6-2040
  ...
  .. INFO .. Unix Host Agent 10.1.6.3443 running as
             irSrH8ZGDIqp9ri for appdiscovery (task ID: <TaskID>)
  ..
     INFO .. Hostname: <KubernetesHostName> / Operating System:
             Red Hat Enterprise Linux Server 7.8 x86_64 /
             Python: 3.6.9
  .. INFO .. Discovering disks and mounted volumes
  .. INFO .. Discovering network and initiator information.
  .. INFO .. Discovering application instances.
  .. INFO .. --- Running discovery for k8s agent ---
  .. INFO .. Starting the inventory of a persistent volume.
  .. ERROR.. Failed to complete Kubernetes API call.
             Error message: Not Found
  .. ERROR.. An exception was encountered while running
             application discovery: (404).
  .. INFO .. [<KubernetesHostIP>] Reason: Not Found
  .. INFO .. [<KubernetesHostIP>] HTTP response headers:
             HTTPHeaderDict({'Content-Type': 'text/plain;
             charset=utf-8' 'X-Content-Type-Options': 'nosniff',
             'Date': '<timestamp>', 'Content-Length': '19'})
  .. INFO .. [<KubernetesHostIP>] HTTP response body: b'404 page
             not found\n'
  .. INFO .. Ending the inventory of a persistent volume.
  .. INFO .. --- Finished discovery for k8s agent ---
  ..
  .. INFO .. Job policy Application Server Inventory completed in
             37 sec(s) and ended with status PARTIAL. id <JobID>
  
  In the agent command log the following will be seen :
  
  [<timestamp>] JOBLOG_SUMMARY pid:914 MainThread jobsummary:
   <CTGGK3001> Starting the inventory of a persistent volume.
  .. updateAgentPodLabels: Updated baas:<KubernetesHostName> add
     label app.kubernetes.io/agent-version=10.1.6.20200716D
  .. api_call: Executing the following Kubernetes API:
     /apis/config.openshift.io/v1/clusteroperators/openshift-
     apiserver Method: GET
  .. getOpenshiftVersion: Openshift version not found. This is
     not openshift.
  .. api_call: Executing the following Kubernetes API: /version
     Method: GET
  .. error: <CTGGK3042> Failed to complete Kubernetes API call.
     Error message: Not Found
  .. executeSppCommand: Application agent is raising an exception
     to the host agent, traceback: 'Traceback (most recent call
     last):
     File "/opt/IBM/SPP/agents_irSrH8ZGDIqp9ri/k8s/main.py",
     line 139, in executeSppCommand
     File "/opt/IBM/SPP/agents_irSrH8ZGDIqp9ri/k8s/
     appdiscoveryx.py", line 398, in execute
     File "/opt/IBM/SPP/agents_irSrH8ZGDIqp9ri/k8s/
     appdiscoveryx.py", line 299, in appSdkInstance
     File "/opt/IBM/SPP/agents_irSrH8ZGDIqp9ri/k8s/
     appdiscoveryx.py", line 69, in appSdkDb
     File "/opt/IBM/SPP/agents_irSrH8ZGDIqp9ri/k8s/KubeCalls.py"
     , line 518, in getVolumeSnapshotClassSnapshotters
     File "/opt/IBM/SPP/agents_irSrH8ZGDIqp9ri/k8s/KubeCalls.py"
     , line 507, in getVolumeSnapshotClassSnapshotters
  .. executeSppCommand: Application agent is raising an exception
     to the host agent, traceback: 'Traceback (most recent call
     last):
  ...
     kubernetes.client.rest.ApiException: (404)
     Reason: Not Found
     HTTP response headers: HTTPHeaderDict({'Content-Type':
          'text/plain; charset=utf-8', 'X-Content-Type-Options':
          'nosniff', 'Date': '<timestamp>', 'Content-Length': '19'
          }) HTTP response body: b'404 page not found\n'
  .. ERROR pid:914 MainThread error: <CTGGK3044> An exception was
           encountered while running application discovery: (404)
           Reason: Not Found
  
  The Kubernetes agent is crashing when it fails to find any
  VolumeSnapshotClass objects.
  Instead of failing the Inventory, the process should report a
  warning when there is no v1alpha1 VolumeSnapshotClass object
  found.
  This can be confirmed by running the following command:
     kubectl get volumesnapshotclass
  A volumesnapshotclass is required to both be able to create
  snapshots and transfer data to the vSnap host during backup.
  This is missing from the requirements listed in the
  documentation.
  Despite the missing requirement, registration of the Kubernetes
  Application Server succeeds.
  The connection test towards the Kubernetes Application Server
  completes successfully without detecting the missing
  volumesnapshotclass.
  
  IBM Spectrum Protect Plus Versions Affected:
  IBM Spectrum Protect Plus 10.1.6
  
  Initial Impact: High
  
  Additional Keywords: SPP, SPPLUS, K8s, inventory
  

Local fix

• The missing snapshotclass can be declared for the Ceph Container
  Storage Interface (CSI) driver by following the documentation
  here :
  https://rook.io/docs/rook/v1.4/ceph-csi-snapshot.html
  
  Examples from ceph-csi-rbd are available here :
  https://github.com/ceph/ceph-csi/blob/release-v2.1/examples/rbd/
  snapshotclass.yaml
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * IBM Spectrum Protect Plus Container Backup Support level     *
  * 10.1.6 for Kubernetes                                        *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See ERROR DESCRIPTION.                                       *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply fixing level when available. This problem is currently *
  * projected to be fixed IBM Spectrum Protect Plus Container    *
  * Backup Support level 10.1.7.                                 *
  * Note that this is subject to change at the discretion of     *
  * IBM.                                                         *
  ****************************************************************
  

Problem conclusion

• If there is no available VolumeSnapshotClass object(s) all
  Container Storage Interface (CSI) PersistentVolumeClaim
  object(s) of the Kubernetes cluster will be marked as ineligible
  for backup.
  

Temporary fix

• The missing snapshotclass can be declared for the Ceph Container
  Storage Interface (CSI) driver by following the documentation
  here :
  https://rook.io/docs/rook/v1.4/ceph-csi-snapshot.html
  https://github.com/ceph/ceph-csi/blob/master/docs/snap-clone.md
  

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Modules/Macros

• k8s
  

Fix information

• Fixed component name

  SP PLUS

• Fixed component ID

  5737SPLUS

Applicable component levels