APAR status
Closed as program error.
Error description
The SQL backups are failing with following messages at the Spectrum Protect Plus job log: CTGGA2244,Mapping lun on volume spp_1010_2110_123456abcde to host (name:IBM-VM-SQL initiator:abc.234.com.microsoft:IBM-VM-SQL.xx.yyyy.com) CTGGA2245,SPP log dir: /data/log/guestdeployer/2019-12-07/1234009900000/1020304050607/I BM-VM-SQL.xx.yyyy.com CTGGA3052,[IBM-VM-SQL.xx.yyyy.com] Start mount command. CTGGA3054,[IBM-VM-SQL.xx.yyyy.com] Failed to mount for volume[spp_1010_2110_123456abcde] error Volume id/name is not found. CTGGA3053,[IBM-VM-SQL.xx.yyyy.com] Mount command complete CTGGA1987,[IBM-VM-SQL.xx.yyyy.com] Unmapping lun on volume spp_1010_2110_123456abcde from host for initiator:iqn.1991-05.com.microsoft:IBM-VM-SQL.xx.yyyy.com CTGGA2073,Failed to perform backup of db instance IBM-VM-SQL on application server IBM-VM-SQL.xx.yyyy.com : Failed to mount backup target volume . At the vSnap server used for the SQL backup, the message log indicates following: Dec 18 03.45:00 VSNAP kernel: WARNING: CPU: 0 PID: 8 at kernel/workqueue.c:2911 __flush_work.isra.30+0x1be/0x1d0 [... truncated ...] Dec 18 03.45:00 VSNAP kernel: Call Trace: Dec 18 03.45:00 VSNAP kernel: ? target_show_cmd+0x23d/0x280 [target_core_mod] Dec 18 03.45:00 VSNAP kernel: __cancel_work_timer+0x103/0x190 Dec 18 03.45:00 VSNAP kernel: ? __switch_to_asm+0x41/0x70 Dec 18 03.45:00 VSNAP kernel: ? __switch_to_asm+0x35/0x70 Dec 18 03.45:00 VSNAP kernel: ? __switch_to_asm+0x41/0x70 Dec 18 03.45:00 VSNAP kernel: ? __switch_to_asm+0x35/0x70 Dec 18 03.45:00 VSNAP kernel: core_tmr_lun_reset+0x551/0x7a0 [target_core_mod] Dec 18 03.45:00 VSNAP kernel: ? __switch_to_asm+0x35/0x70 Dec 18 03.45:00 VSNAP kernel: target_tmr_work+0x106/0x150 [target_core_mod] Dec 18 03.45:00 VSNAP kernel: process_one_work+0x171/0x370 Dec 18 03.45:00 VSNAP kernel: worker_thread+0x49/0x3f0 Dec 18 03.45:00 VSNAP kernel: kthread+0xf8/0x130 Dec 18 03.45:00 VSNAP kernel: ? max_active_store+0x80/0x80 Dec 18 03.45:00 VSNAP kernel: ? kthread_bind+0x10/0x10 Dec 18 03.45:00 VSNAP kernel: ret_from_fork+0x35/0x40 . That kernel issue is causing the LIO (Linux-IO) subsystem on the vSnap server to get hang. Since LIO is handing the iSCSI traffic, the SQL backup were failing because the iSCSI mount from the vSnap was failing. IBM Spectrum Protect Versions Affected: IBM Spectrum Protect Plus 10.1.4.x Initial Impact: Medium Additional Keywords: SPP, SPPLUS, TS002978359
Local fix
Reboot of the vSnap server used for the SQL backup.
Problem summary
**************************************************************** * USERS AFFECTED: * * IBM Spectrum Protect Plus level 10.1.4 and 10.1.5. * **************************************************************** * PROBLEM DESCRIPTION: * * See Error Description * **************************************************************** * RECOMMENDATION: * * Apply fixing level when available. This problem is currently * * projected to be fixed in IBM Spectrum Protect Plus levels * * 10.1.5 patch1 and 10.1.6. Note that this is subject to * * change at the discretion of IBM. * ****************************************************************
Problem conclusion
The newer Linux kernel 4.19 incorporated into vSnap contained some bugs introduced by upstream Linux kernel developers. Under certain conditions these bugs caused hangs of the LIO (Linux-IO) subsystem which is responsible for handling vSnap iSCSI traffic. The bugs were subsequently fixed in a later version of the upstream Linux kernel. The problem has been resolved by incorporating this newer kernel into vSnap.
Temporary fix
Comments
APAR Information
APAR number
IT31792
Reported component name
SP PLUS
Reported component ID
5737SPLUS
Reported release
A10
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
2020-02-20
Closed date
2020-02-21
Last modified date
2020-02-21
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
SP PLUS
Fixed component ID
5737SPLUS
Applicable component levels
[{"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SSNQFQ","label":"IBM Spectrum Protect Plus"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"A10","Line of Business":{"code":"LOB26","label":"Storage"}}]
Document Information
Modified date:
30 January 2024