APAR status
Closed as program error.
Error description
The backup of Oracle databases located in an Oracle Real Application Clusters (RAC) environment were the Database control files are housed in a shared location can inttermittently fail with the following messages seen in the virgo log : [<timestamp>] INFO .. <jobID> <DBName>: Completed datafile backups; total output bytes = 4639995985920 (4.22TB) [<timestamp>] INFO .. <jobID> <DBName>: Backing up archived logs to /mnt/spp/vsnap/vpool1/fs<xx>/<DBName>/arch for SCN range 634905906961 to 634916403408. [<timestamp>] INFO .. <jobID> <DBName>: Completed archived log backups; total output bytes = 3060793344 (2.85GB) [<timestamp>] INFO .. <jobID> <DBName>: Backing up current controlfile. [<timestamp>] ERROR .. <jobID> [<OracleHostIP>] <DBName>: RMAN-03009: failure of backup command on spp1 channel at <timestamp> [<timestamp>] ERROR .. <jobID> [<OracleHostIP>] <DBName>: ORA-00245: control file backup failed; in Oracle RAC, target might not be on shared storage [<timestamp>] ERROR .. <jobID> Database <DBName>: Backup failed: Failed to backup current controlfile [<timestamp>] INFO .. <jobID> Completed backup operation in 269m 54s. 0 database(s) succeeded and 1 failed. in the command.log for the control file backup : [<timestamp>] JOBLOG pid:<PID> Thread-<xx> worker_backup: <CTGGF0026> <DBName>: Backing up current controlfile. [<timestamp>] INFO pid:<PID> Thread-<xx> writeInputFile: <DBName> set command id to 'SPP_BACKUP_<SLA_ID>_<jobID>'; [<timestamp>] INFO pid:<PID> Thread-<xx> writeInputFile: <DBName> allocate channel spp1 type disk format '/mnt/spp/tmp/<DBName>1_spp_backup_<SLA_ID>.ctl'; [<timestamp>] INFO pid:<PID> Thread-<xx> writeInputFile: <DBName> backup as copy current controlfile reuse tag 'SPP_BACKUP_<SLA_ID>'; [<timestamp>] ERROR pid:<PID> Thread-<xx> processOutput: <DBName>: RMAN-03009: failure of backup command on spp1 channel at <timestamp> [<timestamp>] ERROR pid:<PID> Thread-<xx> processOutput: <DBName>: ORA-00245: control file backup failed; in Oracle RAC, target might not be on shared storage [<timestamp>] INFO pid:<PID> Thread-<xx> backupControlfile: Traceback (most recent call last): File "/opt/IBM/SPP/agents_bckadmin/oracleagent/runner.py", line 331, in backupControlfile Exception: Unexpected messages in command output [<timestamp>] INFO pid:<PID> Thread-<xx> worker_backup: Traceback (most recent call last): File "/opt/IBM/SPP/agents_bckadmin/oracleagent/runner.py", line 331, in backupControlfile Exception: Unexpected messages in command output During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/opt/IBM/SPP/agents_bckadmin/oracleagent/backup.py", line 389, in worker_backup File "/opt/IBM/SPP/agents_bckadmin/oracleagent/runner.py", line 338, in backupControlfile oracleagent.runner.RmanRunnerError: Failed to backup current controlfile both virgo.log and command.log files will be found in the job log bundle collected from the Spectrum Protect GUI. IBM Spectrum Protect Versions Affected: IBM Spectrum Protect Plus 10.1.x Initial Impact: High Additional Keywords: SPP, SPPLUS, TS003084275
Local fix
Problem summary
**************************************************************** * USERS AFFECTED: * * IBM Spectrum Protect Plus level 10.1.5. * **************************************************************** * PROBLEM DESCRIPTION: * * See Error Description * **************************************************************** * RECOMMENDATION: * * Apply fixing level when available. This problem is currently * * projected to be fixed in IBM Spectrum Protect Plus level * * 10.1.5 patch1 and 10.1.6. Note that this is subject to * * change at the discretion of IBM. * ****************************************************************
Problem conclusion
In an Oracle RAC clustered environment, SPP mounts the backup volume from vSnap on one of the RAC nodes in order to perform the backup. When creating a backup copy of the controlfile, if another node is modifying the controlfile at the same time, RMAN throws error ORA-00245 indicating that the target of the copy operation is not on shared storage. The issue has been resolved by improving the way SPP performs backup of the controlfile in a RAC environment. Instead of creating the copy on a non-shared vSnap location, SPP first creates a temporary copy in a shared location (the same ASM diskgroup where the original controlfile resides) and then copies this temporary copy from the shared ASM location to the non-shared vSnap location.
Temporary fix
Comments
APAR Information
APAR number
IT31886
Reported component name
SP PLUS
Reported component ID
5737SPLUS
Reported release
A15
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
2020-02-18
Closed date
2020-02-20
Last modified date
2020-02-20
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
SP PLUS
Fixed component ID
5737SPLUS
Applicable component levels
[{"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SSNQFQ","label":"IBM Spectrum Protect Plus"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"A15","Line of Business":{"code":"LOB26","label":"Storage"}}]
Document Information
Modified date:
31 January 2024