Informix Imported Restore failing when using CommVault Storage Manager and character devices

Problem

You attempt to complete an Imported Restore using the CommVault Storage Manager after configuring character devices on the target machine. After executing the onbar command, you see something similar to the following in the bar_act.log: 2015-09-01 13:57:07 8978504 7536712 onbar_d -r 2015-09-01 13:57:08 8978504 7536712 Working with cvsm as generic storage manager. 2015-09-01 13:57:10 8978504 7536712 Successfully connected to Storage Manager. 2015-09-01 13:57:20 8978504 7536712 Begin reserved pages restore (level 0 of rootdbs, Storage Manager copy ID: 10092784 1441125575). 2015-09-01 13:57:23 8978504 7536712 Completed reserved pages restore (level 0 of rootdbs). 2015-09-01 13:57:25 8978504 7536712 Successfully connected to Storage Manager. 2015-09-01 13:57:36 8978504 7536712 Begin cold level 0 restore rootdbs (Storage Manager copy ID: 10092784 1441125575). 2015-09-01 14:02:39 8978504 7536712 Unable to write storage space restore data to the database server: . 2015-09-01 14:02:39 8978504 7536712 (-43391) Skipped backup/restore of space 'rootdbs'. 2015-09-01 14:02:40 8978504 7536712 (-43140) Due to the previous error, logical restore will not be attempted. 2015-09-01 14:02:40 8978504 7536712 onbar_d complete, returning 131 (0x83) This is not very helpful in finding the cause of the problem and the bar_debug.log is equally non-descriptive in addressing the cause of the problem. Looking at the CommVault Storage Manager log, you see the following: ..... 8978504 1 09/01 14:02:39 62604 IfxBackupRestoreBase::CloseArchive() - Job[62604] thread[8978504]: Got bad eof buffer type:2 8978504 1 09/01 14:02:40 62604 CCVAPipelayer::discardTransitBuffers()() - Read All has been aborted 8978504 1 09/01 14:02:40 62604 IfxBackupRestoreBase::CloseArchive() - Job[62604] thread[8978504]: m_archive->AbortReadAll succeed 8978504 1 09/01 14:02:40 62604 IfxBackupRestoreBase::SendMsgToPH() - Job[62604] thread[8978504]: To Pipe Head msgType=FSR_MSG_CLOSE_ARCHIVE_AND_RELEASE_VOL. 8978504 1 09/01 14:02:40 62604 IfxBackupRestoreBase::CloseArchive() - CVArchive::CloseArchiveFile() succeed 8978504 1 09/01 14:02:40 62604 CVArchive::ReadBuffer() - PL_FS_CLOSE_AFILE(2, 9, 149729) 8978504 1 09/01 14:02:40 62604 IfxBackupRestoreBase::CloseArchive() - Job[62604] thread[8978504]: Received PL_FS_CLOSE_AFILE buffer. 8978504 1 09/01 14:02:40 62604 BSAEndData: Data transfer Complete, total = 122880 bytes 8978504 1 09/01 14:02:40 62604 BSAEndData: Archive File Name /onlineipc_prod/rootdbs/0.1441125575.10092784, Size = 122880 bytes, Informix I/O Time = 300.078736 secs, MA I/O Time = 2.677134 secs, Total I/O Time = 302.755870 secs Informix Throughput = 0.001373 GB/Hr, MA I/O Throughput = 0.153891 GB/Hr, Total I/O Throughput = 0.001361 GB/Hr 8978504 1 09/01 14:02:40 62604 BSAEndData: Exit,status=0 8978504 1 09/01 14:02:40 62604 BSAEndTxn: Enter, vote:2 8978504 1 09/01 14:02:40 62604 BSAEndTxn: Error: Transfer Incomplete 8978504 1 09/01 14:02:40 62604 BSAEndTxn: XBSA PS: Informix I/O Throughput = 0.001373 GB/Hr, Media Agent I/O Throughput =0.153891 GB/Hr, Total I/O Throughput = 0.001361 GB/Hr 8978504 1 09/01 14:02:40 62604 BSAEndTxn: Exit, status:3 8978504 1 09/01 14:02:40 62604 BSATerminate: Enter - PID:7536712, TID:8978504, xbsaParams.pid:7536712 8978504 1 09/01 14:02:40 62604 IfxBackupRestoreBase::ExitHere() - Job[62604] thread[8978504]: Enter - phase=[27]. 8978504 1 09/01 14:02:40 62604 IfxBackupRestoreBase::ExitHere() - Job[62604] thread[8978504]: Going to delete Archive Object: ... 8978504 1 09/01 14:02:40 62604 CVArchive::ClosePipeline() - Closing Pipeline 8978504 1 09/01 14:02:40 62604 CVArchive::ClosePipeline() - Shutting down head session 8978504 1 09/01 14:02:40 62604 CCVAPipelayer::discardTransitBuffers()() - DataMover has been destroyed 8978504 1 09/01 14:02:40 62604 CCVAPipelayer::ClosePipeline() - About to shut down RESTORE pipeline 8978504 1 09/01 14:02:40 62604 CPipelayer::ShutdownPipeline() - stat- Pipeline [21693a78] [duration - 310 seconds] 8978504 1 09/01 14:02:40 62604 CPipelayer::ShutdownPipeline() - Sending STOP_PL when Shutting down pipeline 8978504 1 09/01 14:02:40 62604 [PIPELAYER ] sendStopPlMessage(): Sending STOP_PL 8978504 1 09/01 14:02:40 62604 [PIPELAYER ] sendStopPlMessage(): plId='1441133850_8978504_1' [Shutting down pipeline] 8978504 1 09/01 14:02:40 62604 CPipelayer::ShutdownPipeline() - Waiting for STOP_PL message 8978504 1 09/01 14:02:40 62604 [PIPELAYER ] Pipeline already received STOP_PL 8978504 1 09/01 14:02:40 62604 [PIPELAYER ] Returning from waiting for pipeline shutdown because the pipeline is no longer running 8978504 1 09/01 14:02:40 62604 CPipelayer::ShutdownPipeline() - pipeline [21693a78] Shutdown completed. 8978504 1 09/01 14:02:40 62604 [PIPELAYER ] Detaching shared memory 8978504 1 09/01 14:02:40 62604 IfxBackupRestoreBase::ExitHere() - Job[62604] thread[8978504]: deleted m_archive 8978504 1 09/01 14:02:40 62604 IfxBackupRestoreBase::ExitHere() - Job[62604] thread[8978504]: m_totalFilesTrans=0 m_totalFilesTransferred=1 8978504 1 09/01 14:02:40 62604 IfxBackupRestoreBase::ExitHere() - Job[62604] thread[8978504]: RemoveThread() ... 8978504 1 09/01 14:02:40 62604 IfxXbsaJobUnit::RemoveThread() - for threadId=8978504 channel=1 m_threadNumber=0 8978504 1 09/01 14:02:40 62604 IfxBackupRestoreBase::ExitHere() - Job[62604] thread[8978504]: Remove Job Unit ... 8978504 1 09/01 14:02:40 62604 IfxXbsaJobUnit::~IfxXbsaJobUnit() - [62604]: deleted m_jobControl 8978504 1 09/01 14:02:40 62604 IfxXbsaJobUnit::~IfxXbsaJobUnit() - [62604] : deleted m_ifxThread 8978504 1 09/01 14:02:40 62604 IfxBackupRestoreBase::ExitHere() - Job[62604] thread[8978504]: RemoveJobUnit succeed 8978504 1 09/01 14:02:40 62604 IfxBackupRestoreBase::Finalize() - Job[62604] thread[8978504]: Exit. 8978504 1 09/01 14:02:40 62604 IfxAgentBase::Finalize() - delete m_hostName 8978504 1 09/01 14:02:40 62604 ~IfxJobObject() - call EnableAppMgrClientMessageCache(false) 8978504 1 09/01 14:02:40 62604 IfxAgentBase::Finalize() - deleted m_jobObject 8978504 1 09/01 14:02:40 62604 BSATerminate: Exit, status=0, exstatus=3