IBM Support

IC66801: CRASH IN AFMIGRVOLUMETHREAD DURING MIGRATION

Subscribe

You can track all active APARs for this component.

 

APAR status

  • Closed as program error.

Error description

  • There can be a case where Tivoli Storage Manager server can
crash during client backups to a storage pool, when migration is
triggered, but there is nothing currently migratable. There is a
timing window where it may access memory that had been freed by
a sub-thread. This problem happens when the server is very busy
and the sub-thread found nothing to do and exits very quickly.

Core stacks from migration:

(dbx) where
AfRclmOnsiteVols(??) at 0x100559ba4
AfMigrVolumeThread(??) at 0x10055f844
StartThread(??) at 0x1000151ac

or

(dbx) where
StdPutMsg(??, ??, ??, ??, ??, ??, ??, ??) at 0x10000e810
outRptf(0x1cc000000000, 0x3e9000003e9, 0x27dbe, 0x162736628,
  0x0, 0x0, 0x1016a0bb, 0x0) at 0x100013cd0
EndCurMigrVolumeProc(??, ??) at 0x10055bb30
AfMigrVolumeThread(??) at 0x10055f534
StartThread(??) at 0x1000151a






























L2 Diagnostics:
From an afmove bfcreate server trace for the migration crash:

11:06:32.027
[69][bfcputil.c][4107][bfBeginStgPoolMigration]:bfCopyControl
allocated at 627168d8
        ====> Structure with address 627168d8 allocated by
thread 69 (in bfBeginStgPoolMigration).

11:06:32.027 [69][afmigr.c][696][AfBeginStgPoolMigration]:Begin
automatic migration for pool POOL1(9).
11:06:32.027 [69][afmigr.c][709][AfBeginStgPoolMigration]:rc=0
from CheckStgPoolMigration for pool POOL1.
11:06:32.027 [69][afmigr.c][2121][ObtainVolumes]:rc=0 obtainning
reclaimable volumes for pool POOL1(9). rclmType=0, volListP=0.
11:06:32.028 [69][afmigr.c][2159][ObtainVolumes]:rc=0 obtainning
migratable volumes for pool POOL1(9). volListP=6271f098.
11:06:32.028 [69][output.c][6385][PutConsoleMsg]:ANR0984I
Process 395619 for MIGRATION started in the BACKGROUND at
11:06:32.~
11:06:32.029 [69][output.c][6385][PutConsoleMsg]:ANR1000I
Migration process 395619 started for storage pool POOL1
automatically, highMig=0, lowMig=0, duration=None.~
11:06:32.029
[450029][afmigr.c][871][AfMigrVolumeThread]:AfMigrVolumeThread
(procNum=395619) started for pool POOL1(9).
         ====> Sub-thyread 450029 started here.
11:06:32.030 [450029][afmigr.c][1035][AfMigrVolumeThread]:Proc
395619 starting migration for pool POOL1(9).
11:06:32.030 [450029][afmigr.c][2958][LeaseMigrVol]:Proc 395619
leased migratable Volume AB1234.
11:06:32.030 [450029][output.c][6385][PutConsoleMsg]:ANR1100I
Migration started for volume AB1234, storage pool POOL1 (process
number 395619).~
11:06:32.030 [450029][output.c][6385][PutConsoleMsg]:ANR1102I
Removable volume AB1234 is required for migration.~
11:06:32.031 [450029][afmove.c][2017][AfMoveVolume]:Target
poolid 4 collocation is 1, source poolid 0 is not an active data
pool.
11:06:32.031 [450029][output.c][6385][PutConsoleMsg]:ANR1101I
Migration ended for volume AB1234.~
11:06:32.031 [450029][afmigr.c][2679][SelectMigrVolFromList]:No
more volume to be migrated for pool POOL1(9).
11:06:32.032 [450029][afmigr.c][1041][AfMigrVolumeThread]:Proc
395619 ended migration for pool POOL1(9). rc=0.
11:06:32.032 [450029][output.c][6385][PutConsoleMsg]:ANR1001I
Migration process 395619 ended for storage pool POOL1.~
11:06:32.032 [450029][output.c][6385][PutConsoleMsg]:ANR0985I
Process 395619 for MIGRATION running in the BACKGROUND completed
with completion state SUCCESS at 11:06:32.~
11:06:32.036 [450029][output.c][6385][PutConsoleMsg]:ANR4935I
Migration of primary storage pool POOL1 has ended. Files
migrated: 0, Bytes migrated: 0, Unreadable Files: 0.~
11:06:32.037
[450029][afmigr.c][3498][EndMigrVolumeThread]:Logging migration
statistics for pool POOL1(9).
11:06:32.040 [435854][bfcreate.c][2797][AddAggrBitfile]:Adding a
aggregate bitfile to the list. bf id is 0.406667070 head is
627125b8 tail is 62715c18.
11:06:32.044 [435854][bfcreate.c][2849][AddAggrBitfile]:List
updated head is 627125b8 tail is 62717058.
11:06:32.044 [435854][bfcreate.c][511][bfCreate]:bfCreate:
isNASTocCreate Flag = False
11:06:32.044 [435854][bfcreate.c][601][bfCreate]:bfCreate:
Entry Point.  SW Path flag is False.
11:06:32.044
[435854][bfcreate.c][2976][CreateBitfile]:CreateBitfile:
Creating bitfile 0.406667071 in storage pool POOL2, cluster
key=28.3, aggregate=1, estSize=0.23966072,
estBitfileSize=0.3523045, aggrSize=0.20448121.
11:06:32.045
[435854][bfcreate.c][2989][CreateBitfile]:CreateBitfile:
Aggregate open, appending bitfile 0.406667071.
11:06:32.045
[450029][bfcputil.c][711][BfFreeCopyControl]:bfCopyControl at
627168d8 freed
        ====>  Sub-thread 450029 ended very quickly and freed
627168d8.

11:06:32.046 [69][output.c][6385][PutConsoleMsg]:ANR0984I
Process 395620 for MIGRATION started in the BACKGROUND at
11:06:32.~
11:06:32.046 [69][output.c][6385][PutConsoleMsg]:ANR1000I
Migration process 395620 started for storage pool POOL1
automatically, highMig=0, lowMig=1644936208, duration=None.~
11:06:32.047
[450030][afmigr.c][871][AfMigrVolumeThread]:AfMigrVolumeThread
(procNum=395620) started for pool POOL1(9).
11:06:32.047 [450030][afmigr.c][887][AfMigrVolumeThread]:Proc
395620 starting onsite reclamation for pool POOL1(9
        ====>  Sub-thread 450030 will try to free 627168d8
again, which had been freed.


Platform/Version affected :
Tivoli Storage Manager 5.4, 5.5 and 6.1 on All platforms


Additional keywords: zz61 crash core abend migration relcaim
   reclamation cpyCtrlP afmigr.c afrclm.c

Local fix

  • 



Problem summary


    

  • ****************************************************************
* USERS AFFECTED: All Tivoli Storage Manager server users.     *
****************************************************************
* PROBLEM DESCRIPTION: See error description.                  *
****************************************************************
* RECOMMENDATION: Apply fixing level when available. This      *
*                 problem is currently projected to be fixed   *
*                 in levels 5.5.5, 6.1.5, and 6.2.2. Note      *
*                 that this is subject to change at the        *
*                 discretion of IBM.                           *
****************************************************************
*

    



Problem conclusion


    

  • This problem was fixed.
Affected platforms:  AIX, HP-UX, Sun Solaris, Linux, z/OS, and
Windows.

    



Temporary fix


    

  • 



Comments


    

  • 



APAR Information




    

  • APAR number
    IC66801
    • 

  • Reported component name
    TSM SERVER
    • 

  • Reported component ID
    5698ISMSV
    • 

  • Reported release
    55A
    • 

  • Status
    CLOSED  PER
    • 

  • PE
    NoPE
    • 

  • HIPER
    NoHIPER
    • 

  • Special Attention
    NoSpecatt
    • 

  • Submitted date
    2010-03-02
    • 

  • Closed date
    2010-06-23
    • 

  • Last modified date
    2011-06-22
    • 





    

  • APAR is sysrouted FROM one or more of the following:
    

    • 

  • APAR is sysrouted TO one or more of the following:
    

    • 



Fix information


    

  • Fixed component name
    TSM SERVER
    • 

  • Fixed component ID
    5698ISMSV
    • 



Applicable component levels


    

  • R55A  PSY
       UP
    • 

  • R55H  PSY
       UP
    • 

  • R55L  PSY
       UP
    • 

  • R55S  PSY
       UP
    • 

  • R55W  PSY
       UP
    • 

  • R61A  PSY
       UP
    • 

  • R61H  PSY
       UP
    • 

  • R61L  PSY
       UP
    • 

  • R61S  PSY
       UP
    • 

  • R61W  PSY
       UP
    • 

  • R62A  PSY
       UP
    • 

  • R62H  PSY
       UP
    • 

  • R62L  PSY
       UP
    • 

  • R62S  PSY
       UP
    • 

  • R62W  PSY
       UP
    • 








      
              
                            

              

              [{"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SSGSG7","label":"Tivoli Storage Manager"},"Component":"","ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"55A","Edition":"","Line of Business":{"code":"LOB26","label":"Storage"}}]
              

                                                                    

          
 
        

      

    

      

        

          

            
Document Information


            

            


                        

            Modified date:
            

            22 June 2011
            

            
                      

        


        

        


      

    

  



    

  

  
    
            

          
Page Feedback