IT48681: IBM STORAGE PROTECT SERVER TARGET SERVER CRASHES DURING STGRULE FOR REPLICATION, STACK POINTS TO MALLOC_Y

8.2.1.000-IBM-SPSRV-WindowsX64
8.2.1.000-IBM-SPSRV-Linuxx86_64
8.2.1.000-IBM-SPSRV-Linuxs390x
8.2.1.000-IBM-SPSRV-Linuxppc64le
8.2.1.000-IBM-SPSRV-AIX

APAR status

• Closed as program error.

Error description

• IBM Storage Protect Server target server crashes during stgrule
  for replication
  
  
  Stack shows below information, pointing to malloc_and is
  related to GSK issue :
  
  Segmentation fault in malloc_y at 0x90000000004c7fc ($t95221)
  0x90000000004c7fc (malloc_y+0x125c) f8040000 std r0,0x0(r4)
  (dbx)(dbx) where
  malloc_y(0x16df581c0, 0x0, 0x9001000a007fb48, 0x0, 0x0,
  0x134b02b7, 0x134b02b7, 0x20) at 0x90000000004c7fc
  malloc_common@AF104_87(??) at 0x90000000002ef28
  newop.init_pres(unsigned long)(??) at 0x900000000b29fa8
  _ct_19GSKBufferAttributesFv(??) at 0x90000000899b7bc
  _ct_9GSKBufferFUlPCc(??, ??, ??) at 0x90000000899bed0
  Send__14TLSV13ProtocolFPCvi(??, ??, ??) at 0x900000007cb5d4c
  Send__18SSLProtocolManagerFPCvi(??, ??, ??) at 0x900000007cba968
  
  gsk_secure_soc_write(??, ??, ??, ??) at 0x900000007edc770
  tlsSend(??, ??, ??) at 0x1010ff3f0
  smFlushBufferData(??) at 0x1005991f8
  smRecvBufferData(??, ??, ??) at 0x1005b0ec8
  smRecvBufData@AF60_56(??, ??, ??, ??, ??, ??) at 0x1005b19f4
  read__15tls13ReadRecordFv(??) at 0x9000000080efe60
  read__20tls13RecordProcessorFv(??, ??) at 0x9000000080e99bc
  receive__22tls13ProtocolProcessorFR13GSKFastBuffer(??, ??) at
  0x900000008194d34
  Receive__14TLSV13ProtocolFPvib(??, ??, ??, ??) at
  0x900000007cb6218
  Receive__14TLSV13ProtocolFPvi(??, ??, ??) at 0x900000007cb6d10
  Receive__18SSLProtocolManagerFPvi(??, ??, ??) at
  0x900000007cba6c8
  gsk_secure_soc_read(??, ??, ??, ??) at 0x900000007ed9d64
  tlsRecv(??, ??, ??) at 0x1010ff908
  IPRA.$ReceiveVerb(??, ??) at 0x1005962b0
  SmRecvVerbX(??) at 0x10058d1f8
  SmReplServerSession(??) at 0x100c1ed38
  IPRA.$DoReplServer(??, ??) at 0x1009845ec
  smExecuteSession(??, ??, ??, ??, ??, ??, ??, ??) at 0x100973480
  psSessionThread(??) at 0x10110532c
  StartThread(0x0) at 0x100011a6c
  
  (dbx) registers
  $r0:0x000000013ad72fa0 $stkp:0x000000016df58040
  $toc:0x09001000a00912f0
  $r3:0x0000000000000020 $r4:0x0000000000000050
  $r5:0x0000000000000000
  $r6:0x0000000000000060 $r7:0x0000000000000060
  $r8:0x0000000000000060
  $r9:0x0000000000000000 $r10:0x0000000000000020
  $r11:0x0000000000000000
  $r12:0x09001000a00b1e58 $r13:0x000000016df68800
  $r14:0x0000000000000008
  $r15:0x0000000000000009 $r16:0x000000000000002a
  $r17:0x00000001005b0c80
  $r18:0x0900000007a659c0 $r19:0x000000000000000d
  $r20:0x0000000111274050
  $r21:0x000000013b0732d0 $r22:0x000000013b377000
  $r23:0x09001000a00b1e68
  $r24:0x0000000000000000 $r25:0x0000000000000033
  $r26:0x09001000a00b0a30
  $r27:0x0000000000010ea0 $r28:0x0000000000000000
  $r29:0x000000000000000a
  $r30:0x0000000000000060 $r31:0x0000000000000040
  $iar:0x090000000004c7fc $msr:0xa00000000200d032 $cr:0x44204222
  $link:0x090000000004c848 $ctr:0x0900000000591600 $xer:0x20040000
  
  
  Condition status = 0:g 1:g 2:e 4:g 5:e 6:e 7:e
  [unset $noflregs to view floating point registers]
  [unset $novregs to view vector registers]
  [unset $novsregs to view vector scalar registers]
  in malloc_y at 0x90000000004c7fc ($t95221)
  0x90000000004c7fc (malloc_y+0x125c) f8040000 std r0,0x0(r4)
  
  IBM Storage Protect Server version Affected: IBM Storage Protect
  Server 8.1.25.x and later on AIX platforms.
  

Local fix

• Make these changes at the bottom of the rc.dsmserv file by
  adding new line for MALLOCBUCKETS (Replace existing block with
  the following mentioned block) :
  
  if [ "$OS" = "AIX" ]; then
  If MALLOCOPTIONS isn't set by invoker, set it now
  if [ "$MALLOCOPTIONS" = "" ]; then
  MALLOCBUCKETS=number_of_buckets:128,bucket_sizing_factor:64,blo
  cks_per_bucket:1024
  MALLOCOPTIONS=multiheap:16
  fi
  LDR_CNTRL is 1 environment variable. This isn't 5 environment
  variables
  envVars="LDR_CNTRL=TEXTPSIZE=64K@DATAPSIZE=64K@STACKPSIZE=64K@S
  HMPSIZE=64K"
  envVars="${envVars} MALLOCBUCKETS=$MALLOCBUCKETS"
  envVars="${envVars} MALLOCOPTIONS=$MALLOCOPTIONS"
  if [ "$instanceUser" != "" ]; then
  exec su - $instanceUser $bashFlag "-c $envVars
  $serverBinDir/dsmserv $cmdParms"
  else
  exec /usr/bin/env $envVars $serverBinDir/dsmserv $cmdParms
  fi
  else
  if [ "$instanceUser" != "" ]; then
  exec su - $instanceUser -c "nohup $serverBinDir/dsmserv
  $cmdParms"
  else
  exec nohup $serverBinDir/dsmserv $cmdParms
  fi
  fi
  
  You will then have to halt and restart the SP server to pick up
  these changes.
  Please refer https://www.ibm.com/docs/en/aix/7.2.0?topic=subsys
  tem-malloc-multiheap
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * All IBM Storage Protect server users                         *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See error description.                                       *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply fixing level when available. This problem is currently *
  * projected to be fixed in levels 8.2.1. Note that this is     *
  * subject to change at the discretion of IBM.                  *
  ****************************************************************
  

Problem conclusion

• This problem was fixed.
  Platforms fixed:  AIX
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  TSM SERVER

• Fixed component ID

  5698ISMSV

Applicable component levels