IJ46155: SMALL FILE CREATE PERFORMANCE IMPROVEMENT

APAR status

• Closed as program error.

Error description

• Error Description:
  
  Performance is improved when create a lot of small files,
  especially if you see lots of waiters as below:
  
  Waiting 0.0042 sec since 2023-03-05_20:16:51, monitored,
  thread 103072 CloseHandlerThread: on ThMutex
  0x7F1D9420A1D8 (SGAllocMapDeallocMutex)
  Waiting 0.0037 sec since 2023-03-05_20:16:51, monitored,
  thread 104305 CloseHandlerThread: on ThCond
  0x7F1D9420A200 (DeallocQueueCondvar),
  reason 'waiting for dealloc queue flush'
  Waiting 0.0037 sec since 2023-03-05_20:16:51, monitored,
  thread 105412 CloseHandlerThread: on ThCond
  0x7F1D9420A200 (DeallocQueueCondvar),
  reason 'waiting for dealloc queue flush'
  Waiting 0.0036 sec since 2023-03-05_20:16:51, monitored,
  thread 105062 CloseHandlerThread: on ThCond
  0x7F1D9420A200 (DeallocQueueCondvar),
  reason 'waiting for dealloc queue flush'
  Waiting 0.0036 sec since 2023-03-05_20:16:51, monitored,
  thread 104238 CloseHandlerThread: on ThCond
  0x7F1D9420A200 (DeallocQueueCondvar),
  reason 'waiting for dealloc queue flush'
  Waiting 0.0036 sec since 2023-03-05_20:16:51, monitored,
  thread 105299 CloseHandlerThread: on ThCond
  0x7F1D9420A200 (DeallocQueueCondvar),
  reason 'waiting for dealloc queue flush'
  Waiting 0.0036 sec since 2023-03-05_20:16:51, monitored,
  thread 104188 CloseHandlerThread: on ThCond
  0x7F1D9420A200 (DeallocQueueCondvar),
  reason 'waiting for dealloc queue flush'
  Waiting 0.0023 sec since 2023-03-05_20:16:51, monitored,
  thread 103940 CloseHandlerThread: on ThCond
  0x7F19A8007788 (LkObjCondvar),
  reason 'waiting for LX lock'
  Waiting 0.0023 sec since 2023-03-05_20:16:51, monitored,
  thread 103225 CloseHandlerThread: on ThCond
  0x7F19A8007788 (LkObjCondvar),
  reason 'waiting for LX lock'
  Waiting 0.0023 sec since 2023-03-05_20:16:51, monitored,
  thread 103385 CloseHandlerThread: on ThCond
  0x7F19A8007788 (LkObjCondvar),
  reason 'waiting for LX lock'
  Waiting 0.0023 sec since 2023-03-05_20:16:51, monitored,
  thread 104110 CloseHandlerThread: on ThCond
  0x7F19A8007788 (LkObjCondvar),
  reason 'waiting for LX lock'
  ..........................
  
  Reported in:
  Spectrum Scale 5.1.4.1 on RHEL8
  
  Known Impact:
  Performance downgraded
  
  Verification steps:
  mmdiag --waiters
  
  Recovery action:
  N/A
  

Local fix

• N/A
  

Problem summary

• A bunch of files creations are finally closing the files but
  pending on the thread CloseHandlerThread with long waiter
  "waiting for dealloc queue flush", then the slow down the
  performance of files creations.
  

Problem conclusion

• This problem is fixed in 5.1.2.14
  To see all Spectrum Scale APARs and their respective
  Fix solutions refer to page:
  https://public.dhe.ibm.com/storage/spectrumscale/spectrum_scale_
  apars.html
  
  Benefits of the solution:
  Improve the performance of deallocation queue flush, so that
  won't block the CloseHandlerThread too much time on the "waiting
  for dealloc queue flush".
  
  Work around:
  None
  
  Problem trigger:
  Lots of file creations and closes while there are many other
  process doing space deallocations.
  
  Symptom:
  The small files creations are pending on closes, then the
  performance of files creations is slowing down.
  
  Platforms affected:
  All Operating Systems
  
  Functional Area affected:
  All Scale Users
  
  Customer Impact:
  Suggested
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  SPEC SCALE ADV

• Fixed component ID

  5737F35AP

Applicable component levels