IV63654: JVM fails to start up with non-persistent shared class cache after system reboot or IPL

APAR status

• Closed as program error.

Error description

• Error Message:
  Following error messages are possible:
  
  When running without -Xshareclasses:nonFatal option:
  JVMSHRC020E An error has occurred while opening semaphore
  JVMSHRC336E Port layer error code = -197210
  JVMSHRC337E Platform error message: semget : No such file or
  directory
  JVMSHRC619E Semaphore control file is read only.
  
  When running with -Xshareclasses:nonFatal option:
  JVMSHRC022E Error creating shared memory region
  JVMSHRC336E Port layer error code = -393818
  JVMSHRC337E Platform error message: shmget : No such file or
  directory
  JVMSHRC627I Recreation of shared memory control file is not
  allowed when running in read-only mode.
  
  Stack Trace: N/A
  .
  
  LOCAL FIX:
  

Local fix

• After an IPL or reboot, manually remove the JVM semaphore and
  shared memory control files associated with the non-persistent
  shared class cache.
  

Problem summary

• This problem occurs when the IPC semaphore set and/or shared
  memory associated with a non-persistent shared class cache are
  destroyed, but their JVM control files are left behind. This can
  happen in the following cases:1) if the user manually removes
  the IPC semaphore set and/or shared memory using "ipcrm"
  command, or2) on system reboot (or IPL in case of z/OS), at
  which point the operating system cleans up all IPC semaphore
  sets and shared memory regions, or3) the JVM is started with
  -Xshareclasses:destroy and the JVM control file has read-only
  permission for the user. In such case the JVM may have
  successfully removed the semaphore set and/or shared memory but
  the control file will not be removed.After the IPC semaphore set
  and/or shared memory are removed, if a user other than creator
  tries to access the cache, the JVM fails to startup. However if
  creator of the shared cache attempts to startup, then JVM
  removes the stale control files and recreates the cache, if
  required.This difference in behavior between creator of the
  cache and other users is because the JVM creates control files
  with read-only permission for 'group' and 'others' and it is
  programmed not to remove the read-only control files to avoid a
  race condition while recreating the control files. Therefore, in
  such scenarios when the JVM is started by a user other than the
  creator of the cache, in particular which may occur due to
  timing when starting up JVMs after a reboot or IPL, the JVM does
  not remove the stale read-only control files and fails with an
  error message as mentioned in the Error description. If such a
  user attempts to start the JVM using the -Xshareclasses:nonfatal
  option, then the JVM throws an error message, as mentioned in
  the Error description, and continues without using a shared
  class cache.
  

Problem conclusion

• The JVM has been updated with following changes:1) The JVM has
  been updated to create control files with read-write access for
  users in the same group as the creator of the cache when the
  cache is created with -Xshareclasses:groupAccess option. This
  allows a JVM launched by such users to recover from certain
  inconsistent situations, such as when the IPC semaphore set or
  shared memory does not exist or wherein the properties of the
  semaphore set or shared memory do not match with that in the
  control file, by deleting the control files and re-creating
  them.Since the control files are still read-only for users not
  in same group as the creator of the cache, such users will
  continue to get error message as mentioned in the Error
  description if they attempt to use the cache. In such case the
  JVM control files need to be manually removed from the system
  for the JVM to continue.2) In certain rare scenarios if the JVM,
  during startup, finds control files belonging to a previous
  version of the JVM (i.e. which does not have this fix) which
  would have created them as read-only for group users, then it
  will remove the control files irrespective of the permission and
  start up normally by creating a new shared class cache. Note
  that if existing control files created by a previous version of
  the JVM are read-only for the current JVM process, then multiple
  JVMs operating on the control files may get into a race
  condition which may cause one or more JVMs to fail to start up
  the shared class cache, or it may cause the JVMs to create their
  own separate shared memory regions while using same semaphore
  set, possibly impacting each other's performance while writing
  to the shared class cache. Once the control files created by
  older JVMs are removed, the race condition will no longer be
  possible. Therefore it is advised to manually remove the control
  files after system reboot (or IPL in case of z/OS).Note that
  when the JVM attempts to delete the control files, restrictions
  imposed by the Operating System apply as usual.Specifically, if
  the user does not have write permission for the cache directory,
  the Operating System won't allow the JVM to delete the control
  files, irrespective of their permissions. Moreover, if the
  sticky bit is set on the cache directory, then only the owner of
  the control files can delete them. These scenarios can happen if
  the creator of the cache has specified
  -Xshareclasses:cacheDirPerm=<mode> option when creating the
  cache, where <mode>specifies the permission on the cache
  directory, or the cache directory permission was modified
  manually using "chmod" command. In such cases, the JVM will fail
  to startup unless -Xshareclasses:nonfatal option is specified,
  in which case it will startup without using a shared class
  cache.
  .
  This APAR will be fixed in the following Java Releases:
     7    SR8       (7.0.8.0)
     6 R1 SR8 FP2   (6.1.8.2)
     6    SR16 FP2  (6.0.16.2)
     7 R1 SR2       (7.1.2.0)
  .
  Contact your IBM Product's Service Team for these Service
  Refreshes and Fix Packs.
  For those running stand-alone, Java maintenance is available
  from:
             https://www.ibm.com/developerworks/java/jdk/
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  J9 COMMON CODE

• Fixed component ID

  620700127

Applicable component levels

• R260 PSY

     UP

• R600 PSY

     UP

• R270 PSY

     UP