Readme and Release notes for release 3.3.0.5 (GPFS) IBM General Parallel File System 3.3.0.5 GPFS-3.3.0.5-x86-Linux Readme

Unless specifically noted otherwise, this history of problems fixed for GPFS 3.3.x applies for all supported platforms.

Problems fixed in GPFS 3.3.0.5 [April 1, 2010]

• Fix problem where metadata-update intensive workloads (e.g., file creates and deletes) running on systems with large inode cache (large maxFilesToCache) would periodically pause for several seconds.
• Fix spurious EIO errors accessing hidden .snapshots directories enabled via "mmsnapdir -a".
• Allow case insensitive node identifier in specfile.
• Fix fsck code so that it doesnt report the corrupt addresses problem that it claimed to have fixed during the previous fsck run.
• Improve linux trace performance.
• Fix mmtrace to return non-zero value and report error when lxtrace binary for current kernel is not installed.
• Fix performance problems when reading large files from NFS clients.
• Correct problems with mmaddcallback -N clustermanager & mmlscallback.
• Fix missing out-of-memory check in get inode routine.
• Improve performance of large file create when DIO is used.
• Fix buffer calculation in dm_get_events when buffer size is greater than 64K.
• Fix potential loss of events in dm_get_event() call when buffer size is greater than 64K.
• Notify dm_get_events that the session already failed after quorum lost in the cluster.
• Fixed potential assert when writing small files via NFS under heavy load.
• Fixed hardlink assertion problem when upgrading file system from 2.3 to 3.3 or later.
• Remove stat() calls in the mmshutdown path.
• Fixed an allocation loop which could occur during mount and rebalance of a filesystem.
• Fix mmbackup to divide up the list of files that need backup based on filesize when numberOfProcessesPerClient=2 (or more).
• A progress indicator is added in the case of mmchfs -F if it leads to the expansion of preallocated inodes.
• Fix a race condition between deldisk and deallocation of surplus indirect blocks that could result in dangling block pointers.
• Fix disklease inconsistencies between cluster manager resetting lastLeaseProcessed and the client resetting lastLeaseReplyReceived.
• When token_revoke results in a downgrade for a device file, call invalidate so that device-specific cleanup occurs.
• Fix allocation code which can cause "No space left on device" error on initial mount after filesystem creation.
• Fixed a file sharing check that was causing an incorrect "access denied" error.
• Restart mmsdrserv after installing new code on Windows.
• Remove redundant preMount and Mount user callback events.
• Stop GPFS trace automatically when doing upgrade to GPFS 3.3 on Linux. Added more detailed error messages when GPFS kernel extentions can not be unloaded.
• Fix a quota problem that fails to translate invalid fileset ids.
• Fix initial run of mmbackup recall in UTC+ timezone to avoid recall of unchanged files that are already on the TSM server.
• Fix assert s_magic == GPFS_SUPER_MAGIC on kernel 2.6.16.60-0.59.1 and above.
• Fixed allocation code which caused an assert during daemon shutdown.
• Load policy file on sgmgr when file system is mounted so that low space threshold always set when file system is mounted.
• Return EMEDIUMTYPE rather than ELNRNG for incompatible format errors on Linux.
• Cleanup flag beingRestriped if inode is deleted while restriping.
• Fix an assert in kxCommonReclock on AIX node.
• Error conditions returned due to failed metadata flush operation are handled appropriately preventing the restripe operation from asserting due to failed checks.
• Fix a problem in quota file creation when file system pool has metadataOnly disks.
• Fix to avoid holding mutex twice while revoking token encounters SGPanic.
• Fixed a rare race condition during Windows GPFS initialization that could cause a system fault.
• Fix code to avoid unreasonable checking for socket.
• Fix exception using a spin_lock in fasync_helper during fcntl revoke.
• Fix server side token issue in failure cases.
• Ensure that online fsck is not held for ever trying to steal buffers from inode range that is currently locked for online fsck.
• Fixed the parallel inode traversal code which can cause signal 11 during restripe and replace disk.
• Fix code to remove unnecessary assertion when a token is revoked while a large file is being restriped.
• Fix problem where in a file system with large snapshots a failure of the file system manager during the first phase of an mmrestripefs or mmdeldisk command could under certain timing conditions cause corruption.
• Fix problem where filesystems created by GPFS release 2.3 or older were not mountable by GPFS release 3.2 or 3.3.
• Fix for a very rare race condition where a non-DIO read from a cached buffer may transiently return partially incorrect data.
• Fixes issues in minorityQuorum clusters that have leaseDuration set and have migrated from 2.3 to 3.2.
• Fix to tolerate an inconsistent state of Windows security settings on an inode following a failed TSM restore.
• This update addresses the following APARs: IZ68715 IZ68725 IZ69476 IZ70073 IZ70074 IZ70396 IZ70409 IZ70599.

Problems fixed in GPFS 3.3.0.4 [January 28, 2010]

• Fix problem where mmrestripe command might not correctly detect I/O errors during the first phase of restripe.
• Correct the resetting of config parameters to default on a subset of the nodes.
• Replace usage of the lsvg command with getlvodm.
• Fix function checkIntRange error message when checking negative numbers.
• Clear the tiebreaker disk parameter after mmexportfs all.
• Fix ioctl opcode conflict with FIGETBSZ on Linux kernel 2.6.31 and later.
• Fix fsck to avoid incorrectly reporting and fixing of filesystem corruptions in a heterogeneous cluster.
• If the file system is internally forced to unmount (file system panic), invoke the preunmount user exit if one is installed.
• Avoid confusion when using a local fcntl lock versus an NLM one.
• Give customers using mmbackup more flexibility by allowing alternate install location for TSM.
• Fix determining filename length when filename contains invalid UTF8 characters.
• Fix data corruption when using mmap.
• Fix assert due to invalid fcntl acquire sleep element found on the kernel queue.
• Keep FS descriptors off of excluded disks even if they come online.
• Fixed a race condition by serializing the xattr object in inode properly.
• Fix hang between node failure thread and events exporter request handler thread.
• Fix mmapplypolicy to estimate correctly the number of GPFS storage pool bytes freed by migrating to an external/HSM pool. Introduce MM_POLICY_MIGRATION_STUBSIZE environment variable to allow users to directly control size for migration.
• Fix mmbackup to avoid giving file name length and file size to TSM for inclusion in backup list.
• Fix async recovery to let mounts succeed while also processing deffered deletions.
• Fix assert failure on FS manager node when unmountOnDiskFailure=yes and a disk fails after 3.2.1.14-16 installed.
• Prevent HSM and NFS from asking to open inodes that are system metadata nodes.
• Do not let socket get stuck in reconn_cleanup state following repeated breaks that occur just after connection handshake completes.
• Reduce the pagepool usage by inode allocation segments during FS manager initialization or recovery.
• Fix a problem with cutting traces in a CNFS setup.
• Fix filesystem panic when a failed disk holds a FS descriptor and returns unexpected error codes.
• Fix problem with mmlsfileset when expanding inodes is running concurrently.
• Ignore un-supported permission flags passed to gpfs_i_permission on SLES11.
• Fix for a SIGSEGV on Windows caused by a race in accessing the ACL file.
• Fix a condition where mm commands can exit with errors if CWD is unavailable.
• Fix for a rare failed assert in the main process thread on Linux.
• Fix a race condition where node may be deleted right after it started up.
• Fix code to correct backward compatibility of non-blocking token request between gpfs 3.2 and gpfs 3.3.
• Make trace recycle timeout message more descriptive and avoid recycle file being overwritten when trace recycles next time (Linux nodes only).
• Succedent tscrfs command will unset some flags unexpectedly even if it cannot get the permission to run. It will cause a daemon assert. Clear flags only if the command has set it before.
• When open of the directory fails and not all fields are set, do not call back into GPFS to do close (release). This may cause an invalid assert due to attempting to reference uninitialized fields.
• Fix signal 11 due to bad RDMA index and cookie received from the TcpConn in verbs::verbsClient_i.
• Fix remote startup on Windows.
• Fix a race condition between an mmexpelnode and mmchmgr.
• Correctly cleanup tmp files on remote nodes.
• Fix a problem in mmdf where number of free inodes may become negative.
• Fix race condition that occurs due to disk failure during clmgr election while using tiebreaker disks.
• Fixed inode expansion code which can cause restripe to fail with an assert. This problem only happens when restripe and inode expansion run concurrently.
• Several sample script and configuration files are now included with the GPFS for Windows installation. These can be found in %SystemRoot%\SUA\usr\lpp\mmfs\samples. Only the files appropriate for use on Windows are included; additional samples are available with UNIX installations.
• Fix assert "offset mappedLen" when reading dirs.
• Fix allocation manager problem that caused pool to not be deleted when it should have been.
• Initialize allocSize variable during the initialization phase of file repair to prevent assert.
• Fix a rare bug that occurs during nsd config change along with earlier disk issues to another deleted nsd.
• Fixed a GPFS on Windows failure that can occur on systems with a large number of cores (e.g. 8 or more) running a workload with thousands of threads. When this error occurs, /var/adm/ras/mmfs.log.* shows "logAssertFailed: tid >= 0 && tid = MAX_GPFS_KERNEL_TID". The fix for this problem removes any assumption on the maximum thread ID.
• Fix a problem that can lead to loss of an intermediate SSL key file.
• Fix mmbackup to accurately reflect the error encountered on the TSM server.
• Fix a problem with interpreting the syncnfs mount option.
• Fix fsck so that it reports duplicate fragments and its count correctly and also prevent a possible fsck crash due to count overflow.
• Added %myNode as callback parameters.
• Fix an assertion during mount that could happen when quota management is enabled and snapshot is being used.
• Fix fsck so that it detects problems and fixes them without encountering struct assert errors even if the 'assertOnStructureError' config option is turned on.
• This update addresses the following APARs: IZ67659 IZ67660 IZ67661 IZ67662 IZ67663 IZ67664 IZ67665 IZ67666 IZ67667 IZ67723 IZ67746 IZ68028.

Problems fixed in GPFS 3.3.0.3 [December 10, 2009]

• On x86_64 Linux when special encoding flag is set in a functions debugging frame section, an extra offset should be added during decoding. Otherwise, the thread traceback can not be decoded correctly.
• Fix Stripe group configuration change so data block loss cannot occur if data is being ingested along with configuration changes.
• On Windows, consolidated separate -msi and -sh installation logs into a single file(gpfs-install.2009.10.20.11.23.58.gpfs-n70-win.log). Also, eliminated Command window popups that were appearing during GPFS installation.
• Fix problem where the new inode scan function gpfs_next_inode64() would return incorrect values for some gpfs_iattr64_t fields on file systems originally created prior to GPFS 2.3.
• Change to GPFS inode scan api for gpfs_ireaddir64. Directory entry structure returned by call now contains a flag field and allows directory entry names to be 1020 bytes (ie 255 characters encoded in UTF-16 or other Unicode encodings).
• Fix policy handling of rules of the form "EXCLUDE FROM POOL" to prevent LOW_SPACE events from incorrectly being logged.
• Extended Attributes (EAs) on Windows have been changed so that internally they are stored with a "user." prefix. This change supports compatibility with Linux and improves file system security.
• Fix problem on systems configured with large maxFilesToCache that could cause file systems to be unmounted on some client nodes when running recovery after a manager node failure.
• Fixed GPFS hang when recalling files from HSM.
• When running lsvg do not wait for the volume group lock.
• Fix mmbackup to give users early warning and exit when unlinked filesets are present during backup. It also prevents further processing of files that would otherwise give the user misleading error information.
• Limit the number of attempts made to destroy nfsd threads in mmnfsquorumloss in case an nfsd thread is stuck waiting for IO to complete in GPFS.
• Don't stop NFS or unexport fs on quorum loss, and kill NFSDs that are stuck during setNfsdProcs.
• Fix mmdelnode syntax error checking.
• Fixed the allocation code which caused a loop during metadata allocation. This problem only affects filesystem with metadata replication enabled.
• Fix mmbackup incremental to handle conversion from short filename records to new longer records after upgrade to 3.2.1.14 or later.
• Fix Linux "mmnfsinit start" command and return correct return code.
• Fix problem where a multi-threaded workload reading extended attributes from a large number of files could cause accumulation of a large number of byte range tokens leading to slowdown and spurious ENOMEM errors.
• Upon mmfsd daemon failure, change the way of debug data collection to asynchronous script execution.
• Fix a quota initialization problem that could allow quota files in storage pools, while they really belong in system pool.
• Fix for a rare race condition that may cause an assert in the invalid fileset object disposal path.
• Added new command line option "--oneerror" to mmaddcallabck command.
• Enable mmapplypolicy on fs with 8MB blocksize.
• Correct processing to prevent quota requests from being performed while the quota manager operations are being quiesced.
• Converted GPFS to use the Windows sockets library (WinSock) rather than the SUA library. This change fixed an issue with large data transfers that appeared when a file system's block sizes was larger than 256KB. The WinSock library also significantly reduces the CPU consumption required to perform network data transfers.
• Fixed error handling when registering pagepool memory to Infiniband.
• Correct quota hard limit processing to check grace time.
• Correct a rare problem (due to an error encountered writing quota files) that can prevent a newly created filesystem from being mounted.
• Improve stability when encountering hostname resolve issues.
• Prevent callback command path in GPFS file system.
• Fixed a problem which prevents filesystem remount after a forced umount due to error(ie. filesystem panic,quorum loss, etc).
• Fix panic handler code to ensure that the right fsck cleanup code path is chosen by looking at the workerNode flag in the fsck data structure.
• Correct processing during restoring filesets to allow more than the KSH array limit of 1024.
• Fix quota manager cleanup when file system manager migrates to another node.
• Fix a file structure error caused by SetAllocationSize.
• Enable kdump to retrieve kernel thread's backtrace on IA64.
• Handle IB port event of LID change.
• Correct a problem when verifying that the daemon is down from a Windows node.
• Resolved an issue with mmwinservctl where the command would fail to set the account name and password. This error would occur if Windows is not installed in same location on all nodes in the cluster (e.g. some nodes have Windows installed on the C: drive and other have it installed on the D: drive.)
• Fix to avoid assertion when calculating the next valid data block number of low level file.
• Removed some commands and programs that were included in the Windows installation, but not supported on Windows.
• Parse the 'ro' mount option and pass it explicitly to gpfsMount to prevent Windows write on readonly filesystem.
• Fix mmbackupconfig in Windows to give the customers the correct mmbackupconfig behavior and exit gracefully.
• Fix mmapplypolicy to run on a large number of files with the -g and -N flags.
• Make filesystem restore messages for mmrestoreconfig more descriptive.
• Fix mmbackup to backup snapshots that are older than the latest filesystem backup.
• Fix mmbackup to ensure that the only desired TSM servers will be processed.
• Package mmbackup32 for use with 3.2 clients.
• Add support for -N nodeList option in mmbackup version 3.1 and 3.2.
• Fix possible deadlock restriping a file system with data replication enabled under application load and with small pagepool.
• Warning messages on conflicting opertaions are sent to stderr to avoid littering to stdout.
• Resolved an issue that in rare cases could cause GPFS to terminate when tracing is enabled.
• Fixed some mmwinservctl operations which were causing GPFS to start inadvertently when GPFS was configured with autoload=yes.
• Fix mmchconfig trace command to load kernel extensions if not already loaded.
• Increase the default maximum size of the shared segment on 64-bit AIX to 1G (32-bit AIX is architecturally limited to 256M).
• This update addresses the following APARs: IZ63333 IZ65179 IZ65379 IZ65416.

Problems fixed in GPFS 3.3.0.2 [November 12, 2009]

• On Windows, mmtracectl no longer requires ActiveState Python in order to collect and format traces.
• During a stripe group mount, ignore the 'exclDisks' mount option for a filesystem in a remote cluster.
• Fix direct I/O path to set the Windows archive bit only for writes, not reads.
• Add more trace information and ifdef for problems with Linux NFS fast lookup.
• Ensure mmexportfs does not remove tiebreaker disks unless device name is "all".
• Fix kernel exception in fifo_open due to an invalid i_pipe pointer.
• Increase prefetchThreads+worker1Threads+nsdMaxWorkerThreads to 1500 on AIX 64bit systems.
• Added two new calls to the inode scan api to allow dmapi attributes to be backed up and restored for disaster recovery. The dmapi attributes are also saved in a snapshot of the original file on which they were set.
• Collect the output of systeminfo.exe command instead of the command itself.
• Fix trace record missing problem. The _STrace function should pass non-blocking flag as 0 instead of 1 to rl_trc_write.
• Improve SMP scalability in the DIO code path.
• Fix small window where message send will hang if destination list includes the local node and all other nodes reply before the local send can start.
• Prevent force unmounts when disks in different failure groups (FGs) fail, but are in different pools. Prevent marking disks down in multiple FGs when disks die simultaneously.
• Allow command line case insensitive hostname.
• Add an new file system option --filesetdf.
• Fixed assert when deleting files from a dmapi enabled filesystem.
• Dmapi locks are ignored from lock file operation during offline fsck as offline fsck doesnt have any dmapi context. This will prevent offline fsck from crashing the deaom while fixing orphans in a dmapi enabled filesystem.
• Fix mmapplypolicy to accept a policy that includes a 3-value THRESHOLD(hi,lo,pre) clause in a migrate to external pool rule.
• Ensure mmfsctl syncFSconfig does not affect free disks unless device name is "all".
• Ensure mmaddnode fails if the IP address already appears in the cluster.
• Fix node crash from destroy event when accessing stalled stripe group.
• Correct mmcrnsd disk sector size (>2T) error on 32bit Linux.
• Fix mmpolicyExec-hsm.sample to handle characters \ " and ' in filenames properly so that they work in HSM file list.
• Fix deadlock during FS manager takeover if previous FS manager and its disks (site failure) fail at the same time.
• Fix DMAPI enabled filesystems when they are mounted on top of another GPFS filesystem.
• Fixed performance problem while migrating files.
• To avoid 32-bit integer overflow in case of huge sparse file, argument dataBlockNum in repairDataBlock is changed to Int64 instead of int.
• Modified GPFS installation on Windows to prevent GPFS from being started after a package update, but before the mmwinserv service is configured. This problem can result in permission errors when running GPFS administrative operations.
• Fixed a problem in GPL layer makefile so that no warning messages will appear when upgrading from 3.3 GA code on Linux platform when GPL layer has never been compiled before.
• Acquire the stripe group descriptor mutex before changing the quota files inode information in the stripe group descriptor.
• On Windows, error messages related to the mmwinserv service were improved to provide clearer indication of the problem and how it might be corrected.
• Fixed the allocation code which caused an infinite loop when running out of full metadata block.
• Fix for a rare race condition that may result in a "Busy inodes after unmount" syslog message on Linux.
• Fsck generates false positives for bad repication status in user metadata files after a failed PIT operation. The fix ensures that fsck does not generate the false positives.
• When fixing orphans, fsck now prints the fileset name from which the orphan is generated.
• Fix mmbackupconfig to present a clearer error message when they attempt to run the command when the filesystem is mounted.
• Fix mmapplypolicy command to indicate it is making progress.
• Hold the stripe group descriptor mutex only while actually accessing/updating the stripe group descriptor when updating the quota file information in the descriptor.
• Avoid crash in rare cases of concurrent multi-node file creates.
• Prevent customer files that have been backed up to the TSM server from expiring in the same session.
• Enabled global cluster wide events from remote cluster in user exit callbacks.
• Fix "mmtrace noformat" to work on linux nodes.
• Correctly propagate authorization key files to new node in admin central cluster.
• Fixed repair code which can caused snapshot file corruption which could happen when filesystem contains fileset, snapshots and deleting disk resulted in not enough failure group for proper replication of metadata.
• Clarify mmwinserv error messages.
• Fix rare deadlock that occurs during recovery of large filesystem.
• Fixed GPFS trace control on Windows which in some scenarios was not restarting trace collection correctly.
• Fix backward compatibility problem that caused file creates to fail on file systems that were originally created with GPFS version 2.2 or earlier.
• This update addresses the following APARs: IZ63058 IZ63080 IZ63171 IZ63307 IZ63308 IZ63320.