IBM Support

IBM Java for AIX HowTo: Java application leaves .nfsxxxx files behind

Question & Answer


Question

IBM Java for AIX HowTo: Java application leaves .nfsxxxx files behind

Answer


Under UNIX, if you remove a file that a currently running process still has open, the file isn't really removed. Once the process closes the file, the OS then removes the file handle and frees up the disk blocks. This process is complicated slightly when the file that is open and removed is on an NFS mounted filesystem.
Overview
Details
Explanation of .nfsxxxx files


The way NFS keeps track of open files in an NFS mounted filesystem is with .nfsxxxx files.
If you try to remove one of these file, and the file is still open, it will just reappear with a different number.
The .nfsxxxx files are used by NFS clients to manage the deletion of open files.
If an open file is deleted, then the client renames it to .nfsxxxx.
If the last open to this file is closed then the client should send a request to remove it.
However, if the client crashes before it can clean up then you'll be left with these files.
So, in order to remove the file completely you must kill the process that has it open.
Once you have located and killed the process that has the file open, the .nfs file will go away automatically.

*If you want to know what process has this file open you can, however you will only be able to if you are on the machine where the processes that has the file open is running.

Example of how to determine the process referring to the .nfsxxx files


If you try to remove a file and receive output similar to the following:

[root@test-host ~]# rm -rf /opt/WebSphere/AppServer/*
rm: cannot remove `/opt/WebSphere/AppServer/.nfs0000000000004abf00000001': Device or resource busy
rm: cannot remove /opt/WebSphere/AppServer/.nfs0000000000005c7a00000002': Device or resource busy
rm: cannot remove /opt/WebSphere/AppServer/.nfs0000000000006bcf00000003': Device or resource busy


Then it means that one or more processes were still referring to these files.
You have to stop these processes before you can remove these .nfsxxxx files.


You can use the command 'lsof' to find the processes using specific files.
Agin, if you want to know what process has the file(s) open, you can use the 'lsof' command, but this will only work if you are on the machine where the processes that has the file open is running.

[root@test-host ~]# lsof |grep nfs0000000000004abf00000001
java 2956 emcadm mem REG 0,21 1095768 19135 /opt/WebSphere/AppServer/.nfs0000000000004abf00000001 (slce49sn-nas:/export/C9QA123_DC1/tas_central_shared)
/opt/WebSphere/AppServer/.nfs0000000000004abf00000001 (slce49sn-nas:/export/C9QA123_DC1/tas_central_shared)


So from here you can see the process with PID 2956 is still using file /opt/WebSphere/AppServer/.nfs0000000000004abf00000001.


Some systems do not have the 'lsof' command/software installed, so you can install it or use the alternative command of 'fuser':

[root@test-host ~]# fuser -cu /opt/WebSphere/AppServer/
/opt/WebSphere/AppServer/: 2956m(emcadm) 7358c(aime)


Here you can see the processes with PIDs 2956 and 7358 are referring to the directory /opt/WebSphere/AppServer/.


So you'll need stop the process first by killing it (or stop it using the processes own stop() method if defined):

[root@test-host ~]# kill -9 2956

How to get rid of the .nfsxxx files

You'll need stop the process first by killing it (or stop it using the processes own stop() method if defined):

[root@test-host ~]# kill -9 2956


Then remove the .nfsxxx file(s) from the directory
For example:
[root@test-host ~]# rm /opt/WebSphere/AppServer/.nfs*

*NOTE: If you simply remove the .nfsxxxx file(s) from the directory, they will be automatically regenerated with a new random number after the .nfs in the file name

Section 4
Section 5
Section 6
Section 7
Section 8
Section 9
Section 10
Section 11
Section 12
Section 13
Section 14
Section 15
Section 16
Section 17
Section 18
Section 19
Section 20

Contact IBM Support


If, after reading and following the above instructions, further assistance is required, please complete the following steps:

1. Confirm that you have review and completed all of the above steps.

2. Contact IBM and open a new IBM service request (i.e., a new IBM PMR).

3. Collect and upload data as per the data collection procedures noted in the above sections or package and upload the current data and details by following the instructions on this web page:


http://www-01.ibm.com/support/docview.wss?uid=isg3T1022619" target="_blank">IBM Java for AIX MustGather: How to upload diagnostic data and testcases to IBM

Document Type: Technical Document
Content Type: General
Hardware: all Power
Operating System: all AIX Versions
IBM Java: all Java Versions
Author(s): Christopher C. Peters
Reviewer(s): Rama Tenjarla
[{"Product":{"code":"SG9NGS","label":"IBM Java"},"Business Unit":{"code":null,"label":null},"Component":"--","Platform":[{"code":"PF002","label":"AIX"}],"Version":"Version Independent","Edition":"","Line of Business":{"code":"LOB08","label":"Cognitive Systems"}}]

Document Information

More support for:
IBM Java

Software version:
Version Independent

Operating system(s):
AIX

Document number:
630251

Modified date:
17 June 2018

UID

isg3T1024656

Manage My Notification Subscriptions