Question & Answer
Question
IBM Java for AIX HowTo: Java application leaves .nfsxxxx files behind
Answer
Under UNIX, if you remove a file that a currently running process still has open, the file isn't really removed. Once the process closes the file, the OS then removes the file handle and frees up the disk blocks. This process is complicated slightly when the file that is open and removed is on an NFS mounted filesystem.
The way NFS keeps track of open files in an NFS mounted filesystem is with .nfsxxxx files.
If you try to remove one of these file, and the file is still open, it will just reappear with a different number.
The .nfsxxxx files are used by NFS clients to manage the deletion of open files.
If an open file is deleted, then the client renames it to .nfsxxxx.
If the last open to this file is closed then the client should send a request to remove it.
However, if the client crashes before it can clean up then you'll be left with these files.
So, in order to remove the file completely you must kill the process that has it open.
Once you have located and killed the process that has the file open, the .nfs file will go away automatically.
*If you want to know what process has this file open you can, however you will only be able to if you are on the machine where the processes that has the file open is running.
If you try to remove a file and receive output similar to the following:
[root@test-host ~]# rm -rf /opt/WebSphere/AppServer/*
rm: cannot remove `/opt/WebSphere/AppServer/.nfs0000000000004abf00000001': Device or resource busy
rm: cannot remove /opt/WebSphere/AppServer/.nfs0000000000005c7a00000002': Device or resource busy
rm: cannot remove /opt/WebSphere/AppServer/.nfs0000000000006bcf00000003': Device or resource busy
Then it means that one or more processes were still referring to these files.
You have to stop these processes before you can remove these .nfsxxxx files.
You can use the command 'lsof' to find the processes using specific files.
Agin, if you want to know what process has the file(s) open, you can use the 'lsof' command, but this will only work if you are on the machine where the processes that has the file open is running.
[root@test-host ~]# lsof |grep nfs0000000000004abf00000001
java 2956 emcadm mem REG 0,21 1095768 19135 /opt/WebSphere/AppServer/.nfs0000000000004abf00000001 (slce49sn-nas:/export/C9QA123_DC1/tas_central_shared)
/opt/WebSphere/AppServer/.nfs0000000000004abf00000001 (slce49sn-nas:/export/C9QA123_DC1/tas_central_shared)
So from here you can see the process with PID 2956 is still using file /opt/WebSphere/AppServer/.nfs0000000000004abf00000001.
Some systems do not have the 'lsof' command/software installed, so you can install it or use the alternative command of 'fuser':
[root@test-host ~]# fuser -cu /opt/WebSphere/AppServer/
/opt/WebSphere/AppServer/: 2956m(emcadm) 7358c(aime)
Here you can see the processes with PIDs 2956 and 7358 are referring to the directory /opt/WebSphere/AppServer/.
So you'll need stop the process first by killing it (or stop it using the processes own stop() method if defined):
[root@test-host ~]# kill -9 2956
You'll need stop the process first by killing it (or stop it using the processes own stop() method if defined):
[root@test-host ~]# kill -9 2956
Then remove the .nfsxxx file(s) from the directory
For example:
[root@test-host ~]# rm /opt/WebSphere/AppServer/.nfs*
*NOTE: If you simply remove the .nfsxxxx file(s) from the directory, they will be automatically regenerated with a new random number after the .nfs in the file name
Contact IBM Support
If, after reading and following the above instructions, further assistance is required, please complete the following steps:
1. Confirm that you have review and completed all of the above steps.
2. Contact IBM and open a new IBM service request (i.e., a new IBM PMR).
3. Collect and upload data as per the data collection procedures noted in the above sections or package and upload the current data and details by following the instructions on this web page:
http://www-01.ibm.com/support/docview.wss?uid=isg3T1022619" target="_blank">IBM Java for AIX MustGather: How to upload diagnostic data and testcases to IBM
| Document Type: | Technical Document |
| Content Type: | General |
| Hardware: | all Power |
| Operating System: | all AIX Versions |
| IBM Java: | all Java Versions |
| Author(s): | Christopher C. Peters |
| Reviewer(s): | Rama Tenjarla |
Was this topic helpful?
Document Information
More support for:
IBM Java
Software version:
Version Independent
Operating system(s):
AIX
Document number:
630251
Modified date:
17 June 2018
UID
isg3T1024656