IBM Support

/opt file system is getting full

Question & Answer


Question

The issue is /opt on one of our Puredata System For Analytics (AKA netezza) is getting full and hitting the threshold (80%). 'df' command output: Filesystem 1K-blocks Used Available Use% Mounted on ... /dev/sda8 8256952 6285272 1552252 81% /opt ...

Cause

One of the common cause for this; is the 'nz-snapshot' program is creating log files that occupied the file system space through a long period of time.

nz-snapshot (run under /opt/nz-snapshot) is ISOdx diagnostic troubleshooting tool. Each snapshot takes about 5-10 minutes and is nice'd to 5; nice minimise any impact on the system.

The output file size for each snapshot is approximately 50 MB. nz-snapshot stores a maximum of 30 snapshots and the original manufacturing image on the host.

Answer

You should first check the version of the utility using this command:
'cat /opt/nz-snapshot/versions.txt' ; you should make sure that is running 1.8 which is the latest version for now.

The log files commonly stored in /opt/nz-snapshot/isodx/custom01/Linux/out/hostname/# ; can be safely remove / move to another directory with bigger file-system space. For example, /nzscratch. 

Here is the example of the log files (using 'root' user) :-

# ls -ldrt /opt/nz-snapshot/isodx/custom01/Linux/out/hostname
drwxr-xr-x 2 root root 4096 Dec 18 2014 1418896502
drwxr-xr-x 2 root root 4096 Dec 25 2014 1419505145
drwxr-xr-x 2 root root 4096 Jan 1 2015 1420111264
drwxr-xr-x 2 root root 4096 Jan 8 06:35 1420710124
drwxr-xr-x 2 root root 4096 Jan 15 09:41 1421325545
drwxr-xr-x 2 root root 4096 Jan 22 06:22 1421918166
drwxr-xr-x 2 root root 4096 Jan 29 06:56 1422524585
drwxr-xr-x 2 root root 4096 Feb 5 06:30 1423127582
drwxr-xr-x 2 root root 4096 Feb 12 07:35 1423735742
drwxr-xr-x 2 root root 4096 Feb 19 10:18 1424349843
drwxr-xr-x 2 root root 4096 Feb 26 08:37 1424948044
drwxr-xr-x 2 root root 4096 Mar 5 07:23 1425547986
drwxr-xr-x 2 root root 4096 Mar 12 06:53 1426150682
drwxr-xr-x 2 root root 4096 Mar 19 06:52 1426754882
drwxr-xr-x 2 root root 4096 Mar 26 06:46 1427358902
drwxr-xr-x 2 root root 4096 Apr 2 08:03 1427967782
drwxr-xr-x 2 root root 4096 Apr 9 08:58 1428578762
drwxr-xr-x 2 root root 4096 Apr 16 12:00 1429193343
drwxr-xr-x 2 root root 4096 Apr 23 11:37 1429796465
drwxr-xr-x 2 root root 4096 Apr 30 10:17 1430395986
drwxr-xr-x 2 root root 4096 May 7 10:20 1431000662
drwxr-xr-x 2 root root 4096 May 14 09:05 1431600362
drwxr-xr-x 2 root root 4096 May 21 12:22 1432216143
drwxr-xr-x 2 root root 4096 May 28 12:57 1432821902
drwxr-xr-x 2 root root 4096 Jun 4 10:05 1433416505
drwxr-xr-x 2 root root 4096 Jun 11 10:34 1434022382
drwxr-xr-x 2 root root 4096 Jun 18 10:54 1434627842
drwxr-xr-x 2 root root 4096 Jun 25 08:44 1435224423
drwxr-xr-x 2 root root 4096 Jul 2 12:19 1435841463
drwxr-xr-x 2 root root 4096 Jul 9 13:33 1436450043

Inside those directory you should see files like:
-rw-r--r--  1 root root      1257 Sep 16 04:29 cdr.log                    
-rw-r--r--  1 root root      2114 Sep 16 04:29 ex_report.sh.err           
-rw-r--r--  1 root root      1385 Sep 16 04:29 ex_report.sh.log           
-rw-r--r--  1 root root       577 Sep 16 04:29 ex_report_vars             
-rw-r--r--  1 root root      3240 Sep 16 04:29 nzlocal.sh.err             
-rw-r--r--  1 root root     21966 Sep 16 04:29 nzlocal.sh.log             
-rw-r--r--  1 root root       567 Sep 16 04:29 nzlocal_vars               
-rw-r--r--  1 root root  56418806 Sep 16 04:29 nz.sh.err                  
-rw-r--r--  1 root root 178091310 Sep 16 04:29 nz.sh.log                  
-rw-r--r--  1 root root       542 Sep 16 04:23 nz_vars                    
-rw-r--r--  1 root root    214732 Sep 16 04:23 os.sh.err                  
-rw-r--r--  1 root root   3351579 Sep 16 04:23 os.sh.log      

The easiest way is to move some of the older directory (consist of log files) ; to /nzscratch/nz-snapshot-backup directory or simply remove them.

For example,
mkdir /nzscratch/nz-snapshot-backup
mv 1418896502 /nzscratch/nz-snapshot-backup

You may wish to zip or remove those file in /nzscratch/nz-snapshot-backup when no longer needed. (/nzscratch have much bigger file-system space compared to /opt ; you can also backup to other directory with sufficient space)

If in doubt, please contact IBM Support and our Support Engineer will assist you on this.

[{"Product":{"code":"SSULQD","label":"IBM PureData System"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Component":"Not Applicable","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"1.0.0","Edition":"","Line of Business":{"code":"LOB10","label":"Data and AI"}}]

Document Information

Modified date:
17 October 2019

UID

swg21961829