Question & Answer
Question
The issue is /opt on one of our Puredata System For Analytics (AKA netezza) is getting full and hitting the threshold (80%). 'df' command output: Filesystem 1K-blocks Used Available Use% Mounted on ... /dev/sda8 8256952 6285272 1552252 81% /opt ...
Cause
One of the common cause for this; is the 'nz-snapshot' program is creating log files that occupied the file system space through a long period of time.
nz-snapshot (run under /opt/nz-snapshot) is ISOdx diagnostic troubleshooting tool. Each snapshot takes about 5-10 minutes and is nice'd to 5; nice minimise any impact on the system.
The output file size for each snapshot is approximately 50 MB. nz-snapshot stores a maximum of 30 snapshots and the original manufacturing image on the host.
Answer
You should first check the version of the utility using this command:
'cat /opt/nz-snapshot/versions.txt' ; you should make sure that is running 1.8 which is the latest version for now.
The log files commonly stored in /opt/nz-snapshot/isodx/custom01/Linux/out/hostname/# ; can be safely remove / move to another directory with bigger file-system space. For example, /nzscratch.
Here is the example of the log files (using 'root' user) :-
# ls -ldrt /opt/nz-snapshot/isodx/custom01/Linux/out/hostname
drwxr-xr-x 2 root root 4096 Dec 18 2014 1418896502
drwxr-xr-x 2 root root 4096 Dec 25 2014 1419505145
drwxr-xr-x 2 root root 4096 Jan 1 2015 1420111264
drwxr-xr-x 2 root root 4096 Jan 8 06:35 1420710124
drwxr-xr-x 2 root root 4096 Jan 15 09:41 1421325545
drwxr-xr-x 2 root root 4096 Jan 22 06:22 1421918166
drwxr-xr-x 2 root root 4096 Jan 29 06:56 1422524585
drwxr-xr-x 2 root root 4096 Feb 5 06:30 1423127582
drwxr-xr-x 2 root root 4096 Feb 12 07:35 1423735742
drwxr-xr-x 2 root root 4096 Feb 19 10:18 1424349843
drwxr-xr-x 2 root root 4096 Feb 26 08:37 1424948044
drwxr-xr-x 2 root root 4096 Mar 5 07:23 1425547986
drwxr-xr-x 2 root root 4096 Mar 12 06:53 1426150682
drwxr-xr-x 2 root root 4096 Mar 19 06:52 1426754882
drwxr-xr-x 2 root root 4096 Mar 26 06:46 1427358902
drwxr-xr-x 2 root root 4096 Apr 2 08:03 1427967782
drwxr-xr-x 2 root root 4096 Apr 9 08:58 1428578762
drwxr-xr-x 2 root root 4096 Apr 16 12:00 1429193343
drwxr-xr-x 2 root root 4096 Apr 23 11:37 1429796465
drwxr-xr-x 2 root root 4096 Apr 30 10:17 1430395986
drwxr-xr-x 2 root root 4096 May 7 10:20 1431000662
drwxr-xr-x 2 root root 4096 May 14 09:05 1431600362
drwxr-xr-x 2 root root 4096 May 21 12:22 1432216143
drwxr-xr-x 2 root root 4096 May 28 12:57 1432821902
drwxr-xr-x 2 root root 4096 Jun 4 10:05 1433416505
drwxr-xr-x 2 root root 4096 Jun 11 10:34 1434022382
drwxr-xr-x 2 root root 4096 Jun 18 10:54 1434627842
drwxr-xr-x 2 root root 4096 Jun 25 08:44 1435224423
drwxr-xr-x 2 root root 4096 Jul 2 12:19 1435841463
drwxr-xr-x 2 root root 4096 Jul 9 13:33 1436450043
Inside those directory you should see files like:
-rw-r--r-- 1 root root 1257 Sep 16 04:29 cdr.log
-rw-r--r-- 1 root root 2114 Sep 16 04:29 ex_report.sh.err
-rw-r--r-- 1 root root 1385 Sep 16 04:29 ex_report.sh.log
-rw-r--r-- 1 root root 577 Sep 16 04:29 ex_report_vars
-rw-r--r-- 1 root root 3240 Sep 16 04:29 nzlocal.sh.err
-rw-r--r-- 1 root root 21966 Sep 16 04:29 nzlocal.sh.log
-rw-r--r-- 1 root root 567 Sep 16 04:29 nzlocal_vars
-rw-r--r-- 1 root root 56418806 Sep 16 04:29 nz.sh.err
-rw-r--r-- 1 root root 178091310 Sep 16 04:29 nz.sh.log
-rw-r--r-- 1 root root 542 Sep 16 04:23 nz_vars
-rw-r--r-- 1 root root 214732 Sep 16 04:23 os.sh.err
-rw-r--r-- 1 root root 3351579 Sep 16 04:23 os.sh.log
The easiest way is to move some of the older directory (consist of log files) ; to /nzscratch/nz-snapshot-backup directory or simply remove them.
For example,
mkdir /nzscratch/nz-snapshot-backup
mv 1418896502 /nzscratch/nz-snapshot-backup
You may wish to zip or remove those file in /nzscratch/nz-snapshot-backup when no longer needed. (/nzscratch have much bigger file-system space compared to /opt ; you can also backup to other directory with sufficient space)
If in doubt, please contact IBM Support and our Support Engineer will assist you on this.
Was this topic helpful?
Document Information
Modified date:
17 October 2019
UID
swg21961829