Topic
  • 2 replies
  • Latest Post - ‏2013-03-21T19:47:00Z by SystemAdmin
SystemAdmin
SystemAdmin
1283 Posts

Pinned topic WebGUI 7.3.1 FP4 too many files open

‏2013-03-12T18:52:35Z |
Hello,

We have installed WebGUI 7.3.1 fix pack 4 on a Linux x86-64 bit system. After a few days WebGUI crashes (no javacores or heapdumps). I see the following errors in the server1 SystemErr.log file,

3/7/13 23:44:24:732 PST 00000053 SystemErr R java.util.logging.ErrorManager: OPEN_FAILURE
3/7/13 23:44:24:732 PST 00000053 SystemErr R Error message - Error happened when open log file.
3/7/13 23:44:24:732 PST 00000053 SystemErr R Exception - java.io.FileNotFoundException: /opt/IBM/tivoli/tipv2/profiles/TIPProfile/logs/ncw/ncw.0.trace (Too many open files)

The strange thing is that we have adjusted the system's file limits with the following values,

/etc/security/limits.conf is:
ncoadmin soft nofile 32768
ncoadmin hard nofile 32768

The contents of the /proc/sys/fs/file-max

343784

We have also tried to increase the amount of memory the WebGUI JVM is using with no luck. The operating system is Redhat Linux 5.x and here is the uname -a output,

Linux XXXXX 2.6.18-308.11.1.el5 #1 SMP Fri Jun 15 15:41:53 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux

Output from the ulimit command (for the user who runs webgui),

unlimt -Hn is 32768
ulimit -Sn is 32768
While WebGUI was up, we ran lsof -p <java_pid> | wc -l and the count of open files for this PID was 22410 and climbing. I have attached the logs which where written to (there are no other log files, maybe webgui crashed before it could write to them?).

Thanks for any help,

Marc
Updated on 2013-03-21T19:47:00Z at 2013-03-21T19:47:00Z by SystemAdmin
  • SystemAdmin
    SystemAdmin
    1283 Posts

    Re: WebGUI 7.3.1 FP4 too many files open

    ‏2013-03-13T05:47:49Z  
    Hi Marc,

    The way that you followed for fixing too many files open is absolutely correct.
    Are you observing any CPU spikes when this is happening?

    Thanks
    Vamsi
  • SystemAdmin
    SystemAdmin
    1283 Posts

    Re: WebGUI 7.3.1 FP4 too many files open

    ‏2013-03-21T19:47:00Z  
    Hello,

    We ended up re-cloning the VM and re-installed everything. This has resolved our problem. Thanks for checking :)