IBM InfoSphere Streams Version 4.1.1

Guidelines for configuring Linux ulimit settings for InfoSphere Streams

The default Linux user limit (ulimit) settings might be too small for some InfoSphere® Streams environments, which can cause processing element (PE) failures. You can use ulimit settings to control the use of system resources.

Hard and soft ulimit settings

There are two types of ulimit settings:

The hard limit is the maximum value that is allowed for the soft limit. Any changes to the hard limit require root access.
The soft limit is the value that Linux uses to limit the system resources for running processes. The soft limit cannot be greater than the hard limit.

InfoSphere Streams provides tools that you can use to verify that the hard and soft ulimit settings for the maximum number of processes and open files are compatible with InfoSphere Streams.

Tested ulimit settings for InfoSphere Streams

When testing InfoSphere Streams on RHEL, CentOS, and SLES systems, the following soft ulimit settings were used:

open files value = 100000
max user processes value = pending signals value
pending signals value = 100000

Notes:

The InfoSphere Streams dependency checker script issues warning messages if the max user processes value is less than 5000, or the open files value is less than 1024.
The InfoSphere Streams dependency checker script and streamtool checkhosts command issue warning messages if the max user processes value is less than or equal to 1024, or the open files value is less than 1024.
These minimum values might need to be increased based on your InfoSphere Streams applications and environment. Most InfoSphere Streams services use about 30 file handlers and 30 to 50 user processes, but these services can grow to consume over 500 file handlers on each resource under certain usage conditions. Additionally, ensure that the ulimit setting for user processes is set to 5000 or higher due to the process demands that the number of InfoSphere Streams services can place on each resource.

Verifying that hard and soft ulimit settings are compatible with InfoSphere Streams

You can use the InfoSphere Streams dependency checker script and streamtool checkhosts command to verify that the hard and soft ulimit settings for the maximum number of processes and open files are compatible with InfoSphere Streams. If your ulimit settings are not compatible with InfoSphere Streams, messages are displayed in the command output.

For a dependency checker script example, see Dependency checker script output example for an x86_64 system running RHEL 6.7.

Reviewing hard and soft ulimit settings

Use the following commands to review the ulimit settings for the InfoSphere Streams host.

To review the hard ulimit settings, enter the following command:

ulimit -aH

Command output similar to the following example is displayed:

user1@host1 ˜$ ulimit -aH
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 100000
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 100000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) unlimited
cpu time               (seconds, -t) unlimited
max user processes              (-u) 257262
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

To review the soft ulimit, enter the following command:

ulimit -aS

Command output similar to the following example is displayed:

user1@host1 ˜$ ulimit -aS
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 100000
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 100000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 257262
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Updating hard and soft ulimit settings

Attention:

Before you change the ulimit settings, contact your system administrator.
If you change the ulimit settings, you must restart your system for the changes to take effect.

There are several ways to change these settings. The following examples show one way to change the hard and soft ulimit settings for InfoSphere Streams administrators that belong to the @streamsadmin user group.

Note: If a .conf file in the following examples does not exist on your system, create the file using the same name in the example and then edit the file as shown in the example.

To change the open files value on your operating system:
- On RHEL and CentOS, edit the /etc/security/limits.d/91-nofile.conf file as shown in the following example:
```
@streamsadmin - nofile open-files-value
```
- On SLES, edit the /etc/security/limits.conf file as shown in the following example:
```
@streamsadmin - nofile open-files-value
```
To change the max user processes value on your operating system:
- On RHEL and CentOS, edit the /etc/security/limits.d/90-nproc.conf file as shown in the following example:
```
@streamsadmin hard nproc max-user-processes-value
@streamsadmin soft nproc max-user-processes-value
```
- On SLES, edit the /etc/security/limits.conf file as shown in the following example:
```
@streamsadmin hard nproc max-user-processes-value
@streamsadmin soft nproc max-user-processes-value
```
To set the hard stack and soft stack values, add the following lines to the /etc/security/limits.conf file:
```
@streamsadmin hard stack unlimited 
@streamsadmin soft stack 20480
```

Use the following ulimit commands to verify the updated settings:

To verify the updated hard limit, enter the following command:
```
ulimit -aH
```
To verify the updated soft limit, enter the following command:
```
ulimit -aS
```