Use of the tunable parameters in the DataStage server engine UVCONFIG file

MFILES

This parameter defines the size of DSEngine's rotating file pool. It is a per process setting for tracking files opened by DSEngine. It does not include files opened directly at OS level by the parallel engine (PXEngine running osh).

The server engine logically opens and closes files at the DataStage application level, and physically closes them at the OS level when the need arises.

Increase this value when DataStage server jobs use many files. The default value of 200 is usually suitable though. If the value is set too low, then performance issues might occur. That is, DSEngine makes more calls to open and close at the physical OS level in order to map the logical pool to the physical pool.

NOTE: The operating system parameter of nofiles needs to be set higher than MFILES. In a correctly configured Information Server environment, this requirement is not a concern since the recommended value for ulimit -n is a minimum of 10240.

On most UNIX systems, the proc file system can be used to monitor the file handles opened by a process.

T30FILE

This parameter determines the maximum number of dynamic hashed files that can be opened system-wide by DSEngine. Many of the job control files in projects are dynamic hashed files such RT_LOG, RT_STATUS, RT_CONFIG, and VOC.

The default value of 512 can be inadequate. If this value is too low, expect to find an error similar to Unable to allocate Type 30 descriptor, table is full in phantom records. Look for these messages under the &PH& directory of projects.

After the DSEngine environment file is sourced, the following command shows the number of dynamic files in use when the engine tier is on UNIX platforms.

. /opt/IBM/InformationServer/Server/DSEngine/dsenv

Use this command to assist with tuning the T30FILE parameter. Another technote explores this topic in more detail (www.ibm.com/support/docview.wss?uid=swg21390117).

At run time, a DataStage job requires at least 3 slots in this table for its associated RT_CONFIG, RT_LOG, and RT_STATUS files. When multiple instances of a job are running simultaneously, they share slots for these files. That is, each instance just increments a usage counter in the table when the file is already open to another instance.

On AIX®, don't set the T30FILE value higher than the system setting ulimit -n.

This parameter defines the number of slots in each row of record lock tables. From a DataStage job point of view, this value affects the number of DataStage jobs that can be executed concurrently, and the number of DataStage clients that can connect.

Sample smat output when one Director client is logged in to the dstage1 project, and 2 instances of a parallel job are running:

In the report, Item-ID of RT_CONFIG43 identifies a running instance of job number 43. A compiled job's RT_CONFIG file is locked at run time to prevent it from being recompiled or deleted. A job’s number within its project can also be seen in the Detail view of Director.

The numerical value in the Lmode column relates to a row number within a lock table. Each row can hold RLTABSZ locks. In the example, 3 slots out of 300 (default value for RLTABSZ) are used in row 63. When the number of entries for a row gets close to the RLTABSZ value, it is time to consider retuning the system. Two symptoms are jobs failing to start or generating -14 errors, and DataStage clients reporting an error message similar to DataStage Project locked by Administrator. Note the latter error message can be misleading; it means in this case that a lock cannot be acquired because the lock table is full, and not that another user already has the lock.

The row containing the project locks is the most likely one to approach the limit. Use the following smat command to monitor the project locks.

On UNIX:

$DSHOME/bin/smat -r|grep DS.ADMIN|wc -l

On Windows:

SH -c "C:/IBM/InformationServer/Server/DSEngine/bin/smat -r | grep DS.ADMIN | wc -l"

MAXRLOCK

Set this parameter to the value of RLTABSZ – 1.

Each DSD.RUN process takes a record lock on a key named project&!DS.ADMIN!& of the UV.ACCOUNT file in $DSHOME as seen in the previous smat output. Each DataStage client connection (for example, Designer, Director, Administrator, dsjob command) takes this record lock as well. This lock is the mechanism by which DataStage determines whether operations such as project deletion are safe. That is, operations cannot proceed while a project lock is held by any process.

MAXRLOCK needs to be set to accommodate the maximum number of jobs and sequences plus client connections that are used at any time. And RLTABSZ needs to be set to MAXRLOCK + 1.

Keep in mind that changing RLTABSZ greatly increases the amount of memory needed by the disk shared memory segment.

DMEMOFF, PMEMOFF, CMEMOFF, NMEMOFF

These parameters are the shared memory address offset values for each of the four DataStage shared memory segments (disk, printer, catalog, and NLS). The disk shared memory segment contains the dynamic file and record lock tables. Depending upon the platform, changes to T30FILE and RLTABSZ might require adjustments to PMEMOFF, CMEMOFF, and NMEMOFF to allow for larger disk shared memory space.

These values are set to 0x0 on Windows, which means the OS takes care of managing the offsets, and never need to be revised. All of the current UNIX platforms require nonzero offsets. The AIX offset values are adequate and aren't usually revised. Otherwise, on Linux64 and Linux390 platforms, when the size required for disk shared memory exceeds PMEMOFF - DMEMOFF, changes are required. This type of message is reported by uvregen when there is not enough space:

Don't use the shmtest utility. It frequently provides invalid offsets (values exceeding 32 bits) that result in random faults.