IBM Support

Managing and Deleting Persistent Data Sets within IBM InfoSphere Datastage

Question & Answer


Data Sets sometimes take up too much disk space. This technote describes how to obtain information about datasets and how to delete them.


Data sets can be managed using the Data Set Management tool, invoked from the Tools > Data Set Management menu option within DataStage Designer (DataStage Manager in the 7.5 releases.) Alternatively, the 'orchadmin' command line program can be used to perform the same tasks.

The files which store the actual data persist in the locations identified as resource disks in the configuration files. These files are named according to the pattern below:

descriptor: Name of the data set descriptor file.
user: Your user name.
host: Hostname from which you invoked the job which created the data set.
ssss: 4-digit segment identifier (0000-9999)
pppp: 4-digit partition identifier (0000-9999)
nnnn: 4-digit file identifier (0000-9999) within the partition
pid: Process ID of the job on the host from which you invoked the jop that creates the data set.
time: 8-digit hexadecimal time stamp in seconds.
index: 4-digit number incremented for each file.
random: 8 hexadecimal digits containing a random number to insure unique file names.

For example, suppose that your configuration file contains the following node definitions:

    node node0
         fastname "host1"
         pools ""
         resource disk "/orch/s0" {pools ""}
         resource scratchdisk "/scratch" {pools ""}
    node node1
         fastname "host1"
         pools ""
         resource disk "/orch/s0" {pools ""}
         resource scratchdisk "/scratch" {pools ""}

A data set named mydata.ds created by a job using this configuration file will contain data in two partitions, one for each processing node declared in the configuration file. Because each processing node contains only a single disk specification, each partition of data would be stored in a single file on each processing node. Following the naming convention shown above, the data file for partition 0 would be located on the host1 machine, in the /orch/s0 filesystem, and the file would be named:


The data file for partition 1 data would be similarly named:


It is important to understand that the file referenced in the job, called mydata.ds in our example, does not contain any actual data. It is a data set descriptor file, and it contains information about how the data set is constructed. In order for DataStage jobs to access the data, both the descriptor and the actual segment files must exist.

Cleaning up Data Sets

A good plan for managing data sets is to identify the Data Sets that are no longer required, and to use the Data Set Management tool to delete them. If you have the jobs that reference the data sets, you can open each of the data set descriptor files using the Data Set Management tool and then view and delete the data set. If you do not have the jobs, another possible method is to look in the resource disk locations for segment files with very old modification dates. Once you have identified the segment files, you can determine what the data set descriptor file name was.


In this example segment file shown above, the highlighted "mydata.ds" is the file name of the data set descriptor. You can then locate this file in your computer with the find command.

find /my_projects/datasets/ -name "mydata.ds" -print

Once you have located the descriptor file, you can then use the Data Set Management tool to view and delete the data set. If someone has already deleted the descriptor file, then the segments have been orphaned. There is no utility or function to recreate the descriptor file. In this situation, you can safely delete all the segment files named with the "mydata.ds" in the file name.

Cleaning up Data Sets from the command line

It is also possible to use the orchadmin executable program to delete data sets. This program is located in $APT_ORCHHOME/bin.

To delete a data set using orchadmin, the environment has to be setup properly, and the descriptor file has to exist. Follow these steps to use delete a data set.

$ cd $DSHOME
$ . ./dsenv
$ APT_CONFIG_FILE=<config file path>; export APT_CONFIG_FILE
$ $DSHOME/../PXEngine/bin/orchadmin delete <full path to descriptor file>

Note: adjust the steps accordingly for your platform, for example use LIBPATH instead of LD_LIBRARY_PATH on the AIX platform.

[{"Product":{"code":"SSVSEF","label":"IBM InfoSphere DataStage"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Component":"--","Platform":[{"code":"PF002","label":"AIX"},{"code":"PF010","label":"HP-UX"},{"code":"PF016","label":"Linux"},{"code":"PF027","label":"Solaris"},{"code":"PF033","label":"Windows"}],"Version":"9.1;8.7;8.5;8.1;8.0.1;7.5;11.5;11.3","Edition":"Enterprise","Line of Business":{"code":"LOB10","label":"Data and AI"}}]

Document Information

Modified date:
23 June 2018