Concepts and terminology

The following concepts and terminology are new and specific for LSF Data Manager.

LSF data manager

The LSF data manager runs on dedicated LSF server hosts. The LSF data manager hosts are configured to run the LSF data manager daemon (dmd). The LSF data manager daemon communicates with the clusters it serves, and manages the transfer of data in the staging area.

Tip: Configure the LSF data manager host as an LSF server host to enable dmd to start automatically when LIM starts.

Query the LSF data manager with the bdata command to get information about the required data files, LSF data manager configuration, cluster connections, transfer status, and other information. LSF data manager administrators can use the bdata command to reconfigure and shut down dmd.

LSF data manager administrator

The administrator of IBM® Spectrum LSF Data Manager must be an LSF administrator for all clusters that are connected to the data manager. LSF data manager administrators make sure that the dmd daemon is operating smoothly and reconfigure LSF data manager as needed.

LSF data manager administrators manage the LSF data manager:
  • Manage the LSF data manager data transfer queue in the lsb.queues file
  • Run the bdata admin reconfig command to reconfigure LSF data manager
  • Run the bdata admin shutdown command to shut down LSF data manager
  • Run the bdata tags command to list or clean intermediate files that are associated with a tag for users

Configure the LSF data manager administrators with the ADMINS parameter in the lsf.datamanager file.

lsf.datamanager file

The lsf.datamanager file controls the operation of LSF Data Manager features. Each cluster has one LSF data manager configuration file, named lsf.datamanager.cluster_name. The cluster_name suffix is the name of the cluster that is defined in the Cluster section of lsf.shared. The file is read by the LSF data management daemon dmd during startup and reconfiguration.

Data transfer node

A data transfer node, also referred to as an I/O node, is an LSF server host in the cluster that is mounted with direct read/write access to the cluster staging area. This host can access the source of staged-in data and the destination of staged-out data.

Data transfer job

LSF data manager submits transfer jobs to copy required data files for stage in or stage out operations. Transfer jobs run on data transfer nodes as the execution user of the job that triggered the transfer.

Transfer jobs have the following function:
  • To pre-stage files that are requested in the bsub –data option from their source location into the staging area cache.
  • Stage out files that are requested by the bstage out command from the staging area cache to their remote destination.

Data transfer queue

LSF data manager submits transfer jobs to a transfer queue, which is configured to accept transfer jobs only.

Data transfer tool command

Transfer jobs run the transfer tool command that is specified in the FILE_TRANSFER_CMD parameter in the lsf.datamanager file.

Data staging

A staging area, also known as a data cache, is a managed file system on an LSF server host or mounted directory that is accessible to the cluster compute nodes. The staging area stores files that jobs request for staging in or out. Each staging area must have one LSF data manager instance.

Data staging is the transfer of data to the location where it is used. Input data for a job is staged in in two steps:
  1. By copying input data files from the data source repository to the data staging area cache through a data transfer job.
  2. By copying data that is required by an application from the staging area to the job execution host.

Output data that is produced by a job is staged out by copying files back to a location that the job submitter can use it.

Stage-out operations asynchronously copy job output from the cache area to the data destination that you specify.

Data specification file

A data specification file is a text file that is used for specifying many data requirement files for one job.

Each line in a data specification file specifies the name of the path to a source file to be transferred to the staging area before a job is submitted and scheduled. The path can point to a file or a directory.

The following example contains lines for three files. Each line specifies a host_name:file_path pair:

#@dataspec
datahost:/proj/userA/input1.dat
datahost:/proj/userA/input2.dat
datahost:/proj/userA/input3.dat

Data tags

A data tag can be created for a job with a data staging requirement with the bstage out command. A tag allows users to transfer files from the job's current working directory to the staging area, associate those files with a chosen name. Data manager commands can report the existence of that tag if it is queried later.

Data queries

File-based cache query with the bdata cache command displays the job IDs of jobs that request the file under the REF_JOB column. The REF_JOB column is not displayed for job-based query in the bdata cache command.