Samples for programming a pipeline

See samples for programming a pipeline with Bash scripts, such as installing third party packages, and more.

Using CPDCTL

Refer to samples to get started with some end-to-end example code, including how to deploy and monitor pipelines.

Programming with Bash

There are various development tools you can use by running Bash scripts in pipelines.

Installing packages with Bash

Download the required (psql) packages on any Linux machine outside the runtime pod, then copy the libraries into the volume mounted by PXRuntime.

Note:

The image RH UBI 9 is used in Pipelines 4.8.3 or later, while RH UBI 8 is used in Pipelines 4.8.2 or earlier. You need to extract the psql package from Red-hat Package Manager.

  1. Download PostgreSQL RPM packages in a separate machine:
    yum install --downloadonly --downloaddir=/tmp/psql postgresql
  2. Extract and reconstruct RPM packages:
    cd /tmp/psql
mkdir postgresql
cd postgresql
rpm2cpio ../openssh-*.rpm | cpio -idmv
# Repeat for each downloaded RPM package (cannot use wildcards)
  3. Create tarball:
    cd /tmp/psql
tar -zcvf psql.tar.gz ./postgresql
  4. Copy to storage volume:
    # Copy files to px-runtime pod
oc cp psql.tar.gz <px-runtime-pod>:/px-storage/
  5. Extract in runtime pod:
    # Remote shell into px-runtime pod
oc -n <cp4d-namespace> rsh <px-runtime-pod>
cd /px-storage
tar -xzf psql.tar.gz

For more information, see: Installing third-party libraries and creating custom images in DataStage.

Environment configuration

  1. Create extra_config.sh script:
    # Create /px-storage/extra_config.sh with:
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/px-storage/postgresql/usr/lib64:/px-storage/postgresql/usr/lib:/px-storage/postgresql/usr/share
export PATH=$PATH:/px-storage/postgresql/usr/bin
  2. Apply configuration to deployment:
    oc set env deployment/<instance-name>-ibm-datastage-px-runtime DS_EXTRA_CONFIG_SH=/px-storage/extra_config.sh
oc set env sts/<instance-name>-ibm-datastage-px-compute DS_EXTRA_CONFIG_SH=/px-storage/extra_config.sh

Testing and verification

  1. Test in bash script node:
    # Verify installation
psql --version
pg_dump --version

# Test database connection
psql -U ${USER} -h ${HOST_IP} -p ${PORT} -d ${DB} -c "\l"

Using API samples

See some samples for using Pipelines API for cases such as:

  • List all pipelines in a project.
  • Finds a specific pipeline by ID.
  • Deletes a pipeline and its pipeline versions.
  • Lists all pipeline versions of a given pipeline.
  • Gets a pipeline version by pipeline version ID.
  • Deletes a pipeline version by pipeline version ID.
  • Upload a pipeline file and create a new pipeline
  • Upload a pipeline file and create a new pipeline version
  • Commit a pre-existing volatile default version to finished state.
  • Create a new pipeline from existing pipeline.