Samples for programming a pipeline (Pipelines)

See samples for programming a pipeline with Bash scripts, such as installing third party packages, and more.

Using CPDCTL

Refer to samples to get started with some end-to-end example code, including how to deploy and monitor pipelines.

Programming with Bash

There are various development tools you can use by running Bash scripts in pipelines.

Using Bash scripts for utility functions

The following are scripts you can run in the Run Bash script node. You must have DataStage installed and have created a persistent storage volume for your Pipelines as the scripts are installed to ds-storage persistent volume. See Storage and data access for IBM Orchestration Pipelines.

DSAddEnvVar.sh

Add an environment key value pair to a runtime environment in your current project.

Example

DSAddEnvVar.sh -n <env_name> --env=key1 --value=value1

DSDeleteEnvVar.sh

Delete an environment variable from a runtime environment.

Example

DSDeleteEnvVar.sh -n <env_name> --env key1

DSGetIdForJob.sh

Retrieve the job id for a job.

Example

DSGetIdForJob.sh <job_name>

DSGetJobInfo.sh

Retrieve the job information for a job run.

Example

DSGetJobInfo.sh <DSJ.JOBSTATUS|DSJ.JOBNAME|DSJ.PARAMLIST|DSJ.STAGELIST|DSJ.USERSTATUS|DSJ.JOBDESC|DSJ.JOBELAPSED> --job-id=<id of job> --run-id=< id of job run>

DSGetLinkInfo.sh

Retrieve the URL information of a flow.

Example

DSGetLinkInfo.sh <link_name> <link_metric_name> --job-id=<id of job>--run-id=< id of job run>

link_metric_name is "DSJ.LINKNAME" or "DSJ.LINKROWCOUNT".

DSGetParamInfo.sh

Get the job parameter value from a DataStage job run.

Example

DSGetParamInfo.sh <param_name> --job-id= --run-id=<id of job run>

DSGetStageInfo.sh

Get the stage info of a DataStage job run.

Example

rows=DSGetStageInfo.sh "DSJ.STAGEINROWNUM" "Peek_1" --job-id=<id of job> --run-id=< id of job run>
links=DSGetStageInfo.sh "DSJ.LINKLIST" "Peek_1" --job-id=<id of job>--run-id=< id of job run>

DSGetUserStatus.sh

Get the status of user from a DataStage job run.

Example

DSGetUserStatus.sh --job-id= --run-id=< id of job run>

DSGetVersionInfo.sh

Get the DataStage service version info.

Example

DSGetVersionInfo.sh DSJ.VER.DS

DSJobNameFromJobId.sh

Get a job name by job id.

Example

DSJobNameFromJobId.sh --job-id=

DSListEnvVars.sh

List all environment variable for a runtime environment.

Example

DSListEnvVars.sh -n

DSRunJob.sh

Run a job by calling the job id.

Example

DSRunJob.sh --job-id=

DSSetParam.sh

Update job parameter value for a job.

Example

DSSetParam.sh <param_name> <param_value>

DSStopJob.sh

Stop a job run by calling its ID.

Example

DSStopJob.sh --job-id= --run-id=< id of job run>

DSTranslateCode.sh

Translate numeric status code to status string message.

Example

DSTranslateCode.sh <numeric_code>

DSWaitForJob.sh

Wait for job run to complete with timeout.

Example

DSWaitForJob.sh "${jobid1},${jobid2}" "${runid1},${runid2}" "600"

UtilityRunJob.sh

Run a DataStage job with run options.

Example

UtilityRunJob.sh <param1=value1|param2=value2>

UtilityAbortToLog.sh

Log error message and exit the execution while returning value 1.

Example

UtilityAbortToLog.sh "error happened"

UtilityMessageToLog.sh

Log info message and exit the execution with return value 0.

Example

UtilityAbortToLog.sh "info message"

UtilityWarningToLog.sh

Log warning message and exit the execution while returning value 0.

Example

UtilityAbortToLog.sh "warning message"

Installing packages with Bash

You can install third party packages with the Run Bash script node. In the following example, you will learn to install a package (PSQL) to connect to an external database.

Note:

The image RH UBI 9 is used in Pipelines 4.8.3 or later, while RH UBI 8 is used in Pipelines 4.8.2 or earlier. You need to extract the psql package from Red-hat Package Manager.

  1. Download postgresql*.rpm and libpq*.rpm from the proper repository for RH UBI:

    dnf download postgresql
dnf download libpq

  2. Extract the rpm file from postgresql:

    rpm2cpio postgresql-*.....*_64.rpm | cpio -idmv

  3. Verify that the psql file exists in extracted files:

    find . -name psql -print

    Your output should be: ./usr/bin/psql

  4. Extract the rpm file from libpq:

    rpm2cpio libpq-13.11-1.el8.x86_64.rpm | cpio -idmv

  5. Verify that the libpq.so.private13-5 file exists in extracted files.

    1. Run the following code:

      find . -name libpq.so.private13-5* -print

      Your output should be:
      ./usr/lib64/libpq.so.private13-5.13
      ./usr/lib64/libpq.so.private13-5

    2. Rename ./usr/lib64/libpq.so.private13-5.13 to ./usr/lib64/libpq.so.private13-5 as it is a link to the latter.

  6. Verify that the plibpq.so.5 file exists in extracted files.

    1. Run the following code:

      find . -name libpq.so.5* -print

      Your output should be:
      ./usr/lib64/libpq.so.5
      ./usr/lib64/libpq.so.5.13

    2. Rename libpq.so.5.13 to libpq.so.5 as it is a link to the latter.

  7. Copy files to the storage connected to the project. For example, if you have a project named TEST-PSQL with storage connection FVT to volume: cpd-instnace::FVT:

    1. In the file browser, create a new folder psql and upload files psql and libpq.so.5.

    2. Upload test.sql file with the following content:

      \l 
\q

    If you do not have storage or storage connection, please see:
    - Managing storage volumes
    - Storage volume connection
    - Storage volume connection

  8. In the Run Bash script node, add the following

    • environment variables:
      -- LD_LIBRARY_PATH = /mnts/fvt/psql
-- USER 
-- DB
-- PORT 
-- HOST_IP
-- PGPASSWORD
    • script code:
      chmod a+x ${LD_LIBRARY_PATH}/psql 
${LD_LIBRARY_PATH}/psql -U ${USER} -h ${HOST_IP} -p ${PORT}  -d ${DB} -a -f ${LD_LIBRARY_PATH}/test.sql

  9. Run the pipeline. Your output should be the following:

    \l 
                                List of databases
Name    |  Owner   | Encoding |   Collate   |    Ctype    |   Access privileges   
-----------+----------+----------+-------------+-------------+-----------------------
postgres  | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | 
template0 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +
        |          |          |             |             | postgres=CTc/postgres
template1 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +
        |          |          |             |             | postgres=CTc/postgres
(3 rows)

\q

Using API samples

See some samples for using Pipelines API for cases such as:

  • List all pipelines in a project.
  • Finds a specific pipeline by ID.
  • Deletes a pipeline and its pipeline versions.
  • Lists all pipeline versions of a given pipeline.
  • Gets a pipeline version by pipeline version ID.
  • Deletes a pipeline version by pipeline version ID.
  • Upload a pipeline file and create a new pipeline
  • Upload a pipeline file and create a new pipeline version
  • Commit a pre-existing volatile default version to finished state.
  • Create a new pipeline from existing pipeline.

Parent topic: Programming IBM Orchestration Pipelines