Samples for programming a pipeline (Pipelines)

See samples for programming a pipeline with Bash scripts, such as installing third party packages, and more.

Installing packages with Bash

You can install third party packages with the Run Bash script node. In the following example, you will learn to install a package (PSQL) to connect to an external database.

Note:

The image RH UBI 9 is used in Pipelines 4.8.3 or later, while RH UBI 8 is used in Pipelines 4.8.2 or earlier. You need to extract the psql package from Red-hat Package Manager.

  1. Download postgresql*.rpm and libpq*.rpm from the proper repository for RH UBI:

    dnf download postgresql
dnf download libpq

  2. Extract the rpm file from postgresql:

    rpm2cpio postgresql-*.....*_64.rpm | cpio -idmv

  3. Verify that the psql file exists in extracted files:

    find . -name psql -print

    Your output should be: ./usr/bin/psql

  4. Extract the rpm file from libpq:

    rpm2cpio libpq-13.11-1.el8.x86_64.rpm | cpio -idmv

  5. Verify that the libpq.so.private13-5 file exists in extracted files.

    1. Run the following code:

      find . -name libpq.so.private13-5* -print

      Your output should be:
      ./usr/lib64/libpq.so.private13-5.13
      ./usr/lib64/libpq.so.private13-5

    2. Rename ./usr/lib64/libpq.so.private13-5.13 to ./usr/lib64/libpq.so.private13-5 as it is a link to the latter.

  6. Verify that the plibpq.so.5 file exists in extracted files.

    1. Run the following code:

      find . -name libpq.so.5* -print

      Your output should be:
      ./usr/lib64/libpq.so.5
      ./usr/lib64/libpq.so.5.13

    2. Rename libpq.so.5.13 to libpq.so.5 as it is a link to the latter.

  7. Copy files to the storage connected to the project. For example, if you have a project named TEST-PSQL with storage connection FVT to volume: cpd-instnace::FVT:

    1. In the file browser, create a new folder psql and upload files psql and libpq.so.5.

    2. Upload test.sql file with the following content:

      \l 
\q

      {: .codeblock} If you do not have storage or storage connection, please see:

  8. In the Run Bash script node, add the following

    • environment variables:
      -- LD_LIBRARY_PATH = /mnts/fvt/psql
-- USER 
-- DB
-- PORT 
-- HOST_IP
-- PGPASSWORD
    • script code:
      chmod a+x ${LD_LIBRARY_PATH}/psql 
${LD_LIBRARY_PATH}/psql -U ${USER} -h ${HOST_IP} -p ${PORT}  -d ${DB} -a -f ${LD_LIBRARY_PATH}/test.sql

  9. Run the pipeline. Your output should be the following:

    \l 
                                List of databases
Name    |  Owner   | Encoding |   Collate   |    Ctype    |   Access privileges   
-----------+----------+----------+-------------+-------------+-----------------------
postgres  | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | 
template0 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +
        |          |          |             |             | postgres=CTc/postgres
template1 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +
        |          |          |             |             | postgres=CTc/postgres
(3 rows)

\q

Using CPDCTL

Refer to samples to get started with some end-to-end example code, including how to deploy and monitor pipelines.

Parent topic: Programming IBM Watson Pipelines