Important:

IBM Cloud Pak® for Data Version 4.7 will reach end of support (EOS) on 31 July, 2025. For more information, see the Discontinuance of service announcement for IBM Cloud Pak for Data Version 4.X.

Upgrade to IBM Software Hub Version 5.1 before IBM Cloud Pak for Data Version 4.7 reaches end of support. For more information, see Upgrading IBM Software Hub in the IBM Software Hub Version 5.1 documentation.

Capabilities on Linux on IBM Z and IBM LinuxONE

When Cloud Pak for Data is installed on the IBM Z and IBM LinuxONE platforms, Watson Studio and Watson Machine Learning users can create, train, and deploy machine learning models.

Note:

The capabilities described for Watson Studio are distinct from a standalone solution called IBM Watson Machine Learning for z/OS (WML for z/OS or WMLz). Watson Machine Learning for z/OS is an enterprise machine learning solution that runs on IBM Z and Red Hat OpenShift Container Platform (OCP). You can run WML for z/OS as a standalone solution or infuse it into your enterprise AI capability as a scalable platform. For details, see Overview of WML for z/OS.

Review the following for Cloud Pak for Data installed on IBM Z or IBM LinuxONE platforms:

Capabilities

Using Watson Studio, Watson Machine Learning and Analytics Engine powered by Apache Spark services, Linux on IBM Z and LinuxONE users can do the following:

  • Use the Cloud Pak for Data resource specification injection (RSI) feature to apply patches directly to pods that are associated with Kubernetes deployments, StatefulSets, replicasets, replica controllers, jobs, and cronjobs.

  • Interactively discover, cleanse, and transform your data using Data Refinery.

  • Create projects to organize assets such as data, environments, and notebooks.

  • Use data that is uploaded in a project, or connect to remote data.

  • Build and train machine learning models, or code Python functions or scripts using Jupyter notebooks or Jupyter Lab.

  • Train predictive models using AutoAI, a graphical tool in Watson Studio that analyses your data and uses data algorithms, transformations, and parameter settings to create the best predictive model. AutoAI displays various potential models as model candidate pipelines and ranks them on a leaderboard for you to review. Save a pipeline as a model asset that you can deploy.

  • Create deployment spaces for organizing the resources to deploy models and run Python functions and scripts.

  • Create jobs to run notebooks or deployments.

  • Evaluate deployments using Watson OpenScale.

  • Create Python 3.9 runtimes in Watson Studio.

  • Create and run Spark applications on an Analytics Engine powered by Apache Spark instance on Cloud Pak for Data. The supported Spark and language versions include:

    • Spark 3.3 & Python 3.9
    • Spark 3.2 & Python 3.9 (Spark applications with Spark 3.2 have been deprecated. Choose a Spark application with Spark 3.3 instead.)
    • Spark 3.3 & Python 3.10

    To run Spark jobs, see Getting started with Spark applications.

  • Leverage IBM z16 on-chip AI accelerator on IBM Z and IBM LinuxONE:

    • Support for SnapML 1.9 and higher
    • Support for Tensorflow 2.7.2 and higher (Tech preview)

  • Backup and restore is supported for Cloud Pak for Data on Linux on IBM Z and LinuxONE.

Supported frameworks

Build assets using these popular machine learning frameworks and software.

Python 3.9 runtimes (runtime-22.1-py3.9) with these frameworks:

  • scikit-learn 1.0.2
  • xgboost 1.5.2
  • snapml 1.13.1
  • tensorflow 2.7.2
  • pytorch 1.10.2
  • Opencv, torchvision

Python 3.9 runtimes (runtime_22.1-py3.9-nnpa) with IBM z16 on-chip AI accelerator capabilities on IBM Z and IBM LinuxONE:

  • snapml 1.13.1

  • tensorflow 2.7.2 (Tech preview)

  • runtime_22.1-py3.9-nnpa also supports these frameworks:

    • scikit-learn 1.0.2
    • xgboost 1.5.2
    • pytorch 1.10.2
    • Opencv, torchvision

Python 3.10 runtimes (runtime-22.2-py3.10) with these frameworks:

  • scikit-learn 1.1.1
  • xgboost 1.6.2
  • snapml 1.11.1
  • tensorflow 2.9.3
  • pytorch 1.12.1
  • Opencv, torchvision, and lightgbm
  • AutoAI 0.1

Framework notes:

  • Snap ML 1.11.1 introduces the Graph Feature Preprocessor, a Scikit-learn compatible preprocessor that enables scalable, real-time feature extraction from graph-structured data. The Graph Feature Preprocessor can generate a very rich set of edge features by analyzing the graph topology and extracting meaningful substructures. Such additional features can significantly increase the predictive power of machine learning models in practical settings, such as when monitoring suspicious transactions and account activities for money laundering.
  • This release does not support rt23_1 software specifications for IBM Z and IBM LinuxONE. For details, see Supported frameworks

Supported storage

For clusters that are installed on Linux on IBM Z and LinuxONE, these storage solutions are supported:

  • Openshift Container Storage (OCS) and OpenShift Container Platform version 4.9 or later, or Openshift Data Foundation (ODF). Requires OpenShift Container Platform version 4.8 or later.
  • Network File System (NFS)
  • IBM Storage Scale Container Native

Restrictions for Linux on IBM Z and LinuxONE users

When Cloud Pak for Data is installed on the Linux on IBM Z and LinuxONE platforms, Watson Studio and Watson Machine Learning users will not be able to use, run, or deploy the following types of assets:

  • Assets trained using RStudio, such as RShiny apps or assets based on the R framework
  • Assets based on these runtimes: Spark, Python 3.7 or ONNX 1.8.1
  • Deep Learning assets built with deprecated or discontinued versions of frameworks; for example, assets built with TensorFlow 2.4 or PyTorch 1.7 frameworks

Limitations:

  • Attempting to use, train, or deploy unsupported assets on Cloud Pak for Data running on an Linux on IBM Z or LinuxONE platform will fail with an error.
  • On an upgraded Cloud Pak for Data cluster where the release version is 4.7.0 after the upgrade and the Jupyter notebook runtime is "Runtime 22.1 on Python 3.9", the highest supported value for version provided in the wml_credential object for authentication using the Watson Machine Learning python client is 4.5. To use the latest version for the release, upgrade the Runtime 22.1 on Python 3.9.
  • Backup and restore is supported for Cloud Pak for Data on Linux on IBM Z and LinuxONE. However, currently cpd-cli oadp commands must be run from an x86_64 machine.
  • Offline volume backup is not supported for IBM z and LinuxONE. This feature is deprecated and will be removed in a future release.
  • The cpdbr-agent is not supported on Linux on IBM Z and IBM LinuxONE.
  • With the default runtimes, models trained on other platforms and deployed on Linux on IBM Z and Linux ONE might not work as expected. A potential solution is to deploy the model on a custom Python runtime.
  • ORC (Optimized Row Columnar)input format is not supported for evaluating batch deployments using Watson OpenScale.
  • Parquet file data format is not supported for as input for an AutoAI time series experiment.
  • Batch processing in Watson OpenScale using IBM Analytics Engine with OpenShift Data Foundation (ODF) storage class storage is not supported. Use NFS storage instead.
  • Running generated code to add data from a connection on IBM Z can cause kernel failure.

Using an AutoAI generated notebook outside of Cloud Pak for Data

To use an AutoAI generated notebook outside Cloud Pak for Data, use these steps to install the required packages for Linux on zSystem.

  1. Install Conda on the zLinux operating system.

  2. Prepare the pyarrow 8.0.0 Python package installation dependencies with Open-CE.

    Note: This package is not available on the zLinux conda channel.

    1. Create a conda environment.

      Note:

      This example uses /opt as the working directory for the local conda channel.

       

      cd /opt
conda create -y -n pyarrow-env
conda activate pyarrow-env
conda install -y conda-build
conda install -y -c open-ce open-ce-builder python=3.10

    2. Create a Git clone of the Open-CE project and check out the release to the open-ce-r1.7 branch.

      git clone -b open-ce-r1.7 https://github.com/open-ce/open-ce.git

    3. Install a patch package with the following command:

      yum install patch

    4. Navigate to the Open-CE project and build the pyarrow dependencies.

      cd open-ce
open-ce build env --python_versions 3.10 --build_types cpu envs/arrow-env.yaml

    5. Check the condabuild folder to find the pyarrow dependencies

      ls -1 condabuild/
channeldata.json
index.html
linux-s390x
noarch
opence-conda-env-py3.10-cpu-openmpi.yaml
**************************************************
ls -1 condabuild/linux-s390x/
abseil-cpp-20210324.2-he499f12_0.conda
arrow-cpp-8.0.0-py310h63c5c10_2_cpu.conda
arrow-cpp-proc-8.0.0-cpu.conda
boost-cpp-1.65.1-hebff1d6_4.conda
current_repodata.json
current_repodata.json.bz2
gflags-2.2.2-heb72281_0.conda
glog-0.5.0-he499f12_0.conda
grpc-cpp-1.41.0-hae1bb83_pb3.19_4.conda
index.html
libboost-1.65.1-h896dd0f_4.conda
libevent-2.1.10-h8df5d65_2.conda
libthrift-0.13.0-h333d347_6.conda
libutf8proc-2.6.1-h63165b7_0.conda
orc-1.7.4-hb4085e4_1.conda
pyarrow-8.0.0-py310he9dcaf6_2_cpu.conda
rapidjson-1.1.0-h4cc523a_0.conda
re2-2020.11.01-hcac3278_0.conda
repodata_from_packages.json
repodata_from_packages.json.bz2
repodata.json
repodata.json.bz2
thrift-compiler-0.13.0-h333d347_6.conda
thrift-cpp-0.13.0-6.conda

    6. Deactivate the pyarrow-env conda virtual environment and navigate back to the /opt directory.

  3. Prepare the Tensorflow Python package installation dependencies with Open-CE.

    Note: This package is not available on the zLinux conda channel.
    1. Create a conda environment.
      cd /opt
conda create -y -n tensorflow-env
conda activate tensorflow-env
conda install -y conda-build
conda install -y -c open-ce open-ce-builder python=3.10
    2. Navigate to the Open-CE project and build the Tensorflow dependencies.
      cd /opt/open-ce
open-ce build env --python_versions 3.10 --build_types cpu envs/tensorflow-env.yaml
    3. Check the condabuild folder to find the tensorflow-cpu dependencies.
      ls -1 condabuild/
channeldata.json
icons
index.html
linux-s390x
noarch
opence-conda-env-py3.10-cpu-openmpi.yaml
******************************************************
ls -1 condabuild/linux-s390x/
bazel-5.1.1-h447df78_2.conda
clang-14.0.6-0.conda
clang-14-14.0.6-default_hc034eec_0.conda
clangdev-14.0.6-default_hc034eec_0.conda
clang-format-14.0.6-default_hc034eec_0.conda
clang-format-14-14.0.6-default_hc034eec_0.conda
clang-tools-14.0.6-default_hc034eec_0.conda
clangxx-14.0.6-default_h050e89a_0.conda
current_repodata.json
current_repodata.json.bz2
dm-tree-0.1.5-py310h17e39e8_5.conda
flatbuffers-2.0.0-he499f12_0.conda
googleapis-common-protos-1.52.0-py310hb7eb58e_0.conda
googleapis-common-protos-grpc-1.52.0-h785a63c_0.conda
h5py-3.6.0-py310ha7896db_0.conda
index.html
libclang13-14.0.6-default_h99f1993_0.conda
libclang-14.0.6-default_hc034eec_0.conda
libclang-cpp-14.0.6-default_hc034eec_0.conda
libclang-cpp14-14.0.6-default_hc034eec_0.conda
libtensorflow-2.9.3-hd9d4759_cpu_pb3.19_4.conda
promise-2.3-py310h6d39d64_0.conda
pybind11-2.9.2-py310he3a6269_0.conda
pybind11-global-2.9.2-py310h0c46cd6_0.conda
repodata_from_packages.json
repodata_from_packages.json.bz2
repodata.json
repodata.json.bz2
scipy-1.8.1-py310h5fdd5b3_0.conda
tensorflow-addons-0.17.0-py310h88a663b_2_cpu.conda
tensorflow-addons-proc-0.17.0-cpu.conda
tensorflow-base-2.9.3-cpu_py310_pb3.19_4.conda
tensorflow-cpu-2.9.3-py310_1.conda
tensorflow-model-optimization-0.7.3-py310_2.conda
tensorflow-probability-0.17.0-py310_3.conda
tensorflow_select-1.0-cpu_2.conda
tensorflow-text-2.9.0-hf137b60_py310_pb3.19_3.conda

  4. Run the conda index command on the condabuild folder.

    conda index condabuild

  5. Add the following lines in the ~/.condarc file to make use of the condabuild folder as a local conda channel.

    channels:
- /opt/open-ce/condabuild
- defaults

  6. Check the Conda channel to find the pyarrow and tensorflow-cpu installation package.

    conda search pyarrow
conda search tensorflow-cpu

Loading channels: done
# Name             Version       Build                 Channel             
pyarrow            8.0.0         py310he9dcaf6_2_cpu   condabuild 
(tensorflow) [root@recorder1 open-ce]# conda search tensorflow-cpu
Loading channels: done
# Name              Version           Build           Channel             
tensorflow-cpu      2.9.3             py310_1         condabuild          
(tensorflow) [root@recorder1 open-ce]#

  7. Deactivate the tensorflow-env conda virtual environment and navigate back to the /opt directory.

    conda deactivate
cd /opt

  8. Create a Conda virtual environment for autoai installation and activate it.

    conda create -y -n autoai python=3.10
conda activate autoai

  9. Create requirements.linux-s390x.txt file and add this package to it:

    cython
matplotlib
pandas>=0.24.2,<1.6
numpy>=1.20.3,<1.24
packaging
psutil
importlib-metadata
coverage
urllib3<2
xgboost>=1.5.1,<1.7
typing_extensions
pyarrow==8.0.0
scikit-learn>=1.0.2,<=1.1.1
tensorflow-cpu>=2.7.0,<2.10
joblib>=0.11
statsmodels<0.14
dill>=0.3.1.1
networkx>=2.5.1
py4j>=0.10.9,<0.10.10
jpype1>=1.3.0
simplejson==3.17.6
PyWavelets
pytz
traitlets<6

  10. Install requirements.linux-s390x.txt file.

    conda install -y --file requirements.linux-s390x.txt

  11. Create requirements.linux-s390x.pypi.txt file and add these packages to it:

    gensim==4.1.2
pytest
pytest-cov
pep8
pytest-pep8
ibm-watson-machine-learning
mlxtend>=0.17.0
lale>=0.6.8,<0.8

  12. Install requirements.linux-s390x.pypi.txt file.

    pip install -r requirements.linux-s390x.pypi.txt

  13. Install autoai-libs and autoai-ts-libs:

    pip install autoai-libs==1.15.2 --no-deps
pip install autoai-ts-libs==2.0.13 --no-deps

Learn more

Deploying and managing models and functions

Parent topic: Projects