Installing the MLDL frameworks

Find instructions for installing the machine learning and deep learning (MLDL) frameworks.

Setting up the software repository

The WML CE MLDL packages are distributed as conda packages in an online conda repository. conda must be configured to give priority to installing packages from this channel.

Add the WML CE channel to the conda configuration by running the following command:

conda config --prepend channels \
https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/

Creating conda environments (recommended)

With conda, you can create environments that have different versions of Python or packages installed in them. Conda environments are optional but recommended. If not used, packages are installed in the default environment called base, which often has a higher risk of containing conflicting packages or dependencies. Switching between environments is called activating the environment.

The syntax to create and activate a conda environment is:

conda create --name <environment name> python=<python version>
conda activate <environment name>
Note: You must specify the Python version when creating a new environment. If you do not specify the version, Python 3.7.4 is installed.

The only valid Python versions with WML CE are Python 3.6 and 3.7.

For example, to create an environment named wmlce_env with Python 3.6:

conda create --name wmlce_env python=3.6
conda activate wmlce_env

For more information on what you can do with conda environment see https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html.

Note: WML CE should be run as a non-privileged user and not root. The WML CE components are designed to be usable by normal users, and the pre-installed docker images provide a non-root user by default. Some of the WML CE components will give warnings or will fail when run as root.

Installing all frameworks at the same time

All the MLDL frameworks except RAPIDS packages can be installed at the same time by using the powerai meta-package. All the RAPIDS packages can be installed using the powerai-rapids meta-package.

  • To install powerai GPU packages in the conda environment, run:
    conda install powerai
  • To install powerai CPU packages in the conda environment, run:
    conda install powerai-cpu
  • To install RAPIDS packages, run:
    conda install powerai-rapids
Note: The Python 3.6 or 3.7 version of the package installed is determined by the Python version of the active environment.

Installing frameworks individually

You can install the MLDL frameworks individually. The framework packages include the following versions.

Table 1. Framework packages
Package Description Version Available on ppc64le Available on x86_64
caffe IBM-optimized version of Berkeley Vision and Learning Center Caffe 1.0.0 X X
caffe-cpu IBM-optimized Caffe CPU-only package 1.0.0 X X
cudf RAPIDS cuDF 0.9.0 X  
cuml RAPIDS cuML 0.9.1 X  
ddl Distributed Deep Learning 1.5.0 X X
pai4sk WML CE Snap ML 1.5.0 X  
pytorch PyTorch 1.2.0 X X
pytorch-cpu PyTorch 1.2.0 X X
snapml-spark WML CE Snap ML Spark 1.4.0 X  
tensorflow TensorFlow CPU-only package 1.15 X X
tensorflow-gpu TensorFlow with GPU support 1.15 X X
tensorflow-serving TensorFlow Serving 1.15 X X
py-xgboost-gpu xgboost with GPU support 0.90 X X
py-xgboost-cpu xgboost CPU-only package 0.90 X X

With the conda environment activated, run the following command:

conda install <package name>

Accepting the WML CE license agreement

During the conda install, the packages are downloaded from the internet and after downloading, the license agreement is presented. Read the license agreement and accept the terms and conditions to complete the install. If you decline the license agreement the packages are not installed.

After you finish reading the license agreement, future installations can be automated to silently accept the license agreement by running the following command before running the conda install command:

export IBM_POWERAI_LICENSE_ACCEPT=yes

The license accept has to be done only once on a per user basis.

Helping the conda installer: strict channel priority and the powerai-release package

The conda installer uses a set of rules to determine which packages to install. Channel priorities and package versions are weighted heavily, but the installer also considers factors such as the number of packages that would need to be installed, whether any packages would need to be upgraded or removed, and so on.

The conda installer will sometimes come up with a surprising installation solution. It may prefer to install:
  • Packages from Anaconda channels over the WML CE channel in spite of channel priorities.
  • Packages from an older release of WML CE in spite of newer versions being available.

You can guide the conda installer to ensure that it chooses the desired WML CE package using the strict channel priority option and the powerai-release meta-package.

Strict channel priority

The strict channel priority option forces the conda installer to give additional weight to the priority of channels defined in the configuration. It is useful in cases where the conda installer is preferring packages from lower-priority channels. The simplest use is just to add --strict-channel-priority to the install command:

(my-wmlce-env) $ conda install --strict-channel-priority tensorflow
You can check the priority of the channels in the configuration by running the following:
(my-wmlce-env) $ conda config --show
...
channel_priority: flexible
channels:
  - https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/
  - defaults
...

You could permanently change the channel priority setting to strict:

(my-wmlce-env) $ conda config --set channel_priority strict

powerai-release meta-package

The powerai-release meta-package can be used to specify the WML CE release you want to install from. It is useful when the installer prefers packages from an earlier release, or if you intentionally want to install packages from an older release.

Examples:
(my-wmlce-env) $ conda install pytorch powerai-release=1.6.2
(my-wmlce-env) $ conda install pytorch powerai-release=1.6.1

The --strict-channel-priority option can be used with powerai-release for greater control:

(my-wmlce-env) $ conda install --strict-channel-priority pytorch powerai-release=1.6.2

For more information on the conda installer's rules, see Running the solver at: Understanding and Improving Conda’s performance.

Additional conda channels

The main WML CE conda channel is described above. That channel includes the formal, supported WML CE releases.

Additional conda channels are available to complement the main channel. Packages in these channels are not formally supported. Both of these channels are optional. WML CE will install and run fine without either. They can also be used independently of each other (Supplementary does not need Early Access or vice versa). Use them if you want the packages they provide and do not need formal support.

  • The WML CE Supplementary channel is available at: https://anaconda.org/powerai/.

    This channel includes packages that are not part of WML CE, but which may be useful to WML CE users. The packages are built from recipes in the WML CE GitHub repository: https://github.com/ibm/powerai.

    Problem reports and recipe contributions from the community are welcome. More information about the Supplementary channel can be found in the PowerAI Supplementary Channel README.

  • The WML CE Early Access channel is available at: https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda-early-access/.

    This channel is updated occasionally with latest versions of various packages included in WML CE. The purpose of the channel is to make new versions of frameworks available in advance of formal WML CE releases. Packages published in the Early Access channel may not exactly match a later WML CE release. For example, package and prerequisite versions may differ.

    Packages in the Early Access channel might depend on packages in the main channel, so both channels might be needed in the conda config.
    Note: Easy upgrade between packages in the main and Early Access channels is not guaranteed. Using a separate conda environment for Early Access packages is recommended.
When using these optional channels we recommend the following priority:
  • Early Access channel (optional, higher priority)
  • WML CE main channel
  • Supplementary channel (optional, lower priority)