Getting started with MLDL frameworks

After your installation is complete, it is time to get started with the machine learning and deep learning (MLDL) frameworks.

General setup

Most of the PowerAI packages install outside the normal system search paths (to /opt/DL/...), so each framework package provides a shell script to simplify environmental setup (for example, PATH, LD_LIBRARY_PATH, PYTHONPATH).

Consider updating your shell rc file (such as .bashrc) to source the wanted setup scripts. For example, run the following command:

source /opt/DL/<framework>/bin/<framework>-activate

Each framework also provides a test script to verify some of its functions. These test scripts include tests and examples that are sourced from the various communities. Note that some of the included tests rely on data sets (for example, MNIST) that are available in the community and are downloaded at run time.

Note: Access and availability to this data is subject to the community and can change at any time.

To run the test script for a particular framework, run:

<framework>-test

Note about dependencies

A number of the PowerAI frameworks (Caffe-BVLC, Caffe-IBM, TensorFlow, TensorBoard, and PyTorch) have their dependencies that are satisfied through Anaconda packages. These dependencies are validated by the <framework>-activate script to ensure that they are installed and, if not, the script fails.

For these frameworks, the /opt/DL/<framework>/bin/install_dependencies script must be run before activation to install the required packages.

For example:

source /opt/DL/tensorflow/bin/tensorflow-activate

Missing dependencies ['backports.weakref', 'mock', 'protobuf']
Run "/opt/DL/tensorflow/bin/install_dependencies" to resolve this problem.

/opt/DL/tensorflow/bin/install_dependencies

Fetching package metadata ...........
Solving package specifications: .

Package plan for installation in environment /home/rhel/anaconda2:

The following NEW packages will be INSTALLED:

    backports.weakref: 1.0rc1-py27_0
    libprotobuf:       3.5.2-hdbcaa40_0
    mock:              2.0.0-py27_0
    pbr:               1.10.0-py27_0
    protobuf:          3.5.2-py27hf484d3e_1

Proceed ([y]/n)? y

libprotobuf-3. 100% |###############################| Time: 0:00:02   2.04 MB/s
backports.weak 100% |###############################| Time: 0:00:00  12.83 MB/s
protobuf-3.5.2 100% |###############################| Time: 0:00:00   2.20 MB/s
pbr-1.10.0-py2 100% |###############################| Time: 0:00:00   3.35 MB/s
mock-2.0.0-py2 100% |###############################| Time: 0:00:00   3.26 MB/s

source /opt/DL/tensorflow/bin/tensorflow-activate

Note: As of PowerAI 1.5.4, BVLC Caffe and IBM enhanced Caffe support execution through use of Anaconda packages, consistent with other PowerAI frameworks such as PyTorch and Tensorflow.