Installing custom and open-source Python packages

You can install custom open-source packages into the TLS container, and the Conda environment that Jupyter runs in.

Before you begin, ensure that you have all the legal rights and approvals necessary.

Install into the persistent volume claim (PVC), shared directory (NFS drive), so your packages will survive container restarts. To do this, specify the --target=/opt/ibm/fcai/TLS/external/packages/python3.8/site-packages flag during the pip3 package installations for the pipeline and --target=/opt/ibm/fcai/TLS/external/packages/python3.7/site-packages flag during the pip installations for the Jupyter Notebook usage.

Some packages will have dependencies on more recent versions than what is installed in your TLS container. You may need to first upgrade these packages to get your open-source package working by running pip uninstall and pip install.

Important: Installing dependencies via Jupyter can result in a version incompatibility with already installed TLS dependencies. For more information, see Python environment and packages. When this happens, IBM recommends that you have both - TLS and Jupyter version match.

Use oc exec -it {container_name} bash to run these commands inside the TLS container.

No Internet Access

This section explains how to install custom or open-source Python packages if your Kubernetes Cluster does not have direct internet connectivity. This is typically the case in secure production environments.

You can install directly into the TLS container for TLS pipeline use or for use with the TLS Jupyter Notebook.

Perform the following steps to download your files:
- Search for your package at the URL https://pypi.org/project/{your_desired_package}. This package includes the .whl and .tar files for each package your code requires.
- Click Download Files.
  
  If you can't find a pre-built distribution .whl for your Python package, you can create your own .tar file, or .whl file: https://packaging.python.org/tutorials/packaging-projects/#generating-distribution-archives
  
  Note: If the TLS container doesn’t have all the necessary prerequisites for your specific package, you’ll need to repeat this process for each dependency.

Run the oc cp commands to upload the distribution package to your Kubernetes cluster and install into the TLS container with pip. The following commands are examples of copy commands you can use.

oc cp {download_dir}/shap-0.34.0.tar.gz fcai-ibm-fci-alert-triage-prod-tls-analytics-{container_id}:/opt/ibm/fcai/TLS/external/packages/python3.8/site-packages/shap-0.34.0.tar.gz

oc cp {download_dir}/more_itertools-8.7.0-py3-none-any.whl fcai-ibm-fci-alert-triage-prod-tls-analytics-{container_id}:/opt/ibm/fcai/TLS/external/packages/python3.8/site-packages/more_itertools-8.7.0-py3-none-any.whl

You must ensure that you have all the legal rights and approvals necessary to use open-source packages. Many open-source packages already have either a .tar or .whl file associated with them.

Install into TLS Container for Pipeline Use

Check to see if the system already has your package version.

To see what packages are already on this container, run the following command:
```
pip3 list
```
To see a specific package, run the following command:
```
python3.8 -m pip show $pkg_name_version
```
where $pkg_name_version is in the following format:
```
pkg_name_version = package_name==version
```
For example, $pkg_name_version could be defined as follows:
```
pkg_name_version = more-itertools==8.7.0
```
If you need to uninstall first, run the following command:
```
python3.8 -m pip uninstall -y <package_name>
```
The following example assumes that shap .tar file and more-itertools .whl were uploaded via oc cp.

Extract the .tar file as in the following shap & more-itertools examples. Ensure your packages were uploaded to the correct directory:

ls -lat /opt/ibm/fcai/TLS/external/packages/python3.8/site-packages/shap-0.34.0.tar.gz
ls -lat /opt/ibm/fcai/TLS/external/packages/python3.8/site-packages/more_itertools-8.7.0-py3-none-any.whl

Install with .tar file.

python3.8 -m pip install --no-cache-dir --target=/opt/ibm/fcai/TLS/external/packages/python3.8/site-packages /opt/ibm/fcai/TLS/external/packages/python3.8/site-packages/shap-0.34.0.tar.gz --upgrade

Install with .whl file.

python3.8 -m pip install --no-cache-dir --target=/opt/ibm/fcai/TLS/external/packages/python3.8/site-packages /opt/ibm/fcai/TLS/external/packages/python3.8/site-packages/more_itertools-8.7.0-py3-none-any.whl --upgrade

Test the installation.
```
python3.8 -c "import <package_name>"
```
For example:
```
python3.8 -c "import shap"
```
No errors should appear.

Installing new packages for use in Jupyter Notebooks is very similar. For installing packages for Jupyter, use python3.7 and pip.

Install into TLS Container for Jupyter Notebook Use

Check to see if the system already has your package version.

To see what packages are already on this container, run the following command:
```
pip list
```
To see a specific package, run the following command:
```
python -m pip show $pkg_name_version
```
where $pkg_name_version is in the following format:
```
pkg_name_version = package_name==version
```
For example, $pkg_name_version could be defined as follows:
```
pkg_name_version = more-itertools==8.7.0
```
If you need to uninstall first, run the following command:
```
python -m pip uninstall -y <package_name>
```
The following example assumes shap .tar file and more-itertools .whl were uploaded via oc cp.

Extract the .tar file as in the following shap & more-itertools examples Ensure your packages were uploaded to the correct directory:

ls -lat /opt/ibm/fcai/TLS/external/packages/python3.7/site-packages/shap-0.34.0.tar.gz
ls -lat /opt/ibm/fcai/TLS/external/packages/python3.7/site-packages/more_itertools-8.7.0-py3-none-any.whl

Install with a .tar file.

python -m pip install --no-cache-dir --target=/opt/ibm/fcai/TLS/external/packages/python3.7/site-packages /opt/ibm/fcai/TLS/external/packages/python3.7/site-packages/shap-0.34.0.tar.gz --upgrade

Install with a .whl file.

python -m pip install --no-cache-dir --target=/opt/ibm/fcai/TLS/external/packages/python3.7/site-packages /opt/ibm/fcai/TLS/external/packages/python3.7/site-packages/more_itertools-8.7.0-py3-none-any.whl --upgrade

Test the installation.
```
python -c "import <package_name>"
```
For example:
```
python -c "import shap"
```
No errors should appear.

Internet Access

This section explains how to install custom or open-source Python packages if your Kubernetes Cluster has direct internet connectivity.

Install into TLS Container for Pipeline Use

Check to see if the system already has your package version.

To see what packages are already on this container, run the following command:
```
pip3 list
```
To see a specific package, run the following command:
```
python3.8 -m pip show $pkg_name_version
```
where $pkg_name_version is in the following format:
```
pkg_name_version = package_name==version
```
For example, $pkg_name_version could be defined as follows:
```
pkg_name_version = more-itertools==8.7.0
```
If you need to uninstall first, run the following command:
```
python3.8 -m pip uninstall -y <package_name>
```

Install what you need directly as follows:

python3.8 -m pip install --no-cache-dir --target=/opt/ibm/fcai/TLS/external/packages/python3.8/site-packages $pkg_name_version --upgrade

Install into TLS Container for Jupyter Notebook Use

Check to see if the system already has your package version.

To see what packages are already on this container, run the following command:
```
pip list
```
To see a specific package, run the following command:
```
python -m pip show $pkg_name_version
```
where $pkg_name_version is in the following format:
```
pkg_name_version = package_name==version
```
For example, $pkg_name_version could be defined as follows:
```
pkg_name_version = more-itertools==8.7.0
```
If you need to uninstall first, run the following command:
```
python -m pip uninstall -y <package_name>
```

Install what you need directly using the following command:

python -m pip install --no-cache-dir --target=/opt/ibm/fcai/TLS/external/packages/python3.7/site-packages $pkg_name_version --upgrade

Add a package to the notebook path

When you add packages for the Jupyter Notebook use, you will need to add the installation location to your path.

In your Notebook, you must add this code before any code that needs to import the new package.

import sys
sys.path.insert(0,\"/opt/ibm/fcai/TLS/external/packages/python3.7/site-package\")

After installing or upgrading any Python packages, you'll need to restart your notebook kernel.