Run with a serverless container runtime offering

Running on a Serverless platform offers nearly effortless app setup, at the expense of losing fine tune control over your app's configuration. In the context of the Watson NLP Runtime, the most important configuration is where to load model resources from. While some serverless platforms allow configuring access to remote storage, we find it's easier and more reliable to build a new runtime image that contains the pre-trained models you wish to use.

Choose a runtime offering such as IBM Cloud Code Engine, AWS Fargate, or Microsoft Azure Serverless. Then, create a runtime container image that you can deploy to your runtime offering of choice.

Create a runtime container image

The IBM Entitled Registry contains various container images for Watson Runtime. Once you've obtained the entitlement key from the container software library, you can login to the registry with the key, and pull the runtime images to your local machine.

The Watson Runtime on its own doesn't have any models included. However, you can easily build a runtime container image to include one or more pretrained models, which are also stored as container images in the IBM Entitled Registry.

  1. Login to the IBM Entitled Registry

    docker login cp.icr.io --username cp --password <your_entitlement_key>

  2. Download models to a local directory

    mkdir models
REGISTRY=cp.icr.io/cp/ai
export MODELS="watson-nlp_syntax_izumo_lang_en_stock:1.4.1 watson-nlp_syntax_izumo_lang_fr_stock:1.4.1"
for i in $MODELS
do
  image=$REGISTRY/$i
  docker run -it --rm -e ACCEPT_LICENSE=true -v `pwd`/models:/app/models $image
done

  3. Create a Dockerfile using a text editor of your choice

    ARG TAG=1.0
FROM cp.icr.io/cp/ai/watson-nlp-runtime:${TAG}
COPY models /app/models

  4. Build the image

    docker build -t my-watson-nlp-runtime:latest

  5. You can now upload your custom runtime image to a registry of your choice, and start configuring your serverless application.

IBM Cloud Code Engine

Once the runtime image is created, put it on IBM Cloud Container Registry (ICR), so that it can be used for deployment to IBM Cloud Code Engine.

  1. Install the IBM Cloud Command Line Interface

  2. Install the Container Registry plug-in

    ibmcloud plugin install cr

  3. Log in to your IBM Cloud account

    ibmcloud login

    Note: Use ibmcloud login --sso to login if you have a federated ID.

  4. Create a namespace

    You'll need to create a namespace before you can upload your image, and make sure you're targeting the correct ICR region.

    Set a target region for the ibmcloud cr commands.

    ibmcloud cr region-set

    Choose a name for your namespace, specified as ${NAMESPACE}, and create the namespace.

    ibmcloud cr namespace-add ${NAMESPACE}

  5. Login to ICR

    ibmcloud cr login

  6. Upload the runtime image to ICR

    Set ${REGISTRY} to the listed Domain name for the local registry regions (or icr.io for the global registry), and run the following commands.

    # Tag the image
docker tag my-watson-nlp-runtime:latest ${REGISTRY}/${NAMESPACE}/my-watson-nlp-runtime:latest

# Push the image
docker push ${REGISTRY}/${NAMESPACE}/my-watson-nlp-runtime:latest

You can now deploy the uploaded runtime image to IBM Cloud Code Engine. For a complete tutorial, including deployment steps, see Serve Watson NLP Models with IBM Cloud Code Engine on GitHub.

Amazon AWS Fargate

Once the runtime image is created, upload it to Amazon Elastic Container Registry (ECR), so that it can be used for deployment on AWS Fargate.

Each AWS account is provided with a default private registry (aws_account_id.dkr.ecr.region.amazonaws.com).

  1. Login to the default registry

    aws ecr get-login-password | docker login --username AWS --password-stdin ${DEFAULT_REGISTRY}

  2. Create a repository in the default registry

    aws ecr create-repository --repository-name my-watson-nlp-runtime

  3. Upload the image to Amazon ECR

    # Tag the image
docker tag my-watson-nlp-runtime:latest ${DEFAULT_REGISTRY}/my-watson-nlp-runtime:latest

# Push the image
docker push ${DEFAULT_REGISTRY}/my-watson-nlp-runtime:latest

For a complete tutorial, including deployment steps, please see Serve Models on Amazon ECS with AWS Fargate on GitHub.

Microsoft Azure Container Instances

Once the runtime image is created, upload it to Azure Container Registry, so that it can be used for deployment on Azure Container Instances.

Azure Container Registry (ACR) is a private registry service for building, storing, and managing container images and related artifacts.

  1. Login to the ACR registry

    az acr login --name $REGISTRY

  2. Copy the container image(s) to ACR

    REGISTRY1=cp.icr.io/cp/ai
REGISTRY2=${REGISTRY}.azurecr.io
IMAGES="watson-nlp-runtime:1.1.36 watson-nlp_syntax_izumo_lang_en_stock:1.4.1"
for i in $IMAGES
do
  image1=${REGISTRY1}/$i
  image2=${REGISTRY2}/$i
  docker pull $image1
  docker tag $image1 $image2
  docker push $image2
done

For a complete tutorial, including deployment steps, please see Serve Watson NLP Models with Azure Container Instances on GitHub.

Next steps

Once you have your runtime server working, see Accessing client libraries and tools to continue.