Installing NVIDIA NeMo Retriever Library

The Content-Aware Storage (CAS) Document Processor Engine can either use IBM Docling Multimodal or NVIDIA NeMo Retriever Extraction for document extraction.

Note: NVIDIA Ingest (nv-ingest) has been renamed to NeMo Retriever Library.

For more information about installing the GPU operator, see Enabling GPUs in OpenShift.

For more information about installing NVIDIA NeMo Retriever Library, see the NVIDIA documentation.

Verify the PodPidsLimit

To ensure compatibility and stability, you must update your cluster and deployment settings when using NeMo Retriever Library v26.3.0 with CAS v1.1.5 or newer.

  1. To retrieve the current configuration, run the following command:
    oc get KubeletConfig
  2. To increase the pod PID limit, modify the kubeletConfig by adding the following podPidsLimit to the spec section:
    spec:
  kubeletConfig:
    podPidsLimit: 12228
    Note: After updating the configuration, all worker nodes in the cluster get restarted which can cause a temporary disruption.

    For more information on changing the podPidsLimit when using the ROSA environment, see the Red Hat documentation.

Install the NIM Operator

NeMo Retriever Library requires that the NIM Operator is deployed and running. It is required to install the NIM Operator first before running the NeMo Retriever Library Helm chart. The operator can be installed from the OpenShift® marketplace. Search for The NVIDIA NIM Operator for Kubernetes, and install using the default options.

Select NVIDIA Inference Microservices (NIMs)

Before installing NIMs from the Helm chart, select the NIMs that are needed to extract your documents. For example, if your documents include audio files, installation of the audio NIM is required. Each NIM requires one GPU resource, which can either be a full GPU, or a GPU slice using a Multi-Instance GPU (MIG). The audio, reranking, and image captioning NIMs require a full GPU to run properly. By default, the NeMo Retriever Library Helm chart deploys five NIMs as indicated in the following table. For more information about MIG, see Configuring a Multi-Instance GPU (MIG) with CAS.

NIM Usage Parameter in Helm chart Default setting
llama-nemotron-embed-1b-v2 Text embedding nimOperator.embedqa.enabled true
nemotron-page-elements-v3 Object detection within documents nimOperator.page_elements.enabled true
nemotron-graphic-elements-v1 Chart extraction nimOperator.graphic_elements.enabled true
nemotron-table-structure-v1 Table extraction nimOperator.table_structure.enabled true
nemotron-ocr-v1 Optical character recognition (OCR) nimOperator.ocr.enabled true
nemotron-nano-12b-v2-v1 Vision language model (VLM) image captioning .nemotron_nano_12b_v2_vl.enabled false
audio Audio extraction nimOperator.audio.enabled false
llama-nemotron-rerank-1b-v2 Reranker for query accuracy nimOperator.rerankqa.enabled false

Install NeMo Retriever Library

To install NeMo Retriever Library, follow these steps:
  1. Create a new project called nv-ingest:
    oc new-project nv-ingest
  2. In your command line session, set the following bash variables:
    NGC_API_KEY=<api key>
NAMESPACE=nv-ingest
    Tip: To generate an NVIDIA NGC API key, see Generate Your NGC Keys.
  3. Add helm repositories:
    # Nvidia nemo-microservices NGC repository
helm repo add nemo-microservices https://helm.ngc.nvidia.com/nvidia/nemo-microservices --username='$oauthtoken' --password=<NGC_API_KEY>

# Nvidia NIM NGC repository
helm repo add nvidia-nim https://helm.ngc.nvidia.com/nim/nvidia --username='$oauthtoken' --password=<NGC_API_KEY>

# NVIDIA NIM Base repository
helm repo add nim https://helm.ngc.nvidia.com/nim --username='$oauthtoken' --password=<NGC_API_KEY>

# Nvidia NIM baidu NGC repository
helm repo add baidu-nim https://helm.ngc.nvidia.com/nim/baidu --username='$oauthtoken' --password=<NGC_API_KEY>
  4. Deploy NeMo Retriever Library using the helm chart. To select NIMs, see Select NVIDIA Inference Microservices (NIMs).

    Example 1: Deploy NeMo Retriever Library with the five default NIM services deployed

    helm upgrade \
   --install \
   nv-ingest \
   https://helm.ngc.nvidia.com/nvidia/nemo-microservices/charts/nv-ingest-26.3.0.tgz \
   -n ${NAMESPACE} \
   --username '$oauthtoken' \
   --password "${NGC_API_KEY}" \
   --set ngcImagePullSecret.create=true \
   --set ngcImagePullSecret.password="${NGC_API_KEY}" \
   --set ngcApiSecret.create=true \
   --set ngcApiSecret.password="${NGC_API_KEY}" \
   --set image.repository="nvcr.io/nvidia/nemo-microservices/nv-ingest" \
   --set image.tag="26.3.0" \
   --set milvusDeployed=false

    Example 2: Deploy NeMo Retriever Library with the default NIM services (same as Example 1) along with the audio Riva NIM

    helm upgrade \
   --install \
   nv-ingest \
   https://helm.ngc.nvidia.com/nvidia/nemo-microservices/charts/nv-ingest-26.3.0.tgz \
   -n ${NAMESPACE} \
   --username '$oauthtoken' \
   --password "${NGC_API_KEY}" \
   --set ngcImagePullSecret.create=true \
   --set ngcImagePullSecret.password="${NGC_API_KEY}" \
   --set ngcApiSecret.create=true \
   --set ngcApiSecret.password="${NGC_API_KEY}" \
   --set nimOperator.audio.enabled=true \
   --set image.repository="nvcr.io/nvidia/nemo-microservices/nv-ingest" \
   --set image.tag="26.3.0" \
   --set milvusDeployed=false

    Example 3: Deploy NeMo Retriever Library with the default NIM services (same as example 1) along with the audio Riva NIM and the reranking NIM

    helm upgrade \
   --install \
   nv-ingest \
   https://helm.ngc.nvidia.com/nvidia/nemo-microservices/charts/nv-ingest-26.3.0.tgz \
   -n ${NAMESPACE} \
   --username '$oauthtoken' \
   --password "${NGC_API_KEY}" \
   --set ngcImagePullSecret.create=true \
   --set ngcImagePullSecret.password="${NGC_API_KEY}" \
   --set ngcApiSecret.create=true \
   --set ngcApiSecret.password="${NGC_API_KEY}" \
   --set nimOperator.audio.enabled=true \
   --set nimOperator.rerankqa.enabled=true \
   --set image.repository="nvcr.io/nvidia/nemo-microservices/nv-ingest" \
   --set image.tag="26.3.0" \
   --set milvusDeployed=false

    Example 4: Minimum configuration

    For a minimum configuration installation that extracts only text from files (text or PDF), the text embedding NIM is required. However, the other NIM services can be disabled.

    helm upgrade \
   --install \
   nv-ingest \
   https://helm.ngc.nvidia.com/nvidia/nemo-microservices/charts/nv-ingest-26.3.0.tgz \
   -n ${NAMESPACE} \
   --username '$oauthtoken' \
   --password "${NGC_API_KEY}" \
   --set ngcImagePullSecret.create=true \
   --set ngcImagePullSecret.password="${NGC_API_KEY}" \
   --set ngcApiSecret.create=true \
   --set ngcApiSecret.password="${NGC_API_KEY}" \
   --set nimOperator.graphic_elements.enabled=false \
   --set nimOperator.page_elements.enabled=false \
   --set nimOperator.table_structure.enabled=false \
   --set nimOperator.ocr.enabled=false \
   --set image.repository="nvcr.io/nvidia/nemo-microservices/nv-ingest" \
   --set image.tag="26.3.0" \
   --set milvusDeployed=false
  5. Check the PVC sizes created for the NIMCache:
    oc get pvc -n nv-ingest
    Note: You are recommended to increase the audio NIM to 250Gi, reranker NIM to 100Gi, and the embedqa NIM to 100Gi to avoid issues downloading the required models.
    Expand the following PVCs:
    1. embedqa NIM PVC 
      oc patch pvc llama-nemotron-embed-1b-v2-pvc  -n nv-ingest --type merge -p '{"spec":{"resources":{"requests":{"storage":"100Gi"}}}}'
    2. Audio NIM PVC (only applicable when the audio NIM is enabled)
      oc patch pvc audio-pvc  -n nv-ingest --type merge -p '{"spec":{"resources":{"requests":{"storage":"250Gi"}}}}'
    3. Reranker NIM PVC (only applicable when the reranking NIM is enabled):
      oc patch pvc llama-nemotron-rerank-1b-v2-pvc  -n nv-ingest --type merge -p '{"spec":{"resources":{"requests":{"storage":"100Gi"}}}}'
  6. Verify that the pods are running:
    oc get deployments -n nv-ingest
    Example output:
    NAME                                READY   UP-TO-DATE   AVAILABLE   AGE
audio                               1/1     1            1           5d23h
llama-nemotron-embed-1b-v2          1/1     1            1           5d23h
llama-nemotron-rerank-1b-v2         1/1     1            1           5d23h
nemotron-graphic-elements-v1        1/1     1            1           5d23h
nemotron-nano-12b-v2-vl             1/1     1            1           5d23h
nemotron-ocr-v1                     1/1     1            1           5d23h
nemotron-page-elements-v3           1/1     1            1           5d23h
nemotron-table-structure-v1         1/1     1            1           5d23h
nv-ingest                           1/1     1            1           5d18h
nv-ingest-opentelemetry-collector   1/1     1            1           5d18h
nv-ingest-zipkin                    1/1     1            1           5d18h