Registrazione di modelli di fondazione personalizzati per l'implementazione globale

Prerequisiti

Architetture dei modelli supportati

È possibile implementare architetture di modelli di base personalizzate basate sul vLLM runtime a livello globale.

Le seguenti architetture modello sono supportate per l'implementazione a livello globale.

Procedura

Creazione di un ConfigMap file

File di ConfigMap esempio

Crea un ConfigMap file per il modello di base personalizzato copiando uno dei seguenti esempi ConfigMaps e sostituendo le variabili nel modello con i valori appropriati per il tuo modello di base.

apiVersion: v1
kind: ConfigMap
metadata:
  name: meta-llama-3-1-8b-instruct
  namespace: cpd-instance
  labels:
    syom: watsonxaiifm_extra_models_config
  finalizers:
    - watsonxaiifm.cpd.ibm.com/finalizer
data:
  model: |
    meta_llama_3_1_8b_instruct:
      pvc_name: meta-llama-3-1-8b-instruct-pvc
      pvc_size: 62Gi
      command: []
      isvc_yaml_name: isvc.yaml.j2
      dir_name: model
      force_apply: no
      serving_runtime: vllm-serving-runtime
      storage_uri: pvc://meta-llama-3-1-8b-instruct-pvc/
      env:
        - name: MODEL_NAME
          value: /mnt/models
        - name: CUDA_VISIBLE_DEVICES
          value: "0"
        - name: TRANSFORMERS_CACHE
          value: /mnt/models/
        - name: HUGGINGFACE_HUB_CACHE
          value: /mnt/models/
        - name: DTYPE_STR
          value: "float16"
        - name: MAX_SEQUENCE_LENGTH
          value: "2048"
        - name: MAX_BATCH_SIZE
          value: "256"
        - name: MAX_CONCURRENT_REQUESTS
          value: "1024"
        - name: MAX_NEW_TOKENS
          value: "2048"
        - name: FLASH_ATTENTION
          value: "true"
        - name: DEPLOYMENT_FRAMEWORK
          value: "tgis_native"
        - name: HF_MODULES_CACHE
          value: /tmp/huggingface/modules
        - name: SERVED_MODEL_NAME
          value: meta-llama/meta-llama-3-1-8b-instruct
        - name: NUM_GPUS
          value: "1"
        - name: PORT
          value: "3000"
        - name: VLLM_CACHE_ROOT # Additional environment parameters for 5.3.0
          value: /tmp/vllm_cache
      annotations:
        cloudpakId: 5e4c7dd451f14946bc298e18851f3746
        cloudpakName: IBM watsonx.ai
        productChargedContainers: All
        productCloudpakRatio: "1:1"
        productID: 3a6d4448ec8342279494bc22e36bc318
        productMetric: VIRTUAL_PROCESSOR_CORE
        productName: IBM Watsonx.ai
        productVersion: 12.0.0
        cloudpakInstanceId: cd686e30-7b77-4256-a9be-c25e97f5f838
        model-id: "meta-llama/meta-llama-3-1-8b-instruct" # Additional annotation for 5.3.0
      labels_syom:
        app.kubernetes.io/managed-by: ibm-cpd-watsonx-ai-ifm-operator
        app.kubernetes.io/instance: watsonxaiifm
        app.kubernetes.io/name: watsonxaiifm
        icpdsupport/addOnId: watsonx_ai_ifm
        icpdsupport/app: api
        release: watsonxaiifm
        icpdsupport/module: meta-llama-3-1-8b-instruct
        app: text-meta-llama-3-1-8b-instruct
        component: fmaas-inference-server
        bam-placement: colocate
        syom_model: meta--llama-3-1-8b-instruct
      args: []
      wx_inference_proxy:
        meta-llama/meta-llama-3-1-8b-instruct:
          global_custom_foundation_model: true
          enabled:
            - "true"
          label: "meta-llama-3-1-8b-instruct"
          provider: "meta-llama"
          source: "Hugging Face"
          functions:
            - text_generation
            - text_chat
          tags:
            - vllm_runtime  # Additional tags for 5.3.0
          short_description: "A large language model from Meta's LLaMA 3 series, fine-tuned to follow instructions and perform a wide range of natural language understanding and generation tasks."
          long_description: "A powerful 8-billion parameter model from Meta's LLaMA 3 series, specifically fine-tuned to enhance instruction-following capabilities, making it effective for a wide range of NLP tasks."
          task_ids:
            - question_answering
            - generation
            - summarization
            - classification
            - extraction
          tasks_info:
            question_answering:
              task_ratings: { quality: 0, cost: 0 }
            generation:
              task_ratings: { quality: 0, cost: 0 }
            summarization:
              task_ratings: { quality: 0, cost: 0 }
            classification:
              task_ratings: { quality: 0, cost: 0 }
            extraction:
              task_ratings: { quality: 0, cost: 0 }
          min_shot_size: 1
          tier: "class_2"
          number_params: "8b"
          lifecycle:
            available:
              since_version: "9.1.0"
    meta_llama_3_1_8b_instruct_resources:
      limits:
        cpu: "2"
        memory: 128Gi
        nvidia.com/gpu: "1"
        ephemeral-storage: 1Gi
      requests:
        cpu: "1"
        memory: 4Gi
        nvidia.com/gpu: "1"
        ephemeral-storage: 10Mi
    meta_llama_3_1_8b_instruct_replicas: 1

Esempio ConfigMap per modelli di riconoscimento vocale automatico (ASR):

apiVersion: v1
data:
  model: |
    whisper_tiny:
      pvc_name: <pvc name>
      svc_name: whisper-tiny
      pvc_size: 100Gi
      dir_name: .
      force_apply: "no"
      isvc_yaml_name: isvc.yaml.j2
      image: http://registry.redhat.io/rhoai/odh-vllm-cuda-rhel9@sha256:751e2359439161babb9ad8e93e16251888a8c07aed895ffa55e4dfaf2a45f89d
      serving_runtime: vllm-serving-runtime
      storage_uri: pvc://<pvc name>/      # keep the trailing dash after the pvc name
      annotations:
        model-id: "openai/whisper-tiny"
      labels_syom:
        app.kubernetes.io/managed-by: ibm-cpd-watsonx-ai-ifm-operator
        app.kubernetes.io/instance: watsonxaiifm
        app.kubernetes.io/name: watsonxaiifm
        icpdsupport/addOnId: watsonx_ai_ifm
        icpdsupport/app: api
        release: watsonxaiifm
        icpdsupport/module: whisper-tiny
        #app: text-llava-next-video-7b-hf
        component: fmaas-inference-server
        bam-placement: colocate
        syom_model: openai--whisper-tiny  # double dash after model provider name
      env:
        - name: SERVED_MODEL_NAME
          value: openai/whisper-tiny
        - name: VLLM_CACHE_ROOT
          value: /tmp/vllm_cache
        - name: MODEL_NAME
          value: /mnt/models
        - name: SERVED_MODEL_NAME
          value: openai/whisper-tiny
        - name: LANGUAGE
          value: "en"
        - name: NUM_GPUS
          value: "1"
        - name: CUDA_VISIBLE_DEVICES
          value: "0"
        - name: HUGGINGFACE_HUB_CACHE
          value: /mnt/models/
        - name: HF_MODULES_CACHE
          value: /tmp/huggingface/modules
        - name: PORT
          value: "3000"
        - name: MAX_LOG_LEN
          value: "100"
      volumeMounts:
        - name: home
          mountPath: /home/vllm
        - name: tmp
          mountPath: /tmp
        - name: shm
          mountPath: /dev/shm
      volumes:
        - name: home
          emptyDir: {}
        - name: tmp
          emptyDir: {}
        - name: shm
          emptyDir:
            medium: Memory
            sizeLimit: 4Gi
      wx_inference_proxy:
        openai/whisper-tiny:
          enabled:
          - "true"
          label: openai/whisper-tiny
          provider: IBM
          source: IBM
          functions:
          - audio_transcriptions
          short_description: Whisper is a Transformer based encoder-decoder model, also referred to as a sequence-to-sequence model. It was trained on 680k hours of labelled speech data annotated using large-scale weak supervision.
          long_description:  Whisper is a Transformer based encoder-decoder model, also referred to as a sequence-to-sequence model. It was trained on 680k hours of labelled speech data annotated using large-scale weak supervision.
          tags:
          - vllm_runtime
          - consumer_public
          min_shot_size: 1
          input_tier: class_14
          output_tier: class_15
          number_params: 805k
          lifecycle:
            available:
              since_version: 11.0.0
          versions:
          - version: 1.0.0
            since_version: 11.0.0
    whisper_tiny_resources:
      limits:
        cpu: "3"
        memory: 96Gi
        nvidia.com/gpu: "1"
        ephemeral-storage: 1Gi
      requests:
        cpu: "2"
        memory: 85Gi
        nvidia.com/gpu: "1"
        ephemeral-storage: 10Mi
    whisper_tiny_replicas: 1
kind: ConfigMap
metadata:
  finalizers:
  - watsonxaiifm.cpd.ibm.com/finalizer
  labels:
    syom: watsonxaiifm_extra_models_config
  name: whisper-tiny

apiVersion: v1
data:
  model: |
    jina_reranker_v2_base_multilingual:
      pvc_name: jina-reranker-v2-base-multilingual-pvc
      pvc_size: 62Gi
      isvc_yaml_name: isvc.yaml.j2
      embedding_model: "true"
      dir_name: .
      force_apply: no
      serving_runtime: vllm-serving-runtime
      storage_uri: pvc://jina-reranker-v2-base-multilingual-pvc/
      env:
        - name: MODEL_NAME
          value: /mnt/models
        - name: CUDA_VISIBLE_DEVICES
          value: "0"
        - name: TRANSFORMERS_CACHE
          value: /mnt/models/
        - name: HUGGINGFACE_HUB_CACHE
          value:  /mnt/models/
        - name: DTYPE_STR
          value: "float16"
        - name: MAX_SEQUENCE_LENGTH
          value: "2048"
        - name: MAX_NEW_TOKENS
          value: "2048"
        - name: HF_MODULES_CACHE
          value: /tmp/huggingface/modules
        - name: SERVED_MODEL_NAME
          value: jinaai/jina-reranker-v2-base-multilingual
        - name: NUM_GPUS
          value: "1"
        - name: PORT
          value: "3000"
        - name: VLLM_CACHE_ROOT
          value: /home/vllm
        - name: VLLM_USE_V1
          value: "0"
      volumeMounts:
        - name: home
          mountPath: /home/vllm
        - name: tmp
          mountPath: /tmp
        - name: shm
          mountPath: /dev/shm
      volumes:
        - name: home
          emptyDir: {}
        - name: tmp
          emptyDir: {}
        - name: shm
          emptyDir:
            medium: Memory
            sizeLimit: 4Gi
      annotations:
        cloudpakId: 5e4c7dd451f14946bc298e18851f3746
        cloudpakName: IBM watsonx.ai
        productChargedContainers: All
        productCloudpakRatio: "1:1"
        productID: 3a6d4448ec8342279494bc22e36bc318
        productMetric: VIRTUAL_PROCESSOR_CORE
        productName: IBM Watsonx.ai
        productVersion: 12.0.0
        cloudpakInstanceId: cd346dc7-29fd-46db-a874-7e33cb8cf1ac
        model-id: jinaai/jina-reranker-v2-base-multilingual
      labels_syom:
        app.kubernetes.io/managed-by: ibm-cpd-watsonx-ai-ifm-operator
        app.kubernetes.io/instance: watsonxaiifm
        app.kubernetes.io/name: watsonxaiifm
        icpdsupport/addOnId: watsonx_ai_ifm
        icpdsupport/app: api
        release: watsonxaiifm
        icpdsupport/module: jina-reranker-v2-base-multilingual
        app: text-jina-reranker-v2-base-multilingual
        component: fmaas-inference-server
        bam-placement: colocate
        syom_model: jina--reranker-v2-base-multilingual
      command:
        - "/bin/sh"
      args:
        - "-c"
        - "vllm serve /mnt/models --served-model-name jinaai/jina-reranker-v2-base-multilingual --port 3000 --trust-remote-code"
      wx_inference_proxy:
        jinaai/jina-reranker-v2-base-multilingual:
          global_custom_foundation_model: true
          enabled:
          - "true"
          label: "jina-reranker-v2-base-multilingual"
          provider: "jinaai"
          source: "Hugging Face"
          tags:
          - vllm_runtime
          functions:
          - rerank
          short_description: "A multilingual transformer-based reranker that scores query-document relevance with high accuracy"
          long_description: "The Jina Reranker v2 is a cross-encoder model fine-tuned for multilingual text reranking, capable of handling up to 1024 tokens with a sliding window for longer texts. It delivers state-of-the-art performance across retrieval, code, and text-to-SQL reranking tasks."
          tier: "class_c1"
          number_params: 278m
          lifecycle:
            available:
              since_version: "10.1.0"
    jina_reranker_v2_base_multilingual_resources:
      limits:
        cpu: "2"
        memory: 128Gi
        nvidia.com/gpu: "1"
        ephemeral-storage: 1Gi
      requests:
        cpu: "1"
        memory: 4Gi
        nvidia.com/gpu: "1"
        ephemeral-storage: 10Mi
    jina_reranker_v2_base_multilingual_replicas: 1
kind: ConfigMap
metadata:
  finalizers:
  - watsonxaiifm.cpd.ibm.com/finalizer
  labels:
    syom: watsonxaiifm_extra_models_config
  name: jina-reranker-v2-base-multilingual
  namespace: cpd-instance

Scalabilità delle implementazioni globali

Per scalare la distribuzione globale, è necessario modificare il file ConfigMap della distribuzione:

1. Apri il file ConfigMap nel tuo editor di testo predefinito:

   oc edit configmap <configmap name> -n <your namespace>

   Ad esempio:

   oc edit configmap meta-llama-3-1-8b-instruct -n cpd-instance

2. Modifica il <model_name>_replicas valore. Ad esempio, cambia meta_llama_3_1_8b_instruct_replicas: 1 in meta_llama_3_1_8b_instruct_replicas: 2
3. Riconciliazione trigger:

   oc patch watsonxaiifm watsonxaiifm-cr -n <your namespace> --type merge -p '{"spec":{"syom_update_at":"'$(date -u +%Y-%m-%dT%H:%M:%SZ)'"}}'

Cosa fare successivamente

Per testare un modello di base personalizzato distribuito globalmente da un progetto o da uno spazio di distribuzione, inviare una richiesta di inferenza al modello. Per ulteriori informazioni, consulta Implementazione di modelli di base personalizzati.