Skip to main content

vLLM (Containerized)

Learn how to run Granite models with vLLM in a container.

Overview​

In this guide, we'll use vLLM running inside a container to serve Granite models.

Prerequisites​

1. Pull the vLLM container image​

docker pull vllm/vllm-openai:latest

(For stability, you may pin a version tag, e.g., vllm/vllm-openai:v0.18.0. Granite models require vllm version 0.18.0 and above.)

2. Run Granite in the container​

Run the container with your Hugging Face cache mounted:

docker run --runtime nvidia --gpus all     -v ~/.cache/huggingface:/root/.cache/huggingface     -p 8000:8000     vllm/vllm-openai:latest     --model ibm-granite/granite-4.0-h-small
note

You can pre-download the model into ~/.cache/huggingface using:

hf download ibm-granite/granite-4.0-h-small

If not pre-downloaded, the model will be fetched automatically when vLLM starts and cached in the mounted directory.

3. Run a sample request​

curl -X POST http://localhost:8000/v1/chat/completions   -H "Content-Type: application/json"   -d '{
        "model": "ibm-granite/granite-4.0-h-small",
        "messages": [
          {"role": "user", "content": "How are you today?"}
        ]
      }'

4. Enabling tool calling and other extended capabilities​

To run vLLM with the Granite 4.0 models and enabling capabilities such as tool calling, use the additional parameters --tool-call-parser granite4 and --enable-auto-tool-choice. Refer to the vLLM documentation here for more details on these and other parameters.

Now run the container with the added parameters:

docker run --runtime nvidia --gpus all     -v ~/.cache/huggingface:/root/.cache/huggingface     -p 8000:8000     vllm/vllm-openai:latest     --model ibm-granite/granite-4.0-h-small --tool-call-parser granite4 --enable-auto-tool-choice

Once the container is up, you can start running requests using the OpenAI API. Refer to the documentation on OpenAI API tool calling for examples.

To run vLLM with the Granite 3 models and tool calling, use the additional parameters specified in the vLLM documentation here as part of the docker run command share in Section 2.