Quick: Deploy with vLLM¶

In this tutorial, we will deploy a vLLM endpoint in a few easy steps. vLLM has become one of the leading libraries for LLM-serving and inference, supporting many architectures and models that use them.

Model Weights¶

vLLM depends on the model weights being fetched from Hugging Face.

In this tutorial we are loading Qwen/Qwen2.5-1.5B-Instruct model from Hugging Face.

You will also require the User Access Token in order to fetch the weights. You can obtain the Access Token in your Hugging Face account by clicking the Profile icon (top right corner) and selecting Access Tokens.

For deploying the vLLM endpoint, the READ permissions are sufficient.

Choose a compatible vLLM tag¶

The image tag selects a vLLM release and its bundled CUDA libraries. The model, Qwen/Qwen2.5-1.5B-Instruct, is selected separately in the startup command.

1. Check the current NVIDIA driver version for Serverless Containers.
2. Choose an explicit stable release tag from the official vLLM image tags. Avoid latest and nightly tags so the selected release does not change unexpectedly.
3. Read the matching vLLM release notes and that release's documentation for its CUDA build and driver requirements. If the tag does not identify CUDA, check the release's image build configuration or inspect the image's bundled libraries; the vLLM version number alone does not identify its CUDA version.
4. Compare those requirements with NVIDIA's compatibility table and Verda's documented driver. An image that requires a newer driver or a particular unconfirmed patch version needs further verification before deployment. Do not assume that a newer CUDA image works through forward compatibility unless that release explicitly supports your GPU and driver combination.
5. In the documentation for the selected release, confirm support for your GPU and the model's Qwen2ForCausalLM architecture. Follow the deployment steps below, then verify the health check and test request before using the image for production traffic.

Create the Deployment¶

In this tutorial, we will deploy Qwen/Qwen2.5-1.5B-Instruct on a General Compute (24 GB VRAM) GPU type. For larger models, you may need to choose one of the other GPU types we offer.

1. Log in to the Verda cloud console, and go to Containers -> New deployment. Name your deployment and select the Compute Type.
2. We will be using the official vLLM Docker container, set Container Image to docker.io/vllm/vllm-openai
3. Toggle on the Public location for your image
4. We suggest Tag v0.29.0 as a starting point; this tag was used to test the model. You can also use another compatible release following Choose a compatible vLLM tag.
5. Set the Exposed HTTP port to 8000
6. Set the Healthcheck port to 8000
7. Set the Healthcheck path to /health
8. Toggle Start Command on
9. Add the following parameters to CMD: --model Qwen/Qwen2.5-1.5B-Instruct --gpu-memory-utilization 0.9 --model-loader-extra-config '{"enable_multithread_load": true}'
10. Add your Hugging Face User Access Token to the Environment Variables as HF_TOKEN
11. Deploy container

(You can leave the Scaling options to their default values.)

Wait for the replica to become healthy using the /health check configured above, then send the test request. A healthy process alone does not establish that model inference works.

If the logs report an insufficient NVIDIA driver or an unsupported PTX version, recheck the image's CUDA requirements against the driver guidance. Select a compatible image or contact support with the tag, GPU, location, and error message. Errors about model loading, memory, or unsupported startup arguments need to be checked against the selected vLLM release separately.

Connect to the Endpoint¶

Before you can connect to the endpoint, you will need to generate an authentication token, by going to Credentials -> Inference API Keys, and click Create.

The base endpoint URL for your deployment is in the Containers API section in the top left of the screen.

Test Request¶

Below is an example cURL command for running your test request:

curl -X POST <YOUR_CONTAINERS_API_URL>/v1/chat/completions \
--header 'Authorization: Bearer <YOUR_INFERENCE_API_KEY>' \
--header 'Content-Type: application/json' \
--data '{
        "model": "Qwen/Qwen2.5-1.5B-Instruct",
        "messages": [
          { "role": "system", "content": "You are a helpful writer assistant." },
          { "role": "user", "content": "What is deep learning?" }
        ],
        "stream": false
    }'

Example Response¶

You should see a response that looks like this:

{
  "id": "chatcmpl-adf2c6933f6ffde4",
  "object": "chat.completion",
  "created": 1789736418,
  "model": "Qwen/Qwen2.5-1.5B-Instruct",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Deep learning is a subset of machine learning that involves training artificial neural networks with multiple layers to learn and extract features from raw data, such as images or audio signals. The term \"deep\" refers to the number of hidden layers in the network, which allows it to capture complex patterns and relationships in the input data.\nOne of the key advantages of deep learning is its ability to automatically extract hierarchical representations of data, which can be used for tasks such as image recognition, speech recognition, natural language processing, and more. This makes it particularly well-suited for applications where there is not enough labeled data available to train traditional machine learning models.\nThere are many different types of deep learning architectures, including convolutional neural networks (CNNs), recurrent neural networks (RNNs), and transformers. These architectures have been applied to a wide range of domains, including computer vision, natural language processing, speech recognition, and robotics.",
        "refusal": null,
        "annotations": null,
        "audio": null,
        "function_call": null,
        "reasoning": null
      },
      "logprobs": null,
      "finish_reason": "stop",
      "stop_reason": null,
      "token_ids": null,
      "routed_experts": null
    }
  ],
  "service_tier": null,
  "system_fingerprint": "vllm-0.29.0-a3f4f4c1",
  "usage": {
    "prompt_tokens": 25,
    "total_tokens": 209,
    "completion_tokens": 184,
    "prompt_tokens_details": null,
    "completion_tokens_details": null
  },
  "prompt_logprobs": null,
  "prompt_token_ids": null,
  "prompt_text": null,
  "kv_transfer_params": null,
  "ec_transfer_params": null,
  "metrics": null
}