---
description: "Deploy a language model on Verda serverless containers with vLLM in just a few minutes, using a prebuilt image and the standard console deployment settings."
revision_date: 11.11.2026
---

# Quickstart: deploy with vLLM

In this tutorial, we will deploy a vLLM endpoint in a few easy steps. [vLLM](https://docs.vllm.ai/) has become one of the leading libraries for LLM-serving and inference, supporting [many architectures](https://docs.vllm.ai/en/stable/models/supported_models.html) and models that use them.

## Model Weights

vLLM depends on the model weights being fetched from Hugging Face.

In this tutorial we are loading `Qwen/Qwen2.5-1.5B-Instruct` [model from Hugging Face](https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct).

!!! tip
    Some models on Hugging Face require the user to accept their usage policy, so please verify this for any model you are deploying. If you have not Agreed to the policy previously, you will see a similar dialog on the model page on Hugging Face:


You will also require the `User Access Token` in order to fetch the weights. You can obtain the Access Token in your [Hugging Face account](https://huggingface.co/) by clicking the Profile icon (top right corner) and selecting **Access Tokens**.

For deploying the vLLM endpoint, the `READ` permissions are sufficient.

!!! tip
    Please store the obtained token safely. You will need it for the next steps!


## Choose a compatible vLLM tag

The image tag selects a vLLM release and its bundled CUDA libraries. The model, `Qwen/Qwen2.5-1.5B-Instruct`, is selected separately in the startup command.

1. Check the [current NVIDIA driver version for Serverless Containers](https://docs.verda.com/serverless-containers/#gpu-drivers-and-cuda-compatibility).
2. Choose an explicit stable release tag from [the official vLLM image tags](https://hub.docker.com/r/vllm/vllm-openai/tags). Avoid `latest` and nightly tags so the selected release does not change unexpectedly.
3. Read the matching [vLLM release notes](https://github.com/vllm-project/vllm/releases) and that release's documentation for its CUDA build and driver requirements. If the tag does not identify CUDA, check the release's image build configuration or inspect the image's bundled libraries; the vLLM version number alone does not identify its CUDA version.
4. Compare those requirements with [NVIDIA's compatibility table](https://docs.nvidia.com/deploy/cuda-compatibility/minor-version-compatibility.html) and Verda's documented driver. An image that requires a newer driver or a particular unconfirmed patch version needs further verification before deployment. Do not assume that a newer CUDA image works through forward compatibility unless that release explicitly supports your GPU and driver combination.
5. In the documentation for the selected release, confirm support for your GPU and the model's `Qwen2ForCausalLM` architecture. Follow the deployment steps below, then verify the health check and [test request](#test-request) before using the image for production traffic.

## Create the Deployment

In this tutorial, we will deploy `Qwen/Qwen2.5-1.5B-Instruct` on a General Compute (24 GB VRAM) GPU type. For larger models, you may need to choose one of the other GPU types we offer.

1. Log in to the [Verda cloud console](https://console.verda.com/signin), and go to **Containers -> New deployment.** Name your deployment and select the Compute Type.
2. We will be using the official [vLLM Docker container](https://hub.docker.com/r/vllm/vllm-openai), set **Container Image** to `docker.io/vllm/vllm-openai`
3. Toggle on the **Public** location for your image
4. We suggest **Tag** `v0.29.0` as a starting point; this tag was used to test the model. You can also use another compatible release following [Choose a compatible vLLM tag](#choose-a-compatible-vllm-tag).
5. Set the Exposed HTTP port to `8000`
6. Set the Healthcheck port to `8000`
7. Set the Healthcheck path to `/health`
8. Toggle **Start Command** on
9. Add the following parameters to **CMD**: `--model Qwen/Qwen2.5-1.5B-Instruct --gpu-memory-utilization 0.9 --model-loader-extra-config '{"enable_multithread_load": true}'`
10. Add your Hugging Face User Access Token to the **Environment Variables** as `HF_TOKEN`
11. **Deploy container**

(You can leave the **Scaling** options to their default values.)

Wait for the replica to become healthy using the `/health` check configured above, then send the [test request](#test-request). A healthy process alone does not establish that model inference works.

If the logs report an insufficient NVIDIA driver or an unsupported PTX version, recheck the image's CUDA requirements against the [driver guidance](https://docs.verda.com/serverless-containers/#gpu-drivers-and-cuda-compatibility). Select a compatible image or contact support with the tag, GPU, location, and error message. Errors about model loading, memory, or unsupported startup arguments need to be checked against the selected vLLM release separately.

!!! warning
    For production use, we recommend authenticating/using private registries to avoid potential rate limits imposed by public container registries.


## Connect to the Endpoint

<!-- --8<-- [start:njoin-inference-request] -->

Before you can connect to the endpoint, you will need to generate an authentication token, by going to **Credentials -> Inference API Keys**, and click **Create.**

The **base endpoint URL** for your deployment is in the **Containers API** section in the top left of the screen.

### Test Request

Below is an example cURL command for running your test request:

!!! info "Endpoint path"
    Add the `/v1/chat/completions` subpath to the base endpoint URL.


```bash
curl -X POST <YOUR_CONTAINERS_API_URL>/v1/chat/completions \
--header 'Authorization: Bearer <YOUR_INFERENCE_API_KEY>' \
--header 'Content-Type: application/json' \
--data '{
        "model": "Qwen/Qwen2.5-1.5B-Instruct",
        "messages": [
          { "role": "system", "content": "You are a helpful writer assistant." },
          { "role": "user", "content": "What is deep learning?" }
        ],
        "stream": false
    }'
```

### Example Response

You should see a response that looks like this:

```json
{
  "id": "chatcmpl-adf2c6933f6ffde4",
  "object": "chat.completion",
  "created": 1789736418,
  "model": "Qwen/Qwen2.5-1.5B-Instruct",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Deep learning is a subset of machine learning that involves training artificial neural networks with multiple layers to learn and extract features from raw data, such as images or audio signals. The term \"deep\" refers to the number of hidden layers in the network, which allows it to capture complex patterns and relationships in the input data.\nOne of the key advantages of deep learning is its ability to automatically extract hierarchical representations of data, which can be used for tasks such as image recognition, speech recognition, natural language processing, and more. This makes it particularly well-suited for applications where there is not enough labeled data available to train traditional machine learning models.\nThere are many different types of deep learning architectures, including convolutional neural networks (CNNs), recurrent neural networks (RNNs), and transformers. These architectures have been applied to a wide range of domains, including computer vision, natural language processing, speech recognition, and robotics.",
        "refusal": null,
        "annotations": null,
        "audio": null,
        "function_call": null,
        "reasoning": null
      },
      "logprobs": null,
      "finish_reason": "stop",
      "stop_reason": null,
      "token_ids": null,
      "routed_experts": null
    }
  ],
  "service_tier": null,
  "system_fingerprint": "vllm-0.29.0-a3f4f4c1",
  "usage": {
    "prompt_tokens": 25,
    "total_tokens": 209,
    "completion_tokens": 184,
    "prompt_tokens_details": null,
    "completion_tokens_details": null
  },
  "prompt_logprobs": null,
  "prompt_token_ids": null,
  "prompt_text": null,
  "kv_transfer_params": null,
  "ec_transfer_params": null,
  "metrics": null
}
```
<!-- --8<-- [end:njoin-inference-request] -->
