Skip to content

Advanced Guide

Each topic below is a short summary of a reference page: what it is for and the key steps. Follow the link at the end of each topic for the full guide, with every option and edge case.

Instances

Using Jupyter

JupyterLab runs on your instance on port 8888. Never expose that port to the internet; reach it over an SSH tunnel instead.

  1. Deploy an instance with JupyterLab as the Operating System, and secure it.
  2. Open a tunnel from your own machine and keep it open while you work:

    ssh -L 8888:localhost:8888 root@IP_OF_YOUR_INSTANCE
    
  3. Open http://127.0.0.1:8888/ and enter the token. Copy it from the token=... part of the URL behind Open JupyterLab on the instance card.

To work from VS Code instead, keep the tunnel open, install the Jupyter extension, choose Existing Jupyter Server, and enter http://127.0.0.1:8888 with the same token.

See Using Jupyter for the full guide.

Confidential Computing

Confidential VMs (CVMs) keep your data encrypted while it is being processed. They run inside a hardware-enforced Trusted Execution Environment that encrypts CPU and GPU memory and isolates your workload from the hypervisor and from Verda. Available on RTX PRO 6000 (single GPU), B200, and B300.

  • Verify before you trust it. Check CPU memory encryption with sudo dmesg | grep -i sev-snp, attest the CPU with snpguest, and attest the GPU with nvattest attest --device gpu --verifier remote. A helper script verifies the full measured boot chain.
  • Encrypt data at rest with an encrypted home folder (adduser --encrypt-home), or boot your own LUKS2-encrypted OS that only unlocks after attestation.

CVMs cannot reboot from inside the guest

An in-guest reboot will not bring a CVM back. To restart, use Shutdown, then Start in the console or API.

See Confidential Computing for the full guide.

Startup environment and variables

To set environment variables that persist across logins, add a startup script that appends them to /etc/environment:

#!/bin/bash
cat << 'EOF' >> /etc/environment
SECRET1='SECRET1VALUE'
EOF

See Startup environment and variables for the full guide.

Copy files between block devices

Find each volume's device path in the instance's Storage tab, then pick one of two tools:

  • dd makes a full copy of a volume, for example dd if=/dev/vda of=/dev/vdb bs=96M status=progress. Double-check if= (input) and of= (output): a mistake destroys data.
  • rclone copies folders between mounted volumes, for example rclone copy --progress SRC DST.

See Copy files between block devices for the full guide.

Profile with Nsight

Current Verda images come with NVIDIA Nsight Systems (nsys) and Nsight Compute (ncu) preinstalled, and neither needs sudo. Use nsys to find where your application spends time, then ncu to inspect a single kernel:

nsys profile --trace=cuda -o timeline ./your_app
ncu --kernel-name 'regex:KERNEL_NAME' --launch-count 1 -o kernel ./your_app

If you're not signed in as root, run export PATH=/usr/local/cuda/bin:$PATH first.

See Profile with Nsight for the full guide.

Private Networks

Pilot feature

Private Networks is in a limited pilot.

Inside an instance

  • Only the private IP is on the interface. The public IP is applied as 1:1 NAT outside the instance, so bind services to the private IP or 0.0.0.0.
  • Read your own addresses from the metadata service: curl http://169.254.169.254/latest/meta-data/. It has no authentication, so block TCP port 80 to that address if you run untrusted code.
  • No public IP means no outbound internet until you set up internet access.

See Inside an instance for the full guide.

Security model

Each private network is isolated from other customers, even when they use the same address ranges. Networks belong to one project; changing them requires the developer role or above.

See Security model for the full guide.

Clusters

Architecture

A cluster has one login node, one service node, and up to 16 worker nodes with 8 GPUs each.

  • Login node (hostname-login): the only node reachable from the internet, and the SSH entry point.
  • Service node (hostname-service): runs Slurm, Kubernetes, and monitoring. Grafana is at https://<login-ip>.
  • Workers (hostname-1, hostname-2, ...): connected by InfiniBand.
  • Storage: /home is shared by all nodes and deleted with the cluster. Each worker has fast local scratch at /mnt/local_disk.

See Architecture for the full guide.

Networking and ports

  • Open ports on the login node: 22 (SSH), 80 and 443 (Grafana over HTTPS), and 2222 (seamless SSH, Slinky only). Everything else is blocked.
  • Workers aren't reachable from the internet. SSH to the login node, then to a worker by name (ssh hostname-1).
  • NCCL in containers: on H200 and B200 clusters, run containers with -e NCCL_IB_PKEY=1. B300 clusters need no extra setting.

See Networking and ports for the full guide.

Tutorial: deploying vLLM inference on Instant Cluster using Ray

Serve a model across more than 8 GPUs by running vLLM on Ray across several nodes. This runs directly on the nodes, outside Slurm or Kubernetes.

  1. Install vLLM and Ray on each node, and download the model.
  2. Start Ray on the first node with ray start --head, then join each worker with ray start --address="FIRST_NODE_IP:6379".
  3. Start vLLM on the first node with --tensor-parallel-size set to GPUs per node and --pipeline-parallel-size to the number of nodes.

See Deploying vLLM inference on Instant Cluster using Ray for the full guide.

Gang-scheduled Multi-node Training with SkyPilot + Kueue

Launch multi-node training on a Kubernetes cluster with a single sky launch. Kueue starts all pods of a job together or not at all, and queues the rest.

  1. Install SkyPilot locally (pip install 'skypilot[kubernetes]'). Recent cluster images already include Kueue.
  2. Apply a queue manifest with your cluster's GPU, CPU, and memory quota.
  3. Set your kubeconfig namespace to the queue's namespace. This is the most commonly missed step.
  4. Launch the job with sky launch, and release its quota afterwards with sky down.

See Gang-scheduled Multi-node Training with SkyPilot + Kueue for the full guide.

Deploying NVIDIA Dynamo on a Kubernetes Instant Cluster

NVIDIA Dynamo serves LLMs at scale behind an OpenAI-compatible endpoint, splitting prefill and decode across pods. You need an NGC API key, a Hugging Face token, helm, and the shared-path StorageClass set as default.

  1. Run Dynamo's pre-deployment check, and install the GPU Operator if it is missing.
  2. Install the Dynamo CRDs and platform with Helm.
  3. Create secrets for your NGC key and Hugging Face token.
  4. Apply a deployment request for your model and GPU type, then test the endpoint with curl.

To clean up, delete the generated deployment (DGD) first; deleting the request (DGDR) alone leaves the pods running.

See Deploying NVIDIA Dynamo on a Kubernetes Instant Cluster for the full guide.

Storage

Tag immutability rules

Immutability rules stop images with matching tags from being overwritten or deleted, so a tag always points to the same image. Each rule matches repositories and tags using the tag rules syntax. For example, repositories matching ** and tags matching ** makes every tag immutable.

See Tag immutability rules for the full guide.

Tag retention rules and runs

Retention rules decide which images to keep, by age, tag, and repository. A retention run deletes every image that no rule keeps. Immutable tags are never deleted. Run a dry run first to preview what would be removed.

See Tag retention rules and runs for the full guide.

Tag rules syntax

Tag rules use glob patterns, not regex:

Pattern Matches
* Any characters except /
** Any characters including /, only as a whole path segment (**/redis)
? One character
[ab] One character from the set
{beta,rc} Any one of the alternatives

Warning

prefix-** does not match nested paths. Write prefix-*/** instead.

See Tag rules syntax for the full guide.

Serverless Containers

Quickstart: deploy with vLLM

Deploy Qwen/Qwen2.5-1.5B-Instruct with the official vLLM image. You need a Hugging Face token with READ permission.

  1. Go to Containers → New deployment, name it, and pick a General Compute (24 GB VRAM) GPU.
  2. Set the image to docker.io/vllm/vllm-openai:v0.29.0 (Public), the port to 8000, and the health check path to /health.
  3. Turn on Start Command with this CMD, add HF_TOKEN as an environment variable, and deploy:

    --model Qwen/Qwen2.5-1.5B-Instruct --gpu-memory-utilization 0.9 --model-loader-extra-config '{"enable_multithread_load": true}'
    
  4. Create a key under Credentials → Inference API Keys and send a test request:

    curl -X POST <YOUR_CONTAINERS_API_URL>/v1/chat/completions \
    --header 'Authorization: Bearer <YOUR_INFERENCE_API_KEY>' \
    --header 'Content-Type: application/json' \
    --data '{"model": "Qwen/Qwen2.5-1.5B-Instruct", "messages": [{"role": "user", "content": "What is deep learning?"}]}'
    

See Quickstart: deploy with vLLM for the full guide.

Quickstart: migrate from Runpod

Serverless Containers serve a plain HTTP API, so replace the runpod handler with a small web server.

  1. Wrap your existing handler in FastAPI with a /health route and a route for requests (for example /query), served by Uvicorn on port 8000.
  2. In your Dockerfile, replace the runpod package with fastapi and uvicorn, then build and push the image with a version tag.
  3. Deploy it with port 8000, health check path /health, and HF_TOKEN if the model needs it. Set HF_HOME to a path under /data so weights aren't downloaded again on every start.

See Quickstart: migrate from Runpod for the full guide.

Quickstart: GPT-OSS 120B with Ollama

Serve OpenAI's GPT-OSS 120B from your own Ollama image.

  1. Publish your Ollama image with a version tag; :latest is not allowed.
  2. Deploy it on a GPU with at least 80 GB of VRAM, with port 8000 and health check path /api/tags. An incorrect health check is the most common reason for a failed deployment.
  3. Wait for the first start: the weights download once to the persistent /data storage, so later restarts are much faster.
  4. Send requests to /v1/chat/completions with "model": "gpt-oss:120b".

See Quickstart: GPT-OSS 120B with Ollama for the full guide.

Async Whisper inference

Transcribe long audio without keeping the client waiting. Add the Prefer: respond-async header to a request:

curl -X POST https://containers.verda.com/<deployment-name>/generate \
--header 'Authorization: Bearer <YOUR_INFERENCE_API_KEY>' \
--header 'Content-Type: application/json' \
--header 'Prefer: respond-async' \
--data '{"url": "<mp3 url>"}'

The response returns an Id straight away. Poll https://containers.verda.com/status/<deployment-name> with the header X-Inference-Id: <Id> until Status is 3, then fetch the transcript from https://containers.verda.com/result/<deployment-name> with the same header.

See Async Whisper inference for the full guide, and Async inference for every request mode.

Publish your first Docker image

Deployments pull your image from a container registry. To publish one to a private Docker Hub repository:

  1. Create a private repository in Docker Hub, and a personal access token with Read, Write, Delete permissions.
  2. Build, log in with the token as the password, then tag and push with a version tag:

    docker build -t my-app .
    docker login -u <YOUR USERNAME>
    docker tag my-app your-username/my-app:1.0
    docker push your-username/my-app:1.0
    

Always use a specific version tag such as :1.0; :latest can be overwritten and is not allowed in deployments. GHCR, Google Artifact Registry, and Amazon ECR also work; see Container registries.

See Publish your first Docker image for the full guide.

Each product has more guides in its own section of the docs: