Tutorial: Secrets Management with OpenBao¶

What you'll build¶

In this guide we will build a secrets manager that hands short-lived credentials to your inference workloads, so no long-lived key sits in a container image or a deployment's environment variables.

• A Hugging Face token your Serverless Container fetches at start-up rather than carrying in its configuration, so the deployment holds no credential at all.
• A credential you change in one place, which every workload picks up as its replicas cycle, however many deployments share it.
• Access scoped per workload, so adding one is a policy and an AppRole, and revoking its access is one command rather than a hunt through every deployment that might hold a copy.

Several tools can do this. The common self-hostable options are HashiCorp Vault, OpenBao and Infisical. This guide uses OpenBao, an open-source project under the Linux Foundation.

Setting HF_TOKEN or DATABASE_URL as a deployment environment variable works, and for a single prototype it is the right amount of effort. It stops being enough once the same credential is in more than one place, for three reasons:

All three follow from one cause: the credential lives inside the thing that uses it. A secrets manager holds it outside, so the workload fetches it at start-up instead of carrying a copy.

Architecture¶

   +---------------------+   1. log in    +--------------------------+
   |                     | -------------> |                          |
   |      bao agent      |                |         OpenBao          |
   |                     | <------------- |   kv/        API keys    |
   |  runs beside your   |  2. a token,   |   database/  DB users    |
   |     application     |  then secrets  |                          |
   +----------+----------+                +--------------------------+
              |                             CPU instance
              | 3. write
              v
   +---------------------+
   |  your application   |   an env var, in a Serverless Container
   +---------------------+

OpenBao is the only part that has to run all the time. Your workloads keep whatever schedule they already have, and the agent runs inside them as an extra process.

The agent is what makes this practical. It logs in, keeps the token renewed, and puts the secrets where your application already looks for them, which here is the environment of the process it starts. Your application code does not change: it still reads an environment variable.

The Serverless Container reaches OpenBao over its public address, across the internet rather than inside your account, since a container deployment has no interface on a subnet of yours. That is why the listener in Step 3 carries TLS and why every credential here is scoped and short-lived.

Prerequisites¶

• Register an SSH key in your Verda account, if you have not already.
• Create an Inference API Key, which is how you call the deployment in Step 7.
• Create Container Registry credentials, on the Container Registry or Credentials page. Step 7 builds a container image and pushes it there.
• Have ready a Hugging Face token. You store it in OpenBao in Step 5, and the container fetches it from there rather than from the deployment.

Everything here is done in the Verda console and over SSH.

Step 1: Create the instance¶

OpenBao runs on one CPU instance, named bao throughout this guide. The workload it serves is a Serverless Container, so this is the only machine you create.

Step 2: Install OpenBao¶

SSH into the bao instance. Everything in Steps 2 to 7 runs there.

Install the .deb package. This guide is written against v2.7.0; newer releases are on the releases page, and the commands here do not change between versions:

curl -fsSL -o openbao.deb \
  https://github.com/openbao/openbao/releases/download/v2.7.0/openbao_2.7.0_linux_amd64.deb

sudo dpkg -i openbao.deb

Check which version you have:

bao version

OpenBao v2.7.0

Create the data directory and give the service ownership of it:

sudo mkdir -p /opt/openbao/data
sudo chown -R openbao:openbao /opt/openbao

Your secrets will live here, on the instance's OS volume. The store only holds credentials, so it stays lightweight even with thousands of them.

Step 3: Create a certificate and start the server¶

OpenBao serves plain HTTP by default, so tokens travel unencrypted and anyone who can read the network traffic can read them too. A certificate encrypts that traffic and proves the server's identity, so a client knows it is sending its credentials to your OpenBao and not to an impostor.

We will create the certificate before the config file, because the config points at the certificate files and the service will not start until they exist.

A certificate is issued to an address, and a client checks that the address it connected to is named in it. Everything here reaches the bao instance on its public address, so that is the address the certificate carries.

sudo mkdir -p /etc/openbao/tls

Create the certificate. On the last line, replace <bao-public-ip> with the bao instance's public address, keeping both the IP: in front of it and the quotation marks around the whole value:

sudo openssl req -x509 -newkey rsa:2048 -nodes -days 365 \
  -keyout /etc/openbao/tls/key.pem \
  -out    /etc/openbao/tls/cert.pem \
  -subj   "/CN=openbao" \
  -addext "subjectAltName=IP:<bao-public-ip>"

Restrict the key to the service:

sudo chown -R openbao:openbao /etc/openbao/tls
sudo chmod 600 /etc/openbao/tls/key.pem

Check that the address made it in:

openssl x509 -in /etc/openbao/tls/cert.pem -noout -text | grep -A1 "Alternative"

X509v3 Subject Alternative Name:
    IP Address:203.0.113.20

This address has to match the bao instance's public address. If it does not, run the certificate command again with the right one, because clients refuse to connect until it matches.

This certificate signs itself, so nothing trusts it until you say so: every client in the rest of this guide is given a copy of cert.pem to check it against.

The server reads its settings from /etc/openbao/openbao.hcl, and the package already wrote one that uses different storage and certificate paths. Move it aside and write your own:

sudo mv /etc/openbao/openbao.hcl /etc/openbao/openbao.hcl.default

Then save this as /etc/openbao/openbao.hcl, replacing <bao-public-ip> on the last two lines with the bao instance's address:

ui = true

storage "raft" {
  path    = "/opt/openbao/data"
  node_id = "bao-1"
}

listener "tcp" {
  address       = "0.0.0.0:8200"
  tls_cert_file = "/etc/openbao/tls/cert.pem"
  tls_key_file  = "/etc/openbao/tls/key.pem"
}

api_addr     = "https://<bao-public-ip>:8200"
cluster_addr = "https://<bao-public-ip>:8201"

0.0.0.0:8200 listens on every interface the instance has.

Start the service. --now starts it immediately, and enable makes it come back on every reboot:

sudo systemctl enable --now openbao

Check it came up:

sudo systemctl status openbao

The service is running if the output says Active: active (running). If it says failed instead, the cause is usually an error in the config file, which journalctl -u openbao -n 30 will show you.

The rest of this guide uses the bao command on this instance, and it needs to know where the server is and which certificate to check it against. Set both as environment variables, and set them again in every new shell, including after reconnecting over SSH:

export BAO_ADDR='https://<bao-public-ip>:8200'

export BAO_CACERT='/etc/openbao/tls/cert.pem'

Then check the server's state:

bao status

Key                Value
---                -----
Seal Type          shamir
Initialized        false
Sealed             true
...

Initialized false is expected at this point, and Step 4 changes it.

Step 4: Initialize and unseal¶

A freshly started OpenBao cannot read its own data, because everything it stores is encrypted on disk and the key that decrypts it is not in memory yet. OpenBao calls this being sealed. Initializing creates that key and splits it into shares, and unsealing is giving enough of those shares back so it can rebuild the key.

bao operator init -key-shares=5 -key-threshold=3

-key-shares=5 splits the key five ways and -key-threshold=3 means any three of those shares can rebuild it, so no one person can unseal alone and two share-holders can be unavailable. Seal and unseal covers the mechanism.

Unseal Key 1: nB7x...
Unseal Key 2: 4Kp2...
Unseal Key 3: qW9s...
Unseal Key 4: Lm3v...
Unseal Key 5: Zt8h...

Initial Root Token: s.a1b2c3...
...

Save the five unseal keys and the root token now. This output appears once and cannot be retrieved later, so losing it makes the data permanently unreadable, by design. Give each unseal key to a different person, and keep the root token in a password manager.

Unsealing needs three of the five, so run this three times. The command asks for the key rather than taking it as an argument, and hides what you type:

bao operator unseal

Unseal Key (will be hidden):

Each run prints the status table. Sealed stays true after the first two keys, then turns false on the third:

Key                     Value
---                     -----
Seal Type               shamir
Initialized             true
Sealed                  false
Total Shares            5
...

Log in with your own root token, from the output above:

bao login <root-token>

Success! You are now authenticated. The token information displayed below is
already stored in the token helper. You do NOT need to run "bao login" again.
...

The token is cached on the instance, so the commands in the steps that follow use it without you passing it again.

It re-seals on every restart, so three key holders have to be available after each reboot. Production deployments configure auto-unseal instead, where a key management service holds the key and OpenBao opens itself.

Step 5: Store the secret¶

Enable a versioned key-value store:

bao secrets enable -path=inference kv-v2

Success! Enabled the kv-v2 secrets engine at: inference/

Store your Hugging Face token. This is the credential the serving container needs, and it is the one every other vLLM guide asks you to type into the deployment's environment variables. The label beside it is not secret, and the container prints it at start-up so you can see which version it fetched:

bao kv put inference/hf token='<your-hugging-face-token>' label='v1'

Read it back to check it stored:

bao kv get inference/hf

==== Secret Path ====
inference/data/hf

======= Metadata =======
...

====== Data ======
Key      Value
---      -----
label    v1
token    <your-hugging-face-token>

You wrote inference/hf, but the Secret Path line shows inference/data/hf, because a KV v2 store keeps secrets one level deeper and the bao kv commands hide that data/ part for you. Policies do not hide it, which is why the one in Step 6 spells out the longer path.

Step 6: Give a workload access¶

Access control here has two pieces. A policy says what may be read, and an AppRole is how a machine proves which policy applies to it.

Write a policy¶

Save this as chat-policy.hcl in your home directory on the bao instance:

path "inference/data/hf" {
  capabilities = ["read"]
}

bao policy write chat chat-policy.hcl

Success! Uploaded policy: chat

The data/ in that path is not a typo, and leaving it out is the most common mistake in this step. A KV v2 mount rewrites paths internally, so the secret you wrote as inference/hf is stored at inference/data/hf, and a policy has to name the stored path rather than the one you typed.

The policy names one path and nothing else, so this workload can read that secret and no other, however many you store later.

Create an AppRole¶

bao auth enable approle

Success! Enabled approle auth method at: approle/

Then create the role itself by running this in the terminal on the bao instance:

bao write auth/approle/role/chat \
  token_policies="chat" \
  token_ttl=20m \
  token_max_ttl=1h \
  secret_id_ttl=24h

Success! Data written to: auth/approle/role/chat

token_policies="chat" attaches the policy you just wrote, so anything logging in with this role gets exactly that one read. token_ttl=20m means a token issued from the role dies twenty minutes after issue, and the agent renews it silently, so a token stolen from a container is useless almost immediately. secret_id_ttl=24h does the same for the credential the role logs in with.

Logging in with this role takes two values, a role id and a secret id. The role id names which role is logging in, it stays the same for the life of the role, and it is not a secret. The command below reads it:

bao read auth/approle/role/chat/role-id

Key        Value
---        -----
role_id    8f2c1e40-3b7a-4d15-9c62-1a5e7b3d0f89

Then generate a secret id. This one is the credential, and it expires 24 hours after you create it. -f tells bao write to send the request with no data fields, which is how you ask the role to mint one:

bao write -f auth/approle/role/chat/secret-id

Key                   Value
---                   -----
secret_id             3d9b7f21-8e4c-42a6-b105-6f2d8c9a4e73
secret_id_accessor    0c2b5a84-7d13-4e9f-8a60-2f4c1b9d6e05
secret_id_ttl         24h

Keep both values to hand for the next step.

Step 7: Inject into a Serverless Container¶

Add the agent to your image and let it start your server as a child process, with the secret in the environment.

The example serves a chat model with vLLM. What goes into the image:

Any image that serves HTTP works the same way, and only the exec command below changes.

This is the one step that needs a container image of your own, because the agent has to be inside it, and you push that image to the Verda Container Registry, which Serverless Containers pulls from without any extra configuration. The image is built from four files in one directory. This guide builds it on the bao instance, where the certificate already is, while in your own work you would build it wherever you build images now and copy the certificate across.

Docker comes with the instance image chosen in Step 1, so there is nothing to install.

The vLLM base image takes around 30 GB on the instance's OS volume, so check there is room before you start.

Save this as agent.hcl in your home directory on the bao instance:

auto_auth {
  method "approle" {
    config = {
      role_id_file_path                   = "/tmp/role_id"
      secret_id_file_path                 = "/tmp/secret_id"
      remove_secret_id_file_after_reading = true
    }
  }
}

env_template "HF_TOKEN" {
  contents = "{{ with secret \"inference/data/hf\" }}{{ .Data.data.token }}{{ end }}"
}

env_template "SECRET_LABEL" {
  contents = "{{ with secret \"inference/data/hf\" }}{{ .Data.data.label }}{{ end }}"
}

exec {
  command = ["/bin/sh", "-c",
             "echo \"openbao secret label: $SECRET_LABEL\"; exec vllm serve Qwen/Qwen3-8B --served-model-name chat --max-model-len 16384"]
  restart_on_secret_changes = "never"
}

Save this as entrypoint.sh in the same directory. It writes the two AppRole values where the agent expects them, then hands over:

#!/bin/sh
set -e
printf '%s' "$BAO_ROLE_ID"   > /tmp/role_id
printf '%s' "$BAO_SECRET_ID" > /tmp/secret_id
exec bao agent -config=/etc/bao/agent.hcl

The agent checks the server against the certificate you created in Step 3, so the image needs a copy of it. It is already on this instance, so copy it into the build directory:

sudo cp /etc/openbao/tls/cert.pem ./ca.pem

The copy is owned by root, so the build cannot read it. Change the owner to ubuntu:

sudo chown ubuntu:ubuntu ca.pem

If you build on your own machine instead, pull the same file down with scp ubuntu@<bao-public-ip>:/etc/openbao/tls/cert.pem ./ca.pem.

Then save this as Dockerfile, alongside the other three files:

FROM vllm/vllm-openai:v0.27.1

COPY --from=openbao/openbao:2.7.0 /usr/bin/bao /usr/local/bin/bao
COPY agent.hcl      /etc/bao/agent.hcl
COPY ca.pem         /etc/bao/ca.pem
COPY entrypoint.sh  /entrypoint.sh
RUN chmod +x /entrypoint.sh

ENV BAO_CACERT=/etc/bao/ca.pem

ENTRYPOINT ["/entrypoint.sh"]

Build the image from the four files, with the trailing dot pointing at the current directory:

docker build -t chat-with-agent:1 .

The first build takes several minutes, since it pulls the vLLM base image. It ends with two lines naming the image, and the id will differ from this one:

Successfully built 1d853adea068
Successfully tagged chat-with-agent:1

If the build returns a permission error instead, add your user to the docker group, start a shell with the new group applied, and run the build again:

sudo usermod -aG docker ubuntu
newgrp docker

Log in to the Verda Container Registry. Replace all three placeholders, <registry-secret>, <project-id> and <credentials-name>, keeping the quotes. The secret and the credentials name come from the registry credentials you created in the console, and <project-id> appears twice:

echo '<registry-secret>' | docker login vccr.io/<project-id> \
  -u 'vcr-<project-id>+<credentials-name>' --password-stdin

Login Succeeded

Docker also warns that it stored the credentials unencrypted in config.json. That is expected, and it is why the secret went in through --password-stdin rather than the command line, where it would also land in your shell history.

Tag the image with the registry address, which is how docker push knows where to send it:

docker tag chat-with-agent:1 vccr.io/<project-id>/chat-with-agent:1

Then push it. This also takes a few minutes, and it ends with the tag and the image's digest:

docker push vccr.io/<project-id>/chat-with-agent:1

1: digest: sha256:... size: 4283

Create a deployment with these settings:

The first start takes a few minutes while the weights, about 16 GB, are pulled and loaded. While you wait, the deployment's system log shows the normal sequence: Scheduled, Pulling, Pulled, Created, Started, then several Startup probe failed: connection refused entries while the weights load. Those failures are expected, because nothing is listening on the port yet, and the deployment turns healthy once vLLM finishes loading.

The Hugging Face token is not in that table, which is the difference from every other vLLM deployment guide. The agent fetches it at start-up and it exists only in the environment of the process the agent starts, so the console shows no credential and nobody browsing your deployments can read it.

The deployment's log is where you watch the whole chain happen. The agent logs in with the AppRole, renders both values, prints the label, then hands over to vLLM:

[INFO]  agent.auth.handler: authentication successful, sending token to sinks
[INFO]  agent: (runner) rendered "(dynamic)" => "HF_TOKEN"
[INFO]  agent: (runner) rendered "(dynamic)" => "SECRET_LABEL"
[INFO]  agent.exec.server: [INFO] (child) spawning: /bin/sh -c echo "openbao secret label: $SECRET_LABEL"; exec vllm serve Qwen/Qwen3-8B ...
openbao secret label: v1

The label line is the proof that this container fetched this version of the secret.

Once the deployment reports healthy, call it with your Verda Inference API Key, which is how Verda authenticates callers to any container endpoint. The address copied from the console ends with a /, so delete that before adding the path:

curl -sL -X POST <endpoint-address>/v1/chat/completions \
  --header "Authorization: Bearer <your-inference-api-key>" \
  --header 'Content-Type: application/json' \
  --data '{"model": "chat", "messages": [{"role": "user", "content": "Say hello."}],
           "chat_template_kwargs": {"enable_thinking": false}}'

{"id":"chatcmpl-bec240c2fd9d1250","object":"chat.completion","model":"chat",
 "choices":[{"index":0,"message":{"role":"assistant","content":"Hello! How can I assist you today?"},
 "finish_reason":"stop"}],
 "usage":{"prompt_tokens":15,"total_tokens":27,"completion_tokens":12}}

enable_thinking: false keeps Qwen3 from prefixing every reply with its reasoning notes.

The deployment does still hold the secret id, which is the trade rather than a gap. One scoped credential that expires in 24 hours replaces a permanent token, the secret itself is never in a deployment, and secret_id_num_uses caps how many logins that id allows.

Congratulations! Your deployment pulls its model with a token it does not store, and the only credential in its configuration is a login that expires in 24 hours. The rest of this page covers what to adjust once it is running.

After it works¶

Changing a secret¶

Changing a credential is a write to OpenBao, with no deployment to edit. On the bao instance, store the new token and move the label on:

bao kv put inference/hf token='<new-hugging-face-token>' label='v2'

== Secret Path ==
inference/data/hf

======= Metadata =======
Key        Value
---        -----
version    2

Replicas already running keep the value they fetched at start-up, so nothing changes for them. A new replica reads version 2 the moment it starts, which happens on the next cold start after the deployment has been idle, when you scale up, or when you restart the deployment.

The quickest way to see it is to scale up after the write. The new replica logs the new label while the old one is still serving on the old value:

openbao secret label: v2

Each replica prints its own line, so a deployment mid-rotation shows both, which is also how you tell that a rotation has finished.

Before revoking the old credential, check how your workload uses it, because a replica keeps the value it fetched until it is replaced. A credential sent on every call stops working the moment you revoke it, so wait until the replicas holding it have cycled. One read only at start-up, like this token, is safe to revoke as soon as the weights are loaded.

Adding more secrets¶

Access control and delivery are two different changes, and only one of them is instant. A policy takes effect on the next login, while the list of secrets a container reads lives in agent.hcl inside the image, so changing that list means building the image again.

Another secret for the same container. Say the chat application starts calling an external API and needs that credential too. Store it first:

bao kv put inference/tool-api key='<the-api-credential>'

Then add a second path block to chat-policy.hcl and write the policy again:

path "inference/data/hf" {
  capabilities = ["read"]
}

path "inference/data/tool-api" {
  capabilities = ["read"]
}

bao policy write chat chat-policy.hcl

The policy only grants access. To deliver the value, add an env_template block to agent.hcl beside the ones from Step 7, where the name in quotes is the environment variable and the path is the secret it comes from:

env_template "TOOL_API_KEY" {
  contents = "{{ with secret \"inference/data/tool-api\" }}{{ .Data.data.key }}{{ end }}"
}

Rebuild the image, push it, and point the deployment at the new tag. The policy change on its own does nothing visible, because the agent renders only the templates that were in the image when it was built.

A second deployment with its own secret. Say you add chat-2, serving another model with a Hugging Face token of its own. It repeats Steps 5 to 7 under different names: store the token at inference/hf-2, write chat-2-policy.hcl granting read on that one path, create a chat-2 AppRole, and build its image with its own templates, then put BAO_ADDR and the new role's two values in that deployment. The policy and AppRole commands are Step 6 with the names changed:

bao policy write chat-2 chat-2-policy.hcl

bao write auth/approle/role/chat-2 \
  token_policies="chat-2" \
  token_ttl=20m \
  token_max_ttl=1h \
  secret_id_ttl=24h

The chat deployment is untouched, because its policy and AppRole are separate. That is the property that makes this scale: workloads are added beside each other rather than by editing anything that already works.

Revoking access. Deleting a role stops the workloads using it at their next login, and nothing else:

bao delete auth/approle/role/chat-2

Their current tokens keep working until they expire, which is twenty minutes at most with the token_ttl used here.

For running OpenBao in production, check the OpenBao documentation, which covers more on auto-unseal, clustering, audit logging and certificates.

Conclusion¶

You now have OpenBao storing credentials on a Verda instance, a Serverless Container that fetches its Hugging Face token at start-up instead of carrying it, and a credential you can change without editing any deployment.

To clean up. Instances keep billing while they exist. When you are finished, discontinue the bao instance and delete the container deployment.

Related guides¶

• Container Registry: pushing images, tag rules and retention policies.
• Serverless Containers: how deployments work.
• Scaling and health checks: replica behaviour during scale-down, which affects the agent's child process.
• Vector database: a Postgres deployment you can issue dynamic credentials against.