How to Add Vault Secrets with obs

Your app's secrets (API keys, passwords, connection strings, certificates) are stored in HashiCorp Vault at vault.rit.services. You don't work in Vault directly: you request secrets through the observability platform, obs (obs.rit.services), and obs sets everything up in Vault for you.

Every change to a secret needs approval. A change only runs once key holders approve it with their Vault unseal keys: 3 keys in total, from at least 2 different people. No single person, including whoever made the request, can change, read or delete a secret alone.

Who	What they do
You (the person who needs the secret)	Request a new secret, a change, a deletion, or to see the current values
Key holders (currently Hammad, Matthias and Sven)	Approve or reject requests with their unseal keys

Info

This is not the same as Vaultwarden, our password manager for personal logins. Vault (this guide) holds the secrets your apps use when they run in Kubernetes.

Before you start

You need an obs account with operator access to your app's namespace. Without it you won't see the Vault secrets card described below. Ask a platform administrator to give you access.
Your app should already be deployed, or about to be, through ArgoCD. See How to Setup ArgoCD with your project.

1. Request a new secret

Sign in to obs.rit.services.
Open your namespace:
- Namespace users: click My Namespaces in the sidebar, then your namespace.
- Administrators: click Clusters, then the cluster, then Namespaces, then the namespace.
Find the Vault secrets card and click Add secret.
Fill in the form:
- Secret name: lowercase letters, numbers and hyphens, e.g. orderservice. The secret is stored in Vault at secret/<namespace>/<name>. You can't rename it later.
- How your app gets the secret: pick one.
  - Vault Secrets Operator (recommended, and what most of our projects use). The secret becomes a normal Kubernetes Secret, which your app reads as environment variables. No code changes are needed.
  - Agent Injector (file). The secret is written as files inside your pod. Use this for config files, certificates or keys your app reads from disk.
- Secret data: one row per value. The key is the variable name your app reads, e.g. DATABASE_URL, and the value is the secret itself. Use Agent Injector if you need to upload files instead.
- Advanced (optional): leave it closed unless you know you need it. By default obs names the ServiceAccount <name>-vault, names the Kubernetes Secret <name>-secret, re-reads the secret every 30s, and gives your app read-only access.
Click Request approval.

You'll see Sent to the key holders. The key holders are emailed straight away. The request expires after 24 hours if it isn't approved; if that happens, simply request it again.

Warning

Never paste secrets into Slack, email or tickets to "help" the key holders. They don't need to see your values to approve: they only approve that the change may run.

2. Wait for approval

Follow your request in Vault approvals (sidebar) → My requests.

Status	Meaning
Waiting for approvals	No key holder has approved yet.
Collecting keys	Some keys are in; more are needed.
Done	Approved and carried out. You're also emailed.
Rejected	A key holder rejected it; the reason is shown on the request.
Expired	Not approved within 24 hours. Request it again.
Failed	Approved, but Vault reported an error. The error is shown on the request.

You can withdraw a request that's still waiting with Cancel request.

Tip

If it's urgent, let a key holder know a request is waiting. Two of them need to approve it.

3. Connect your app to the secret

Once the request is Done, the secret appears on your namespace's Vault secrets card.

Click View guide next to the secret.
Copy the manifests it shows into your project's Kubernetes folder (usually k8s/), and commit them:
- Vault Secrets Operator: the manifests (a ServiceAccount, VaultConnection, VaultAuth and VaultStaticSecret) plus a short envFrom snippet to add to your Deployment. Your app then sees every key as an environment variable.
- Agent Injector: the annotations to add to your pod template. Your app then finds the files at the path shown.
ArgoCD deploys the change as usual.

You only need to do this once per secret. Later changes to the values reach your app automatically. With the Vault Secrets Operator, the Kubernetes Secret is updated within the refresh interval (30 seconds by default). Restart your pods if your app only reads its environment at startup.

Change a secret

On the Vault secrets card, click the pencil icon next to the secret.
Enter the values and click Request approval.

An approved change replaces the whole secret

The form starts empty, because obs can't read the current values without approval. Enter the complete set of keys and values, including the ones that aren't changing. Any key you leave out is deleted when the change is approved.

If you don't have the current values, request to see them first (next section), then request the change.

See the current values

On the Vault secrets card, click the eye icon next to the secret, then Request reveal.
Once it's approved, open Vault approvals → My requests, and click View values on the request.
Click Show values once and copy what you need.

Warning

The values can be viewed once, within 15 minutes of approval. After you close the window they're gone from obs. If you miss it, request again.

Delete a secret

Click the bin icon next to the secret, then Request deletion. Once it's approved, the secret, its access policy and its Kubernetes login role are removed from Vault for good.

Danger

Remove the secret from your app's manifests first. Any workload still using it will fail to authenticate on its next sync.

For key holders: approving requests

You're emailed when a request needs approval.

Open Vault approvals → Awaiting approval.
Check what's being asked: who asked, which namespace and secret, and whether it's a create, change, reveal or delete. If you didn't expect it, ask the requester before approving.
Click Approve with my keys, then either:
- tick your saved keys and enter your obs passphrase for them, or
- paste a key from your password manager.
Click Approve with N keys (it shows how many you picked).

How the keys add up:

A request needs 3 keys from at least 2 different people, so one person can't approve a request alone, even with 2 keys.
You can give several of your keys in one go. The dialog shows how many more are needed and how many you can add.
The requester can contribute their own keys too, if they're a key holder.
Vault collects keys for one request at a time. Others wait until it finishes or is cancelled, so don't leave a half-approved request sitting.

To refuse a request, click Reject and give a reason. The requester sees it.

Setting up your keys (once): in Vault approvals → My keys, enroll each of your unseal keys separately. Keep "Save it so I can pick it when approving" ticked to approve with a passphrase instead of pasting the key each time. The passphrase is at least 12 characters and never leaves your browser. An administrator must first mark you as a key holder in Admin → Vault.

Troubleshooting

Problem	What to do
I don't see a Vault secrets card on my namespace	You don't have operator access to that namespace. Ask a platform administrator.
The card says Vault isn't connected yet	Ask a platform administrator to connect Vault in Admin → Vault.
The button says Create secret, not Request approval	Key-holder approval is switched off, so changes run straight away. Ask a platform administrator if that's expected.
My request Expired	Nobody approved it within 24 hours. Request it again and let a key holder know.
A key holder sees You are not a Vault key holder	An administrator must mark them as a key holder in Admin → Vault.
A key is refused when approving	Only keys enrolled under your account in My keys are accepted. Enroll it first.
My app doesn't see a changed value	With the Vault Secrets Operator, wait for the refresh interval, then restart the pods if the app reads its environment only at startup.