The Recovery Runbook You Should Have Written Six Months Ago

Halfway through writing the recovery procedure for a stack you've been running for two years, you realize you don't know where the .env file actually lives, whether the named volume is app_data or app-data, or what network_mode: host is doing on that one service and whether it matters on restore. The mental model is complete. The procedure isn't.

This is the documentation session — scoped to a 31-container Proxmox and Synology environment. What to extract, in what order, what to write down, and how to know when you're done. The output is a runbook that passes one test: someone technically competent who has never seen this environment can execute it under pressure.

Jump to Commands Reference →

Inventory what's actually running before you write a word

The documentation session starts with a full inventory pass, not with writing. The gap between what you think is running and what's actually running is where the first surprises live.

On each Proxmox host, pull the full container list with status and ports:

docker ps --format "table {{.Names}}	{{.Image}}	{{.Status}}	{{.Ports}}"

Then pull the list of active compose projects:

docker compose ls

The output of docker compose ls tells you the project name, the compose file path, and the status. Any container running without a corresponding compose project is running from a bare docker run command — which means its configuration exists only in the daemon's runtime state. Those are the first items that need documentation. If the daemon restarts, the intent behind those containers is gone.

On Proxmox, list the VMs:

qm list

Do this on every node. Cross-reference against what you know is running. Any VM not in your mental model is worth investigating before you assume it can be ignored in the runbook.

Extract the rendered Docker stack configuration

The compose file in your Git repo or working directory is the declared state. The rendered config — with environment variables substituted and defaults applied — is what's actually running. For recovery purposes, the rendered config is what matters.

From each compose project directory:

docker compose config

The output is the fully resolved compose definition: every environment variable substituted, every default made explicit. Save this alongside the source compose file. The difference between them is exactly the documentation gap that kills recoveries — the variables that live in the .env file and nowhere else.

While you're in each project directory, note the .env file location. These files are frequently absent from file-level backups because they look like configuration scaffolding, not data. If your backup job covers /working/docker/ but not the compose project directories where you keep stack definitions, the env file isn't backed up.

find /opt/stacks -name ".env" -type f 2>/dev/null
find /home -name ".env" -type f 2>/dev/null

For each .env file found, document its path and confirm it's inside a backed-up directory. If it isn't, move it or add the path to the backup job now, before continuing.

Document named volumes, networks, and the things compose files don't show

Named volume documentation is where most runbooks fail. The backup covers the data inside the volume. The runbook needs the volume name and mount path — because the container starts against an empty filesystem if either is wrong on restore, with no error that points to the cause.

docker volume ls
docker volume inspect <volume_name>

For each named volume associated with a container you care about: record the volume name exactly as it appears in docker volume ls, the mountpoint on the host, and the container path it maps to. These three pieces together are what you verify on restore.

Network documentation matters for any container using a custom bridge network. The default bridge and host networking reconstruct automatically. Custom bridge networks with specific names don't — they need to be created before the containers that reference them can start.

docker network ls
docker network inspect <network_name>

Record the name, driver, and subnet for every non-default bridge network. In the runbook, these network creation steps go before any stack deployment that references them.

The start-order dependency is the documentation item most likely to be missing and most painful to reconstruct under pressure. Which stacks must be up before others work? If your reverse proxy, database, or auth service needs to be running before downstream containers can initialize correctly, write that dependency down explicitly. It won't appear in any compose file.

Export Proxmox VM configuration

For each VM on each node, export the config:

qm config <vmid> > vm-<vmid>-config-$(date +%Y%m%d).txt

The exported config contains disk assignments, network bridge names, memory and CPU allocation, and boot order. The critical annotation to add alongside it: what the storage IDs in that config map to physically, because Proxmox storage IDs are local to the node. A PBS backup restored to a different node requires remapping those storage pool references manually — the restore UI gives no indication of the correct mapping if you haven't documented it.

For each node, document the storage pool layout:

pvesm status

Record the storage ID, type (dir, LVM, ZFS, NFS), and what it physically represents. This is the translation table that makes a cross-node restore coherent. Also document network bridge assignments per node: which bridge connects to which physical interface or VLAN. On restore to a different node, bridge names carry no inherent meaning — only the topology does.

Document the Synology configuration

Three things belong in the runbook for Synology recovery: what Hyper Backup covers, what the share layout is, and the DSM configuration backup.

In the Hyper Backup interface, for each backup job: the source shares and folders, the destination, the schedule, and the retention policy. Then explicitly note what's not covered. Shares created after the last time you reviewed the job configuration are a common gap — Hyper Backup doesn't automatically include new shares.

Document each share: name, path, permissions model (public, user-based, group-based), and whether SMB and NFS are enabled. On a fresh DSM install, shares need to be recreated before any Hyper Backup restore can complete — and the share name must match exactly what the backup job expects.

The DSM configuration backup is separate from data backup. It covers system configuration: users, groups, network settings, application settings, and share definitions. In DSM: Control Panel → Update & Restore → Configuration Backup → Back Up Configuration. The output is a .dss file. Store it somewhere that's included in a backup job.

Vaultwarden gets its own section

Every other service in the environment can wait while you piece together a recovery procedure under pressure. Vaultwarden cannot — it holds the credentials you need to access everything else. Its restore procedure is separate from the file-level backup covering the rest of the environment, and specific enough that discovering the details during a real incident is a serious problem.

The Vaultwarden backup that matters is the SQLite database file, typically at /working/docker/vaultwarden/data/db.sqlite3. Hyper Backup covering the Docker data directory will include this file — but verify it explicitly, because the backup job likely predates your documentation of exactly what it covers.

The restore sequence: stop the container, replace the database file, restart. No import step — the container reads the database on startup. The variable to document explicitly is the admin token: if it's set via environment variable in the compose file, it survives the restore. If it was set interactively and isn't in the compose file, the restore produces a running container with no admin access and no error indicating why.

docker compose -f /path/to/vaultwarden/docker-compose.yml down
cp /path/to/backup/db.sqlite3 /working/docker/vaultwarden/data/db.sqlite3
docker compose -f /path/to/vaultwarden/docker-compose.yml up -d

Document the admin token location. Document the HIBP API key if in use. Document the SMTP configuration. None of these are stored in the database — they're runtime configuration that won't survive a restore if they aren't in the compose file.

Apply the quality test before calling it done

Read through every section with this question: could someone technically competent who has never seen this environment execute it to a running state? Not a stranger off the street — a competent operator who simply doesn't know your specific setup.

The consistent failure modes: steps that reference a path without saying how to find it; steps that name a service without saying where its config lives; steps that say "configure the service" without specifying what the default gets wrong for this environment; steps that assume the reader knows the sequence without documenting it. If anything in the runbook requires the reader to figure something out rather than execute a documented step, it isn't finished.

Run one restore before you trust any of it

Pick the least consequential service in the environment — something where an hour offline creates no real problem. Stop it. Delete its named volume. Follow the recovery procedure exactly, without improvising or filling gaps from memory.

The gaps you find will be consistent across most of the other procedures, because the missing information falls into the same categories: volume paths, env variable sources, network configuration, service-specific gotchas that aren't documented anywhere. One restore test reveals the systemic gaps. Fix the pattern, then update the individual procedures.

If you won't run the test, the documentation isn't finished. The writing phase and the validation phase are both part of the procedure. A runbook that has never been executed is a hypothesis, not a procedure.

Commands Reference

# Inventory — Docker
docker ps --format "table {{.Names}}	{{.Image}}	{{.Status}}	{{.Ports}}"
docker compose ls

# Inventory — Proxmox (run on each node)
qm list

# Rendered compose config (run from each project directory)
docker compose config

# Find .env files
find /opt/stacks -name ".env" -type f 2>/dev/null
find /home -name ".env" -type f 2>/dev/null

# Named volumes
docker volume ls
docker volume inspect <volume_name>

# Networks
docker network ls
docker network inspect <network_name>

# Proxmox VM config export (run for each VM ID on each node)
qm config <vmid> > vm-<vmid>-config-$(date +%Y%m%d).txt

# Proxmox storage pool status (run on each node)
pvesm status

# Synology DSM config backup
# DSM → Control Panel → Update & Restore → Configuration Backup → Back Up Configuration
# Save the resulting .dss file to a backed-up location

# Vaultwarden database location check
ls -lh /working/docker/vaultwarden/data/db.sqlite3

# Vaultwarden restore sequence
docker compose -f /path/to/vaultwarden/docker-compose.yml down
cp /path/to/backup/db.sqlite3 /working/docker/vaultwarden/data/db.sqlite3
docker compose -f /path/to/vaultwarden/docker-compose.yml up -d