How Deep Does the Recovery Documentation Need to Go?
The Architecture post for this idea draws the line between data resilience and operational resilience , between the backup job completing and actually being able to rebuild your environment. That line is the reason you're here. The decision this post is about is what comes next: given that the recovery procedure needs to exist, what does it actually look like, and how much of it do you genuinely need to write?
This is a homelab running 31 containers across Proxmox hosts, backed by Synology storage. It has been running for years. It has no RTO requirement, no SLA, no team who might need to execute the recovery in your absence. The standard is self-set, which in practice means undefined. That's both the problem and the design space.
Procedure-level vs. architecture-level: the core tradeoff
There are two recognisable approaches to recovery documentation and most operators land somewhere between them without consciously choosing.
Procedure-level documentation is step-by-step. "Boot from USB. Configure storage pools in this order. Pull compose files from backup. Run docker compose up -d in this sequence." It survives a 3am incident when your mental capacity is reduced and the system you're recovering is the thing that hosts your notes. It also goes stale the fastest, any change to the environment that isn't reflected in the procedure introduces a gap, and gaps compound.
Architecture-level documentation is systems understanding. It describes what the environment is, how it's connected, what depends on what. A competent operator who reads it can figure out the procedure themselves. It ages better because the architecture changes less often than the configuration. It fails badly under pressure because "figure out the procedure" is cognitive work you don't want to do at 3am when something is broken.
In enterprise environments, this choice is often made for you. RTO requirements demand procedure-level specificity because the person executing the runbook may not be the person who built the system. The bar is "a junior ops engineer can execute this at 2am", which forces you to write it down in enough detail that it actually works. Homelab has no equivalent forcing function. The standard is whatever you set, and most operators set nothing.
The honest answer is that architecture-level documentation is what most people have , because it's faster to write and feels like enough, and procedure-level is what actually survives a crisis. The question isn't which philosophy is correct. It's which parts of the environment need which level.
Not everything earns the same treatment
Trying to write procedure-level runbooks for 31 containers is the kind of project that doesn't get finished. The maintenance burden alone would be unsustainable. The better approach is triage , and triage requires a clear criterion.
The criterion is: what's the consequence of getting this wrong under pressure? Not "how important is this service to daily use" , that biases toward things you'd notice immediately, which aren't necessarily the things most dangerous to recover incorrectly. The criterion is consequence of failure during recovery specifically.
By that logic, Vaultwarden gets procedure-level documentation first. Its recovery procedure is entirely separate from the backup job that covers the rest of the environment, it's non-obvious in ways that will hurt you if you discover them during a real incident, and it contains the credentials you need to access everything else. If you get Vaultwarden recovery wrong, or discover mid-incident that you don't actually know the procedure, the cascading effect is severe. Everything else can wait while you figure it out. Vaultwarden cannot.
The next tier is anything whose recovery has non-obvious dependencies or gotchas , the things where "someone technically competent who's never seen this environment" would genuinely get stuck. In a Proxmox setup: storage pool naming and ID mapping on restore, especially if the target node differs from the source. In Docker: named volume paths, which must match exactly or the container starts against an empty filesystem; custom bridge network names that don't reconstruct automatically; environment variables from .env files that look unimportant and frequently aren't backed up. These aren't complex procedures , they're landmines that require documentation only because they're non-obvious, not because they're hard.
Architecture-level documentation is appropriate for the rest , the services that a competent operator could figure out from the compose files and a list of what was running. You still need to write it, but you don't need to write it at procedure depth.
In enterprise, this triage maps to business impact and blast radius ,the services where a recovery failure creates the most downstream exposure get the most detailed runbooks. The criterion differs but the logic is identical: not equal coverage, tiered coverage based on the cost of failure.
Where IaC earns its documentation claim and where it doesn't
A well-structured compose file is a form of documentation. It declares what was intended. An operator who has never seen the environment can read it, understand the declared state, and reconstruct most of the running system from it. That's real value and it's worth investing in.
The documentation claim breaks in two places. First, anywhere the running state has drifted from the declared state , interactive docker exec sessions that set environment variables, manual network configuration changes, volume mounts adjusted outside the compose context. In a homelab that has been running for years, the drift between compose file and production state is almost never zero. The compose file documents intent. It doesn't document what's actually running.
Second, it breaks anywhere the implicit reasoning matters. The compose file captures what was configured. It doesn't capture why. Why network_mode: host is set on that specific container and what breaks if you remove it during a recovery. Why that particular version pin is there and what the upgrade path looks like. Why Traefik has those labels in that exact form. A compose file that makes sense to the person who wrote it can be genuinely opaque to the person recovering at 3am , including the person who wrote it six months ago.
The practical approach is to treat compose files as the foundation, not the ceiling. They cover structure and declared config. Annotated comments within compose files cover the non-obvious reasoning. A separate document covers the procedure for anything the compose file can't communicate: start order, gotchas, the specific things that will be wrong if you don't know about them.
Homelab vs. enterprise shifts here on maintenance. Enterprise change management creates a forcing function , the compose file in version control gets reviewed when things change. Homelab is you remembering. The gap between compose file and production state grows faster in homelab environments not because homelabs are less disciplined by nature, but because there's no external mechanism to close it.
Keeping it current without making it a second job
Staleness is the real enemy of procedure-level documentation. A runbook that's six months out of date is worse than no runbook in some ways, it creates false confidence and adds cognitive load when you discover mid-recovery that the steps don't match what you're seeing.
The least sustainable approach is discipline-based freshness: reviewing and updating docs on a schedule. It requires remembering, it competes with everything else, and it has no forcing function. It works until it doesn't.
The more durable approach is change-triggered updates. Not a review schedule, a rule: when the environment changes, the doc changes before the change is considered complete. This pairs naturally with composing changes through version-controlled files, because the commit is the natural moment to update the recovery note. It doesn't require scheduling; it requires treating the doc as part of the definition of done for any configuration change.
For the things that genuinely don't change , the Proxmox storage topology, the Synology share layout, the core network architecture , a lighter review cadence is reasonable. The problem is knowing which parts of the environment fall into that category. The honest answer is you find out by asking: "if I had to rebuild this from the doc right now, would the doc be right?" That question surfaces staleness faster than a calendar reminder.
Enterprise solves this through post-incident review and change management processes that have runbook updates as a required output. Homelab needs a substitute, and the most reliable substitute is making the update cost low enough that you actually do it at the time of change, rather than batching it into a review that never happens.
The validation test you probably haven't run
There is one reliable way to know whether the recovery documentation is good: use it. Not in a real incident, that's the wrong time to discover it's incomplete, but in a deliberate test on a non-critical service.
Pick the least consequential service in the environment. Something that going offline for an hour causes no real problem. Write the recovery procedure for it. Then follow the procedure exactly, without improvising or filling gaps from memory. Treat it as if you are the person who has never seen this environment. Note every place where the procedure is incomplete, assumes knowledge it didn't document, or tells you to do something that isn't quite right.
The gaps you find will be embarrassingly consistent across every service in the environment. The same categories of missing information appear everywhere: named volume paths, environment variable sources, network configuration, service-specific gotchas. One test tells you most of what you need to know about the quality of documentation everywhere.
This is the homelab equivalent of an enterprise game day, smaller in scope, achievable alone, actionable from the output. Enterprise runs these on a schedule and produces formal reports. Homelab runs this once and learns what the common failure modes are. The value-per-hour is high; the investment is one afternoon on something low-stakes.
If you won't run the test, the documentation isn't really finished. The writing phase and the testing phase are both part of the procedure. A runbook that's never been executed is a hypothesis, not a procedure.
What to carry into the build
The decisions this post walks aren't a one-time exercise, they define what recovery documentation is in your environment and how much it can be trusted. A few positions worth carrying forward: triage by consequence of failure, not by service importance; use compose files as the structural foundation, not the documentation ceiling; treat change-triggered updates as part of the definition of done; and validate at least once with a real restore test before trusting any procedure you've written.
How you execute against those positions in the specific context of a 31-container Proxmox and Synology environment, that's what the Build post is for.
