Certificate Rotation in Self-Hosted Infrastructure — Silent Failures and What to Do Before One Finds You

There's a version of this story that ends with a late-night page, a Kubernetes control plane that won't accept connections, and a cluster that looks up but is effectively down. The certificates expired. Nobody set a reminder. The setup was configured once, at install, and then left to run.

That story plays out more than it should. Certificate expiry is one of the most preventable outages in self-hosted infrastructure and one of the most reliably overlooked. Not because it's complicated — the underlying mechanics are well understood. But because in most self-hosted environments, there's no system that reminds you the clock is ticking until the clock has already run out.

This post isn't about certificate management theory. It's about the decision that nobody's documentation quite covers: when you're running your own CA, managing your own Kubernetes cluster, and you can't just drop cert-manager in and call it done — what do you actually do, and why does the answer depend on factors that most posts don't mention.

The Situation

In cloud-managed Kubernetes, the control plane certificates are someone else's problem. In most managed distributions, certificate renewal is automatic, invisible, and not a thing you need to think about. The tradeoff is that you're inside someone else's operational model.

In self-hosted environments — RKE2, k3s, bare metal clusters, Proxmox nodes running internal services — you own the full PKI chain. That means you issued the CA. You issued the leaf certificates. You configured the trust anchors. And unless you built something to watch them, you are the only thing standing between those certificates and their expiry date.

Kubernetes control plane certificates, by default, are issued with a one-year validity period. The cluster renews them automatically at startup if they're within 90 days of expiry — but only on restart. A cluster that's been running without a restart for over a year will have certificates that have quietly crossed the expiry line while everything appeared normal. When the certificates actually matter — when you try to hit the API server, when a sidecar needs to verify identity, when a new node tries to join — you find out.

That's the first failure mode. The second is more subtle.

Where I Started

The reasonable first instinct is cert-manager. It's the standard answer to certificate automation inside Kubernetes, it supports multiple issuers, and the documentation is thorough. For environments with outbound access to ACME endpoints or an accessible internal CA, it's a solid fit. You define a Certificate resource, cert-manager watches the validity window, and renewal happens automatically before expiry.

The problem shows up at the boundary conditions.

In air-gapped environments, cert-manager's default issuer integrations assume network reachability that doesn't exist. The Vault issuer works if you're running Vault, and getting Vault into an air-gapped environment is its own project. The CA issuer works for self-signed or internally-issued certificates, but you're now managing your CA credentials as a Kubernetes secret, which means the security model of your PKI is as strong as your secret management — and in many self-hosted setups, that's not a position you want to be in.

There's also the question of scope. cert-manager manages certificates for workloads running inside the cluster. It does not manage the Kubernetes control plane certificates themselves — the API server cert, the controller manager cert, the scheduler cert, the etcd peer certificates. Those are managed by kubeadm, or by your distribution's equivalent tooling. They live outside cert-manager's reach.

This is the piece that catches people. They see cert-manager handling ingress certificates and assume the rotation problem is solved. It isn't. The workload certificates are rotating. The control plane certificates are not.

What Didn't Work

The first attempt at a rotation process in this kind of environment usually looks like: find the documentation for your distribution, run the renewal command, confirm the new expiry. For RKE2, that's rke2 certificate rotate. For kubeadm, it's kubeadm certs renew all. The commands exist. They work. The problem is that they only run when you remember to run them.

A cron job is the obvious next step. Schedule the renewal command, log the output, move on. This is where the failure modes get more interesting.

Certificate rotation for Kubernetes control plane components usually requires the components to restart to pick up the new certificates. Some distributions handle this automatically; some don't. Running the renewal command without a subsequent restart leaves you in a state where the files on disk are rotated but the running processes are still holding the old certificates. Everything looks fine until the next time those processes check their in-memory certificate — which, depending on the implementation, might be never until the next restart.

mTLS between sidecars adds another layer. In a service mesh environment, the sidecar proxies handle their own certificate lifecycle, often through their own control plane (Linkerd's identity component, Istio's istiod). Those certificates are typically short-lived — hours to days rather than months — and rotation is automatic. But the root CA that signs those leaf certificates has a much longer validity period and is not automatically rotated. When that root CA expires, the entire trust chain breaks. Sidecars stop accepting connections from each other. Services that were communicating fine yesterday start refusing connections today. The error messages are often not intuitive — you see connection refused or certificate validation failures rather than "root CA expired."

The problem with the sidecar failure mode specifically is that it can start surfacing before the certificate technically expires. If the trust anchor isn't being refreshed in the sidecars' trust bundle and the root CA is approaching expiry, some implementations will start rejecting connections early. You get failures that look like networking problems, not certificate problems. The investigation goes in the wrong direction for a while before someone checks the cert dates.

The Shift

What changed the thinking here was separating the rotation problem into three distinct layers, each of which has different tooling, different risk profiles, and different rotation windows:

Control plane certificates — annual validity, distribution-specific renewal tooling, requires component restart. The risk here is operational: expired certificates break cluster administration and control plane communication. The mitigation is scheduled rotation with a tested procedure, not automation for its own sake.

Workload and ingress certificates — variable validity, managed by cert-manager or similar, generally well-covered in connected environments. The risk in air-gapped environments is that the renewal mechanism itself has no path to an external issuer, so the CA configuration and the secret management around it become critical.

Service mesh trust anchors — long-lived but easy to forget. The rotation procedure for these is usually documented in the service mesh's operational guides, but it's non-trivial and the consequences of getting it wrong are service-wide. Linkerd's trust anchor rotation, for example, requires coordinated steps — you have to add the new root before removing the old one to maintain continuity in the trust bundle across the fleet.

Once the layers were separated, the decision about what to automate and what to run as a scheduled procedure became cleaner. Automation makes sense where the rotation is routine, the failure mode of a bad rotation is contained, and the process is well-understood. Scheduled procedures with human oversight make sense where the rotation touches the control plane, where the failure blast radius is large, or where the environment doesn't support the infrastructure that automation requires.

Where It Landed

The approach that held up in practice looked like this:

For control plane certificate rotation: a tested manual procedure, scheduled on a calendar, run before the 90-day renewal window. Not a cron job. A documented runbook with pre-rotation checks (current expiry dates, etcd health, API server responsiveness), the rotation command, post-rotation verification, and a record of when it was done. The manual step is intentional — control plane rotation is high-stakes enough that you want a human confirming the environment is healthy before and after.

For workload certificates in air-gapped environments: cert-manager with the CA issuer, with the CA key material stored outside the cluster in something with access controls — a secrets manager if one exists in the environment, or a dedicated CA management host if it doesn't. The CA certificate itself is monitored separately for expiry.

For service mesh trust anchors: the rotation procedure documented in the service mesh's own operational guide, scheduled before the anchor's expiry with enough lead time to run the coordinated multi-step process without urgency. Linkerd documents this well for their trust anchor rotation; the key is running it before you're under pressure, because the procedure requires careful coordination and a degraded trust state partway through is normal and expected.

For monitoring: something that alerts before expiry, not at it. openssl x509 -enddate -noout -in <cert> is the primitive. Wrapping it in a script that checks all relevant certificates and alerts at 90 days, 30 days, and 14 days gives enough lead time to schedule rotation without urgency. Prometheus and cert-manager both expose certificate expiry metrics if you have the monitoring infrastructure for them.

The inventory piece is worth calling out separately. You can't monitor what you haven't catalogued. One of the more useful early exercises is building a complete list of every certificate in the environment — control plane certs, etcd certs, service mesh trust anchors, any internal services with their own TLS — with the issuing CA, the expiry date, and the rotation procedure for each. It's tedious. It's also the thing that tells you which rotation problem you actually have.

If You're Navigating This

The first question worth asking is whether the rotation problem you're solving is a tooling problem or an inventory problem. Most environments that get caught by certificate expiry don't fail because they chose the wrong automation strategy. They fail because nobody knew the certificate existed, or nobody knew when it expired, or the rotation procedure was never written down and tested before it was needed.

cert-manager solves a real problem in the right environment. If you're in a connected environment, running cert-manager with ACME or an internal CA issuer, and you're only trying to manage workload certificates — it's the right tool. If you're in an air-gapped environment, or trying to manage control plane certificates, or dealing with service mesh trust anchors, you're in territory that cert-manager doesn't cover.

The tendency in these situations is to reach for more automation. That instinct is worth examining. Automation that runs a rotation procedure you haven't tested is automation that can fail silently in a way that leaves you with rotated files and running processes that haven't restarted. The most expensive certificate failures are often not the ones where nothing rotated — they're the ones where something rotated incorrectly, and the environment appeared healthy until it didn't.

A tested manual procedure on a schedule is not a failure of automation maturity. For high-stakes, low-frequency operations in environments without the infrastructure to support robust automated rotation, it's often the more reliable choice. The goal is not to automate everything — it's to ensure that rotation happens before expiry, that the procedure is understood before it's needed, and that someone is watching the dates.

What failure mode is acceptable in your environment, and are you actually protected against the one that isn't?

That's the question the tooling choices need to answer. The tools are secondary to that.

Technology referenced: cert-manager, RKE2, kubeadm, Linkerd, Prometheus. Validity periods, default behaviours, and tooling capabilities described here reflect current documentation as of early 2026. These things shift — verify against the current docs for your distribution and version before acting on anything here