Contrary to my hopes in the previous post, Vault did not magically come back up again. I’m not sure why, but the ZVols created on TrueNAS by the previous cluster had truly gone away: thanks to zfs get, I could see that the creation times of all iSCSI ZVols was in the last week or so, i.e. while I’ve been rebuilding, not from the previous cluster. I guess when the StorageClass was deleted, it also wiped out all associated ZVols? That doesn’t quite ring true to me, as the cluster itself was inoperable due to unavailability of the Cluster Config (which was stored on an NFS mount on the drive that failed) - but, eh, can’t argue with results. The data was gone¹, so, time to rebuild!

A couple of days ago, the Hard Drive that I’d initially been using as my NAS (before investing in a beefy TrueNAS setup) failed. Since my homelab Kubernetes cluster predated the TrueNAS, this was the drive on which I’d stored configuration files for the cluster itself, which meant the cluster immediately went down. I probably could have salvaged it while flying the plane, but this was a great opportunity to start again from scratch using the various lessons I’d learned over the years.

Observant readers of this blog, refreshing every day desperate for new content, will have noticed that the last blog post - dated 2022-12-31 - actually went live in the middle of January. My k3s cluster, which had always been a bit rickety, finally gave up the ghost in late December, and two of the nodes needed to be fully reimaged before I could start it back up again.