A couple of days ago, the Hard Drive that I’d initially been using as my NAS (before investing in a beefy TrueNAS setup) failed. Since my homelab Kubernetes cluster predated the TrueNAS, this was the drive on which I’d stored configuration files for the cluster itself, which meant the cluster immediately went down. I probably could have salvaged it while flying the plane, but this was a great opportunity to start again from scratch using the various lessons I’d learned over the years.

You can follow along with the setup here.

What’s changed

Out with Gitea, In with GitHub

I still resonate with the reasons why I chose to self-host Gitea in the first place - one of my primary motivations for self-hosting is a(n admittedly-symbolic) gesture against centralization of technical tools. That said - as detailed here - there are pretty convincing reasons to compromise on that particular point. It makes me sad, but it is what it is; gotta pick your battles.

High-Availability Cluster

Although I don’t think I ever found this explicitly stated in k3s’s documentation, I’m 99% sure that it’s not possible to upgrade from a single-control-plane-node cluster to a high-availability one. This was a nice opportunity to do so!

As usual with k3s, installation couldn’t have been simpler - just one prerequisite step of setting up an external Datastore, and an extra --datastore-endpoint parameter to the launch commands, and that was that!

Docker made spinning up Postgres super-easy¹:

# docker-compose.yaml
services:
  postgres:
    image: postgres
    ports:
      - "5432:5432"
    environment:
      POSTGRES_PASSWORD: FILL_IN_YOUR_DESIRED_PASSWORD_HERE
    volumes:
      - /path/to/a-dir/on-your-host-machine-owned-by-root:/var/lib/postgresql/data

# Probably a more-sophisticated way to do this would be with `systemctl`,
# but I don't know much about that other than basic commands yet
# `$ docker compose up -d`

Using IngressRoute for Argo

Per instructions here (more docs here) - although I haven’t yet set up a CertResolver, so used a self-signed certificate.

Really I don’t have any use-case that makes an IngressRoute preferable to a plain ol’ Ingress - not using any middleware or complex Rules, nor observability - but this’d be a nice opportunity to try some of those out if desired!

One cloudflared per application

I found out that the pattern I’d been using for cloudflared previously - one installation that fanned out to various services - was not one envisioned or encouraged by the developers. For now, I’ve semi-manually created a deployment².

Ideally, of course, I’d use Vercel’s CDN, but it doesn’t currently support the ability to serve from a locally-running application.

Everything³ in app-of-apps

Since I was learning about Kubernetes tooling as I went, you can chart my learning by looking at the sophistication of my configuration mechanisms - from cat-ing config files straight to kubectl apply, through kubectl apply -f <url>, via k8s manifests intended to be installed via Argo⁴, into Helm Charts, finishing off with a (beautiful, if I do say so myself) abstraction of common installation patterns so they can be expressed in just a handful of lines in a single directory that Argo will scan app definitions. Upgrading any of these applications to “the new way” always felt like more trouble than it was worth; but now I’m spring-cleaning, everything can be cleaner and more uniform!

Everything on TrueNAS, no Longhorn

I experimented with Longhorn a couple of years back and found it frankly unsatisfactory. When it worked, it worked pretty well; but when it went wrong, the errors were inscrutable, and even when you knew what to do, recovery was difficult and time-consuming. There were several nights when I stayed awake for hours, shuffling replicas around and whack-a-moling erroring pods. Maybe I’d done something wrong in my setup, who knows; but I’m not sad to see it gone.

Next priorities

• Jellyfin. The only service that matters, as far as the rest of my household’s concerned! Done.
  • A very important prerequisite for that, which I’m glad I remembered first - installing the TrueNAS-linked Storage Classes! Thankfully, that was pretty easy - I just followed the instructions linked from the main GitHub (explicitly - this (though that was mostly about setting up TrueNAS itself, which I didn’t have to do this time around) and this). I had to do a bit of finangling on the config - see everything from here to here - but not too much.
  • Interestingly, after Jellyfin was set up, it consistently asked me for a password when I hit it via Domain Name or k port-forward to the service, but when I got in with a direct port-forward to the pod the wizard ran fine. Odd! I guess it makes sense for that to be blocked from non-local addresses, though - shouldn’t be giving arbitrary people setup permissions just because you accidentally left it exposed on reinstall!
  • One slight hiccup in that the config directory for my Jellyfin library contains metadata for the library - so, with my digital-packrat tendencies, required a larger PV than the 100M I’d originally allocated. Weird flex, but I’ll take it.
• Backups of data from the k3s HA postgres database. It would be ironic and frustrating to lose this cluster in a similar way to the previous one!
• Double-checking backups from the TrueNAS. I get regular confirmations that they’re running, but you know what they say about untested backups…
• A LetsEncrypt Cert Resolver⁵
• Set up a catch-all website on gitea.scubbo.org acknowledging that it’s gone away
  • I might do a find-and-replace on blog posts that mention it. Or, that’d be a cool use-case for some kind of rendering built-in by Hugo itself…
• Automation (probably via Operators, which I haven’t yet played with; or Crossplane, which I have, and love) for:
  • Updates on my router’s UnboundDNS for certain Ingresses⁶
  • per-service Cloudflared tunnel
• Reinstallation of Vault and the GitHub Token Plugin - I’ve hard-coded a secret to allow this blog’s CD pipeline to succeed, but I don’t want to do that any more than once. I’m really hoping that the installation will magically come right back up again, pre-configured, when I reconnect to the PVC - fingers crossed… (Spoiler alert - it didn’t…)
• Monitoring. Grafana was one of the first systems I tried to install on the Homelab, but newbie that I was, I couldn’t really get my head around it⁷.

• Cloudflare-Tunnels
• Homelab
• K8s
• Meta
• Starting-Over