(Disclaimer - as with previous posts, this post was originally based on output generated by an LLM which I used to investigate and resolve the problem, though every word of it has been reviewed by my actual human brain, and meaningful edits were made)

I noticed something odd in my AdGuard Home dashboard: one host — culex¹, a Kubernetes control-plane node — appeared to be responsible for nearly 80% of all DNS queries on my network. My first thought was that something had gone wrong: a runaway process, a crashlooping pod making repeated lookups, or even something more sinister.

The reality was more mundane, though a neat opportunity to understand my tools a little better.

Remember when I set up Keycloak for SSO? Well, I’ve switched to Authentik, and it was surprisingly painless.

A couple of days ago, the Hard Drive that I’d initially been using as my NAS (before investing in a beefy TrueNAS setup) failed. Since my homelab Kubernetes cluster predated the TrueNAS, this was the drive on which I’d stored configuration files for the cluster itself, which meant the cluster immediately went down. I probably could have salvaged it while flying the plane, but this was a great opportunity to start again from scratch using the various lessons I’d learned over the years.

Continuing my pattern of writing “week"notes every fortnight. It’s not intentional, I swear, it’s just working out that way!

I had intended to write this weeknotes on the amusing rabbit-hole of yak-shaving I’d fallen down:

As I hoped in my last post, I’ve set up Gitea Actions on my homelab, with a view to completely replacing Drone which I’ve found to be pretty buggy and missing some core features¹. The process was reasonably smooth, but not entirely turnkey, so I’ve laid out the steps I took in the hopes that they’ll help someone else.

I know just enough about computer security to know that I don’t know enough about computer security, so I default to keeping my systems as closed-off from the outside world as possible. I use Cloudflare Tunnels for the few systems that I want to make externally available¹ (like Gitea), and Tailscale to access “internal” services or ssh while on-the-go.

In my previous post, I had figured out how to inject Vault secrets into Kubernetes Secrets using the Vault Secrets Operator. My runthrough of the walkthrough worked, but I swiftly ran into namespacing issues when trying to use it “in production”.

Continuing my recent efforts to make authentication on my homelab cluster more “joined-up” and automated, this weekend I dug into linking Vault to Kubernetes so that pods could authenticate via shared secrets without me having to manually create the secrets in Kubernetes.

Setting up regular backup for my Keycloak installation was a lot trickier than I expected!

I just configured OIDC login for the first service on my Homelab.

I’ve been annoyed sufficiently-often by the fact that there is no single kubectl command to “create a pod, and attach a PVC to it” that I threw together the following script:

This post is going to be a bit of a meander. It starts with the description of a bug (and appropriate fix, in the hopes of helping a fellow unfortunate), continues on through a re-consideration of software engineering practice, and ends with a bit of pretentious terminological philosophy. Strap in, let’s go!

Observant readers of this blog, refreshing every day desperate for new content, will have noticed that the last blog post - dated 2022-12-31 - actually went live in the middle of January. My k3s cluster, which had always been a bit rickety, finally gave up the ghost in late December, and two of the nodes needed to be fully reimaged before I could start it back up again.

I was surprised to find that there’s not much discussion of putting Kubernetes pods behind a VPN. Given how useful both tools are, you’d think more people would use them in concert.

I’ve had several instability issues with my Kubernetes cluster recently, and so I wanted to install some monitoring to notify me of incipient issues. I’m already using Grafana dashboards to visualize the state of my cluster (using some of my own hand-crafted dashboards along with some pre-existing Kubernetes-specific ones), but that’s only useful if I happen to be looking at it at the time a problem is happening - it won’t warn me of a brewing problem (and, if the problem results in my VPN becoming unavailable while I’m away from home, that could result in complete disconnection).

I use Cloudflare Tunnels to expose services (like this blog!) to the public Internet while remaining protected by Cloudflare’s infrastructure. While attempting to add a new service, I noticed that there were two steps required:

• Updating the configuration deployed to the tunnel daemon, mapping the internal service to its externally-accessible name
• Updating Cloudflare’s DNS entries to map the external name to the Cloudflare tunnel

Although the first step is easily automated with the cloudflare/cloudflared image, the second isn’t so simple - there’s no single command to update all exposed sites, so the logic would need to parse the config file to determine the set of all sites, and the cloudflared image doesn’t include tools to do so.

Way back in this post, I talked about enabling Analytics Tracking on this blog. I disabled it a while back, as the move to an actually self-hosted blog behind Cloudflare Tunnels (as opposed to an AWS-hosted one) messed that up a bit, and I was more incentivized to have a self-hosted blog without analytics, than vice versa. This post is the story of how I got self-hosting analytics working.

Part of the self-hosted setup that supports this blog (along with all my other homelab projects) is a Docker Registry to hold the images built and used in the CI/CD pipeline. Recently I tried to install TLS certificates to secure interaction with the Registry, and it was a fair bit harder to figure out than I expected, so I wanted to write it up both for future-me and for anyone else struggling with the same problem.