I had intended to write this weeknotes on the amusing rabbit-hole of yak-shaving I’d fallen down:

As I hoped in my last post, I’ve set up Gitea Actions on my homelab, with a view to completely replacing Drone which I’ve found to be pretty buggy and missing some core features¹. The process was reasonably smooth, but not entirely turnkey, so I’ve laid out the steps I took in the hopes that they’ll help someone else.

I know just enough about computer security to know that I don’t know enough about computer security, so I default to keeping my systems as closed-off from the outside world as possible. I use Cloudflare Tunnels for the few systems that I want to make externally available¹ (like Gitea), and Tailscale to access “internal” services or ssh while on-the-go.

I’ve fallen out of the habit of blogging, recently, due to some personal/family stuff going down. In an effort to kickstart that process again, I’m taking on a smaller task that requires significantly less effortful thought - a rudimentary “Uses” page, inspired by the general practice of listing the stuff™️ used.

I just set up a step in my publication pipeline to automatically post on Mastodon when I publish a new blog post.

In my previous post, I had figured out how to inject Vault secrets into Kubernetes Secrets using the Vault Secrets Operator. My runthrough of the walkthrough worked, but I swiftly ran into namespacing issues when trying to use it “in production”.

Continuing my recent efforts to make authentication on my homelab cluster more “joined-up” and automated, this weekend I dug into linking Vault to Kubernetes so that pods could authenticate via shared secrets without me having to manually create the secrets in Kubernetes.

Setting up regular backup for my Keycloak installation was a lot trickier than I expected!

I just configured OIDC login for the first service on my Homelab.

In my greatest display yet of over-engineering and procrastinating-with-tooling, I’ve started self-hosting OpenProject to track the tasks I want to carry out on my homelab (and their dependencies).

This post is going to be a bit of a meander. It starts with the description of a bug (and appropriate fix, in the hopes of helping a fellow unfortunate), continues on through a re-consideration of software engineering practice, and ends with a bit of pretentious terminological philosophy. Strap in, let’s go!

When working on my personal projects, I typically just push straight to main - opening a PR just to approve it seems entirely pointless, as if I had been able to find any issues in my own work, I wouldn’t wait to do it in a PR! However, this does mean that, if I forget to run any quality checkers (linters, tests, etc.), I won’t find out about it until on: push GitHub Action runs, and even then I might not see the failure until several commits later.

As I’ve discussed before, this blog is hosted on a k3s cluster which runs on 3 Raspberries Pi in a nifty little case. The router that powers our home network is in my partner’s office, with the Pi cluster nearby so that it can benefit from a fast stable wired Ethernet connection.

Observant readers of this blog, refreshing every day desperate for new content, will have noticed that the last blog post - dated 2022-12-31 - actually went live in the middle of January. My k3s cluster, which had always been a bit rickety, finally gave up the ghost in late December, and two of the nodes needed to be fully reimaged before I could start it back up again.

I was surprised to find that there’s not much discussion of putting Kubernetes pods behind a VPN. Given how useful both tools are, you’d think more people would use them in concert.

Since leaving Amazon ~4 months ago and dedicating more time to my own personal projects (and actually trying to ship things instead of getting distracted a few days in by the next shiny project!), I’ve learned a lot more about the Open Source tools that are available to software engineers; which, in turn, has highlighted a few areas of ignorance about CI/CD Pipelines. Emulating Julia Evans, I’m writing this blog both to help lead others who might have similar questions, and to rubber-duck my own process of answering the questions.

I’ve had several instability issues with my Kubernetes cluster recently, and so I wanted to install some monitoring to notify me of incipient issues. I’m already using Grafana dashboards to visualize the state of my cluster (using some of my own hand-crafted dashboards along with some pre-existing Kubernetes-specific ones), but that’s only useful if I happen to be looking at it at the time a problem is happening - it won’t warn me of a brewing problem (and, if the problem results in my VPN becoming unavailable while I’m away from home, that could result in complete disconnection).

I use Cloudflare Tunnels to expose services (like this blog!) to the public Internet while remaining protected by Cloudflare’s infrastructure. While attempting to add a new service, I noticed that there were two steps required:

• Updating the configuration deployed to the tunnel daemon, mapping the internal service to its externally-accessible name
• Updating Cloudflare’s DNS entries to map the external name to the Cloudflare tunnel

Although the first step is easily automated with the cloudflare/cloudflared image, the second isn’t so simple - there’s no single command to update all exposed sites, so the logic would need to parse the config file to determine the set of all sites, and the cloudflared image doesn’t include tools to do so.

Way back in this post, I talked about enabling Analytics Tracking on this blog. I disabled it a while back, as the move to an actually self-hosted blog behind Cloudflare Tunnels (as opposed to an AWS-hosted one) messed that up a bit, and I was more incentivized to have a self-hosted blog without analytics, than vice versa. This post is the story of how I got self-hosting analytics working.

I’ve written before about setting up my ssh config so that I’ll automatically join an existing screen session when ssh-ing to certain hosts, by setting RemoteCommand screen -D -RR -p +

However, this has a couple of issues:

• It will always create a new window within the session, even if an idle window exists. More often than not, I find myself immediately killing the new window and switching to an existing one.
• It doesn’t restrict the rejoin to a named session - in my current usage, I typically only have a single screen session open at once, but that could change!

Part of the self-hosted setup that supports this blog (along with all my other homelab projects) is a Docker Registry to hold the images built and used in the CI/CD pipeline. Recently I tried to install TLS certificates to secure interaction with the Registry, and it was a fair bit harder to figure out than I expected, so I wanted to write it up both for future-me and for anyone else struggling with the same problem.

screen (Wikipedia) is a Unix tool that starts a persistent session on a remote machine, allowing you to detach from that session while keeping any running processes alive. It’s really useful when executing a long-running process over an unstable ssh connection. There are other ways to achieve that aim (like Background Processes), and other features of screen itself (like fitting multiple panels in a single window), but that’s what I primarily use it for.

Despite this blog being initially set up to primarily talk about self-hosting, I’d actually been hosting it on AWS until very recently. This was due to caution - I know just enough about security to know that I know next-to-nothing about security, and so I didn’t want to expose any ports on my own network to the Internet. Instead, I set up an AWS CodePipeline to build the blog and deploy to S3 anytime I pushed a new change. Admittedly, this was a pretty cool project in itself that taught me a lot more about CDK and some AWS services; but it didn’t feel like true self-hosting, even though I wasn’t using anything like Medium or WordPress.

Update: I’m preserving the post below for posterity, but I had the obvious solution in the final sentence - I changed my setup to run Grafana from Docker and mount a folder from my external Hard Drive (I haven’t saved up for a NAS yet!), and now my dashboard definition is persistent across restarts/re-images.

I fully intend to write a full blog-post as a follow-up to my previous post at some point, detailing some of the quirks of this setup and issues that I ran into - but I just got a timely reminder of the importance of checking backups, and wanted to pass it on to you.