Self-Hosted Analytics

Way back in this post, I talked about enabling Analytics Tracking on this blog. I disabled it a while back, as the move to an actually self-hosted blog behind Cloudflare Tunnels (as opposed to an AWS-hosted one) messed that up a bit, and I was more incentivized to have a self-hosted blog without analytics, than vice versa. This post is the story of how I got self-hosting analytics working.

(If you just want to jump down to the How To, click here)

Evaluation of options

There are a lot of Self-Hosted Analytics options out there - Plausible, Umami, Shynet, Matomo, the list goes on. I picked Plausible because of their focuses on:

• compliance - ain’t nobody got time to implement GDPR for themselves, they want a ready-built solution (as my old colleagues on the Amazon Music Privacy & Compliance team knew all too well!)
• privacy, anti-capitalism ("We’re not interested in raising funds or taking investment. We choose the subscription business model rather than surveillance capitalism. We’re operating a sustainable project funded solely by the fees that our subscribers pay us. And we donate 5% of our revenue."), and environmentalism ("A site with 10,000 monthly visitors can save 4.5 kg of CO2 emissions per year by switching.")
• lightweight simplicity

And, if I’m being honest, their website was way more impressive than the competition’s, too!

Getting it working

Despite being a primarily subscription-based service, Plausible went the extra mile by providing a Github repo dedicated to self-hosting, complete with a docker-compose file and extra documentation. Sweet! So I can just follow this guide and it will work?

Well, not quite. Plausible has a few dependencies - namely, Bytemark (a mail server), Postgres, and Clickhouse (both databases - the former for configuration options, the latter for actual click/hit-tracking). Sadly, neither Plausible itself nor Bytemark have published images that work on arm64 (the architecture of Raspberries Pi), and the latest commits of Clickhouse and Plausible at the time of this experiment appeared to be broken, too. I am much indebted to Ștefan Stănciulescu who wrote a great blog post on their experience setting up Plausible on a Raspberry Pi, and who helpfully responded and updated their guidance after I reached out with more questions. After building Plausible (at commit 3242327d) and Bytemark on ARM architecture, and pinning the Clickhouse image to tag clickhouse-server:22.3.3-alpine, I was able to get the Docker Compose solution working on a Raspberry Pi. Awesome!

Getting it working on Kubernetes

But I’m a sucker for punishment always keen to push my learning further, so I wanted to get this working on Kubernetes. I faced a few challenges with this:

Configuration upload

The docker-compose solution relies on mounting a couple of configuration files as “volumes”. We can replicate this directly using a ConfigMap.

Pre-chowning

On my first attempt, spin-up of the Postgres and Clickhouse containers would fail with an error message like chown: changing ownership of '<path>': Operation not permitted. Apparently NFS-provided volumes on Kubernetes have stricter permission control than local volumes provided by Docker(-compose).

Fortunately, this was an easy fix - this SO answer suggested using an initContainer (docs) to pre-emptively chown the directory to the container’s user before the container starts. This also seemed to require setting no_root_squash on the NFS export (previously I’d been using all_squash) - thanks to this comment for pointing that out.

depends_on

Docker-compose has a neat feature called depends_on that lets you delay the start of one container until another container is ready and available, and this is used in Plausible’s docker-compose to ensure that the main app doesn’t start until the dependencies are ready. No such built-in feature exists in Kubernetes, but we can hack it in with another use of initContainers, since a pod’s main containers will not start until all its initContainers have successfully completed. So, by creating two initContainers that will not successfully complete until Postgres/Clickhouse (respectively) are available, I can replicate this behaviour in Kubernetes.

I didn’t bother implementing this for the mail container, since so far as I can tell I don’t actually need it - Plausible uses email to go through the sign-up/forgot-password flow for users, but since there’ll only be a single user of the app (me!) and I can (re)set my password manually with config options, I have intentionally left the mail container in an unconfigured (and probably non-functional) state.

ulimits

The docker-compose file sets some limits on the maximal number of open files, and from a quick skim of the issue it seems that this isn’t implement in Kubernetes, so I just…ignored it :D in fairness, this answer suggests that “[t]here is almost no software in existence that can handle more than about 20,000 files open at a time”, and the limit in docker-compose was an order of magnitude higher than that, so probably functionally equivalent to an infinite limit.

Service creation

In Docker-compose, every container is nicely addressable (to other containers) by the container name. In Kubernetes, it’s not so simple, and you need to create a Service to expose containers to one another (or the outside world). Even though I usually hate video tutorials, this guide was the best resource I found for grokking how they work. One really neat feature of Services that I had missed until now is that, if you create a service named foo-service in namespace bar, Kubernetes¹ will set up the following DNS records within the cluster to resolve to that service:

• foo-service or foo-service.svc.cluster.local, for any resource² in bar namespace
• foo-service.bar or foo-service.bar.svc.cluster.local, for any resource

Annoyingly, Plausible’s default configuration assumes that the database is addressable at DNS name plausible_db, which is not a legal Kubernetes service name (underscores are forbidden), so I both had to create a service and provide a configuration option telling Plausible where to find the database (and ditto for Clickhouse).

I ran into a really confusing bug while setting this up, because I had foolishly configured both Pods (the dependencies - Bytemark, Postgres, and Clickhouse - and the Plausible pod itself) with identical labels, so the Service could not distinguish between them and, for reasons I don’t fully understand, seemed to consistently prioritize the main-app pod (where no Postgres/Clickhouse containers were running). Once I figured out what was happening³, it was an easy fix to add distinguishing labels between the two pods and correctly direct the service.

Email verification

By default, when you create a user in Plausible, you cannot log in as that user until the system has sent out a verification email and you’ve responded to it. Ștefan gave me a handy code snippet to bypass Email verification for all registered users of Plausible - psql -U postgres -d plausible_db -c "UPDATE users SET email_verified = true;". This allows you to log in with username and password directly, without verification (and, so, this lets you run Plausible without a properly configured mail server).

Sadly, I haven’t found a way to automate running this command during the Kubernetes deployment - it needs to be run after the Plausible web app runs /entrypoint.sh db init-admin (otherwise no admin user will exist to be modified), but the Plausible web app image lacks psql or any other command-line Postgres interface, and Postgres doesn’t have a built-in HTTP API⁴. If I really wanted to I could probably modify the Plausible app image and add an /entrypoint.sh db set-admin-email-verified method using the Postgrex[sic] connector that the app uses - but I don’t fancy learning a whole new framework (Phoenix) in a whole new language (Elixir) just for that. For now, I’m just running that single command once manually in the Kubernetes-provided shell in the Postgres container.

Secrets

Lots of sensitive information - username, password, email, etc. - are encoded into the plausible-env.conf, which would have ended up as env: variables in YAML definition of pods. Even though my threat model shouldn’t seriously care too much about attackers getting inside my home network (if that happens, I will have much bigger concerns than their access to my blog analytics!), I still wanted to learn the “proper” way to reference secret values.

Turns out this is super easy! A Secret in Kubernetes is a collection of key-value pairs, you can create it with kubectl -n <namespace> create secret generic --from-file=key1=./path_to_file_containing_value_1 --from-file=key2=./path_to_file_containing_value_2 (remember to use echo -n to create the files, to prevent extra newlines being added), and secrets are referenced in env definitions of pods as:

env:
  - name: env_variable_name
    valueFrom:
      secretKeyRef:
        name: secret-name
        key: key

You can even concatenate values, which came in handy when creating the connection string for the Postgres database (which includes the Postgres password)

Cloudflared

It’s no good creating the Plausible app if blog-readers’ browsers can’t report to it! I’ve been using Cloudflare Tunnels to expose my services safely to the Internet (inspired by this blog post). Recently, I’d converted this Tunnel to also be run on Kubernetes (previously, it was running as a systemd service), and working with this new service taught me an important gotcha - updating a ConfigMap does not appear to force a refresh of the Pods in the Deployment, you will need to forcibly delete them so they will be recreated with the fresh configuration!

I haven’t experimented enough to figure out whether I need to include logic in my Tunnel Containers to call cloudflared tunnel route dns ... to set up new DNS names, or if that will happen automagically with new config. That’ll be another interesting thing to trip over in a later project!

Conclusion and next steps

You can see the commit that introduced this application to my Kubernetes cluster here; or, for a more user-friendly view, look at the directory here which contains a README.md.

A lot of the issues that I tripped over when setting up Plausible on Kubernetes were either due to images not being available for ARM64, or due to my own lack of knowledge with Kubernetes. In hindsight, the whole process was pretty straightforward and simple.

Plausible also has a drop-in page to let users opt-out of tracking - I’ve implemented it here. It does seem that Plausible does not respect the Do Not Track header, though with pretty justifiable reasons.

• homelab
• k8s
• meta