Raspberry Pi Temperature Monitoring

As I’ve discussed before, this blog is hosted on a k3s cluster which runs on 3 Raspberries Pi in a nifty little case. The router that powers our home network is in my partner’s office, with the Pi cluster nearby so that it can benefit from a fast stable wired Ethernet connection.

The fans are pretty quiet, but not silent. A couple of weeks back I wondered - are the fans necessary? Our home’s not terribly hot, the Pi’s aren’t in direct sunlight, and it’s not like I’m running intensive workloads - maybe I could disable the fans and give my partner a bit more quiet in her working day? But on the other hand, I didn’t want to fry my precious (and increasingly rare) Pi boards. This sounds like a job for Observability!

Temperature metrics aren’t exported from the Node metrics of the Prometheus stack I’m using (not surprising, as the stack is presumably hardware-independent), but it was a breeze to add them: this article described how to access the Pi’s internal temperature sensors, and with a little finangling I had a teeny-tiny app ready to deploy as a DaemonSet to my nodes (note the ServiceMonitor which tells Prometheus to pick up metrics from this Service). A Prometheus Query of max by (node_name) (avg_over_time(sys_cpu_temp_celsius_degrees[$__interval])) gave a lovely Grafana visualization:

I find it particularly pleasing that you can see a representation of their physical layout - there are 3 Pi’s in a 4-bay case, so rasnu2 is closest to an open space with airflow and rassigma is furthest. Neat!

The temperature’s generally hovering between 35°C and 45°C - that’s pretty reasonable! Since they’re apparently specced to go up to 85°C, there should be no risk of disabling the fan.

Unfortunately, the power to the case-fans comes from an adapter that’s plugged into one of the Pi’s USB-C ports, and I couldn’t find a way to selectively depower it¹, so I had to power off rassigma (the cluster control-plane), physically remove it, and disconnect the power cables to carry out this experiment. Despite using Longhorn Storage Class, Prometheus seems to lose metric history over this disconnection, so I don’t have the data to show the graph, but the temperature only jumped up about 10 degrees - still well within the bounds of safety!

…and then the fans on the PoE hats themselves powered up, which were even louder and more irritating than the case fans, so it was back to the default setup pretty quickly. Ah well - not every experiment has the desired outcome, it’s still a success if you learn something!

• homelab
• observability