Authentication in the New World

Contrary to my hopes in the previous post, Vault did not magically come back up again. I’m not sure why, but the ZVols created on TrueNAS by the previous cluster had truly gone away: thanks to zfs get, I could see that the creation times of all iSCSI ZVols was in the last week or so, i.e. while I’ve been rebuilding, not from the previous cluster. I guess when the StorageClass was deleted, it also wiped out all associated ZVols? That doesn’t quite ring true to me, as the cluster itself was inoperable due to unavailability of the Cluster Config (which was stored on an NFS mount on the drive that failed) - but, eh, can’t argue with results. The data was gone¹, so, time to rebuild!

I documented the process a little better this time around. Installation of the GitHub token was pretty smooth, though I did hit a new roadblock - since I’m using GitHub rather than self-hosted Gitea, the runners don’t have access to Vault (I don’t trust my security well enough to expose Vault to the public Internet!), so I had to set up a self-hosted runner to run the jobs.

Naturally I tried the Helm-based approach first, but got an error while test-installing it (i.e. not from IaC):

$ helm install arc --namespace arc-systems --create-namespace oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set-controller
Error: INSTALLATION FAILED: GET "https://ghcr.io/v2/actions/actions-runner-controller-charts/gha-runner-scale-set-controller/tags/list": GET "https://ghcr.io/token?scope=repository%3Aactions%2Factions-runner-controller-charts%2Fgha-runner-scale-set-controller%3Apull&service=ghcr.io": unexpected status code 403: denied: denied

And, to be honest - it’s nearly 11pm, I’m tired, and the fact that that’s the first oci://-based Helm chart that I’ve encountered means I really don’t want to try debugging that right now; I need a win!

So I adapted the instructions from here and here to make a Dockerfile and ./start.sh that I could dump onto one of the bare nodes:

# Dockerfile

# Note I had to use a more-modern Ubuntu version than the example, due to
# https://old.reddit.com/r/linux4noobs/comments/1crp9f4/node_libx86_64linuxgnulibcso6_version_glibc_228/
FROM ubuntu:20.04

ARG RUNNER_VERSION="2.326.0"

RUN apt-get update -y && apt-get upgrade -y && useradd -m docker
RUN DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
    curl jq build-essential libssl-dev libffi-dev python3 python3-venv python3-dev python3-pip
RUN cd /home/docker && mkdir actions-runner && cd actions-runner \
    && curl -O -L https://github.com/actions/runner/releases/download/v${RUNNER_VERSION}/actions-runner-linux-x64-${RUNNER_VERSION}.tar.gz \
    && tar xzf ./actions-runner-linux-x64-${RUNNER_VERSION}.tar.gz

# Must install tzdata in a special way to make it noninteractive
# https://serverfault.com/a/992421
RUN DEBIAN_FRONTEND=noninteractive TZ=Etc/UTC apt-get -y install tzdata

RUN chown -R docker ~docker && /home/docker/actions-runner/bin/installdependencies.sh

# We need git for the actual operation of the workflow!
# `apt-get install git` from within the workflow gives a permission error - which I guess is expected as it's running
# as `docker`, not `root`
RUN apt-get install -y git

COPY start.sh start.sh
RUN chmod +x start.sh


USER docker
ENTRYPOINT ["./start.sh"]

# start.sh
#!/bin/bash

cd /home/docker/actions-runner

./config.sh --url https://github.com/scubbo/blog-content --token <my_token>

cleanup() {
    echo "Removing runner..."
    ./config.sh remove --unattended --token <my_token>
}

trap 'cleanup; exit 130' INT
trap 'cleanup; exit 143' TERM

./run.sh & wait $!

, ran it with docker run $(docker build -q .) --network host --name actions_runner, and it seems to be working. At least - if this is published, you’ll know it is!

• CI/CD
• Homelab
• Meta
• Starting-Over
• Vault