OIDC on K8s

I just configured OIDC login for the first service on my Homelab.

The first step was picking a provider - but thanks to the awesome self-hosting guide, I’d already narrowed it down to a shortlist, and this post helped me pick Keycloak over Authentik.

Unusually, there’s no Helm chart listed in the getting started guide, but the old standby of Bitnami had an offering (though they did weirdly change the admin username from admin to user, which threw me off at first). Installation via GitOps was a breeze now that I’m using jsonnet to extract common Application setup boilerplate - though I did have to upgrade my ArgoCD installation from 2.7 to 2.10 to make use of valuesObject configuration.

The first application I wanted to integrate was Argo itself¹, and thankfully there’s a step-by-step guide available, which…mostly worked².

It’s not DNS…

I did run into a problem, though - I’d entered an override on my OpnSense router (running Upbound DNS) for keycloak.avril³ pointing to the k8s cluster, so that I could access it from my browser - but, apparently, the pods on the cluster don’t delegate to that resolver, so I got an error Failed to query provider "http://keycloak.avril/realms/avril": Get "http://keycloak.avril/realms/avril/.well-known/openid-configuration": dial tcp: lookup keycloak.avril on 10.43.0.10:53: no such host when trying to login via SSO. At first I tried setting the issuer value in Argo’s oidc.config to http://keycloak.keycloak rather than http://keycloak.avril (i.e. using the k8s internal DNS name for the service), which allowed Argo to talk to Keycloak, but then gave a DNS error when my browser tried to connect to that host. I could have worked around this by also setting a keycloak.keycloak DNS override on the OpnSense Unbound resolver, but that felt hacky - and, besides, I wanted to understand Kubernetes’ DNS setup a little better.

This SO answer looked promising as a way to set overrides for k8s’ CoreDNS - but, since my ConfigMap already had a hosts entry (presumably provided by k3s?):

data:
  Corefile: |
    .:53 {
        errors
        health
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          fallthrough in-addr.arpa ip6.arpa
        }
        hosts /etc/coredns/NodeHosts {
          ttl 60
          reload 15s
          fallthrough
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
        import /etc/coredns/custom/*.override
    }
    import /etc/coredns/custom/*.server
  NodeHosts: |
    192.168.1.70 host1
    192.168.1.71 host2
    ...

I got an error plugin/hosts: this plugin can only be used once per Server Block when trying to add another (I’m not sure why that restriction exists tbh - the docs make it clear that the plugin can be limited to a zone, so it seems reasonable to have multiple entries for multiple zones?). Handily, though, the plugin also allows listing overrides inline, so I was able to add an entry for keycloak.avril and everything worked as-desired!

...
hosts /etc/coredns/NodeHosts {
  192.168.1.70 keycloak.avril
  ttl 60
  reload 15s
  fallthrough
}
...
...

That worked, but still felt hacky. Now I was managing DNS overrides in two places rather than one. The docs do list a forward plugin which looks like it should do what I want - but adding that (and removing the manual override in hosts):

...
 forward avril 192.168.1.1 # My OpnSense router IP
 forward . /etc/resolv.conf
...

…gave a slightly different error failed to get token: Post "http://keycloak.avril/realms/avril/protocol/openid-connect/token": dial tcp: lookup keycloak.avril on 10.43.0.10:53: no such host during callback in the OIDC process. Even opening up the forward to operate for all names (.) failed (though in this case it was back to the Failed to query provider... error).

🤷🏻‍♂️ at some point you’ve just gotta take a working solution (the inlined entry in hosts) and move forwards with it! This duplication isn’t too bad - I doubt there’ll be another system (other than OIDC) where I’ll need both pods and my browser to be able to use the same DNS name. If there is, I’ll return to this problem and try to crack it.

I do also see an import /etc/coredns/custom/*.override line in that configuration, which would be another promising avenue of investigation - and, hey, if I realize my intention of managing Unbound DNS entries via Crossplane, both the Router Overrides and the CoreDNS configuration could be generated from the same source.

• dns
• homelab
• keycloak
• k8s