In my previous post, I had figured out how to inject Vault secrets into Kubernetes Secrets using the Vault Secrets Operator. My runthrough of the walkthrough worked, but I swiftly ran into namespacing issues when trying to use it “in production”.

The Problem

The setup can be divided into two parts¹:

• Creation of a Vault Role (with boundServiceAccountNamespaces corresponding with the k8s namespaces that should be permitted to access it) and Policy, and a k8s VaultAuth object telling the Vault Secrets Operator how to access the Vault Role.
• Creation of a VaultStaticSecret (referencing the VaultAuth object) in the app’s -deployment repo, which results in a k8s secret.

As I started trying to extend my initial installation to other apps, I realized that simply adding more k8s namespaces to the boundServiceAccountNamespaces of a single Vault Role would not be a secure solution - it would allow any pods in any of the bound namespaces to access any secret of any of the (other) applications. Ideally, each application-stage (or, equivalently, each k8s namespace²) would have its own resources created, with the Vault Role only accessible from that namespace³.

Why do I care?

You may be wondering why I care about Least Privilege - after all, it’s only my own homelab, surely I know and trust every application that’s running on it? Well, to an extent. I trust them enough to install them, but it still doesn’t hurt to limit their privileges so that any unforeseen misbehaviour - whether deliberate or accidental - has limited impact. More importantly, my primary motivation in running this homelab is to learn and practice technical skills - the tasks don’t have to be entirely practical, so long as they are educational! In fact, as you’ll see shortly, this problem is almost-exactly equivalent to one I’m going to be solving at work soon, so doing this “right” is a good head-start.

The solution

Ideally, I’d be able to automate (via extracted-and-parameterized logic) the creation of these resources as part of the application definition, since many apps will have similar requirements and I want to minimize any manual or imperative setup.

Thankfully, this is pretty close to a problem that I’ve been looking into at work, so I have a solution ready to go - Crossplane, a tool that allows:

• management of “External Resources” (i.e. resources in systems outside Kubernetes, like Vault, Argo, etc.) via Kubernetes objects - i.e. you can declaratively create and update a Kubernetes object (a “Managed Resource”) which represents the External Resource, and the Kubernetes reconciliation loop will keep the External Resource up-to-date.
• “bundling” of resources into Compositions - parameterized and inter-related collections of resources, analagous to Constructs in CDK.

With Crossplane in hand, the solution becomes simple:

• (while wearing my “Platform Team” hat) install a Provider (the interface between Crossplane and an external service) for Vault, and create a Composition which bundles the Vault resources that are necessary for Vault Secrets Operator setup.
• (wearing my “App team” hat) whenever I install an app which requires secret injection, do so alongside a Composite Resource (an instance of a Composition). All from the convenience of a single deployment repo, and with only a few extra lines of configuration!

Walkthrough

You can see the solution here.

XRD

(Lines 1-26) A Composite Resource Definition (or “XRD” - yeah, I know, but Kubernetes had already taken the term “CRD”) is like (in Programming Language terms) the Interface to a Composition’s Implementation, or (in Web Service) the API Spec or schema. It defines how a consumer can invoke a Composition - the name they should use, and the parameters they should pass. Consumers can either invoke this by its name if creating a cluster-scoped Composite Resource, or in a namespaced context via a Claim.

This definition is saying:

• (Lines 6-8) “There’s a Composition that can be addressed as xbaseapplicationinfrastructures.scubbo.org…”
• (Lines 10-12) “…(which can also be addressed by the Claim Name BaseAppInfra)…”
• (Lines 13-25) “…which has only a single version defined, which takes a single string parameter named appName”

It is apparently possible to provide multiple schema versions - but since “new required fields are a ‘breaking change.’” and “Only one version can be referenceable […which…] indicates which version of the schema Compositions use”, I’m not really sure how that is actually useful - and this is borne out by the fact that “Crossplane recommends implementing breaking schema changes as brand new XRDs.”.

Top-level Composition

The only point to note in lines 29-36 is that spec.compositeTypeRef.apiVersion and spec.compositeTypeRef.kind must match the values set on 6, 8, and 14.

Vault Resources

Lines 37-136 define Vault Resources, provided by the Vault Provider. These create a Vault Role, Policy, and KV Secrets Mount roughly as described in the walkthrough. Note the use of patches and transforms to set values in the Managed Resources based on properties of the Claim (the Kubernetes namespace and the parameter appName)

Kubernetes Resource

The Vault Secrets Operator walkthrough also requires the creation of a VaultAuth object (specifying how the Secrets Operator should authenticate to Vault - i.e. which Role to use), and that is not an object provided by the Vault Provider⁴, so I also needed to use the Kubernetes Provider to create an arbitrary Kubernetes object as part of the Composition.

Actual usage

After deploying this Composition to my cluster, actual usage was a doddle:

$ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Namespace
metadata:
  name: example-namespace-for-crossplane-vault-secrets-demo
---
apiVersion: scubbo.org/v1alpha1
kind: BaseAppInfra
metadata:
  name: example-app-base-infra
  namespace: example-namespace-for-crossplane-vault-secrets-demo
spec:
  appName: example-app
EOF
namespace/example-namespace-for-crossplane-vault-secrets-demo created
baseappinfra.scubbo.org/example-app-base-infra created

$ kubectl ns example-namespace-for-crossplane-vault-secrets-demo
Context "default" modified.
Active namespace is "example-namespace-for-crossplane-vault-secrets-demo".

$ kubectl get BaseAppInfra example-app-base-infra
NAME                     SYNCED   READY   CONNECTION-SECRET   AGE
example-app-base-infra   True     True                        29s

$ vault secrets list | grep 'example-app'
app-example-app-kv/    kv           kv_d4b378a7           KV storage for app example-app

$ vault read auth/kubernetes/role/vault-secrets-operator-example-app-role
Key                                 Value
---                                 -----
alias_name_source                   serviceaccount_uid
audience                            vault
bound_service_account_names         [default]
bound_service_account_namespaces    [example-namespace-for-crossplane-vault-secrets-demo]
token_bound_cidrs                   []
token_explicit_max_ttl              0s
token_max_ttl                       0s
token_no_default_policy             false
token_num_uses                      0
token_period                        0s
token_policies                      [vault-secrets-operator-example-app-policy]
token_ttl                           24h
token_type                          default

$ vault kv put -mount app-example-app-kv example-secret key=value-but-make-it-secret
============= Secret Path =============
app-example-app-kv/data/example-secret

======= Metadata =======
Key                Value
---                -----
created_time       2024-05-09T05:53:59.20680794Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

$ kubectl get secrets
No resources found in example-namespace-for-crossplane-vault-secrets-demo namespace.

$ cat <<EOF | kubectl apply -f -
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: vault-kv-app
  namespace: example-namespace-for-crossplane-vault-secrets-demo
spec:
  type: kv-v2
  mount: app-example-app-kv
  path: example-secret

  destination:
    name: secretkv
    create: true
  refreshAfter: 30s
  vaultAuthRef: vault-auth-example-app
EOF
vaultstaticsecret.secrets.hashicorp.com/vault-kv-app created

$ kubectl get VaultStaticSecret
NAME           AGE
vault-kv-app   6s

$ kubectl get secrets
NAME       TYPE     DATA   AGE
secretkv   Opaque   2      23s

$ kubectl get secret secretkv -o jsonpath='{.data.key}' | base64 -d
value-but-make-it-secret

Almost all of the steps above were executed “as if” I was a memeber of the App Team. The Platform Team (or, more accurately, automation owned by the Platform Team, but triggered during Application Creation via the Developer Platform like Backstage) should take care of creating the Namespace, but everything else - creating the BaseAppInfra, populating the Vault Secret, and creating the VaultStaticSecret - are tasks that the App Team can handle.

Next Steps and Further Thoughts

• Unwinding my yak-shaving-stack by another level, my motivation for injecting secrets from Vault was to be able to set up Velero with AWS Credentials so I can back up my PVs to S3. Most of my pods are using my TrueNAS cluster as a persistent storage provider (thanks to this great walkthrough), with RAID for redundancy⁵, so they should be reasonably durable - but, backups are still important!
  • I should probably export the ZFS Snapshots off-site as well. The task stack never ends…
• My system’s getting complex enough that an architecture diagram in the README would be useful - at least, as a reminder to myself of what tools I have running, even if no-one else would be interested!
• Because I’m using an App Of Apps Pattern, I don’t need to create Argo Applications⁶ and Policies for the applications - but, for setups that don’t use that pattern (like, say, my work :P ), those resources should also be part of the Base Infrastructure. Assumable (Vault) Roles for the application itself to use would also be good.
• This setup defines a Composition that any App Team can create (via Claim), but I haven’t looked into how to prevent (non-admin) users from creating arbitrary Managed Resources (outside the scope of a Composition). That is, there’s nothing to prevent a user from using Crossplane to create a Vault Policy that has global access, creating a Vault Role using that Policy that’s available to their namespace, and wreaking havoc. I suspect this would be a use-case for Kyverno, OpenPolicyAgent, or other policy tools.
• Several fields in the Composition are mutually-dependent. For instance, the name of the Vault Role (line 71) must be referenced by the VaultAuth on line 166, and the name of the Vault Policy (line 128) must be assigned to the Role on line 79. I’d love to use cdk8s to define the resources instantiated by Crossplane, so that these dependencies can be made explicit, rather than incidental. As a coworker of mine is fond of proclaiming, “YAML is the assembly language of Cloud-Native” - although it’s a universally-comprehended language that tools can use to communicate, we as human developers should be using higher-level tools and abstractions.
• I’ve still only used the Secrets Operator to inject static secrets. I’d be interested to see how Dynamic Secrets - for secret values which have to change over time, such as TTL’d creds for other services - would work. According to the docs, “updates to existing Secret objects are propagated automatically to Pods that use the data”, which is pretty cool.
  • An alternative would be to use the Vault Sidecar Injector service to directly inject and update Vault secret values into the Pod. At first glance, I’d be averse to running both this and Vault Secrets Operator - I’d prefer to have one-and-only-one way of getting Vault Secrets into Kubernetes, and VSO (plus native Secret mounting) seems to cover all use-cases whereas Vault Sidecare only covers injection (and not setting Secret values as env variables) - but, it’s always good to know the alternative options!

• Crossplane
• Homelab
• K8s
• SDLC
• Vault