Giving My Kubernetes Cluster Its Own DNS Identity
- 10 minutes read - 2013 words - Page Source(Disclaimer - as with previous posts, this post was originally based on output generated by an LLM which I used to investigate and resolve the problem, though every word of it has been reviewed by my actual human brain, and meaningful edits were made)
I noticed something odd in my AdGuard Home dashboard: one host — culex1, a Kubernetes control-plane node — appeared to be responsible for nearly 80% of all DNS queries on my network. My first thought was that something had gone wrong: a runaway process, a crashlooping pod making repeated lookups, or even something more sinister.
The reality was more mundane, though a neat opportunity to understand my tools a little better.
Why Does This Happen?
Kubernetes clusters use CoreDNS as their internal DNS resolver. Pods don’t talk to your home DNS server directly — they talk to CoreDNS, which handles internal cluster names (like my-service.my-namespace.svc.cluster.local) and forwards everything else upstream to your real DNS server2.
Pods run in their own network namespace, with their own IP addresses on a virtual overlay network3. When a pod sends a packet to the outside world — including a DNS query to AdGuard — the kernel applies a SNAT (Source Network Address Translation) rule that rewrites the source IP from the pod’s virtual IP to the physical IP of the node it’s running on. So from AdGuard’s perspective, every DNS query from every pod in my cluster looks like it’s coming from the node that runs CoreDNS.
This is completely normal behaviour. It’s not a bug or a misconfiguration — it’s how pod networking works. But it does make AdGuard’s client statistics less useful: I can’t tell whether a DNS query originated from a pod, from the k3s/kubelet process, or from the OS on culex itself.
The Goal
Give Kubernetes cluster DNS queries their own IP address, so AdGuard can display them as a distinct client — separate from culex-the-host.
The approach has three parts:
- Add a virtual IP alias to
culex’s network interface — a second IP address the host responds to, which we’ll use exclusively for cluster DNS traffic - Add an iptables SNAT rule that rewrites DNS packets from the cluster’s pod CIDR to use our new alias IP as the source, intercepting them before Flannel’s generic masquerade rule
- Register the alias IP in AdGuard Home as a named client
Understanding the Packet Flow
Before making changes, it helps to visualize what happens to a DNS packet from a pod:
Pod (10.42.x.y)
└─► CoreDNS (10.42.1.105) via cluster DNS (10.43.0.10)
└─► iptables POSTROUTING chain:
├─ [our rule] src=10.42.0.0/16, dst=AdGuard:53 → SNAT to 192.168.1.200
└─ FLANNEL-POSTRTG: src=10.42.0.0/16 → MASQUERADE to 192.168.1.13
└─► AdGuard Home (192.168.1.1)
The key insight is that iptables rules are evaluated in order, and the first matching rule wins. By inserting our specific rule before Flannel’s catch-all masquerade, DNS traffic from the cluster takes our path (→ 192.168.1.200) while all other pod traffic continues to use the standard masquerade (→ 192.168.1.13).
The Process
I’m using all my own values below - culex, 192.168.1.13 or .200, 10.42.0.0/16 for my pod IP range, etc. Substitute your own if you’re following along!
Step 1: Add an IP Alias
The interface on culex is eno1. The alias IP I chose is 192.168.1.200 — a static address outside the DHCP range.
There’s a subtlety here worth understanding: eno1’s primary address (192.168.1.13) is DHCP-assigned. If you add the alias via NetworkManager (nmcli connection modify eno1 +ipv4.addresses 192.168.1.200/24), NM treats the explicitly-configured static address as the primary and demotes the DHCP address to secondary. This matters because:
- Linux’s MASQUERADE target uses the interface’s primary address
- If
.200becomes primary, Flannel would masquerade all pod traffic to.200, not just DNS - The culex host’s own outbound connections would also source from
.200
That defeats the whole point — everything would appear as .200 and we’d have no separation at all. I made that mistake initially!
The fix is to add .200 directly with ip addr add after the DHCP address is already in place. Linux automatically marks any new address on an already-occupied subnet as secondary, meaning the kernel won’t spontaneously use it for outbound traffic:
sudo ip addr add 192.168.1.200/24 dev eno1
Verify the ordering is correct:
ip addr show eno1 | grep inet
You want to see:
inet 192.168.1.13/24 ... scope global dynamic ... eno1
inet 192.168.1.200/24 ... scope global secondary eno1
.13 primary, .200 secondary. The secondary flag is load-bearing — it’s what ensures .200 only appears as a source when our SNAT rule explicitly forces it.
Making the Alias Persistent
Since we can’t use NM to manage this address (it would become primary), we use a NetworkManager dispatcher script that re-adds it idempotently whenever eno1 comes up:
sudo tee /etc/NetworkManager/dispatcher.d/99-culex-dns-alias << 'EOF'
#!/bin/bash
INTERFACE="$1"
EVENT="$2"
if [ "$INTERFACE" = "eno1" ] && [ "$EVENT" = "up" ]; then
ip addr show eno1 | grep -q "192.168.1.200" || ip addr add 192.168.1.200/24 dev eno1
fi
EOF
sudo chmod +x /etc/NetworkManager/dispatcher.d/99-culex-dns-alias
The grep -q ... || ip addr add pattern means “add the address only if it’s not already there” — safe to run multiple times.
Step 2: Add the iptables SNAT Rules
We need one rule each for UDP and TCP on port 53 (DNS uses UDP by default but falls back to TCP for large responses, e.g. DNSSEC):
sudo iptables -t nat -I POSTROUTING 1 \
-s 10.42.0.0/16 -d 192.168.1.1 -p udp --dport 53 \
-j SNAT --to-source 192.168.1.200
sudo iptables -t nat -I POSTROUTING 2 \
-s 10.42.0.0/16 -d 192.168.1.1 -p tcp --dport 53 \
-j SNAT --to-source 192.168.1.200
Inserting at position 1 (rather than just before FLANNEL-POSTRTG) is intentional — see the Gotchas section for why. Verify:
sudo iptables -t nat -L POSTROUTING -n --line-numbers
You should see the two SNAT rules near the top, with FLANNEL-POSTRTG appearing later in the chain:
1 SNAT 17 -- 10.42.0.0/16 192.168.1.1 udp dpt:53 to:192.168.1.200
2 SNAT 6 -- 10.42.0.0/16 192.168.1.1 tcp dpt:53 to:192.168.1.200
...
N FLANNEL-POSTRTG 0 -- 0.0.0.0/0 0.0.0.0/0 /* flanneld masq */
Making the Rules Persistent
iptables rules are lost on reboot. The naive approach — iptables-save + a restore service — has a timing problem: k3s creates the FLANNEL-POSTRTG chain at startup, and if our restore runs before k3s, the chain ordering could end up wrong on the next reboot.
The clean solution is a systemd oneshot service that runs after k3s, using -C (check) before each -I (insert) to stay idempotent:
sudo tee /etc/systemd/system/k8s-dns-snat.service << 'EOF'
[Unit]
Description=SNAT cluster DNS traffic to dedicated alias IP
After=k3s.service
Requires=k3s.service
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/bash -c '\
iptables -t nat -C POSTROUTING -s 10.42.0.0/16 -d 192.168.1.1 -p udp --dport 53 -j SNAT --to-source 192.168.1.200 2>/dev/null || \
iptables -t nat -I POSTROUTING 1 -s 10.42.0.0/16 -d 192.168.1.1 -p udp --dport 53 -j SNAT --to-source 192.168.1.200; \
iptables -t nat -C POSTROUTING -s 10.42.0.0/16 -d 192.168.1.1 -p tcp --dport 53 -j SNAT --to-source 192.168.1.200 2>/dev/null || \
iptables -t nat -I POSTROUTING 2 -s 10.42.0.0/16 -d 192.168.1.1 -p tcp --dport 53 -j SNAT --to-source 192.168.1.200'
[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable k8s-dns-snat.service
One thing worth knowing: after running systemctl enable, checking systemctl status shows inactive (dead) — which looks alarming but is correct. A Type=oneshot service with RemainAfterExit=yes is “inactive” until it has actually been run; it will activate on the next boot (after k3s starts). The rules are already in place from when we added them manually earlier, so nothing is broken in the meantime.
Step 3: Register the Alias in AdGuard Home
In AdGuard Home, navigate to Settings → Client Settings → Add Client:
- Name:
k8s-cluster - Identifier:
192.168.1.200
Now DNS queries that appear to come from 192.168.1.200 will be shown as k8s-cluster in the query log and statistics — cleanly separated from culex’s own host-level DNS activity.
Verification
Trigger some cluster DNS activity and check AdGuard’s query log:
kubectl run -it --rm dns-test --image=busybox:1.36 --restart=Never -- nslookup example.org
You should see the query attributed to k8s-cluster (or 192.168.1.200 if you haven’t added the client yet) rather than culex.
After a reboot, verify both pieces survived:
# Alias IP should still be secondary
ip addr show eno1 | grep inet
# SNAT rules should be at positions 5 and 6
sudo iptables -t nat -L POSTROUTING -n --line-numbers
# Service should now show as active
sudo systemctl status k8s-dns-snat.service
Gotchas
Primary vs. secondary address ordering matters more than you’d think. The secondary flag in Linux networking isn’t cosmetic — it controls whether the kernel will spontaneously use an address as the source for outbound connections. Getting this wrong (by using NM to add the alias, which makes it primary) causes all pod traffic to masquerade to the wrong IP. Always add your alias with ip addr add after the primary address is in place.
Pod CIDR, not just CoreDNS. This approach SNATs all DNS traffic from any pod in the cluster (10.42.0.0/16), not just the CoreDNS pod. That’s intentional — we want all cluster-originated DNS to appear under the alias IP, regardless of which pod initiates it (e.g., pods using hostNetwork: true that bypass CoreDNS and query AdGuard directly).
Don’t hardcode the insertion position. My first attempt used -I POSTROUTING 5 (just before FLANNEL-POSTRTG). This worked interactively but failed on reboot with iptables: Index of insertion too big — because when our service ran, k3s had started but Flannel hadn’t yet added its jump rule to POSTROUTING, leaving the chain shorter than expected. The fix is to always insert at position 1. Our rule is specific enough (src=pod-cidr, dst=adguard, port=53) that placing it before the other chain entries (CNI portfwd, kube postrouting, Netavark) causes no interference — they handle completely different traffic.
Rule ordering vs. k3s restarts. k3s reinjects its iptables rules when it restarts, but it doesn’t flush other rules — so our SNAT rules survive k3s restarts as long as they were already in the chain. The After=k3s.service in our systemd unit handles the boot ordering case.
Why Bother?
Mostly, because aesthetically I wanted the AdGuard stats to be more meaningful - I don’t realistically think that this distinction will help me much in my homelabbing.
More practically, though, this was an opportunity to learn concretely about how some aspects of networking - especially Kubernetes pod networking - actually works; SNAT, iptables chain ordering, the primary/secondary address distinction, all that jazz. In particular, it was an eye-opening observation (though retroactively obvious) that culex would “see” packets from k3s pods as originating “from” 10.42.0.0 IPs - the idea of pods having IP addresses hadn’t occurred to me prior to this, as I only ever thought of them as being “within” a node, but of course if they’re addressable via the network they must have an IP even if they’re local!
Coda: CoreDNS Cache TTL
After getting attribution working, the cluster DNS traffic was now clearly visible in AdGuard — and it was still a lot. CoreDNS does cache upstream responses, but k3s ships with a default of cache 30 (30 seconds). For stable records that rarely change, that means a fresh upstream query every half-minute per unique hostname per pod.
Bumping it to 5 minutes is a one-line change to the CoreDNS ConfigMap:
kubectl edit configmap coredns -n kube-system
# change: cache 30
# to: cache 300
Then restart CoreDNS to pick it up:
kubectl rollout restart deployment/coredns -n kube-system
For a homelab where DNS records change infrequently, this is a free win. The only tradeoff is that if you do change a DNS record, pods won’t see the update for up to 5 minutes — which is almost never a problem in practice.
Note that this only affects external lookups forwarded upstream to AdGuard. Intra-cluster service names (my-service.my-namespace.svc.cluster.local) are resolved directly by CoreDNS’s kubernetes plugin and never touch AdGuard regardless of the cache setting.