saved to localStorage
$ nslookup web-service.default.svc.cluster.local

Life of a DNS Request
in K8s

From pod to CoreDNS and back
Vivek Bhadauria · Engineer @ AWS EKS viveksb007
02 / 17 Agenda

$ cat agenda.md

01

DNS Architecture
in K8s

How pods get their DNS configuration
04 / 17 DNS Architecture

DNS Infrastructure Overview

  • Pod yaml has a field "dnsPolicy" which tells kubelet what goes into /etc/resolv.conf
  • ClusterFirst (default) — resolv.conf points to kube-dns ClusterIP (CoreDNS)
  • Default — Pod inherits node's /etc/resolv.conf directly
Kubernetes DNS infrastructure diagram
05 / 17 resolv.conf

$ cat /etc/resolv.conf

Pod resolv.conf contents
06 / 17 ndots:5

ndots:5 Explained

App resolves: "web-service" — 0 dots, < 5 → try search domains first
1. web-service.default.svc.cluster.local → FOUND!
2. web-service.svc.cluster.local (stop here)
3. web-service.cluster.local
4. web-service.us-west-2.compute.internal
5. web-service (as-is)
✓ Resolved in 1 query — efficient!
App resolves: "api.example.com" — 2 dots, < 5 → try search domains first
1. api.example.com.default.svc.cluster.local → NXDOMAIN
2. api.example.com.svc.cluster.local → NXDOMAIN
3. api.example.com.cluster.local → NXDOMAIN
4. api.example.com.us-west-2.compute.internal → NXDOMAIN
5. api.example.com → FOUND!
⚠ 4 wasted queries before resolving — ndots:5 penalty!
App resolves: "api.example.com." — trailing dot = FQDN, skip search domains
1. api.example.com → FOUND!
✓ Trailing dot bypasses search domains — 1 query!
> press arrow / space to step through (0/6)
07 / 17 Query Types

Three types of DNS queries

Service Resolution

Same ns: web-service
Cross ns: web-service.prod
FQDN: web-service.prod.svc.cluster.local
Returns ClusterIP address

Pod Resolution

Format: <pod-ip-dashed>.<ns>.pod.cluster.local
Example: 10-244-1-4.default.pod.cluster.local
Returns PodIP address

External DNS

CoreDNS proxies to upstream resolvers
VPC+2 for EKS clusters
ndots:5 can cause extra lookups!

02

The Packet Walk

Client to DNS Resolver and back
09 / 17 E2E Flow

End-to-End DNS Resolution Flow

End-to-end DNS resolution flow diagram
10 / 17 Phase 1

Phase 1: From application to wire

  • App calls getaddrinfo() "web-service"
  • libc resolver reads /etc/resolv.conf
  • Checks /etc/hosts first (no match)
  • Constructs UDP packet:
    src=PodIP:random, dst=10.100.0.10:53
  • Packet leaves pod via veth pair
  • Arrives at node's network namespace
Node 1 (Worker)
Client Pod (10.244.2.5)
App: lookup web-service
↓
libc resolver
reads /etc/resolv.conf
checks /etc/hosts (no match)
↓
Construct UDP packet
src: 10.244.2.5:random
dst: 10.100.0.10:53
↓
veth pair
pod ns → node ns
↓
Node 1 network namespace
→ iptables processing
> press arrow / space to step through (0/5)
11 / 17 Phase 2

Phase 2: iptables translates the destination

  • Packet hits PREROUTING -> KUBE-SERVICES chain
  • Rule matches: dst=10.100.0.10, port=53
  • Jumps to KUBE-SVC-xxx (service chain)
  • Random probability selects a KUBE-SEP-xxx (endpoint)
  • DNAT rewrites dest: 10.100.0.10 -> 10.244.0.4
  • Packet routed to actual CoreDNS pod
Node 1 (Worker)
Packet arrives: dst=10.100.0.10:53
↓
PREROUTING
↓
KUBE-SERVICES
match: -d 10.100.0.10 -p udp --dport 53
↓
KUBE-SVC-TCOU7JCQXEZGVUNU
kube-dns service chain
p=0.50
KUBE-SEP-aaa
DNAT to
10.244.0.4:53
remainder
KUBE-SEP-bbb
DNAT to
10.244.0.5:53
↓
After DNAT: src: 10.244.2.5
dst: 10.244.0.4:53
→
Node 2
CoreDNS Pod
10.244.0.4
> press arrow / space to step through (0/5)
12 / 17 Phase 3

Phase 3: CoreDNS processes & returns

Node 2 (Worker)
CoreDNS Pod (10.244.0.4)
Query arrives on port 53
↓
cache plugin (30s TTL)
↓
kubernetes plugin
*.cluster.local?
YES → resolve via K8s API cache
↓
forward plugin
forwards to VPC+2
↓
10.0.0.2
VPC+2 upstream (AmazonDNS)
response crosses
VPC network
→
Node 1 (Worker)
conntrack table
ORIGINAL: src=10.244.2.5, dst=10.100.0.10:53
REPLY: src=10.244.0.4, dst=10.244.2.5
↓
Reverse NAT
src: 10.244.0.4 → 10.100.0.10 (svc IP)
↓
Client Pod receives response
from src=10.100.0.10 (service IP)
Never knows actual CoreDNS pod
> press arrow / space to step through (0/7)
03

CoreDNS Internals

Configuration and plugin chain
14 / 17 Corefile

$ kubectl get cm coredns -n kube-system -o yaml

CoreDNS Corefile configuration
15 / 17 Plugins

Key CoreDNS plugins

kubernetes

Watches Services, Endpoints/EndpointSlices, etc.
Maintains a local cache using informers.
Resolves *.svc.cluster.local names.
Returns ClusterIP for service queries.

forward

Proxies queries to upstream DNS.
Uses node's /etc/resolv.conf.
Handles all external domain lookups.

cache

Default TTL: 30 seconds.
Reduces repeated plugin chain processing.
Reduces load on upstream DNS.

16 / 17 Takeaways

$ echo "Key Takeaways"

Thank you!

Feedback for presentation
Feedback QR Code
Vivek Bhadauria · Engineer @ AWS EKS viveksb007