Kubernetes Network Deep Dive - CoreDNS, kube-proxy, and Istio Working Together

Posted on In

Kubernetes networking is often described as “simple” in its model but complex in its implementation. Once you look under the hood, you’ll find multiple independent components—CNI plugins, kube-proxy, CoreDNS, and optionally a service mesh like Istio—all cooperating to make a Pod-to-Service call work transparently. This post builds a mental model of how they fit together and traces a real request through the entire stack.

The Kubernetes Network Model

Before diving into individual components, the fundamental contract K8s networking must fulfill:

  1. Every Pod gets a unique cluster-wide IP — no NAT between Pods on the same cluster.
  2. Pods can reach any other Pod by IP — across nodes, without knowing which node they’re on.
  3. Agents on a node (kubelet, kube-proxy) can reach all Pods on that node.

This contract is intentionally underspecified — Kubernetes doesn’t implement it; it delegates to a CNI plugin (Flannel, Calico, Cilium, etc.). The plugin is responsible for:

  • Assigning IP addresses to Pods from a per-node subnet (e.g., 10.244.1.0/24)
  • Programming routes/tunnels so traffic between nodes is forwarded correctly
1
2
3
4
5
6
7
8
9
Node A (10.244.1.0/24)          Node B (10.244.2.0/24)
┌─────────────────────────┐     ┌─────────────────────────┐
│  Pod-A (10.244.1.5)     │     │  Pod-B (10.244.2.7)     │
│    eth0 ─── veth ──┐    │     │    eth0 ─── veth ──┐    │
│                   cbr0  │     │                   cbr0  │
│                    │    │     │                    │    │
│               eth0(node)│────▶│               eth0(node)│
└─────────────────────────┘     └─────────────────────────┘
        (VXLAN/BGP/host-gw route programmed by CNI)

Pod-to-Pod traffic stays at L3. The CNI plugin ensures that 10.244.1.5 → 10.244.2.7 is routable without NAT.

kube-proxy: Making Services Work

A Service is a stable virtual IP (ClusterIP) that load-balances to a dynamic set of Pod endpoints. Pods come and go; the Service IP stays fixed. kube-proxy bridges this gap by watching the API server for Service and Endpoints objects and translating the virtual ClusterIP into real Pod IPs.

iptables Mode (default)

For each Service, kube-proxy writes iptables rules in the PREROUTING and OUTPUT chains via KUBE-SERVICES:

1
2
3
4
5
6
PREROUTING → KUBE-SERVICES
  → match dst=ClusterIP:port
    → KUBE-SVC-XXXXX (per-service chain)
      → KUBE-SEP-AAA (30% probability) → DNAT to Pod-1 IP:port
      → KUBE-SEP-BBB (50% probability) → DNAT to Pod-2 IP:port
      → KUBE-SEP-CCC (100% remainder)  → DNAT to Pod-3 IP:port

The kernel handles the DNAT transparently — the application sends to ClusterIP:port, and by the time the packet leaves the host’s network stack it’s already been rewritten to the chosen Pod’s IP. The return path is handled via conntrack (SNAT on the way back to restore the original source).

Limitation: iptables rule count grows O(n) with endpoint count. A service with 1000 endpoints means 1000+ rules evaluated sequentially. For large clusters this creates latency and rule management overhead.

IPVS Mode

IPVS (IP Virtual Server) is a kernel-level L4 load balancer designed for exactly this use case. kube-proxy in IPVS mode:

  1. Creates a virtual server (ipvsadm -A) for each ClusterIP:port
  2. Adds real servers (ipvsadm -a) for each endpoint
  3. Uses iptables only for hairpin/masquerade rules, not for the core load-balancing

IPVS uses hash tables — O(1) lookup regardless of endpoint count. It also supports richer algorithms: round-robin, least-connection, source-hash (for session affinity without iptables conntrack).

NodePort and LoadBalancer

These extend the same mechanism:

  • NodePort: kube-proxy adds rules to accept traffic on a fixed port on every node’s eth0, then DNAT to the service’s endpoints (on any node).
  • LoadBalancer: Same as NodePort, but an external cloud LB (or MetalLB) distributes traffic to node:NodePort first.
1
2
3
External → Cloud LB → Node:30080 (any node)
                          ↓ iptables/IPVS
                       Pod endpoint (any node)

CoreDNS: Service Discovery

kube-proxy deals with the routing half of the problem. But how does a client know the ClusterIP to begin with? That’s CoreDNS’s job.

How CoreDNS Serves K8s DNS

CoreDNS runs as a Deployment in kube-system behind a Service (typically 10.96.0.10). Every Pod’s /etc/resolv.conf is set by kubelet to point to this address:

1
2
3
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

When your app does http://my-svc/api, the resolver expands it:

1
my-svc → my-svc.default.svc.cluster.local (search domain applied)

CoreDNS uses the kubernetes plugin to serve records from the API server (it watches Services and Endpoints via the API):

Query type Example query Response
A (ClusterIP) my-svc.default.svc.cluster.local ClusterIP (e.g., 10.96.1.50)
SRV _http._tcp.my-svc.default.svc.cluster.local port + FQDN
A (Headless) my-headless.default.svc.cluster.local All endpoint Pod IPs
A (Pod) 10-244-1-5.default.pod.cluster.local 10.244.1.5

Headless Services (clusterIP: None) are special — CoreDNS returns the actual Pod IPs rather than a ClusterIP, enabling client-side load balancing (used by StatefulSets for stable DNS names per Pod like redis-0.redis.default.svc.cluster.local).

KubeDNS vs CoreDNS

KubeDNS (the original) was a three-container sidecar: kubedns, dnsmasq (cache), sidecar (health/metrics). It had known memory leak issues and the dnsmasq cache layer introduced bugs. CoreDNS replaced it in K8s 1.11 (GA 1.13) as a single Go binary that’s plugin-driven, easier to configure, and more stable.

The ndots:5 Problem

The default ndots:5 means: if a hostname has fewer than 5 dots, try all search domains before the bare name. For a query to www.google.com (3 dots), the resolver tries:

1
2
3
4
www.google.com.default.svc.cluster.local  → NXDOMAIN
www.google.com.svc.cluster.local          → NXDOMAIN
www.google.com.cluster.local              → NXDOMAIN
www.google.com.                           → answer

This causes 3 unnecessary DNS round-trips on every external lookup. Solutions:

  • Use FQDNs with trailing dot in app config: www.google.com.
  • Set dnsConfig.options[ndots: 2] on latency-sensitive Pods
  • Use NodeLocal DNSCache (a DaemonSet that caches locally to avoid crossing to CoreDNS every time)

Putting It Together: Full Network Flow

Let’s trace a request from Pod-A calling http://backend-svc:8080/api:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Pod-A                    kube-proxy               CoreDNS
  │                      (iptables/IPVS)            │
  │─── DNS query ────────────────────────────────────▶│
  │    "backend-svc.default.svc.cluster.local"        │
  │◀── A record: 10.96.1.50 ─────────────────────────│
  │
  │─── TCP SYN dst=10.96.1.50:8080 ──▶ (kernel netfilter)
  │                      │               DNAT: 10.96.1.50:8080
  │                      │             → 10.244.2.7:8080 (Pod-B)
  │
  │─────────────────────────────────────────────────────▶ Pod-B
  │                                          (via CNI routing)
  │◀────────────────────────────────────────────────────  response
  │                      │               un-DNAT via conntrack
  │◀── response from 10.96.1.50:8080 (appears to come from ClusterIP)

Key insight: CoreDNS gives you the ClusterIP. kube-proxy DNAT’s the ClusterIP to a real Pod. The CNI plugin routes the rewritten packet between nodes. The application sees none of this — it talks to a stable ClusterIP.

Istio: The Service Mesh Layer

kube-proxy + CoreDNS provide basic connectivity and discovery. Istio adds observability, traffic management, and security — without changing application code — by intercepting traffic at the sidecar level.

Architecture

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
┌─────────────────────────────────────────────────────┐
│  Istio Control Plane (istiod)                       │
│  ┌──────────┐  ┌──────────┐  ┌──────────────────┐  │
│  │  Pilot   │  │  Citadel │  │    Galley         │  │
│  │(xDS API) │  │  (certs) │  │  (config valid.)  │  │
│  └──────────┘  └──────────┘  └──────────────────┘  │
└─────────────────────────────────────────────────────┘
         ▲  xDS (LDS/RDS/CDS/EDS)  ▲
         │                         │
┌────────┴──────────┐   ┌──────────┴──────────┐
│  Pod-A            │   │  Pod-B              │
│  ┌─────────────┐  │   │  ┌─────────────┐   │
│  │ App process │  │   │  │ App process │   │
│  └──────┬──────┘  │   │  └──────┬──────┘   │
│    iptables redir │   │    iptables redir   │
│  ┌──────▼──────┐  │   │  ┌──────▼──────┐   │
│  │Envoy sidecar│  │   │  │Envoy sidecar│   │
│  └─────────────┘  │   │  └─────────────┘   │
└───────────────────┘   └────────────────────┘

Traffic Interception via iptables

When Istio injects a sidecar, an init container (istio-init) runs first and adds iptables rules in the Pod’s network namespace:

1
2
3
4
5
6
7
8
# Redirect all outbound traffic to Envoy (port 15001)
iptables -t nat -A OUTPUT -p tcp -j REDIRECT --to-port 15001

# Redirect all inbound traffic to Envoy (port 15006)
iptables -t nat -A PREROUTING -p tcp -j REDIRECT --to-port 15006

# Except: Envoy's own traffic (uid 1337), DNS (port 53)
iptables -t nat -A OUTPUT -m owner --uid-owner 1337 -j RETURN

The application has no idea it’s talking through Envoy. Envoy runs as user 1337, which is excluded from the redirect rules — otherwise it would create an infinite loop.

What Envoy Does

Envoy receives the original destination address via SO_ORIGINAL_DST (the kernel preserves it through REDIRECT). It then:

  1. Outbound: Looks up the destination in its config (received from istiod via xDS), applies load balancing, circuit breaking, retries, mutual TLS, and forwards to the actual upstream.
  2. Inbound: Validates mTLS certs, applies rate limits/authz policies, then forwards to the local app.

The xDS protocol is how istiod pushes config to Envoy:

  • LDS (Listener Discovery Service): “listen on port 15001 for outbound”
  • RDS (Route Discovery Service): “route /api to cluster backend-svc“
  • CDS (Cluster Discovery Service): “cluster backend-svc has these endpoints”
  • EDS (Endpoint Discovery Service): “endpoints are 10.244.1.5:8080, 10.244.2.7:8080“

Istio + kube-proxy: Do They Conflict?

Istio bypasses kube-proxy for service routing. Here’s why:

  • kube-proxy DNAT’s ClusterIP → Pod IP at the node level (before the packet enters the Pod)
  • Istio’s Envoy intercepts traffic inside the Pod’s network namespace

By the time Envoy sees the traffic, kube-proxy has already rewritten the destination if it matched a Service IP. But Istio’s EDS gives Envoy the actual endpoint IPs directly — so Envoy can connect directly to Pod IPs, bypassing kube-proxy’s load balancing entirely.

In practice:

  • kube-proxy is still needed for non-sidecar traffic (DaemonSets without injection, system components)
  • Istio replaces kube-proxy’s load balancing for meshed traffic
  • Cilium + Istio can be configured to disable kube-proxy entirely (eBPF handles the rest)

Full Flow with Istio

1
2
3
4
5
6
7
8
9
10
11
12
Pod-A app                  Pod-A Envoy             istiod
  │                             │                    │
  │─ connect to ClusterIP ──────▶ (intercept)        │
  │                             │─ xDS lookup ───────▶
  │                             │◀ endpoints ─────────
  │                             │
  │                             │─── mTLS handshake ────────▶ Pod-B Envoy
  │                             │    (cert from istiod/Citadel)    │
  │                             │─── request (encrypted) ─────────▶│
  │                             │                             │─▶ Pod-B app
  │                             │◀── response (encrypted) ────│
  │◀── response ────────────────│

CoreDNS still provides the initial ClusterIP resolution. But Envoy uses the Pod IPs from EDS rather than the ClusterIP for the actual TCP connection.

Component Summary

Component Layer Responsibility
CNI Plugin L3 (IP routing) Pod IP allocation, cross-node routing
kube-proxy L4 (iptables/IPVS) ClusterIP→Pod DNAT, NodePort, LoadBalancer
CoreDNS DNS (L7) Service name → ClusterIP resolution
Istio/Envoy L7 (sidecar proxy) mTLS, observability, traffic management, direct endpoint routing

They are designed to compose — you can run all four simultaneously. The typical layering:

1
2
3
4
5
6
7
App request
  → CoreDNS (name → IP)
  → [Envoy outbound intercept if Istio present]
  → [kube-proxy DNAT if no Istio, or for non-meshed traffic]
  → CNI routing (cross-node packet delivery)
  → [Envoy inbound intercept if Istio present]
  → Destination app

Common Troubleshooting Patterns

DNS not resolving inside Pod:

1
2
3
4
5
6
kubectl exec -it <pod> -- nslookup kubernetes.default
# Check resolv.conf
kubectl exec -it <pod> -- cat /etc/resolv.conf
# Check CoreDNS pods
kubectl -n kube-system get pods -l k8s-app=kube-dns
kubectl -n kube-system logs -l k8s-app=kube-dns

Service ClusterIP unreachable:

1
2
3
4
5
6
# Check if kube-proxy is running on the node
kubectl -n kube-system get pods -l k8s-app=kube-proxy
# Verify iptables rules
iptables -t nat -L KUBE-SERVICES | grep <service-ip>
# Check endpoints exist
kubectl get endpoints <svc-name>

Istio traffic not flowing:

1
2
3
4
5
6
7
# Check sidecar injection
kubectl get pod <pod> -o jsonpath='{.spec.containers[*].name}'
# Check Envoy config
istioctl proxy-config clusters <pod>
istioctl proxy-config listeners <pod>
# Check mTLS mode
istioctl authn tls-check <pod> <service>

Summary

Kubernetes networking is a layered system where each component has a clear, narrow responsibility:

  • CNI ensures every Pod is routable by IP across nodes — the physical fabric.
  • kube-proxy virtualizes Services as stable IPs via kernel DNAT — the L4 load balancer.
  • CoreDNS translates human-readable names to ClusterIPs — the directory service.
  • Istio/Envoy adds L7 intelligence, security, and observability by intercepting traffic inside each Pod — the application-aware overlay.

Understanding where each component operates and how they hand off to each other makes debugging network issues far less mysterious. The key insight: a single Pod-to-Service call crosses all four layers, each doing its part invisibly.