AWS Configuration for Setting Up a Kubernetes Cluster - EKS Deep Dive

Setting up a Kubernetes cluster on AWS involves far more than just clicking “Create Cluster” in the EKS console. IAM roles, VPC topology, security groups, node groups, and storage classes all need to be wired together correctly. This post walks through every AWS-specific configuration decision you’ll encounter when building a production-grade EKS cluster.

Overview: What EKS Manages vs What You Manage

EKS is a managed control plane. AWS runs the API server, etcd, scheduler, and controller manager across multiple AZs. You are responsible for:

  • VPC and networking — subnets, route tables, NAT gateways, security groups
  • Node groups — EC2 instances (managed or self-managed) that run your workloads
  • IAM — roles for the control plane, nodes, and pods (IRSA)
  • Add-ons — CoreDNS, kube-proxy, VPC CNI, EBS/EFS CSI drivers
  • Storage — StorageClasses backed by EBS or EFS
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
┌─────────────────────────────────────────────────────────┐
│ AWS Managed (EKS Control Plane) │
│ ┌────────────┐ ┌──────────┐ ┌──────────────────────┐│
│ │ API Server │ │ etcd │ │ Controller Manager ││
│ │ (HA, 3 AZ) │ │ (HA, 3AZ)│ │ Scheduler ││
│ └────────────┘ └──────────┘ └──────────────────────┘│
└─────────────────────────────────────────────────────────┘
│ kubectl / kubeconfig
┌────────┴────────────────────────────────────────────────┐
│ Your VPC (Customer Managed) │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ Node Group │ │ Node Group │ │ Fargate │ │
│ │ (us-east-1a)│ │ (us-east-1b)│ │ Profile │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ │
└─────────────────────────────────────────────────────────┘

1. VPC Design

The VPC is the most critical foundational decision — it’s hard to change later.

Subnet Layout

EKS requires at minimum two subnets in different AZs. The recommended layout for production:

Subnet Type Purpose Typical CIDR
Public Load balancers, NAT gateway EIPs /24 per AZ
Private Worker nodes, pods /19 or /18 per AZ
Intra (optional) RDS, ElastiCache (no internet) /24 per AZ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
# terraform/vpc.tf
module "vpc" {
source = "terraform-aws-modules/vpc/aws"
version = "~> 5.0"

name = "my-eks-vpc"
cidr = "10.0.0.0/16"

azs = ["us-east-1a", "us-east-1b", "us-east-1c"]
private_subnets = ["10.0.1.0/19", "10.0.33.0/19", "10.0.65.0/19"]
public_subnets = ["10.0.100.0/24", "10.0.101.0/24", "10.0.102.0/24"]

enable_nat_gateway = true
single_nat_gateway = false # true = cheaper, false = HA
one_nat_gateway_per_az = true

# CRITICAL: These tags are required for EKS to discover subnets
# and for the AWS Load Balancer Controller to provision ALBs/NLBs
public_subnet_tags = {
"kubernetes.io/cluster/my-cluster" = "shared"
"kubernetes.io/role/elb" = "1"
}
private_subnet_tags = {
"kubernetes.io/cluster/my-cluster" = "shared"
"kubernetes.io/role/internal-elb" = "1"
}
}

Why subnet tags matter: The AWS Load Balancer Controller reads these tags to know which subnets to place internet-facing vs internal load balancers in. Without them, Service: LoadBalancer and Ingress resources will fail to provision.

VPC CNI IP Exhaustion

The default AWS VPC CNI assigns each Pod a real VPC IP from your subnet (not an overlay). A m5.large supports up to 29 pods (3 ENIs × 10 IPs each, minus 1 primary per ENI). Plan your subnet sizes accordingly:

1
Nodes × max-pods-per-node < available IPs in private subnets

For large clusters, enable prefix delegation to get /28 prefixes per ENI slot (16× more IPs):

1
2
kubectl set env daemonset aws-node -n kube-system ENABLE_PREFIX_DELEGATION=true
kubectl set env daemonset aws-node -n kube-system WARM_PREFIX_TARGET=1

Or use a custom networking mode where pods use a secondary CIDR (e.g., <IP_ADDRESS>/10) entirely separate from node IPs, avoiding subnet exhaustion.


2. IAM Roles

EKS requires three distinct IAM role types. Mixing them up is the #1 source of permission errors.

2a. EKS Cluster Role (Control Plane)

This role is assumed by the EKS service itself to call EC2, ELB, and CloudWatch APIs on your behalf.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
resource "aws_iam_role" "eks_cluster_role" {
name = "eks-cluster-role"

assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [{
Action = "sts:AssumeRole"
Effect = "Allow"
Principal = { Service = "eks.amazonaws.com" }
}]
})
}

resource "aws_iam_role_policy_attachment" "eks_cluster_policy" {
role = aws_iam_role.eks_cluster_role.name
policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
}

2b. Node Group Role (EC2 Worker Nodes)

Every EC2 node needs an instance profile with permissions to join the cluster, pull ECR images, and report metrics:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
resource "aws_iam_role" "eks_node_role" {
name = "eks-node-role"

assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [{
Action = "sts:AssumeRole"
Effect = "Allow"
Principal = { Service = "ec2.amazonaws.com" }
}]
})
}

# Required managed policies for nodes
resource "aws_iam_role_policy_attachment" "node_policies" {
for_each = toset([
"arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy",
"arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy", # VPC CNI
"arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly",
"arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy", # optional: metrics
])
role = aws_iam_role.eks_node_role.name
policy_arn = each.value
}

2c. IRSA — IAM Roles for Service Accounts (Pod-Level Permissions)

Never give your node role broad S3/DynamoDB/Secrets access. Instead, use IRSA to bind an IAM role directly to a Kubernetes ServiceAccount. Pods using that ServiceAccount get temporary credentials via the OIDC token.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
# Step 1: Create OIDC provider for the cluster
data "tls_certificate" "eks" {
url = aws_eks_cluster.main.identity[0].oidc[0].issuer
}

resource "aws_iam_openid_connect_provider" "eks" {
client_id_list = ["sts.amazonaws.com"]
thumbprint_list = [data.tls_certificate.eks.certificates[0].sha1_fingerprint]
url = aws_eks_cluster.main.identity[0].oidc[0].issuer
}

# Step 2: Create IAM role with OIDC trust policy
locals {
oidc_provider = replace(aws_iam_openid_connect_provider.eks.url, "https://", "")
}

resource "aws_iam_role" "s3_reader" {
name = "eks-s3-reader"

assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [{
Effect = "Allow"
Action = "sts:AssumeRoleWithWebIdentity"
Principal = { Federated = aws_iam_openid_connect_provider.eks.arn }
Condition = {
StringEquals = {
"${local.oidc_provider}:aud" = "sts.amazonaws.com"
"${local.oidc_provider}:sub" = "system:serviceaccount:my-namespace:my-service-account"
}
}
}]
})
}

# Step 3: Annotate the Kubernetes ServiceAccount
# kubectl annotate serviceaccount my-service-account \
# -n my-namespace \
# eks.amazonaws.com/role-arn=arn:aws:iam::ACCOUNT:role/eks-s3-reader

The annotation tells the EKS pod identity webhook to inject AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE env vars into matching pods, which the AWS SDK picks up automatically.


3. EKS Cluster Configuration

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
resource "aws_eks_cluster" "main" {
name = "my-cluster"
role_arn = aws_iam_role.eks_cluster_role.arn
version = "1.30"

vpc_config {
subnet_ids = module.vpc.private_subnets
endpoint_private_access = true
endpoint_public_access = true # set false for fully private clusters
public_access_cidrs = ["YOUR_OFFICE_IP/32"] # restrict public endpoint
security_group_ids = [aws_security_group.cluster_additional.id]
}

# Enables envelope encryption of K8s Secrets with KMS
encryption_config {
provider {
key_arn = aws_kms_key.eks.arn
}
resources = ["secrets"]
}

# Ship control plane logs to CloudWatch
enabled_cluster_log_types = ["api", "audit", "authenticator", "controllerManager", "scheduler"]

depends_on = [aws_iam_role_policy_attachment.eks_cluster_policy]
}

Cluster Endpoint Access Modes

Mode endpoint_public_access endpoint_private_access Use case
Public only true false Dev/test, simple setup
Public + Private true true Most production clusters
Private only false true Air-gapped, high security

For private-only, kubectl traffic must originate from within the VPC (VPN, bastion, or AWS SSM).


4. Security Groups

EKS creates a cluster security group automatically, but you often need additional rules.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
# Additional SG attached to the cluster (control plane)
resource "aws_security_group" "cluster_additional" {
name = "eks-cluster-additional"
vpc_id = module.vpc.vpc_id
}

# Node security group
resource "aws_security_group" "nodes" {
name = "eks-nodes"
vpc_id = module.vpc.vpc_id

# Nodes communicate with each other (all ports)
ingress {
from_port = 0
to_port = 0
protocol = "-1"
self = true
}

# Control plane → nodes (webhook calls, metrics)
ingress {
from_port = 443
to_port = 443
protocol = "tcp"
security_groups = [aws_eks_cluster.main.vpc_config[0].cluster_security_group_id]
}

ingress {
from_port = 1025
to_port = 65535
protocol = "tcp"
security_groups = [aws_eks_cluster.main.vpc_config[0].cluster_security_group_id]
}

# All outbound
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
}

Critical ports to know:

Port Direction Purpose
443 Control plane → nodes Webhook admission, exec/logs
10250 Control plane → nodes Kubelet API
4443 Control plane → nodes Metrics server
53 (UDP/TCP) Nodes → nodes CoreDNS
All Node ↔ node Pod-to-Pod (VPC CNI, no overlay)

5. Managed Node Groups

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
resource "aws_eks_node_group" "main" {
cluster_name = aws_eks_cluster.main.name
node_group_name = "main-ng"
node_role_arn = aws_iam_role.eks_node_role.arn
subnet_ids = module.vpc.private_subnets

instance_types = ["m5.xlarge"] # or ["m5.xlarge", "m5a.xlarge"] for flexibility
capacity_type = "ON_DEMAND" # or "SPOT" for cost savings (non-critical)
ami_type = "AL2_x86_64" # Amazon Linux 2; or AL2023_x86_64_STANDARD

scaling_config {
min_size = 2
max_size = 10
desired_size = 3
}

update_config {
max_unavailable_percentage = 33 # rolling update: at most 1/3 nodes down
}

# Use launch template for custom AMI, user data, or extra EBS volumes
launch_template {
id = aws_launch_template.eks_node.id
version = aws_launch_template.eks_node.latest_version
}

labels = {
role = "general"
}

taint {
key = "dedicated"
value = "gpu"
effect = "NO_SCHEDULE"
}

depends_on = [aws_iam_role_policy_attachment.node_policies]
}

Launch Template for Custom Configuration

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
resource "aws_launch_template" "eks_node" {
name_prefix = "eks-node-"
instance_type = "m5.xlarge"

block_device_mappings {
device_name = "/dev/xvda"
ebs {
volume_size = 100
volume_type = "gp3"
iops = 3000
throughput = 125
encrypted = true
kms_key_id = aws_kms_key.ebs.arn
delete_on_termination = true
}
}

# Custom kubelet arguments via user data (AL2)
user_data = base64encode(<<-EOF
#!/bin/bash
/etc/eks/bootstrap.sh my-cluster \
--kubelet-extra-args '--max-pods=110 --node-labels=eks.amazonaws.com/nodegroup=main-ng' \
--b64-cluster-ca '${aws_eks_cluster.main.certificate_authority[0].data}' \
--apiserver-endpoint '${aws_eks_cluster.main.endpoint}'
EOF
)

metadata_options {
http_endpoint = "enabled"
http_tokens = "required" # IMDSv2 only — disables IMDSv1
http_put_response_hop_limit = 1
}

tag_specifications {
resource_type = "instance"
tags = { Name = "eks-node" }
}
}

http_tokens = required (IMDSv2) is a critical security setting — it prevents container escape attacks that steal the node’s instance role via IMDS.


6. EKS Add-ons

Manage core cluster components as EKS-native add-ons (instead of self-managed Helm charts) so AWS handles version upgrades:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
locals {
addons = {
"vpc-cni" = "v1.18.1-eksbuild.1"
"coredns" = "v1.11.1-eksbuild.9"
"kube-proxy" = "v1.30.0-eksbuild.3"
"aws-ebs-csi-driver" = "v1.31.0-eksbuild.1"
}
}

resource "aws_eks_addon" "addons" {
for_each = local.addons

cluster_name = aws_eks_cluster.main.name
addon_name = each.key
addon_version = each.value
resolve_conflicts_on_update = "OVERWRITE"

# EBS CSI driver needs IRSA
service_account_role_arn = each.key == "aws-ebs-csi-driver" ? aws_iam_role.ebs_csi.arn : null
}

EBS CSI Driver IRSA

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
resource "aws_iam_role" "ebs_csi" {
name = "eks-ebs-csi-driver"

assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [{
Effect = "Allow"
Action = "sts:AssumeRoleWithWebIdentity"
Principal = { Federated = aws_iam_openid_connect_provider.eks.arn }
Condition = {
StringEquals = {
"${local.oidc_provider}:sub" = "system:serviceaccount:kube-system:ebs-csi-controller-sa"
"${local.oidc_provider}:aud" = "sts.amazonaws.com"
}
}
}]
})
}

resource "aws_iam_role_policy_attachment" "ebs_csi" {
role = aws_iam_role.ebs_csi.name
policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy"
}

7. StorageClasses

After installing the EBS CSI driver, create StorageClasses:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
# Default StorageClass — gp3 (better price/performance than gp2)
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: gp3
annotations:
storageclass.kubernetes.io/is-default-class: "true"
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer # waits for Pod scheduling before provisioning
reclaimPolicy: Delete
allowVolumeExpansion: true
parameters:
type: gp3
iops: "3000"
throughput: "125"
encrypted: "true"
kmsKeyId: "arn:aws:kms:us-east-1:ACCOUNT:key/KEY-ID"
---
# High-performance StorageClass for databases
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: io2-high-perf
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
parameters:
type: io2
iops: "16000"
encrypted: "true"

WaitForFirstConsumer is important in multi-AZ clusters — it delays EBS volume creation until a Pod is scheduled, ensuring the volume is created in the same AZ as the node.


8. aws-auth ConfigMap — Node and User Access

EKS uses the aws-auth ConfigMap in kube-system to map IAM roles/users to Kubernetes RBAC identities:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
apiVersion: v1
kind: ConfigMap
metadata:
name: aws-auth
namespace: kube-system
data:
mapRoles: |
# Worker nodes (required — without this, nodes cannot join)
- rolearn: arn:aws:iam::ACCOUNT:role/eks-node-role
username: system:node:{{EC2PrivateDNSName}}
groups:
- system:bootstrappers
- system:nodes

# Allow a CI/CD role to deploy
- rolearn: arn:aws:iam::ACCOUNT:role/github-actions-role
username: github-actions
groups:
- deployers # bound to a ClusterRoleBinding

mapUsers: |
# Individual developer access
- userarn: arn:aws:iam::ACCOUNT:user/alice
username: alice
groups:
- system:masters # cluster-admin

Important: The node role entry must be present before nodes can register. If you see nodes stuck in NotReady with Unauthorized errors in kubelet logs, this mapping is missing.


9. Cluster Autoscaler

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
# IRSA for Cluster Autoscaler
resource "aws_iam_role" "cluster_autoscaler" {
name = "eks-cluster-autoscaler"
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [{
Effect = "Allow"
Action = "sts:AssumeRoleWithWebIdentity"
Principal = { Federated = aws_iam_openid_connect_provider.eks.arn }
Condition = {
StringEquals = {
"${local.oidc_provider}:sub" = "system:serviceaccount:kube-system:cluster-autoscaler"
"${local.oidc_provider}:aud" = "sts.amazonaws.com"
}
}
}]
})
}

resource "aws_iam_role_policy" "cluster_autoscaler" {
name = "cluster-autoscaler"
role = aws_iam_role.cluster_autoscaler.id

policy = jsonencode({
Version = "2012-10-17"
Statement = [{
Effect = "Allow"
Action = [
"autoscaling:DescribeAutoScalingGroups",
"autoscaling:DescribeAutoScalingInstances",
"autoscaling:DescribeLaunchConfigurations",
"autoscaling:DescribeScalingActivities",
"autoscaling:DescribeTags",
"autoscaling:SetDesiredCapacity",
"autoscaling:TerminateInstanceInAutoScalingGroup",
"ec2:DescribeImages",
"ec2:DescribeInstanceTypes",
"ec2:DescribeLaunchTemplateVersions",
"ec2:GetInstanceTypesFromInstanceRequirements",
"eks:DescribeNodegroup"
]
Resource = "*"
}]
})
}

Node group ASG tags required for Cluster Autoscaler discovery:

1
2
3
4
5
6
7
8
9
10
11
12
resource "aws_autoscaling_group_tag" "ca_tags" {
for_each = {
"k8s.io/cluster-autoscaler/enabled" = "true"
"k8s.io/cluster-autoscaler/my-cluster" = "owned"
}
autoscaling_group_name = aws_eks_node_group.main.resources[0].autoscaling_groups[0].name
tag {
key = each.key
value = each.value
propagate_at_launch = false
}
}

10. AWS Load Balancer Controller

Replace the legacy in-tree cloud controller with the AWS Load Balancer Controller for ALB (Ingress) and NLB (Service) support:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
resource "aws_iam_role" "aws_lb_controller" {
name = "eks-aws-lb-controller"
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [{
Effect = "Allow"
Action = "sts:AssumeRoleWithWebIdentity"
Principal = { Federated = aws_iam_openid_connect_provider.eks.arn }
Condition = {
StringEquals = {
"${local.oidc_provider}:sub" = "system:serviceaccount:kube-system:aws-load-balancer-controller"
"${local.oidc_provider}:aud" = "sts.amazonaws.com"
}
}
}]
})
}
# Policy: download from https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/main/docs/install/iam_policy.json

Install via Helm:

1
2
3
4
5
6
7
helm repo add eks https://aws.github.io/eks-charts
helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
-n kube-system \
--set clusterName=my-cluster \
--set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=arn:aws:iam::ACCOUNT:role/eks-aws-lb-controller \
--set region=us-east-1 \
--set vpcId=vpc-XXXXXXXX

Ingress annotation to use ALB:

1
2
3
4
5
6
7
8
9
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/target-type: ip # routes directly to Pod IPs (no NodePort hop)
alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-1:ACCOUNT:certificate/CERT-ID
alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS": 443}]'

Common Gotchas

Issue Root Cause Fix
Nodes NotReady Missing aws-auth node role mapping Add node role to aws-auth ConfigMap
ALB not provisioning Missing subnet tags Add kubernetes.io/role/elb tag to public subnets
EBS volume in wrong AZ StorageClass uses Immediate binding Switch to WaitForFirstConsumer
Pod can’t access S3 Node role used instead of IRSA Create IRSA role + annotate ServiceAccount
IMDSv1 access from container http_tokens not set to required Set metadata_options.http_tokens = "required"
IP exhaustion on nodes Default VPC CNI, dense clusters Enable prefix delegation or custom networking
Cluster Autoscaler not scaling Missing ASG tags Add k8s.io/cluster-autoscaler/enabled tags

Minimal Quickstart Checklist

1
2
3
4
5
6
7
8
9
10
11
12
13
□ VPC with public/private subnets across 3 AZs
□ Subnet tags for ELB discovery
□ EKS cluster role with AmazonEKSClusterPolicy
□ Node role with WorkerNode + CNI + ECR policies
□ EKS cluster with private endpoint + KMS encryption
□ Managed node group with IMDSv2 enforced
□ aws-auth ConfigMap with node role mapping
□ OIDC provider created for IRSA
□ EBS CSI driver add-on with IRSA role
□ gp3 StorageClass as default (WaitForFirstConsumer)
□ CoreDNS + kube-proxy + VPC CNI add-ons managed
□ Cluster Autoscaler with IRSA + ASG tags
□ AWS Load Balancer Controller installed

Getting these right from day one saves significant pain during scaling and security audits.