Setting up a Kubernetes cluster on AWS involves far more than just clicking “Create Cluster” in the EKS console. IAM roles, VPC topology, security groups, node groups, and storage classes all need to be wired together correctly. This post walks through every AWS-specific configuration decision you’ll encounter when building a production-grade EKS cluster.
Overview: What EKS Manages vs What You Manage EKS is a managed control plane. AWS runs the API server, etcd, scheduler, and controller manager across multiple AZs. You are responsible for:
VPC and networking — subnets, route tables, NAT gateways, security groups
Node groups — EC2 instances (managed or self-managed) that run your workloads
IAM — roles for the control plane, nodes, and pods (IRSA)
Add-ons — CoreDNS, kube-proxy, VPC CNI, EBS/EFS CSI drivers
Storage — StorageClasses backed by EBS or EFS
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 ┌─────────────────────────────────────────────────────────┐ │ AWS Managed (EKS Control Plane) │ │ ┌────────────┐ ┌──────────┐ ┌──────────────────────┐│ │ │ API Server │ │ etcd │ │ Controller Manager ││ │ │ (HA, 3 AZ) │ │ (HA, 3AZ)│ │ Scheduler ││ │ └────────────┘ └──────────┘ └──────────────────────┘│ └─────────────────────────────────────────────────────────┘ │ kubectl / kubeconfig ┌────────┴────────────────────────────────────────────────┐ │ Your VPC (Customer Managed) │ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │ │ Node Group │ │ Node Group │ │ Fargate │ │ │ │ (us-east-1a)│ │ (us-east-1b)│ │ Profile │ │ │ └─────────────┘ └─────────────┘ └─────────────┘ │ └─────────────────────────────────────────────────────────┘
1. VPC Design The VPC is the most critical foundational decision — it’s hard to change later.
Subnet Layout EKS requires at minimum two subnets in different AZs. The recommended layout for production:
Subnet Type
Purpose
Typical CIDR
Public
Load balancers, NAT gateway EIPs
/24 per AZ
Private
Worker nodes, pods
/19 or /18 per AZ
Intra (optional)
RDS, ElastiCache (no internet)
/24 per AZ
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 # terraform/vpc.tf module "vpc" { source = "terraform-aws-modules/vpc/aws" version = "~> 5.0" name = "my-eks-vpc" cidr = "10.0.0.0/16" azs = ["us-east-1a", "us-east-1b", "us-east-1c"] private_subnets = ["10.0.1.0/19", "10.0.33.0/19", "10.0.65.0/19"] public_subnets = ["10.0.100.0/24", "10.0.101.0/24", "10.0.102.0/24"] enable_nat_gateway = true single_nat_gateway = false # true = cheaper, false = HA one_nat_gateway_per_az = true # CRITICAL: These tags are required for EKS to discover subnets # and for the AWS Load Balancer Controller to provision ALBs/NLBs public_subnet_tags = { "kubernetes.io/cluster/my-cluster" = "shared" "kubernetes.io/role/elb" = "1" } private_subnet_tags = { "kubernetes.io/cluster/my-cluster" = "shared" "kubernetes.io/role/internal-elb" = "1" } }
Why subnet tags matter: The AWS Load Balancer Controller reads these tags to know which subnets to place internet-facing vs internal load balancers in. Without them, Service: LoadBalancer and Ingress resources will fail to provision.
VPC CNI IP Exhaustion The default AWS VPC CNI assigns each Pod a real VPC IP from your subnet (not an overlay). A m5.large supports up to 29 pods (3 ENIs × 10 IPs each, minus 1 primary per ENI). Plan your subnet sizes accordingly:
1 Nodes × max-pods-per-node < available IPs in private subnets
For large clusters, enable prefix delegation to get /28 prefixes per ENI slot (16× more IPs):
1 2 kubectl set env daemonset aws-node -n kube-system ENABLE_PREFIX_DELEGATION=true kubectl set env daemonset aws-node -n kube-system WARM_PREFIX_TARGET=1
Or use a custom networking mode where pods use a secondary CIDR (e.g., <IP_ADDRESS>/10) entirely separate from node IPs, avoiding subnet exhaustion.
2. IAM Roles EKS requires three distinct IAM role types. Mixing them up is the #1 source of permission errors.
2a. EKS Cluster Role (Control Plane) This role is assumed by the EKS service itself to call EC2, ELB, and CloudWatch APIs on your behalf.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 resource "aws_iam_role" "eks_cluster_role" { name = "eks-cluster-role" assume_role_policy = jsonencode({ Version = "2012-10-17" Statement = [{ Action = "sts:AssumeRole" Effect = "Allow" Principal = { Service = "eks.amazonaws.com" } }] }) } resource "aws_iam_role_policy_attachment" "eks_cluster_policy" { role = aws_iam_role.eks_cluster_role.name policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy" }
2b. Node Group Role (EC2 Worker Nodes) Every EC2 node needs an instance profile with permissions to join the cluster, pull ECR images, and report metrics:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 resource "aws_iam_role" "eks_node_role" { name = "eks-node-role" assume_role_policy = jsonencode({ Version = "2012-10-17" Statement = [{ Action = "sts:AssumeRole" Effect = "Allow" Principal = { Service = "ec2.amazonaws.com" } }] }) } # Required managed policies for nodes resource "aws_iam_role_policy_attachment" "node_policies" { for_each = toset([ "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy", "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy", # VPC CNI "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly", "arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy", # optional: metrics ]) role = aws_iam_role.eks_node_role.name policy_arn = each.value }
2c. IRSA — IAM Roles for Service Accounts (Pod-Level Permissions) Never give your node role broad S3/DynamoDB/Secrets access. Instead, use IRSA to bind an IAM role directly to a Kubernetes ServiceAccount. Pods using that ServiceAccount get temporary credentials via the OIDC token.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 # Step 1: Create OIDC provider for the cluster data "tls_certificate" "eks" { url = aws_eks_cluster.main.identity[0].oidc[0].issuer } resource "aws_iam_openid_connect_provider" "eks" { client_id_list = ["sts.amazonaws.com"] thumbprint_list = [data.tls_certificate.eks.certificates[0].sha1_fingerprint] url = aws_eks_cluster.main.identity[0].oidc[0].issuer } # Step 2: Create IAM role with OIDC trust policy locals { oidc_provider = replace(aws_iam_openid_connect_provider.eks.url, "https://", "") } resource "aws_iam_role" "s3_reader" { name = "eks-s3-reader" assume_role_policy = jsonencode({ Version = "2012-10-17" Statement = [{ Effect = "Allow" Action = "sts:AssumeRoleWithWebIdentity" Principal = { Federated = aws_iam_openid_connect_provider.eks.arn } Condition = { StringEquals = { "${local.oidc_provider}:aud" = "sts.amazonaws.com" "${local.oidc_provider}:sub" = "system:serviceaccount:my-namespace:my-service-account" } } }] }) } # Step 3: Annotate the Kubernetes ServiceAccount # kubectl annotate serviceaccount my-service-account \ # -n my-namespace \ # eks.amazonaws.com/role-arn=arn:aws:iam::ACCOUNT:role/eks-s3-reader
The annotation tells the EKS pod identity webhook to inject AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE env vars into matching pods, which the AWS SDK picks up automatically.
3. EKS Cluster Configuration 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 resource "aws_eks_cluster" "main" { name = "my-cluster" role_arn = aws_iam_role.eks_cluster_role.arn version = "1.30" vpc_config { subnet_ids = module.vpc.private_subnets endpoint_private_access = true endpoint_public_access = true # set false for fully private clusters public_access_cidrs = ["YOUR_OFFICE_IP/32"] # restrict public endpoint security_group_ids = [aws_security_group.cluster_additional.id] } # Enables envelope encryption of K8s Secrets with KMS encryption_config { provider { key_arn = aws_kms_key.eks.arn } resources = ["secrets"] } # Ship control plane logs to CloudWatch enabled_cluster_log_types = ["api", "audit", "authenticator", "controllerManager", "scheduler"] depends_on = [aws_iam_role_policy_attachment.eks_cluster_policy] }
Cluster Endpoint Access Modes
Mode
endpoint_public_access
endpoint_private_access
Use case
Public only
true
false
Dev/test, simple setup
Public + Private
true
true
Most production clusters
Private only
false
true
Air-gapped, high security
For private-only, kubectl traffic must originate from within the VPC (VPN, bastion, or AWS SSM).
4. Security Groups EKS creates a cluster security group automatically, but you often need additional rules.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 # Additional SG attached to the cluster (control plane) resource "aws_security_group" "cluster_additional" { name = "eks-cluster-additional" vpc_id = module.vpc.vpc_id } # Node security group resource "aws_security_group" "nodes" { name = "eks-nodes" vpc_id = module.vpc.vpc_id # Nodes communicate with each other (all ports) ingress { from_port = 0 to_port = 0 protocol = "-1" self = true } # Control plane → nodes (webhook calls, metrics) ingress { from_port = 443 to_port = 443 protocol = "tcp" security_groups = [aws_eks_cluster.main.vpc_config[0].cluster_security_group_id] } ingress { from_port = 1025 to_port = 65535 protocol = "tcp" security_groups = [aws_eks_cluster.main.vpc_config[0].cluster_security_group_id] } # All outbound egress { from_port = 0 to_port = 0 protocol = "-1" cidr_blocks = ["0.0.0.0/0"] } }
Critical ports to know:
Port
Direction
Purpose
443
Control plane → nodes
Webhook admission, exec/logs
10250
Control plane → nodes
Kubelet API
4443
Control plane → nodes
Metrics server
53 (UDP/TCP)
Nodes → nodes
CoreDNS
All
Node ↔ node
Pod-to-Pod (VPC CNI, no overlay)
5. Managed Node Groups 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 resource "aws_eks_node_group" "main" { cluster_name = aws_eks_cluster.main.name node_group_name = "main-ng" node_role_arn = aws_iam_role.eks_node_role.arn subnet_ids = module.vpc.private_subnets instance_types = ["m5.xlarge"] # or ["m5.xlarge", "m5a.xlarge"] for flexibility capacity_type = "ON_DEMAND" # or "SPOT" for cost savings (non-critical) ami_type = "AL2_x86_64" # Amazon Linux 2; or AL2023_x86_64_STANDARD scaling_config { min_size = 2 max_size = 10 desired_size = 3 } update_config { max_unavailable_percentage = 33 # rolling update: at most 1/3 nodes down } # Use launch template for custom AMI, user data, or extra EBS volumes launch_template { id = aws_launch_template.eks_node.id version = aws_launch_template.eks_node.latest_version } labels = { role = "general" } taint { key = "dedicated" value = "gpu" effect = "NO_SCHEDULE" } depends_on = [aws_iam_role_policy_attachment.node_policies] }
Launch Template for Custom Configuration 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 resource "aws_launch_template" "eks_node" { name_prefix = "eks-node-" instance_type = "m5.xlarge" block_device_mappings { device_name = "/dev/xvda" ebs { volume_size = 100 volume_type = "gp3" iops = 3000 throughput = 125 encrypted = true kms_key_id = aws_kms_key.ebs.arn delete_on_termination = true } } # Custom kubelet arguments via user data (AL2) user_data = base64encode(<<-EOF #!/bin/bash /etc/eks/bootstrap.sh my-cluster \ --kubelet-extra-args '--max-pods=110 --node-labels=eks.amazonaws.com/nodegroup=main-ng' \ --b64-cluster-ca '${aws_eks_cluster.main.certificate_authority[0].data}' \ --apiserver-endpoint '${aws_eks_cluster.main.endpoint}' EOF ) metadata_options { http_endpoint = "enabled" http_tokens = "required" # IMDSv2 only — disables IMDSv1 http_put_response_hop_limit = 1 } tag_specifications { resource_type = "instance" tags = { Name = "eks-node" } } }
http_tokens = required (IMDSv2) is a critical security setting — it prevents container escape attacks that steal the node’s instance role via IMDS.
6. EKS Add-ons Manage core cluster components as EKS-native add-ons (instead of self-managed Helm charts) so AWS handles version upgrades:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 locals { addons = { "vpc-cni" = "v1.18.1-eksbuild.1" "coredns" = "v1.11.1-eksbuild.9" "kube-proxy" = "v1.30.0-eksbuild.3" "aws-ebs-csi-driver" = "v1.31.0-eksbuild.1" } } resource "aws_eks_addon" "addons" { for_each = local.addons cluster_name = aws_eks_cluster.main.name addon_name = each.key addon_version = each.value resolve_conflicts_on_update = "OVERWRITE" # EBS CSI driver needs IRSA service_account_role_arn = each.key == "aws-ebs-csi-driver" ? aws_iam_role.ebs_csi.arn : null }
EBS CSI Driver IRSA 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 resource "aws_iam_role" "ebs_csi" { name = "eks-ebs-csi-driver" assume_role_policy = jsonencode({ Version = "2012-10-17" Statement = [{ Effect = "Allow" Action = "sts:AssumeRoleWithWebIdentity" Principal = { Federated = aws_iam_openid_connect_provider.eks.arn } Condition = { StringEquals = { "${local.oidc_provider}:sub" = "system:serviceaccount:kube-system:ebs-csi-controller-sa" "${local.oidc_provider}:aud" = "sts.amazonaws.com" } } }] }) } resource "aws_iam_role_policy_attachment" "ebs_csi" { role = aws_iam_role.ebs_csi.name policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy" }
7. StorageClasses After installing the EBS CSI driver, create StorageClasses:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: gp3 annotations: storageclass.kubernetes.io/is-default-class: "true" provisioner: ebs.csi.aws.com volumeBindingMode: WaitForFirstConsumer reclaimPolicy: Delete allowVolumeExpansion: true parameters: type: gp3 iops: "3000" throughput: "125" encrypted: "true" kmsKeyId: "arn:aws:kms:us-east-1:ACCOUNT:key/KEY-ID" --- apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: io2-high-perf provisioner: ebs.csi.aws.com volumeBindingMode: WaitForFirstConsumer parameters: type: io2 iops: "16000" encrypted: "true"
WaitForFirstConsumer is important in multi-AZ clusters — it delays EBS volume creation until a Pod is scheduled, ensuring the volume is created in the same AZ as the node.
8. aws-auth ConfigMap — Node and User Access EKS uses the aws-auth ConfigMap in kube-system to map IAM roles/users to Kubernetes RBAC identities:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 apiVersion: v1 kind: ConfigMap metadata: name: aws-auth namespace: kube-system data: mapRoles: | # Worker nodes (required — without this, nodes cannot join) - rolearn: arn:aws:iam::ACCOUNT:role/eks-node-role username: system:node:{{EC2PrivateDNSName}} groups: - system:bootstrappers - system:nodes - rolearn: arn:aws:iam::ACCOUNT:role/github-actions-role username: github-actions groups: - deployers mapUsers: | # Individual developer access - userarn: arn:aws:iam::ACCOUNT:user/alice username: alice groups: - system:masters # cluster-admin
Important: The node role entry must be present before nodes can register. If you see nodes stuck in NotReady with Unauthorized errors in kubelet logs, this mapping is missing.
9. Cluster Autoscaler 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 # IRSA for Cluster Autoscaler resource "aws_iam_role" "cluster_autoscaler" { name = "eks-cluster-autoscaler" assume_role_policy = jsonencode({ Version = "2012-10-17" Statement = [{ Effect = "Allow" Action = "sts:AssumeRoleWithWebIdentity" Principal = { Federated = aws_iam_openid_connect_provider.eks.arn } Condition = { StringEquals = { "${local.oidc_provider}:sub" = "system:serviceaccount:kube-system:cluster-autoscaler" "${local.oidc_provider}:aud" = "sts.amazonaws.com" } } }] }) } resource "aws_iam_role_policy" "cluster_autoscaler" { name = "cluster-autoscaler" role = aws_iam_role.cluster_autoscaler.id policy = jsonencode({ Version = "2012-10-17" Statement = [{ Effect = "Allow" Action = [ "autoscaling:DescribeAutoScalingGroups", "autoscaling:DescribeAutoScalingInstances", "autoscaling:DescribeLaunchConfigurations", "autoscaling:DescribeScalingActivities", "autoscaling:DescribeTags", "autoscaling:SetDesiredCapacity", "autoscaling:TerminateInstanceInAutoScalingGroup", "ec2:DescribeImages", "ec2:DescribeInstanceTypes", "ec2:DescribeLaunchTemplateVersions", "ec2:GetInstanceTypesFromInstanceRequirements", "eks:DescribeNodegroup" ] Resource = "*" }] }) }
Node group ASG tags required for Cluster Autoscaler discovery:
1 2 3 4 5 6 7 8 9 10 11 12 resource "aws_autoscaling_group_tag" "ca_tags" { for_each = { "k8s.io/cluster-autoscaler/enabled" = "true" "k8s.io/cluster-autoscaler/my-cluster" = "owned" } autoscaling_group_name = aws_eks_node_group.main.resources[0].autoscaling_groups[0].name tag { key = each.key value = each.value propagate_at_launch = false } }
10. AWS Load Balancer Controller Replace the legacy in-tree cloud controller with the AWS Load Balancer Controller for ALB (Ingress) and NLB (Service) support:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 resource "aws_iam_role" "aws_lb_controller" { name = "eks-aws-lb-controller" assume_role_policy = jsonencode({ Version = "2012-10-17" Statement = [{ Effect = "Allow" Action = "sts:AssumeRoleWithWebIdentity" Principal = { Federated = aws_iam_openid_connect_provider.eks.arn } Condition = { StringEquals = { "${local.oidc_provider}:sub" = "system:serviceaccount:kube-system:aws-load-balancer-controller" "${local.oidc_provider}:aud" = "sts.amazonaws.com" } } }] }) } # Policy: download from https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/main/docs/install/iam_policy.json
Install via Helm:
1 2 3 4 5 6 7 helm repo add eks https://aws.github.io/eks-charts helm install aws-load-balancer-controller eks/aws-load-balancer-controller \ -n kube-system \ --set clusterName=my-cluster \ --set serviceAccount.annotations."eks\.amazonaws\.com/role-arn" =arn:aws:iam::ACCOUNT:role/eks-aws-lb-controller \ --set region=us-east-1 \ --set vpcId=vpc-XXXXXXXX
Ingress annotation to use ALB:
1 2 3 4 5 6 7 8 9 apiVersion: networking.k8s.io/v1 kind: Ingress metadata: annotations: kubernetes.io/ingress.class: alb alb.ingress.kubernetes.io/scheme: internet-facing alb.ingress.kubernetes.io/target-type: ip alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-1:ACCOUNT:certificate/CERT-ID alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS": 443}]'
Common Gotchas
Issue
Root Cause
Fix
Nodes NotReady
Missing aws-auth node role mapping
Add node role to aws-auth ConfigMap
ALB not provisioning
Missing subnet tags
Add kubernetes.io/role/elb tag to public subnets
EBS volume in wrong AZ
StorageClass uses Immediate binding
Switch to WaitForFirstConsumer
Pod can’t access S3
Node role used instead of IRSA
Create IRSA role + annotate ServiceAccount
IMDSv1 access from container
http_tokens not set to required
Set metadata_options.http_tokens = "required"
IP exhaustion on nodes
Default VPC CNI, dense clusters
Enable prefix delegation or custom networking
Cluster Autoscaler not scaling
Missing ASG tags
Add k8s.io/cluster-autoscaler/enabled tags
Minimal Quickstart Checklist 1 2 3 4 5 6 7 8 9 10 11 12 13 □ VPC with public/private subnets across 3 AZs □ Subnet tags for ELB discovery □ EKS cluster role with AmazonEKSClusterPolicy □ Node role with WorkerNode + CNI + ECR policies □ EKS cluster with private endpoint + KMS encryption □ Managed node group with IMDSv2 enforced □ aws-auth ConfigMap with node role mapping □ OIDC provider created for IRSA □ EBS CSI driver add-on with IRSA role □ gp3 StorageClass as default (WaitForFirstConsumer) □ CoreDNS + kube-proxy + VPC CNI add-ons managed □ Cluster Autoscaler with IRSA + ASG tags □ AWS Load Balancer Controller installed
Getting these right from day one saves significant pain during scaling and security audits.