Disk Quota Controller Analysis

Posted on In

The disk-quota controller is a Kubernetes component that manages disk quotas for various resources in a Kubernetes cluster, including:

  1. emptyDir volumes
  2. Container root directories
  3. Container log directories

Architecture Overview

The controller uses XFS project quotas to limit disk usage. It follows a reconciliation pattern common in Kubernetes controllers, maintaining a desired state of the world (what quotas should be applied) and an actual state of the world (what quotas are currently applied), then reconciling the differences.

Code Flow Diagram

Code Flow Diagram

Key Components

1. QuotaController

  • Main controller that orchestrates all components
  • Watches for pod events and updates the desired state
  • Initializes and starts all other components

2. QuotaManagement

  • Manages XFS project quotas
  • Provides methods to set and clear quotas
  • Tracks project IDs for paths

3. DesiredStateOfWorld

  • In-memory cache of paths that should have quotas applied
  • Updated when pods are added, updated, or deleted

4. ActualStateOfWorld

  • In-memory cache of paths that currently have quotas applied
  • Updated when quotas are applied or removed

5. Reconciler

  • Periodically compares desired state with actual state
  • Applies quotas to paths in desired state but not in actual state
  • Removes quotas from paths in actual state but not in desired state

6. DesiredStateOfWorldPopulator

  • Periodically refreshes the desired state based on current pods
  • Removes paths from desired state if pods no longer exist
  • Removes paths from desired state if paths no longer exist

7. MetricsCollector

  • Collects quota usage metrics
  • Exposes metrics via Prometheus

Quota Types

The controller manages three types of quotas:

  1. EmptyDir Quotas: Limits the size of emptyDir volumes
    • Path: /var/mnt/kubelet/pods/<pod-uid>/volumes/kubernetes.io~empty-dir/<volume-name>
    • Size: Configurable via annotation or flag
  2. Container Root Quotas: Limits the size of container root filesystems
    • Path: Container’s overlayfs upper directory
    • Size: Configurable via annotation or flag
  3. Log Quotas: Limits the size of container logs
    • Path: /var/log/pods/<pod-uid> or /var/log/pods/<namespace>_<pod-name>_<pod-uid>
    • Size: Configurable via flag

Implementation Details

XFS Project Quotas

The controller uses XFS project quotas to limit disk usage. Each path gets assigned a unique project ID, and a quota is set for that project ID.

Quota Application Process

  1. A path is added to the desired state when a pod is created
  2. The reconciler detects the path needs a quota
  3. A project ID is assigned to the path
  4. The project ID is set on the path using SetProjectID
  5. A quota is set for the project ID using SetProjectQuota
  6. The path is added to the actual state

Quota Removal Process

  1. A path is removed from the desired state when a pod is deleted
  2. The reconciler detects the path no longer needs a quota
  3. The quota is cleared using ClearQuota
  4. The path is removed from the actual state

Metrics

The controller exposes the following metrics:

  • emptyDir_volume_used: Current usage of emptyDir volumes
  • emptyDir_volume_limit: Size limit of emptyDir volumes
  • emptyDir_volume_usage: Usage percentage of emptyDir volumes