/* * This is the extent on-disk structure. * It's used at the bottom of the tree (leaf nodes). */ structext4_extent { __le32 ee_block; /* first logical block extent covers */ __le16 ee_len; /* number of blocks covered by extent */ __le16 ee_start_hi; /* high 16 bits of physical block */ __le32 ee_start_lo; /* low 32 bits of physical block */ };
/* * This is index on-disk structure. * It's used at all the levels except the bottom (internal nodes). */ structext4_extent_idx { __le32 ei_block; /* index covers logical blocks from 'block' */ __le32 ei_leaf_lo; /* pointer to the physical block of the next * level. leaf or next index could be there */ __le16 ei_leaf_hi; /* high 16 bits of physical block */ __u16 ei_unused; };
/* * Each block (leaves and indexes), even inode-stored has header. */ structext4_extent_header { __le16 eh_magic; /* probably will support different formats */ __le16 eh_entries; /* number of valid entries */ __le16 eh_max; /* capacity of store in entries */ __le16 eh_depth; /* has tree real underlying blocks? * depth == 0 means the tree is a leaf */ __le32 eh_generation; /* generation of the tree */ };
structext4_ext_path { ext4_fsblk_t p_block; /* physical block of this level */ __u16 p_depth; /* depth of this level */ __u16 p_maxdepth; structext4_extent *p_ext;/* pointer to extent (leaf) */ structext4_extent_idx *p_idx;/* pointer to index (internal) */ structext4_extent_header *p_hdr;/* header of this block */ structbuffer_head *p_bh;/* buffer head for this block */ };
Understanding CIMaster: Intelligent CI Cluster Coordination at Scale
In modern cloud-native development, continuous integration (CI) pipelines are the backbone of software delivery. At scale, managing shared test infrastructure becomes a critical challenge. This is where CIMaster comes in—a sophisticated cluster management service designed to coordinate access to shared CI test clusters, ensuring efficient resource utilization and preventing test conflicts.
The Problem: Shared CI Infrastructure at Scale
In large organizations running hundreds or thousands of CI jobs daily, test clusters are expensive resources that need to be shared efficiently. Key challenges include:
Resource Contention: Multiple CI jobs competing for limited test clusters
Cluster State Management: Tracking which clusters are available, occupied, or held for debugging
Manual Intervention: Developers needing to hold clusters for investigation without blocking others
Dynamic Provisioning: Creating new clusters on-demand when capacity is insufficient
Lifecycle Management: Automatically releasing clusters after use or expiration
CIMaster addresses all these challenges through a centralized coordination service.
Architecture Overview
CIMaster is a Kubernetes-native service written in Go that provides a REST API for cluster lifecycle management. It consists of several key components:
Optimistic Locking prevents race conditions during concurrent updates using Kubernetes ResourceVersion.
Integration with Prow’s Manual Trigger
One of CIMaster’s powerful features is its integration with Prow through the manual-trigger component. This enables dynamic cluster provisioning when existing capacity is insufficient.
What is Prow Manual Trigger?
Prow is Kubernetes’ CI/CD system. The manual-trigger component (/Users/tashen/test-infra/prow/cmd/manual-trigger) is an HTTP server that allows programmatic creation of ProwJobs outside the normal GitHub webhook flow.
Key Capabilities:
Accepts HTTP POST requests with job specifications
Creates ProwJob custom resources in Kubernetes
Supports presubmit, postsubmit, and periodic job types
Injects environment variables (like AUTHOR) into jobs
// 1. Receives the request func(s *server) handleManualTrigger(w http.ResponseWriter, r *http.Request) { var req triggerRequest json.NewDecoder(r.Body).Decode(&req)
// 2. Looks up the job definition from config postsubmits := cfg.PostsubmitsStatic[req.Org+"/"+req.Repo] for _, p := range postsubmits { if p.Name == req.ProwJob { prowJob = createProwJobFromPostsubmit(p, req) break } }
// 4. Creates ProwJob in Kubernetes prowJobClient.Create(ctx, prowJob, metav1.CreateOptions{})
// 5. Waits for BuildID and returns status link statusLink := fmt.Sprintf("https://prow.tess.io/prowjob?prowjob=%s", prowJob.Name) logLink := fmt.Sprintf("https://prow.tess.io/log?job=%s&id=%s", req.ProwJob, buildID) }
The triggered ProwJob typically runs infrastructure-as-code (like Terraform or Ansible) to provision a new Kubernetes cluster, which is then added to CIMaster’s pool once ready.
This ensures clusters don’t remain locked indefinitely if developers forget to release them.
3. Multi-Purpose Cluster Support
CIMaster supports different cluster types:
tess-ci: Standard CI test clusters
tnet-ci: Network-specific test clusters with OS image selection
Allocation respects purpose and OS image requirements:
1 2 3 4 5 6
if cluster.Purpose != purpose { continue// skip incompatible clusters } if cluster.Purpose == TnetCI && cluster.OSImage != osimage { continue// skip wrong OS image }
4. Admin Authorization
Protected endpoints use a simple file-based authorization:
1 2 3 4 5 6 7 8 9 10
funccheckUser(h http.HandlerFunc, users []string) http.HandlerFunc { returnfunc(w http.ResponseWriter, r *http.Request) { userName := r.URL.Query().Get("name") if !contains(users, userName) { fmt.Fprintf(w, "user %s is not authorized", userName) return } h(w, r) } }
Admin users are loaded from /botadmin/users file (semicolon-separated).
5. Observability
Prometheus Metrics: Exposed on port 8090 (/metrics)
Structured Logging: All operations logged with correlation IDs
Graceful Shutdown: 120-second grace period to handle in-flight requests
API Examples
Allocating a Cluster for CI
1 2 3 4 5 6 7 8
# Get a vacant cluster for build #123 CLUSTER=$(curl -s "http://cimaster:8080/getvacant?build=123&job=e2e-test&email=ci-bot@ebay.com") echo"Using cluster: $CLUSTER"
# Run tests...
# Return cluster to pool curl "http://cimaster:8080/finishtest?cluster=$CLUSTER"
Debugging Workflow
1 2 3 4 5 6 7 8
# Hold cluster for investigation curl "http://cimaster:8080/holdcluster?cluster=cluster-05&name=alice&desc=debugging+network+issue"
# Investigate... kubectl get pods -n test-namespace
# Release when done curl "http://cimaster:8080/releasecluster?cluster=cluster-05&name=alice"
Retry Overhead: 50-200ms per retry (max 3 attempts)
Hold Expiration Check: Every 10 minutes (default)
Concurrency: Safe for multiple replicas via optimistic locking
Real-World Impact
At eBay’s TESS platform, CIMaster manages:
20+ shared test clusters across multiple regions
Hundreds of CI jobs daily from various teams
6-hour automatic hold expiration preventing resource lock-ups
Sub-second allocation for most requests
Dynamic scaling through Prow integration
Future Enhancements
Potential improvements being considered:
Priority Queues: Allow critical jobs to jump the allocation queue
Cluster Health Checks: Automatic disabling of unhealthy clusters
Usage Analytics: Track allocation patterns and optimize capacity
Webhook Notifications: Slack/email alerts for hold expirations
Multi-Cluster Federation: Coordinate across multiple Kubernetes clusters
Conclusion
CIMaster demonstrates how a relatively simple coordination service can solve complex resource management challenges in CI/CD infrastructure. By combining:
Stateful cluster tracking in Kubernetes ConfigMaps
Optimistic locking for safe concurrent access
Automatic expiration for abandoned holds
Prow integration for dynamic provisioning
REST API for easy integration
…it provides a robust foundation for shared test infrastructure at scale.
The integration with Prow’s manual-trigger component is particularly elegant—CIMaster doesn’t need to know how to create clusters, only when to request them. This separation of concerns allows infrastructure teams to evolve cluster provisioning strategies independently.
Whether you’re building CI infrastructure for a large organization or looking to optimize resource utilization in your Kubernetes platform, the patterns demonstrated by CIMaster offer valuable insights into distributed system coordination.
This article explores the internal architecture of CIMaster, a production cluster coordination service. All code examples are from the actual implementation.