Managing fleet configuration at scale
If you aren't using KubeFed, how do you deploy applications to 50 clusters consistently? The answer the industry settled on is GitOps: Git is the single source of truth, and a controller running inside each cluster continuously reconciles the cluster's actual state with what Git says it should be. The two dominant tools are ArgoCD and Flux.
The architectural question is: where do you install the GitOps controller?
1. Hub-and-Spoke (Centralized)
You install ArgoCD on one dedicated "Management Cluster". This instance connects to all other clusters using their API server credentials (stored as Kubernetes Secrets containing kubeconfig data).
Management Cluster (ArgoCD)
├── credentials for cluster-us-east → deploys to cluster-us-east
├── credentials for cluster-eu-west → deploys to cluster-eu-west
└── credentials for cluster-asia → deploys to cluster-asia
Pros:
- Single pane of glass. One ArgoCD UI shows the sync status of every application across every cluster.
- Centralized RBAC. You manage who can deploy what in one place.
- Easier auditing. All deployment events flow through one system.
Cons:
- Security blast radius. The Management Cluster holds cluster-admin credentials for every spoke. A compromise of the Management Cluster is a compromise of your entire fleet.
- Single point of failure. If the Management Cluster's ArgoCD is down, no cluster can receive new deployments — even if the clusters themselves are healthy.
- Scalability ceiling. One ArgoCD instance managing hundreds of clusters can become a bottleneck.
2. Independent (Decentralized)
You install a separate ArgoCD (or Flux) instance inside every cluster. Each instance watches the same Git repository and applies the configuration relevant to its own cluster, typically by reading from a cluster-specific directory or branch.
cluster-us-east/
└── ArgoCD → watches git/clusters/us-east/
cluster-eu-west/
└── ArgoCD → watches git/clusters/eu-west/
cluster-asia/
└── ArgoCD → watches git/clusters/asia/
Pros:
- Blast radius containment. Cluster A has no knowledge of Cluster B. Compromising one cluster yields no credentials for any other.
- Resilience. Each cluster is self-sufficient. A network partition between clusters doesn't stop deployments.
- Scales linearly. Adding a new cluster means adding a new ArgoCD instance — no central bottleneck.
Cons:
- No global view. You have 50 dashboards. Getting a fleet-wide status requires additional tooling (e.g., ArgoCD ApplicationSets with a notification system, or a dedicated fleet observability tool).
- Configuration drift risk. If the GitOps controller itself needs updating, you must update it in 50 places.
Hybrid approaches
In practice, many organizations use a hybrid: an Independent model for production clusters (security-first) combined with a lightweight fleet observability layer (like ArgoCD's ApplicationSet controller or Grafana dashboards fed by each cluster's metrics) to provide a global view without centralizing credentials.
Interviewers frequently ask: "What are the security trade-offs between Hub-and-Spoke and Independent GitOps?" The key insight to lead with is blast radius: in Hub-and-Spoke, the management cluster is a high-value target because it holds credentials for every spoke. In Independent, a breach is contained to a single cluster. Then mention the operational trade-off: Independent sacrifices the single pane of glass. A strong answer acknowledges both and describes the hybrid approach as the production-grade middle ground.