As a reply to Zalando's "Running 80+ clusters in production" post, someone asked whether this (80+ clusters) would not defeat "the purpose"? My answer would not fit in a Tweet, so here it is as a blog post.

RnL writes on Twitter:

"I don’t know any reasoning behind running 80 clusters I am sure smarter people than me thought about it. However, wasn’t the purpose of schedulers to scale nodes and use resources efficiently?"

This was not the first question I got, here a LinkedIn comment by Daniel D'Abate regarding our Zalando Kubernetes story from 4 months ago:

"I have a question and your insight will be very helpful for our next steps. Why are you using 100 clusters instead of 1 huge cluster? Is it related with better cost isolation or are any other reasons involved?"

Zalando runs 100+ Kubernetes clusters on AWS. Each cluster runs in its own AWS account. We always create a pair of prod/non-prod clusters per "product community" [1], i.e. only half of our clusters (50+) are marked as "production" and have full 24x7 on-call support.

We decided to go with "many" (that's relative) clusters for various reasons:

• Kubernetes has no strong story for multi-tenancy, having "smaller" clusters mitigates part of this problem

• some infrastructure is shared per cluster, e.g. Prometheus and the Ingress proxy (Skipper) --- this requires appropriate (vertical) scaling of these components, smaller clusters make this easier to handle

• the blast radius is limited --- anything going wrong in one cluster (outage, security incident, ..) does not necessarily affect the whole organization

• cost attribution is easier (every cluster belongs to a cost center) [2]

• the cluster (and its AWS account) serves as a natural trust boundary for access control (you can either deploy via CI/CD to a cluster or not)

It just overall better fits our world view. Smaller clusters would also be possible, but produce too much overhead [3] and do not leverage the advantages (in-cluster communication, better utilization).

You can find some Kubernetes Failure Stories mentioning cluster size, e.g.:

• On Infrastructure at Scale: A Cascading Failure of Distributed Systems - Target

• Breaking Kubernetes: How We Broke and Fixed our K8s Cluster - Civis Analytics

My colleague Sandor reminded me of the fact that AWS rate limits are another reason against "too large" clusters: all AWS integrations call AWS APIs and count towards these rate limits: Controller-Manager, kube2iam, External DNS, kube-ingress-aws-controller, kube-static-egress-controller, ..

To summarize:

• you want big clusters for utilization, in-cluster communication, and reduced overhead

• you want small clusters for isolation, reduced blast radius, and less challenges around scaling

We have 1200+ developers at Zalando, should they all share the same huge cluster? Probably not! Should every team (~4-8 people) get its own cluster? Probably not! So the truth is somewhere in the middle and the story might be different for your organization.