Uber Details ServiceScale, a Second Kubernetes Scaling Controller Built to Let Failover Reuse Idle Capacity
Uber describes a new ServiceScale controller that lets multiple orchestrators set replica counts on the same Kubernetes workloads, and the stale-cache and multi-writer bugs its year-long rollout exposed.
Overview
Uber has published an account of a new Kubernetes controller, called ServiceScale, that allows multiple orchestrators to safely control the scale of the same workload, according to a September 9 post on the Uber Engineering Blog by senior software engineers Egor Grishechko and Srikar Paruchuru. InfoQ covered the post on September 28, describing how Uber separated scaling intent from execution to support regional failover without carrying reserved idle capacity.
What We Know
The platform and the problem
Uber’s Container Platform team manages over 100 compute clusters across data centers and cloud providers including Oracle and Google. Per the same post, roughly 4,000 services run on 3 million cores, and the clusters handle 1.5 million daily pod launches.
According to Uber, the company runs active-active data centers across regions. When a region becomes degraded or unavailable, traffic can be rerouted to another region, which needs enough idle compute to absorb it. Historically, Uber kept reserved idle capacity in all data centers. The new approach was to scale down low-tier workloads and scale up high-tier workloads during a failover, reusing the capacity of the former for the latter.
That created a second source of scaling intent. Uber’s internal platform, Up, and its Uber Deployment Controller (UDC) still owned the normal desired state, while a failover orchestrator now also needed to influence replica counts, as InfoQ reports.
Why a new controller
The engineers considered extending UDC but decided against it. UDC already sat on the hot path for service lifecycle operations, and, in the authors’ words, “a regression in failover handling wouldn’t stay isolated to failover.” They instead introduced a custom resource definition called ServiceScale and a Service Scale Controller (SSC). Each orchestrator writes its own scaling desire, and SSC reconciles the combined intent into Kubernetes objects.
The authors say they kept the model deliberately simple: “We didn’t want an additional external database, a separate coordination service, or a control plane that’d become harder to debug under incident pressure.” Storing intent in the Kubernetes API also made the system easier to inspect, since engineers can check ServiceScale to see which orchestrator wants what, and it simplified failback because steady-state and temporary failover information both live in the CRD spec.
Production lessons
Stale caches. Uber’s controllers read resources through informer caches that, per the post, can lag reality by a few seconds. Because Up treated a status field as a terminal input that could trigger the next irreversible workflow step, stale reads were costly. Uber added a read-your-own-write guardrail: a controller attaches its current generation as an annotation on downstream resources and, before reporting status, verifies that its cached data reflects at least that generation. InfoQ notes that Kubernetes v1.36, released in April 2026, introduced staleness mitigation for controllers using a comparable approach.
Multiple writers. Once UDC and SSC began updating the same resource, specific timing conditions made a ReplicaSet’s metadata (annotations) drift from its spec. Uber says the inconsistency was later identified as a bug in the proportional scaling logic of the upstream Kubernetes deployment controller. It broke proportional scaling needed for zero-downtime rolling updates and sometimes left workloads stuck until manually corrected. The team added fleet-wide observability to detect the drift and built an automated healer in UDC that periodically scans for it and patches affected ReplicaSets, while pursuing a long-term fix in the scaling path.
Rollout. The post describes a year-long rollout designed to be invisible to service owners, using staging environments, canary deployments and integration tests built on the kind framework. The new scaling path had to support native Kubernetes Deployments and OpenKruise CloneSets, and it completed without customer-impacting outages.
Broader failover results
InfoQ also points to an academic paper on Uber’s failover architecture, which it says reports that the broader Unified Failover Architecture reduced steady-state provisioning from 2x to 1.3x and eliminated over one million CPU cores. That figure describes the wider architecture rather than the ServiceScale controller alone.
What We Don’t Know
- Neither source says whether ServiceScale will be open-sourced; it is described as part of Uber’s internal platform.
- The Uber post says a long-term fix for the ReplicaSet drift was pursued in the scaling path, but the sources reviewed here do not say whether or when that fix shipped upstream.
- The post does not attribute a specific cost saving to ServiceScale itself.
Analysis
The post’s closing line summarizes its thesis: “multi-orchestrator systems aren’t hard because of the APIs. They’re hard because of everything that happens between writes.” The two problems Uber describes, cache staleness and concurrent writers, are general to Kubernetes controllers, and InfoQ suggests that the v1.36 staleness mitigation shows the problems are gaining recognition at the platform level.