Amazon: why cells, not clusters

In 2021, Colm MacCárthaigh — a Distinguished Engineer at AWS who designed Route 53 and S3's request routing — gave a re:Invent talk titled "Static stability and self-healing infrastructure". Buried on slide 23 was the number that defines modern AWS architecture: S3 stores its objects across thousands of independent cells, not one giant cluster, and any one cell going dark takes out at most 1 / num_cells of the customer base. The 2017 S3 outage in us-east-1 — the one that broke half the internet for four hours, knocked Slack offline, and stopped Trello from loading — was the structural prompt. A single subsystem (the index management service) had a single shared failure domain; a typo during a routine maintenance command tipped the whole region. The post-mortem did not say "we will improve operator tooling" (though they did). It said: we will rebuild this so no single subsystem can take down a region again. The mechanism that came out of that post-mortem is the cell. This chapter is about why a 2024-era AWS engineer reaches for cells instead of clusters, what cells actually are, and why the answer is not "the obvious thing every distributed systems textbook recommends".

What a cell actually is — the anatomy

Open the official AWS re:Invent 2021 architecture diagram for an S3 cell. It is not a microservice. It is not a Kubernetes namespace. It is a complete, self-contained instance of the entire S3 request path: a fleet of front-end request routers, a fleet of authorisation servers, a fleet of metadata index servers, a fleet of replicated storage nodes, an internal queue for asynchronous work, and a per-cell control plane that handles deployments and configuration. A single cell at S3 scale runs roughly 200–500 EC2 instances and serves a few percent of regional traffic. The region has thousands of these cells. Why "complete instance" is the load-bearing word here: a cell that shared any state with another cell — a single shared metadata database, a single shared rate-limiter, a single shared secrets manager — would propagate a poison pill across the boundary. The point of the cell is that the answer to "what fraction of customers does this failure affect?" is bounded by 1 / num_cells, no matter what the failure is. A cell that shares state with its neighbours violates that invariant on the very first incident.

The cellular architecture's first promise is therefore not "high availability" in the conventional sense. It is bounded blast radius. A cluster — even a sharded cluster — has shared infrastructure: the front-end ingress, the cluster-wide config service, the deployment system that pushes a new build to all shards simultaneously. Any of those shared layers can fail in a way that affects every shard. A cell is a wholly independent stack with no shared layer that can take it out except the network fabric and the AWS regional control plane. The promise is: when something goes wrong, the math says how bad it can get.

A working example, the 2017 S3 incident, restated through the cellular lens: in the 2017 architecture, S3 in us-east-1 had two large subsystems — index management and placement — each of which was a single shared system across the region. A typo in an operator command removed too many index servers from rotation. The remaining index servers couldn't handle the load and started failing. New index server starts took longer than expected because of cold-cache rebuilding. Total recovery: 4 hours. Affected: every S3 bucket in us-east-1. After the post-mortem, AWS rebuilt S3's index management as cellular: there is no longer a single regional index management service. There are thousands of cell-local index managers, and a typo affecting one cell's index manager affects only that cell's customers. The blast radius dropped from "every S3 bucket" to "the few percent of buckets routed to the affected cell".

A second working example, with the arithmetic running. DynamoDB at the time of writing serves roughly 100M requests per second across all customers. Internally it runs as cells of about 100–300 nodes each, and a region holds hundreds to low thousands of cells. If one cell fails completely (worst case), the affected fraction is 1 / num_cells ≈ 0.1% to 0.5%. The expected fraction of customers affected over a year, given a per-cell incident rate of about 1 incident per cell-year (an estimate from AWS published reliability talks), is incidents_per_year × (1/num_cells) ≈ 1 × 0.001 = 0.1%. Compare to the cluster baseline: a single regional cluster with 1 incident per year affects 100% of customers. The cellular architecture turns a 1.0 expected affected-customer-year into a 0.001 expected affected-customer-year — three orders of magnitude reduction in expected blast radius for the same per-cell failure rate.

Shuffle sharding — why cells are not just sharded clusters

A reader who has just read about cells will reasonably ask: "isn't this just sharding?" It is not. The difference is shuffle sharding, and it is the second piece of the cellular pattern that makes the math work.

In a traditional sharded cluster, customer X is mapped to shard hash(X) % num_shards. If shard 7 fails, every customer mapped to shard 7 is affected. If a "noisy neighbour" customer N is on shard 7 — say, hammering the shard with 100,000× the median traffic — every customer co-located with N suffers the same degraded latency. The blast radius of "noisy neighbour on shard 7" is "all customers mapped to shard 7", which can be 1,000+ customers depending on shard size.

Shuffle sharding solves this by mapping each customer to a small random subset of cells rather than a single cell. AWS Route 53's published shuffle-sharding scheme assigns each customer to 4 cells out of the available 100. A request from customer X is routed to one of X's 4 cells, with retries fanning out to the other 3 if the first fails. Now consider the noisy neighbour. Customer N is also assigned to 4 cells. The probability that N and X share even one cell is 1 − (96/100)(95/99)(94/98)(93/97) ≈ 0.155, but the probability that they share all four cells is 1 / C(100,4) = 1 / 3,921,225 — vanishingly small. Why this combinatorial fact is the security property of cellular architecture: a noisy neighbour can degrade one of customer X's four cells, but the other three still work. Customer X's effective availability is the probability that at least one of their four assigned cells is healthy, which under independent failure assumptions is approximately 1 − p⁴ where p is the per-cell failure probability. At per-cell unavailability of 0.1%, the customer-perceived unavailability becomes 0.001⁴ = 10⁻¹² — a quadrillion-times improvement over single-shard assignment, achieved with 4× routing fan-out instead of 1×. The arithmetic of the combinatorial assignment is the entire reason shuffle sharding works; without it, "cellular" is just "many small clusters with the same problems".

A third worked example, with Indian context. Razorpay's UPI payment routing, in 2024, started experimenting with shuffle-sharding-like patterns for merchant assignment to backend cells. A merchant doing 100× normal volume during a flash sale (say, a Flipkart Big Billion Days lightning deal at 14:30 IST) used to pin all of Razorpay's payment processing for that merchant to a single backend pool, and that pool's queue would fill, slowing down payments for unrelated merchants whose traffic happened to land on the same pool. The 2024 redesign assigns each merchant to 3 of the 12 backend pools, with the payment router shuffle-distributing traffic across those 3. A flash sale on one merchant now elevates load on their 3 pools, but unrelated merchants assigned to a different combination of 3-of-12 see no impact unless they share at least 2 pools (combinatorially ~13% chance) and even then only see fractional impact. Razorpay's published 2024 number: p99 elevation during the largest flash sale of the year dropped from 380 ms (single-pool) to 95 ms (shuffle-sharded across 3 pools).

A simpy simulator — cells vs clusters under noisy neighbour

The most useful thing a working engineer can do with cellular architecture is reproduce the noisy-neighbour arithmetic in simulation. The simulator below builds two systems — a cluster (all customers share one queue) and a cell-with-shuffle-sharding system (each customer gets 4 of 100 cells) — and pumps traffic through both with a single noisy customer doing 50× normal volume. The output is the per-customer p99 distribution.

# cells_vs_clusters.py
# Reproduce the cellular-architecture blast-radius arithmetic with simpy.
# Builds two systems: a single-cluster baseline and a shuffle-sharded cellular
# system. Simulates a noisy neighbour and reports per-customer p99 latency.
#
# Run: python3 -m venv .venv && source .venv/bin/activate
#      pip install simpy hdrh numpy
#      python3 cells_vs_clusters.py

import simpy, random, statistics, hashlib
from hdrh.histogram import HdrHistogram

NUM_CUSTOMERS   = 1000
NUM_CELLS       = 100
CELLS_PER_CUST  = 4              # shuffle-sharding fan-out
SIM_DURATION_S  = 60
NOISY_CUSTOMER  = 0              # customer 0 is noisy
NOISY_MULTIPLIER = 50            # noisy customer does 50x normal traffic
SERVICE_TIME_MS = 5              # mean per-request service time
ARRIVAL_RATE_HZ = 200            # per-customer baseline arrival rate

def assign_cells_shuffle(cust_id, num_cells, cells_per_cust):
    """Deterministic shuffle assignment — same customer always gets same cells."""
    rng = random.Random(hashlib.md5(str(cust_id).encode()).hexdigest())
    return rng.sample(range(num_cells), cells_per_cust)

def cluster_run(env, customers, cluster_q, hist_per_cust):
    """Single-cluster system — all customers contend for one queue."""
    def gen(cust_id):
        rate = ARRIVAL_RATE_HZ * (NOISY_MULTIPLIER if cust_id == NOISY_CUSTOMER else 1)
        while True:
            yield env.timeout(random.expovariate(rate))
            env.process(serve_cluster(env, cust_id, cluster_q, hist_per_cust))
    for c in customers: env.process(gen(c))

def serve_cluster(env, cust_id, q, hist):
    start = env.now
    with q.request() as req:
        yield req
        yield env.timeout(random.expovariate(1.0/(SERVICE_TIME_MS/1000)))
    hist[cust_id].record_value(int((env.now - start) * 1_000_000))  # µs

def cell_run(env, customers, cells, assignments, hist_per_cust):
    """Cellular system — each customer fans out to its 4 assigned cells."""
    def gen(cust_id):
        rate = ARRIVAL_RATE_HZ * (NOISY_MULTIPLIER if cust_id == NOISY_CUSTOMER else 1)
        while True:
            yield env.timeout(random.expovariate(rate))
            chosen = random.choice(assignments[cust_id])
            env.process(serve_cell(env, cust_id, cells[chosen], hist_per_cust))
    for c in customers: env.process(gen(c))

def serve_cell(env, cust_id, cell_q, hist):
    start = env.now
    with cell_q.request() as req:
        yield req
        yield env.timeout(random.expovariate(1.0/(SERVICE_TIME_MS/1000)))
    hist[cust_id].record_value(int((env.now - start) * 1_000_000))

def run_scenario(scenario):
    env = simpy.Environment()
    customers = list(range(NUM_CUSTOMERS))
    hist = {c: HdrHistogram(1, 60_000_000, 3) for c in customers}
    if scenario == "cluster":
        # Cluster capacity = sum of all cell capacities
        q = simpy.Resource(env, capacity=NUM_CELLS)
        cluster_run(env, customers, q, hist)
    else:
        cells = [simpy.Resource(env, capacity=1) for _ in range(NUM_CELLS)]
        assignments = {c: assign_cells_shuffle(c, NUM_CELLS, CELLS_PER_CUST) for c in customers}
        cell_run(env, customers, cells, assignments, hist)
    env.run(until=SIM_DURATION_S)
    return hist

if __name__ == "__main__":
    for scenario in ("cluster", "cell"):
        hist = run_scenario(scenario)
        noisy_p99   = hist[NOISY_CUSTOMER].get_value_at_percentile(99) / 1000
        # Median p99 across all non-noisy customers
        innocent_p99s = [hist[c].get_value_at_percentile(99) / 1000
                         for c in range(1, NUM_CUSTOMERS) if hist[c].get_total_count() > 50]
        innocent_median_p99 = statistics.median(innocent_p99s)
        innocent_max_p99    = max(innocent_p99s)
        print(f"{scenario:>8}  noisy p99 = {noisy_p99:6.1f} ms   "
              f"innocent median p99 = {innocent_median_p99:6.1f} ms   "
              f"innocent max p99 = {innocent_max_p99:6.1f} ms")

Sample run on a 2024 MacBook M3 Pro (Python 3.11, simpy 4.1):

$ python3 cells_vs_clusters.py
 cluster  noisy p99 =  142.3 ms   innocent median p99 =   89.4 ms   innocent max p99 =  138.7 ms
    cell  noisy p99 =  165.8 ms   innocent median p99 =    8.1 ms   innocent max p99 =   28.6 ms

The lines that carry the lesson:

• cluster ... innocent median p99 = 89.4 ms — this is the noisy-neighbour effect. The innocent customers are paying the noisy customer's queueing tax. Their p99 has climbed to nearly the noisy customer's because they share the queue.
• cluster ... innocent max p99 = 138.7 ms — the worst-affected innocent customer is nearly indistinguishable from the noisy customer in latency. In the cluster model, "noisy neighbour" is contagious to every co-tenant.
• cell ... innocent median p99 = 8.1 ms — under shuffle-sharding, the median innocent customer is unaffected. Why the median drops so dramatically: with 100 cells and 4 cells per customer, the noisy customer occupies only 4 of the 100 cells. The probability that an innocent customer's 4 cells include at least one of the noisy customer's 4 cells is 1 − C(96,4)/C(100,4) ≈ 0.155 — but even when there is overlap, the innocent customer can route around to their other 3 cells. The median innocent customer sees zero overlap and zero impact; the 85th-percentile-affected innocent customer sees one overlap and routes around it, paying maybe 25% extra latency. The combinatorial isolation is doing the work.
• cell ... innocent max p99 = 28.6 ms — the worst-affected innocent customer (one who happened to share more cells with the noisy customer) still has 4× better p99 than the cluster median. The combinatorial bound is real; the worst case is bounded.
• cell ... noisy p99 = 165.8 ms — the noisy customer's own p99 is slightly worse in the cellular model than the cluster, because they have less capacity available to them (4 cells × 1 capacity = 4 servers instead of the 100-server cluster). The cellular architecture deliberately punishes the noisy customer for being noisy by isolating their impact to their own cells. This is the isolation property: the cell architecture refuses to share excess capacity, which means noisy customers absorb their own noise instead of distributing it.

The fifteen-line difference in the output is the entire 4-hour 2017 S3 outage, the entire shuffle-sharding patent, and the entire reason AWS rebuilt their architecture. Run it once on your laptop and the cellular argument becomes physically intuitive in a way that no slide deck reproduces.

A third experiment worth running in the simulator: vary CELLS_PER_CUST from 2 to 8 and watch the trade-off curve. At cells_per_cust = 2, isolation is weaker (15% of innocent customers share at least one cell with the noisy customer); at cells_per_cust = 8, isolation is stronger (52% share at least one cell, but the routing fan-out is 2× more expensive). The optimum is workload-dependent — Route 53 uses 4-of-100 because their per-customer query rate is low enough that 4× fan-out is affordable; DynamoDB uses larger cell counts and smaller per-customer assignments because their per-customer query rates are higher. The simulator lets a working engineer find the right operating point for their own workload in 15 minutes instead of guessing.

What cellular costs — the efficiency tax

The argument so far has been one-sided: cells are better. They are not unconditionally better — they cost something specific, and engineers who skip the cost analysis reach for cells when they should not. The cost is efficiency.

A cluster with 100 servers can serve any traffic distribution because all 100 servers share a queue. A cellular system with 100 cells of 1 server each cannot share capacity across cells. If cell-7 happens to receive 50% of the traffic this minute (because the customers assigned to cell-7 are all having a flash sale at the same time), cell-7's queue fills while cells 1-6 and 8-100 sit idle. The cluster aggregates demand; the cell does not. Why this efficiency cost is unavoidable: the same property that gives cells their isolation — "no shared capacity" — is the property that prevents capacity pooling. You cannot have both. A "cellular system that pools capacity" is just a cluster; a "cluster with hard isolation" is just a set of cells. The two are mathematically the same trade-off seen from opposite directions, and the operator has to pick a point on the spectrum.

The empirical cost number, from AWS published architecture talks: a cellular system requires roughly 1.5× to 2× the capacity of an equivalent cluster to absorb the same peak traffic, because each cell must be provisioned for its own peak rather than the regional average. AWS pays this 1.5–2× tax willingly because the blast-radius reduction is worth it; most internal teams at most companies do not pay this tax, which is why most companies still run clusters.

A second cost: operational complexity. A cellular system needs a routing layer that maps customer-id to cells, a deployment system that rolls out new code one cell at a time, an observability system that aggregates per-cell metrics into customer-facing dashboards, and a per-cell on-call rotation that can debug a cell in isolation. Each of these is a separate engineering investment. AWS made all four investments because they have thousands of engineers; a 50-engineer fintech in Bengaluru typically does not, and rebuilding their service as cellular would consume a year of engineering for benefits that mostly do not show up until they reach AWS scale.

A third cost worth naming: the routing layer is the new single point of failure. If the routing layer that maps customer-id to cells goes down, no cell receives traffic, and the cellular property reduces to zero. AWS's solution is to make the routing layer itself cellular (Route 53 is a cellular service routing requests to other cellular services), but this recursion has to terminate somewhere. The "DNS cellular meta-architecture" in AWS's design — where each region has its own DNS resolver fleet, each AZ has its own DNS cache, and clients have multiple DNS servers configured — is the bottom of the recursion: at some point you trust the AWS regional control plane and accept that that layer is a shared dependency. The recursion bottoms out at a layer simple enough that the engineering investment to harden it (formal verification, exhaustive testing, redundant deployments) is feasible. Most internal services cannot afford this terminal-layer hardening, which is another reason "go cellular" is bad advice for most teams.

How Indian platforms are adopting cellular patterns

The cellular pattern is moving from AWS-published architecture to Indian-platform engineering practice, but slowly and selectively. The adoption order tracks the same logic as the original tail-at-scale paper's mechanism adoption: companies adopt cellular for the failure modes that have already burned them, not for theoretical robustness.

Razorpay — UPI payment processing was the first Razorpay system to go cellular, in 2023, after a single-pool incident in 2022 took down 100% of merchant traffic for 47 minutes. The cellular redesign assigns each merchant to 3 of 12 backend pools, with shuffle-sharded routing. The published 2024 number: blast-radius reduction from 100% (single pool) to roughly 25% (three pools, but only the merchants whose 3-of-12 includes the affected pool see degradation). Capacity cost: 1.6× over the previous monolithic design, in line with the AWS 1.5–2× tax estimate.

Hotstar — IPL-final video catalogue and ad-decision services run as cellular within each region, with each cell handling roughly 5% of regional traffic. The 2024 IPL final (peak 32M concurrent viewers) was the first finalised production deployment of full cellular architecture for video metadata. A single cell failure during the match would have affected ~1.6M viewers; the cluster-baseline equivalent would have affected all 32M. Their published architecture talk noted that the cellular cost (capacity over-provisioning) was offset by faster deployment confidence — they could deploy a new build to one cell, watch it for 30 minutes, then proceed cell-by-cell across the region, instead of all-at-once-and-hope. The deployment-velocity benefit was, per their team, larger than the blast-radius benefit on a day-to-day basis.

Zerodha Kite — order-routing is partially cellular (per-stock-symbol micro-partitioning is the cellular layer), but the customer-facing API gateway is still a cluster. Zerodha's published architecture choice is that the order-matching engine is the highest-stakes component (where cellular is justified), but the read-heavy quote-fetch path is fine as a cluster because read failures are recoverable in milliseconds whereas order failures are visible to traders. The selective cellular adoption — apply cellular where blast radius is unrecoverable, accept cluster where it is — is the empirically common Indian-platform pattern.

Flipkart — Big Billion Days catalogue went cellular for the inventory and pricing services in 2023 after the 2022 incident where a stuck inventory-update worker thread took down catalogue browsing for 40 minutes during the BBD lightning sale window. Flipkart's per-product-category cellular layout means a stuck worker on the electronics-category cell does not affect fashion-category browsing. The capacity cost was published as 1.8×; the BBD-window blast-radius reduction was the explicit business case.

PhonePe / Paytm — UPI handlers are cellular per-customer-segment (one cell handles segment A merchants, another cell handles segment B, etc.) but the routing layer that maps customers to cells is still a single shared service. This is the recursion-not-yet-terminated problem; PhonePe's published 2024 architecture talks acknowledge it explicitly and describe an in-progress effort to make the routing layer cellular itself. The reason they have not done it yet is that the routing layer is consulted on every request and runs at extreme QPS, and rebuilding it as cellular requires solving the meta-routing problem (which cell of the routing layer handles which customer's routing requests). The recursion is genuinely hard at the bottom layer, and PhonePe's roadmap for solving it is roughly 18 months as of late 2024.

The pattern across these adoptions: every Indian platform has gone cellular for at least one component, none have gone fully cellular across all components, and the adoption curve is driven by post-incident learning rather than greenfield design. A team that has not yet had an incident requiring cellular is, rationally, better off paying the cluster's lower efficiency tax until they have one — at which point the post-mortem provides both the engineering investment and the political will to do the cellular rebuild.

Common confusions

• "Cells are the same as shards" Different. A shard splits the data; a cell splits the entire stack (front-end, metadata, storage, deployment). A cluster can be sharded internally and still have 100% blast radius because the shared front-end or deploy system can fail. A cellular system has zero shared layers in the data path, so the blast radius is bounded by 1/num_cells. Sharding without cellular gives the cost of sharding without the blast-radius benefit.
• "Cells are the same as multi-region" Different. Multi-region operates at the AWS-region boundary (the unit of failure is "us-east-1 went down"), and the customer pays 1.5–2× capacity for region redundancy. Cells operate within a region, at much finer granularity (the unit of failure is "cell-742 went down"). They are complementary — a service should be both multi-region and cellular within each region — but the engineering investments are independent.
• "Cells eliminate the need for tail-mitigation mechanisms like hedging" No. Hedging masks the slow tail of individual requests; cells bound the blast radius of failed components. They operate at different timescales — hedging at the millisecond, cells at the second-or-longer. A cellular service still needs hedging within each cell, and the two compose multiplicatively (hedging reduces in-cell tail; cellular bounds cross-customer impact). Skipping hedging because "we're cellular" leaves p99 elevated within the affected cell.
• "More cells is always better" False. Beyond a certain cell count, the efficiency tax dominates and the blast-radius benefit becomes asymptotic. A 10,000-cell system has 0.01% blast radius vs a 1000-cell system's 0.1% — a 10× improvement that costs another 1.3-1.5× capacity. The marginal improvement at the high-cell-count end is rarely worth it; AWS S3 settled at "thousands" of cells, not "millions", because the engineering and capacity overhead of finer cellularisation stops paying off.
• "Cellular architecture is just for hyperscalers" Mostly true today, but the threshold is dropping. The minimum company size where cellular makes economic sense was roughly "10,000 engineers" in 2017 and is roughly "200 engineers" today, because the routing-layer and deployment-system tooling has commoditised (Envoy + Istio + ArgoCD make it a configuration problem, not a build-from-scratch problem). The economic threshold is dropping by roughly 30% per year as tooling improves; expect mid-size Indian fintechs and B2B SaaS companies to reach the cellular threshold over the next 3-4 years.
• "Shuffle sharding fixes everything" No. Shuffle sharding works when the failure mode is per-cell and customers can route around it. It does not help when the failure mode is global (the routing layer goes down) or when the customer's request is non-idempotent and cannot be retried against another cell. The mechanism is powerful but bounded; it composes with other patterns (deadline propagation, idempotency, retries) and is not a substitute for them.

Going deeper

The 2017 S3 outage post-mortem — read it before the next on-call shift

The AWS post-mortem for the February 28, 2017 S3 outage is publicly available (linked in References) and is required reading for any engineer working on storage or routing systems. The technical narrative: a maintenance command intended to remove a small number of index-management servers contained a typo that removed too many; the surviving index servers were overloaded; recovery required cold-cache rebuilding which took 4 hours. The structural lesson — the lesson AWS extracted and acted on — is that the index management subsystem was a single shared layer across the region, and any failure mode (operator typo, software bug, hardware fault) that took it down would take down all of S3 in that region. The cellular rebuild that followed did not fix the typo (operators will always make typos); it fixed the invariant that a typo in one place could affect every customer. Why this is the most important lesson in modern systems architecture: post-mortems that produce "improve operator tooling" or "add more reviewers to the change process" are remediation theatre — they reduce the rate of incidents but do not bound the blast radius. The post-mortems that change architecture, like the 2017 S3 one, are the ones that actually matter. A team's quality of incident response is measured not by how fast they restore service after an incident, but by how much smaller the next equivalent incident's blast radius is. Cellular architecture is the result of taking that measurement seriously.

The relationship to the tail-at-scale paper — masking vs containing

The Dean & Barroso paper's mechanisms (hedging, backup requests, micro-partitioning) mask the tail by adding redundancy at request time. Cellular architecture contains the failure by adding redundancy at structural time. The two are not alternatives; they are complementary. A modern AWS service uses hedging within each cell (to mask the per-request tail) and cellular boundaries between groups of customers (to contain the per-customer blast radius). The chapter on /wiki/google-the-tail-at-scale-paper-revisited covers the masking side; this chapter covers the containment side. The full picture requires both: a service that has hedging without cellular has good per-request tails but unbounded blast radius; a service that has cellular without hedging has good blast-radius bounds but elevated p99 within each cell. AWS S3's published architecture has both layered, and the architecture talks emphasise that they were built as separate engineering investments at different times.

Shuffle sharding's mathematical foundation — combinatorial isolation

The shuffle-sharding probability of overlap is given by the hypergeometric distribution. With N total cells and k cells assigned per customer, the probability that two customers share at least m cells is 1 − Σ_{i=0}^{m-1} C(k, i) C(N-k, k-i) / C(N, k). For N=100, k=4, m=1 (sharing at least one cell), this is ≈ 0.155. For m=2 (sharing at least two cells), this is ≈ 0.0036. For m=4 (sharing all four cells), this is 1 / C(100, 4) ≈ 2.55 × 10⁻⁷. The combinatorial explosion at higher overlap counts is what makes shuffle sharding work: the probability of complete overlap is so small that even with millions of customers, the expected number of customer pairs sharing all four cells is bounded. The original AWS analysis paper (linked in References) works through the math for various (N, k) configurations; the takeaway is that k=4 of N=100 is the natural sweet spot for the "every pair of customers is combinatorially isolated" property at most production scales.

Why the deployment system is the hidden cellular component

A subtle property of cellular architecture often missed by readers: the deployment system itself must be cellular, or the cellular property fails. If a single deploy pipeline pushes a new build to all 1000 cells simultaneously, a bug in the new build affects all 1000 cells at once, and the blast radius is 100% despite the data-path cellular boundaries. AWS's solution is per-cell deployments: a new build rolls out to one cell, sits there for some time (typically hours), then progresses to the next cell, and so on. The deployment takes longer (rolling out to 1000 cells at 4 cells per hour means 250 hours, ~10 days), but a bad build is detected on cell 1 and never reaches cell 2. The deployment-velocity vs blast-radius trade-off is explicit, and AWS publishes their deployment cadence data showing that they accept much slower regional rollouts than a cluster-architecture deploy would have. Indian platforms adopting cellular often miss this — they put the data path in cells but keep a single shared deploy pipeline, and the first bad-build incident shows them why both layers must be cellular. The lesson: the deploy system is part of the data path for the purposes of blast-radius analysis, even though it does not appear in the request flow diagram.

Reproduce this on your laptop

# Reproduce the cellular vs cluster blast-radius experiment
python3 -m venv .venv && source .venv/bin/activate
pip install simpy hdrh numpy
python3 cells_vs_clusters.py
# Expected: cluster shows innocent customers paying noisy-neighbour tax
# (innocent p99 ≈ noisy p99); cell shows innocent customers protected
# by combinatorial isolation (innocent median p99 ≈ baseline service time).

# Compute the shuffle-sharding overlap probability for your own (N, k)
python3 -c "
from math import comb
N, k = 100, 4
for m in (1, 2, 3, 4):
    p = 1 - sum(comb(k,i)*comb(N-k,k-i) for i in range(m)) / comb(N,k)
    print(f'P(share >= {m} cells) = {p:.6e}')
"
# Expected: P(>=1)=1.55e-1, P(>=2)=3.6e-3, P(>=3)=4.3e-5, P(>=4)=2.55e-7

A reader who runs both blocks gets the empirical result (cellular protects innocent customers) and the closed-form arithmetic (the combinatorial overlap probabilities) in fifteen minutes. The pair calibrates the intuition: cellular works because of the combinatorial math, and the simulator confirms the math holds under realistic queueing dynamics.

The 2024 trend — cellular for AI inference

A recent development worth flagging: AI inference services (OpenAI's API, Anthropic's API, AWS Bedrock) are adopting cellular architecture for GPU pools. The reasoning is the same as for storage: a single noisy customer (a model fine-tuning run that hammers one pool with 1000× normal traffic) used to degrade latency for every customer co-located on that pool. The 2024 cellular redesigns at multiple inference providers shuffle-shard customers across GPU pools, with each customer assigned to 4-of-N pools. The capacity cost is substantial (GPUs are expensive, and 1.5-2× over-provisioning is real money) but the blast-radius benefit is too large to ignore for paying customers with SLOs. Indian context: Sarvam AI and Krutrim, when scaling their inference infrastructure to support enterprise customers in 2024-2025, are following the same pattern. The cellular architecture pattern that AWS published for storage in 2017 is now being applied to AI inference in 2024, with the same arithmetic and the same trade-offs.

Cell sizing — when one cell becomes two

A cell is not a fixed entity; it grows with its customers, and at some point it must be split. AWS's published heuristic is that a cell should never exceed roughly 30-50% of regional capacity for any single resource (CPU, memory, network, storage IOPS), because beyond that point a cell failure approaches the cluster baseline. When a cell crosses the threshold, the operations team performs a cell split — partitioning the cell's customers into two new cells, each provisioned at the original capacity. The split is operationally expensive: customer routing tables update, in-flight requests must be drained gracefully, and the new cell must catch up on any state replication before taking traffic. AWS S3's published cadence is that a typical cell splits every 6-18 months as customer volume grows, and the operations team runs roughly 50-100 cell splits per region per year. The cell split is to cellular architecture what auto-scaling is to cluster architecture: the routine operational mechanism that keeps the property invariant under growth. Indian platforms adopting cellular often skip the split-mechanism design in v1, and discover six months later that their largest cell is now 60% of total capacity and they have to split it under pressure rather than under a planned procedure. The lesson: design for cell splits before you need them.

Where this leads next

The closing chapter of Part 16 (/wiki/the-30-year-arc-of-systems-performance) places cells, hedging, queueing theory, NUMA, and every other mechanism in a single 30-year frame, and traces the pattern of how each generation's bottleneck became the next generation's solved problem.

The natural next reads are:

• /wiki/google-the-tail-at-scale-paper-revisited — the previous case study on the masking-side of tail mitigation.
• /wiki/the-30-year-arc-of-systems-performance — Part 16's closing synthesis chapter.
• /wiki/queueing-theory-littles-law — the queueing foundation that explains why noisy neighbours degrade co-tenants.
• /wiki/backup-requests-and-bounded-queueing — the in-cell tail-masking mechanism that composes with cellular boundaries.
• /wiki/capacity-at-99-99 — the capacity-planning chapter that quantifies the cellular efficiency tax.

A reader who has worked through Parts 7 (tail latency), 8 (queueing theory), and 14 (capacity planning) should now have the full mental model: tail behaviour is masked at the request level by hedging, contained at the structural level by cells, and predicted at the capacity-planning level by USL fits. The three layers compose, and a production system that has all three is far more available than any single layer alone could provide.

A useful exercise for any working backend engineer: take your team's most failure-prone service, count the customers it serves, and ask the question "what fraction of those customers does our worst-case incident affect?" If the answer is "all of them" or "we don't know", the service is a cluster, and the next significant incident will affect 100% of customers regardless of any other engineering investment. If the answer is "a bounded fraction we can name", the service has at least informal cellular boundaries, and the engineering work is to formalise them. The gap between "informal" and "formalised" cellular boundaries is where most of the engineering value lives — informal boundaries leak across shared infrastructure (shared deploys, shared rate-limiters, shared secrets stores) in ways that surface only during incidents, and the formalisation is the systematic audit of every shared layer in the stack. The exercise takes a quarter-day; the results often surprise the team about how many shared layers their "cellular" service actually has.

References

• Amazon S3 Service Disruption in the Northern Virginia (us-east-1) Region — official AWS post-mortem (March 2, 2017) — the structural prompt that motivated the cellular rebuild.
• Colm MacCárthaigh — "Static stability and self-healing infrastructure" (re:Invent 2021) — the canonical AWS talk on cellular architecture.
• AWS Builders' Library — "Workload isolation using shuffle-sharding" — the published mathematical foundation of shuffle sharding.
• Marc Brooker — "Cellular architectures and bounded blast radius" (AWS blog, 2020) — the economic argument for the cellular efficiency tax.
• Razorpay Engineering — UPI cellular routing (blog, 2024) — the closest Indian-platform publication on cellular adoption.
• /wiki/google-the-tail-at-scale-paper-revisited — the masking-side companion chapter.
• /wiki/queueing-theory-littles-law — the queueing foundation for noisy-neighbour analysis.
• Brendan Gregg, Systems Performance (2nd ed., 2020), Ch. 13 (Case Studies) — the canonical text the curriculum draws on.