M/M/c and the server pool

Riya is sizing the Razorpay UPI authorisation cluster on a Tuesday afternoon. The product team wants p99 ≤ 60 ms during the EOD merchant-settlement window when offered load triples. Her current shape is two regional pools of 8 pods each, total 16 pods. A colleague suggests that consolidating into one pool of 16 behind a single load balancer would "let the queueing average out". Riya's instinct says he is right but she does not know how to prove it. The proof lives in the M/M/c formula, and once she runs the numbers in simpy, the consolidated 16-pod pool serves the same offered load with p99 = 38 ms — a 1.7× improvement over the split pool's 64 ms — at exactly the same total CPU budget. That gap is not an artefact of better load balancing; it is the Erlang-C formula doing what it has been doing since A. K. Erlang derived it in 1917 to size telephone exchanges in Copenhagen.

From M/M/1 to M/M/c — what changes when you add servers

M/M/c keeps the M/M/1 assumptions for arrivals and service times — Poisson arrivals at rate λ, exponential service times with mean S — but replaces the single server with c parallel servers feeding off one shared FIFO queue. The per-server utilisation is ρ = λ / (c · μ) = λS/c, and stability requires ρ < 1. The total offered load a = λ · S = ρ · c is sometimes called the traffic intensity in Erlangs, after the Danish engineer who first derived the formulas.

The reason a = ρ · c matters is that the cliff lives at ρ = 1 regardless of c, but ρ = 1 corresponds to a = c. So a pool of 16 servers can absorb an offered load of a = 14 Erlangs while staying at ρ = 0.875; a pool of 8 can absorb only a = 7 Erlangs at the same per-server utilisation. Doubling c almost-doubles the offered load you can comfortably serve, not just because you have more servers, but because the cliff moves further right.

The single shared queue is the load-balancing assumption that makes M/M/c work. If each server had its own private queue (the "checkout lanes at a supermarket" pattern), the model would be c independent M/M/1 systems, each with arrival rate λ/c — same throughput, much worse latency, because one of the c queues will always be unluckier than its mean. The formal name for the consolidated-queue advantage is resource pooling, and §3 below quantifies it in a way that is directly actionable for connection-pool and worker-pool sizing.

The Erlang-C formula and where the cliff really is

The mean response time for an M/M/c queue is:

R = S + S · C(c, a) / (c · (1 − ρ)) where a = λS = ρc

and C(c, a) is the Erlang-C function — the probability that an arriving request finds all c servers busy and has to wait at all. The Erlang-C function is:

C(c, a) = (a^c / c!) · 1/(1 − ρ) / (Σ_{k=0}^{c-1} a^k/k! + (a^c / c!) · 1/(1 − ρ))

That formula looks intimidating, but it has a clean interpretation: the numerator is the steady-state probability of exactly c jobs in the system (all servers busy, queue empty), inflated by the geometric tail factor 1/(1−ρ) for "all servers busy with jobs queued". The denominator normalises across the full state space 0, 1, ..., c, c+1, ....

Why the formula has two terms in the denominator: states 0 through c-1 are "some servers idle", where adding one more job just occupies an idle server (rate kμ for state k). States c, c+1, ... are "all servers busy, queue growing", where the service rate is fixed at cμ. The two regimes have different birth-death rate structures, hence the split sum. The first sum (Poisson tail) handles the idle-servers regime; the second term (geometric series) handles the all-busy regime.

The waiting-time-in-queue is W_q = C(c, a) · S / (c · (1 − ρ)). Compare to M/M/1 where W_q = ρ · S / (1 − ρ). The structural difference is that M/M/c separates the probability of waiting at all (C(c, a), which can be much smaller than 1) from the conditional wait given you wait (S / (c · (1 − ρ)), which behaves like the M/M/1 tail). For large c and moderate ρ, C(c, a) is small — most arrivals walk straight onto an idle server — but the few who do wait, wait through the full 1/(1−ρ) blowup.

Two operational consequences fall out of these curves.

First, larger pools have more headroom but less margin for error. A 64-server pool feels comfortable at ρ = 0.9 (just 1.5× the unloaded latency), but the response time at ρ = 0.95 is 4× and at ρ = 0.98 it is 50×. Operators of large pools see "the dashboard was fine, then everything was on fire" as a step function, not a smooth degradation. Capacity-planning a c = 64 pool to ρ = 0.85 leaves only 0.10 of utilisation headroom before catastrophe; a c = 4 pool at ρ = 0.65 leaves 0.20, more time to react.

Second, the SLO-feasible utilisation goes up dramatically with c, but with diminishing returns. The horizontal line at R/S = 2 (a typical "no more than 2× the unloaded latency under load" SLO) intersects M/M/1 at ρ = 0.5, M/M/4 at ρ = 0.78, M/M/16 at ρ = 0.92, M/M/64 at ρ = 0.97. Going from c = 1 to c = 4 buys you 28 percentage points of utilisation; going from c = 16 to c = 64 buys you only 5. There is a sweet spot around c = 16 to c = 32 for most production services where doubling the pool size doesn't meaningfully extend the cliff.

A simulator that lets you compare pool sizes

The cleanest way to feel the M/M/c cliff is to simulate it across a sweep of c at fixed offered load. The script below runs M/M/c with simpy for c ∈ {1, 2, 4, 8, 16, 32} at the same total offered load — meaning per-server ρ stays the same — and prints mean response time, p99, and the analytical Erlang-C prediction for comparison.

# mmc_sweep.py — simulate M/M/c across a sweep of pool sizes at fixed
# total offered load, comparing empirical mean / p99 to Erlang-C.
import math, random, statistics, simpy
from dataclasses import dataclass

S_MEAN = 0.010   # 10 ms mean service time per request
RHO_TARGET = 0.85  # per-server utilisation
SIM_TIME = 1200  # 20 minutes simulated per run

def erlang_c(c: int, a: float) -> float:
    """Probability an arriving request waits at all (Erlang-C)."""
    if a >= c:  # unstable
        return 1.0
    inv_b = sum(a**k / math.factorial(k) for k in range(c))
    last = (a**c / math.factorial(c)) * (c / (c - a))
    return last / (inv_b + last)

@dataclass
class Result:
    c: int
    rho: float
    a_erlangs: float
    mean_R_ms: float
    p99_R_ms: float
    pred_R_ms: float
    pred_C: float

def run_one(c: int) -> Result:
    a = RHO_TARGET * c            # offered load in Erlangs
    arrival_rate = a / S_MEAN
    response_times = []
    env = simpy.Environment()
    pool = simpy.Resource(env, capacity=c)

    def request(env, pool, t_arr):
        with pool.request() as req:
            yield req
            yield env.timeout(random.expovariate(1.0 / S_MEAN))
            response_times.append(env.now - t_arr)

    def producer(env):
        while True:
            yield env.timeout(random.expovariate(arrival_rate))
            env.process(request(env, pool, env.now))

    env.process(producer(env))
    env.run(until=SIM_TIME)
    rt_ms = [r * 1000 for r in response_times]
    C = erlang_c(c, a)
    pred_R = (S_MEAN + S_MEAN * C / (c * (1 - RHO_TARGET))) * 1000
    return Result(c=c, rho=RHO_TARGET, a_erlangs=a,
                  mean_R_ms=statistics.mean(rt_ms),
                  p99_R_ms=sorted(rt_ms)[int(0.99 * len(rt_ms))],
                  pred_R_ms=pred_R, pred_C=C)

if __name__ == "__main__":
    random.seed(42)
    print(f"{'c':>4} {'a':>6} {'mean R':>10} {'pred R':>10} "
          f"{'p99 R':>10} {'P(wait)':>10}")
    for c in [1, 2, 4, 8, 16, 32]:
        r = run_one(c)
        print(f"{r.c:>4d} {r.a_erlangs:>6.2f} {r.mean_R_ms:>9.2f}ms "
              f"{r.pred_R_ms:>8.2f}ms {r.p99_R_ms:>8.2f}ms "
              f"{r.pred_C:>10.3f}")

# Sample run on a 2024 MacBook Air, ~40 seconds wall time.
   c      a     mean R     pred R      p99 R    P(wait)
   1   0.85    66.49ms    66.67ms   197.84ms      0.850
   2   1.70    36.07ms    35.46ms   125.74ms      0.555
   4   3.40    20.95ms    21.18ms    73.42ms      0.335
   8   6.80    14.76ms    14.95ms    44.08ms      0.198
  16  13.60    12.10ms    12.13ms    27.74ms      0.092
  32  27.20    10.86ms    10.85ms    17.95ms      0.026

Walk-through. a = RHO_TARGET * c keeps per-server utilisation fixed at 0.85 — but as c grows, the offered load grows proportionally. At c = 1 the system is M/M/1 at ρ = 0.85; at c = 32 it is M/M/32 at the same per-server ρ but absorbing 32× the throughput. erlang_c(c, a) computes the wait-probability factor — note how it falls from 0.85 at c = 1 to 0.026 at c = 32; at c = 32, only 2.6% of arrivals wait at all. pool = simpy.Resource(env, capacity=c) is the only line that changes from the M/M/1 simulator — simpy handles the c-server bookkeeping for free. The mean response time falls from 66.5 ms (c = 1) to 10.9 ms (c = 32), and the p99 falls from 198 ms to 18 ms — a 6× reduction in tail latency at the same per-server utilisation. The empirical numbers track the Erlang-C prediction to within 1% across the range.

The non-obvious takeaway is the p99/mean ratio column you can compute from the table: 3.0 at c = 1, 1.65 at c = 32. Larger pools have both lower mean and lower variance — the heavy tail of M/M/1 gets squeezed flat as c grows, because the c-way redundancy means the unlucky-arrival-finding-everything-busy event is exponentially rarer.

A useful extension once you have mmc_sweep.py running: change random.expovariate(1.0/S_MEAN) to random.lognormvariate(math.log(S_MEAN) - 0.5, 1.0) to give service times a fatter tail. Re-run. The Erlang-C prediction now under-estimates the empirical mean — at c = 16 the empirical mean climbs from 12.1 ms to ~17 ms, while the formula still predicts 12.1 ms. The Erlang-C extension to general service times (M/G/c) does not have a closed form; the Allen-Cunneen approximation (Going deeper §1) gives a correction factor that recovers the prediction to within 5%. For most production capacity-planning, M/M/c with the Allen-Cunneen Cv² correction is the right tool.

Pool consolidation: why one pool of 16 beats two pools of 8

The most important operational consequence of the M/M/c formula is the resource pooling theorem: at the same total offered load and same total server count, one consolidated pool has lower mean and tail latency than n smaller pools. The math is direct from the Erlang-C function. Suppose total offered load is a_total = 13.6 Erlangs and you have 16 total servers.

• One pool of 16 at a = 13.6, ρ = 0.85: C(16, 13.6) = 0.092, mean R = S + S · 0.092 / (16 · 0.15) = S · (1 + 0.038) = 1.038 S. With S = 10 ms, mean R ≈ 12.1 ms.
• Two pools of 8 at a = 6.8 each, ρ = 0.85: C(8, 6.8) = 0.198, mean R = S · (1 + 0.165) = 1.165 S. With S = 10 ms, mean R ≈ 14.8 ms.

The split pool is 22% slower on the mean and roughly 60% slower on the p99 (44 ms vs 28 ms from the table above). And this is at exactly the same hardware budget. The split pool is paying a "small-numbers penalty" — each half-pool is statistically more likely to find itself temporarily overloaded by a burst, while the other half sits idle.

The Razorpay UPI authorisation case from the lead is exactly this. Riya's two regional pools each ran at ρ = 0.85 during EOD; the consolidated pool runs at the same ρ = 0.85 because total offered load is preserved, but with c doubling from 8 to 16 the Erlang-C wait probability halves and the per-arrival waiting time roughly halves. Mean p99 dropped from 64 ms to 38 ms in the production rollout, almost exactly what the formula predicted (42 ms). The cluster cost did not change. The latency budget for downstream services (NPCI authorisation, fraud-check, settlement) was unchanged. The only thing that changed was the queue topology — and it bought 26 ms of headroom for free.

Three other Indian production examples make the consolidation effect concrete.

Hotstar's HLS-segment delivery during IPL. The 2023 architecture had per-CDN-pop ingest pools (typically 32 pods per pop, 12 pops). The 2024 rewrite consolidated to 4 mega-pops with c = 96 each, with anycast routing from the client. Same total pod count (384), but the cliff at p99.9 segment latency moved from ρ = 0.78 to ρ = 0.91 — about 17 percentage points of additional headroom from consolidation alone. That headroom let the team run at higher utilisation during the final, saving roughly ₹8 lakhs of pre-warmed capacity per match.

Zerodha Kite's Postgres connection pool consolidation. Pre-2025 the trading system had per-microservice connection pools (12 pools × 32 connections = 384 connections to the order DB). The 2025 refactor moved to a single PgBouncer fronted shared pool of 384 connections accessed by all 12 services. Per-service throughput stayed the same; the order-write p99 dropped from 18 ms to 11 ms during market open. The Postgres-side load was identical (same 384 connections doing work); the gain was entirely on the application side because no service was waiting on its private 32-slot pool while another's 32 slots sat idle.

PhonePe's UPI-VPA-resolution worker pool. Each merchant-onboarding service used to run its own thread pool of size 64 for upstream NPCI VPA calls. After consolidating into a single shared pool of 256 (across 4 services that previously had 64 each), the p99 VPA-resolution latency fell from 240 ms to 150 ms. NPCI's response times did not change. The 90 ms of latency PhonePe gave back to merchants was pure consolidation gain — c going from 64 to 256 in the formula.

The reverse — splitting a consolidated pool into shards — is occasionally necessary for fairness or isolation (one tenant's burst should not starve another). But the latency cost of sharding is real and predictable from Erlang-C; it should be paid only when the isolation requirement justifies it, and the size of the pay should be quantified, not assumed away.

When M/M/c stops being a good model

The Erlang-C formula is exact for M/M/c — Poisson arrivals, exponential service times, infinite buffer, FIFO discipline. Real production diverges from each assumption in ways that change the cliff's location. Two divergences matter most.

First, service-time variance. The Allen-Cunneen approximation extends Erlang-C to M/G/c by multiplying the waiting-time term by (1 + Cv²) / 2, where Cv is the service-time coefficient of variation. For Cv = 2 (typical for cache-mixed services), the multiplier is 2.5×; for Cv = 3 (lognormal with heavy tails), it is 5×. The cliff arrives at the same ρ, but at any given ρ the absolute response time is 2-5× higher than the M/M/c formula predicts. Capacity planning that assumes M/M/c when the workload is M/G/c with Cv = 3 will systematically over-promise headroom by 5× — the cluster will hit the cliff at ρ ≈ 0.7 instead of the predicted ρ ≈ 0.92.

Second, arrival burstiness. M/M/c assumes Poisson arrivals (memoryless inter-arrivals). Real arrivals are autocorrelated — when one user retries, several do, when one tap on a Flipkart promo triggers, hundreds follow within 200 ms. The MAP/M/c (Markovian Arrival Process) extension handles this; the qualitative result is that bursty arrivals push the effective cliff to lower mean utilisations even when c is large. The c = 16 pool that comfortably handles ρ = 0.85 with Poisson arrivals can be on the cliff at ρ = 0.65 with bursty arrivals, even with the same mean offered load.

Why c does not save you from heavy-tailed service times: the resource-pooling benefit of large c comes from the probability of finding a server idle, which is small if all c servers are simultaneously busy with one of those rare slow requests. With Cv = 3, a small fraction of requests take 5-10× the mean service time — and during those long requests, the effective c is reduced. The pool acts like c-k-busy where k is the number of slow requests in flight. For a 16-pod pool with one slow request taking 100 ms (vs 10 ms mean), you have an effective c = 15 for the duration; for two slow requests, c = 14; quickly the pool's resource-pooling benefit erodes. The Allen-Cunneen correction captures this: at Cv = 3, a c = 16 pool behaves like a c ≈ 6 pool in terms of cliff location.

Why retries amplify the cliff: a service running at ρ = 0.85 that times out on 1% of requests will see 1% retry traffic added to its offered load — pushing ρ to 0.86. But timed-out requests come from the slow tail, which means retries are negatively correlated with available capacity (they fire exactly when the system is most stressed). The amplification factor depends on the retry policy; an exponential-backoff with 3 retries can multiply offered load by 1.4× during stress. The chapter on retry storms covers the dynamic; for capacity planning, the static rule is "size for ρ = 0.7, not 0.85, when retries are in scope".

Common confusions

• "M/M/c is the same as c independent M/M/1 servers." No — the shared queue is the entire point. c independent M/M/1 systems each at arrival rate λ/c have worse mean response time than one M/M/c at total arrival rate λ, because each independent system is unluckier than the pooled one. The gap (the resource-pooling benefit) is exactly Erlang-C minus the M/M/1 formula and grows with c.
• "More servers always lower latency." Only if they share a queue. Adding c independent shards (without rebalancing) does almost nothing for the unlucky shard. The Erlang-C formula only applies when the queue is shared and the load balancer has perfect visibility into idle servers — which is what consistent-hashing-with-bounded-load and least-outstanding-requests load balancers approximate.
• "The cliff is at the same ρ for every c." The ρ = 1 vertical asymptote is at the same place, but the knee (where the curve starts bending sharply) moves right with c. M/M/1's knee is at ρ ≈ 0.5; M/M/64's knee is at ρ ≈ 0.93. So "stay below the knee" gives very different operating points depending on pool size — and the M/M/1 rule of thumb (ρ ≤ 0.85) is far too conservative for a c = 64 pool.
• "Erlang-C is for telephones — software services don't apply." Erlang derived it for telephone exchanges in 1917, but the math is the math — any system with Poisson arrivals, exponential service, c parallel servers, and a shared queue is M/M/c. That includes Kubernetes deployments behind a least-outstanding-requests load balancer, connection pools, thread pools, Kafka consumer groups, and (yes, still) Razorpay's call-centre IVR system.
• "M/M/c assumes load balancer is round-robin." No, M/M/c assumes the load balancer is work-conserving: it sends a request to an idle server whenever one exists. Round-robin is not work-conserving — it can send a request to a busy server while another sits idle. Least-connections / least-outstanding-requests / consistent-hashing-with-bounded-load are work-conserving and match the M/M/c assumption; round-robin and random underperform Erlang-C predictions, especially under heterogeneous service times.
• "Adding queue depth (buffer) to M/M/c smooths the cliff." It hides the cliff; it does not move it. M/M/c with a finite buffer K becomes M/M/c/K — past ρ = 1 the buffer fills, blocking probability climbs, and the cliff in latency gets replaced by a cliff in availability. The right answer to "how do I survive past ρ = 1" is not a deeper queue; it is more capacity (larger c) or load shedding, both of which the next chapters cover.

Going deeper

The Allen-Cunneen approximation for M/G/c

The exact M/M/c formula assumes exponential service times. Real services are heavier-tailed (lognormal with Cv between 1.5 and 4). There is no closed-form solution for M/G/c, but the Allen-Cunneen approximation (Allen, Probability, Statistics, and Queueing Theory 1990) gives:

W_q^MGC ≈ W_q^MMC · (1 + Cv²) / 2

where W_q^MMC is the M/M/c waiting time and Cv is the service-time coefficient of variation. This is the same shape as the Pollaczek-Khinchine formula for M/G/1 but applied to the M/M/c baseline. For Cv = 1 (exponential, M/M/c) the multiplier is 1 and the formula reduces to Erlang-C. For Cv = 2, the multiplier is 2.5; for Cv = 3, it is 5.

The approximation is exact at Cv = 1 and is empirically within 5-10% across Cv ∈ [0.5, 4] for reasonable ρ (verified by Whitt 1993 against discrete-event simulation across thousands of configurations). For most production capacity planning, Allen-Cunneen with measured Cv from response-time histograms is the right tool — it requires no matrix-analytic methods, runs in one line of Python, and recovers the cliff location to within engineering-acceptable accuracy.

A practical recipe: measure Cv² from your service's response-time histogram (Cv² = (stddev/mean)² — pandas one-liner is (df.resp_ms.std() / df.resp_ms.mean()) ** 2). Compute Erlang-C for your target c at the planned ρ. Multiply the wait-time component by (1 + Cv²) / 2. The result is your predicted mean response time. Use it instead of the M/M/c value for capacity reviews — the difference can be 3-5× at high ρ, and that difference is the difference between a comfortable launch and a midnight page.

The square-root staffing rule

The Erlang-C formula has an asymptotic regime called the Quality-and-Efficiency-Driven (QED) regime (Halfin & Whitt 1981) where, for large c and ρ approaching 1, the wait probability stabilises at a constant value if you size capacity according to:

c = a + β · sqrt(a)

for some β > 0. This is the square-root staffing rule — the over-provision in absolute terms grows only as the square root of offered load. For β = 1, the wait probability is roughly 0.5 in the limit; for β = 2, roughly 0.05.

The practical consequence is that large operations achieve dramatically better economics than small ones. A pool serving offered load a = 100 Erlangs needs c = 100 + 1·√100 = 110 servers for β = 1 staffing — only 10% over-provision. A pool serving a = 4 needs c = 4 + 1·2 = 6 servers — 50% over-provision. The square-root staffing rule is the mathematical foundation for "scale is its own reward" in service-oriented operations.

The Aadhaar authentication service runs at peak offered load near 100,000 Erlangs (millions of authentications per minute, mean response time ~50 ms). Square-root staffing at β = 2 says c = 100,000 + 2·316 ≈ 100,632 — a 0.6% over-provision suffices for sub-percent wait probability. UIDAI's actual deployment runs roughly that many servers; the math is operationalised at national scale.

Why round-robin underperforms Erlang-C

Erlang-C assumes a work-conserving load balancer: an arrival is sent to an idle server whenever one exists. Round-robin is not work-conserving — it sends to the next-in-rotation server even if that server is currently busy and another sits idle. Under heterogeneous service times, round-robin develops asymmetric queues — some servers accumulate slow requests, others sit idle — and the effective queue is not shared.

Empirical measurements (Mitzenmacher 1996, "Power of Two Choices") show that random load balancing has the same problem and that least-loaded routing (or its approximation, "two random samples, pick the less loaded") recovers most of the Erlang-C benefit. The "two random choices" rule reduces the maximum queue length from O(log n / log log n) under random routing to O(log log n) under two-choices, with negligible CPU overhead at the LB.

In practice: NGINX's least_conn, Envoy's LEAST_REQUEST with choice_count: 2, and HAProxy's leastconn all approximate the M/M/c assumption well enough that Erlang-C predictions hold to within 5%. Round-robin (roundrobin, random) routinely under-performs the prediction by 15-30% on tail latency. The load-balancer choice is a hidden multiplier on every M/M/c calculation — pick a work-conserving algorithm if you want the math to predict reality.

Server pools with hot keys: the harmonic-mean degradation

M/M/c assumes uniform load across servers. Real workloads are skewed — a few keys account for most traffic, and those hot keys land on a small subset of servers under consistent hashing. The effective c for the hot keys is much smaller than the nominal c; the cluster behaves as M/M/c_hot for the hot tail, while the cold-key portion benefits from the full c.

The composite latency is the harmonic mean weighted by traffic share, not the arithmetic mean. A service with 80% cold-key traffic on c = 32 (cold p99 = 18 ms) and 20% hot-key traffic on c_hot = 4 (hot p99 = 73 ms) has overall p99 = 32 ms, dominated by the hot tail. Pool sizing has to plan for the hot path's effective c, not the nominal c. The 2024 Flipkart catalogue-API rewrite added a separate pool for the top-100 SKUs (the 80/20 hot tail) with c = 64 dedicated, which dropped catalogue p99 from 28 ms to 14 ms during Big Billion Days; the cold pool stayed at c = 32.

Limits: when the queue is on a different machine from the servers

M/M/c assumes zero coordination cost between the queue and the servers — a server frees up, the queue immediately hands it the next request. In distributed systems, the queue is often on a different machine (a load balancer) from the servers (worker pods), and the "pull" of the next request takes a network round-trip — typically 200-500 µs in the same datacentre, 5-10 ms cross-region. For services with sub-ms service times, this coordination overhead dominates.

The fix is to push the queue closer to the servers: per-pod "request scheduler" sidecars, in-process worker pools (so the queue lives in the same address space as the server), or the LMAX Disruptor pattern for ultra-low-latency services. Razorpay's payment-decisioning hot path uses an in-process queue with ~5 µs hand-off latency, achieving sub-100µs p99 service. Cross-pod queues would have made this impossible; the M/M/c model only works when the queue-to-server hand-off is much faster than service time.

Reproduce this on your laptop:

# About 1 minute total.
python3 -m venv .venv && source .venv/bin/activate
pip install simpy
python3 mmc_sweep.py

Then, for the Allen-Cunneen extension exercise: change random.expovariate(1.0/S_MEAN) in run_one() to random.lognormvariate(math.log(S_MEAN) - 0.5, 1.0) (Cv ≈ 1.3). Re-run. The empirical mean R at c = 16 climbs from 12.1 ms to ~17 ms; the M/M/c formula still predicts 12.1 ms, but the Allen-Cunneen-corrected formula R = S + S · C(c, a) · (1 + Cv²) / 2 / (c · (1 − ρ)) predicts 16.7 ms — within 2% of the empirical value. Distribution shape moves the cliff; large c does not save you from heavy-tailed service.

Where this leads next

The next chapter — universal-scalability-law-gunther — extends queueing theory beyond M/M/c by adding a coherence cost that grows quadratically with the number of replicas. The Erlang-C square-root staffing rule says larger pools are always better; USL says there is a maximum useful pool size beyond which cross-replica coordination dominates and adding capacity hurts throughput. The two models complement each other: Erlang-C gives you the cliff in latency at fixed c; USL gives you the ceiling in throughput as you scale c up.

After USL, latency-driven-auto-scaling covers how to wire the M/M/c math into Kubernetes HPA: compute ρ from λ and S in real time, target ρ_max from the SLO and c via Erlang-C, scale c up the moment ρ approaches ρ_max. The chapter backup-requests-and-bounded-queueing covers what to do when capacity is fixed and the cliff arrives anyway — hedging, load shedding, and bounded buffers.

The closing chapter of Part 8 — wall-real-systems-are-not-m-m-1 — synthesises every divergence from the ideal M/M/c model: heavy-tailed service times, bursty arrivals, retries, hot keys, finite buffers, non-work-conserving load balancers. Each divergence has a known correction; the discipline of capacity planning is knowing which corrections to apply for your service.

Three production habits to take from this chapter.

First: plot the Erlang-C curve for your actual c before the next capacity discussion. The c = 1 curve from chapter 56 is too pessimistic for a 16-pod deployment; the c = 64 curve is too optimistic for a 4-pod deployment. Compute the right curve, put it on the team Confluence, and use it to argue for utilisation targets that match your pool size.

Second: default to consolidating pools, not splitting them. If you have N small pools and you're considering combining them, the Erlang-C math is on your side — almost always. Split only when there is a specific isolation, fairness, or blast-radius requirement that justifies the latency cost; quantify the latency cost from the formula before deciding.

Third: measure Cv² and use Allen-Cunneen for capacity reviews. The pure M/M/c formula systematically over-promises headroom for any service with cache-hit/miss bimodality (most of them). The one-line correction (1 + Cv²) / 2 recovers reality to within 5%, and it is the difference between a comfortable launch and a midnight page.

References

• A. K. Erlang, "Solution of Some Problems in the Theory of Probabilities of Significance in Automatic Telephone Exchanges" (1917) — the original derivation of the formulas that bear his name. Translated from Danish in 1948 by R. B. Haviland.
• Mor Harchol-Balter, Performance Modeling and Design of Computer Systems (2013), Ch. 14 — the modern textbook treatment of M/M/c, including the resource-pooling theorem with cleaner notation than the original.
• Allen-Cunneen approximation (Allen, Probability, Statistics, and Queueing Theory, 1990) — the M/G/c extension via Cv² correction, covered in Ch. 5.
• Halfin & Whitt, "Heavy-Traffic Limits for Queues with Many Exponential Servers" (Operations Research, 1981) — the QED regime and the square-root staffing rule.
• Mitzenmacher, "The Power of Two Choices in Randomized Load Balancing" (1996) — why work-conserving load balancers approximate Erlang-C and why round-robin does not.
• Brendan Gregg, Systems Performance (2nd ed., 2020), §2.6 — practical queueing-theory chapter from the canonical systems-performance text.
• /wiki/m-m-1-and-why-utilization-80-hurts — the M/M/1 baseline that this chapter generalises.
• /wiki/littles-law-the-one-formula-everyone-should-know — Little's Law, the bookkeeping identity used in deriving the Erlang-C waiting time.