When NOT to Use a Distributed Database

Note: Company names, engineers, incidents, numbers, and scaling scenarios in this article are hypothetical — even when they resemble real ones. See the full disclaimer.

The cost-benefit shape, drawn honestly

Before any decision tree, draw the shape. Distributed databases pay off above some scale threshold; below it, they cost more than they earn. The shape is not "linear, slightly worse" — it is "fixed overhead, then crossover, then dominant".

Why distributed has a high fixed minimum: a CockroachDB or Spanner cluster cannot run on one node and call itself distributed — you need at least three nodes for a Raft majority of any single range, and in production you typically run nine or more for tolerance across availability zones. That is three to nine times the per-month VM bill before you serve a single request. Add Enterprise licensing (CockroachDB charges per vCPU once you cross feature gates; Spanner is straight Google Cloud billing per node-hour) and on-call rotation for a system whose failure modes are not Postgres failure modes, and the floor sits at roughly $2,500 per month for the smallest production-credible cluster.

Why single-node is so cheap at the low end: a single managed Postgres on AWS RDS or GCP Cloud SQL at the db.t4g.medium tier — 2 vCPU, 4 GB RAM — runs about $50/month and handles a thousand writes per second sustained, with backups, point-in-time recovery, and automated minor-version upgrades included. You pay for one box, you back up with one snapshot policy, you alert on one CloudWatch dashboard. There is no fan-out cost, no consensus latency, no cross-AZ replication tax. The reason the green curve is so flat at the start is that everything below thirty thousand writes per second fits inside the operational envelope of a single managed instance.

The decision tree — three questions

Before the architecture diagram, run the three questions. They are not subjective. Each has a number, and the number was chosen because it corresponds to a real point at which a single tuned Postgres node stops being enough.

Why these particular numbers: 1 TB is roughly the point at which a single Postgres instance with NVMe storage stops keeping the working set in OS page cache, so query latency stops being predictable. 10K writes per second is roughly where a tuned Postgres with synchronous WAL fsync on a single NVMe drive starts queueing — you can push higher with synchronous_commit=off or batched writes, but ten thousand is the conservative honest number. 100 ms is the threshold at which UPI, web checkout flows, and most user-perceived "instant" actions start to feel sluggish; if your Singapore user must wait 300 ms for a write to clear in a Mumbai primary, you have either a UX problem or you need a regional write footprint.

The hidden costs nobody puts in the slide deck

When a team proposes a distributed database, the slide deck shows scale numbers, partition tolerance, and "future-proofing." It rarely shows the costs. Here are the five categories of cost you actually pay.

Operational cost

A three-node CockroachDB cluster has roughly three times the operational surface area of a single Postgres instance, but the cost is not 3×; it is more like 5×, because the failure modes multiply rather than scale.

You now have to watch range distribution — are any ranges hotter than others, are splits happening cleanly, has any node fallen behind on Raft replication. You watch HLC clock skew, because a node whose clock drifts past the cluster's --max-offset will kill itself, taking its ranges' leadership offline temporarily. You watch range leases — leadership transfers happen continuously and slow leases cascade into latency spikes. You watch gossip, the cluster's metadata propagation layer, because gossip storms are real.

Backups are no longer pg_dump. They are cluster-aware snapshots that have to coordinate consistent timestamps across all nodes, usually using BACKUP SQL extensions or vendor tools. Restoring a backup to a development cluster is a multi-step ceremony; restoring a single table from yesterday's snapshot — trivial in Postgres with pg_restore -t — is a different operation in CockroachDB or Spanner.

Schema migrations become online distributed operations. Adding an index in Postgres is CREATE INDEX CONCURRENTLY and a few hours of background work. In CockroachDB it is the same syntax but the implementation has to coordinate across every range that holds rows of the table, build the index in parallel, and swap atomically. Most of the time it works. Sometimes a migration runs for days because one range is hot and back-pressure stalls the scan.

Latency cost

Every write in a distributed database is a quorum operation. In a three-node Raft group, the leader must persist locally and have at least one follower acknowledge before the write returns. That is one network round-trip minimum. Across availability zones, that is 1-2 ms in the same region; across regions, 50-200 ms.

Cross-shard transactions add another round-trip via two-phase commit (or its CockroachDB-style equivalent: a transaction record on a primary range, then intent resolution). A single insert that touches two ranges costs more than an insert that touches one. A transaction that crosses three ranges costs more again. The cost is not in the SQL you wrote; it is in the physical layout the system chose for your data, which you do not directly control.

Even reads pay a tax. A single-key read on a single shard in the local region is roughly as fast as a Postgres point lookup. A read that needs to consult the leader (because you want strong consistency, not a stale follower read) pays an extra hop to find the leader. A multi-key read that scans across ranges pays per-range overhead. The local-fast-path that single-node databases give you for free is something you have to engineer for in distributed systems.

Monetary cost

Three nodes minimum, often nine, for a production-credible cluster. Across availability zones, often across regions. CockroachDB Enterprise licensing per vCPU once you cross certain feature gates — backup tooling, change data capture, geo-partitioning. Spanner is direct Google Cloud billing per node-hour, and the per-node-hour price assumes you reserved capacity ahead of time; spot pricing does not exist.

Add at least one extra ops engineer salary to maintain the cluster. In Bengaluru that is 25-40 lakh per year fully loaded; in San Francisco it is $250K. Distributed systems debugging is its own skill set, and you cannot cross-train your existing Postgres DBA into it in a quarter.

Cognitive cost

Developers must learn that BEGIN ... COMMIT blocks may need to be retried. CockroachDB pushes timestamps when it detects conflicts and your transaction may receive a 40001 serialization_failure error that did not exist in Postgres. You wrap every transaction in retry logic. You do this even for transactions you "know" do not conflict, because you cannot know which range a key will live on after the next split.

You learn to think about hot ranges. If your primary key is monotonically increasing — a typical id BIGSERIAL or a timestamp-prefixed UUID — every new write goes to the same range, the range that owns the highest key. That range's leader becomes a hotspot, its disk fills, the cluster splits the range, the new range inherits the same problem. You either change your key design (use UUIDs, hash prefixes, or Spanner-style reverse-bit timestamps) or you accept the bottleneck.

You learn the PACELC trade-offs of your chosen system. You learn what "follower read" means and when it is safe. You learn the difference between strong reads and stale reads. None of these were in your Postgres mental model.

Debugging cost

A query that used to be one row in pg_stat_activity is now a distributed trace with spans across three nodes. A correctness anomaly that looks like a bug — "this read returned the old value, even though the write completed" — may be eventual consistency catching up, or it may be a real bug, and telling them apart takes hours and tools. Partial failures — one node up, one node slow, one node returning errors — produce behaviour that no single-node test suite ever surfaced.

The Postgres scaling ladder

Most apps never reach the top of this ladder. The ones that do tend to know it. Each step has a clear signal that it is time to move to the next.

Step 1: Single node, default settings. A db.t4g.medium on RDS, 2 vCPU, 4 GB RAM, ~$50/month. Handles roughly 1,000 writes per second sustained for typical OLTP workloads — small inserts, indexed lookups. Almost every B2B SaaS startup with a few thousand users lives here for the first year. PaisaBridge's earliest billing system, DakWala's early sync layer, every CRUD-shaped app you have ever shipped: this is the floor.

Step 2: Tuned single node. Same box, better choices: connection pooler (PgBouncer in transaction mode), proper composite indexes, EXPLAIN-driven query review, work_mem and shared_buffers tuned to your workload. Same ~$50-100/month. Now you handle 5,000 writes per second. The signal you are bottlenecked here is pg_stat_statements showing your top queries are doing sequential scans, or your connection count is hitting the pooler limit during peak.

Step 3: Read replicas. One primary, two or three async replicas, route reads to replicas via a DNS-based router or your application. Costs scale linearly per replica, but reads scale near-linearly with replica count. The signal: your primary's CPU is dominated by reads, not writes. Caveat: replica lag is real (typically 10-100 ms), and reading from a replica means accepting that occasional staleness — Werner Vogels' Eventually Consistent essay is the right mental model.

Step 4: Vertical scaling. A db.r6i.16xlarge on RDS — 64 vCPU, 512 GB RAM, NVMe SSD — runs about $4,000-5,000/month and handles 30,000 writes per second sustained for OLTP workloads. The signal: even with replicas, your primary is CPU-bound on writes, or your working set no longer fits in RAM and IO latency is your limit. Vertical scaling is unfashionable in a cloud-native era, but the truth is that one big machine is operationally simpler and often cheaper than a sharded cluster of small ones.

Step 5: Application sharding or Citus. Once you exceed what one box can do, you partition the data — by tenant ID, by user ID, by region — across multiple Postgres instances. Citus is the most popular extension that does this transparently; many teams roll their own. Sharded Postgres can serve millions of writes per second, with the cost being that cross-shard transactions are now your problem to design around (or to avoid entirely, the Helland approach).

Step 6: THEN distributed. If your sharded Postgres has become unwieldy — too many shards to manage, cross-shard transactions are killing you, you genuinely need geo-distributed strong consistency — now it is time to consider CockroachDB, Spanner, or TiDB. You will have specific, measured requirements at this point that justify the operational tax.

Most apps never get past Step 3.

When distributed actually wins

Distributed databases are not a mistake. They exist because the workloads at the top of the ladder are real. The cases where distributed genuinely earns its cost are well-defined.

Hyperscale e-commerce. Riverone at peak handles tens of millions of orders per hour during Prime Day; BharatBazaar during Mega Bargain Days does similar. Single-region Postgres simply does not run at that scale; even sharded Postgres becomes operationally untenable when you need the same data accessible with low latency from twenty regions.

Geo-distributed payments at scale. DigiPaisa and DhanWallet during a Diwali payments surge do tens of thousands of UPI transactions per second across multiple Indian regions, with strict consistency requirements (no double spends, no lost reversals). Spanner-class systems exist exactly for this — strong consistency across geographic distance with bounded latency.

Multi-region apps with strict consistency. A multinational bank operating across India, the EU, and the US, with regulators in each jurisdiction demanding that local data stay local but global accounting be consistent, is a textbook fit for distributed databases with geo-partitioning. CockroachDB's row-level region pinning and Spanner's region constraints exist for exactly this case.

Search and ad platforms at hyperscale. Querion's ad platform, Sociogram's news feed ranking, Bing's index — these are at scales where distribution is the only architecture that works. They are also at scales where they often build their own distributed databases rather than buying.

The common thread: specific, measured, large-scale problems with no simpler answer. Not "we might need to scale someday." Not "everyone uses Spanner now." Not "the CTO read a blog post."

The premature distribution timeline

The anti-pattern is so common it has a shape. Here is what it looks like across three years.

Why the rewrite usually wins on every axis: distributed databases are optimised for scale that the rewriting team does not have, and the operations they perform inefficiently — small transactions, small reads, simple joins — are exactly the operations Postgres has been optimised for since the 1990s. Removing the distributed layer removes the per-write quorum cost, removes the gateway-to-leader hop on every read, removes the need for transaction-restart logic in application code, and removes the per-shard fan-out cost on multi-row queries. p99 latency typically drops 30-50% for OLTP workloads, infra cost drops 5-10×, and on-call quietens dramatically because Postgres failure modes are familiar to every engineer in the building.

The Indian SaaS pattern

The Indian SaaS scene has produced a striking number of companies that scaled to hundreds of millions of users on managed Postgres or MySQL before adopting any kind of distribution. The pattern is consistent enough to be a useful prior.

PaisaBridge ran on managed Postgres for years, scaling vertically and adding read replicas, before introducing application-layer sharding for the highest-volume tables. Even today, much of the platform is sharded Postgres rather than a distributed SQL database — they reach for the distributed option only where the workload genuinely demands it (cross-region settlement reconciliation, certain audit pipelines).

NayaWorks built much of its CRM and helpdesk product on MySQL, scaling vertically and across read replicas for years. The shift to sharded MySQL came late, and only for tenants whose data exceeded what a single shard could hold.

DakWala ran its sync layer on Postgres for a long time, adding caching and read replicas before any sharding. The team's public talks have repeatedly returned to the theme: simpler is better, distributed is a last resort.

ParakhTrade famously runs much of its trading infrastructure on simple, single-region setups with aggressive use of Postgres, kdb+ for tick data, and minimal distributed-system complexity. The trading day has a known peak and known shape, and a sized-correctly single-node deployment outperforms a misconfigured distributed cluster handily.

ZeruApp has built a multi-billion-dollar SaaS business with a deliberately conservative database strategy — heavy use of MySQL, careful sharding where needed, zero rush to distributed SQL. They are profitable, they ship reliably, and they have not paid Spanner's bill once.

The lesson is not "Indian SaaS is afraid of distributed databases." The lesson is "Indian SaaS is good at engineering economics." Distribution is for the workloads that need it. Most workloads do not.

Common confusions

• "Distributed databases are always more reliable than single-node ones." No. A managed Postgres on RDS Multi-AZ has a higher measured uptime in practice for small-to-mid workloads than a self-operated three-node CockroachDB cluster, because the failure modes of the former are familiar and well-tooled (failover is one knob, the cloud provider owns it) and the failure modes of the latter are exotic — gossip storms, range-leader oscillation, HLC drift, slow followers cascading into request timeouts. The cluster has more redundancy on paper; it also has more pieces that can break and more skill required to bring it back. Reliability is a function of operator competence, not just architecture.

• "Sharded Postgres is the same as a distributed database." Not really. Sharded Postgres (Citus, Vitess for MySQL, or hand-rolled application sharding) gives you horizontal write scale by partitioning data across instances, but cross-shard transactions are your problem to design around — the system does not give you ACID across shards for free. A distributed SQL database like CockroachDB or Spanner does give you cross-shard ACID, at the cost of consensus on every write. The two solve different parts of the scaling problem; calling them "distributed" lumps them together in a way that obscures what each actually buys you.

• "Once we hit ten thousand writes per second we must go distributed." Ten thousand is the conservative single-NVMe-disk Postgres ceiling with synchronous_commit=on. Turn off synchronous commit (accepting a small window of last-second loss after crash, which many workloads tolerate), batch writes, or move to a db.r6i.16xlarge with provisioned IOPS, and a single Postgres node will sustain thirty to forty thousand writes per second cleanly. The distributed conversation starts at scales much further out than most teams realise.

• "Replication = distribution." A Postgres primary with three streaming-replica followers is replicated but not distributed in the sense this chapter means. The primary still owns all writes; the replicas serve reads and act as failover candidates. There is no consensus, no quorum, no HLC. The operational model is a single-node model with redundant copies — far simpler than a distributed-SQL cluster. Mixing up "we have a follower" with "we are running a distributed database" leads to an alarming amount of wrong architecture.

• "Distributed databases give you free multi-region latency." They give you the ability to write in multiple regions; they do not give you free low latency. A user in Singapore writing to a CockroachDB cluster whose leaseholder for that range is in Mumbai still pays the Mumbai-Singapore round-trip on every write, because the leaseholder must be reached. To actually get fast multi-region writes you need geo-partitioning, region-pinned tables, or follower writes — and these have their own consistency caveats. Distribution is necessary but not sufficient for low-latency multi-region.

• "NoSQL means you skip all of this and just scale." No. The original NoSQL pitch — schemaless, horizontally scalable from day one — collided with the same trade-offs by year three. MongoDB now has multi-document transactions and tunable consistency, DynamoDB now has global tables and conditional writes, Cassandra has lightweight transactions. The distributed-systems trade-offs do not disappear because you chose a different data model; they just show up in different vocabulary. The scaling questions in this chapter apply to NoSQL stacks just as much as to SQL ones.

Going deeper

The advice in this chapter — start single-node, scale up before you scale out — is widely agreed upon by the engineers who actually operate large databases, and widely disagreed with by people who have not yet operated one. The disagreement has a few specific shapes worth knowing about, and understanding them is what separates the engineer who has internalised the lesson from the one who has only memorised the slogan.

The "we'll need it eventually" argument, examined

The most common objection to single-node-first is "but we'll outgrow it eventually, so we should start with the architecture we'll end up with." This sounds reasonable and is wrong. The cost of migrating from managed Postgres to a sharded or distributed system, at the point you actually need it, is roughly one to three quarters of focused engineering work. The cost of operating a distributed system you do not need, for the years before you would have needed it, is strictly larger in almost every actual case — five to ten times the infra spend, ongoing operational tax, a more complex data model that constrains feature velocity. Migrations are bounded events; operational tax is unbounded. Build for what you have, plan for what you might need, do not pay today for capability you will earn the right to need three years from now. Joel Spolsky's Things You Should Never Do essay applies in the inverse here — do not pre-rewrite for a scale you have not yet hit.

The hidden assumption: "scale" means "writes"

The decision tree in this chapter privileges write throughput. Reads are easier — a primary with replicas absorbs almost arbitrary read load. The 10K-writes-per-second number is the one that genuinely separates single-node from sharded territory, because writes go through the WAL and the WAL is on a single disk on a single primary. Most teams that think they have a "scale problem" actually have a read problem (slow dashboards, slow analytics queries) that is solved by a read replica plus an index review, not by reaching for distribution. Before invoking the decision tree, classify the bottleneck honestly. If it is reads, you are looking at the wrong tool.

The role of analytical workloads — OLAP is a separate question

This chapter is about OLTP — small transactions, indexed lookups, the bread-and-butter shape of operational data. OLAP — analytical queries scanning millions of rows — is a different problem with a different answer. You do not solve OLAP by sharding Postgres; you solve it by routing analytics to a column-store like ClickHouse, DuckDB, BigQuery, or Snowflake, with a CDC pipeline from Postgres. The "we need a distributed database for our reporting" conversation is almost always a conversation that should end with "you need a column-store, not a distributed OLTP system." Conflating the two is one of the most expensive architectural confusions in the field.

The DigiPaisa / PaisaBridge / ParakhTrade pattern — what they actually run

It is worth being concrete about what large Indian platforms actually run, because the distance between the marketing material ("we use distributed databases at scale") and the operational reality ("we use sharded Postgres or MySQL with careful read-write splits, plus a smaller distributed component for the few workloads that need it") is large. PaisaBridge's public talks describe heavy use of MySQL with application-layer sharding for transaction tables, with Aerospike for low-latency lookups; DigiPaisa has talked about a Kafka-and-Postgres-heavy pipeline with newer distributed components added narrowly; ParakhTrade has been famously open about running on Postgres, kdb+ for tick data, and minimal distributed-system complexity (Kailash Nadh's blog and conference talks are the canonical references). The pattern is narrow adoption of distribution where the workload demands it, not blanket distribution everywhere.

When you are wrong about needing distribution — the Glydex MySQL story

Glydex's much-discussed migration from Postgres to MySQL to Schemaless — and later, partial walk-back to Postgres for some workloads — is often cited as evidence that "real scale needs custom databases." Read the post-mortems carefully and a different lesson emerges: Glydex chose a complex sharded architecture (Schemaless on top of MySQL) at a point where the workload genuinely justified it, but the cost of operating it has been enormous, and several internal teams have since moved back to managed Postgres for sub-services that did not need Schemaless. Even at Glydex scale, the answer is not "distributed everywhere" — it is "distributed exactly where the workload demands it, simpler everywhere else." Marc Brooker's post makes the same case with AWS-internal data points.

Spanner's own caveats from the design paper

Even Querion's Spanner paper — one of the foundational documents of geo-distributed strong consistency — explicitly notes that Spanner is overkill for most workloads and that internal Querion teams use Bigtable, Megastore, or sharded MySQL when the workload does not need TrueTime-level external consistency. The team that built Spanner is more cautious about deploying it than most outsiders are. Read section 7 ("Related Work") and section 8 ("Future Work") with this in mind — the trade-offs are explicit, and the paper does not claim Spanner is universally better.

The cultural pressure problem

The hardest part of following this chapter's advice is not technical. It is cultural. There is real prestige attached to "we run on Spanner" or "we use CockroachDB" — at conferences, on engineering blogs, in interview rooms. There is no equivalent prestige to "we run on a single tuned Postgres." The asymmetry creates a real bias in architectural decisions, especially in early-stage startups whose engineers are also building their resumes. Recognising this bias is half the battle — when you propose distributed-by-default, ask honestly whether the proposal is justified by measured requirements or by aspirational ones. The honest answer is often illuminating.

The right time to revisit

Re-run the decision tree once a quarter. If your write throughput, data size, or geo-latency requirements have crossed any of the gates, now you start the planning conversation. Migrations to sharded Postgres take a quarter; migrations to distributed SQL take a quarter to two quarters. You have plenty of runway from "we measured the gate-crossing" to "we are operating the new system in production." Do not start that work earlier; the option to start it later, with measured data, is genuinely free.

A decision framework, in one sentence

Start simple. Move to distributed only when you have a measured, specific scaling problem that simpler approaches genuinely cannot solve.

That is the entire framework. Everything else in this chapter is supporting evidence. The framework is hard to follow not because it is intellectually difficult but because it cuts against the gravitational pull of resume-driven development, conference-talk-driven architecture, and the cultural prestige of "scale-ready" choices. Resist all three. Ship the simple thing. Measure. When the simple thing breaks at a place you can point to with a number, replace just that piece.

The fourteen builds in this series — through Raft and Paxos, MVCC and snapshot isolation, Spanner and TrueTime, CockroachDB and HLC, Calvin's deterministic ordering, the PACELC trade-offs — were not designed to push you toward distributed databases. They were designed to give you the literacy to know when to choose one, and when not to. The closer for Build 14 is the chapter that closes the loop: most of the time, the answer is "not yet, maybe never, definitely not on day one."

If you remember one number from this chapter, remember the cost ratio: a managed Postgres at 200/month versus a distributed cluster at2,500-$5,000/month plus engineering tax. That is 10-25× the cost. You owe that ratio a real reason before you cross it.

References

1. Pat Helland — Life Beyond Distributed Transactions: An Apostate's Opinion (CIDR 2007). The classic case for designing applications that avoid distributed transactions, from one of the architects of the field.
2. Marc Brooker — You Probably Don't Need a Distributed Database. The blog post that names the pattern this chapter elaborates.
3. Werner Vogels — Eventually Consistent. The Riverone CTO's foundational essay on consistency models and the costs of strong consistency.
4. Citus Data — Distributed Postgres goes full open source. The most popular Postgres-sharding extension and a worked example of step 5 on the scaling ladder.
5. PgBouncer — Project documentation. Connection pooling, the unsexy single most impactful Postgres scaling intervention.
6. Eric Brewer — CAP Twelve Years Later. Brewer's own retraction of the "two of three" framing, useful context for why distributed-by-default is the wrong default.