Separation of Storage and Compute (Snowflake's Bet)

Note: Company names, engineers, incidents, numbers, and scaling scenarios in this article are hypothetical — even when they resemble real ones. See the full disclaimer.

You have spent the last two chapters in Build 16 designing the warehouse — star schemas, dimensions, slowly changing dimensions, the grain decision. The implicit assumption was that the warehouse runs on a cluster, and the cluster has disks, and the disks have your data on them. That assumption was true for every serious analytical database from 1984 to roughly 2015, and it shaped everything: the size of your cluster, the cost of your bills, the latency of your queries, and — most painfully — the political negotiations between the BI team and the data engineering team about whose query got to run first.

Then a startup that nobody had heard of, founded by three Oracle veterans in 2012 and named Snowflake after the schema variant that nobody used, made a bet. The bet was that S3 had quietly eliminated the technical reason for tying compute to storage, and that whoever built the first warehouse around that observation would reset the economics of analytics for the next decade. They published the architecture in a SIGMOD 2016 paper called The Snowflake Elastic Data Warehouse, IPO'd at a $70 billion valuation in 2020, and forced every other vendor to copy the shape. This chapter is about the bet, the shape, and the new problems it created.

The classical model: storage and compute glued together

Walk into a data centre running Vertica or Greenplum or pre-2018 Redshift in 2014. You see a rack of machines, maybe 16 of them, each one a 2U server with 24 cores, 256 GB RAM, and 16 spinning disks of 4 TB each. Total cluster capacity: 384 cores, 4 TB RAM, 1 PB of disk. The data is sharded across the disks — fact_sales partitioned by date_id, with 2026 data on nodes 1-4, 2025 on nodes 5-8, and so on. When a query runs, the planner sends a fragment to every node; each node reads from its local disks (fast — sub-millisecond seeks on the SSDs that arrived later, single-digit milliseconds on spinners), runs its share of the work, and ships partial results to a coordinator.

This shape — shared-nothing, in Stonebraker's 1986 taxonomy — was the right answer for the hardware available. Network bandwidth between machines (1 Gbit/s, later 10 Gbit/s) was an order of magnitude slower than local-disk bandwidth, so you wanted compute next to its data. Hadoop took the same idea and made it the headline feature: move the computation to the data. MapReduce schedulers tried hard to assign each map task to the node that already held the input block.

The coupling produced three operational pathologies that everyone running a warehouse in 2014 knew by heart:

Forced bundled scaling. You bought the cluster sized for one of compute or storage, and overprovisioned the other. A telco running fraud detection had 50 TB of call records but needed every CPU it could get to run the ML jobs — so it bought a 32-node cluster with 1.5 PB of disk it would never use. A media company had 800 TB of historical viewer events but ran maybe two queries an hour against them — so it bought a 60-node cluster with 1500 cores it could not keep busy.

Workload contention. Every cluster had exactly one fleet of CPUs and exactly one set of disks, so every team's queries fought for the same resources. The classic 9 AM scenario: BI analyst opens a DashView dashboard and kicks off a 10-minute scan of fact_sales; data engineer's hourly ETL job runs at 9:05 and queues behind it; the dashboard renders, the ETL drains the queue, the next dashboard query collides with the ETL — and so on, all day. Workload management features (Vertica's resource pools, Redshift's WLM queues) tried to mitigate this with priorities and CPU shares, but the contention was structural: there was one set of disks and they were shared.

No pause. When the cluster was idle (nights, weekends, between batch windows), the disks still held the data, so the cluster still had to be running to make the data accessible. You paid for 168 hours a week of compute even if you used 40 of them. Cloud-hosted variants — early Redshift, EMR — let you tear the cluster down, but then you lost the data and had to reload from S3 on restart, which made teardown unattractive.

Why coupling made sense in 1986 and not in 2014: shared-nothing architecture was invented when network bandwidth was a tenth of disk bandwidth and S3 did not exist. By 2014, 10 Gbit/s networks were standard in datacentres (40 Gbit/s in cloud datacentres), S3 offered 11 nines of durability and effectively unbounded throughput, and the bandwidth gap had inverted in some cases — pulling Parquet from S3 to a worker over 10 Gbit/s was faster than reading the same data from a single spinning disk. The technical justification for storage-on-the-worker had quietly evaporated; only inertia kept the architecture in place.

The bet: put the data in S3, make compute disposable

Snowflake's founders — Benoit Dageville, Thierry Cruanes, and Marcin Zukowski (who had built Vectorwise earlier) — looked at this picture in 2012 and made a different design choice. The data, they decided, would live in S3, full stop. Not as a backup, not as a cold tier, but as the primary, sole, authoritative copy. Compute would be a fleet of ephemeral clusters — what Snowflake calls virtual warehouses — that you spin up to run a query and tear down when you are done. Each virtual warehouse pulls the data it needs from S3, caches what it can locally for the duration of its life, and bills by the second.

The architecture is a three-layer cake. The Snowflake architecture documentation names them like this:

• Storage layer. S3 (or GCS, or Azure Blob, depending on which cloud you run in). Tables are stored as immutable, columnar micro-partitions — 50-500 MB compressed Parquet-like files, automatically partitioned by Snowflake based on insertion order. Once written, a micro-partition is never modified; updates and deletes produce new micro-partitions and tombstone the old ones. Storage is metered separately at object-storage rates (cents per GB-month).
• Cloud services layer. A multi-tenant fleet of stateless services that does query parsing, planning, optimisation, transaction management, security, and metadata bookkeeping. The metadata itself — which micro-partitions belong to which table, which columns they have, which versions are current, which time-travel snapshots reference them — lives in a transactional store. Snowflake famously built this on top of FoundationDB once Pearle open-sourced it in 2018.
• Compute layer. Virtual warehouses. Each is a cluster of EC2 instances (a "T-shirt size" — XS, S, M, L, XL, 2XL, 3XL, 4XL — doubling cores at each step). You can have many warehouses running concurrently, each owned by a different team, each sized to its workload. A warehouse spins up in seconds, runs queries, caches micro-partitions on its local NVMe disks for the life of the cluster, and shuts down when idle (configurable auto-suspend, default 10 minutes).

What falls out of this layout is striking, and explains the marketing pitch that won Snowflake $70 billion of market cap.

Independent scaling. You add storage by writing more bytes to S3 — there is no node to provision. You add compute by spinning up a bigger warehouse — there is no data to re-shard. The two axes that were welded together for thirty years come apart. A team with 500 TB of historical data and one analyst running one query a day pays for 500 TB of S3 (12 per TB-month, so ~6000/month) and a few warehouse-hours, not for a 50-node cluster sized to hold 500 TB.

Workload isolation. This is the feature that ends the political war between BI and data engineering. The BI team owns Warehouse A (Medium, auto-suspend 5 min). The data science team owns Warehouse B (Large, manual control). The ETL pipeline owns Warehouse C (XLarge, runs 2-6 AM only). All three teams query the same fact_sales table because it lives in S3, and none of them contend with each other for CPU, RAM, or local-disk IO. A heavy ad-hoc scan in Warehouse B does not slow down a dashboard refresh in Warehouse A. They are physically different fleets of EC2 instances reading the same S3 bucket.

Why isolation works without coordination: the data in S3 is immutable. Once a micro-partition is written, no warehouse can modify it; updates produce new micro-partitions and update the metadata layer atomically. So all the warehouses can read the same files concurrently without locks, without consensus, without coordination — exactly the property S3 was designed to give. The transactional metadata layer (FoundationDB) handles the rare case where two warehouses commit conflicting writes; it serialises them through a single source of truth, but the read path never touches it.

Pause when idle. When a warehouse auto-suspends, you pay $0 for compute. The data is still in S3, accessible immediately when the next query comes in. A cluster that is busy 8 hours a day costs you 8 warehouse-hours, not 24. For typical BI workloads — bursty during work hours, near-zero overnight and weekends — this is a 3-5x reduction in compute spend.

Time travel and zero-copy clones. Because data lives in immutable S3 objects and the metadata layer tracks which objects were live at which point in time, Snowflake can offer two features that would have been impossible on a coupled cluster. Time travel lets you query a table as of any timestamp in the last N days (SELECT * FROM orders AT (TIMESTAMP => '2026-04-20 09:00')) — the metadata layer just hands the warehouse the micro-partition list as of that timestamp. Zero-copy clones let you create a full logical copy of a table or database (CREATE TABLE orders_dev CLONE orders) by copying only the metadata pointers, not the underlying micro-partitions. A clone of a 100 TB table takes a few seconds and adds zero bytes of storage; only when you modify the clone do new micro-partitions get written.

Many warehouses, one source of truth

The third diagram is the one that closes the deal for most enterprises. Once compute is decoupled from storage and a warehouse can spin up in seconds, you stop having one warehouse and start having one warehouse per workload. Each team picks a size that matches its query latency target; each team picks an auto-suspend window that matches its bursty pattern; each team's bill is its own, billed to its own cost centre, visible in its own dashboard.

Why this changes organisation design, not just architecture: the political question "whose query gets to run during business hours" was the dominant friction in 2014-era data teams, and it had no good technical answer because there was only one cluster. Decoupled compute reduces it to a billing question, which finance teams already know how to solve with cost centres and chargebacks. The platform team stops being a queue manager and starts being a librarian — guarantee the data is correct, fresh, and discoverable; let each team buy whatever compute it needs.

The new problems the architecture created

Decoupling solved the three coupled-architecture pathologies, but it created three new ones that the Cloud Data Warehouse Maturity Model and a decade of practitioner experience now name explicitly.

Object-store latency. S3 GET latency is 10-100 ms per request, versus sub-millisecond for a local SSD. A query that scans a thousand micro-partitions cold from S3 spends a measurable fraction of its time waiting on TCP. Snowflake fights this with two layers of caching: each warehouse node keeps a local NVMe SSD cache of recently-read micro-partitions (so a hot table is essentially memory-speed after the first query), and the cloud-services layer keeps a result cache (so identical repeated queries return in milliseconds). The first run of a new query against a cold dataset is slow; the second is fast; this is the bargain.

Transactional metadata. Hadoop's HDFS had real POSIX-ish semantics — rename was atomic, directory listing was consistent. S3 does not: it is eventually consistent for some operations, and concurrent writes need explicit coordination if you want serialisability. So the warehouse needs a separate transactional metadata service to record "what is the current version of table T", "which micro-partitions belong to it", "what was its state at timestamp X". Snowflake built one on FoundationDB; Databricks open-sourced Delta Lake (a transaction log on S3); Apache Iceberg and Apache Hudi are alternative open-source approaches. None of them are free; all of them are simpler than rebuilding HDFS.

Cost-model surprises. When compute is metered by the second and storage by the GB-month, your bill becomes a function of your queries, not your cluster. A new analyst who writes SELECT * FROM events ORDER BY timestamp on a 50 TB table can rack up a four-figure bill in an afternoon. BigQuery's per-byte-scanned pricing is even more direct: a poorly-written query that scans a partition it didn't need to scan costs real money. Modern Snowflake and BigQuery deployments need cost guardrails — query timeouts, credit quotas per warehouse, per-user spending alerts, and SQL linters that warn on SELECT * over large tables. The cost is real; it is just visible at a different granularity than before.

The pattern is now industry-standard

Snowflake pioneered the shape commercially, but they were not alone in the bet. By 2026 the architecture is the default for any new analytical system.

Google BigQuery had separated storage and compute from its 2010 launch, predating Snowflake. Storage is Colossus (Querion's successor to GFS), compute is Dremel (a serverless query engine that allocates "slots" on demand). The user does not provision clusters at all; they buy slots (units of CPU+RAM) on a flat-rate or per-query-byte basis, and Dremel dispatches their query across thousands of cores transparently. The two systems converged on the same architecture from opposite directions: Snowflake started with the cluster model and made it elastic; BigQuery started with the serverless model and added clusters.

Databricks built the Lakehouse on the same shape: data in S3 as Parquet files governed by a Delta Lake transaction log, compute as Spark clusters (or the newer C++ Photon engine), and a metadata catalog (Unity Catalog) tying it together. The pitch is that the same data layer can serve both ML/data-science workloads (which want raw files) and BI workloads (which want SQL warehouse semantics).

AWS retrofitted Redshift twice. The 2019 RA3 node type introduced "managed storage" — data lives in S3 and is cached on local NVMe in the cluster, decoupling storage scaling from compute. The 2021 Redshift Serverless launch went further: no nodes at all, just RPUs (Redshift Processing Units) you pay for by the second, with auto-scaling. Both were defensive responses to Snowflake taking large enterprise accounts.

Athena is the extreme case in the AWS lineup: pure compute (a managed Presto/Trino fleet) that runs SQL directly against your S3 bucket, with no warehouse to manage and no data to load. You write a CREATE EXTERNAL TABLE over your existing files and pay $5 per TB scanned. There is no "cluster" at all from the user's point of view; storage and compute are not just separated, the compute is invisible.

And DuckDB-WASM (which you met in chapter 127) is the architectural endpoint: the compute is your laptop, in the browser tab, and the storage is whatever Parquet file is on the open internet. The same separation pattern, miniaturised, with the caching layer being your browser cache and the metadata layer being a HTTP range-read into the file's footer.

The bet that storage and compute belong on different machines won completely. The interesting questions in 2026 are about the consequences — open table formats (Iceberg vs Delta vs Hudi), governance across warehouses, multi-cloud portability, and the rise of the data mesh organisational pattern that separation makes possible.

Worked example: sizing an Indian e-commerce BI workload

Common confusions

• "Snowflake stores your data in its own private storage system." It does not. Your tables live in S3 (or GCS, or Azure Blob) buckets that Snowflake operates on your behalf. The bytes are object-store objects with the same 11-nines durability and infinite-scalability properties you would get if you wrote them yourself. What is proprietary is the layout (Snowflake's micro-partition format) and the metadata service, not the underlying storage substrate. This is why Snowflake can offer cross-region replication and disaster recovery so cheaply — they are using the same S3 features you would.

• "Decoupled compute means queries are free when nothing is running." Storage is never free — your S3 bill keeps ticking at 23/TB-month even when every warehouse is suspended. For a 50 TB historical dataset that is1,150/month before any query runs. The right mental model is compute is elastic, storage is fixed; you can drive compute spend to zero overnight, but the data itself is a continuous monthly subscription. This trips up first-time Snowflake architects who assume "auto-suspend" means "auto-zero".

• "S3 is slower than local SSD, so cloud warehouses must be slower than on-prem MPP." The first time you scan a cold table, yes — S3 GETs are 10-100 ms versus sub-millisecond local SSD. But Snowflake and Databricks cache hot micro-partitions on each warehouse node's local NVMe SSD, so a frequently-queried table is essentially memory-speed after the first run. For repeated dashboard queries (the dominant BI workload), the cache hit rate is >95% and the architecture is competitive with local-disk MPP. The latency penalty shows up only on truly cold scans of huge fact tables — exactly the workload the warehouse is sized to handle in bulk anyway.

• "BigQuery and Snowflake have the same architecture." They share the separation principle but the implementations differ. Snowflake exposes virtual warehouses you provision and resize; BigQuery has slots that the scheduler allocates from a shared pool, and you do not see clusters at all. Snowflake bills by warehouse-second; BigQuery (in on-demand mode) bills per byte scanned. Snowflake caches data on warehouse-local NVMe; BigQuery's Dremel relies on Querion's network being so fast (Jupiter datacentre fabric, ~petabit-class bisection bandwidth) that local caching is less critical. Same shape, very different engineering choices.

• "You can just point any query engine at S3 and have a Snowflake." No — the transactional metadata layer is doing real work. If two ETL jobs commit conflicting writes to the same table, who wins? S3 alone cannot answer this; you need a transaction log (Snowflake's FoundationDB-backed catalog, or open-source Delta Lake / Iceberg / Hudi) sitting in front of S3. Athena pointed at raw Parquet does not give you ACID; Athena pointed at an Iceberg table does. The metadata service is the unsung hero of the architecture — without it, "S3 + compute" is a data lake, not a warehouse.

• "Snowflake's MERGE updates rows in place." It cannot — micro-partitions in S3 are immutable. A MERGE (or UPDATE, or DELETE) reads the affected micro-partitions, produces new micro-partitions with the merged data, and atomically swaps the metadata pointers to point at the new files. The old files become garbage that gets cleaned up after the time-travel window expires (default 1 day, configurable up to 90 days on Enterprise). This is why "small frequent updates" is an anti-pattern in Snowflake: each one rewrites entire micro-partitions. Batch your DML.

Going deeper

The micro-partition is not just a Parquet file. Snowflake's storage format is its own thing — proprietary, closed-source, and highly optimised for the specific access patterns of an analytical warehouse. Each micro-partition is roughly 50-500 MB compressed, columnar, with per-column min/max statistics in the file footer for partition pruning, and per-column dictionary and run-length encoding chosen automatically based on the column's actual data. The crucial property is automatic clustering: Snowflake reorders data within a micro-partition to maximise compression and minimise scan overhead, without the user specifying a sort key. For very large tables you can also set an explicit CLUSTER BY (analogous to a partition key) and Snowflake will background-reorganise the table to honour it — re-clustering is itself a metered cost, billed in credits, which is one of the gotchas. The Snowflake micro-partition documentation is the canonical read.

The metadata layer is the actual moat. S3 is a commodity. Compute is a commodity (Snowflake runs on the same EC2 instances anyone can buy). The thing that is genuinely hard, and that Snowflake spent five years building before they had a viable product, is the transactional metadata service — the thing that makes "list all current micro-partitions of table T as of timestamp X" answerable in milliseconds across thousands of concurrent queries. They built it first on a custom service, then migrated to FoundationDB after Pearle open-sourced it, because FoundationDB's strict-serialisable transactions over geographically replicated key-value storage are exactly what you need. The 2021 FoundationDB paper is one of the cleanest descriptions of how to build a transactional layer on top of fast key-value storage; it is the engine room of Snowflake.

Open table formats are unbundling Snowflake. If the metadata layer is the moat, what happens if someone open-sources it? That is exactly the bet behind Apache Iceberg (Streamora, 2017), Delta Lake (Databricks, 2019), and Apache Hudi (Glydex, 2017). Each one defines a transaction log and metadata format that sits on top of S3/Parquet, allowing multiple engines (Spark, Trino, Presto, Flink, DuckDB, even Snowflake itself via Iceberg tables) to read and write the same data with full ACID semantics. The 2024 Snowflake announcement that they would natively support Iceberg as a table format was an acknowledgement that the moat was leaking. Chapter 133 walks the Iceberg internals.

The BharatRail analogy. When BharatRail opened its tatkal booking window at 10 a.m. for 1 lakh hits in the first minute, the legacy architecture was a single Sybase / Oracle cluster sized for the worst minute of the year and idle for the other 1439 minutes of the day. The cloud-warehouse architecture lets a system like that run a small warehouse for normal hours and spin up a 4XL for the tatkal window — the data does not move, the schema does not change, only the compute is elastic. For Indian e-commerce specifically, this is what makes Mega Bargain Days (BharatBazaar, ~1 million orders/hour at peak) and IPL streaming analytics (SetuStream, 25 million concurrent viewers) economically tractable. You no longer have to provision for the peak; you provision for the steady state and burst on demand.

The cost-attribution war. Once each team gets its own warehouse, finance has a new problem: how do you attribute the storage bill, which is shared? Snowflake's answer is per-table storage metering with tag-based cost attribution — you tag tables with a cost-centre tag, and the billing report aggregates by tag. This sounds boring until you realise that without it, the data platform team becomes the dumping ground for storage costs that should have been billed to the team that produced the data. Modern Snowflake deployments at companies like PaisaBridge or DigiPaisa spend significant engineering time on cost-tagging infrastructure precisely because separation makes this question answerable; the coupled architecture made it impossible to even ask.

The Photon and Dremel parallels. Snowflake's compute engine, Databricks' Photon, and Google BigQuery's Dremel converge on the same execution-engine ideas that the vectorised execution chapter covers: columnar in-memory representation, SIMD instructions per column, late materialisation, predicate pushdown into the scan layer. None of these would be cost-effective on an HDD-backed coupled cluster — the disk is the bottleneck — but on top of S3 + NVMe cache + 10 Gbit/s networking, the bottleneck moves to the CPU, and vectorised execution is the right answer there. The hardware shift (from spinning disk to SSD to NVMe to network-attached object storage) is what makes the architectural shift economic; the two are tightly coupled.

Where this leads next

• /wiki/data-lakes-on-object-storage — chapter 132. Push the architecture further: drop the warehouse semantics, keep the raw files.
• /wiki/iceberg-and-the-open-table-format-war — chapter 133. The open-source unbundling of the metadata layer.
• /wiki/duckdb-and-the-embedded-analytics-renaissance — the laptop-as-warehouse extreme of the same pattern.
• /wiki/column-chunks-and-per-type-encoding — what is actually inside a micro-partition.
• /wiki/vectorised-execution-and-simd-in-warehouse-engines — why decoupled compute is fast despite the network in the middle.
• /wiki/star-and-snowflake-schemas — the modelling layer that runs on top.

References

1. Dageville, B., Cruanes, T., Zukowski, M., et al. (2016). The Snowflake Elastic Data Warehouse. SIGMOD 2016. [link]
2. Snowflake. Key Concepts and Architecture. docs.snowflake.com
3. Melnik, S., et al. (2010). Dremel: Interactive Analysis of Web-Scale Datasets. VLDB 2010. [link]
4. Snowflake. The Cloud Data Warehouse Maturity Model. [link]
5. Armbrust, M., et al. (2021). Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics. CIDR 2021. [link]
6. AWS. Amazon Redshift RA3 Nodes with Managed Storage. [link]