Schema registries and the evolution problem

It is 2:14 a.m. at PhonePe's Bengaluru data centre. A producer team rolls out a new version of the UPI-events service that adds an optional device_fingerprint field to the payload, ships through canary, and starts emitting v2 messages onto the upi.events Kafka topic. Twelve seconds later, the fraud-scoring consumer — a Flink job that has been quietly running for fourteen months on schema v1 — begins throwing AvroTypeException: Found upi.Event, expecting upi.Event on every message it pulls. The bytes look fine; the topic is healthy; nothing crashed. But the schema embedded in the message is one the consumer was never told about, the consumer cannot decode it, and the lag on the partition starts climbing at 3,000 messages per second. By the time the on-call data engineer logs in, the consumer group is 2 million messages behind and the SOC team is asking why fraud scoring has gone dark during peak fraud hour. The producer did nothing wrong. The consumer did nothing wrong. The piece nobody wired up was the small, boring service that would have told the consumer what device_fingerprint means before the first v2 message landed.

Why bytes alone are not enough

A Kafka message is a sequence of bytes; an Avro or Protobuf payload is a sequence of bytes that means something only when interpreted against a schema. The schema is what tells you that bytes 5–12 are a 64-bit integer named amount_paise, not a 32-bit float and a 32-bit string ID. Without the schema, you cannot decode the payload — and the schema lives nowhere inside the bytes themselves.

There are three places the schema could live, and each has a problem. Inline in every message (what JSON does) wastes 80–95% of the wire bytes on field names, which is fatal at PhonePe's 100,000 events/sec — the bandwidth alone would cost lakhs per month. Compiled into the consumer (the protobuf approach with .proto files checked into the consumer repo) means every schema change requires recompiling and redeploying every consumer, which is impossible at 200-microservice scale. In a separate registry, referenced by ID is the third option: the message carries a 5-byte prefix (1 magic byte + 4-byte schema ID), the consumer looks up the schema by ID the first time it sees that ID, caches it forever, and decodes. Why the registry-by-ID design wins: the wire overhead is constant (5 bytes regardless of schema size), schema changes are decoupled from consumer deploys (the consumer fetches the new schema lazily), and the registry becomes the one place where compatibility rules can be enforced before any producer ships a breaking change.

The evolution problem is the operational reality that producers and consumers ship on independent cadences. The producer team rolls out v2; the consumer is still on v1; the consumer needs to keep working anyway, because you cannot freeze every consumer team every time a producer adds a column. A schema registry doesn't solve evolution by itself — it provides the substrate (versioned schemas, compatibility checks, schema-by-ID resolution) on top of which a coherent evolution policy becomes possible. Without the registry, every team negotiates evolution in Slack; with the registry, evolution is a POST /subjects/upi.events-value/versions request that either succeeds or fails CI deterministically.

What the registry stores and how messages reference it

A schema registry is, mechanically, a small CRUD service over schemas. It exposes a REST API; every Kafka client library (Java, Python, Go, .NET) speaks to it; the data model has three concepts.

A subject is a logical grouping of schemas — almost always one subject per Kafka topic per key/value side. The Confluent default subject naming is <topic>-key and <topic>-value, so the upi.events topic has two subjects, upi.events-key and upi.events-value, holding the key schema and value schema respectively. A schema is the actual Avro/Protobuf/JSON-Schema document. A version is the (subject, schema) pairing — version 1 is the first schema registered under a subject, version 2 is the next compatible schema, and so on. Every version also has a globally unique integer ID across the entire registry (so version 5 of upi.events-value might have global ID 4827).

When a producer sends a message, it Avro-encodes the payload, then prepends a 5-byte header: 1 magic byte (always 0x00) and 4 bytes of big-endian schema ID. The wire payload looks like [magic][schema_id_be4][avro_payload...]. When a consumer pulls the message, it reads the first byte (must be 0x00, otherwise the message wasn't produced by a registry-aware client), reads the next 4 bytes as the schema ID, looks up the schema in its local cache (or fetches from the registry on cache miss), and decodes the rest of the payload against that schema. The cache miss happens once per schema ID per consumer process; after that every decode is local.

The compatibility check is the most operationally important piece. When a producer registers a new schema under a subject, the registry compares it against the latest registered version (or all registered versions, depending on the policy) and checks one of four compatibility modes:

• BACKWARD — the new schema can decode data written with the old schema. This is what you want when consumers upgrade after producers — a v2 consumer must be able to read v1 messages still on the topic.
• FORWARD — the old schema can decode data written with the new schema. This is what you want when producers upgrade after consumers — a v1 consumer must be able to read v2 messages.
• FULL — both BACKWARD and FORWARD. Strictest; safest; most restrictive on what evolutions are allowed.
• NONE — no compatibility check; the producer is on their own.

Most production Kafka deployments at Indian fintechs run BACKWARD as the default, because the typical deploy cadence is "producers ship first, consumers catch up later". The compatibility mode is configured per-subject, so the high-stakes payments.captured topic can run FULL while a logging topic runs NONE.

Building a tiny registry

The mechanics are simple enough that a 60-line Python service is enough to understand the contract. The real Confluent Schema Registry adds clustering, ACLs, schema references, JSON-schema support, and a 5,000-page list of operational features — but the core is what fits below.

# tiny_registry.py — a 60-line schema registry that demonstrates the core contract.
# Stores schemas in memory; in production replace dict with Postgres / Kafka log.
from flask import Flask, request, jsonify
from fastavro.schema import parse_schema, fullname
import json, hashlib

app = Flask(__name__)

# subject_name -> [ {id, version, schema} ... ]
subjects = {}
# global_id -> schema_dict (every registered schema has a global ID)
schemas_by_id = {}
next_id = 1

def is_backward_compatible(old, new):
    """new schema must be able to decode payloads written with old schema.
    Concretely: every required field in old must still exist in new, and
    every required field in new must have existed in old or have a default."""
    old_fields = {f["name"]: f for f in old.get("fields", [])}
    new_fields = {f["name"]: f for f in new.get("fields", [])}
    for name, of in old_fields.items():
        if name not in new_fields:
            return False, f"required field '{name}' removed"
    for name, nf in new_fields.items():
        if name in old_fields: continue
        if "default" not in nf:
            return False, f"new field '{name}' added without default"
    return True, "ok"

@app.post("/subjects/<subject>/versions")
def register(subject):
    global next_id
    new_schema = json.loads(request.json["schema"])
    parse_schema(new_schema)  # validates it parses as Avro
    versions = subjects.setdefault(subject, [])
    if versions:
        latest = versions[-1]["schema"]
        ok, reason = is_backward_compatible(latest, new_schema)
        if not ok:
            return jsonify({"error": "incompatible", "reason": reason}), 409
    schema_id = next_id; next_id += 1
    schemas_by_id[schema_id] = new_schema
    versions.append({"id": schema_id, "version": len(versions)+1,
                     "schema": new_schema})
    return jsonify({"id": schema_id})

@app.get("/schemas/ids/<int:schema_id>")
def get_by_id(schema_id):
    s = schemas_by_id.get(schema_id)
    if not s: return jsonify({"error": "not found"}), 404
    return jsonify({"schema": json.dumps(s)})

# Sample run — register two compatible schemas, then a breaking one:
$ curl -sX POST localhost:5000/subjects/upi.events-value/versions \
    -H 'content-type: application/json' \
    -d '{"schema":"{\"type\":\"record\",\"name\":\"Event\",\"fields\":[{\"name\":\"id\",\"type\":\"string\"},{\"name\":\"amount_paise\",\"type\":\"long\"}]}"}'
{"id": 1}

$ curl -sX POST localhost:5000/subjects/upi.events-value/versions \
    -H 'content-type: application/json' \
    -d '{"schema":"{\"type\":\"record\",\"name\":\"Event\",\"fields\":[{\"name\":\"id\",\"type\":\"string\"},{\"name\":\"amount_paise\",\"type\":\"long\"},{\"name\":\"device_fp\",\"type\":\"string\",\"default\":\"\"}]}"}'
{"id": 2}

$ curl -sX POST localhost:5000/subjects/upi.events-value/versions \
    -H 'content-type: application/json' \
    -d '{"schema":"{\"type\":\"record\",\"name\":\"Event\",\"fields\":[{\"name\":\"id\",\"type\":\"string\"}]}"}'
{"error": "incompatible", "reason": "required field 'amount_paise' removed"}

The walkthrough is short because the design is small. is_backward_compatible() is the heart of the registry — it implements the BACKWARD rule by checking two conditions: every field in the old schema is still present (else old payloads can't be decoded by the new schema if a field was renamed-via-removal), and every newly-added field has a default value (else the new schema can't decode old payloads which lack that field). Why the default-value rule matters: when a v2 consumer reads a v1 payload that's missing the new field, the Avro decoder substitutes the default. Without a default, the decoder has no value to insert and throws — turning what should be a forwards-compatible read into a runtime crash for every old message still on the topic.

/subjects/<subject>/versions is the registration endpoint. The producer's deploy pipeline POSTs the new schema before rolling out the producer code. If the registry returns 409, the deploy is gated — the producer team has to fix the schema (add a default, keep the renamed field as a deprecated alias) before they can ship. Why this gating matters in CI: the registration request runs in the producer's deploy pipeline, before any producer instance is updated. A breaking schema is rejected at deploy time, in a CI step the producer team owns and reads. Pushing the gate any later — into Kafka itself, into the consumer — makes the failure mode worse: the producer is already serving traffic when the breakage is discovered.

/schemas/ids/<id> is the consumer's lookup endpoint. The consumer's Avro deserializer uses this to resolve schema IDs it hasn't seen before. The lookup is read-heavy and cacheable forever per ID (schemas are immutable once registered), so the consumer caches by ID in process memory after the first miss. The actual Confluent registry adds cache-control headers and supports up-to-30-second freshness on subject-level metadata, but the per-ID schema mapping is genuinely immutable — the only way to change what schema ID 4827 points to is to take down the registry and corrupt its storage.

The real Confluent Schema Registry adds three features the toy version skips. Persistence via a compacted Kafka topic — schemas are written to a _schemas topic and replayed on registry restart, giving durability without an external database. Subject-level configuration — the compatibility mode (BACKWARD, FORWARD, FULL, NONE) is set per subject via PUT /config/<subject>. Schema references — one schema can $ref another, useful when many topics share a common header type. None of these change the basic contract; they make the registry production-operable.

Compatibility modes in production

Picking a compatibility mode is the most consequential schema-registry decision a team makes. The mode determines the deploy order producers and consumers can use — get it wrong and a routine deploy becomes an outage.

The PhonePe outage from the lead — adding device_fingerprint to the topic — would have been gated under BACKWARD if the new field had a default. The producer team's actual mistake was registering the schema without a default, which makes adding a field FORWARD-compatible (new producers, old consumers) but not BACKWARD-compatible (the v2 consumer cannot decode any old v1 message). When the on-call read the registry log the next morning, they saw the v2 schema had been registered with a null default explicitly disabled — a subtle Avro union ordering issue where ["string", "null"] (a string with a fallback to null) is BACKWARD-compatible but ["null", "string"] (a null-or-string with no default) is not. The fix took ten minutes; the diagnosis took ninety. Why the union ordering matters: Avro defaults must match the type of the first branch of a union, so ["string", "null"] defaults to a string (and adding default: "" works) while ["null", "string"] defaults to null (and default: null works only if the field is genuinely nullable in the receiving consumer). Teams that don't internalise this rule re-encounter the same outage every six months.

Choosing a mode per subject is the next-level discipline. A high-stakes topic consumed by external partners (e.g. PhonePe's settlement-events topic consumed by HDFC Bank's reconciliation system) should run FULL — the partner cannot coordinate deploys with PhonePe, so any change must be safe in either direction. An internal-only topic where the consumer team and producer team ship together can run BACKWARD or FORWARD depending on whose deploy lands first. A development topic where the team is iterating on schema design can run NONE temporarily, with the understanding that it must flip to BACKWARD before any data lands in production. Mixing modes per subject — instead of running one global mode — is what lets a single registry serve dev, internal-prod, and external-prod with one piece of infrastructure.

Which evolution operations are actually safe is worth memorising as a working table:

Operation	BACKWARD	FORWARD	FULL
Add optional field with default	safe	safe	safe
Add required field (no default)	unsafe	safe	unsafe
Remove optional field	unsafe	safe	unsafe
Remove required field	unsafe	unsafe	unsafe
Rename field	unsafe (registers as remove + add)	unsafe	unsafe
Widen int → long	safe	safe	safe
Narrow long → int	unsafe	unsafe	unsafe
Add enum value	unsafe under strict resolvers	safe	unsafe
Reorder fields	safe (Avro resolves by name, not index)	safe	safe

The only "always safe" operations are adding-with-default and widening numeric types. Everything else requires choosing the right mode and the right deploy order. Teams memorising this table on first adoption avoid most evolution incidents in their first year — the remainder are semantic-but-not-structural shifts that contracts (chapter 31), not the registry, are designed to catch.

The compatibility mode also informs the deprecation window the team commits to. Under BACKWARD, removing a field requires a two-phase migration: first mark the field as deprecated and add a default to it (BACKWARD-compatible), wait long enough for all producers to stop writing to it (typically 30–90 days at fintech scale because retention windows on the topic are 7 days but downstream warehouses need history), then finally remove the field (also BACKWARD-compatible since by then no payload has the field). Skipping the wait means consumers reading historical messages from a replay see the field missing, which they may handle correctly or incorrectly depending on how they coded the deserializer.

Where registries break and how teams operate them

A schema registry is a small piece of infrastructure with an outsized blast radius. When the registry is down, every Avro consumer that hits a cache miss on a new schema ID can't decode messages — and since cache misses happen on every consumer-process restart, a registry outage during a deploy can take down every consumer that scaled up during the outage. The operational pattern that production teams converge on has four pieces.

Run the registry as a multi-AZ cluster with leader election — Confluent's reference deployment runs three nodes across availability zones, with leader election via the underlying Kafka cluster. Reads scale across replicas; writes go to the leader. PhonePe's 2025 incident review traced a cascade outage to a single-node registry that fell over during a region-wide cache flush; the fix was a three-node cluster.

Prime caches on consumer startup — instead of letting the consumer hit the registry on the first message containing a new schema ID, consumer processes can call GET /subjects/<subject>/versions and pre-fetch every schema ever registered for the topics they consume. The first message decode is then a local cache hit. The cost is one bulk fetch on startup; the benefit is that the registry becoming unavailable mid-run doesn't kill the consumer, only its ability to decode future new schemas.

Pin the registry as a hard dependency in the producer's deploy gate, not the consumer's runtime — the producer cannot ship a new schema without the registry being available; the consumer can keep decoding old schemas without the registry being available. This asymmetry is correct: a deploy can wait for the registry to come back, but a running production consumer cannot.

Monitor registration latency and the size of the schema-ID space — registration latency above 100ms means the registry is leader-election-thrashing or the underlying Kafka log is slow. The schema-ID counter growing unboundedly means a producer is registering a new schema on every message (almost always a bug — usually a non-deterministic schema generator that adds a random ID to a doc field). At Razorpay, an alert on "schema registrations per hour > 50 on any subject" caught a misconfigured Avro generator before it filled the registry's compacted topic.

The other failure mode is subjects that nobody owns. A producer team registered a schema two years ago, the team got reorganised, the topic is still live, the schema can't be modified because nobody knows what consumes it. The fix that scales is to require an ownership_team tag on every subject, gate registration on the tag being present and matching a known team in the catalog, and run a quarterly audit that flips orphaned subjects to NONE compatibility (which won't break running pipelines but will surface the lack-of-ownership the next time anyone tries to evolve them).

A subtler failure mode is schema-ID exhaustion through accidental churn. The registry's global ID counter is typically a 32-bit integer; at 2.1 billion possible IDs, exhaustion sounds impossible — until a misbehaving producer in a dev environment registers a fresh schema for every event because the schema includes a timestamp in a docstring. The Cred team observed this once on a load-testing topic that consumed 4 million IDs in a weekend before alerting fired. The discipline is to compute schema fingerprints (Avro's parsing_canonical_form SHA-256) and reject re-registrations of equivalent schemas at the API layer — a check Confluent added in 5.5 and that every Apicurio deployment ships with by default. The first defence against ID exhaustion is making the registry idempotent: registering the same logical schema twice returns the existing ID rather than allocating a new one.

Common confusions

• "A schema registry is the same as a data contract." Related, not equivalent. A schema registry stores and gates the structural schema (fields, types, evolution rules). A data contract (chapter 31) wraps the schema with semantics, freshness guarantees, governance class, and ownership — and lives in version control alongside the producer's code. The registry is the runtime substrate; the contract is the producer's promise.
• "Schema registry is just for Kafka." It started Kafka-specific (Confluent's 2014 design), but the same pattern now ships with Pulsar, Redpanda, and AWS Glue. Iceberg and Delta have their own equivalent (the table-level schema metadata). The pattern — central authority on schemas, ID-based lookup, compatibility gating — generalises beyond streaming.
• "BACKWARD compatible means the new schema can be safely deployed." Only if you upgrade producers before consumers. BACKWARD says new-schema-can-read-old-data, which protects consumers reading historical messages after they upgrade. If you upgrade consumers first under BACKWARD, the consumer reads new producer messages it doesn't know about — that's a FORWARD scenario and BACKWARD does not protect it.
• "You can change the schema for a given ID." No. Schema IDs are immutable. To evolve, register a new schema under the same subject and get a new ID. This is what makes consumer caches safe to keep forever.
• "Once we have a registry, we don't need data contracts." The registry tells you the schema is BACKWARD-compatible; the contract tells you the field amount_paise is in INR, refunds emit a separate row, and the semantics haven't changed. A backwards-compatible structural change can still break consumers semantically — e.g. the field stays the same but the meaning shifts from gross to net amount.
• "NONE compatibility is fine for internal topics." Almost never. NONE means the next producer deploy can register {"id": "string"} over today's {"id": "long"} and every consumer will start failing to decode. NONE is a development tool, not a production operating mode.

Going deeper

How Avro's resolution algorithm decides field-by-field compatibility

When an Avro consumer reads a message, it has two schemas: the writer's schema (what produced the message, fetched from the registry by ID) and the reader's schema (what the consumer was compiled against). Resolution walks both schemas field by field. For every field in the reader's schema, Avro looks for a same-named field in the writer's schema; if absent, it uses the reader's default; if the types differ, it applies a type-promotion rule (int → long is allowed, long → int is not). For every field in the writer's schema not present in the reader's schema, Avro skips the bytes (it knows the size from the writer schema). This is what makes BACKWARD evolution work: the v2 reader looks at v1 writer bytes, finds device_fp missing, substitutes the default "", and continues. The whole protocol is in the Avro Schema Resolution section of the spec, four paragraphs long, and worth reading once. Confluent's KafkaAvroDeserializer is essentially an implementation of those four paragraphs plus a registry client.

Protobuf, JSON-Schema, and the registry-pluralism problem

Avro was the original Confluent registry format, but production teams in 2026 increasingly want to register Protobuf and JSON-Schema definitions in the same registry. The Confluent registry added Protobuf support in 2018 and JSON-Schema in 2020; Apicurio (Red Hat's open-source registry) supports all three from the start. The compatibility rules differ — Protobuf's optional is naturally backwards-compatible because every field is optional in proto3, while Avro requires explicit defaults. Mixing formats in one organisation is fine if every team picks one and sticks with it; the failure mode is teams switching mid-stream, which leaves the consumer unable to decode old messages because the new format is structurally different. Razorpay standardised on Avro for streaming and Protobuf for gRPC service-to-service contracts, with the registry holding both — a clean split that avoids the format-drift problem.

How the registry composes with stream processing engines

Flink, Kafka Streams, and ksqlDB all integrate with the schema registry directly — when a Flink job reads from upi.events, it queries the registry to fetch the writer schema and configures the Avro deserializer to decode against the SQL types Flink expects. The integration point is the SchemaRegistryClient interface that every JVM-based stream engine consumes. The interesting twist: Flink's SQL planner uses the registry's schema to generate the SQL row type, so adding an optional field to the topic automatically becomes a new column in the Flink source table — without any Flink code change. The discipline is then to make sure the Flink job does not select *, because a v2 producer adding a field would silently widen every downstream operator's row type. Selecting explicit columns gives the same robustness as in any SQL system: schema additions don't propagate where they aren't asked for.

Multi-region registries and the consistency tradeoff

A registry serving consumers in Mumbai and Singapore needs to satisfy both with low read latency. Confluent's reference architecture runs the registry as a single global cluster with one leader; reads can be served from local replicas but writes go cross-region, which adds 80–120ms to schema-registration latency. Apicurio takes a different approach with multi-leader replication via Kafka's compacted topic — every region can register a schema locally, and conflicts are resolved by Kafka's offset ordering. The Apicurio model is more available but admits the possibility of two regions briefly disagreeing on what schema ID 4827 means, which is a correctness bug if a producer in one region writes a message with that ID and a consumer in the other region reads it before the registries converge. The PhonePe team chose the single-leader model because correctness mattered more than the 80ms registration latency.

A related correctness wrinkle is Kafka topic compaction. Compacted topics retain the latest message per key indefinitely, which means a fresh consumer joining the topic in 2026 may decode messages produced under schema v1 in 2022. Time-retention topics expire data after 7 days so the historical window is bounded; compacted topics have no such bound. PhonePe's discipline is to enforce FULL compatibility on every compacted topic — no ambiguity about whether old messages can be read — while leaving BACKWARD as the default for time-retention topics where the older-than-retention-window data is gone anyway.

Schema-registry-as-source-of-truth for downstream tooling

By 2026, the registry is increasingly the substrate for tools that need a machine-readable view of every topic's schema: data catalogs (chapter 30) ingest from the registry to populate field-level descriptions; column-level lineage tools (chapter 29) read the registry to know what fields exist on what topic at what point in time; data contracts (chapter 31) reference registry-assigned schema IDs in their YAML to pin the contract to a specific schema version. The registry stops being just a Kafka helper and becomes a structural data-asset itself, with its own backup/DR plan, its own access control, and its own SLA. At PhonePe, the registry's RPO target is 0 and RTO is 60 seconds — tighter than most production databases.

Where this leads next

The next chapter (33) covers freshness SLAs and the meaning of "late" — the operational guarantee that lives inside a contract's freshness block and is monitored against the registry-promised schema. Chapter 34 covers data-quality testing layered on top of contracts and registries — Great Expectations, Soda, dbt-test — for the row-level checks that contracts and registries together don't cover. Build 7 (chapters 76–95) builds the message log itself, where the registry's role as a schema source-of-truth becomes structural to the pipeline.

• Data contracts: the producer/consumer boundary — chapter 31, the producer-promise layer that wraps the registry's structural schema.
• Schema drift: when the source changes under you — chapter 17, the failure mode the registry's compatibility gate prevents at the source.
• Data catalogs and the "what does this column mean" problem — chapter 30, the discovery layer that ingests from the registry.

The registry is one of those pieces of infrastructure where its presence is unremarkable and its absence is catastrophic. Teams that have a schema registry consider it boring and rarely think about it; teams that don't have one spend Friday nights diagnosing AvroTypeException. The ₹2 lakh per year of operational cost to run a three-node registry is a rounding error against the cost of one production incident at PhonePe scale — and the gating discipline it enforces (every schema change passes through one place, gated by mechanical rules that don't get tired at 11 p.m.) is the thing that lets a 200-microservice fintech keep evolving its data layer without grinding the engineering org to a halt.

A practical bar: pick a random producer team and ask what happens when they add a field to a Kafka payload. If they describe a registry call that succeeds or fails in CI, the substrate is working. If they describe "we just deploy and watch the consumer dashboards" — the registry hasn't crossed from infrastructure into operational reflex.

References

1. Confluent Schema Registry documentation — the canonical reference for the registry's API, compatibility modes, and operational patterns.
2. Apache Avro Schema Resolution specification — the four-paragraph algorithm that defines how a writer schema and reader schema are reconciled.
3. Martin Kleppmann, "Schema evolution in Avro, Protocol Buffers and Thrift" (2012) — the classic comparative essay that framed the evolution problem before registries were standard.
4. Apicurio Registry project — the open-source Red Hat registry supporting Avro, Protobuf, and JSON-Schema with multi-leader replication.
5. KIP-69: Kafka Schema Registry — the Kafka improvement proposal that documented the design rationale for ID-based schema reference.
6. Gwen Shapira, "Schemas, Contracts, and Compatibility" (Confluent blog, 2020) — practitioner-level guidance from one of the registry's original designers on picking compatibility modes.
7. Data contracts: the producer/consumer boundary — chapter 31, the layer above the registry where semantics and governance live.
8. Confluent Schema Registry security guide — RBAC, TLS, and the operational hardening every fintech registry deployment uses.