Note: Company names, engineers, incidents, numbers, and scaling scenarios in this article are hypothetical — even when they resemble real ones. See the full disclaimer.

Context propagation across protocols

It is 02:14 IST. Riya, on-call at PaySetu, has a clear flame graph for the synchronous half of a failed ₹62,400 payout — five HTTP hops, all green, total 312 ms. Then the trace just stops. The payout's actual settlement happened on a Kafka consumer 47 seconds later, and that consumer ran in a span with no parent — orphaned, root-of-its-own-trace, unconnected to the customer's request. The payout failed inside that orphan span. Riya can see the failure; she cannot see which customer triggered it without grepping by amount across 18 GB of logs. The bug is one missing line in the Kafka producer: the traceparent header was never copied into the message. Context propagation is what makes a trace a trace; the moment it stops, you are back to log archaeology.

What gets propagated, and why each piece exists

A propagation context is a tiny tuple of fields that travels with every request across every hop. The OpenTelemetry specification splits it into two layers: the span context (the W3C traceparent quartet — version, trace ID, parent span ID, flags) and baggage (arbitrary key-value pairs the application chooses to carry, like user_tier=gold or region=ap-south-1). On top of those, most production systems also propagate a correlation ID (one flat string sampled at 100%; see correlation IDs) and a deadline (the absolute wall-clock time by which the request must finish; see deadlines and deadline propagation).

The fields have different lifetimes. Why the parent span_id is the only field rewritten on outbound: the trace is a tree, and each service's outbound RPC is a new edge in the tree whose source is the current service's span. If service A forwarded the inbound traceparent unchanged to service B, then B's span would point to A's parent as its parent, not to A — the tree would collapse, B would appear as a sibling of A, and the flame graph would lie about who called whom. The parent-pointer overwrite is what preserves causality. Trace ID, sampling flag, correlation ID, and baggage are immutable end-to-end — every hop sees the same values, by design. Deadline is the unusual one: it carries an absolute timestamp (e.g. 2026-04-29T02:14:18.504Z), and each hop computes its remaining budget by subtracting now(). Why absolute, not relative: if you propagate "300 ms remaining" instead, every hop has to subtract its own service time before forwarding, and any clock skew or arithmetic bug compounds. With an absolute deadline, the propagation contract is "copy this header verbatim", and only the receiving service does the math against its own local clock — one subtraction per hop, no compounding error.

The propagation contract has exactly three rules, and they are the same regardless of protocol: (1) extract the context from the inbound carrier (HTTP headers, Kafka record headers, gRPC metadata, AMQP properties); (2) make it the active context for the duration of the work; (3) inject it into every outbound carrier on every downstream call. Skipping any one rule breaks the trace at that hop.

How propagation differs across the four protocol families you'll meet

In production, one user request crosses up to four different families of boundary, and each family has its own carrier:

1. HTTP / gRPC — the easy case. The carrier is request headers (HTTP) or initial metadata (gRPC). Both libraries' OpenTelemetry instrumentations auto-propagate the W3C traceparent header. You usually do nothing — except confirm that no proxy or middleware strips it.
2. Message queues — Kafka, RabbitMQ, NATS, Redis Streams — the hard case. Each message carries its own headers (Kafka record headers, AMQP properties, NATS message header field). The producer must inject traceparent into those headers; the consumer must extract on poll. No instrumentation library does this for free in every language — the python confluent-kafka client, for instance, requires you to wire it manually unless you use OpenTelemetry's confluent-kafka instrumentation package.
3. In-process async hops — Python asyncio tasks, Java CompletableFuture chains, Go goroutines. The carrier is the language's context-propagation primitive: contextvars.ContextVar (Python), Context parameter (Go), ScopedValue (Java 21+), thread-local + MDC.copy() (Java pre-21). Skipping the handoff produces the most insidious bug — the trace continues in production as long as work is synchronous, then silently stops the moment a loop.create_task(...) is called without copying context.
4. Storage and side-effect boundaries — database writes, S3 events, scheduled cron jobs. The carrier is whatever your storage layer can attach to a record. The pattern is to write trace_id, span_id, and cid as columns or object metadata, then re-create a new root span in the consumer with span links back to the original — see "span links" in distributed tracing.

The hardest boundary is almost always the message queue, because it is the one where the trace can outlive the originating request and still want to point back at it.

# propagation.py — extract / inject context across HTTP, Kafka, and asyncio.
# Demonstrates the three carriers most production systems hit.
import asyncio, contextvars, secrets, time, json

ctx_var: contextvars.ContextVar[dict] = contextvars.ContextVar("ctx", default={})

def extract(carrier: dict) -> dict:
    """Read traceparent + baggage out of any string-keyed carrier."""
    tp = carrier.get("traceparent", "")
    parts = tp.split("-") if tp else []
    if len(parts) == 4 and parts[0] == "00":
        ctx = {"trace_id": parts[1], "parent_span_id": parts[2], "flags": parts[3]}
    else:
        ctx = {"trace_id": secrets.token_hex(16), "parent_span_id": None, "flags": "01"}
    bag = carrier.get("baggage", "")
    ctx["baggage"] = dict(p.split("=", 1) for p in bag.split(",") if "=" in p) if bag else {}
    ctx["cid"] = carrier.get("x-correlation-id") or secrets.token_hex(8)
    return ctx

def inject(ctx: dict, my_span_id: str, carrier: dict):
    carrier["traceparent"] = f"00-{ctx['trace_id']}-{my_span_id}-{ctx['flags']}"
    if ctx["baggage"]:
        carrier["baggage"] = ",".join(f"{k}={v}" for k, v in ctx["baggage"].items())
    carrier["x-correlation-id"] = ctx["cid"]

async def serve(name: str, inbound: dict):
    ctx = extract(inbound)
    my_span = secrets.token_hex(8)
    ctx_var.set({**ctx, "span_id": my_span, "service": name})
    print(f"[{name}] trace={ctx['trace_id'][:8]} parent={(ctx['parent_span_id'] or '----')[:8]} "
          f"span={my_span[:8]} cid={ctx['cid']} tier={ctx['baggage'].get('user_tier','-')}")
    outbound: dict = {}
    inject(ctx, my_span, outbound)
    return outbound

async def main():
    edge_inbound = {"baggage": "user_tier=gold,region=ap-south-1", "x-correlation-id": "c8f4a201"}
    h_gw = await serve("api-gateway", edge_inbound)
    h_order = await serve("order-svc", h_gw)
    h_kafka_msg = await serve("kafka-producer", h_order)        # injects into "Kafka headers"
    # 47 seconds later, on a different host, the consumer extracts:
    h_consumer = await serve("settlement-consumer", h_kafka_msg)

asyncio.run(main())

Sample run:

[api-gateway] trace=9a4f7c1b parent=---- span=7d9e1f2a cid=c8f4a201 tier=gold
[order-svc] trace=9a4f7c1b parent=7d9e1f2a span=1c3e5a7b cid=c8f4a201 tier=gold
[kafka-producer] trace=9a4f7c1b parent=1c3e5a7b span=8b6d4f2e cid=c8f4a201 tier=gold
[settlement-consumer] trace=9a4f7c1b parent=8b6d4f2e span=4a2c6e80 cid=c8f4a201 tier=gold

Walkthrough. The extract function treats the carrier polymorphically — it works on any dict-like object, which is why one function handles HTTP headers, Kafka record headers, AMQP properties, and gRPC metadata. The inject function is the inverse, with one subtle but critical detail: it sets parent_span_id in the outbound to my_span_id (the current service's own span), not to the inbound parent. Why this single line is the most-violated rule in propagation: a tired engineer often writes carrier["traceparent"] = inbound_traceparent, just forwarding the header. That preserves the trace ID but loses the parent-child structure — every downstream span ends up parented to the upstream's parent, the tree collapses to a flat list, and the flame graph stops showing causality. Always overwrite the parent slot with your own span ID. The ctx_var.set(...) stores the active context per asyncio task; child tasks await-ed from this task inherit it via contextvars's task-local copy. The kafka-producer → settlement-consumer edge crosses a 47-second wall-clock gap and a process boundary on a different host, but the trace tree is still intact because the same headers were injected into the Kafka message and extracted on the other side.

The four boundaries that drop context, and how to fix each

After running this in production for two years, the bug pattern is always the same: the trace dies at one specific boundary your team didn't instrument. The four most common offenders, in descending order of frequency:

Kafka producers forgetting headers is the most common failure. The fix is either OpenTelemetry's confluent-kafka auto-instrumentation (which monkey-patches Producer.produce to inject headers from the active context) or a one-line manual call: producer.produce(topic, value, headers=[("traceparent", current_traceparent.encode())]). Why this is so easy to miss: most teams write the producer code once, on day one, before they have an observability stack. Three years later they have OpenTelemetry shipping HTTP traces beautifully, and nobody has revisited the Kafka producer. The auto-instrumentation packages bridge this gap precisely because retrofitting manual headers= calls across 47 producers is tedious — but the auto-instrumentation must actually be installed and imported.

asyncio.create_task losing context is a Python-specific subtlety. create_task does copy contextvars at task creation — but loop.run_in_executor, which is what most code reaches for to run blocking work in a thread pool, does not unless you explicitly wrap with contextvars.copy_context().run(fn, *args). The same pattern bites Java's old ExecutorService.submit (where you need MDC.getCopyOfContextMap() + restore in the runnable) and Go's errgroup (where you must explicitly forward ctx context.Context rather than relying on a captured outer scope).

Third-party HTTP libraries and proxies stripping unknown headers is a configuration trap. CloudFront, Akamai, and many Nginx setups by default whitelist known headers and drop the rest. traceparent and tracestate are unknown to a 2018-era CDN config. The fix is one line in the proxy: explicitly allowlist traceparent, tracestate, baggage, and x-correlation-id. Verify by curl-ing through the proxy and inspecting headers received downstream.

Cron jobs and scheduled processors are the conceptual hard case — the cron job genuinely is a new root, because it's not triggered by a user request. The right pattern is: when the row was written, the writing service stored trace_id and cid as columns. When the cron picks up the row, it starts a new root span (its own trace), but adds a span link (see "span links" in distributed tracing) pointing back at the original trace. The reader can navigate from the cron-job trace to the original-request trace via the link.

The diagnostic for all four of these is a single SLO: the orphan-span ratio — spans whose parent_span_id is null but whose service is not an entry point (i.e. not your API gateway, mobile SDK, or cron scheduler). In a system with healthy propagation this ratio is under 1% of all spans. KapitalKite's platform team caught their first major propagation regression — a Spring Boot upgrade had silently changed the default RestTemplate to a version without OTel instrumentation — when this metric jumped from 0.4% to 11% within an hour of the deploy.

Common confusions

• "OpenTelemetry's auto-instrumentation handles all my propagation" It handles HTTP and gRPC for the languages it instruments, on the libraries it knows about. It does not automatically instrument every Kafka client (Python confluent-kafka needs the dedicated package), every async runtime, every custom RPC framework you wrote in 2019, or every proxy in your network. Audit boundaries explicitly; don't assume.

• "traceparent is enough — I don't need a correlation ID" A trace ID is sampled at 1–5% in production; a correlation ID is logged at 100%. When the trace you need is in the un-sampled 99%, the correlation ID is the only thing left to grep on. Most mature systems propagate both — the cid in x-correlation-id, the trace ID in traceparent — and the log line carries both as fields. See correlation IDs.

• "Baggage is just extra trace tags" No — baggage is propagated to every downstream service in the request, while trace attributes (http.status, db.statement) only attach to the local span. Baggage is what lets the recommendation service know "this user is in tier=gold" without having to re-fetch from the user-profile service. The cost: every byte in baggage rides on every outbound request, so it's tempting to overload it. Cap your baggage at a few hundred bytes; do not put PII in it.

• "Propagating the deadline as 300ms is the same as the absolute timestamp" It is structurally different. Propagating a relative timeout of 300ms means every hop must subtract its own service time before forwarding — and any clock skew or arithmetic bug compounds across hops. An absolute deadline (epoch_ms = 1714326858504) is "copy this verbatim", and only the receiving service does math against its local clock — one subtraction per hop, zero compounding.

• "tracestate is for tags I want to set" No — tracestate is for vendor extensions and is governed by strict format rules (vendor=value, comma-separated, ≤512 bytes). User-application data goes in baggage, which is what OpenTelemetry's Baggage API writes to. Don't write into tracestate from application code.

• "If I propagate carefully, I never need span links" You do, the moment you have a fan-in async pattern. A queue consumer that batches 100 messages from 100 different traces into one processing span has no single parent — there are 100 logical parents, all at the same level. Span links are exactly that: one parent (which is null in this case, making the span a root) and an unbounded list of links to upstream spans. Propagation gives you the tree; span links give you the DAG.

Going deeper

W3C Trace Context's design choices, and why they matter

The 2020 W3C Trace Context Recommendation was deliberately minimal — traceparent carries four fixed-position fields, no extensibility, and tracestate is the escape hatch for everything else. Why this rigidity is a feature: a fixed-format, version-prefixed binary-friendly header is parseable in 50 nanoseconds without allocations on every hop, in every language. Extensible JSON-in-header schemes (which earlier proposals favoured) require allocation-heavy parsing on every middlebox. At scale (a CDN at 10M req/sec parsing every header), the allocation cost dominates — so W3C deliberately chose rigidity over expressiveness, and pushed expressiveness into a separate tracestate header that vendors could parse only if they cared. This is why traceparent is precisely 55 bytes (00- + 32 hex + - + 16 hex + - + 2 hex + checksum), always — it's parseable as a slice operation, not a parser invocation.

The PaySetu Kafka-headers retrofit

PaySetu shipped its OpenTelemetry rollout in three phases over six months. Phase 1 wired up HTTP and gRPC — the wins were immediate, and the orphan-span ratio dropped from a baseline of unknown to about 8%. The team initially celebrated. Phase 2 was an audit: where did the remaining 8% live? Almost entirely on Kafka consumer entry points — every settlement, every notification, every reconciliation job. Phase 3 was the painful retrofit: 31 producer call sites across 14 services, each requiring a headers=[("traceparent", ...), ("baggage", ...), ("x-correlation-id", ...)] parameter. The team automated 90% of this with a wrapper class (InstrumentedProducer) that pulled the active context and injected automatically; the remaining 10% were producers wrapped in custom abstractions that needed bespoke fixes. After phase 3, orphan ratio dropped to 0.6%. The lesson: HTTP/gRPC is the easy 80% — the remaining 20% is where the engineer-hours live.

Span links: the DAG, not the tree

Pure context propagation gives you a tree — every span has exactly one parent. Real systems have DAGs. A Kafka consumer that batches 100 messages, a fan-in aggregator that joins 5 upstream RPCs, a Saga that retries with a new attempt all have multiple causal predecessors. OpenTelemetry models this with span links — a span has one parent (or null, marking it a root) and an arbitrary list of Link objects pointing at predecessors. The flame graph UIs render this as "this span was caused by: [trace1, trace2, ...]" with click-to-pivot. Why span links are not "multiple parents": parents propagate sampling decisions and form the primary tree shape; links are pure references. If a span had multiple parents in the formal sense, the tree's sampling logic would have to choose between conflicting flags, the duration model (parent contains child) would break, and rendering would have to handle cycles. Links sidestep all of this — they're an annotation, not a structural edge.

Reproduce this on your laptop

# Reproduce this on your laptop
python3 -m venv .venv && source .venv/bin/activate
pip install opentelemetry-api opentelemetry-sdk opentelemetry-instrumentation-confluent-kafka confluent-kafka
# minimal propagation script — extracts/injects context across HTTP, Kafka, asyncio
python3 propagation.py
# observe orphan-span ratio in your collector by querying spans with parent_span_id=null
# excluding entry-point services (api-gateway, mobile-sdk, cron-scheduler).

Where this leads next

Context propagation is the plumbing that makes the next chapters possible. Tail-latency aggregation (next chapter) only makes sense when you can attribute a slow span to one specific trace, then pivot to that trace via the cid. Service dependency graphs (chapter 124) are built by aggregating parent-child edges across millions of traces — they only exist if propagation works at every hop, including the queue and async ones. Debugging cross-service outages (chapter 125) is the payoff: at 03:00 IST when the on-call has six minutes to find which of 47 services is the culprit, the trace ID propagated cleanly across every boundary is the difference between a one-query answer and a two-hour grep.

The deeper pattern: propagation is one of the few distributed-systems disciplines where the cost is paid by the person writing the producer, but the benefit goes to the person reading the consumer's logs at 3 AM. That asymmetry is why it always degrades unless someone owns it — usually the platform team, with the orphan-span SLO as their canary.

References

1. W3C, Trace Context (Recommendation 2020) — the binding spec for traceparent and tracestate.
2. W3C, Baggage (Recommendation 2021) — the spec for the baggage header carrying user-defined key-value context.
3. OpenTelemetry Specification, Context Propagation — the canonical propagator API and the carrier-injection contract.
4. Sigelman et al., "Dapper, a Large-Scale Distributed Systems Tracing Infrastructure" (Google Tech Report 2010) — propagation of metadata via the RPC framework as the original mechanism.
5. OpenTelemetry, Kafka instrumentation for Python — the auto-instrumentation that closes the most common propagation gap.
6. Cindy Sridharan, Distributed Systems Observability (O'Reilly 2018), Chapter 6 — the propagation-discipline framing that motivates orphan-span SLOs.
7. Yuri Shkuro, Mastering Distributed Tracing (Packt 2019), Chapter 8 — propagation across messaging and async boundaries, with a worked Kafka example.
8. See also: correlation IDs, distributed tracing (W3C, Dapper, Jaeger), deadlines and deadline propagation.