B3, W3C Trace Context: two wire formats, one tree

It is 02:40 AM in Bengaluru. Karan, an SRE on the Swiggy ordering platform, is staring at Tempo. A delivery-rider's order took 11 seconds; the trace tree shows only six spans when there should be twenty. Five of the six are from one Java service still running an old Brave-Zipkin instrumentation that emits b3 headers. The newer Python services downstream all speak traceparent. Somewhere on that boundary, a span is being minted with a fresh trace_id because the receiving service did not parse the header it was sent. The bug is not in the application; the bug is in the wire format mismatch.

This chapter takes the two propagation formats that almost every production trace pipeline carries — Zipkin's older b3 (single-header and multi-header variants) and the W3C traceparent standard — pulls them apart byte-by-byte, builds a working multi-format propagator in Python, and shows the exact bug Karan was looking at.

B3 propagates trace identity through either four separate headers (X-B3-TraceId, X-B3-SpanId, X-B3-ParentSpanId, X-B3-Sampled) or one packed b3: header (b3: TraceId-SpanId-Sampled-ParentSpanId). W3C traceparent is one fixed-format header — traceparent: 00-<trace_id>-<span_id>-<flags>. They encode the same trace_id and parent span_id, but B3 carries the parent_span_id explicitly while W3C derives it from receiver context. Mixed environments need a multi-format propagator that reads either and writes both — the OpenTelemetry SDK ships exactly this, but configuring it wrong fragments trace trees at the format boundary.

What each header actually carries — byte by byte

Every modern tracer has to solve the same problem: when service A makes an HTTP call to service B, A must hand B enough information for B to claim its first span has the right trace_id and the right parent_span_id. Both B3 and W3C Trace Context solve it; they just disagree on the wire syntax. Reading the bytes side by side is what makes the disagreement concrete.

The B3 multi-header format (Zipkin's original) ships four separate HTTP headers. X-B3-TraceId is 16 hex chars (64-bit, the original Zipkin design) or 32 hex chars (128-bit, added in B3 v1.1 — 2019). X-B3-SpanId is 16 hex chars (64-bit). X-B3-ParentSpanId is 16 hex chars (64-bit) and carries the caller's parent, not the caller itself — this is the trip-up. X-B3-Sampled is 0 or 1. There is also X-B3-Flags for debug (1 = force-sample). Multi-header B3 is what Brave (Java), Zipkin's reference instrumentation, and the original Spring Cloud Sleuth emit by default.

The B3 single-header format (b3: header, Zipkin spec 2018) packs the same fields into one string: b3: <TraceId>-<SpanId>-<Sampled>-<ParentSpanId>. The trailing -<ParentSpanId> is optional — if the caller is itself a root span, it is omitted, giving b3: <TraceId>-<SpanId>-<Sampled>. There is also a debug short form: b3: 0 means "do not sample, no trace identity". Single-header B3 is what Envoy and Istio's service mesh emit by default for their internal hops because one HTTP header is cheaper to parse than four.

The W3C traceparent format is one header with a strict shape: traceparent: <version>-<trace_id>-<parent_id>-<flags>. Version is 00 (only one defined, with strict-mode parsing). trace_id is mandatory 32 hex chars (128-bit; the spec disallows 64-bit entirely). parent_id is mandatory 16 hex chars (64-bit) — and this is the caller's own span_id, which the callee will record as its parent_span_id. flags is two hex chars carrying the sampled bit (01) and seven reserved bits. There is a sibling tracestate header for vendor-specific extensions and a separate baggage header for application-level key-value propagation, but traceparent alone is what stitches the tree.

B3 multi-header, B3 single-header, and W3C traceparent — same payload, three syntaxesThree boxed panels showing the byte-level structure of B3 multi-header (four headers stacked), B3 single-header (one packed string with hyphenated fields), and W3C traceparent (one header with version-prefixed fields). Each field labelled with its bit width.Three wire formats, one set of identifiersB3 multi-header (four HTTP headers)X-B3-TraceId: 9f4e2a0bdc3f7261d4e8b75c821ae8a2128-bit (or 64)X-B3-SpanId: 3d51b07ef2c9981464-bit, caller's spanX-B3-ParentSpanId: a0c47def81b2532064-bit, caller's parent (optional)X-B3-Sampled: 10 or 1B3 single-header (one packed string)b3: 9f4e2a0b...e8a2-3d51b07ef2c99814-1-a0c47def81b25320format: TraceId - SpanId - Sampled - ParentSpanId(optional)if caller is root: "b3: TraceId-SpanId-Sampled" (no trailing parent)W3C traceparent (one header, strict format)traceparent: 00-9f4e2a0b...e8a2-3d51b07ef2c99814-01format: version(00) - trace_id(128) - parent_id(64) - flags(8)parent_id = caller's span_id, becomes callee's parent_span_idflags: 01 = sampled, 00 = not sampled (re-decision forbidden)
Illustrative — same trace, three wire formats. Bytes are equivalent; framing differs. B3 ships parent-of-caller; W3C ships caller's own span_id and lets the callee derive parent at receive time.

The single point that trips engineers reading both specs for the first time is what parent_span_id (B3) and parent_id (W3C) actually mean. In B3, X-B3-ParentSpanId is the caller's own parent — i.e. the grandparent of the next span. The callee uses X-B3-SpanId (the caller's span_id) as its parent_span_id. In W3C, parent_id already is the caller's span_id — there is no field for the caller's parent, because the caller does not need to send it. Same data structure, opposite framing. Why W3C dropped the parent-of-parent field: it is redundant. The receiving service does not care who the caller's parent was — it only cares about who its own parent is, which is the caller. B3's extra field is a historical accident from Zipkin's early Dapper-era model where the entire ancestor chain was sometimes propagated for debugging; the W3C working group decided in 2018 that one ancestor was enough, and the rest is reconstructable at query time.

Build a multi-format propagator — runnable in 90 lines

The fastest way to internalise both formats is to build a propagator that reads either, writes both, and shows you which one was used at each hop. The Python script below stands up two Flask processes: a gateway that emits a traceparent outbound (modern), and a legacy_payments service that responds to either format. Run it with both formats enabled and watch the logs.

# multi_format_propagator.py — read B3 (multi or single) or W3C traceparent;
# write both on outbound. Shows incoming/outgoing headers per hop.
# pip install flask requests
import json, threading, time, uuid, random
from flask import Flask, request, g
import requests

SPAN_LOG = "/tmp/b3_w3c_spans.jsonl"
open(SPAN_LOG, "w").close()
def emit(s): open(SPAN_LOG, "a").write(json.dumps(s) + "\n")

# ---- parsers: return (trace_id_hex, parent_span_id_hex, sampled_bool) ----
def parse_w3c(h):
    if not h: return None
    p = h.split("-")
    if len(p) != 4 or p[0] != "00" or len(p[1]) != 32 or len(p[2]) != 16:
        return None
    return (p[1], p[2], p[3] == "01")

def parse_b3_single(h):
    if not h or h == "0": return None
    p = h.split("-")
    if len(p) < 3: return None
    tid = p[0].rjust(32, "0")        # promote 64-bit to 128-bit
    return (tid, p[1], p[2] == "1")

def parse_b3_multi(headers):
    tid = headers.get("X-B3-TraceId")
    sid = headers.get("X-B3-SpanId")
    samp = headers.get("X-B3-Sampled", "0") == "1"
    if not tid or not sid: return None
    return (tid.rjust(32, "0"), sid, samp)

def extract(headers):
    for src, fn in [("w3c",   lambda h: parse_w3c(h.get("traceparent"))),
                    ("b3-1h", lambda h: parse_b3_single(h.get("b3"))),
                    ("b3-mh", parse_b3_multi)]:
        ctx = fn(headers)
        if ctx: return src, ctx
    return "none", None

# ---- writers: emit both formats on outbound for compatibility ----
def inject(trace_id, span_id, sampled):
    return {
        "traceparent": f"00-{trace_id}-{span_id}-{'01' if sampled else '00'}",
        "b3": f"{trace_id}-{span_id}-{'1' if sampled else '0'}",
        "X-B3-TraceId":   trace_id,
        "X-B3-SpanId":    span_id,
        "X-B3-Sampled":   "1" if sampled else "0",
    }

def serve(name, port, downstream=None):
    app = Flask(name)
    @app.route("/handle")
    def handle():
        src, ctx = extract(request.headers)
        if ctx is None:
            tid, parent_sid, sampled = uuid.uuid4().hex, "0"*16, True
        else:
            tid, parent_sid, sampled = ctx
        my_sid = uuid.uuid4().hex[:16]
        t0 = time.time_ns()
        time.sleep(random.uniform(0.01, 0.04))
        if downstream:
            requests.get(downstream, headers=inject(tid, my_sid, sampled),
                         timeout=2)
        emit({"service": name, "received_via": src,
              "trace_id": tid, "span_id": my_sid,
              "parent_span_id": parent_sid,
              "sampled": sampled,
              "duration_ms": (time.time_ns() - t0) / 1e6})
        return {"ok": True, "trace_id": tid}
    threading.Thread(target=lambda: app.run(port=port, use_reloader=False),
                     daemon=True).start()

serve("legacy_payments", 9202)            # speaks any format
serve("gateway",         9201, downstream="http://localhost:9202/handle")
time.sleep(0.4)

# Case A: client sends only W3C (modern Python service calling gateway)
print("--- Case A: incoming traceparent only ---")
requests.get("http://localhost:9201/handle", headers={
    "traceparent": "00-9f4e2a0bdc3f7261d4e8b75c821ae8a2-aaaaaaaaaaaaaaaa-01"})
time.sleep(0.2)

# Case B: client sends only B3 multi (legacy Java service calling gateway)
print("--- Case B: incoming X-B3-* only ---")
requests.get("http://localhost:9201/handle", headers={
    "X-B3-TraceId": "1111111111111111bbbbbbbbbbbbbbbb",
    "X-B3-SpanId": "cccccccccccccccc",
    "X-B3-Sampled": "1"})
time.sleep(0.3)

print()
for line in open(SPAN_LOG):
    s = json.loads(line)
    print(f"[{s['service']:16}] via={s['received_via']:5} "
          f"tid={s['trace_id'][:8]} sid={s['span_id'][:8]} "
          f"pid={s['parent_span_id'][:8]} samp={s['sampled']}")

A representative run produces:

--- Case A: incoming traceparent only ---
--- Case B: incoming X-B3-* only ---

[gateway         ] via=w3c   tid=9f4e2a0b sid=3d51b07e pid=aaaaaaaa samp=True
[legacy_payments ] via=w3c   tid=9f4e2a0b sid=a0c47def pid=3d51b07e samp=True
[gateway         ] via=b3-mh tid=11111111 sid=4f8e02bc pid=cccccccc samp=True
[legacy_payments ] via=w3c   tid=11111111 sid=82c7e103 pid=4f8e02bc samp=True

Per-line walkthrough. The extract() function tries the three formats in priority order — W3C first, then single-header B3, then multi-header B3 — which is the order the OpenTelemetry SDK uses when configured with W3CMultiPropagator and B3MultiFormat together. Why W3C wins ties: when both traceparent and b3 headers are present (a legacy service in the middle of a migration injects both), preferring W3C is the OpenTelemetry default because it is the standard. If you reversed the order, a service that overwrote b3 but left a stale traceparent would silently fork the trace. The priority is a safety choice, not a performance one. The inject() function emits both formats on every outbound call. This is the migration trick: a service that speaks W3C but writes B3 too can talk to legacy receivers without breaking trees, and the cost is two extra header bytes per request. The line tid.rjust(32, "0") promotes a 64-bit B3 trace_id to a 128-bit W3C trace_id by left-padding with zeros — both representations point at the same trace, and Tempo / Jaeger treat them as equivalent at query time. The line if ctx is None: ... parent_sid = "0"*16 is the root-span fallback: if no header parses (all three returned None), the gateway is the entry point and starts a fresh trace. This is exactly what the gateway should do for a curl request from outside; for an internal service, it would be a propagation bug. Production teams alert on internal_service_root_span_count going non-zero.

A subtle but important property of this propagator is that the parsers are defensive but not pedantic. The W3C parser checks length and version (p[0] != "00"), but does not verify that every character is hex — a real production parser must also reject non-hex characters with strict-mode handling. The SDK does this; the demo above does not, for clarity. Why the demo skips strict hex-validation: the goal here is to show the bidirectional flow, not to ship a hardened propagator. In production, you would never roll your own — from opentelemetry.propagators.tracecontext import TraceContextTextMapPropagator and from opentelemetry.propagators.b3 import B3MultiFormat give you parsers that handle every edge case in the W3C and B3 specs, including all-zero IDs, version-mismatch fallback, and the tracestate co-evolution. Educational code shows the shape; production code reuses the SDK. The demo's value is making the data flow visible — via=w3c versus via=b3-mh is information the SDK does not log by default, and adding that column to your own propagator wrapper is one of the highest-leverage changes you can make for debugging migration windows.

The output shows what matters for debugging. The via= column tells you which format the receiver actually parsed. In Case A, every hop sees w3c because the gateway received traceparent, recognised it, and wrote traceparent (and b3) onward; the legacy service preferred traceparent. In Case B, the gateway received only X-B3-*, parsed it as b3-mh, then wrote both formats outbound — and the downstream service, given the choice, picked W3C. This is the correct migration shape: you can have arbitrary mixtures of legacy and modern services, and as long as every service writes both formats, the tree stitches. The only thing you cannot do is have a service that only speaks B3 receive a request from a service that only writes W3C; that boundary fragments the trace.

Real-system tie-ins — three production failure modes

Format mismatch is responsible for a surprising fraction of "incomplete trace" tickets. Three concrete failure shapes show up repeatedly.

The first is the silent fork. A service in the middle of the call chain runs old Brave-Zipkin instrumentation that emits only b3 and ignores traceparent. When called by a modern Python service that emits only traceparent, the legacy service does not parse anything, treats itself as the entry point, mints a fresh trace_id, and starts a new trace. The user's one logical request becomes two traces in Tempo with no link between them. Querying for the customer's complaint by trace_id returns half the spans; on-call cannot reconstruct the story. Razorpay's payments platform retro from 2024 documented exactly this — a pre-OpenTelemetry Spring Boot service in the UPI mandate flow, running Spring Cloud Sleuth 2.x with B3-only output, was forking 12% of payment traces during a migration. The fix was to upgrade to Sleuth 3.x with W3C support, which they pushed in two weeks once they realised the cost. A senior SRE called it "the most expensive missing dash on a header in our history" — they had spent six weeks debugging a "p99 spike" that was actually fragmented traces being mis-bucketed by their span-aggregation pipeline.

The second failure mode is flag re-evaluation. The W3C spec is explicit: the sampled flag must propagate untouched through every hop. B3's X-B3-Sampled is the same. But poorly-written intermediaries — old Envoy filters, custom service-mesh proxies, some early API gateways — sometimes re-decide sampling at the edge ("only sample 1% even of incoming-sampled traces"). This produces partial trees: the leaf services have the spans but the middle layer dropped them, leaving a hole. Tempo renders the hole as a gap in the waterfall. Hotstar's IPL-2024 traces had a one-week window with ~3% of traces showing 200ms gaps in the middle that lined up exactly with one Envoy version that was downsampling the sampled flag — a known bug in that release. The fix was a single config knob (tracing.disable_resampling: true); the diagnostic was 18 hours.

The third failure mode is trace_id width mismatch. B3 originally specified 64-bit X-B3-TraceId (16 hex chars). B3 v1.1 in 2019 added 128-bit support (32 hex chars). Some services still emit 64-bit B3; their trace_id collides with another service's 128-bit trace_id whose lower 64 bits happen to match. In a backend that does not normalise widths (older Tempo versions before 2.3, some Jaeger setups), the spans land in the wrong tree. Two unrelated user requests can end up in one displayed trace, and the on-call sees a Frankenstein span tree that does not correspond to any real request. The fix is to normalise — left-pad 64-bit IDs to 128-bit at ingest, treat them as equivalent. Modern Tempo does this automatically; older deployments need explicit configuration. Zerodha Kite's tracing migration in 2024 hit this: their old Java order-routing service was emitting 64-bit B3, the new Python risk-engine emitted 128-bit W3C, and Tempo (then on 2.1) did not normalise — for two days, on-call engineers reported "this trace shows orders that I am sure did not happen" before the team realised what was going on.

A fourth, smaller-scale concrete pattern is encoding drift. The W3C spec mandates lowercase hex; B3's spec is case-insensitive. Some backends (older Jaeger, some Datadog ingest paths) reject traceparent headers with uppercase hex characters with a 400 error, dropping the trace entirely. The fix is at the SDK: every modern OpenTelemetry SDK lowercases on emit, but if a service is hand-rolling the header — say, a Bash wrapper around curl that constructs traceparent from environment variables — it must lowercase explicitly. Swiggy's internal CI infrastructure had a shell-based instrumentation hack for cron jobs that was emitting uppercase trace_ids; for six months, the cron-job traces were being silently dropped at the Tempo edge. The fix was one tr 'A-Z' 'a-z' in the wrapper.

A fifth pattern, the most insidious, is header-stripping proxies. Some load balancers, WAFs, and CDNs (Cloudflare's older WAF rulesets, AWS ALB pre-2022, some on-prem F5 configurations) strip headers they do not recognise. traceparent lands in the recognised-header list because it has a registered IANA name as of 2021; older X-B3-* headers do not, and some strict proxies drop them. The result is a trace that stitches across proxied hops if W3C is in use but fragments if only B3 is in use. The diagnosis is that the fragmentation is path-dependent — a request that goes through the WAF loses spans, a direct call (e.g. internal service-to-service traffic that bypasses the WAF) keeps them. Why this is hard to debug: the trace fragmentation only appears for the small fraction of requests that traverse the WAF. The bulk of internal traffic looks fine. Engineers see "most traces are complete, some are not" and assume sampling is the cause — but sampling either keeps or drops a whole trace, never half of one. The signature of header-stripping is "every spans-from-the-WAF-onward is missing", which is structurally different from "every span has a 1% chance of being missing". Pattern-matching on this signature is the diagnostic. Cred's 2024 outage retro mentioned exactly this — their CloudFront distribution was stripping X-B3-* headers but passing traceparent, and during a migration window where some services were still on B3-only, only the WAF-traversing fraction of their request volume had broken traces. The fix was to either (a) configure CloudFront to whitelist X-B3-* or (b) speed up the W3C migration. They did both.

The silent fork — a B3-only legacy service in a W3C call chainA horizontal swimlane diagram of four services. The first three propagate via traceparent and form a connected tree. The third service (legacy, B3-only) ignores traceparent and starts a fresh trace_id, splitting the tree into two unconnected halves. Both halves are valid traces but neither is complete.The silent fork — one B3-only service breaks the treegatewayW3CcheckoutW3Clegacy-svcB3-onlypaymentsW3Ctraceparent: 00-9f4e..-3d51..-01traceparent: 00-9f4e..-a0c4..-01(legacy-svc ignores it)traceparent: 00-NEW..-bb..-01(fresh trace_id, fork happens)trace 9f4e..: gateway → checkout (2 spans)trace NEW..: legacy-svc → payments (2 spans)
Illustrative — the user made one request. The trace tree should have four spans. Because the legacy service does not parse `traceparent`, it mints a fresh trace_id and the tree splits into two unrelated traces of two spans each. Neither is complete; on-call cannot reconstruct the request.

Beyond HTTP — gRPC, Kafka, and the propagator-per-transport problem

The W3C and B3 specs both define HTTP-header bindings, but a real production trace crosses many transports. gRPC carries metadata as binary key-value pairs, not HTTP headers — the OpenTelemetry gRPC instrumentation injects traceparent and b3 keys into the gRPC metadata bag, which renders identically on the wire as HTTP/2 trailers, but the encoding is the same: lowercase hex, hyphen-separated. Kafka has no HTTP layer at all; the modern pattern (since OTel 1.20) is to inject traceparent into the Kafka record headers, a feature added in the Kafka client at version 0.11. Async message-queue trace propagation is where the "links" concept from the data model kicks in — a consumer that batches 100 records receives 100 different traceparent headers and uses span links rather than parent-pointers to record the influence (see Span, trace, context — the data model).

The non-obvious failure mode is transport-specific propagator gaps. A service that uses OpenTelemetry's HTTP auto-instrumentation but also makes Redis calls without the Redis instrumentation will produce HTTP spans with correct propagation and Redis spans with no parent — orphan spans inside what looks like a complete trace. This is the per-transport-propagator problem: each transport needs its own injection / extraction code, and a service is only as well-instrumented as its weakest transport. PhonePe's UPI fraud-detection pipeline hit this in 2024 — their HTTP traces from the API gateway through the fraud-scoring service stitched perfectly, but the Redis hops where the fraud service cached customer-risk-tier scores produced orphan spans. The fix was three lines of code (RedisInstrumentor().instrument()); the diagnostic was nine days because the team was looking for an HTTP propagation bug. The lesson: when you see orphan spans, audit every outbound network call your service makes — HTTP, gRPC, Redis, Postgres, Kafka, RabbitMQ, S3 — and verify each one has a propagator wired in. The OpenTelemetry "auto-instrumentation" packages cover most of these; the gaps are the ones where you need to know.

A concrete shape for the Kafka case: a Dream11 leaderboard pipeline (T20 World Cup 2024) had Python producers writing scores to a Kafka topic, a Spark Structured Streaming consumer reading them, and a downstream Postgres-write service consuming the Spark output. The Python producers used OTel's KafkaInstrumentor() and correctly injected traceparent into each record's headers. The Spark consumer (a Scala job) used an older kafka-clients 0.10 build that silently dropped record headers — the API existed at 0.11+, and 0.10 happily ignored injected headers without error. Every trace fragmented at the Kafka boundary. The fix was a kafka-clients upgrade and explicit propagator wiring on the Scala side. Time-to-fix once the issue was named: 30 minutes; time-to-name-it: six days. The lesson generalises — whenever a transport has had a propagation feature added at a specific version, every consumer of that transport must run at that version or later, and the diagnostic is to pin the version of every Kafka / gRPC / Redis client in your fleet as a precondition for trusting trace stitching across them.

AWS X-Ray runs its own propagator on top of all of this. Its header is X-Amzn-Trace-Id: Root=1-abc-def;Parent=xyz;Sampled=1, and the Root field is X-Ray's 96-bit trace_id (8 hex epoch + 24 hex random — different layout from W3C's 128-bit). Bridging X-Ray traces to OpenTelemetry traces requires the AWS Distro for OpenTelemetry (ADOT) translator, which left-pads X-Ray IDs into W3C-compatible 128-bit form. If you run AWS-native services (Lambda, API Gateway, ALB) alongside OTel-instrumented services, this bridging is mandatory — without it, traces split at the AWS boundary the same way they split at a B3-only legacy service. AWS publishes a reference propagator implementation; teams that try to write their own typically get the epoch-prefix wrong.

Common confusions

Going deeper

The W3C tracestate header — vendor extensions without forking the standard

traceparent is fixed-format; tracestate is the W3C extension point for vendor-specific data. The format is tracestate: vendor1=value1,vendor2=value2 (up to 32 entries, each key under 256 chars). Datadog uses it to ship internal sampling priorities (tracestate: dd=s:1;p:abc123); New Relic uses it for inter-account routing; AWS X-Ray uses it for sampling rules. The key property is that vendors must not break each other — every vendor reads only its own key, leaves others untouched, and re-emits the full header. This makes it possible for a request to flow through Datadog APM, then OpenTelemetry Collector, then AWS X-Ray, with each adding its own tracestate entry without trampling the others. The cost is wire-format complexity: the tracestate parser must handle commas, equals signs, and quoting rules, and most home-grown propagators get this wrong. Use the SDK's parser; do not write your own.

B3 debug mode — the b3: 0 and the debug flag

B3 has two debug shortcuts that no other format has. First, b3: 0 (just the literal string 0) means "do not sample, no trace identity propagated" — useful for health-check requests that should never produce traces. Second, X-B3-Flags: 1 (the debug flag) means "force-sample this trace regardless of any other sampling decision" — the trace must be kept even if the sampling rate is 0%. Modern OpenTelemetry maps the latter to the W3C sampled=01 flag (no debug-vs-normal distinction at the wire level), but Brave-instrumented services still emit it. If your tracing backend treats debug-flagged traces specially (some teams send them to a separate, never-sampled storage tier), translating B3 debug to OTel correctly matters. The W3C draft for "Trace Context Level 2" includes a debug flag in the reserved bits but it has not landed in any production SDK as of 2025.

A concrete tracestate example from a production trace at Cleartrip: a request that flows from their booking gateway (instrumented with Datadog APM) through their payments service (instrumented with OpenTelemetry, exporting to Tempo) carries tracestate: dd=s:1;p:abc123,otel=k1:v1 on every hop. Datadog's collector reads only the dd= entry to extract its sampling priority; Tempo's collector reads only the otel= entry; both are happy. If you grep for tracestate in your incident-response runbooks, you should also be tracking the entries each tool adds — when dd= disappears, Datadog has lost the trace; when otel= disappears, OpenTelemetry has. Treating tracestate entries as a per-tool heartbeat is a small but useful discipline.

Migration cookbook — moving a Java fleet from B3 to W3C without breaking trees

The OpenTelemetry community has settled on a three-phase migration pattern that minimises trace breakage.

Phase 1 (months 1–2): every service runs the OTel SDK with both B3MultiFormat and W3CTraceContextPropagator configured for both reading and writing. This is the "read both, write both" state — every outbound request carries both traceparent and b3 headers. The cost is roughly 60 extra bytes per HTTP call. The gate to Phase 2: every service in the fleet has the SDK upgrade deployed and a metric showing it is emitting both formats.

Phase 2 (months 2–6): services flip their read priority from B3-first to W3C-first as they upgrade their instrumentation. Trees still stitch because writing remains both-formats. The gate to Phase 3: aggregate access-log telemetry across your entire fleet shows that 100% of incoming traffic carries traceparent. Until that is true, some upstream is still B3-only and dropping B3 reading would fragment its traces.

Phase 3 (months 6+): services drop B3 emission once you can prove no upstream service is sending B3-only — a query against your access-log telemetry showing zero X-B3-* headers received over a 30-day window is the gate. This three-phase approach is what Flipkart, Cred, and Dream11 have all publicly documented for their fleet-wide migrations. The mistake to avoid: Phase 1 + drop B3 reading before Phase 3 — that is the silent-fork trap.

What B3 got right and W3C kept

W3C Trace Context inherited several B3 design choices: the parent-pointer-based tree assembly, the per-process span_id minting, the propagated sampled flag. The bits W3C changed are the version byte (B3 has none — making future-proofing impossible), the strict 128-bit width (B3 had a width-confusion era from 2010 to 2019), and the dropping of the redundant grandparent field. The bits W3C left out are B3's debug flag (subsumed into sampled) and the B3 single-header debug literal (b3: 0). For a greenfield system, W3C is strictly better; for a fleet that has years of B3-instrumented services, B3 plus the OTel SDK's bidirectional propagator is the path. The two are not competing on technical merit — they are competing on installed base, and W3C is winning because OpenTelemetry chose it as the default.

Pinned to a timeline: B3 single-header (the packed format) shipped in early 2018, W3C Trace Context Level 1 became a Recommendation in February 2020, and OpenTelemetry's general availability in late 2021 made W3C the de facto standard. Every cloud-native CNCF project incubated since 2022 (Linkerd, Istio recent versions, Kuma, Cilium) emits traceparent by default and treats B3 as opt-in legacy. The arc is closing: by 2027–2028, most fleets currently mid-migration will have completed Phase 3 and dropped B3 emission entirely, leaving B3 as a curiosity in archived Zipkin instances. Understanding both formats today is still load-bearing for any production engineer; in five years it will be a piece of historical context. The skill that lasts longer is the meta-skill: when a new wire format eventually replaces W3C (as it inevitably will, perhaps for post-quantum signed traces or for OpenTelemetry 2.x), the migration shape — read both, write both, drop old when telemetry says zero usage — will be the same.

Why neither format propagates the full ancestor chain, and what goes wrong if you write your own parser

A reasonable question on first reading both specs is "why not propagate the entire span_id chain from root to current — wouldn't that make trace assembly easier?" The answer is wire-format cost. A trace with 47 spans and seven hops would carry 47 × 16 hex chars = 752 bytes of ancestor IDs on every internal HTTP call, plus the 32-byte trace_id and the current span_id — over 800 bytes of telemetry overhead per request. That is non-trivial on a high-QPS path. Tree assembly at query time is cheap (a hashmap walk over the spans returned for a trace_id query) and only happens when someone is actually looking at the trace; propagation overhead is paid on every hop of every request. The W3C trade-off — propagate only the immediate parent, derive the rest at query time — is the right one. It is the same trade-off Dapper made in 2010, and 15 years of production experience has confirmed it.

A related sub-question: why not let services write their own parsers? A surprising amount of production trace breakage comes from teams that wrote their own traceparent parser instead of using the SDK's. The W3C grammar is short — four hyphen-separated fields, each a fixed length — and feels easy to parse with split("-"). The trap is the strict-mode rules that Section 3.3.2 of the spec spells out: a parser must reject a header that has wrong field counts, wrong field lengths, non-hex characters, or all-zero trace_id or parent_id. A naive split("-") accepts all of these silently, producing a parsed context with garbage values — which then propagates onward, corrupting every downstream trace. The OpenTelemetry SDK's parser implements all eight reject conditions; a hand-rolled parser typically implements two or three. The right answer is always to use opentelemetry.propagators.tracecontext.TraceContextTextMapPropagator() from the SDK rather than rolling your own. The exception is a Bash or shell-script consumer where Python is not available — even then, prefer to invoke a tiny Python helper rather than parsing in shell.

A practical cheat sheet for trace_id format conversion when you are pasting between tools at 02:40 AM: a 128-bit hex trace_id is 32 lowercase hex chars, no separators (9f4e2a0bdc3f7261d4e8b75c821ae8a2); a B3 64-bit trace_id is the lower 16 chars of the 128-bit form (d4e8b75c821ae8a2); some tools display with hyphen separators every 8 chars (9f4e2a0b-dc3f7261-d4e8b75c-821ae8a2) — strip them before pasting into Tempo; Datadog renders trace_ids as decimal integers (15341432175033030818 is d4e8b75c821ae8a2 in hex). Knowing all three representations cold lets you copy a trace_id from a Slack incident channel into any backend without losing time to format mismatch.

Where this leads next

One concrete experiment to run before moving on: enable the OpenTelemetry SDK's composite propagator with both W3C and B3 readers, then look at the access logs of any one service in your fleet. Count the ratio of incoming requests that carry traceparent, b3 (single-header), and X-B3-* (multi-header). That ratio is your migration progress dashboard — if 80% of traffic is W3C and 20% is still B3, you are mid-migration; if 99% is W3C, you are ready for Phase 3 (drop B3 emission). If 50% of traffic is none (no propagation header at all), you have a propagation bug at an upstream service — that is the signal to look for next, before it costs you a debugging session at 02:40 AM.

A second instrumentation hygiene rule: emit a metric called incoming_propagation_format with a label like format=w3c|b3-1h|b3-mh|none and increment it on every received request. This single metric, plotted as a stacked time series, makes migration progress visible at a glance and catches regressions immediately — when a deploy of one service drops the W3C propagator and reverts to B3-only, the stacked area chart shifts colour the moment the deploy lands. Razorpay's observability platform team added exactly this metric in late 2024 and credited it with cutting their migration debugging time in half. The metric costs almost nothing (one counter, four label values, no high-cardinality risk) and is the kind of self-instrumented telemetry that the next chapter on Tempo / Jaeger / Zipkin assumes you have.

The next chapter sits one layer up. Once you know how a traceparent flows on the wire, the question becomes: which backend should receive it? Zipkin (B3-native, simple, file-backed), Jaeger (B3 and OTLP, Cassandra/Elasticsearch), or Tempo (OTLP-only, columnar, Grafana-tied)? Each made different choices about indexing, retention, and query — and the right answer depends on your read pattern. That is what chapter 24 covers.

References

# Reproduce this on your laptop
python3 -m venv .venv && source .venv/bin/activate
pip install flask requests
python3 multi_format_propagator.py
# Expected: two cases. Case A — incoming traceparent only — both spans
# show via=w3c. Case B — incoming X-B3-* only — gateway shows via=b3-mh,
# legacy_payments shows via=w3c (because the gateway wrote both formats
# outbound, and the receiver picked W3C in the priority order). Trace IDs
# stitch across both cases — pid of one hop matches sid of the previous.
#
# Extension exercise (do this yourself): comment out the `inject()` writer
# for B3 (remove the b3 / X-B3-* keys). Now Case B will show that the
# gateway parsed b3-mh on input but downstream legacy_payments cannot find
# any header it understands — it mints a fresh trace_id and the tree forks.
# This is exactly the silent-fork bug from the Razorpay 2024 retro, in
# 30 seconds on your laptop.