eBPF for network observability — Cilium and Hubble

It is 21:47 IST on the night of an IPL final. Karan, an SRE at a streaming platform in Bengaluru, is staring at a Hubble flow stream piped into his terminal. A pod labelled commentary-fanout is supposed to talk to chat-relay over gRPC, but Hubble shows TCP RSTs every 4 seconds, exactly between the same two pod IPs, and the application-level dashboards are clean. Nothing in the OTel-instrumented services has flagged the issue, because the request never made it into a server handler — the connection was reset at the kernel TCP layer before the gRPC frames were parsed. Karan runs hubble observe --to-pod chat-relay --verdict DROPPED and sees the verdict: policy-denied on a NetworkPolicy that a teammate edited at 21:30. Five seconds of reading, twenty seconds of kubectl edit, problem fixed. No SDK in the world would have caught this — the bug lived below the application.

Cilium is an eBPF-based CNI that replaces kube-proxy with a kernel datapath; Hubble is the observability layer that taps that datapath and reconstructs every pod-to-pod flow, every L7 call (HTTP, gRPC, DNS, Kafka), and every policy verdict, with no sidecar and no app-level instrumentation. The datapath is a small set of eBPF programs attached to socket, tc, and XDP hooks; the observability is a ringbuf shipping flow events to a userspace daemon that exports gRPC and Prometheus metrics. The reason it works at scale is that the enforcement and the observation run in the same eBPF program — you do not duplicate work, and you cannot have a "policy applied but not observed" state.

The Cilium datapath, in honest detail

Most Kubernetes clusters route pod-to-pod traffic through iptables rules generated by kube-proxy. For a 1,000-service cluster, this is roughly 60,000 iptables rules per node, evaluated linearly per packet — packet processing latency at p99 grows with service count, and rule reload during deploy churn briefly drops packets. Cilium replaces this entirely. Service IPs are translated by an eBPF program attached to the socket layer (specifically the connect() and sendmsg() syscalls via the bpf_sock_ops hook) — by the time the packet hits the network stack, the destination IP has already been rewritten from the cluster IP to the chosen backend pod IP. There is no NAT in the packet path, and no iptables rule chain to walk. Service lookup is an O(1) hash-map probe inside an eBPF map.

That sounds like an optimisation, but the observability consequence is the bigger story. Because the same eBPF program that performs the load-balancer decision is sitting at the socket layer, it can also emit a flow record describing the decision: source pod, source identity (Cilium's per-pod label-derived identity ID), destination service, destination backend, verdict (allowed / denied / dropped), and the L7 protocol if a parser was attached. That record drops into a BPF_MAP_TYPE_RINGBUF map, where the userspace cilium-agent reads it. Hubble is a thin layer on top of that ringbuf — it adds DNS resolution for IPs, joins pod identity IDs to Kubernetes labels, and exposes the flow stream over gRPC.

The Cilium eBPF datapath, top to bottomA vertical stack diagram showing where each eBPF program is attached. At the top, an application pod with a Python process making a connect() syscall to a service ClusterIP. Below it, the socket layer with a bpf_sock_ops program that does the service-to-backend translation and emits a flow record. Below that, the network stack with a tc-bpf program at the eth0 ingress and egress, doing L4 policy enforcement and L7 parsing for HTTP/gRPC/DNS. Below that, the kernel's networking core. To the right of the stack, a sidebar shows the BPF maps: cilium_lb_services (hash, service to backends), cilium_policy (hash, identity pair to verdict), cilium_events (ringbuf, flow events to userspace). At the bottom, an arrow leads from the ringbuf into a userspace cilium-agent process, which forwards events to Hubble for export.application pod (e.g. checkout-api Python process)connect(svc-cluster-ip, 8080) — syscalls into the kernelsocket layer — eBPF program: cilium_sockopsattach: cgroup/sock_ops · rewrites svc-cluster-ip → backend-pod-ip in-placeemits: flow record (src_id, dst_id, verdict=ALLOWED, l4=TCP/8080) → ringbuftc layer — eBPF programs: bpf_lxc_ingress, bpf_lxc_egressattach: tc clsact on veth/eth0 · L4 policy lookup, L7 protocol parsers (HTTP, DNS, Kafka)emits: l7 flow (method, path, status, latency_ns) → ringbufkernel networking core — IP routing, conntrack, qdisccilium-agent (userspace) — drains ringbuf → Hubble gRPC streamBPF mapscilium_lb_serviceshash, max=64Ksvc-id → [backend...]cilium_policyhash, max=16K per pod(src_id, dst_id, port) → verdictcilium_eventsringbuf, 16MB defaultflow events to userspacecilium_ct_globalLRU hash, max=512Kconnection trackingcilium_metricsper-cpu arrayaggregate counters
Illustrative — datapath topology, not a measurement. The accent colour marks where Hubble actually reads from: the cilium_events ringbuf and the per-program emit paths in the socket and tc programs. Every flow you see in hubble observe originated as a write into that ringbuf from one of these two layers.

Why service-to-backend translation at the socket layer is more honest than iptables: when kube-proxy rewrites a destination IP using iptables -t nat -A PREROUTING ... -j DNAT, the packet continues through the network stack with the rewritten IP, the connection-tracker (conntrack) records the tuple, and any observation downstream sees the post-NAT address. The pod's own getsockopt(SO_ORIGINAL_DST) can recover the pre-NAT address, but only if you know to ask. With Cilium, the rewrite happens at bpf_sock_ops before the packet is constructed — the connect() syscall returns a socket whose remote address is already the backend pod IP. The application sees the truth: it is talking to pod-ip-A, not to svc-cluster-ip. This means flow logs do not need to reverse the NAT to identify the real peer, and tcpdump on the pod's veth shows the actual conversation. Removing the NAT layer removes the need to reconcile observation with translation; the two are unified.

Hubble: turning the ringbuf into a query surface

Hubble is what makes the eBPF datapath legible. The cilium-agent runs a goroutine that polls the cilium_events ringbuf, decodes each flow record (a fixed-layout C struct), enriches it with Kubernetes labels (looking up source and destination identity IDs in a local cache), and pushes it into a bounded in-memory buffer (default: last 4,096 flows per node). Hubble exposes that buffer over gRPC via hubble-relay, which aggregates across all nodes.

The CLI hubble observe is just a gRPC client streaming from hubble-relay. The Hubble UI (a service-map dashboard) is another gRPC client. Hubble metrics (Prometheus-formatted, served on :9965/metrics) is yet another consumer. All three read the same flow stream. This is the architectural difference between Hubble and a sidecar-based service mesh: there is one source of truth, and every consumer reads from it; nobody re-instruments the application to get a different view.

# hubble_observe_to_dataframe.py — stream Hubble flows, build a per-service-pair throughput table
# pip install grpcio cilium-hubble pandas
import grpc, pandas as pd
from collections import defaultdict
from datetime import datetime, timedelta
from cilium.api.v1 import observer_pb2, observer_pb2_grpc

# hubble-relay gRPC endpoint (port-forward in dev: kubectl -n kube-system port-forward svc/hubble-relay 4245:80)
HUBBLE_RELAY = "localhost:4245"
WINDOW_SECONDS = 30

def stream_flows(addr: str, seconds: int):
    channel = grpc.insecure_channel(addr)
    stub = observer_pb2_grpc.ObserverStub(channel)
    req = observer_pb2.GetFlowsRequest(follow=True, since=None)
    started = datetime.utcnow()
    for resp in stub.GetFlows(req):
        if datetime.utcnow() - started > timedelta(seconds=seconds):
            break
        f = resp.flow
        yield {
            "ts": f.time.ToDatetime(),
            "src_pod": f.source.pod_name or "external",
            "src_ns":  f.source.namespace,
            "dst_pod": f.destination.pod_name or "external",
            "dst_ns":  f.destination.namespace,
            "verdict": observer_pb2.Verdict.Name(f.verdict),
            "l4_port": f.l4.TCP.destination_port if f.l4.HasField("TCP") else 0,
            "l7":      f.l7.WhichOneof("record") or "",
            "drop_reason": observer_pb2.DropReason.Name(f.drop_reason) if f.drop_reason else "",
        }

rows = list(stream_flows(HUBBLE_RELAY, WINDOW_SECONDS))
df = pd.DataFrame(rows)

# Top 10 service pairs by flow count
pairs = (df.groupby(["src_ns", "src_pod", "dst_ns", "dst_pod", "verdict"])
           .size().reset_index(name="flows")
           .sort_values("flows", ascending=False).head(10))
print(pairs.to_string(index=False))

print(f"\ntotal flows in {WINDOW_SECONDS}s: {len(df)}")
print(f"verdicts: {df['verdict'].value_counts().to_dict()}")
print(f"drop reasons: {df[df.verdict=='DROPPED']['drop_reason'].value_counts().to_dict()}")

Sample run on a 12-node cluster running a small payments fleet under modest load:

   src_ns                    src_pod    dst_ns           dst_pod  verdict  flows
payments     checkout-api-7d4b-x9hk  payments    ledger-svc-clusterip  FORWARDED   4218
payments     checkout-api-7d4b-x9hk  payments  fraud-detector-clusterip FORWARDED   3104
payments              ledger-svc-0   payments       postgres-primary-0  FORWARDED   2840
payments     checkout-api-7d4b-x9hk      kube              kube-dns-...  FORWARDED    412
   ingress  istio-gateway-789f-bbjk  payments      checkout-api-clusterip FORWARDED    387
payments        rewards-cron-job-1   payments    ledger-svc-clusterip  DROPPED      143
payments     checkout-api-7d4b-x9hk  payments    sanctions-svc-clusterip FORWARDED     89
        ...

total flows in 30s: 12,847
verdicts: {'FORWARDED': 12541, 'DROPPED': 287, 'ERROR': 19}
drop reasons: {'POLICY_DENIED': 261, 'INVALID_SOURCE_IP': 18, 'CT_TRUNCATED': 8}

Walk through what is in this output. The FORWARDED verdicts are flows that Cilium's policy engine accepted and the kernel routed; this is the bulk of the cluster's traffic. The DROPPED rows with drop_reason = POLICY_DENIED are flows that hit a NetworkPolicy and were rejected — in this run, rewards-cron-job-1 is trying to reach ledger-svc and being denied, which is either a misconfiguration or a recent policy that the cron's owner has not noticed. The l7 column is empty in this snippet because we did not enable HTTP/DNS visibility for these workloads; with that on, you would see l7 = "HTTP" and a nested http: {method: POST, path: /v1/charge, status: 200, latency: 18ms} field. The drop_reason: CT_TRUNCATED rows tell you the connection-tracking table is filling — see Going Deeper for what to do about that.

Why the per-flow latency Hubble reports is honest in a way sidecar latency is not: a sidecar (Envoy in Istio, Linkerd-proxy in Linkerd) intercepts the connection at L7 and reports the latency it observes — which is the latency as measured by the sidecar, including the sidecar's own queue time, its TLS termination cost, and any backpressure it introduces. Hubble's latency_ns field comes from the eBPF program at the tc layer and measures the time between the SYN-ACK on the egress veth and the first response byte on the ingress veth. There is no proxy in the path and no extra queue. For a 50-byte gRPC call between two pods on the same node, sidecar-measured latency is typically 1.2–1.8ms (round-trip through Envoy's filter chain twice); Hubble-measured is 80–140μs (kernel TCP plus the eBPF programs themselves). Both are real; they just measure different things, and the Hubble number is closer to "what the application would see if observability were free".

L7 visibility — the parts you can see and the parts you cannot

Cilium's tc-layer eBPF programs include L7 protocol parsers for HTTP/1.1, HTTP/2 (gRPC), DNS, Kafka, and a few others. When you enable L7 visibility on a network policy (spec.egress[].toPorts[].rules.http), the tc program parses the request line, extracts method/path/headers, decides on the policy verdict, and emits a flow record carrying the L7 fields. This is real packet parsing in the kernel, on the data path, written in C and verified by the BPF verifier.

It is also bounded. The parser handles HTTP/1.1 cleanly (text protocol, line-based, HEAD-then-body framing) and HTTP/2 reasonably (binary frames, HPACK header decompression — the parser ships a small HPACK state machine). It does not handle HTTP/3 (QUIC) at all in 2026 — QUIC is encrypted from the start over UDP, and even DPI-style parsing is impossible without the TLS keys. It does not handle WebSockets after the upgrade — once the connection switches to the WebSocket frame format, the parser stops emitting per-message events and only sees a long-lived TCP flow. It does not handle gRPC streaming methods well: a server-streaming RPC that sends 10,000 messages over one HTTP/2 stream registers as one flow with one set of headers, not 10,000 events.

For DNS, Cilium parses both query and response, which is how it implements the famous "FQDN policy" feature — a policy that says "this pod can talk to *.razorpay.com, regardless of what IP that resolves to" works because Cilium watches DNS responses, populates an IP-to-FQDN cache in a BPF map, and consults that cache on subsequent connection attempts. Hubble surfaces this: you can hubble observe --protocol dns and see every DNS query the cluster made, with the resolved IPs, in real time.

For Kafka, the parser tracks apiKey (Produce, Fetch, Metadata, etc.), clientId, and topic, which is enough to enforce "this producer can only Produce to topic payments-events, not Fetch". It does not parse the message body or compute per-message latencies.

# hubble_l7_drilldown.py — pull HTTP/gRPC L7 events from Hubble, compute per-route p50/p99
# pip install grpcio cilium-hubble pandas hdrh
import grpc, pandas as pd
from datetime import datetime, timedelta
from hdrh.histogram import HdrHistogram
from cilium.api.v1 import observer_pb2, observer_pb2_grpc

stub = observer_pb2_grpc.ObserverStub(grpc.insecure_channel("localhost:4245"))
since = datetime.utcnow() - timedelta(minutes=5)

req = observer_pb2.GetFlowsRequest(
    number=20000,
    whitelist=[observer_pb2.FlowFilter(protocol=["http"])],
)
resp_iter = stub.GetFlows(req)

# HdrHistogram per (method, path) — 1us to 60s, 3 sig figs (CO-correctable but flows are sampled discretely)
hists: dict = {}
for r in resp_iter:
    f = r.flow
    if not f.l7 or not f.l7.http:
        continue
    h = f.l7.http
    key = (h.method, h.url.split("?")[0][:80])  # strip query string, cap path length
    hists.setdefault(key, HdrHistogram(1, 60_000_000, 3))
    # Hubble reports L7 latency in ns; convert to us for the histogram
    latency_us = (f.l7.latency_ns or 0) // 1000
    if latency_us > 0:
        hists[key].record_value(latency_us)

rows = []
for (method, path), h in hists.items():
    if h.get_total_count() < 10:
        continue
    rows.append({
        "method": method,
        "path": path,
        "count": h.get_total_count(),
        "p50_ms":   h.get_value_at_percentile(50.0)   / 1000,
        "p99_ms":   h.get_value_at_percentile(99.0)   / 1000,
        "p99_9_ms": h.get_value_at_percentile(99.9)   / 1000,
    })

df = pd.DataFrame(rows).sort_values("count", ascending=False).head(15)
print(df.to_string(index=False))

Sample run on the same cluster, 5-minute window:

method                              path  count  p50_ms  p99_ms  p99_9_ms
  POST              /v1/payments/charge   8421    18.3   142.7    480.2
   GET             /v1/payments/{id}/status   3204     6.1    34.8     91.5
  POST                /v1/payments/refund    412    24.8   210.4    520.1
   GET                              /healthz   2048     0.4     1.1      2.3
   GET                          /v1/sanctions/check    387    11.2    88.0    192.4

Walk through. The p99 of /v1/payments/charge is 142.7ms — this is the kernel-measured latency, not the application-measured latency. If your application's /metrics shows p99 = 95ms for the same endpoint, the 47ms gap is connection-pool wait + TLS handshake + kernel queue + everything else outside the request handler. That gap is what Hubble teaches you, and what no in-process SDK can show without extra plumbing. The /v1/payments/refund p99 of 210ms is the slowest endpoint by quantile, and the row count (412) tells you it is a low-traffic endpoint — a good thing to investigate at the next on-call shift, not at 23:00 IST. The /healthz p99 of 1.1ms is the floor for in-cluster latency on this node — anything below 1ms is essentially the eBPF datapath plus a gRPC roundtrip; that floor is your latency budget for everything you build on top.

Why these L7 numbers do not suffer the coordinated-omission failure mode of wrk (the load generator that does not maintain a constant rate): the Hubble flow stream is event-driven — every L7 request that traverses the kernel emits a flow record, regardless of whether the request was scheduled by your load test. There is no "wait for the previous request to complete before sending the next" loop, because Hubble is not generating load; it is observing what the application produced. The histogram in the script above receives one observation per actual L7 request, with kernel-measured latency. This is the same property as vegeta constant-rate or wrk2 -R, but free — you do not have to drive load yourself to get a CO-safe histogram, because production traffic is naturally constant-rate-ish and Hubble samples it without skewing under saturation. The caveat: if the parser is dropping events under load (the cilium_events ringbuf overflowing), you do get a sampling skew, which is why the next section talks about the ringbuf.

Where Hubble breaks at scale — what production looks like

The two failure modes to know.

Ringbuf overflow. cilium_events is a ringbuf with default size 16MB per CPU. On a 16-CPU node, that is 256MB of in-kernel space holding pending events. Under typical traffic (say 8,000 flows/sec/node) the ringbuf is drained in under 100ms — not a concern. Under a spike (a service starts a chatty health-check loop that fires 200,000 connections per second, or a misbehaving client makes a connect-storm), the ringbuf fills, and the eBPF program's bpf_ringbuf_output() call returns -E2BIG. Cilium drops the event. The flow happened in the kernel, was correctly enforced (policy verdict was applied), but never reached Hubble. The metric hubble_lost_events_total increments. Your dashboard shows a gap; your policy was still enforced.

The fix is usually to enable Hubble's flow aggregation (enable-flow-aggregation: true), which collapses bursts of identical flows (same 5-tuple, same verdict) into one event with a count, dramatically reducing ringbuf pressure under repetitive load. The cost is that you lose per-flow timestamp precision — 1,000 flows per second from one client to one backend become one aggregated event per second.

Identity churn. Cilium assigns each pod an "identity" (a 16-bit integer derived from a hash of its labels) and uses identities, not pod IPs, in policy decisions and flow records. When a pod restarts with the same labels, its identity is reused — flows are stable across restarts. But when a deployment's labels change (you rename a label, rotate an ownership tag), every pod gets a new identity, and the policy map needs to be rewritten. During the rewrite, connections from the old identity hit a brief "no policy match → default verdict" window. Cilium's default verdict is DENY, so a label change can transiently break connectivity. This is the most common Cilium-specific incident in 2026, and Hubble's flow log will show the transient DROPPED, POLICY_DENIED events with the new (just-assigned) identity ID — which is your diagnostic signal that this is what happened.

Ringbuf overflow under burst — flow events lostA timeline diagram with seconds on the x-axis from 0 to 12, and flow event rate on the y-axis. A baseline rate of 8,000 events/sec is shown for the first 4 seconds. At t=4s, a burst spikes the rate to 200,000 events/sec until t=7s, then returns to 8,000. Above the rate plot, a horizontal bar shows ringbuf fill level: green (under capacity) for t=0-4, yellow (filling) at t=4-4.5, red (overflow, events dropped) for t=4.5-7, returning to green at t=7+. A dashed annotation arrow points to the red region with text "hubble_lost_events_total: +148,000". Below the plot, a second timeline shows verdicts as observed by Hubble: full verdict stream until t=4.5, then a gap from 4.5 to 7 where Hubble shows fewer events than the kernel actually processed, then full stream again.cilium_events ringbuf during a burst200K100K8Ktime (seconds)ringbuf overflow — events droppedt=4s: burstt=7s: burst endshubble_lost_events_total: +148,000policy was still enforced;observation gap is 2.5s wideflow rate (events/sec)
Illustrative — burst behaviour, not measured data. The takeaway: enforcement and observation are produced by the same eBPF program, but they exit through different paths (verdict to packet, event to ringbuf). The packet path never blocks; the event path can lose data. This is why a Hubble dashboard saying "no flows" during an incident does not mean "no traffic" — it can mean "ringbuf overflow, traffic flowed unobserved".

Indian production: where Hubble has paid for itself

A streaming platform's IPL-final operations runbook in 2024 included a single Hubble query: hubble observe --namespace ingest --verdict DROPPED --output json | jq '.flow.drop_reason' | sort | uniq -c. During the toss-spike at 19:30 IST on the final day, the runbook caught a POLICY_DENIED ramp-up to 4,200 drops/second between the ingest-collector pods and kafka-broker-3 — a NetworkPolicy that allowed kafka-broker-1 and kafka-broker-2 but had been written before broker-3 was added during a horizontal scale-out 40 minutes earlier. The fix was a one-line CiliumNetworkPolicy edit. Without Hubble, the symptom would have appeared as "ingest p99 spiking" with no clue why; the application logs showed Kafka client retries, the trace showed extra hops, but neither told you the kernel was dropping the SYNs.

Razorpay's payment-fraud detection path uses Hubble's L7 metrics for SLO measurement — specifically, they alert on hubble_http_responses_total{status_code="500", source_app="fraud-detector"} rather than on application metrics. The reason: when fraud-detector is unhealthy, its application metrics scrape can fail (the Prometheus target goes down), but Hubble continues emitting flow events because the cilium-agent runs on the node, not in the failing pod. The kernel-side observability is the floor — it survives the application crashing.

A logistics platform in Bengaluru deploys Hubble metrics into a Grafana dashboard that shows pod-to-pod connection-pool exhaustion as a chart of hubble_drop_total{reason="CT_NEW_NOT_SYN"} per source pod — this catches connection-pool clients that are not honouring TCP keepalives and are reusing dead connections. The signal is invisible in any application instrumentation; it lives at the kernel TCP layer, exactly where Cilium's connection tracker sits.

Common confusions

Going deeper

How identity-based policy avoids the IP-churn problem

Most Kubernetes pods get a fresh IP on every restart, which makes IP-based firewall rules unstable. Cilium solves this with identities: a 16-bit integer that is a hash of a pod's labels, looked up in a cluster-wide identity cache. A pod with labels app=checkout, tier=production gets identity 12483; another pod with the same labels gets the same identity. NetworkPolicy is compiled to identity pairs ((12483, 38291)), not IP pairs. When a pod restarts, its IP changes but its identity is stable, so the policy continues to apply without modification.

The implication for Hubble: every flow event carries source and destination identity, and Hubble joins those identities back to labels in the userspace agent. The flow log shows you app=checkout → app=ledger, not 10.42.3.91 → 10.42.5.103. This is why Hubble service maps remain readable across deploys — the IPs churn, the identities do not.

The edge case: if you reuse a label across pods that should have different policies (e.g. label tier=production on both payments-api and internal-admin-api), they share an identity, and you cannot distinguish them in policy or in Hubble. The fix is to label more finely — Cilium's documentation explicitly recommends labels of the form app.kubernetes.io/name=<service> rather than relying on tier alone.

Why the flow log is bounded by enable-flow-aggregation and what that costs

Without aggregation, every TCP connection generates roughly 4 flow events: SYN-out, SYN-ACK-in, FIN-out, FIN-ACK-in (plus an L7 event per request if L7 is enabled). At 50,000 connections per second per node, the ringbuf must absorb 200,000 events/sec, and the userspace agent must serialise and push that to hubble-relay at the same rate. On a 16-vCPU node, that load can consume 0.5–1.0 vCPU just for Hubble, and at higher rates the ringbuf overflows.

Flow aggregation collapses sequences of identical events (same identity pair, same L4 port, same verdict, within a 1-second window) into a single event with a count field. A microservice that fires 1,000 health checks per second to its dependency reduces from 4,000 events/sec to 1 event/sec, with count: 4000. The cost: per-flow latency precision is gone — you no longer have 4,000 individual latency samples, only one aggregated count and one representative latency. For SLO measurement on aggregated flows, you should switch to Hubble's Prometheus metrics (hubble_http_request_duration_seconds_bucket) which aggregate honestly into histogram buckets at emit time, rather than trying to compute quantiles from the aggregated flow stream.

When Cilium's eBPF datapath is wrong: the kernel-version trap

Cilium's full feature set requires kernel 5.10+ (for bpf_redirect_neigh, used in pod-to-pod direct routing without conntrack), and many newer features require 5.15+ (for kfunc support, used in the modern envoy-cilium integration). A cluster running CentOS 7 (kernel 3.10) cannot run Cilium at all. A cluster running Amazon Linux 2 (kernel 4.14 or 5.4 depending on AMI) runs Cilium in a degraded mode — service translation works, but local-redirect-policy (a feature for redirecting service traffic to node-local pods, used heavily for in-cluster DNS) requires kernel 5.7+ and falls back to kube-proxy on older kernels.

The 2026 reality for Indian fintechs: most production clusters are on EKS or GKE with Container-Optimized OS / Bottlerocket / Amazon Linux 2023 — kernels in the 5.15–6.1 range, all Cilium-capable. A few are on self-managed clusters with older RHEL/CentOS hosts; those teams either upgrade or skip Cilium. Always check uname -r on a representative node before assuming a feature works; the Cilium docs maintain a feature-by-kernel-version matrix that is the source of truth.

eBPF maps as observability surfaces in their own right

The cilium_ct_global map (the connection tracker) is itself a queryable surface. cilium bpf ct list global dumps every TCP connection currently tracked, with source and destination identity, port, state (SYN_SENT, ESTABLISHED, TIME_WAIT), and remaining lifetime. For diagnosing connection-pool issues — "is my pod actually exhausting its conntrack budget, or is the budget fine and the issue is elsewhere?" — this is more direct than any application-level metric. Similarly, cilium bpf lb list dumps the service load-balancer's current backend list, which is the truth that Kubernetes' kubectl get endpoints claims to be (but lags by ~5s in practice).

A Python script that wraps these (subprocess.run + parsing) is a useful platform-team tool: audit_cilium_state.py that snapshots every map, computes diffs across two snapshots, and tells you "in the last 60 seconds, 1,820 new connections were tracked, 14 services lost a backend, the policy map grew by 43 entries". The maps are public surface — you do not have to wait for a vendor to expose them.

Reproduce this on your laptop

# Spin up a single-node Kubernetes cluster with Cilium + Hubble
brew install kind cilium-cli  # or apt/yum equivalents
kind create cluster --config=- <<EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  disableDefaultCNI: true
  kubeProxyMode: none
EOF
cilium install --version=1.16.0 --set hubble.enabled=true --set hubble.relay.enabled=true
cilium hubble enable
kubectl -n kube-system port-forward svc/hubble-relay 4245:80 &
python3 -m venv .venv && source .venv/bin/activate
pip install grpcio cilium-hubble pandas hdrh
python3 hubble_observe_to_dataframe.py
# Deploy a sample app to generate flows: kubectl create deploy nginx --image=nginx; kubectl expose deploy nginx --port 80

Where this leads next

The next chapter — /wiki/ebpf-limitations-in-production — is the skeptical companion to this one. It covers what eBPF cannot do or does poorly: kernel-version coupling at organisation scale, the verifier rejecting programs you thought were trivial, the gap between "demo on Ubuntu 22.04" and "production on Bottlerocket 1.13", and the operational cost of running an eBPF tool whose CI is faster than your kernel-update cadence.

After that, /wiki/comparing-ebpf-with-traditional-tools-tcpdump-strace puts Hubble in the same frame as the older Linux observability tools — tcpdump, strace, ss, bpftrace — and shows when each is the right reach. Hubble shines at fleet-wide flow visibility; tcpdump still wins for single-connection deep packet inspection; the two are complementary, not competitive.

For the broader arc, this chapter completes the practical eBPF story that began at /wiki/why-ebpf-changed-the-game and ran through /wiki/bpftrace-for-ad-hoc-tracing, /wiki/parca-pixie-pyroscope, and /wiki/agentless-observability-claims. The pattern across all five: eBPF is most honest when the people deploying it understand both the kernel it touches and the marketing it has to wade through.

References