Network, Service & Security Observability for Kubernetes

• What is Hubble?
• Getting Started
• Features
  • Service Dependency Graph
  • Metrics & Monitoring
  • Flow Visibility
• Get in touch / Community
• Authors

What is Hubble?

Hubble is a fully distributed networking and security observability platformfor cloud native workloads. It is built on top of Cilium and eBPF to enabledeep visibility into the communication and behavior of services as well as thenetworking infrastructure in a completely transparent manner.

Hubble can answer questions such as:

Service dependencies & communication map:

• What services are communicating with each other? How frequently? What doesthe service dependency graph look like?
• What HTTP calls are being made? What Kafka topics does a service consumefrom or produce to?

Operational monitoring & alerting:

• Is any network communication failing? Why is communication failing? Is itDNS? Is it an application or network problem? Is the communication broken onlayer 4 (TCP) or layer 7 (HTTP)?
• Which services have experienced a DNS resolution problems in the last 5minutes? Which services have experienced an interrupted TCP connectionrecently or have seen connections timing out? What is the rate of unansweredTCP SYN requests?

Application monitoring:

• What is the rate of 5xx or 4xx HTTP response codes for a particular serviceor across all clusters?
• What is the 95th and 99th percentile latency between HTTP requests andresponses in my cluster? Which services are performing the worst? What isthe latency between two services?

Security observability:

• Which services had connections blocked due to network policy? What serviceshave been accessed from outside the cluster? Which services have resolved aparticular DNS name?

Why Hubble?

The Linux kernel technology eBPF is enabling visibility into systems andapplications at a granularity and efficiency that was not possible before. Itdoes so in a completely transparent way, without requiring the application tochange or for the application to hide information. By building on top ofCilium, Hubble can leverage eBPF for visibility. By leveraging eBPF, allvisibility is programmable and allows for a dynamic approach that minimizesoverhead while providing deep and detailed insight where required. Hubble hasbeen created and specifically designed to make best use of these new eBPFpowers.

Releases

Since the release of v0.8, the Hubble CLI is backward compatible with allsupported Cilium releases. For this reason, only the latest Hubble CLI versionis maintained.

Component Stability

Hubble project consists of several components (see Architecture section).

While the core Hubble components have been running in production in multipleenvironments, new components continue to emerge as the project grows andexpands in scope.

Some components, due to their relatively young age, are still considered betaand have to be used with caution in critical production workloads.

Architecture

Getting Started

• Introduction to Cilium & Hubble
• Networking and Security Observability with Hubble

Features

Service Dependency Graph

Troubleshooting microservices application connectivity is a challenging task.Simply looking at "kubectl get pods" does not indicate dependencies betweeneach service or external APIs or databases.

Hubble enables zero-effort automatic discovery of the service dependency graphfor Kubernetes Clusters at L3/L4 and even L7, allowing user-friendlyvisualization and filtering of those dataflows as a Service Map.

See Hubble Service Map Tutorialfor more examples.

Metrics & Monitoring

The metrics and monitoring functionality provides an overview of the state ofsystems and allow to recognize patterns indicating failure and other scenariosthat require action. The following is a short list of example metrics, for amore detailed list of examples, see theMetrics Documentation.

Networking Behavior

Network Policy Observation

HTTP Request/Response Rate & Latency

DNS Request/Response Monitoring

Flow Visibility

Flow visibility provides visibility into flow information on the network andapplication protocol level. This enables visibility into individual TCPconnections, DNS queries, HTTP requests, Kafka communication, and much more.

DNS Resolution

Identifying pods which have received DNS response indicating failure:

hubble observe --since=1m -t l7 -j \
   | jq 'select(.l7.dns.rcode==3) | .destination.namespace + "/" + .destination.pod_name' \
   | sort | uniq -c | sort -r
  42 "starwars/jar-jar-binks-6f5847c97c-qmggv"

Successful query & response:

starwars/x-wing-bd86d75c5-njv8k            kube-system/coredns-5c98db65d4-twwdg      DNS Query deathstar.starwars.svc.cluster.local. A
kube-system/coredns-5c98db65d4-twwdg       starwars/x-wing-bd86d75c5-njv8k           DNS Answer "10.110.126.213" TTL: 3 (Query deathstar.starwars.svc.cluster.local. A)

Non-existent domain:

starwars/jar-jar-binks-789c4b695d-ltrzm    kube-system/coredns-5c98db65d4-f4m8n      DNS Query unknown-galaxy.svc.cluster.local. A
starwars/jar-jar-binks-789c4b695d-ltrzm    kube-system/coredns-5c98db65d4-f4m8n      DNS Query unknown-galaxy.svc.cluster.local. AAAA
kube-system/coredns-5c98db65d4-twwdg       starwars/jar-jar-binks-789c4b695d-ltrzm   DNS Answer RCode: Non-Existent Domain TTL: 4294967295 (Query unknown-galaxy.starwars.svc.cluster.local. A)
kube-system/coredns-5c98db65d4-twwdg       starwars/jar-jar-binks-789c4b695d-ltrzm   DNS Answer RCode: Non-Existent Domain TTL: 4294967295 (Query unknown-galaxy.starwars.svc.cluster.local. AAAA)

HTTP Protocol

Successful request & response with latency information:

starwars/x-wing-bd86d75c5-njv8k:53410      starwars/deathstar-695d8f7ddc-lvj84:80    HTTP/1.1 GET http://deathstar/
starwars/deathstar-695d8f7ddc-lvj84:80     starwars/x-wing-bd86d75c5-njv8k:53410     HTTP/1.1 200 1ms (GET http://deathstar/)

TCP/UDP Packets

Successful TCP connection:

starwars/x-wing-bd86d75c5-njv8k:53410      starwars/deathstar-695d8f7ddc-lvj84:80    TCP Flags: SYN
deathstar.starwars.svc.cluster.local:80    starwars/x-wing-bd86d75c5-njv8k:53410     TCP Flags: SYN, ACK
starwars/x-wing-bd86d75c5-njv8k:53410      starwars/deathstar-695d8f7ddc-lvj84:80    TCP Flags: ACK, FIN
deathstar.starwars.svc.cluster.local:80    starwars/x-wing-bd86d75c5-njv8k:53410     TCP Flags: ACK, FIN

Connection timeout:

starwars/r2d2-6694d57947-xwhtz:60948   deathstar.starwars.svc.cluster.local:8080     TCP Flags: SYN
starwars/r2d2-6694d57947-xwhtz:60948   deathstar.starwars.svc.cluster.local:8080     TCP Flags: SYN
starwars/r2d2-6694d57947-xwhtz:60948   deathstar.starwars.svc.cluster.local:8080     TCP Flags: SYN

Network Policy Behavior

Denied connection attempt:

starwars/enterprise-5775b56c4b-thtwl:37800   starwars/deathstar-695d8f7ddc-lvj84:80(http)   Policy denied (L3)   TCP Flags: SYN
starwars/enterprise-5775b56c4b-thtwl:37800   starwars/deathstar-695d8f7ddc-lvj84:80(http)   Policy denied (L3)   TCP Flags: SYN
starwars/enterprise-5775b56c4b-thtwl:37800   starwars/deathstar-695d8f7ddc-lvj84:80(http)   Policy denied (L3)   TCP Flags: SYN

Community

Join the Cilium Slack #hubble channel to chatwith Cilium Hubble developers and other Cilium / Hubble users. This is a goodplace to learn about Hubble and Cilium, ask questions, and share yourexperiences.

Learn more about Cilium.

Authors

Hubble is an open source project licensed under the Apache License. Everybodyis welcome to contribute. The project is following the Governance Rules ofthe Cilium project. See CONTRIBUTING for instructions on how to contributeand details of the Code of Conduct.