Cilium and Hubble¶

Cilium is the CNI (Container Network Interface): it gives every pod an IP address, routes traffic, and enforces network policy using eBPF programs in the Linux kernel. Hubble is its observability layer: it records every flow along with the identity (namespace and labels) of both ends and the verdict (forwarded or dropped).

K8TRE mandates Cilium; KID installs it with the Cilium CLI in bootstrap.sh (see CILIUM_FLAGS).

Identity-based policy¶

Traditional firewalls match IP addresses. Pod IPs change every time a pod restarts, so Cilium gives each set of labels a numeric security identity and matches on that. Rules say "pods labelled component=singleuser-server may reach pods labelled component=hub in namespace jupyterhub on 8081" and stay correct however often pods come and go.

The rules¶

Once any policy selects a pod for a direction (ingress or egress), everything not explicitly allowed in that direction is dropped.

Policy	Where	Source	Effect
`workspace-baseline`	each `project-*`	`gitops/projects/chart/templates/networkpolicy.yaml`	Workspaces: ingress only from hub/proxy on 8888; egress only to DNS and hub:8081
`default-deny`	each `project-*`	generated by Kyverno (`gitops/kyverno/generate-default-deny.yaml`)	Every pod: deny all, both directions
`internet-allowlist`	projects that set `internetAllowlist`	`gitops/projects/chart/templates/internet-allowlist.yaml`	Workspaces: HTTPS to named FQDNs only
`jupyterhub-baseline`	`jupyterhub`	`gitops/network-policies/jupyterhub.yaml`	Hub/proxy: DNS, API server, Keycloak, workspaces; ingress from Traefik and workspaces

Policies are additive: a flow is allowed if any policy selecting the pod allows it. That is why default-deny and workspace-baseline can sit side by side.

DNS-aware rules¶

The workspace egress rule for DNS has an L7 dns section:

rules:
  dns:
    - matchPattern: "*"

This sends DNS lookups through Cilium's DNS proxy. Two things follow:

Hubble can show which names a workspace looked up, not just that it talked to port 53.
toFQDNs rules work: Cilium learns which IPs pypi.org resolved to and allows exactly those (lab 4).

Useful commands¶

cilium status                                  # health of agent, operator, Hubble
kubectl get cnp -A                             # all CiliumNetworkPolicies
kubectl -n project-alpha describe cnp workspace-baseline

# What the agent is enforcing, per endpoint (pod)
kubectl -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg endpoint list
kubectl -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg fqdn cache list

# Hubble
tre hubble                                     # starts the port-forward once
hubble observe --namespace project-alpha --follow
hubble observe --namespace project-alpha --verdict DROPPED
hubble observe --namespace project-alpha --protocol dns
hubble observe --from-pod project-alpha/jupyter-researcher1 --to-namespace project-beta

Hubble UI¶

Open /hubble/ from the portal, pick a namespace (e.g. project-alpha) and you get a live service map. Dropped flows show in red; click a flow for its source/destination identities and the verdict.

Warning

Hubble UI has no authentication in KID. In a real TRE, network flow data is sensitive (it shows who talks to what) and must sit behind authentication.

In K8TRE¶

K8TRE also runs Cilium with Hubble, uses the Cilium Gateway API instead of Traefik, and replaces kube-proxy. KID keeps kube-proxy (kubeProxyReplacement=false) because the eBPF replacement is fiddly inside nested Docker containers.