Skip to content

Lab 3: Project isolation

Objectives: show that workspaces in different projects can't talk to each other, see the dropped traffic in Hubble, and understand the policies responsible.

Setup

You need two running workspaces:

  • researcher1 in project-alpha (from lab 2)
  • researcher2 in project-beta: log in as researcher2 in a private window and start a server
kubectl get pods -A -l component=singleuser-server -o wide

Note the IP of jupyter-researcher2.

1. Start watching the network

tre hubble
hubble observe --namespace project-alpha --follow

Leave this running in its own terminal (VS Code: split the terminal).

2. Try to reach the other project

As researcher1, open a notebook (File → New → Notebook, Python 3) and run:

import socket
beta_ip = "<IP of jupyter-researcher2>"
try:
    socket.create_connection((beta_ip, 8888), timeout=3)
    print("CONNECTED - isolation is broken!")
except OSError as e:
    print("blocked:", e)

You should see blocked: timed out. The Hubble terminal shows something like (abridged):

project-alpha/jupyter-researcher1:43210 (ID:...) <> 10.42.0.57:8888 (ID:...) policy-verdict:none EGRESS DENIED (TCP Flags: SYN)
project-alpha/jupyter-researcher1:43210 (ID:...) <> 10.42.0.57:8888 (ID:...) Policy denied DROPPED (TCP Flags: SYN)

3. Also blocked: the Kubernetes API

try:
    socket.create_connection(("kubernetes.default.svc", 443), timeout=3)
    print("CONNECTED")
except OSError as e:
    print("blocked:", e)

A workspace that could talk to the API server could, with a leaked token, try to change the cluster.

4. Allowed: DNS and the hub

print(socket.gethostbyname("hub.jupyterhub.svc.cluster.local"))
socket.create_connection(("hub.jupyterhub.svc.cluster.local", 8081), timeout=3)
print("hub reachable")

Both work because they are on the allow-list.

5. Read the policies

kubectl -n project-alpha get ciliumnetworkpolicies
kubectl -n project-alpha get cnp workspace-baseline -o yaml
kubectl -n project-alpha get cnp default-deny -o yaml
  • workspace-baseline came from Git (Argo CD app project-alpha).
  • default-deny was generated by Kyverno (see the app.kubernetes.io/managed-by: kyverno label) because the namespace is labelled as a project.

Now look at what Cilium actually enforces for the pod:

kubectl -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg endpoint list | grep -E "^[0-9]+ +Enabled +Enabled"

POLICY (ingress/egress) ENFORCEMENT reads Enabled / Enabled for workspace endpoints.

6. Hubble UI

Open Hubble UI from the portal (/hubble/) and choose namespace project-alpha. Re-run the cell from step 2 and watch a red (dropped) flow appear. Click it to see source and destination identities.

7. Try to remove the safety net

kubectl -n project-alpha delete cnp default-deny
sleep 10
kubectl -n project-alpha get cnp

default-deny is back: the Kyverno generate rule has synchronize: true. Now try workspace-baseline:

kubectl -n project-alpha delete cnp workspace-baseline
kubectl -n project-alpha get cnp -w

Argo CD restores it (selfHeal). While it's missing, what can the workspace still do? Remember default-deny denies everything.

What just happened

  • Cilium turned the policies into eBPF programs attached to each pod's network interface.
  • With a policy selecting the pod, anything not allowed was dropped in the kernel before leaving the pod.
  • Hubble recorded the drop along with the identities (namespace/labels) of both ends.

Check your understanding

  1. researcher1 and researcher3 are both in project-alpha. Can their workspaces talk to each other? Test it. Why? (Look at the ingress rule in workspace-baseline.)
  2. Why is it useful to have both a Git-managed policy and a Kyverno-generated one?
  3. What would an auditor want from Hubble in a real TRE, and where should that data be kept?