Implementation plan¶

This page records the plan used to turn the original karectl-demo dev container into the KARECTL in Docker (KID) training environment. It is kept in the docs so facilitators can see why things are built the way they are.

1. Goals and non-goals¶

Goals: a hands-on demonstrator that lets junior research software engineers see, touch and break the core mechanisms of a Kubernetes-based Trusted Research Environment (TRE), modelled on the K8TRE reference implementation:

Non-goals: this is not a production TRE. There is no TLS inside the cluster, no secrets manager, no airlock or egress review, no backup, no HA, and credentials are hard-coded demo values.

2. Constraints¶

• Runs in GitHub Codespaces (4 cores / 16 GB) and locally in VS Code Dev Containers.
• Must fit a 16 GB laptop; the target is ≤ 6–7 GB of RAM in steady state with two notebooks running (Docker Desktop should be given 8–10 GB).
• Works on amd64 and arm64 (Apple Silicon) hosts.
• Trainees can fork (full GitOps loop) or clone (read-only GitOps, everything else works).

3. Review of the starting point¶

What already existed and worked well (kept):

• Docker Compose dev container with a privileged k3s server container (flannel and the k3s network-policy controller disabled so Cilium owns networking), bpffs mounted for Cilium.
• bootstrap.sh that installs Cilium, waits for Traefik, installs Argo CD and applies a root app.
• Argo CD app-of-apps with Keycloak, JupyterHub (Z2JH chart via Kustomize), a portal and a JupyterHub CiliumNetworkPolicy.
• Traefik ingress with path routing: /, /auth, /jupyter, /argocd.

Gaps identified (addressed by this plan):

4. Target architecture¶

flowchart LR
  subgraph host["Codespace / laptop (Docker)"]
    dc["dev container<br/>kubectl · helm · cilium · hubble · argocd · kubescape"]
    subgraph k3s["k3s-server container (single-node Kubernetes)"]
      traefik["Traefik ingress"]
      subgraph kube-system
        cilium["Cilium + Hubble"]
      end
      argocd["argocd"]
      keycloak["keycloak"]
      subgraph jupyterhub
        hub["hub"]; proxy["proxy"]
      end
      kyverno["kyverno"]
      portal["portal"]
      subgraph project-alpha
        nbA["jupyter-researcher1"]; pvcA[("project-shared")]
      end
      subgraph project-beta
        nbB["jupyter-researcher2"]; pvcB[("project-shared")]
      end
    end
  end
  dc -- kubectl --> k3s
  traefik --> proxy --> nbA & nbB
  hub -- spawn --> nbA & nbB
  hub -- OIDC --> keycloak

Components and budget¶

Estimated steady state: ~3 GB for the platform, plus notebooks.

5. Design decisions¶

1. k3s in Docker Compose (kept) rather than kind: it's already working, bundles Traefik, local-path and metrics-server, and matches K8TRE's on-prem K3s target.
2. Projects are namespaces. Following K8TRE (project-<name>), each project gets a namespace labelled k8tre.io/type=project and k8tre.io/project=<name>. A project Helm chart (gitops/projects/chart) stamps out: Namespace (with Pod Security Admission labels), ResourceQuota, LimitRange, shared PVC, hub RoleBinding, and workspace network policy. Projects are declared as a list in gitops/apps/values.yaml; Argo CD creates one Application per project.
3. Project membership comes from Keycloak groups. A user in Keycloak group alpha may launch workspaces in project-alpha. OAuthenticator's manage_groups syncs groups on login.
4. JupyterHub discovers projects from the cluster. The hub lists namespaces labelled k8tre.io/type=project and builds the profile list from namespace annotations (display name, image, size options) filtered by the user's groups. Adding a project needs no hub restart. KubeSpawner re-evaluates the profile list at spawn time, so a user cannot forge a request for a project they are not a member of.
5. Least-privilege hub RBAC. No cluster-wide pod permissions: a ClusterRole is bound per project namespace by a RoleBinding; cluster-wide the hub may only read namespaces.
6. Defence in depth for networking.
7. Project chart ships an explicit workspace-baseline CiliumNetworkPolicy (ingress only from the hub/proxy, egress only to DNS and the hub).
8. Kyverno generates a default-deny CiliumNetworkPolicy into any namespace labelled as a project, even one created by hand, as a safety net.
9. Internet access is granted per project by opt-in FQDN policies (lab exercise).
10. Storage: two named StorageClasses on top of k3s' local-path provisioner: tre-user-home (Delete) and tre-project-shared (Retain). K8TRE's storage spec asks apps to use pre-defined classes, not the default. Single node ⇒ RWO volumes can be shared by pods of the same project (documented caveat).
11. Kyverno policies scoped to project namespaces: restrict image registries, require limits, disallow privileged containers / host namespaces / hostPath, audit :latest, and generate the default-deny policy.
12. Kubescape as a CLI to avoid ~1.5 GB of in-cluster operator footprint.
13. Pinned versions kept from the working baseline (k3s 1.31, Cilium 1.16, Argo CD 3.0, Z2JH 4.0.0, Kyverno chart 3.3.4). Upgrades are a facilitator task, documented in reference/versions.md.

6. Work breakdown (commits)¶

1. Plan (this page).
2. Dev container: multi-arch Dockerfile, add hubble and kubescape CLIs, bootstrap.sh refactor (repo/branch detection, PUBLIC_URL, Hubble, Argo CD trimming, idempotency), scripts/tre helper CLI.
3. GitOps restructure: templated root app, apps Helm chart, platform app (StorageClasses, Hubble ingress, Kyverno RBAC).
4. Projects and JupyterHub: project chart, alpha/beta projects, JupyterHub config (per-project spawning, storage, groups), RBAC, network policies, Keycloak realm (groups, users, group mapper), portal update.
5. Kyverno: Argo app + policies.
6. Documentation: Zensical site: getting started, concepts, component guides, 9 labs, reference, facilitator notes.
7. GitHub Pages workflow + README.

7. Risks and mitigations¶