trevor — Iteration Plan¶

Spec-driven, iterative delivery. Each iteration produces working, tested code and a refined spec layer. Spec artefacts (API schema, data model updates) are written before implementation within each iteration.

Iteration 0 — Project skeleton & local dev¶

Goal: A running trevor with no features, but with the full scaffolding in place.

Deliverables: - uv-managed Python project, ruff config, prek hooks - FastAPI app with health endpoint - SQLModel + Alembic setup (SQLite locally, PostgreSQL path documented) - Keycloak dev container + OIDC login flow (or DEV_AUTH_BYPASS) - MinIO dev container + S3 client abstraction (aioboto3) - ARQ worker + Redis setup - Tilt / k3d local dev environment - Helm chart skeleton - Docker image + GitHub Actions CI (lint, test, build) - spec/ directory with all initial ADRs (this document set)

Iteration 1 — Project sync & user model¶

Goal: trevor knows about projects and users.

Spec to write first: - OpenAPI paths: GET /projects, GET /projects/{id}, GET /users/me - DB migration: Project, User, ProjectMembership tables

Deliverables: - CRD sync CronJob reading CR8TOR project CRDs - User shadow record created/updated on Keycloak login - ProjectMembership CRUD (admin only) - Role conflict validation (researcher ≠ checker on same project) - Admin UI: project list, checker assignment

Iteration 2 — Airlock request lifecycle (researcher side)¶

Goal: Researcher can create a request and upload output objects.

Spec to write first: - OpenAPI paths: POST /requests, GET /requests, GET /requests/{id}, POST /requests/{id}/objects, GET /requests/{id}/objects/{object_id} - DB migration: AirlockRequest, OutputObject, OutputObjectMetadata, AuditEvent - State machine implementation

Deliverables: - Request creation (egress + ingress direction) - File upload with streaming to quarantine S3, checksum computation - Object metadata form (title, description, statbarn, justification, suppression notes) - File preview rendering (markdown, CSV, image, PDF, parquet, code) - Request submission - Researcher dashboard: list requests, view status - Audit event emission on all state transitions

Iteration 3 — Agent review¶

Goal: Automatic statbarn-based review runs on submission.

Spec to write first: - Agent rule engine design (statbarn → rules mapping) - Review table migration - ARQ job: agent_review_job

Deliverables: - Statbarn rule engine (initial rule set from SACRO guide) - Agent review ARQ job triggered on SUBMITTED - Structured report output per object - Optional LLM narrative via configurable provider - Agent report stored as Review record - State transition: SUBMITTED → AGENT_REVIEW → HUMAN_REVIEW - Notification to checkers when agent report is ready

Iteration 4 — Human review (checker side)¶

Goal: Output checkers can review requests and provide structured feedback.

Spec to write first: - OpenAPI paths: POST /requests/{id}/reviews, GET /requests/{id}/reviews - Checker dashboard spec

Deliverables: - Checker dashboard: queue of requests awaiting review - Per-object review form with agent report displayed alongside - Accept agent findings / override per object - Decision: approve / request changes / reject - Checker feedback written to OutputObjectMetadata.checker_feedback - Two-reviewer rule enforcement - State transitions to CHANGES_REQUESTED, APPROVED, REJECTED - Notifications: researcher notified of outcome

Iteration 5 — Revision cycle (researcher response)¶

Goal: Researcher can respond to feedback and resubmit.

Deliverables: - Researcher sees per-object feedback inline with their objects - Replacement object upload (new version of a logical object) - Metadata carry-forward on replacement - Resubmission (returns request to SUBMITTED, triggers agent review again) - Lineage display in UI: version history per logical object

Iteration 6 — Release (RO-Crate + pre-signed URL)¶

Goal: Approved requests are packaged and delivered.

Deliverables: - RO-Crate assembly from approved objects + metadata - Checksum verification before assembly - Upload crate zip to release S3 bucket - Pre-signed URL generation (configurable TTL) - ReleaseRecord created - Email notification with download link - State transition to RELEASED - URL expiry warning notification (CronJob)

Iteration 7 — Admin dashboard & metrics¶

Goal: Admins and senior checkers have full visibility and operational metrics.

Deliverables: - All-projects request overview (status, age, direction) - Metrics: median time to review, requests per checker, approval rates, revision counts - Audit log viewer (filterable by project, user, event type, date range) - Checker assignment management - Stuck request detection (requests waiting > configurable SLA) - Export audit log as CSV

Iteration 7.5 — Datastar UI¶

Goal: Server-rendered Datastar UI covering all backend functionality from iterations 1–7.

Spec to write first: - Template structure, Datastar patterns, SSE endpoints - See spec/iteration-7.5-spec.md for full spec

Deliverables: - Base template shell (nav, project switcher, auth state, flash messages) - Researcher views: request list, create, detail, upload, metadata, replace, resubmit - Checker views: review queue, review form with agent report alongside - Admin views: request overview, metrics dashboard, audit log, membership management - File preview component (CSV, markdown, code, image, PDF) - SSE live updates (request status, review queue) - Minimal custom CSS (no framework, no build step)

Iteration 8 — Ingress flow¶

Goal: Complete the ingress direction (import into TRE).

Deliverables: - Ingress request creation by admin/external submitter - Pre-signed PUT URL for external file submission - Review flow (same as egress) - Delivery to workspace on approval (pre-signed GET URL for workspace to consume) - UI: Ingress-specific views (upload via pre-signed PUT, delivery status)

Iteration 9 — Hardening & observability¶

Deliverables: - Prometheus metrics endpoint (/metrics) - Structured JSON logging - OpenTelemetry tracing - Horizontal scaling smoke tests (multi-pod ARQ worker concurrency) - Security review: CSRF, rate limiting, input validation audit - Pen test checklist - Production Helm values review - Runbook documentation - UI: CSRF token integration, error pages (403, 404, 500), loading states

Iteration 10 — Local development environment¶

Goal: Single-command local dev stack with Tilt, k3d, and all infrastructure dependencies.

Deliverables: - Devcontainer configuration (VS Code / Codespaces / remote) - Bare-metal setup scripts (scripts/dev-setup.sh, scripts/dev-teardown.sh) - Tiltfile rewrite: trevor + PostgreSQL + Redis + SeaweedFS + Keycloak in k3d - Kubernetes dev manifests (deploy/dev/): SeaweedFS, PostgreSQL, Redis, Keycloak - Pre-configured Keycloak realm with test users - SeaweedFS replaces MinIO (ADR-0013 — licensing) - Developer quick-start documentation

Iteration 11 — Production Helm chart completion¶

Goal: Complete Helm chart with all Kubernetes resources for production deployment.

Deliverables: - Service, ServiceAccount, Ingress templates - ARQ worker Deployment template - HorizontalPodAutoscaler template - PodDisruptionBudget template - Network policies (optional, gated) - Alembic migration Job (Helm pre-upgrade hook) - envFromSecrets wired into API and worker containers - Startup probes, init container migration option - NOTES.txt post-install message

Iteration 12 — OIDC login/logout flow & JWT validation¶

Goal: Full browser login/logout UX via Keycloak OIDC and JWT validation for API Bearer tokens.

Spec: spec/iterations/iteration-12-spec.md
ADR: ADR-0007 (existing — covers all auth decisions)

Deliverables: - OIDC client module: discovery, PKCE, token exchange, JWKS cache, ID token validation - Session cookie module: stateless signed cookie via itsdangerous - Auth routes: /auth/login (redirect to Keycloak), /auth/callback (code exchange), /auth/logout - Dual auth in get_auth_context: session cookie for UI, Bearer token for API, dev bypass unchanged - JWT validation against Keycloak JWKS (python-jose) - 401 handler: redirect to login for browser requests, JSON for API - Logout link in nav template - Keycloak realm config: standardFlowEnabled: true - Tests: PKCE, session cookie roundtrip, JWT validation, login/callback/logout flow (mocked Keycloak)

Iteration 12.5 — Checker UI Redesign (Split-Pane Review)¶

Goal: Redesign output checker UI from flat review queue to three-level drill-down (project list → request list → split-pane review) inspired by SACRO Outputs Viewer, adapted for trevor's domain model.

Spec: spec/iterations/iteration-12.5-spec.md

Deliverables: - Project list page: card grid with pending request counts and oldest wait time - Per-project request list with agent decision badges - Full-viewport split-pane review form: - Left sidebar: object list with status indicators, Datastar-powered client-side selection - Right detail panel: metadata grid, researcher justification, agent assessment (statbarn confirmation, disclosure risk, rule checks, narrative), per-object approve/reject controls - Bottom footer: overall decision radios, summary input, submit - Agent findings parsed per-object from Review.findings JSON - Researcher metadata (justification, suppression notes) surfaced to checker - All-or-none release model enforced in UI - 4 new tests (164 total)

Iteration 13 — CRD Sync Reconciler¶

Goal: Automatically sync CR8TOR Project, User, and Group CRDs into trevor's database.

Spec: spec/iterations/iteration-13-spec.md
ADRs: ADR-0012 (CRD sync design), ADR-0016 (official kubernetes client, supersedes ADR-0014)

Deliverables: - kubernetes>=31.0 dependency (official Python client) - CRD client module (crd.py) — CustomObjectsApi list functions wrapped for async via run_in_executor - Sync service (crd_sync_service.py) — parse CRDs, reconcile projects/users/memberships - ARQ cron job running every 5 minutes - RBAC manifest (ClusterRole + ClusterRoleBinding) - Researcher memberships derived from parent + analyst Group CRDs - Checker memberships preserved (never deleted by sync) - Role conflict detection and warning

Iteration 14 — Notification Service: Core + In-App Backend¶

Goal: Pluggable notification system with in-app notification storage and API.

Spec: spec/iterations/iteration-14-spec.md
ADR: ADR-0009 (notification abstraction)

Deliverables: - Notification DB model + Alembic migration - NotificationEvent dataclass, NotificationBackend protocol - NotificationRouter with error-isolated multi-backend dispatch - InAppBackend — writes to Notification table - ARQ job send_notifications_job for async dispatch - API: GET /notifications, PATCH /notifications/{id}/read, GET /notifications/unread-count - Nav bar badge showing unread notification count (static, SSE in iter 16) - Notification triggers wired into review, release, and state-transition flows

Iteration 15 — Email Notification Backend¶

Goal: SMTP email delivery for all notification events.

Spec: spec/iterations/iteration-15-spec.md
ADR: ADR-0009

Deliverables: - aiosmtplib dependency - SmtpBackend implementing NotificationBackend protocol - 7 Jinja2 email templates (HTML + plain-text): submitted, agent_report_ready, changes_requested, approved, rejected, released, url_expiring - Settings: smtp_host, smtp_port, smtp_from_address, smtp_use_tls - Error-isolated dispatch (SMTP failure logged, never blocks) - Registration in NotificationRouter when email_notifications_enabled=True

Iteration 16 — SSE Live Updates¶

Goal: Real-time UI updates via Server-Sent Events using Datastar's native SSE support.

Spec: spec/iterations/iteration-16-spec.md
ADR: ADR-0015 (SSE live updates)

Deliverables: - SSE helper utility for Datastar fragment events - GET /ui/sse/requests/{id}/status — live request status badge - GET /ui/sse/review/queue-count — live review queue count for checkers - GET /ui/sse/notifications/count — live unread notification count - Template updates: data-on-load attributes on status badges and nav badges - Database polling (2s interval, 5min connection timeout) - Auth via session cookie on connection open

Iteration 17 — ARQ Cron Jobs: URL Expiry & Stuck Requests¶

Goal: Proactive alerts for expiring pre-signed URLs and stuck requests.

Spec: spec/iterations/iteration-17-spec.md

Deliverables: - url_expiry_warning_job — detect ReleaseRecords with URLs expiring within 48h, notify researcher - stuck_request_alert_job — detect requests stuck in review > configurable hours, notify admins - expiry_warned_at field on ReleaseRecord (Alembic migration) - request.stuck notification event type - Idempotent — expiry_warned_at prevents duplicate warnings - Settings: url_expiry_warning_hours

Iteration 18 — File Preview Rendering¶

Goal: Server-side file preview for CSV, markdown, code, images in the review and detail views.

Spec: spec/iterations/iteration-18-spec.md

Deliverables: - Preview service (preview_service.py) — render file content to HTML - CSV/TSV/Parquet via polars (first 50 rows) - Markdown via mistune - Code via pygments syntax highlighting - Images via base64 data URI - HTML sanitization via nh3 - Size limit: skip preview for files > 10MB - Graceful degradation when S3 unavailable (dev mode) - Updated file_preview.html template

Iteration 19 — Production Readiness¶

Goal: PostgreSQL migration testing, Helm hardening, CI improvements, operational documentation.

Spec: spec/iterations/iteration-19-spec.md

Deliverables: - CI job running Alembic migrations + tests against PostgreSQL - values.production.yaml with production-appropriate Helm defaults - Helm lint + template validation in CI - Dockerfile scan (trivy) in CI - docs/runbook.md — deployment, migrations, failure modes, monitoring, scaling - docs/security-checklist.md — CSRF, rate limiting, auth, secrets, container security

Iteration 20 — Two-Panel Researcher/Checker UI¶

Goal: Redesign the researcher request detail page and the checker review form into a consistent two-panel layout with inline upload, inline metadata editing, file-type icons, and per-object state dots. Expose the title metadata field in all relevant forms and views.

Spec: spec/iterations/iteration-20-spec.md

Deliverables: - Two-panel researcher detail page: left nav (ft-icon + state dot) + right panel (object detail, inline upload, inline metadata edit, Reviews/Audit tabs) - Datastar signals replacing deprecated data-store in checker review form - title field added to metadata POST endpoint, upload form, and checker review metadata display - _build_request_detail_ctx shared context helper - CSS: ft-icon file-type palette (tabular/figure/code/report/model/other), obj-state-dot state indicators - 10 new tests (262 total)

Later / backlog¶

• Per-user notification preference settings
• Slack / Teams webhook backend
• Multi-language UI (i18n)
• Bulk operations (approve multiple objects in one action)
• Researcher self-service statbarn lookup / guidance
• API key support for programmatic submission
• TRE-specific RO-Crate profile proposal (community contribution)
• CRD watch mode (real-time sync, complement to periodic reconcile)