deploy: machine deployment — quadlets (gitea, nginx, isolated build network), poll+build pipeline, runbook

Reviewed (Opus security review, all findings verified) and tested end-to-end
locally: cold build 23s, no-op poll 45ms, incremental rebuild 4s, prune,
gitea-down grace, real bridge network with container DNS.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012jWfn3RwPfFGTtBddm36Uy

deploy/Containerfile.build

@@ -0,0 +1,7 @@
+# Build image for the CTAO portal demo: pinned Node + git.
+# Why: the host has node 24 but NO git (and we have no sudo); a 2-line image
+# keeps the whole toolchain pinned and independent of host packages.
+# Built ONCE at install, never pulled again at runtime:
+#   podman build -t localhost/ctao-portal-build:1 -f Containerfile.build .
+FROM docker.io/library/node:24-alpine
+RUN apk add --no-cache git

deploy/README.md

@@ -0,0 +1,96 @@
+# Deploy — CTAO portal demo on strapi-experimental.cyfronet
+
+Everything runs rootless as user `strapi`. Footprint on the machine:
+ONE directory (`~/ctao-portal-demo/`) + 4 unit files in `~/.config/`.
+Uninstall restores the machine exactly (see bottom). No secrets in any file.
+
+```
+~/ctao-portal-demo/
+├── gitea-data/    # Gitea state (repos, SQLite, accounts)  ← the ONLY thing worth backing up
+├── gitea-config/  # Gitea app.ini (generated from env)
+├── repo/          # clone of the portal repo (created by first build)
+├── releases/      # <sha>/ dirs + `current` symlink (what nginx serves)
+├── state/         # last-built SHA
+├── bin/build.sh   # copied from deploy/ (source of truth stays in the repo)
+└── config/nginx.conf
+```
+
+| Port | What | Exposed how |
+|---|---|---|
+| 3000 | Gitea | ingress vhost (TODO: ask Hubert) |
+| 8080 | portal (nginx, static) | ingress vhost (TODO: ask Hubert) |
+
+## Install (each step reviewed before running; [W] = writes to the machine)
+
+1. **[W]** `mkdir -p ~/ctao-portal-demo/{gitea-data,gitea-config,releases,state,bin,config} ~/.config/containers/systemd ~/.config/systemd/user`
+2. **[W]** Copy files from this dir (scp from the Mac):
+   - `ctao-demo-gitea.container`, `ctao-demo-web.container`, `ctao-demo.network` → `~/.config/containers/systemd/`
+   - `ctao-portal-build.service`, `ctao-portal-build.timer` → `~/.config/systemd/user/`
+   - `build.sh` → `~/ctao-portal-demo/bin/` (`chmod +x`)
+   - `nginx.conf` → `~/ctao-portal-demo/config/`
+3. **[W]** Pull + pin images (one-time, needs internet):
+   `podman pull docker.io/gitea/gitea:1.27-rootless docker.io/library/nginx:stable-alpine docker.io/library/node:24-alpine`
+   `podman build -t localhost/ctao-portal-build:1 -f Containerfile.build .`
+   Then `podman images --digests` → paste the sha256 digests into both `.container` files.
+4. **[W]** `systemctl --user daemon-reload && systemctl --user start ctao-demo-gitea`
+5. **[W]** Create the Gitea admin — run YOURSELF in your own terminal (password
+   is prompted/printed there only; never goes through chat or shell history):
+   `podman exec -it ctao-demo-gitea gitea admin user create --admin --username <you> --email <you@…> --random-password`
+6. **[W]** In the Gitea UI: create org `ctao`, repo `portal` (public read).
+   Push from the Mac through an SSH port-forward:
+   `ssh -L 3300:localhost:3000 strapi-experimental.cyfronet` then
+   `git remote add machine http://localhost:3300/ctao/portal.git && git push machine main`
+7. **[W]** `systemctl --user enable --now ctao-portal-build.timer` — first run
+   clones + `npm ci` + builds (minutes); later runs are seconds. Wait until
+   `journalctl --user -u ctao-portal-build -n 5` shows `published <sha>`
+   (starting nginx earlier just serves 404s until the first build lands).
+8. **[W]** `systemctl --user start ctao-demo-web`
+9. **[R]** Verify: `curl -s -o /dev/null -w '%{http_code}' http://localhost:8080/`
+   and `journalctl --user -u ctao-portal-build -n 20` (shows measured build times).
+
+## After Hubert assigns the vhosts
+
+1. Replace both `TODO(vhost)` values in `ctao-demo-gitea.container`
+   (ROOT_URL → gitea vhost, CORS `*` → portal origin, add
+   `GITEA__cors__SCHEME=https`); `systemctl --user daemon-reload && systemctl --user restart ctao-demo-gitea`.
+2. In Gitea UI: Settings → Applications → new OAuth2 app for Sveltia
+   (redirect: `https://<portal-vhost>/admin/`), PKCE, no client secret.
+3. Put the Gitea vhost URL + client id into `public/admin/config.yml` in the
+   portal repo, commit, push — the timer publishes it like any other change.
+
+## Ask Hubert (one message)
+
+1. Two ingress vhosts → `192.168.10.15:8080` (portal) and `:3000` (gitea) —
+   same mechanism as `strapi.isl-dev…:1337`.
+2. Are vhosts public-internet or VPN-scopable? (Gitea preferably VPN-only.)
+3. Does Cyfronet offer static-file hosting on the ingress itself? If yes, we
+   drop our nginx container entirely and rsync builds there instead.
+
+## Uninstall (leaves zero traces)
+
+```
+systemctl --user disable --now ctao-portal-build.timer
+systemctl --user stop ctao-demo-web ctao-demo-gitea
+rm ~/.config/containers/systemd/ctao-demo-*.container \
+   ~/.config/containers/systemd/ctao-demo.network \
+   ~/.config/systemd/user/ctao-portal-build.{service,timer}
+systemctl --user daemon-reload
+podman network rm ctao-demo 2>/dev/null || true
+podman rmi localhost/ctao-portal-build:1 docker.io/gitea/gitea:1.27-rootless docker.io/library/nginx:stable-alpine
+rm -rf ~/ctao-portal-demo
+```
+
+## Notes
+
+- Publish latency = poll (≤10 s) + build (measured 1 s on M-series; expect
+  4–8 s on the 2 vCPU VM — every build's time lands in the journal).
+- Internet needed only for: image pulls (install) and `npm ci` when the
+  lockfile changes. Routine rebuilds are fully offline.
+- Memory caps (`MemoryHigh`) keep us polite next to Outline + Strapi;
+  the build container is capped at 1 GB via `podman run --memory`. Watch the
+  first `npm ci` + build in the journal — if it OOMs inside its cgroup
+  (contained, just retries), raise the cap.
+- Build isolation: build containers run on the `ctao-demo` bridge (see
+  `ctao-demo.network`) — repo/npm code cannot reach host loopback services.
+- Secrets inventory: Gitea admin password (typed interactively, lives only
+  in Gitea's DB) — that's the complete list. Build/poll/serve use none.

deploy/build.sh

@@ -0,0 +1,80 @@
+#!/usr/bin/env bash
+# CTAO portal demo — poll the content repo, rebuild, publish atomically.
+# Triggered every 10 s by ctao-portal-build.timer; systemd oneshot semantics
+# guarantee runs never overlap. The 99.9% case is one local curl and exit 0.
+# No secrets anywhere: the portal repo is public-read on the local Gitea.
+set -euo pipefail
+
+# Defaults match the machine; every var is env-overridable so the whole
+# pipeline can be tested locally against a sandbox dir + local Gitea.
+BASE="${BASE:-$HOME/ctao-portal-demo}"
+GITEA_URL="${GITEA_URL:-http://localhost:3000}"   # published by ctao-demo-gitea
+REPO="${REPO:-ctao/portal}"                       # owner/repo in Gitea
+BRANCH="${BRANCH:-main}"
+BUILD_IMAGE="${BUILD_IMAGE:-localhost/ctao-portal-build:1}"
+BUILD_NET="${BUILD_NET:-ctao-demo}"               # bridge from ctao-demo.network
+# Clone URL as seen from INSIDE the build network (container-name DNS);
+# the poll below uses $GITEA_URL because it runs on the host.
+REPO_INTERNAL="${REPO_INTERNAL:-http://ctao-demo-gitea:3000/$REPO.git}"
+KEEP="${KEEP:-3}"                                 # released builds to retain
+mkdir -p "$BASE/repo" "$BASE/releases" "$BASE/state"
+
+# --- 1. Cheap poll: branch head via the local Gitea API (host curl + jq) ---
+sha=$(curl -fsS --max-time 5 "$GITEA_URL/api/v1/repos/$REPO/branches/$BRANCH" \
+      | jq -r '.commit.id' || true)
+# Gitea down/unreachable is a transient, not a unit failure — exit 0 quietly
+# instead of painting the journal red every 10 s.
+if [[ ! "$sha" =~ ^[0-9a-f]{40}$ ]]; then
+  echo "poll failed (gitea unreachable?) — skipping this tick"
+  exit 0
+fi
+[[ "$sha" == "$(cat "$BASE/state/last-built" 2>/dev/null)" ]] && exit 0
+
+echo "building $sha"
+t0=$(date +%s)
+
+# --- 2. Build in the ephemeral container (git + pinned node live there).
+# SECURITY: the container runs npm lifecycle scripts from the repo, so it is
+# confined to the ctao-demo bridge — it reaches Gitea by container name and
+# the internet (for `npm ci` when the lockfile changed), but NOT the host's
+# loopback services (code-server, Strapi). Never use --network=host here.
+# node_modules and .deps-hash are untracked, so they survive checkouts.
+podman run --rm --network="$BUILD_NET" --memory=1g \
+  -e ASTRO_TELEMETRY_DISABLED=1 \
+  -e SHA="$sha" -e REPO_URL="$REPO_INTERNAL" -e BRANCH="$BRANCH" \
+  -v "$BASE/repo:/work/repo:z" \
+  -v "$BASE/releases:/work/releases:z" \
+  -w /work "$BUILD_IMAGE" sh -ec '
+    git config --global safe.directory "*"
+    [ -d repo/.git ] || git clone --branch "$BRANCH" "$REPO_URL" repo
+    git -C repo remote set-url origin "$REPO_URL"   # self-heal if the URL changes
+    git -C repo fetch --quiet origin "$BRANCH"
+    git -C repo checkout --quiet "$SHA"
+    cd repo
+    lock=$(sha256sum package-lock.json | cut -d" " -f1)
+    if [ ! -d node_modules ] || [ "$lock" != "$(cat .deps-hash 2>/dev/null)" ]; then
+      npm ci --no-audit --no-fund
+      echo "$lock" > .deps-hash
+    fi
+    npm run build
+    rm -rf "../releases/$SHA"
+    cp -a dist "../releases/$SHA"
+  '
+
+# --- 3. Atomic publish: symlink flip via rename(2) — no half-published moment.
+# mv -T is GNU (the target host is Rocky); when testing on macOS put a
+# coreutils `mv` (gmv) first in PATH.
+rm -f "$BASE/releases/".current.*   # stale temps from a crash mid-flip
+ln -s "$sha" "$BASE/releases/.current.$$"
+mv -Tf "$BASE/releases/.current.$$" "$BASE/releases/current"
+echo "$sha" > "$BASE/state/last-built"
+
+# --- 4. Prune old releases (never touches `current` — it is always newest).
+# `|| true`: an empty match must not fail the unit after a successful publish
+# (grep exits 1 under pipefail when there is nothing to prune).
+cd "$BASE/releases"
+ls -1t | grep -vx current | tail -n +"$((KEEP + 1))" | while read -r old; do
+  rm -rf -- "$old"
+done || true
+
+echo "published $sha in $(( $(date +%s) - t0 ))s"

deploy/ctao-demo-gitea.container

@@ -0,0 +1,54 @@
+# CTAO portal demo — Gitea (CMS backend: content repo, editor accounts, OAuth for Sveltia)
+# Quadlet unit. Install: copy to ~/.config/containers/systemd/ on the machine,
+# then `systemctl --user daemon-reload` → service name: ctao-demo-gitea.service.
+# Rootless image + SQLite by design: one container, no DB server, all state
+# lives in ~/ctao-portal-demo/gitea-{data,config} — nothing else on the machine.
+
+[Unit]
+Description=CTAO portal demo — Gitea
+Wants=network-online.target
+After=network-online.target
+
+[Container]
+ContainerName=ctao-demo-gitea
+# Same major as the local demo (1.27). Re-pin to the exact digest at install:
+#   podman pull docker.io/gitea/gitea:1.27-rootless
+#   podman images --digests docker.io/gitea/gitea   → paste sha256 here
+Image=docker.io/gitea/gitea:1.27-rootless
+# The rootless image runs as uid 1000 inside; keep-id maps it to the host user
+# so the bind-mounted dirs stay owned by `strapi` (no chown, no root anywhere).
+UserNS=keep-id:uid=1000,gid=1000
+# Shared bridge with the build containers (name-resolvable as ctao-demo-gitea);
+# host loopback stays out of reach for build code — see ctao-demo.network.
+Network=ctao-demo.network
+Volume=%h/ctao-portal-demo/gitea-data:/var/lib/gitea:Z
+Volume=%h/ctao-portal-demo/gitea-config:/etc/gitea:Z
+PublishPort=3000:3000
+
+# Env-driven config — re-applied on every start, no hand-edited app.ini
+# (see .skills/gitea/SKILL.md). Secrets: none here; the admin account is
+# created interactively after first start (README).
+Environment=GITEA__server__HTTP_PORT=3000
+# TODO(vhost): set to the public URL once the ingress vhost exists,
+# e.g. https://gitea-portal-demo.isl-dev.grid.cyfronet.pl/ — then restart.
+Environment=GITEA__server__ROOT_URL=http://localhost:3000/
+Environment=GITEA__server__DISABLE_SSH=true
+Environment=GITEA__database__DB_TYPE=sqlite3
+Environment=GITEA__security__INSTALL_LOCK=true
+Environment=GITEA__service__DISABLE_REGISTRATION=true
+Environment=GITEA__mailer__ENABLED=false
+# Sveltia is served from the portal vhost and calls the Gitea API cross-origin.
+# TODO(vhost): tighten * to the portal origin once known, and add
+# GITEA__cors__SCHEME=https (the [cors] SCHEME default is http).
+Environment=GITEA__cors__ENABLED=true
+Environment=GITEA__cors__ALLOW_DOMAIN=*
+Environment=GITEA__cors__METHODS=GET,HEAD,POST,PUT,PATCH,DELETE,OPTIONS
+
+[Service]
+Restart=on-failure
+# Shared 3.6 GB VM (Outline + Strapi live here too) — keep Gitea on a leash.
+MemoryHigh=400M
+MemoryMax=600M
+
+[Install]
+WantedBy=default.target

deploy/ctao-demo-web.container

@@ -0,0 +1,25 @@
+# CTAO portal demo — static web server. Serves ~/ctao-portal-demo/releases/current
+# (a symlink the build flips atomically). Read-only mounts: nginx can only read.
+# Quadlet unit → service name: ctao-demo-web.service.
+
+[Unit]
+Description=CTAO portal demo — static web server (nginx)
+Wants=network-online.target
+After=network-online.target
+
+[Container]
+ContainerName=ctao-demo-web
+# Official image; re-pin to the exact digest at install (same drill as Gitea).
+Image=docker.io/library/nginx:stable-alpine
+# releases/ is shared with the build container (:z shared label);
+# the config file is exclusive to nginx (:Z).
+Volume=%h/ctao-portal-demo/releases:/srv/releases:ro,z
+Volume=%h/ctao-portal-demo/config/nginx.conf:/etc/nginx/conf.d/default.conf:ro,Z
+PublishPort=8080:80
+
+[Service]
+Restart=on-failure
+MemoryHigh=64M
+
+[Install]
+WantedBy=default.target

deploy/ctao-demo.network

@@ -0,0 +1,11 @@
+# CTAO portal demo — dedicated bridge network (quadlet .network unit).
+# Why it exists: the build container executes npm lifecycle scripts from the
+# repo, so it must NOT see the host's loopback (code-server, Strapi). On this
+# bridge, containers resolve each other by name (aardvark-dns) and have NATed
+# internet egress, but host 127.0.0.1 services are unreachable — unlike
+# --network=host, which this replaces.
+[Unit]
+Description=CTAO portal demo — internal bridge network
+
+[Network]
+NetworkName=ctao-demo

deploy/ctao-portal-build.service

@@ -0,0 +1,15 @@
+# CTAO portal demo — rebuild-on-new-commit worker (fired by the .timer).
+# Install: copy to ~/.config/systemd/user/ → systemctl --user daemon-reload.
+# Type=oneshot + timer means runs can never overlap: the timer will not
+# activate a service that is still running.
+
+[Unit]
+Description=CTAO portal demo — rebuild if the content repo has new commits
+
+[Service]
+Type=oneshot
+ExecStart=%h/ctao-portal-demo/bin/build.sh
+# Polite neighbour on the shared VM:
+Nice=10
+# First-ever npm ci on 2 vCPU can be slow; normal rebuilds are seconds.
+TimeoutStartSec=900

deploy/ctao-portal-build.timer

@@ -0,0 +1,18 @@
+# CTAO portal demo — poll cadence for the build worker.
+# OnUnitInactiveSec: next tick 10 s after the previous run FINISHES (it is
+# deactivation-relative; OnUnitActiveSec would be start-relative). Overlap is
+# impossible either way — a timer never activates a still-running unit.
+# OnActiveSec covers `systemctl --user enable --now` (first tick 5 s later);
+# OnBootSec covers reboot. Enable: systemctl --user enable --now ctao-portal-build.timer
+
+[Unit]
+Description=CTAO portal demo — poll the content repo every 10 s
+
+[Timer]
+OnActiveSec=5
+OnBootSec=30
+OnUnitInactiveSec=10
+AccuracySec=1s
+
+[Install]
+WantedBy=timers.target

deploy/nginx.conf

@@ -0,0 +1,30 @@
+# CTAO portal demo — static serving. The cache split is what makes the
+# 10-second publish loop feel instant: HTML is revalidated on every request
+# (a symlink flip shows up on the next refresh), fingerprinted assets are
+# cached forever.
+server {
+  listen 80;
+  server_name _;
+  root /srv/releases/current;
+  charset utf-8;
+  error_page 404 /404.html;    # Astro emits 404.html at the site root
+
+  # Fingerprinted build assets (/_astro/<name>.<hash>.*) — immutable
+  location /_astro/ {
+    add_header Cache-Control "public, max-age=31536000, immutable";
+  }
+
+  # Editor-uploaded media (stable paths, may be re-uploaded) — short cache
+  location /uploads/ {
+    add_header Cache-Control "public, max-age=3600";
+  }
+
+  # Everything else: HTML pages, feeds, /admin (Sveltia is static files too)
+  location / {
+    try_files $uri $uri/ =404;
+    add_header Cache-Control "no-cache";
+  }
+
+  gzip on;
+  gzip_types text/css application/javascript application/json image/svg+xml application/rss+xml text/xml;
+}