Deployment log — CTAO portal demo on strapi-experimental.cyfronet

Living document: what was done, what broke, what is left. Update it as work proceeds so any agent (or human) can resume without the chat history. Started 2026-07-28.

Goal

Run the git-based CMS demo (Gitea + Sveltia + Astro static build) on the Cyfronet test machine so the SUSS-PORT team can click through it. Shape must be production-viable, not throwaway. Machine footprint: one directory (~/ctao-portal-demo/) + 4 unit files, fully removable (README "Uninstall").

Ground rules (from the user, non-negotiable)

Write every command out in chat, marked read-only / write, BEFORE running it — the user reviews on a phone and cannot see truncated tool calls.
Non-invasive verification first, debugging second: confirm each precondition instead of fixing after the fact.
Never read secrets (~/.ssh/*, tokens). Mask token URLs in any output (sed 's|://[^@]*@|://***@|g'). Secrets stay in 0600 files on the machine.
No sudo, nothing outside $HOME on the machine, no traces for the admin.
Temp/working files under .tmp/<date>-<topic>/ in the repo, never /tmp.
Commit messages: no "Co-Authored-By: Claude", no session links.

Architecture (decided, see also README.md in this dir)

editor → Sveltia (/admin, static) → commit → Gitea :3000 (container, SQLite)
                                                  │ polled every 10 s
                                     build.sh (systemd user timer)
                                     └─ podman run node+git, --network=container:ctao-demo-gitea
                                        npm ci (only if lockfile changed) → astro build
                                        → releases/<sha>/ → atomic symlink flip
                                                  │
                                      nginx :8080 serves releases/current
                                                  │ (TODO) ingress vhost → team access

Machine facts (verified 2026-07-28, read-only)

Rocky Linux 9.7, host outline.openstacklocal, IP 192.168.10.15, 2 vCPU, 3.6 GB RAM, no swap. User strapi, no sudo, Linger=yes, user systemd running.
Podman 5.6.0 rootless. Host has node 24.17, npm, curl, jq, rsync — no git, no nginx (hence the build container / nginx container).
Neighbours: pod-strapi (:1337), Outline (:9091), code-server (127.0.0.1:41787).
VPN does NOT route to machine ports (curl 192.168.10.15:1337 times out); only the admin-managed HTTPS ingress reaches the machine (https://strapi.isl-dev.grid.cyfronet.pl → 204). Hence ports 3000/8080 are reachable for us only via ssh -L, and the team needs vhosts from Hubert.

DONE

1. Local: files authored + reviewed + tested (before touching the machine)

Wrote quadlets, build.sh, systemd service/timer, nginx.conf, Containerfile.build, README.md (runbook incl. uninstall).
Opus security review — 8 findings, every one verified personally: BLOCKER --network=host (build code could reach host loopback: code-server = RCE, Strapi) → replaced; OnUnitActiveSec is start-relative, not end-relative (agent corrected me) → OnUnitInactiveSec; missing mkdir for ~/.config/*; nginx-before-first-build race; prune pipefail landmine; noisy unit failure when Gitea is down; CORS SCHEME note; RAM headroom. All fixed. No hallucinated findings.
Local end-to-end test against the local Gitea (:3010), sandbox .tmp/2026-07-28-deploy-test/: cold build 23 s · no-op poll 45 ms · incremental rebuild 4 s · prune ✓ · Gitea-down grace ✓ · anonymous clone (no token in .git/config) ✓.
Commits (local Gitea repo ctao/portal): ed7a9c0, 80d886c.

2. Machine: install steps 1–7 of README

Created ~/ctao-portal-demo/{gitea-data,gitea-config,releases,state,bin,config} and ~/.config/{containers/systemd,systemd/user}.
Copied unit files, build.sh, nginx.conf, Containerfile.build.
Pulled + pinned by digest: gitea 1.27-rootless, nginx stable-alpine, node:24-alpine; built localhost/ctao-portal-build:1.
Gitea running (:3000, API version 1.27.1), admin ctao created with --random-password → ~/ctao-portal-demo/state/initial-admin.txt (0600), push token → state/push-token.txt (0600). Never printed in chat.
Content pushed via git bundle (78 MB, one-off bootstrap): repo ctao/portal at 80d886c, public, anonymously clonable. Bundle deleted afterwards.
First build on the machine: 44 s (npm ci ~35 s + astro build 6.5 s), 225 pages. Incremental rebuilds should be ~10 s (measure to confirm).
nginx container running, http://localhost:8080/ → 200, <title>CTAO Science Portal</title>.
Poll timer enabled (ctao-portal-build.timer), ticking every ~10 s, journal clean.

3. Problems hit on the machine (and the fixes)

Rootless bridge network impossible: netavark: modprobe ip_tables: Operation not permitted → quadlet restart loop. Root cause: kernel module not loaded, rootless cannot modprobe, no sudo. Fix: dropped ctao-demo.network; build joins Gitea's netns (--network=container:ctao-demo-gitea), so it reaches Gitea on localhost:3000 while the host loopback stays unreachable (pasta). Same security property as the bridge, zero privileges. Lesson written to .skills/podman-quadlet/SKILL.md.
Push rejected: admin was created with --must-change-password → remote: Update your password. Fix: gitea admin user must-change-password --unset ctao.
My rc-chain bug: ... | sed made $? the sed's status, so a failed push printed "push-ok" and the bundle was deleted prematurely → had to re-upload 78 MB. Fix: capture output in a var, check rc of the real command. (Also why the upload "took so long" — it ran twice.)
Push-created repo was private despite DEFAULT_PRIVATE=public — push-to-create has its own key. Fix: PATCH via API to public + GITEA__repository__DEFAULT_PUSH_CREATE_PRIVATE=false in the quadlet.
npm ci failed: ETXTBSY spawning esbuild (postinstall race in a rootless container, ~5 min wasted). Fix: npm ci --ignore-scripts (also removes the malicious-postinstall vector) + persistent npm cache volume ~/ctao-portal-demo/npm-cache. Validated locally, then on machine.

4. Left on the machine (known, not cleaned — user asked to be told, not tidied)

Orphan systemd entry ctao-demo-network.service (not-found/active-exited), ghost of the removed network quadlet. Harmless; clears with systemctl --user stop ctao-demo-network.service.
/tmp/storage-run-1003/ (~136 KB) — podman runroot from ssh calls without XDG_RUNTIME_DIR. Disappears on reboot. (storage-run-1001 is NOT ours.)
Red journal entries from the netavark restart loop (history, not a live fault).
Disk: 11 G → 9.6 G free (images ~590 MB + repo/content). RAM available ~1.3 G.

TODO (next agent starts here)

Commit the last local fix (build.sh: --ignore-scripts + npm cache) to the local Gitea repo, then mirror it to the machine's Gitea so the machine's copy of the repo matches. NOTE: build.sh currently lives in the same repo as the content; pushing to the machine repo re-triggers a build (that is fine and is a good live test of the pipeline).
Measure an incremental rebuild on the machine (push a trivial content change → time from commit to visible change). Expect ~10–20 s total. Record the number here; it is the "publish latency" figure for the demo.
Verify the whole loop through ssh -L with the user watching: ssh -L 8080:localhost:8080 -L 3000:localhost:3000 strapi-experimental.cyfronet → portal at http://localhost:8080, Gitea at http://localhost:3000. Note: forwarding the SAME port numbers keeps Gitea's ROOT_URL valid, so Sveltia's OAuth login can be tested end-to-end before any vhost exists.
Sveltia OAuth app: create in Gitea UI (Settings → Applications), PKCE, redirect to the portal /admin/; put URL + client id in public/admin/config.yml; commit → auto-publishes. Then test a real edit from the browser (the demo's money shot).
Message to Hubert (only after we have seen it working):
- two ingress vhosts → 192.168.10.15:8080 (portal) and :3000 (gitea), same mechanism as strapi.isl-dev…:1337;
- are vhosts public-internet or VPN-scopable? (Gitea preferably VPN-only);
- does Cyfronet offer static-file hosting on the ingress itself? If yes we drop our nginx container and just rsync builds there.
After vhosts arrive: replace both TODO(vhost) in ctao-demo-gitea.container (ROOT_URL, CORS origin + GITEA__cors__SCHEME=https), daemon-reload + restart, update Sveltia config, re-test login.
Backup: nightly tar of ~/ctao-portal-demo/gitea-data (the only stateful thing; everything else is rebuildable). Not yet written.
Decided but not implemented — split into two repos (portal = code, portal-content = markdown + uploads). Rationale: editors get write access ONLY to content, so repo-push cannot alter package.json/templates that the build executes; also keeps code history clean. Cost: build.sh clones two repos and overlays content (~10 lines), polls two SHAs. Do this for the production version, not mid-demo.
Demo choreography: the test article is draft: true; the user creates one live during the demo. With the poller running, a Sveltia save publishes automatically — no manual rebuild needed (unlike the Mac preview setup).

Command cheat-sheet (machine)

export XDG_RUNTIME_DIR=/run/user/$(id -u)          # needed for systemctl --user over ssh
systemctl --user status ctao-demo-gitea ctao-demo-web ctao-portal-build.timer
journalctl --user -u ctao-portal-build -n 20 --no-pager     # build times land here
~/ctao-portal-demo/bin/build.sh                    # manual build
readlink ~/ctao-portal-demo/releases/current       # which sha is live
curl -s -o /dev/null -w '%{http_code}' http://localhost:8080/

10 KiB Raw Blame History Unescape Escape