Deployment log — CTAO portal demo on strapi-experimental.cyfronet

Living document: what was done, what broke, what is left. Update it as work proceeds so any agent (or human) can resume without the chat history. Started 2026-07-28.

Goal

Run the git-based CMS demo (Gitea + Sveltia + Astro static build) on the Cyfronet test machine so the SUSS-PORT team can click through it. Shape must be production-viable, not throwaway. Machine footprint: one directory (~/ctao-portal-demo/) + 4 unit files, fully removable (README "Uninstall").

Ground rules (from the user, non-negotiable)

Write every command out in chat, marked read-only / write, BEFORE running it — the user reviews on a phone and cannot see truncated tool calls.
Non-invasive verification first, debugging second: confirm each precondition instead of fixing after the fact.
Never read secrets (~/.ssh/*, tokens). Mask token URLs in any output (sed 's|://[^@]*@|://***@|g'). Secrets stay in 0600 files on the machine.
No sudo, nothing outside $HOME on the machine, no traces for the admin.
Temp/working files under .tmp/<date>-<topic>/ in the repo, never /tmp.
Commit messages: no "Co-Authored-By: Claude", no session links.

Architecture (decided, see also README.md in this dir)

editor → Sveltia (/admin, static) → commit → Gitea :3000 (container, SQLite)
                                                  │ polled every 10 s
                                     build.sh (systemd user timer)
                                     └─ podman run node+git, --network=container:ctao-demo-gitea
                                        npm ci (only if lockfile changed) → astro build
                                        → releases/<sha>/ → atomic symlink flip
                                                  │
                                      nginx :8080 serves releases/current
                                                  │ (TODO) ingress vhost → team access

Machine facts (verified 2026-07-28, read-only)

Rocky Linux 9.7, host outline.openstacklocal, IP 192.168.10.15, 2 vCPU, 3.6 GB RAM, no swap. User strapi, no sudo, Linger=yes, user systemd running.
Podman 5.6.0 rootless. Host has node 24.17, npm, curl, jq, rsync — no git, no nginx (hence the build container / nginx container).
Neighbours: pod-strapi (:1337), Outline (:9091), code-server (127.0.0.1:41787).
VPN does NOT route to machine ports (curl 192.168.10.15:1337 times out); only the admin-managed HTTPS ingress reaches the machine (https://strapi.isl-dev.grid.cyfronet.pl → 204). Hence ports 3000/8080 are reachable for us only via ssh -L, and the team needs vhosts from Hubert.

DONE

1. Local: files authored + reviewed + tested (before touching the machine)

Wrote quadlets, build.sh, systemd service/timer, nginx.conf, Containerfile.build, README.md (runbook incl. uninstall).
Opus security review — 8 findings, every one verified personally: BLOCKER --network=host (build code could reach host loopback: code-server = RCE, Strapi) → replaced; OnUnitActiveSec is start-relative, not end-relative (agent corrected me) → OnUnitInactiveSec; missing mkdir for ~/.config/*; nginx-before-first-build race; prune pipefail landmine; noisy unit failure when Gitea is down; CORS SCHEME note; RAM headroom. All fixed. No hallucinated findings.
Local end-to-end test against the local Gitea (:3010), sandbox .tmp/2026-07-28-deploy-test/: cold build 23 s · no-op poll 45 ms · incremental rebuild 4 s · prune ✓ · Gitea-down grace ✓ · anonymous clone (no token in .git/config) ✓.
Commits (local Gitea repo ctao/portal): ed7a9c0, 80d886c.

2. Machine: install steps 1–7 of README

Created ~/ctao-portal-demo/{gitea-data,gitea-config,releases,state,bin,config} and ~/.config/{containers/systemd,systemd/user}.
Copied unit files, build.sh, nginx.conf, Containerfile.build.
Pulled + pinned by digest: gitea 1.27-rootless, nginx stable-alpine, node:24-alpine; built localhost/ctao-portal-build:1.
Gitea running (:3000, API version 1.27.1), admin ctao created with --random-password → ~/ctao-portal-demo/state/initial-admin.txt (0600), push token → state/push-token.txt (0600). Never printed in chat.
Content pushed via git bundle (78 MB, one-off bootstrap): repo ctao/portal at 80d886c, public, anonymously clonable. Bundle deleted afterwards.
First build on the machine: 44 s (npm ci ~35 s + astro build 6.5 s), 225 pages. Incremental rebuilds should be ~10 s (measure to confirm).
nginx container running, http://localhost:8080/ → 200, <title>CTAO Science Portal</title>.
Poll timer enabled (ctao-portal-build.timer), ticking every ~10 s, journal clean.

3. Problems hit on the machine (and the fixes)

Rootless bridge network impossible: netavark: modprobe ip_tables: Operation not permitted → quadlet restart loop. Root cause: kernel module not loaded, rootless cannot modprobe, no sudo. Fix: dropped ctao-demo.network; build joins Gitea's netns (--network=container:ctao-demo-gitea), so it reaches Gitea on localhost:3000 while the host loopback stays unreachable (pasta). Same security property as the bridge, zero privileges. Lesson written to .skills/podman-quadlet/SKILL.md.
Push rejected: admin was created with --must-change-password → remote: Update your password. Fix: gitea admin user must-change-password --unset ctao.
My rc-chain bug: ... | sed made $? the sed's status, so a failed push printed "push-ok" and the bundle was deleted prematurely → had to re-upload 78 MB. Fix: capture output in a var, check rc of the real command. (Also why the upload "took so long" — it ran twice.)
Push-created repo was private despite DEFAULT_PRIVATE=public — push-to-create has its own key. Fix: PATCH via API to public + GITEA__repository__DEFAULT_PUSH_CREATE_PRIVATE=false in the quadlet.
npm ci failed: ETXTBSY spawning esbuild (postinstall race in a rootless container, ~5 min wasted). Fix: npm ci --ignore-scripts (also removes the malicious-postinstall vector) + persistent npm cache volume ~/ctao-portal-demo/npm-cache. Validated locally, then on machine.

4. Left on the machine (known, not cleaned — user asked to be told, not tidied)

Orphan systemd entry ctao-demo-network.service (not-found/active-exited), ghost of the removed network quadlet. Harmless; clears with systemctl --user stop ctao-demo-network.service.
/tmp/storage-run-1003/ (~136 KB) — podman runroot from ssh calls without XDG_RUNTIME_DIR. Disappears on reboot. (storage-run-1001 is NOT ours.)
Red journal entries from the netavark restart loop (history, not a live fault).
Disk: 11 G → 9.6 G free (images ~590 MB + repo/content). RAM available ~1.3 G.

5. Pipeline proven end-to-end on the machine (2026-07-28)

Pushed 46ec5e2 to the machine's Gitea (incremental bundle, 6 KB — this is what ongoing syncs cost, vs the 78 MB one-off bootstrap).
The poller published it automatically in 26 s end-to-end (push → visible), of which the build itself was 21 s. That is the demo's publish latency: ~30 s worst case on 2 vCPU (vs 44 s for the very first build which included npm ci). Cold-cache builds only happen when the lockfile changes.
Verified after publish: / 200, /admin/ 200 (Sveltia), an article page 200; neighbours untouched (Strapi 204, Outline 200); RAM available 1.2 G, disk free 9.0 G.

6. Sveltia wired to the machine + editor cycle proven (2026-07-28)

OAuth app created via API, no UI clicking: temp write:user token from gitea admin user generate-access-token (CLI in container) → POST /user/applications/oauth2 → temp token deleted (needed basic auth from initial-admin.txt; token-auth DELETE returns 401). App: "Sveltia CMS (demo)", client_id cf1c44ac-ebe6-4a97-bf2f-f7f26ef1126c, PKCE (confidential_client=false), redirect_uris: http://localhost:18080/admin/ and http://localhost:8080/admin/.
Found & fixed: public/admin/config.yml still pointed at the MAC's Gitea (localhost:3010 + old app_id) — CMS login on the machine could never work. Switched to localhost:3000 + machine app_id (bc02012). localhost:3000 is valid both on the machine and through the tunnel (same-port trick).
Editor cycle simulated exactly like Sveltia does it (Gitea contents API): create article via API → live on the portal in 12 s; delete via API → gone in 21 s. Repo left clean (29385a5). Earlier full-push rebuild: 16 s build. Publish latency ≈ 10–30 s depending on poll-tick alignment.
Tunnel run from the Mac with keepalive (-o ServerAliveInterval=30), portal mapped to 18080 locally (user request: non-default port), Gitea kept at 3000 (must match ROOT_URL). First tunnel died after idle — keepalive fixed.
Push token scope confirmed minimal (write:repository only) — cannot manage users/apps; file format is <name>:<40-hex> (grep the hex out).

TODO (next agent starts here)

~~Commit the --ignore-scripts fix and mirror to the machine~~ — DONE (46ec5e2, and it doubled as the live pipeline test above).
~~Measure incremental publish latency on the machine~~ — DONE: 26 s end-to-end, build 21 s.
~~Verify the loop through ssh -L~~ — mostly DONE: tunnel -L 18080:localhost:8080 -L 3000:localhost:3000 (with ServerAlive keepalive) verified with real content. Remaining: the user clicks "Sign in with Gitea" at http://localhost:18080/admin/ and edits an article in the browser — the only step that needs a human + browser (PKCE login). Credentials: initial-admin.txt on the machine.
~~Sveltia OAuth app~~ — DONE via API (see §6). Config committed and auto-published. Only the browser login test remains (see 3).
Message to Hubert (only after we have seen it working):
- two ingress vhosts → 192.168.10.15:8080 (portal) and :3000 (gitea), same mechanism as strapi.isl-dev…:1337;
- are vhosts public-internet or VPN-scopable? (Gitea preferably VPN-only);
- does Cyfronet offer static-file hosting on the ingress itself? If yes we drop our nginx container and just rsync builds there.
After vhosts arrive: replace both TODO(vhost) in ctao-demo-gitea.container (ROOT_URL, CORS origin + GITEA__cors__SCHEME=https), daemon-reload + restart, update Sveltia config, re-test login.
Backup: nightly tar of ~/ctao-portal-demo/gitea-data (the only stateful thing; everything else is rebuildable). Not yet written.
Decided but not implemented — split into two repos (portal = code, portal-content = markdown + uploads). Rationale: editors get write access ONLY to content, so repo-push cannot alter package.json/templates that the build executes; also keeps code history clean. Cost: build.sh clones two repos and overlays content (~10 lines), polls two SHAs. Do this for the production version, not mid-demo.
Demo choreography: the test article is draft: true; the user creates one live during the demo. With the poller running, a Sveltia save publishes automatically — no manual rebuild needed (unlike the Mac preview setup).

Command cheat-sheet (machine)

export XDG_RUNTIME_DIR=/run/user/$(id -u)          # needed for systemctl --user over ssh
systemctl --user status ctao-demo-gitea ctao-demo-web ctao-portal-build.timer
journalctl --user -u ctao-portal-build -n 20 --no-pager     # build times land here
~/ctao-portal-demo/bin/build.sh                    # manual build
readlink ~/ctao-portal-demo/releases/current       # which sha is live
curl -s -o /dev/null -w '%{http_code}' http://localhost:8080/

12 KiB Raw Blame History Unescape Escape