deploy: npm ci --ignore-scripts (kills postinstall vector + esbuild ETXTBSY race); persistent npm cache; deployment log

deploy/DEPLOY-LOG.md

@@ -0,0 +1,170 @@
+# Deployment log — CTAO portal demo on strapi-experimental.cyfronet
+
+Living document: what was done, what broke, what is left. Update it as work
+proceeds so any agent (or human) can resume without the chat history.
+Started 2026-07-28.
+
+## Goal
+
+Run the git-based CMS demo (Gitea + Sveltia + Astro static build) on the
+Cyfronet test machine so the SUSS-PORT team can click through it. Shape must
+be production-viable, not throwaway. Machine footprint: one directory
+(`~/ctao-portal-demo/`) + 4 unit files, fully removable (README "Uninstall").
+
+## Ground rules (from the user, non-negotiable)
+
+- **Write every command out in chat, marked read-only / write, BEFORE running
+  it** — the user reviews on a phone and cannot see truncated tool calls.
+- Non-invasive verification first, debugging second: confirm each precondition
+  instead of fixing after the fact.
+- Never read secrets (`~/.ssh/*`, tokens). Mask token URLs in any output
+  (`sed 's|://[^@]*@|://***@|g'`). Secrets stay in 0600 files on the machine.
+- No sudo, nothing outside `$HOME` on the machine, no traces for the admin.
+- Temp/working files under `.tmp/<date>-<topic>/` in the repo, never `/tmp`.
+- Commit messages: **no "Co-Authored-By: Claude"**, no session links.
+
+## Architecture (decided, see also README.md in this dir)
+
+```
+editor → Sveltia (/admin, static) → commit → Gitea :3000 (container, SQLite)
+                                                  │ polled every 10 s
+                                     build.sh (systemd user timer)
+                                     └─ podman run node+git, --network=container:ctao-demo-gitea
+                                        npm ci (only if lockfile changed) → astro build
+                                        → releases/<sha>/ → atomic symlink flip
+                                                  │
+                                      nginx :8080 serves releases/current
+                                                  │ (TODO) ingress vhost → team access
+```
+
+## Machine facts (verified 2026-07-28, read-only)
+
+- Rocky Linux 9.7, host `outline.openstacklocal`, IP 192.168.10.15, 2 vCPU,
+  3.6 GB RAM, no swap. User `strapi`, no sudo, `Linger=yes`, user systemd running.
+- Podman 5.6.0 rootless. Host has node 24.17, npm, curl, jq, rsync — **no git**,
+  **no nginx** (hence the build container / nginx container).
+- Neighbours: pod-strapi (`:1337`), Outline (`:9091`), code-server (`127.0.0.1:41787`).
+- **VPN does NOT route to machine ports** (`curl 192.168.10.15:1337` times out);
+  only the admin-managed HTTPS ingress reaches the machine
+  (`https://strapi.isl-dev.grid.cyfronet.pl` → 204). Hence ports 3000/8080 are
+  reachable for us only via `ssh -L`, and the team needs vhosts from Hubert.
+
+## DONE
+
+### 1. Local: files authored + reviewed + tested (before touching the machine)
+- Wrote quadlets, `build.sh`, systemd service/timer, `nginx.conf`,
+  `Containerfile.build`, `README.md` (runbook incl. uninstall).
+- **Opus security review** — 8 findings, every one verified personally:
+  BLOCKER `--network=host` (build code could reach host loopback: code-server
+  = RCE, Strapi) → replaced; `OnUnitActiveSec` is start-relative, not
+  end-relative (agent corrected me) → `OnUnitInactiveSec`; missing `mkdir` for
+  `~/.config/*`; nginx-before-first-build race; prune `pipefail` landmine;
+  noisy unit failure when Gitea is down; CORS `SCHEME` note; RAM headroom.
+  All fixed. No hallucinated findings.
+- **Local end-to-end test** against the local Gitea (`:3010`), sandbox
+  `.tmp/2026-07-28-deploy-test/`: cold build 23 s · no-op poll 45 ms ·
+  incremental rebuild 4 s · prune ✓ · Gitea-down grace ✓ · anonymous clone
+  (no token in `.git/config`) ✓.
+- Commits (local Gitea repo `ctao/portal`): `ed7a9c0`, `80d886c`.
+
+### 2. Machine: install steps 1–7 of README
+- Created `~/ctao-portal-demo/{gitea-data,gitea-config,releases,state,bin,config}`
+  and `~/.config/{containers/systemd,systemd/user}`.
+- Copied unit files, `build.sh`, `nginx.conf`, `Containerfile.build`.
+- Pulled + **pinned by digest**: gitea 1.27-rootless, nginx stable-alpine,
+  node:24-alpine; built `localhost/ctao-portal-build:1`.
+- Gitea running (`:3000`, API version 1.27.1), admin `ctao` created with
+  `--random-password` → `~/ctao-portal-demo/state/initial-admin.txt` (0600),
+  push token → `state/push-token.txt` (0600). **Never printed in chat.**
+- Content pushed via git bundle (78 MB, one-off bootstrap): repo `ctao/portal`
+  at `80d886c`, public, anonymously clonable. Bundle deleted afterwards.
+- **First build on the machine: 44 s** (`npm ci` ~35 s + astro build 6.5 s),
+  225 pages. Incremental rebuilds should be ~10 s (measure to confirm).
+- nginx container running, `http://localhost:8080/` → **200, `<title>CTAO
+  Science Portal</title>`**.
+- Poll timer enabled (`ctao-portal-build.timer`), ticking every ~10 s, journal
+  clean.
+
+### 3. Problems hit on the machine (and the fixes)
+1. **Rootless bridge network impossible**: `netavark: modprobe ip_tables:
+   Operation not permitted` → quadlet restart loop. Root cause: kernel module
+   not loaded, rootless cannot modprobe, no sudo. **Fix:** dropped
+   `ctao-demo.network`; build joins Gitea's netns
+   (`--network=container:ctao-demo-gitea`), so it reaches Gitea on
+   `localhost:3000` while the host loopback stays unreachable (pasta).
+   Same security property as the bridge, zero privileges. Lesson written to
+   `.skills/podman-quadlet/SKILL.md`.
+2. **Push rejected**: admin was created with `--must-change-password` →
+   `remote: Update your password`. **Fix:** `gitea admin user
+   must-change-password --unset ctao`.
+3. **My rc-chain bug**: `... | sed` made `$?` the sed's status, so a failed
+   push printed "push-ok" and the bundle was deleted prematurely → had to
+   re-upload 78 MB. **Fix:** capture output in a var, check rc of the real
+   command. (Also why the upload "took so long" — it ran twice.)
+4. **Push-created repo was private** despite `DEFAULT_PRIVATE=public` —
+   push-to-create has its own key. **Fix:** PATCH via API to public +
+   `GITEA__repository__DEFAULT_PUSH_CREATE_PRIVATE=false` in the quadlet.
+5. **`npm ci` failed: `ETXTBSY` spawning `esbuild`** (postinstall race in a
+   rootless container, ~5 min wasted). **Fix:** `npm ci --ignore-scripts`
+   (also removes the malicious-postinstall vector) + persistent npm cache
+   volume `~/ctao-portal-demo/npm-cache`. Validated locally, then on machine.
+
+### 4. Left on the machine (known, not cleaned — user asked to be told, not tidied)
+- Orphan systemd entry `ctao-demo-network.service` (not-found/active-exited),
+  ghost of the removed network quadlet. Harmless; clears with
+  `systemctl --user stop ctao-demo-network.service`.
+- `/tmp/storage-run-1003/` (~136 KB) — podman runroot from ssh calls without
+  `XDG_RUNTIME_DIR`. Disappears on reboot. (`storage-run-1001` is NOT ours.)
+- Red journal entries from the netavark restart loop (history, not a live fault).
+- Disk: 11 G → 9.6 G free (images ~590 MB + repo/content). RAM available ~1.3 G.
+
+## TODO (next agent starts here)
+
+1. **Commit the last local fix** (`build.sh`: `--ignore-scripts` + npm cache)
+   to the local Gitea repo, then mirror it to the machine's Gitea so the
+   machine's copy of the repo matches. NOTE: `build.sh` currently lives in
+   the same repo as the content; pushing to the machine repo re-triggers a
+   build (that is fine and is a good live test of the pipeline).
+2. **Measure an incremental rebuild on the machine** (push a trivial content
+   change → time from commit to visible change). Expect ~10–20 s total.
+   Record the number here; it is the "publish latency" figure for the demo.
+3. **Verify the whole loop through `ssh -L`** with the user watching:
+   `ssh -L 8080:localhost:8080 -L 3000:localhost:3000 strapi-experimental.cyfronet`
+   → portal at `http://localhost:8080`, Gitea at `http://localhost:3000`.
+   Note: forwarding the SAME port numbers keeps Gitea's `ROOT_URL` valid,
+   so Sveltia's OAuth login can be tested end-to-end before any vhost exists.
+4. **Sveltia OAuth app**: create in Gitea UI (Settings → Applications), PKCE,
+   redirect to the portal `/admin/`; put URL + client id in
+   `public/admin/config.yml`; commit → auto-publishes. Then test a real edit
+   from the browser (the demo's money shot).
+5. **Message to Hubert** (only after we have seen it working):
+   - two ingress vhosts → `192.168.10.15:8080` (portal) and `:3000` (gitea),
+     same mechanism as `strapi.isl-dev…:1337`;
+   - are vhosts public-internet or VPN-scopable? (Gitea preferably VPN-only);
+   - does Cyfronet offer static-file hosting on the ingress itself? If yes we
+     drop our nginx container and just rsync builds there.
+6. **After vhosts arrive**: replace both `TODO(vhost)` in
+   `ctao-demo-gitea.container` (ROOT_URL, CORS origin + `GITEA__cors__SCHEME=https`),
+   `daemon-reload` + restart, update Sveltia config, re-test login.
+7. **Backup**: nightly `tar` of `~/ctao-portal-demo/gitea-data` (the only
+   stateful thing; everything else is rebuildable). Not yet written.
+8. **Decided but not implemented — split into two repos** (`portal` = code,
+   `portal-content` = markdown + uploads). Rationale: editors get write access
+   ONLY to content, so repo-push cannot alter `package.json`/templates that the
+   build executes; also keeps code history clean. Cost: `build.sh` clones two
+   repos and overlays content (~10 lines), polls two SHAs. Do this for the
+   production version, not mid-demo.
+9. Demo choreography: the test article is `draft: true`; the user creates one
+   live during the demo. With the poller running, a Sveltia save publishes
+   automatically — no manual rebuild needed (unlike the Mac preview setup).
+
+## Command cheat-sheet (machine)
+
+```
+export XDG_RUNTIME_DIR=/run/user/$(id -u)          # needed for systemctl --user over ssh
+systemctl --user status ctao-demo-gitea ctao-demo-web ctao-portal-build.timer
+journalctl --user -u ctao-portal-build -n 20 --no-pager     # build times land here
+~/ctao-portal-demo/bin/build.sh                    # manual build
+readlink ~/ctao-portal-demo/releases/current       # which sha is live
+curl -s -o /dev/null -w '%{http_code}' http://localhost:8080/
+```

deploy/build.sh

@@ -19,7 +19,7 @@ BUILD_IMAGE="${BUILD_IMAGE:-localhost/ctao-portal-build:1}"
 BUILD_NETNS="${BUILD_NETNS:-container:ctao-demo-gitea}"
 REPO_INTERNAL="${REPO_INTERNAL:-http://localhost:3000/$REPO.git}"
 KEEP="${KEEP:-3}"                                 # released builds to retain
-mkdir -p "$BASE/repo" "$BASE/releases" "$BASE/state"
+mkdir -p "$BASE/repo" "$BASE/releases" "$BASE/state" "$BASE/npm-cache"
 
 # --- 1. Cheap poll: branch head via the local Gitea API (host curl + jq) ---
 sha=$(curl -fsS --max-time 5 "$GITEA_URL/api/v1/repos/$REPO/branches/$BRANCH" \
@@ -46,6 +46,7 @@ podman run --rm --network="$BUILD_NETNS" --memory=1g \
   -e SHA="$sha" -e REPO_URL="$REPO_INTERNAL" -e BRANCH="$BRANCH" \
   -v "$BASE/repo:/work/repo:z" \
   -v "$BASE/releases:/work/releases:z" \
+  -v "$BASE/npm-cache:/root/.npm:z" \
   -w /work "$BUILD_IMAGE" sh -ec '
     git config --global safe.directory "*"
     [ -d repo/.git ] || git clone --branch "$BRANCH" "$REPO_URL" repo
@@ -55,7 +56,11 @@ podman run --rm --network="$BUILD_NETNS" --memory=1g \
     cd repo
     lock=$(sha256sum package-lock.json | cut -d" " -f1)
     if [ ! -d node_modules ] || [ "$lock" != "$(cat .deps-hash 2>/dev/null)" ]; then
-      npm ci --no-audit --no-fund
+      # --ignore-scripts: (1) kills the malicious-postinstall vector from npm
+      # deps entirely, (2) avoids the esbuild ETXTBSY postinstall race in
+      # rootless containers. esbuild ships its binary as an optional dep, so
+      # nothing needed here actually requires lifecycle scripts.
+      npm ci --ignore-scripts --no-audit --no-fund
       echo "$lock" > .deps-hash
     fi
     npm run build

deploy/ctao-demo-gitea.container

@@ -40,6 +40,8 @@ Environment=GITEA__mailer__ENABLED=false
 # pipeline can clone anonymously — content is the public site anyway.
 Environment=GITEA__repository__ENABLE_PUSH_CREATE_USER=true
 Environment=GITEA__repository__DEFAULT_PRIVATE=public
+# Push-created repos have their OWN default (true = private) — learned the hard way:
+Environment=GITEA__repository__DEFAULT_PUSH_CREATE_PRIVATE=false
 # Sveltia is served from the portal vhost and calls the Gitea API cross-origin.
 # TODO(vhost): tighten * to the portal origin once known, and add
 # GITEA__cors__SCHEME=https (the [cors] SCHEME default is http).