handoff review fixes: deploy log removed from the repo (internal infra notes do not belong in the handover), diagrams moved to the content repo, pages glob non-recursive, dead badge/pill css and stale design-system pointers removed, node engines declared, README states access and node prerequisites, demo-only dev server settings dropped, audit clean
This commit is contained in:
@@ -1,199 +0,0 @@
|
||||
# Deployment log — CTAO portal demo on strapi-experimental.cyfronet
|
||||
|
||||
Living document: what was done, what broke, what is left. Update it as work
|
||||
proceeds so any agent (or human) can resume without the chat history.
|
||||
Started 2026-07-28.
|
||||
|
||||
## Goal
|
||||
|
||||
Run the git-based CMS demo (Gitea + Sveltia + Astro static build) on the
|
||||
Cyfronet test machine so the SUSS-PORT team can click through it. Shape must
|
||||
be production-viable, not throwaway. Machine footprint: one directory
|
||||
(`~/ctao-portal-demo/`) + 4 unit files, fully removable (README "Uninstall").
|
||||
|
||||
## Ground rules (from the user, non-negotiable)
|
||||
|
||||
- **Write every command out in chat, marked read-only / write, BEFORE running
|
||||
it** — the user reviews on a phone and cannot see truncated tool calls.
|
||||
- Non-invasive verification first, debugging second: confirm each precondition
|
||||
instead of fixing after the fact.
|
||||
- Never read secrets (`~/.ssh/*`, tokens). Mask token URLs in any output
|
||||
(`sed 's|://[^@]*@|://***@|g'`). Secrets stay in 0600 files on the machine.
|
||||
- No sudo, nothing outside `$HOME` on the machine, no traces for the admin.
|
||||
- Temp/working files under `.tmp/<date>-<topic>/` in the repo, never `/tmp`.
|
||||
- Commit messages: **no "Co-Authored-By: Claude"**, no session links.
|
||||
|
||||
## Architecture (decided, see also README.md in this dir)
|
||||
|
||||
```
|
||||
editor → Sveltia (/admin, static) → commit → Gitea :3000 (container, SQLite)
|
||||
│ polled every 10 s
|
||||
build.sh (systemd user timer)
|
||||
└─ podman run node+git, --network=container:ctao-demo-gitea
|
||||
npm ci (only if lockfile changed) → astro build
|
||||
→ releases/<sha>/ → atomic symlink flip
|
||||
│
|
||||
nginx :8080 serves releases/current
|
||||
│ (TODO) ingress vhost → team access
|
||||
```
|
||||
|
||||
## Machine facts (verified 2026-07-28, read-only)
|
||||
|
||||
- Rocky Linux 9.7, host `outline.openstacklocal`, IP 192.168.10.15, 2 vCPU,
|
||||
3.6 GB RAM, no swap. User `strapi`, no sudo, `Linger=yes`, user systemd running.
|
||||
- Podman 5.6.0 rootless. Host has node 24.17, npm, curl, jq, rsync — **no git**,
|
||||
**no nginx** (hence the build container / nginx container).
|
||||
- Neighbours: pod-strapi (`:1337`), Outline (`:9091`), code-server (`127.0.0.1:41787`).
|
||||
- **VPN does NOT route to machine ports** (`curl 192.168.10.15:1337` times out);
|
||||
only the admin-managed HTTPS ingress reaches the machine
|
||||
(`https://strapi.isl-dev.grid.cyfronet.pl` → 204). Hence ports 3000/8080 are
|
||||
reachable for us only via `ssh -L`, and the team needs vhosts from Hubert.
|
||||
|
||||
## DONE
|
||||
|
||||
### 1. Local: files authored + reviewed + tested (before touching the machine)
|
||||
- Wrote quadlets, `build.sh`, systemd service/timer, `nginx.conf`,
|
||||
`Containerfile.build`, `README.md` (runbook incl. uninstall).
|
||||
- **Opus security review** — 8 findings, every one verified personally:
|
||||
BLOCKER `--network=host` (build code could reach host loopback: code-server
|
||||
= RCE, Strapi) → replaced; `OnUnitActiveSec` is start-relative, not
|
||||
end-relative (agent corrected me) → `OnUnitInactiveSec`; missing `mkdir` for
|
||||
`~/.config/*`; nginx-before-first-build race; prune `pipefail` landmine;
|
||||
noisy unit failure when Gitea is down; CORS `SCHEME` note; RAM headroom.
|
||||
All fixed. No hallucinated findings.
|
||||
- **Local end-to-end test** against the local Gitea (`:3010`), sandbox
|
||||
`.tmp/2026-07-28-deploy-test/`: cold build 23 s · no-op poll 45 ms ·
|
||||
incremental rebuild 4 s · prune ✓ · Gitea-down grace ✓ · anonymous clone
|
||||
(no token in `.git/config`) ✓.
|
||||
- Commits (local Gitea repo `ctao/portal`): `ed7a9c0`, `80d886c`.
|
||||
|
||||
### 2. Machine: install steps 1–7 of README
|
||||
- Created `~/ctao-portal-demo/{gitea-data,gitea-config,releases,state,bin,config}`
|
||||
and `~/.config/{containers/systemd,systemd/user}`.
|
||||
- Copied unit files, `build.sh`, `nginx.conf`, `Containerfile.build`.
|
||||
- Pulled + **pinned by digest**: gitea 1.27-rootless, nginx stable-alpine,
|
||||
node:24-alpine; built `localhost/ctao-portal-build:1`.
|
||||
- Gitea running (`:3000`, API version 1.27.1), admin `ctao` created with
|
||||
`--random-password` → `~/ctao-portal-demo/state/initial-admin.txt` (0600),
|
||||
push token → `state/push-token.txt` (0600). **Never printed in chat.**
|
||||
- Content pushed via git bundle (78 MB, one-off bootstrap): repo `ctao/portal`
|
||||
at `80d886c`, public, anonymously clonable. Bundle deleted afterwards.
|
||||
- **First build on the machine: 44 s** (`npm ci` ~35 s + astro build 6.5 s),
|
||||
225 pages. Incremental rebuilds should be ~10 s (measure to confirm).
|
||||
- nginx container running, `http://localhost:8080/` → **200, `<title>CTAO
|
||||
Science Portal</title>`**.
|
||||
- Poll timer enabled (`ctao-portal-build.timer`), ticking every ~10 s, journal
|
||||
clean.
|
||||
|
||||
### 3. Problems hit on the machine (and the fixes)
|
||||
1. **Rootless bridge network impossible**: `netavark: modprobe ip_tables:
|
||||
Operation not permitted` → quadlet restart loop. Root cause: kernel module
|
||||
not loaded, rootless cannot modprobe, no sudo. **Fix:** dropped
|
||||
`ctao-demo.network`; build joins Gitea's netns
|
||||
(`--network=container:ctao-demo-gitea`), so it reaches Gitea on
|
||||
`localhost:3000` while the host loopback stays unreachable (pasta).
|
||||
Same security property as the bridge, zero privileges. Lesson written to
|
||||
`.skills/podman-quadlet/SKILL.md`.
|
||||
2. **Push rejected**: admin was created with `--must-change-password` →
|
||||
`remote: Update your password`. **Fix:** `gitea admin user
|
||||
must-change-password --unset ctao`.
|
||||
3. **My rc-chain bug**: `... | sed` made `$?` the sed's status, so a failed
|
||||
push printed "push-ok" and the bundle was deleted prematurely → had to
|
||||
re-upload 78 MB. **Fix:** capture output in a var, check rc of the real
|
||||
command. (Also why the upload "took so long" — it ran twice.)
|
||||
4. **Push-created repo was private** despite `DEFAULT_PRIVATE=public` —
|
||||
push-to-create has its own key. **Fix:** PATCH via API to public +
|
||||
`GITEA__repository__DEFAULT_PUSH_CREATE_PRIVATE=false` in the quadlet.
|
||||
5. **`npm ci` failed: `ETXTBSY` spawning `esbuild`** (postinstall race in a
|
||||
rootless container, ~5 min wasted). **Fix:** `npm ci --ignore-scripts`
|
||||
(also removes the malicious-postinstall vector) + persistent npm cache
|
||||
volume `~/ctao-portal-demo/npm-cache`. Validated locally, then on machine.
|
||||
|
||||
### 4. Left on the machine (known, not cleaned — user asked to be told, not tidied)
|
||||
- Orphan systemd entry `ctao-demo-network.service` (not-found/active-exited),
|
||||
ghost of the removed network quadlet. Harmless; clears with
|
||||
`systemctl --user stop ctao-demo-network.service`.
|
||||
- `/tmp/storage-run-1003/` (~136 KB) — podman runroot from ssh calls without
|
||||
`XDG_RUNTIME_DIR`. Disappears on reboot. (`storage-run-1001` is NOT ours.)
|
||||
- Red journal entries from the netavark restart loop (history, not a live fault).
|
||||
- Disk: 11 G → 9.6 G free (images ~590 MB + repo/content). RAM available ~1.3 G.
|
||||
|
||||
### 5. Pipeline proven end-to-end on the machine (2026-07-28)
|
||||
- Pushed `46ec5e2` to the machine's Gitea (incremental bundle, 6 KB — this is
|
||||
what ongoing syncs cost, vs the 78 MB one-off bootstrap).
|
||||
- **The poller published it automatically in 26 s end-to-end** (push →
|
||||
visible), of which the build itself was 21 s. That is the demo's publish
|
||||
latency: **~30 s worst case** on 2 vCPU (vs 44 s for the very first build
|
||||
which included `npm ci`). Cold-cache builds only happen when the lockfile
|
||||
changes.
|
||||
- Verified after publish: `/` 200, `/admin/` 200 (Sveltia), an article page
|
||||
200; neighbours untouched (Strapi 204, Outline 200); RAM available 1.2 G,
|
||||
disk free 9.0 G.
|
||||
|
||||
### 6. Sveltia wired to the machine + editor cycle proven (2026-07-28)
|
||||
- **OAuth app created via API, no UI clicking**: temp `write:user` token from
|
||||
`gitea admin user generate-access-token` (CLI in container) → POST
|
||||
`/user/applications/oauth2` → temp token deleted (needed basic auth from
|
||||
`initial-admin.txt`; token-auth DELETE returns 401). App: "Sveltia CMS
|
||||
(demo)", `client_id cf1c44ac-ebe6-4a97-bf2f-f7f26ef1126c`, PKCE
|
||||
(`confidential_client=false`), redirect_uris: `http://localhost:18080/admin/`
|
||||
and `http://localhost:8080/admin/`.
|
||||
- **Found & fixed**: `public/admin/config.yml` still pointed at the MAC's Gitea
|
||||
(`localhost:3010` + old app_id) — CMS login on the machine could never work.
|
||||
Switched to `localhost:3000` + machine app_id (`bc02012`). localhost:3000 is
|
||||
valid both on the machine and through the tunnel (same-port trick).
|
||||
- **Editor cycle simulated exactly like Sveltia does it** (Gitea contents API):
|
||||
create article via API → **live on the portal in 12 s**; delete via API →
|
||||
**gone in 21 s**. Repo left clean (`29385a5`). Earlier full-push rebuild:
|
||||
16 s build. Publish latency ≈ **10–30 s** depending on poll-tick alignment.
|
||||
- Tunnel run from the Mac with keepalive (`-o ServerAliveInterval=30`), portal
|
||||
mapped to **18080** locally (user request: non-default port), Gitea kept at
|
||||
**3000** (must match ROOT_URL). First tunnel died after idle — keepalive fixed.
|
||||
- Push token scope confirmed minimal (`write:repository` only) — cannot manage
|
||||
users/apps; file format is `<name>:<40-hex>` (grep the hex out).
|
||||
|
||||
## TODO (next agent starts here)
|
||||
|
||||
1. ~~Commit the `--ignore-scripts` fix and mirror to the machine~~ — DONE
|
||||
(`46ec5e2`, and it doubled as the live pipeline test above).
|
||||
2. ~~Measure incremental publish latency on the machine~~ — DONE: **26 s
|
||||
end-to-end**, build 21 s.
|
||||
3. ~~Verify the loop through `ssh -L`~~ — mostly DONE: tunnel
|
||||
`-L 18080:localhost:8080 -L 3000:localhost:3000` (with ServerAlive
|
||||
keepalive) verified with real content. **Remaining: the user clicks
|
||||
"Sign in with Gitea" at `http://localhost:18080/admin/`** and edits an
|
||||
article in the browser — the only step that needs a human + browser
|
||||
(PKCE login). Credentials: `initial-admin.txt` on the machine.
|
||||
4. ~~Sveltia OAuth app~~ — DONE via API (see §6). Config committed and
|
||||
auto-published. Only the browser login test remains (see 3).
|
||||
5. **Message to Hubert** (only after we have seen it working):
|
||||
- two ingress vhosts → `192.168.10.15:8080` (portal) and `:3000` (gitea),
|
||||
same mechanism as `strapi.isl-dev…:1337`;
|
||||
- are vhosts public-internet or VPN-scopable? (Gitea preferably VPN-only);
|
||||
- does Cyfronet offer static-file hosting on the ingress itself? If yes we
|
||||
drop our nginx container and just rsync builds there.
|
||||
6. **After vhosts arrive**: replace both `TODO(vhost)` in
|
||||
`ctao-demo-gitea.container` (ROOT_URL, CORS origin + `GITEA__cors__SCHEME=https`),
|
||||
`daemon-reload` + restart, update Sveltia config, re-test login.
|
||||
7. **Backup**: nightly `tar` of `~/ctao-portal-demo/gitea-data` (the only
|
||||
stateful thing; everything else is rebuildable). Not yet written.
|
||||
8. **Decided but not implemented — split into two repos** (`portal` = code,
|
||||
`portal-content` = markdown + uploads). Rationale: editors get write access
|
||||
ONLY to content, so repo-push cannot alter `package.json`/templates that the
|
||||
build executes; also keeps code history clean. Cost: `build.sh` clones two
|
||||
repos and overlays content (~10 lines), polls two SHAs. Do this for the
|
||||
production version, not mid-demo.
|
||||
9. Demo choreography: the test article is `draft: true`; the user creates one
|
||||
live during the demo. With the poller running, a Sveltia save publishes
|
||||
automatically — no manual rebuild needed (unlike the Mac preview setup).
|
||||
|
||||
## Command cheat-sheet (machine)
|
||||
|
||||
```
|
||||
export XDG_RUNTIME_DIR=/run/user/$(id -u) # needed for systemctl --user over ssh
|
||||
systemctl --user status ctao-demo-gitea ctao-demo-web ctao-portal-build.timer
|
||||
journalctl --user -u ctao-portal-build -n 20 --no-pager # build times land here
|
||||
~/ctao-portal-demo/bin/build.sh # manual build
|
||||
readlink ~/ctao-portal-demo/releases/current # which sha is live
|
||||
curl -s -o /dev/null -w '%{http_code}' http://localhost:8080/
|
||||
```
|
||||
@@ -40,7 +40,7 @@ Environment=GITEA__service__DISABLE_REGISTRATION=true
|
||||
Environment=GITEA__mailer__ENABLED=false
|
||||
# Repos default to public so the build pipeline can clone anonymously —
|
||||
# content is the public site anyway. The `ctao/portal` repo itself is
|
||||
# created in the UI (README step 6).
|
||||
# created in the UI (README step 7).
|
||||
Environment=GITEA__repository__DEFAULT_PRIVATE=public
|
||||
# Sveltia is served from the portal vhost and calls the Gitea API cross-origin
|
||||
# — CORS locked to exactly that origin. ALLOW_DOMAIN takes FULL origins with
|
||||
|
||||
Reference in New Issue
Block a user