fix: align HWLAB v03 native DB controls
This commit is contained in:
@@ -115,7 +115,7 @@ UniDesk 是一个以主 server 为统一入口的分布式工作平台;本文
|
||||
- P0: 每次开始 G14 HWLAB runtime lane 工作前,目标固定仓库必须先即时快进到目标 remote 分支最新;例如 v0.2 使用 `trans G14:/root/hwlab-v02 script -- 'git fetch origin v0.2 && git pull --ff-only origin v0.2 && git status --short --branch && git remote -v'`。若有未提交并行变更阻碍快进,必须先判断是否能与最新 remote 快速合并;能合并则立即合并,禁止默认 stash、丢弃或绕到落后 worktree;只有确实无法自动合并时才隔离保存并停止交给人工决策。禁止用落后的固定仓库继续预检、创建 worktree、开发、render、polling 或部署。
|
||||
- G14 HWLAB service/runtime 开发必须先以目标 runtime lane 固定 repo 做预检,再按目标 lane 创建独立 worktree/PR;v0.2 CaseRun、短连接 CLI、trace、docs/reference 和配置治理这类无服务任务按 `docs/reference/g14.md` 的 `direct-lightweight` 规则可在 `G14:/root/hwlab-v02` 直接提交推送,不触发 PR/CI/CD/rollout。
|
||||
- P0: G14 已有节点本地 GitHub/Google/DockerHub/npm 等 bootstrap 加速 proxy;在 G14 host、临时 pod、构建 pod 或透传 pod 里拉 GitHub/Google 资源前必须优先使用该 proxy,避免把网络直连失败当作代码阻塞。主入口:HTTP/HTTPS `http://127.0.0.1:10808`、SOCKS `socks5h://127.0.0.1:10808`,备份入口和 NO_PROXY 规则见 `docs/reference/g14.md`。
|
||||
- P0: G14 platform PostgreSQL 是 host-native 底层基础设施,HWLAB v0.3+ 通过 namespace-local `g14-platform-postgres` bridge 使用它;迁移后必须清理旧自有 Postgres Secret/PVC,长期规则见 `docs/reference/g14-platform-db.md`。
|
||||
- P0: G14 platform PostgreSQL 是 host-native 底层基础设施,HWLAB v0.3+ 通过 namespace-local `g14-platform-postgres` bridge 使用它;迁移后必须清理旧自有 Postgres StatefulSet/Service/ConfigMap/Secret/PVC,长期规则见 `docs/reference/g14-platform-db.md`。
|
||||
- 禁止把 master server、D601、`/root/HWLAB`、`/home/ubuntu/hwlab`、`/workspace/hwlab`、retired `/root/hwlab` 或临时 clone 当作当前 G14 runtime lane source truth;master server 也不得运行 HWLAB check、Playwright/browser smoke、镜像构建或其他重型验证,验证必须走目标 runtime lane workspace、G14 k3s/Tekton 或其他获批外部执行面。
|
||||
- 长期细节见 `docs/reference/g14.md`;G14 节点本地也保留 `/root/docs/hwlab-g14-workspace.md`,两处口径必须保持一致。
|
||||
|
||||
|
||||
@@ -16,7 +16,7 @@ CI/CD、GitOps、rollout、artifact 发布、PR 合并后的 runtime lane 滚动
|
||||
|
||||
当现有 CLI 对某个 CI/CD 操作缺字段、缺动作、缺状态或缺权限时,处理顺序是先补 CLI,再执行发布或治理动作。临时低层 route 写操作只允许用于一次性止血,并且必须随后把稳定能力补进 CLI 与本参考文档;不能把手工 `kubectl apply/delete/annotate`、原生 GitHub CLI、手写 REST 请求或 registry shell 脚本沉淀成长期流程。长时观察仍遵守 60 秒短查询和 submit-and-poll 语义,不用单个 `trans`/`tran` 等待完整 PipelineRun 或 Argo rollout 结束。
|
||||
|
||||
`hwlab nodes secret cleanup-owned-postgres --node G14 --lane v03 [--dry-run|--confirm]` 是 v03 迁移到 G14 platform PostgreSQL 后清理旧 repo-owned Postgres Secret/PVC 的受控入口。迁移后的 `hwlab-cloud-api-v03-db` 和 `hwlab-v03-openfga` SecretRef 来自 G14 host platform DB 凭据文件,不再从 lane-local Postgres Secret 派生;旧 `hwlab nodes secret ensure --name hwlab-cloud-api-v03-db` 只适用于仍使用 repo-owned lane-local Postgres 的路径,不是 v03 platform DB 轮换入口。平台 DB 运行、SecretRef 轮换边界和 health 验证见 `docs/reference/g14-platform-db.md`。
|
||||
`hwlab nodes secret cleanup-owned-postgres --node G14 --lane v03 [--dry-run|--confirm]` 是 v03 迁移到 G14 platform PostgreSQL 后清理旧 repo-owned Postgres StatefulSet/Service/ConfigMap/Secret/PVC 的受控入口。迁移后的 `hwlab-cloud-api-v03-db` 和 `hwlab-v03-openfga` SecretRef 来自 G14 host platform DB 凭据文件,不再从 lane-local Postgres Secret 派生;v03+ 的 `hwlab nodes secret ensure --name hwlab-cloud-api-v03-db|hwlab-v03-openfga` 旧路径已删除,status 只做 redacted SecretRef 与 `g14-platform-postgres` bridge 观测。平台 DB 运行、SecretRef 轮换边界和 health 验证见 `docs/reference/g14-platform-db.md`。
|
||||
|
||||
`hwlab nodes secret status|ensure --node G14 --lane v03 --name hwlab-v03-code-agent-provider` 是 v03 Code Agent / MoonBridge provider SecretRef 的受控 bootstrap 入口;`ensure` 只从集群内既有 `hwlab-v02/hwlab-v02-code-agent-provider` 复制 `openai-api-key`、`opencode-api-key` 到 lane-local Secret,输出仅披露 source/target Secret 名、key presence、decoded byte count、mutation 和后续命令,禁止打印 base64、解码值、完整 API key 或可复用凭据。OpenFGA 和 master admin API key 继续使用同一命名空间下的 `hwlab nodes secret ... --name hwlab-v03-openfga|hwlab-v03-master-server-admin-api-key`。
|
||||
|
||||
@@ -62,12 +62,13 @@ CI/CD、GitOps、rollout、artifact 发布、PR 合并后的 runtime lane 滚动
|
||||
创建 PipelineRun 前会读取 `devops-infra` mirror refs,若 `localV02` 未等于当前 source commit,则自动执行一次受控 manual `git-mirror sync` Job 并复核 ref,复核失败时停止触发,避免 Tekton `prepare-source` 已知失败;services 参数只包含 v02 runtime service matrix,`hwlab-cli` 是固定 repo 短连接源码工具,不进入 PipelineRun service build。
|
||||
`--dry-run` 只报告是否会 pre-sync,不创建 Job;confirmed trigger 默认创建 `.state/jobs/` 异步 job 并立刻返回 `job.id`、`statusCommand`、stdout/stderr 路径,避免 git mirror pre-sync 或 PipelineRun 创建期间长时间阻塞;`--wait` 路径也必须向 stderr 输出 `hwlab.v02.trigger.progress` JSON 事件,覆盖 `control-plane-refresh`、`git-mirror-pre-sync` 和 `create-pipelinerun`,避免异步 job 长时间只有启动命令而无法判断卡点;默认 JSON 必须对 `manifest_b64`、长脚本和远端 stdout/stderr 做有界摘要,保留长度与 hash,最终 trigger 结果只返回阶段摘要和关键 tail,完整内容通过 job stdout/stderr 文件渐进披露;只有现场同步调试才显式加 `--wait`;旧 `rerun-current` 只作为输入别名保留。PipelineRun `Completed`、Argo `Synced/Healthy` 和 `webAssets.ok=true` 只证明 G14 runtime 已更新;交付收口还必须用 `hwlab g14 git-mirror status` 查看 `cache.summary.pendingFlush`,若为 true,继续执行受控 `hwlab g14 git-mirror flush --confirm` 并用 job status 轮询到 `pendingFlush=false`。
|
||||
- `hwlab g14 control-plane runtime-migration --lane v02 [--dry-run|--allow-live-db-read --dry-run|--confirm]` 只通过 `hwlab-v02` namespace 当前 `deployment/hwlab-cloud-api -c hwlab-cloud-api` 内 repo-owned migration CLI 执行;不读取或打印 Secret 值、不触碰 PROD、不绕到手工 `psql`。
|
||||
- `hwlab g14 secret status|ensure --lane v02 --name hwlab-v02-openfga|hwlab-v02-master-server-admin-api-key [--dry-run|--confirm]` 和 `hwlab nodes secret status|ensure --node G14 --lane v03 --name hwlab-v03-openfga|hwlab-cloud-api-v03-db|hwlab-v03-master-server-admin-api-key [--dry-run|--confirm]` 是 HWLAB runtime lane SecretRef bootstrap 的历史入口:OpenFGA preset 确保 preshared token、datastore URI 和 Postgres password 存在;Cloud API DB preset 只适用于仍使用 repo-owned lane-local Postgres 的路径。v0.3 迁移到 G14 platform PostgreSQL 后,Cloud API/OpenFGA datastore SecretRef 不再从 `hwlab-v03-postgres` Secret 或 StatefulSet 派生;平台库凭据、桥接 Service 和 SecretRef 轮换边界见 `docs/reference/g14-platform-db.md`。master server admin API key preset 确保本机 `/root/.config/hwlab-v0x/master-server-admin-api-key.env` 以 0600 保存 `HWLAB_API_KEY`,并同步到对应 lane 的 `*-master-server-admin-api-key/api-key`。`status` 只返回 key 是否存在、解码后字节数、key prefix、DB 对象存在性和 rollout exit code,永远不读取、不打印、不回传 secret 明文。`hwlab nodes secret cleanup-owned-postgres --node G14 --lane v03 [--dry-run|--confirm]` 是 v0.3+ 迁移到 G14 平台 Postgres 后的受控残留清理入口,只删除旧 repo-owned `hwlab-v03-postgres` Secret 和 `data-hwlab-v03-postgres-0` PVC;它要求 `g14-platform-postgres` Service 已存在、旧 StatefulSet 不存在,默认 dry-run,不触碰平台数据库、OpenFGA/Cloud API 当前 SecretRef 或 GitOps desired state。`hwlab g14 secret delete --lane v02 --name <obsolete-hwlab-v02-secret> [--dry-run|--confirm]` 只用于删除确认已不被 workload 引用的 v0.2 废弃 Secret,默认 dry-run,拒绝删除 OpenFGA/Postgres/master admin API key 等必需 Secret;共享 device-pod API key 已退出当前授权路径,不再提供 ensure/bootstrap 入口。
|
||||
- `hwlab g14 secret status|ensure --lane v02 --name hwlab-v02-openfga|hwlab-v02-master-server-admin-api-key [--dry-run|--confirm]` 和 `hwlab nodes secret status|ensure --node G14 --lane v03 --name hwlab-v03-master-server-admin-api-key [--dry-run|--confirm]` 是 HWLAB runtime lane SecretRef bootstrap 的保留入口。v03+ Cloud API/OpenFGA datastore SecretRef 已迁移到 G14 platform PostgreSQL,`hwlab nodes secret status --node G14 --lane v03 --name hwlab-cloud-api-v03-db|hwlab-v03-openfga` 只做 redacted SecretRef 与 `g14-platform-postgres` bridge 观测;旧 `ensure` 路径已删除,不再从 `hwlab-v03-postgres` Secret 或 StatefulSet 派生。平台库凭据、桥接 Service 和 SecretRef 轮换边界见 `docs/reference/g14-platform-db.md`。master server admin API key preset 确保本机 `/root/.config/hwlab-v0x/master-server-admin-api-key.env` 以 0600 保存 `HWLAB_API_KEY`,并同步到对应 lane 的 `*-master-server-admin-api-key/api-key`。`status` 只返回 key 是否存在、解码后字节数、key prefix、bridge 存在性和 runtime health 相关结果,永远不读取、不打印、不回传 secret 明文。`hwlab nodes secret cleanup-owned-postgres --node G14 --lane v03 [--dry-run|--confirm]` 是 v0.3+ 迁移到 G14 平台 Postgres 后的受控残留清理入口,精确删除旧 repo-owned `hwlab-v03-postgres` StatefulSet/Service/ConfigMap/Secret 和 `data-hwlab-v03-postgres-0` PVC;它要求 `g14-platform-postgres` Service 已存在,默认 dry-run,不触碰平台数据库、OpenFGA/Cloud API 当前 SecretRef 或 GitOps desired state。`hwlab g14 secret delete --lane v02 --name <obsolete-hwlab-v02-secret> [--dry-run|--confirm]` 只用于删除确认已不被 workload 引用的 v0.2 废弃 Secret,默认 dry-run,拒绝删除 OpenFGA/Postgres/master admin API key 等必需 Secret;共享 device-pod API key 已退出当前授权路径,不再提供 ensure/bootstrap 入口。
|
||||
- `hwlab g14 control-plane cleanup-runs --lane v02|v03|g14|all [--min-age-minutes N] [--limit N] [--dry-run|--confirm]` 是完成态 PipelineRun 工作区 retention 入口;真实清理只删除已完成 PipelineRun,让 Tekton/local-path 回收临时 PVC,不触碰 registry storage、业务 PVC、Secret、runtime workload 或 GitOps desired state。带 `--pipeline-run <name>` 或 `--source-commit <full-sha>` 的定点清理必须先直接查询目标 PipelineRun,而不是只从全量列表过滤;不存在的目标返回 `target-pipelinerun-not-found`,未完成目标返回 `target-pipelinerun-not-terminal`,空查询和读取失败分别返回 `target-pipelinerun-query-empty` / `target-pipelinerun-query-failed`,年龄保护仍返回 `below-min-age`。`hwlab nodes control-plane cleanup-runs --node G14 --lane v03 --pipeline-run <name>` 是 v0.3 failed run 受控重试前的清理入口。
|
||||
- `hwlab g14 control-plane cleanup-released-pvs --lane all [--limit N] [--dry-run|--confirm]` 是 local-path 未自动回收后的补充 retention 入口;只列并删除 `Released`、`local-path`、`Delete`、`claimNamespace=hwlab-ci` 且 claim 名称形如 Tekton 临时 `pvc-*` 的 PV。
|
||||
- `hwlab g14 git-mirror status|apply|sync|flush [--dry-run|--confirm]` 是 `devops-infra` git mirror/relay 的受控维护入口:`apply` 渲染并 server-side apply `devops-infra/git-mirror.yaml`,同时删除遗留 `git-mirror-hwlab-sync` CronJob;`sync` 创建一次性 manual Job,把 GitHub allowlist refs 拉入本地 mirror;`flush` 创建一次性 manual Job,把本地 `v0.2-gitops` 快进推回 GitHub。
|
||||
`status` 返回 read/write URL、last sync/write/flush、本地 ref、GitHub staging ref 和 pending flush 状态,并在 `cache.summary` 给出 `localV02`、`localGitops`、`githubGitops`、`pendingFlush`、`flushNeeded`、`githubInSync` 和下一条受控 `flushCommand`。confirmed `sync` 和 `flush` 默认创建 `.state/jobs/` 异步 job 并立刻返回可查询状态,只有现场同步调试才显式加 `--wait`;mirror 不设置 CronJob。
|
||||
如果 PipelineRun 的 `gitops-promote` 阶段报 `git mirror write rejected: path ... is outside ... GitOps outputs`,优先按 live mirror 控制面漂移处理:先执行 `hwlab g14 git-mirror apply --confirm` 重新应用当前 `devops-infra/git-mirror.yaml` hook/ConfigMap,再执行 `hwlab g14 git-mirror sync --confirm --wait` 复核 refs;失败的同名 PipelineRun 只能通过 `hwlab g14 control-plane cleanup-runs --lane <lane> --pipeline-run <name> --confirm` 受控清理后重试,不要用原生 `kubectl delete` 或手工改 mirror hook。修复后仍必须用 `control-plane status --pipeline-run <name>` 和 `git-mirror status` 分别确认 runtime closeout 与 GitHub flush。
|
||||
如果 PipelineRun 的 `gitops-promote` 阶段报 git mirror 控制面漂移或 refs 不一致,先执行 `hwlab g14 git-mirror apply --confirm` 重新应用当前 `devops-infra/git-mirror.yaml` hook/ConfigMap,再执行 `hwlab g14 git-mirror sync --confirm --wait` 复核 refs;失败的同名 PipelineRun 只能通过 `hwlab g14 control-plane cleanup-runs --lane <lane> --pipeline-run <name> --confirm` 受控清理后重试,不要用原生 `kubectl delete` 或手工改 mirror hook。修复后仍必须用 `control-plane status --pipeline-run <name>` 和 `git-mirror status` 分别确认 runtime closeout 与 GitHub flush。
|
||||
- `platform-infra sub2api plan|apply|status|validate|codex-pool` 是 G14 `platform-infra` namespace 内 Sub2API 的受控入口。镜像版本由 `config/platform-infra/sub2api.yaml` 控制,Codex 上游池、统一 API key Secret、FRP 公网端口和 master `~/.codex` 消费端由 `config/platform-infra/sub2api-codex-pool.yaml` 控制;`codex-pool sync --confirm` 从 YAML 指定的 `~/.codex/config.toml*`/`auth.json*` 导入四个上游账号到同一 Sub2API group,统一消费 key 固定存放在 `platform-infra/sub2api-codex-pool-api-key.API_KEY`。`codex-pool expose --confirm` 只新增明确 FRPS allow port 和 `platform-infra/sub2api-frpc`,不得扩大端口段;`codex-pool configure-local --confirm` 先把当前 `~/.codex/config.toml`、`auth.json` 备份成 `*.pre-sub2api`,再把默认 provider 指向 master 本机 FRP 入口。所有输出只允许打印 key preview/fingerprint、字节数和 Secret 路径,不打印完整 API key。
|
||||
- `hwlab g14 observability status|apply|query|targets|boundary|closeout [--lane v02] [--promql <expr>] [--expect-count N] [--expect-value V] [--dry-run|--confirm]` 是 G14 `devops-infra` 共享监控基础设施和 HWLAB v0.2 监控 closeout 的受控入口。`apply` 固定安装 Prometheus Operator `v0.91.0`、Prometheus `v3.12.0`、Prometheus 发现 RBAC、`devops-infra` 内 Prometheus 实例和 ClusterIP query Service,并给被允许发现的 workload namespace 打低风险 label;它不把 Prometheus、Grafana 或 Alertmanager 部署到 `hwlab-v02`,也不接管 HWLAB runtime Deployment/Service。`status` 只读汇总 CRD、operator Deployment、Prometheus CR/pod/service、`hwlab-v02` ServiceMonitor/PrometheusRule 和 bounded `up` 查询;`query` 只通过 Kubernetes service proxy 查询 Prometheus,支持 `--expect-count` / `--expect-value` 输出 `assertion`、bad values 和 missing/extra series;`targets` 汇总 ServiceMonitor/PrometheusRule、metrics sidecar readiness/restart、三层指标值和 `metrics.k8s.io` 当前 CPU/内存资源快照;`boundary` 验证 workload namespace 没有 Prometheus/Alertmanager,并对 `19666/19667` 公网 `/metrics` 做负向验证;`closeout` 聚合平台 ready、scrape reachable、sidecar serving、business health probe、resource snapshot、namespace boundary 和 public metrics exposure 语义结论。长期边界见 `docs/reference/g14-observability-infra.md`。
|
||||
- `hwlab g14 tools-image status|build --name ci-node-tools --tag <tag> [--dockerfile deploy/ci/hwlab-ci-node-tools.Dockerfile] [--dry-run|--confirm]` 是 G14 固定 HWLAB CI tools image 的受控 host build/push 入口;构建和 push 只发生在 G14 host 与本地 registry,不在 master server 构建,也不把 `apk add`/runtime install 塞进 Tekton PipelineRun。
|
||||
- `trans gh:/owner/repo ...` 把 GitHub issue/PR 映射成只读/受控写入的虚拟文本目录,适合日报、PR 正文和 issue 正文的小补丁维护:`trans gh:/pikasTech/HWLAB ls` 展示 `pr/` 与 `issue/`,`trans gh:/pikasTech/HWLAB/pr ls [--limit N] [--full]` 和 `trans gh:/pikasTech/HWLAB/issue ls [--limit N] [--full]` 展示条目状态、楼层数、正文长度和标题,`trans gh:/pikasTech/HWLAB/pr/507 ls` 展示单个 PR 的一楼正文文件,`trans gh:/pikasTech/HWLAB/505/1 cat|rg|patch-apply` 兼容旧式 issue/PR number route。`patch-apply` 使用 UniDesk 默认 apply-patch v2 的虚拟文件 executor,把正文一楼映射为 `body.md`,写回仍走 `bun scripts/cli.ts gh issue/pr update` 的 guard/concurrency 规则;`rm` 对正文一楼结构化拒绝,避免误删 issue/PR 正文。大正文读取必须展开 UniDesk gh dump 文件,否则 `cat/rg/patch-apply` 会误读为空,这是 `gh:` 虚拟文件接口的 P0 可见性契约。
|
||||
|
||||
@@ -51,7 +51,7 @@ PostgreSQL 只监听 G14 host loopback 与 k3s pod 可达的 node gateway 地址
|
||||
- `Endpoints/g14-platform-postgres`
|
||||
- `EndpointSlice/g14-platform-postgres-host`
|
||||
|
||||
目标地址固定为 `10.42.0.1:5432`。当前 G14 k3s service path 需要同时保留 legacy `Endpoints` 与 `EndpointSlice`;只保留手写 EndpointSlice 时 ClusterIP 转发可能不可用。移除 `Endpoints` 之前必须先用运行面连接验证证明 kube-proxy/EndpointSlice 路径已经稳定。
|
||||
目标地址固定为 `10.42.0.1:5432`。bridge 最低验证标准是 namespace-local `Service/g14-platform-postgres` 存在,并且 `Endpoints` 或 `EndpointSlice` 至少一条路径可用;最终仍以 runtime `/health/live` 和 `hwlab nodes control-plane status` 的真实连接结果为准,不把某一种端点对象形态做成额外门禁。
|
||||
|
||||
业务 SecretRef 固定使用现有应用 Secret 名称,不迁移为共享明文配置:
|
||||
|
||||
@@ -59,7 +59,7 @@ PostgreSQL 只监听 G14 host loopback 与 k3s pod 可达的 node gateway 地址
|
||||
- `hwlab-v03-openfga/datastore-uri` 指向 `openfga_v03`。
|
||||
- `hwpod-v03-db/database-url` 指向 `hwpod_v03`。
|
||||
|
||||
这些 SecretRef 的源凭据来自 G14 host 的 platform DB credentials file,不再从旧 lane-local Postgres Secret 派生。迁移完成后,不要用旧的 repo-owned PostgreSQL bootstrap 路径轮换 `hwlab-cloud-api-v03-db` 或 `hwlab-v03-openfga`。如需重复轮换,必须先补齐平台 DB 专用 UniDesk CLI,让它从 host 凭据文件读取、写入 k3s Secret,并输出 redacted 状态;不得把手写 `kubectl create secret` 沉淀成长期流程。
|
||||
这些 SecretRef 的源凭据来自 G14 host 的 platform DB credentials file,不再从旧 lane-local Postgres Secret 派生。迁移完成后,旧的 repo-owned PostgreSQL bootstrap/ensure 路径已经删除,不能用于轮换 `hwlab-cloud-api-v03-db` 或 `hwlab-v03-openfga`。如需重复轮换,必须先补齐平台 DB 专用 UniDesk CLI,让它从 host 凭据文件读取、写入 k3s Secret,并输出 redacted 状态;不得把手写 `kubectl create secret` 或旧 StatefulSet ensure 路径沉淀成长期流程。
|
||||
|
||||
当前运行面验证使用 control-plane status 和 live health:
|
||||
|
||||
@@ -87,7 +87,7 @@ Argo Application `hwlab-node-v03` 应显示 `Synced/Healthy`,runtime workloads
|
||||
|
||||
## 旧自有 DB 清理
|
||||
|
||||
迁移到平台 DB 后,v0.3 自有 PostgreSQL StatefulSet、Service、ConfigMap、Secret 和 PVC 都必须清理。GitOps 负责移除 desired-state workload;PVC 和旧 admin Secret 由受控 CLI 清理:
|
||||
迁移到平台 DB 后,v0.3 自有 PostgreSQL StatefulSet、Service、ConfigMap、Secret 和 PVC 都必须清理。GitOps render 不再生成旧 `postgres.yaml`;如果运行面仍残留旧对象,直接用受控 CLI 清理精确命名资源,不保留兼容路径:
|
||||
|
||||
```bash
|
||||
bun scripts/cli.ts hwlab nodes secret cleanup-owned-postgres --node G14 --lane v03 --dry-run
|
||||
@@ -97,10 +97,9 @@ bun scripts/cli.ts hwlab nodes secret cleanup-owned-postgres --node G14 --lane v
|
||||
该入口只允许 v0.3+ lane 使用,默认 dry-run。它必须先确认:
|
||||
|
||||
- `g14-platform-postgres` Service 存在。
|
||||
- 旧 `hwlab-v03-postgres` StatefulSet 不存在。
|
||||
- 清理目标仅为旧 `hwlab-v03-postgres` Secret 和 `data-hwlab-v03-postgres-0` PVC。
|
||||
- 清理目标仅为旧 `hwlab-v03-postgres` StatefulSet、Service、ConfigMap、Secret 和 `data-hwlab-v03-postgres-0` PVC。
|
||||
|
||||
它不得删除平台 database、平台 role、当前 `hwlab-cloud-api-v03-db`、当前 `hwlab-v03-openfga`、OpenFGA/Cloud API Deployment 或 GitOps desired state。PVC 删除前必须已经有平台库备份或迁移 dump 可用;迁移 dump 归档在 `/var/backups/g14-platform-db/migration-*`。
|
||||
它不得删除平台 database、平台 role、当前 `hwlab-cloud-api-v03-db`、当前 `hwlab-v03-openfga`、OpenFGA/Cloud API Deployment 或当前 platform DB bridge。PVC 删除前必须已经有平台库备份或迁移 dump 可用;迁移 dump 归档在 `/var/backups/g14-platform-db/migration-*`。
|
||||
|
||||
清理后用以下只读检查确认旧自有 DB 不再存在:
|
||||
|
||||
|
||||
@@ -164,12 +164,14 @@ assertCondition(
|
||||
hwlabHelpUsage.some((line) => line.includes("hwlab nodes secret status --node G14 --lane v03 --name hwlab-v03-code-agent-provider"))
|
||||
&& hwlabHelpUsage.some((line) => line.includes("hwlab nodes secret ensure --node G14 --lane v03 --name hwlab-v03-code-agent-provider --confirm"))
|
||||
&& hwlabHelpUsage.some((line) => line.includes("hwlab nodes secret status --node G14 --lane v03 --name hwlab-cloud-api-v03-db"))
|
||||
&& hwlabHelpUsage.some((line) => line.includes("hwlab nodes secret ensure --node G14 --lane v03 --name hwlab-cloud-api-v03-db --confirm"))
|
||||
&& hwlabHelpUsage.some((line) => line.includes("hwlab nodes secret cleanup-owned-postgres --node G14 --lane v03 --dry-run"))
|
||||
&& hwlabNodeHelpJson.includes("hwlab nodes secret status --node G14 --lane v03 --name hwlab-v03-code-agent-provider")
|
||||
&& hwlabNodeHelpJson.includes("hwlab nodes secret ensure --node G14 --lane v03 --name hwlab-v03-code-agent-provider --confirm")
|
||||
&& hwlabNodeHelpJson.includes("hwlab nodes secret status --node G14 --lane v03 --name hwlab-cloud-api-v03-db")
|
||||
&& hwlabNodeHelpJson.includes("hwlab nodes secret ensure --node G14 --lane v03 --name hwlab-cloud-api-v03-db --confirm"),
|
||||
"v0.3 node-scoped secret help must expose the controlled code-agent provider and cloud-api DB SecretRef bootstrap paths",
|
||||
&& hwlabNodeHelpJson.includes("hwlab nodes secret cleanup-owned-postgres --node G14 --lane v03 --dry-run")
|
||||
&& !hwlabNodeHelpJson.includes("hwlab nodes secret ensure --node G14 --lane v03 --name hwlab-cloud-api-v03-db --confirm")
|
||||
&& !hwlabNodeHelpJson.includes("hwlab nodes secret ensure --node G14 --lane v03 --name hwlab-v03-openfga --confirm"),
|
||||
"v0.3 node-scoped secret help must expose provider bootstrap plus platform DB status/cleanup without old DB ensure paths",
|
||||
{ hwlabHelpUsage, hwlabNodeHelp: hwlabNodeHelp() },
|
||||
);
|
||||
const cloudApiDbStatus = nodeSecretStatusFromTextForTest([
|
||||
@@ -177,44 +179,96 @@ const cloudApiDbStatus = nodeSecretStatusFromTextForTest([
|
||||
"secret\thwlab-cloud-api-v03-db",
|
||||
"key\tdatabase-url",
|
||||
"preset\tcloud-api-db",
|
||||
"action\tensured",
|
||||
"dryRun\tfalse",
|
||||
"mutation\ttrue",
|
||||
"beforeExists\tno",
|
||||
"beforePostgresSecretExists\tyes",
|
||||
"beforeDatabaseUrlPresent\tno",
|
||||
"beforeDatabaseUrlBytes\t0",
|
||||
"action\tobserved",
|
||||
"dryRun\ttrue",
|
||||
"mutation\tfalse",
|
||||
"platformDbMode\ttrue",
|
||||
"afterExists\tyes",
|
||||
"afterDatabaseUrlPresent\tyes",
|
||||
"afterDatabaseUrlBytes\t112",
|
||||
"postgresAdminSecretPresent\tyes",
|
||||
"postgresSecret\thwlab-v03-postgres",
|
||||
"afterDatabaseUrlBytes\t172",
|
||||
"legacyPostgresSecret\thwlab-v03-postgres",
|
||||
"legacyPostgresSecretExists\tno",
|
||||
"platformService\tg14-platform-postgres",
|
||||
"platformServiceExists\tyes",
|
||||
"platformEndpointsExists\tyes",
|
||||
"platformEndpointSliceExists\tyes",
|
||||
"dbName\thwlab_v03",
|
||||
"dbUser\thwlab_v03",
|
||||
"dbHost\thwlab-v03-postgres.hwlab-v03.svc.cluster.local",
|
||||
"dbRoleExistsBefore\tt",
|
||||
"dbDatabaseExistsBefore\tt",
|
||||
"dbProbeExitCodeBefore\t0",
|
||||
"dbRoleExistsAfter\tt",
|
||||
"dbDatabaseExistsAfter\tt",
|
||||
"dbProbeExitCodeAfter\t0",
|
||||
"cloudApiDeployment\thwlab-cloud-api",
|
||||
"applyExitCode\t0",
|
||||
"dbEnsureExitCode\t0",
|
||||
"rolloutRestartExitCode\t0",
|
||||
"rolloutStatusExitCode\t0",
|
||||
"dbUser\thwlab_v03_app",
|
||||
"dbHost\tg14-platform-postgres.hwlab-v03.svc.cluster.local",
|
||||
"dbHostMatchesPlatform\tyes",
|
||||
"dbNameMatchesExpected\tyes",
|
||||
"dbUserMatchesExpected\tyes",
|
||||
].join("\n"), true, 0, "");
|
||||
assertCondition(
|
||||
record(cloudApiDbStatus).ok === true
|
||||
&& record(cloudApiDbStatus).valuesRedacted === true
|
||||
&& record(cloudApiDbStatus).platformDbMode === true
|
||||
&& record(record(cloudApiDbStatus).after).exists === true
|
||||
&& record(record(record(cloudApiDbStatus).after).databaseUrl).valueBytes === 112
|
||||
&& record(cloudApiDbStatus).cloudApiDeployment === "hwlab-cloud-api"
|
||||
&& record(record(record(cloudApiDbStatus).after).databaseUrl).valueBytes === 172
|
||||
&& record(record(cloudApiDbStatus).legacyPostgresSecret).exists === false
|
||||
&& record(record(cloudApiDbStatus).platformService).name === "g14-platform-postgres"
|
||||
&& record(record(cloudApiDbStatus).platformService).endpointsExist === true
|
||||
&& record(cloudApiDbStatus).dbUser === "hwlab_v03_app"
|
||||
&& record(cloudApiDbStatus).dbHost === "g14-platform-postgres.hwlab-v03.svc.cluster.local"
|
||||
&& !JSON.stringify(cloudApiDbStatus).includes("postgres://")
|
||||
&& !JSON.stringify(cloudApiDbStatus).includes("password"),
|
||||
"cloud-api DB Secret status must be redacted while proving DB SecretRef and runtime database are present",
|
||||
"cloud-api DB Secret status must be redacted while proving native platform DB SecretRef and bridge are present",
|
||||
cloudApiDbStatus,
|
||||
);
|
||||
const openFgaPlatformStatus = nodeSecretStatusFromTextForTest([
|
||||
"namespace\thwlab-v03",
|
||||
"secret\thwlab-v03-openfga",
|
||||
"key\tdatastore-uri",
|
||||
"preset\topenfga",
|
||||
"action\tobserved",
|
||||
"dryRun\ttrue",
|
||||
"mutation\tfalse",
|
||||
"platformDbMode\ttrue",
|
||||
"afterExists\tyes",
|
||||
"afterDatastoreUriPresent\tyes",
|
||||
"afterDatastoreUriBytes\t176",
|
||||
"afterAuthnPresent\tyes",
|
||||
"afterAuthnBytes\t64",
|
||||
"afterPostgresPasswordPresent\tyes",
|
||||
"afterPostgresPasswordBytes\t48",
|
||||
"legacyPostgresSecret\thwlab-v03-postgres",
|
||||
"legacyPostgresSecretExists\tno",
|
||||
"platformService\tg14-platform-postgres",
|
||||
"platformServiceExists\tyes",
|
||||
"platformEndpointsExists\tyes",
|
||||
"platformEndpointSliceExists\tyes",
|
||||
"dbName\topenfga_v03",
|
||||
"dbUser\topenfga_v03_app",
|
||||
"dbHost\tg14-platform-postgres.hwlab-v03.svc.cluster.local",
|
||||
"dbHostMatchesPlatform\tyes",
|
||||
"dbNameMatchesExpected\tyes",
|
||||
"dbUserMatchesExpected\tyes",
|
||||
].join("\n"), true, 0, "");
|
||||
assertCondition(
|
||||
record(openFgaPlatformStatus).ok === true
|
||||
&& record(openFgaPlatformStatus).platformDbMode === true
|
||||
&& record(record(openFgaPlatformStatus).legacyPostgresSecret).exists === false
|
||||
&& record(openFgaPlatformStatus).dbName === "openfga_v03"
|
||||
&& record(openFgaPlatformStatus).dbUser === "openfga_v03_app"
|
||||
&& !JSON.stringify(openFgaPlatformStatus).includes("postgres://")
|
||||
&& !JSON.stringify(openFgaPlatformStatus).includes("password"),
|
||||
"OpenFGA Secret status must validate native platform DB datastore-uri without requiring the old lane-local Postgres Secret",
|
||||
openFgaPlatformStatus,
|
||||
);
|
||||
const removedCloudApiDbEnsure = runCommand(["bun", "scripts/cli.ts", "hwlab", "nodes", "secret", "ensure", "--node", "G14", "--lane", "v03", "--name", "hwlab-cloud-api-v03-db", "--dry-run"], process.cwd(), { timeoutMs: 30_000 });
|
||||
assertCondition(
|
||||
removedCloudApiDbEnsure.exitCode !== 0
|
||||
&& `${removedCloudApiDbEnsure.stdout}\n${removedCloudApiDbEnsure.stderr}`.includes("was removed after native platform DB migration"),
|
||||
"v0.3 cloud-api DB ensure must reject the removed lane-local Postgres path",
|
||||
removedCloudApiDbEnsure,
|
||||
);
|
||||
const removedOpenFgaEnsure = runCommand(["bun", "scripts/cli.ts", "hwlab", "nodes", "secret", "ensure", "--node", "G14", "--lane", "v03", "--name", "hwlab-v03-openfga", "--dry-run"], process.cwd(), { timeoutMs: 30_000 });
|
||||
assertCondition(
|
||||
removedOpenFgaEnsure.exitCode !== 0
|
||||
&& `${removedOpenFgaEnsure.stdout}\n${removedOpenFgaEnsure.stderr}`.includes("was removed after native platform DB migration"),
|
||||
"v0.3 OpenFGA ensure must reject the removed lane-local Postgres path",
|
||||
removedOpenFgaEnsure,
|
||||
);
|
||||
const codeAgentProviderStatus = nodeSecretStatusFromTextForTest([
|
||||
"namespace\thwlab-v03",
|
||||
"secret\thwlab-v03-code-agent-provider",
|
||||
|
||||
@@ -10216,14 +10216,13 @@ export function hwlabG14Help(): Record<string, unknown> {
|
||||
"bun scripts/cli.ts hwlab g14 secret ensure --lane v02 --name hwlab-v02-openfga --dry-run",
|
||||
"bun scripts/cli.ts hwlab g14 secret ensure --lane v02 --name hwlab-v02-openfga --confirm",
|
||||
"bun scripts/cli.ts hwlab nodes secret status --node G14 --lane v03 --name hwlab-v03-openfga",
|
||||
"bun scripts/cli.ts hwlab nodes secret ensure --node G14 --lane v03 --name hwlab-v03-openfga --dry-run",
|
||||
"bun scripts/cli.ts hwlab nodes secret ensure --node G14 --lane v03 --name hwlab-v03-openfga --confirm",
|
||||
"bun scripts/cli.ts hwlab g14 secret status --lane v02 --name hwlab-v02-master-server-admin-api-key",
|
||||
"bun scripts/cli.ts hwlab g14 secret ensure --lane v02 --name hwlab-v02-master-server-admin-api-key --confirm",
|
||||
"bun scripts/cli.ts hwlab nodes secret status --node G14 --lane v03 --name hwlab-v03-master-server-admin-api-key",
|
||||
"bun scripts/cli.ts hwlab nodes secret ensure --node G14 --lane v03 --name hwlab-v03-master-server-admin-api-key --confirm",
|
||||
"bun scripts/cli.ts hwlab nodes secret status --node G14 --lane v03 --name hwlab-cloud-api-v03-db",
|
||||
"bun scripts/cli.ts hwlab nodes secret ensure --node G14 --lane v03 --name hwlab-cloud-api-v03-db --confirm",
|
||||
"bun scripts/cli.ts hwlab nodes secret cleanup-owned-postgres --node G14 --lane v03 --dry-run",
|
||||
"bun scripts/cli.ts hwlab nodes secret cleanup-owned-postgres --node G14 --lane v03 --confirm",
|
||||
"bun scripts/cli.ts hwlab nodes secret status --node G14 --lane v03 --name hwlab-v03-code-agent-provider",
|
||||
"bun scripts/cli.ts hwlab nodes secret ensure --node G14 --lane v03 --name hwlab-v03-code-agent-provider --confirm",
|
||||
"bun scripts/cli.ts hwlab g14 secret delete --lane v02 --name <obsolete-hwlab-v02-secret> --dry-run",
|
||||
|
||||
+474
-25
@@ -25,6 +25,8 @@ interface RuntimeSecretSpec {
|
||||
node: string;
|
||||
lane: string;
|
||||
namespace: string;
|
||||
platformDb: boolean;
|
||||
platformPostgresService: string;
|
||||
postgresSecret: string;
|
||||
postgresStatefulSet: string;
|
||||
postgresAdminUser: string;
|
||||
@@ -80,12 +82,11 @@ export function hwlabNodeHelp(): Record<string, unknown> {
|
||||
"bun scripts/cli.ts hwlab nodes control-plane apply --node G14 --lane v03 --dry-run",
|
||||
"bun scripts/cli.ts hwlab nodes control-plane refresh --node G14 --lane v03 --confirm",
|
||||
"bun scripts/cli.ts hwlab nodes control-plane trigger-current --node G14 --lane v03 --confirm",
|
||||
"bun scripts/cli.ts hwlab nodes control-plane allow-endpoint-bridge --node G14 --lane v03 --confirm",
|
||||
"bun scripts/cli.ts hwlab nodes git-mirror status --node G14 --lane v03",
|
||||
"bun scripts/cli.ts hwlab nodes secret status --node G14 --lane v03 --name hwlab-v03-openfga",
|
||||
"bun scripts/cli.ts hwlab nodes secret ensure --node G14 --lane v03 --name hwlab-v03-openfga --confirm",
|
||||
"bun scripts/cli.ts hwlab nodes secret ensure --node G14 --lane v03 --name hwlab-v03-master-server-admin-api-key --confirm",
|
||||
"bun scripts/cli.ts hwlab nodes secret status --node G14 --lane v03 --name hwlab-cloud-api-v03-db",
|
||||
"bun scripts/cli.ts hwlab nodes secret ensure --node G14 --lane v03 --name hwlab-cloud-api-v03-db --confirm",
|
||||
"bun scripts/cli.ts hwlab nodes secret cleanup-owned-postgres --node G14 --lane v03 --dry-run",
|
||||
"bun scripts/cli.ts hwlab nodes secret cleanup-owned-postgres --node G14 --lane v03 --confirm",
|
||||
"bun scripts/cli.ts hwlab nodes secret status --node G14 --lane v03 --name hwlab-v03-code-agent-provider",
|
||||
@@ -96,6 +97,9 @@ export function hwlabNodeHelp(): Record<string, unknown> {
|
||||
|
||||
async function runNodeDelegatedDomain(config: Config, domain: DelegatedNodeDomain, args: string[]): Promise<Record<string, unknown>> {
|
||||
const scoped = parseNodeScopedDelegatedOptions(domain, args);
|
||||
if (domain === "control-plane" && scoped.action === "allow-endpoint-bridge") {
|
||||
return runNodeEndpointBridge(scoped);
|
||||
}
|
||||
if (domain === "control-plane" && scoped.action === "trigger-current" && scoped.confirm && !scoped.dryRun && !scoped.wait) {
|
||||
return startNodeDelegatedJob(scoped);
|
||||
}
|
||||
@@ -265,6 +269,9 @@ function parseSecretOptions(args: string[]): NodeSecretOptions {
|
||||
}
|
||||
if (name === spec.cloudApiDbSecret) {
|
||||
if (key !== undefined && key !== spec.cloudApiDbKey) throw new Error(`secret ${name} supports only key ${spec.cloudApiDbKey}`);
|
||||
if (actionRaw === "ensure" && spec.platformDb) {
|
||||
throw new Error(`secret ensure for ${name} on --lane ${lane} was removed after native platform DB migration; use status plus platform DB SecretRef rotation CLI when it exists`);
|
||||
}
|
||||
return {
|
||||
action: actionRaw,
|
||||
node,
|
||||
@@ -283,6 +290,9 @@ function parseSecretOptions(args: string[]): NodeSecretOptions {
|
||||
if (key !== undefined && key !== OPENFGA_AUTHN_KEY && key !== OPENFGA_DATASTORE_URI_KEY && key !== OPENFGA_POSTGRES_PASSWORD_KEY) {
|
||||
throw new Error(`secret ${name} supports keys ${OPENFGA_AUTHN_KEY}, ${OPENFGA_DATASTORE_URI_KEY}, and ${OPENFGA_POSTGRES_PASSWORD_KEY}`);
|
||||
}
|
||||
if (actionRaw === "ensure" && spec.platformDb) {
|
||||
throw new Error(`secret ensure for ${name} on --lane ${lane} was removed after native platform DB migration; use status plus platform DB SecretRef rotation CLI when it exists`);
|
||||
}
|
||||
return {
|
||||
action: actionRaw,
|
||||
node,
|
||||
@@ -298,23 +308,29 @@ function parseSecretOptions(args: string[]): NodeSecretOptions {
|
||||
|
||||
function runtimeSecretSpec(input: { node: string; lane: string }): RuntimeSecretSpec {
|
||||
const namespace = `hwlab-${input.lane}`;
|
||||
const platformDb = /^v0*[3-9]\d*$/.test(input.lane);
|
||||
const platformPostgresService = "g14-platform-postgres";
|
||||
const legacyPostgresHost = `${namespace}-postgres.${namespace}.svc.cluster.local`;
|
||||
const platformPostgresHost = `${platformPostgresService}.${namespace}.svc.cluster.local`;
|
||||
return {
|
||||
node: input.node,
|
||||
lane: input.lane,
|
||||
namespace,
|
||||
platformDb,
|
||||
platformPostgresService,
|
||||
postgresSecret: `${namespace}-postgres`,
|
||||
postgresStatefulSet: `${namespace}-postgres`,
|
||||
postgresAdminUser: `hwlab_${input.lane}`,
|
||||
openFgaSecret: `${namespace}-openfga`,
|
||||
openFgaDbName: "hwlab_openfga",
|
||||
openFgaDbUser: "hwlab_openfga",
|
||||
openFgaDbHost: `${namespace}-postgres.${namespace}.svc.cluster.local`,
|
||||
openFgaDbName: platformDb ? `openfga_${input.lane}` : "hwlab_openfga",
|
||||
openFgaDbUser: platformDb ? `openfga_${input.lane}_app` : "hwlab_openfga",
|
||||
openFgaDbHost: platformDb ? platformPostgresHost : legacyPostgresHost,
|
||||
masterAdminApiKeySecret: `${namespace}-master-server-admin-api-key`,
|
||||
cloudApiDbSecret: `hwlab-cloud-api-${input.lane}-db`,
|
||||
cloudApiDbKey: CLOUD_API_DB_KEY,
|
||||
cloudApiDbName: `hwlab_${input.lane}`,
|
||||
cloudApiDbUser: `hwlab_${input.lane}`,
|
||||
cloudApiDbHost: `${namespace}-postgres.${namespace}.svc.cluster.local`,
|
||||
cloudApiDbUser: platformDb ? `hwlab_${input.lane}_app` : `hwlab_${input.lane}`,
|
||||
cloudApiDbHost: platformDb ? platformPostgresHost : legacyPostgresHost,
|
||||
cloudApiDeployment: "hwlab-cloud-api",
|
||||
codeAgentProviderSecret: `${namespace}-code-agent-provider`,
|
||||
codeAgentProviderSourceNamespace: CODE_AGENT_PROVIDER_SOURCE_NAMESPACE,
|
||||
@@ -329,11 +345,11 @@ function runNodeSecret(options: NodeSecretOptions): Record<string, unknown> {
|
||||
? readMasterAdminApiKey().key
|
||||
: "";
|
||||
const script = options.preset === "openfga"
|
||||
? openFgaSecretScript(options, spec)
|
||||
? spec.platformDb ? platformDbSecretStatusScript(options, spec) : openFgaSecretScript(options, spec)
|
||||
: options.preset === "master-server-admin-api-key"
|
||||
? masterAdminApiKeySecretScript(options, spec)
|
||||
: options.preset === "cloud-api-db"
|
||||
? cloudApiDbSecretScript(options, spec)
|
||||
? spec.platformDb ? platformDbSecretStatusScript(options, spec) : cloudApiDbSecretScript(options, spec)
|
||||
: options.preset === "owned-postgres-cleanup"
|
||||
? ownedPostgresCleanupScript(options, spec)
|
||||
: codeAgentProviderSecretScript(options, spec);
|
||||
@@ -356,26 +372,220 @@ function runNodeSecret(options: NodeSecretOptions): Record<string, unknown> {
|
||||
mutation: status.mutation === true,
|
||||
result: compactCommandResult(result),
|
||||
valuesRedacted: true,
|
||||
next: ok && options.action === "status" ? undefined : {
|
||||
ensure: options.action === "cleanup-owned-postgres"
|
||||
? `bun scripts/cli.ts hwlab nodes secret cleanup-owned-postgres --node ${options.node} --lane ${options.lane} --confirm`
|
||||
: `bun scripts/cli.ts hwlab nodes secret ensure --node ${options.node} --lane ${options.lane} --name ${options.name}${options.key ? ` --key ${options.key}` : ""} --confirm`,
|
||||
},
|
||||
next: ok && options.action === "status" ? undefined : nextSecretCommand(options, spec),
|
||||
};
|
||||
}
|
||||
|
||||
function nextSecretCommand(options: NodeSecretOptions, spec: RuntimeSecretSpec): Record<string, string> {
|
||||
if (options.action === "cleanup-owned-postgres") {
|
||||
return { ensure: `bun scripts/cli.ts hwlab nodes secret cleanup-owned-postgres --node ${options.node} --lane ${options.lane} --confirm` };
|
||||
}
|
||||
if (spec.platformDb && (options.preset === "cloud-api-db" || options.preset === "openfga")) {
|
||||
return {
|
||||
status: `bun scripts/cli.ts hwlab nodes secret status --node ${options.node} --lane ${options.lane} --name ${options.name}${options.key ? ` --key ${options.key}` : ""}`,
|
||||
controlPlaneStatus: `bun scripts/cli.ts hwlab nodes control-plane status --node ${options.node} --lane ${options.lane}`,
|
||||
};
|
||||
}
|
||||
return { ensure: `bun scripts/cli.ts hwlab nodes secret ensure --node ${options.node} --lane ${options.lane} --name ${options.name}${options.key ? ` --key ${options.key}` : ""} --confirm` };
|
||||
}
|
||||
|
||||
function runTransScript(node: string, script: string, input: string, timeoutSeconds: number): CommandResult {
|
||||
return runCommand(["/root/.local/bin/trans", `${node}:k3s`, "script", "--", script], repoRoot, { input, timeoutMs: timeoutSeconds * 1000 });
|
||||
}
|
||||
|
||||
function runNodeEndpointBridge(options: ReturnType<typeof parseNodeScopedDelegatedOptions>): Record<string, unknown> {
|
||||
if (options.dryRun && options.confirm) throw new Error("control-plane allow-endpoint-bridge accepts only one of --dry-run or --confirm");
|
||||
const dryRun = options.dryRun || !options.confirm;
|
||||
const result = runTransScript(options.node, endpointBridgeScript({ lane: options.lane, dryRun }), "", options.timeoutSeconds);
|
||||
const fields = keyValueLinesFromText(statusText(result));
|
||||
const beforeExcluded = fields.beforeEndpointResourcesExcluded === "yes";
|
||||
const beforeIgnored = fields.beforeEndpointsIgnoreUpdates === "yes" || fields.beforeEndpointSliceIgnoreUpdates === "yes";
|
||||
const afterExcluded = fields.afterEndpointResourcesExcluded === "yes";
|
||||
const afterIgnored = fields.afterEndpointsIgnoreUpdates === "yes" || fields.afterEndpointSliceIgnoreUpdates === "yes";
|
||||
const ok = result.exitCode === 0 && !afterExcluded && !afterIgnored;
|
||||
return {
|
||||
ok: dryRun ? result.exitCode === 0 : ok,
|
||||
command: "hwlab nodes control-plane allow-endpoint-bridge",
|
||||
node: options.node,
|
||||
lane: options.lane,
|
||||
namespace: "argocd",
|
||||
application: fields.application || `hwlab-node-${options.lane}`,
|
||||
mode: dryRun ? "dry-run" : "confirmed-control-plane-update",
|
||||
status: {
|
||||
action: fields.action || null,
|
||||
dryRun,
|
||||
mutation: fields.mutation === "true",
|
||||
before: {
|
||||
endpointResourcesExcluded: beforeExcluded,
|
||||
endpointsIgnoreUpdates: fields.beforeEndpointsIgnoreUpdates === "yes",
|
||||
endpointSliceIgnoreUpdates: fields.beforeEndpointSliceIgnoreUpdates === "yes",
|
||||
},
|
||||
after: {
|
||||
endpointResourcesExcluded: afterExcluded,
|
||||
endpointsIgnoreUpdates: fields.afterEndpointsIgnoreUpdates === "yes",
|
||||
endpointSliceIgnoreUpdates: fields.afterEndpointSliceIgnoreUpdates === "yes",
|
||||
},
|
||||
patchExitCode: numericField(fields.patchExitCode),
|
||||
rolloutRestartExitCode: numericField(fields.rolloutRestartExitCode),
|
||||
rolloutStatusExitCode: numericField(fields.rolloutStatusExitCode),
|
||||
refreshExitCode: numericField(fields.refreshExitCode),
|
||||
exitCode: result.exitCode,
|
||||
stderr: result.exitCode === 0 ? "" : result.stderr.trim().slice(0, 2000),
|
||||
summary: !afterExcluded && !afterIgnored
|
||||
? "Argo tracks HWLAB external Postgres EndpointSlice resources"
|
||||
: "Argo still excludes or ignores HWLAB external Postgres EndpointSlice resources",
|
||||
},
|
||||
result: compactCommandResult(result),
|
||||
};
|
||||
}
|
||||
|
||||
function endpointBridgeScript(options: { lane: HwlabRuntimeLane; dryRun: boolean }): string {
|
||||
const application = `hwlab-node-${options.lane}`;
|
||||
return [
|
||||
"set +e",
|
||||
"namespace=argocd",
|
||||
"configmap=argocd-cm",
|
||||
`application=${shellQuote(application)}`,
|
||||
`dry_run=${shellQuote(options.dryRun ? "true" : "false")}`,
|
||||
"preset=endpoint-bridge-resource-tracking",
|
||||
"cm_data() { kubectl -n \"$namespace\" get configmap \"$configmap\" -o \"go-template={{ index .data \\\"$1\\\" }}\" 2>/dev/null || true; }",
|
||||
"cm_has_key() { kubectl -n \"$namespace\" get configmap \"$configmap\" -o jsonpath=\"{.data.$1}\" >/tmp/hwlab-argocd-cm-key.out 2>/dev/null && [ -s /tmp/hwlab-argocd-cm-key.out ] && printf yes || printf no; }",
|
||||
"endpoint_resources_excluded() { exclusions=$(cm_data resource.exclusions); printf '%s' \"$exclusions\" | grep -Eq '(^|[[:space:]])(Endpoints|EndpointSlice)([[:space:]]|$)' && printf yes || printf no; }",
|
||||
"before_endpoint_resources_excluded=$(endpoint_resources_excluded)",
|
||||
"before_endpoints_ignore_updates=$(cm_has_key 'resource\\.customizations\\.ignoreResourceUpdates\\.Endpoints')",
|
||||
"before_endpoint_slice_ignore_updates=$(cm_has_key 'resource\\.customizations\\.ignoreResourceUpdates\\.discovery\\.k8s\\.io_EndpointSlice')",
|
||||
"action=observed",
|
||||
"mutation=false",
|
||||
"patch_exit=",
|
||||
"rollout_restart_exit=",
|
||||
"rollout_status_exit=",
|
||||
"refresh_exit=",
|
||||
"needs_update=false",
|
||||
"if [ \"$before_endpoint_resources_excluded\" = yes ] || [ \"$before_endpoints_ignore_updates\" = yes ] || [ \"$before_endpoint_slice_ignore_updates\" = yes ]; then needs_update=true; fi",
|
||||
"if [ \"$dry_run\" = true ]; then",
|
||||
" if [ \"$needs_update\" = true ]; then action=would-remove-old-endpoint-exclusions; else action=kept; fi",
|
||||
"elif [ \"$needs_update\" = false ]; then",
|
||||
" action=kept",
|
||||
"else",
|
||||
" patch_file=$(mktemp /tmp/hwlab-argocd-endpoint-bridge.XXXXXX.json)",
|
||||
" python3 - <<'PY' >\"$patch_file\"",
|
||||
"import json",
|
||||
"desired = '''### Internal Kubernetes resources excluded to reduce watch volume",
|
||||
"- apiGroups:",
|
||||
" - coordination.k8s.io",
|
||||
" kinds:",
|
||||
" - Lease",
|
||||
"### Internal Kubernetes Authz/Authn resources excluded to reduce watched events",
|
||||
"- apiGroups:",
|
||||
" - authentication.k8s.io",
|
||||
" - authorization.k8s.io",
|
||||
" kinds:",
|
||||
" - SelfSubjectReview",
|
||||
" - TokenReview",
|
||||
" - LocalSubjectAccessReview",
|
||||
" - SelfSubjectAccessReview",
|
||||
" - SelfSubjectRulesReview",
|
||||
" - SubjectAccessReview",
|
||||
"### Intermediate Certificate Request excluded to reduce watched events",
|
||||
"- apiGroups:",
|
||||
" - certificates.k8s.io",
|
||||
" kinds:",
|
||||
" - CertificateSigningRequest",
|
||||
"- apiGroups:",
|
||||
" - cert-manager.io",
|
||||
" kinds:",
|
||||
" - CertificateRequest",
|
||||
"### Cilium internal resources excluded to reduce UI clutter",
|
||||
"- apiGroups:",
|
||||
" - cilium.io",
|
||||
" kinds:",
|
||||
" - CiliumIdentity",
|
||||
" - CiliumEndpoint",
|
||||
" - CiliumEndpointSlice",
|
||||
"### Kyverno intermediate and reporting resources excluded to reduce watched events",
|
||||
"- apiGroups:",
|
||||
" - kyverno.io",
|
||||
" - reports.kyverno.io",
|
||||
" - wgpolicyk8s.io",
|
||||
" kinds:",
|
||||
" - PolicyReport",
|
||||
" - ClusterPolicyReport",
|
||||
" - EphemeralReport",
|
||||
" - ClusterEphemeralReport",
|
||||
" - AdmissionReport",
|
||||
" - ClusterAdmissionReport",
|
||||
" - BackgroundScanReport",
|
||||
" - ClusterBackgroundScanReport",
|
||||
" - UpdateRequest",
|
||||
"'''",
|
||||
"print(json.dumps({",
|
||||
" 'data': {",
|
||||
" 'resource.exclusions': desired,",
|
||||
" 'resource.customizations.ignoreResourceUpdates.Endpoints': None,",
|
||||
" 'resource.customizations.ignoreResourceUpdates.discovery.k8s.io_EndpointSlice': None,",
|
||||
" }",
|
||||
"}))",
|
||||
"PY",
|
||||
" kubectl -n \"$namespace\" patch configmap \"$configmap\" --type merge --patch-file \"$patch_file\" >/tmp/hwlab-argocd-endpoint-bridge-patch.out 2>/tmp/hwlab-argocd-endpoint-bridge-patch.err",
|
||||
" patch_exit=$?",
|
||||
" rm -f \"$patch_file\"",
|
||||
" if [ \"$patch_exit\" -eq 0 ]; then",
|
||||
" kubectl -n \"$namespace\" rollout restart statefulset/argocd-application-controller >/tmp/hwlab-argocd-endpoint-bridge-rollout-restart.out 2>/tmp/hwlab-argocd-endpoint-bridge-rollout-restart.err",
|
||||
" rollout_restart_exit=$?",
|
||||
" if [ \"$rollout_restart_exit\" -eq 0 ]; then",
|
||||
" kubectl -n \"$namespace\" rollout status statefulset/argocd-application-controller --timeout=180s >/tmp/hwlab-argocd-endpoint-bridge-rollout-status.out 2>/tmp/hwlab-argocd-endpoint-bridge-rollout-status.err",
|
||||
" rollout_status_exit=$?",
|
||||
" fi",
|
||||
" kubectl -n \"$namespace\" annotate application \"$application\" argocd.argoproj.io/refresh=hard --overwrite >/tmp/hwlab-argocd-endpoint-bridge-refresh.out 2>/tmp/hwlab-argocd-endpoint-bridge-refresh.err",
|
||||
" refresh_exit=$?",
|
||||
" if [ \"$rollout_restart_exit\" -ne 0 ]; then action=rollout-restart-failed",
|
||||
" elif [ \"$rollout_status_exit\" -ne 0 ]; then action=rollout-status-failed",
|
||||
" elif [ \"$refresh_exit\" -ne 0 ]; then action=refresh-failed",
|
||||
" else action=removed-old-endpoint-exclusions; mutation=true; fi",
|
||||
" else",
|
||||
" action=patch-failed",
|
||||
" fi",
|
||||
"fi",
|
||||
"after_endpoint_resources_excluded=$(endpoint_resources_excluded)",
|
||||
"after_endpoints_ignore_updates=$(cm_has_key 'resource\\.customizations\\.ignoreResourceUpdates\\.Endpoints')",
|
||||
"after_endpoint_slice_ignore_updates=$(cm_has_key 'resource\\.customizations\\.ignoreResourceUpdates\\.discovery\\.k8s\\.io_EndpointSlice')",
|
||||
"printf 'namespace\\t%s\\n' \"$namespace\"",
|
||||
"printf 'configMap\\t%s\\n' \"$configmap\"",
|
||||
"printf 'application\\t%s\\n' \"$application\"",
|
||||
"printf 'preset\\t%s\\n' \"$preset\"",
|
||||
"printf 'action\\t%s\\n' \"$action\"",
|
||||
"printf 'dryRun\\t%s\\n' \"$dry_run\"",
|
||||
"printf 'mutation\\t%s\\n' \"$mutation\"",
|
||||
"printf 'beforeEndpointResourcesExcluded\\t%s\\n' \"$before_endpoint_resources_excluded\"",
|
||||
"printf 'beforeEndpointsIgnoreUpdates\\t%s\\n' \"$before_endpoints_ignore_updates\"",
|
||||
"printf 'beforeEndpointSliceIgnoreUpdates\\t%s\\n' \"$before_endpoint_slice_ignore_updates\"",
|
||||
"printf 'afterEndpointResourcesExcluded\\t%s\\n' \"$after_endpoint_resources_excluded\"",
|
||||
"printf 'afterEndpointsIgnoreUpdates\\t%s\\n' \"$after_endpoints_ignore_updates\"",
|
||||
"printf 'afterEndpointSliceIgnoreUpdates\\t%s\\n' \"$after_endpoint_slice_ignore_updates\"",
|
||||
"printf 'patchExitCode\\t%s\\n' \"$patch_exit\"",
|
||||
"printf 'rolloutRestartExitCode\\t%s\\n' \"$rollout_restart_exit\"",
|
||||
"printf 'rolloutStatusExitCode\\t%s\\n' \"$rollout_status_exit\"",
|
||||
"printf 'refreshExitCode\\t%s\\n' \"$refresh_exit\"",
|
||||
"if [ \"$dry_run\" != true ] && { [ \"$after_endpoint_resources_excluded\" = yes ] || [ \"$after_endpoints_ignore_updates\" = yes ] || [ \"$after_endpoint_slice_ignore_updates\" = yes ]; }; then exit 46; fi",
|
||||
"if [ -n \"$patch_exit\" ] && [ \"$patch_exit\" != 0 ]; then exit \"$patch_exit\"; fi",
|
||||
"if [ -n \"$rollout_restart_exit\" ] && [ \"$rollout_restart_exit\" != 0 ]; then exit \"$rollout_restart_exit\"; fi",
|
||||
"if [ -n \"$rollout_status_exit\" ] && [ \"$rollout_status_exit\" != 0 ]; then exit \"$rollout_status_exit\"; fi",
|
||||
"if [ -n \"$refresh_exit\" ] && [ \"$refresh_exit\" != 0 ]; then exit \"$refresh_exit\"; fi",
|
||||
].join("\n");
|
||||
}
|
||||
|
||||
function ownedPostgresCleanupScript(options: NodeSecretOptions, spec: RuntimeSecretSpec): string {
|
||||
const pvc = `data-${spec.postgresSecret}-0`;
|
||||
const platformService = "g14-platform-postgres";
|
||||
const postgresService = spec.postgresSecret;
|
||||
const postgresConfigMap = `${spec.postgresSecret}-init`;
|
||||
return [
|
||||
"set +e",
|
||||
`namespace=${shellQuote(spec.namespace)}`,
|
||||
`postgres_secret=${shellQuote(spec.postgresSecret)}`,
|
||||
`postgres_statefulset=${shellQuote(spec.postgresStatefulSet)}`,
|
||||
`postgres_service=${shellQuote(postgresService)}`,
|
||||
`postgres_configmap=${shellQuote(postgresConfigMap)}`,
|
||||
`pvc=${shellQuote(pvc)}`,
|
||||
`platform_service=${shellQuote(platformService)}`,
|
||||
`dry_run=${shellQuote(options.dryRun ? "true" : "false")}`,
|
||||
@@ -388,20 +598,44 @@ function ownedPostgresCleanupScript(options: NodeSecretOptions, spec: RuntimeSec
|
||||
"before_pvc_phase=$(phase_of_pvc)",
|
||||
"before_pv=$(pv_name)",
|
||||
"before_statefulset_exists=$(exists_flag statefulset \"$postgres_statefulset\")",
|
||||
"before_service_exists=$(exists_flag service \"$postgres_service\")",
|
||||
"before_configmap_exists=$(exists_flag configmap \"$postgres_configmap\")",
|
||||
"platform_service_exists=$(exists_flag service \"$platform_service\")",
|
||||
"action=observed",
|
||||
"mutation=false",
|
||||
"delete_statefulset_exit=",
|
||||
"delete_service_exit=",
|
||||
"delete_configmap_exit=",
|
||||
"delete_secret_exit=",
|
||||
"delete_pvc_exit=",
|
||||
"before_any_owned=false",
|
||||
"for flag in \"$before_statefulset_exists\" \"$before_service_exists\" \"$before_configmap_exists\" \"$before_secret_exists\" \"$before_pvc_exists\"; do",
|
||||
" if [ \"$flag\" = yes ]; then before_any_owned=true; fi",
|
||||
"done",
|
||||
"if [ \"$dry_run\" = true ]; then",
|
||||
" if [ \"$before_secret_exists\" = yes ] || [ \"$before_pvc_exists\" = yes ]; then action=would-delete; else action=already-absent; fi",
|
||||
" if [ \"$before_any_owned\" = true ]; then action=would-delete; else action=already-absent; fi",
|
||||
"else",
|
||||
" kubectl -n \"$namespace\" delete statefulset \"$postgres_statefulset\" --ignore-not-found=true >/tmp/hwlab-owned-postgres-statefulset-delete.out 2>/tmp/hwlab-owned-postgres-statefulset-delete.err",
|
||||
" delete_statefulset_exit=$?",
|
||||
" kubectl -n \"$namespace\" delete service \"$postgres_service\" --ignore-not-found=true >/tmp/hwlab-owned-postgres-service-delete.out 2>/tmp/hwlab-owned-postgres-service-delete.err",
|
||||
" delete_service_exit=$?",
|
||||
" kubectl -n \"$namespace\" delete configmap \"$postgres_configmap\" --ignore-not-found=true >/tmp/hwlab-owned-postgres-configmap-delete.out 2>/tmp/hwlab-owned-postgres-configmap-delete.err",
|
||||
" delete_configmap_exit=$?",
|
||||
" kubectl -n \"$namespace\" delete secret \"$postgres_secret\" --ignore-not-found=true >/tmp/hwlab-owned-postgres-secret-delete.out 2>/tmp/hwlab-owned-postgres-secret-delete.err",
|
||||
" delete_secret_exit=$?",
|
||||
" kubectl -n \"$namespace\" delete pvc \"$pvc\" --ignore-not-found=true >/tmp/hwlab-owned-postgres-pvc-delete.out 2>/tmp/hwlab-owned-postgres-pvc-delete.err",
|
||||
" delete_pvc_exit=$?",
|
||||
" if [ \"$delete_secret_exit\" -eq 0 ] && [ \"$delete_pvc_exit\" -eq 0 ]; then",
|
||||
" if [ \"$before_secret_exists\" = yes ] || [ \"$before_pvc_exists\" = yes ]; then action=deleted; mutation=true; else action=already-absent; fi",
|
||||
" for _ in $(seq 1 30); do",
|
||||
" current_statefulset=$(exists_flag statefulset \"$postgres_statefulset\")",
|
||||
" current_service=$(exists_flag service \"$postgres_service\")",
|
||||
" current_configmap=$(exists_flag configmap \"$postgres_configmap\")",
|
||||
" current_secret=$(exists_flag secret \"$postgres_secret\")",
|
||||
" current_pvc=$(exists_flag pvc \"$pvc\")",
|
||||
" if [ \"$current_statefulset\" != yes ] && [ \"$current_service\" != yes ] && [ \"$current_configmap\" != yes ] && [ \"$current_secret\" != yes ] && [ \"$current_pvc\" != yes ]; then break; fi",
|
||||
" sleep 2",
|
||||
" done",
|
||||
" if [ \"$delete_statefulset_exit\" -eq 0 ] && [ \"$delete_service_exit\" -eq 0 ] && [ \"$delete_configmap_exit\" -eq 0 ] && [ \"$delete_secret_exit\" -eq 0 ] && [ \"$delete_pvc_exit\" -eq 0 ]; then",
|
||||
" if [ \"$before_any_owned\" = true ]; then action=deleted; mutation=true; else action=already-absent; fi",
|
||||
" else",
|
||||
" action=delete-failed",
|
||||
" fi",
|
||||
@@ -411,8 +645,13 @@ function ownedPostgresCleanupScript(options: NodeSecretOptions, spec: RuntimeSec
|
||||
"after_pvc_phase=$(phase_of_pvc)",
|
||||
"after_pv=$(pv_name)",
|
||||
"after_statefulset_exists=$(exists_flag statefulset \"$postgres_statefulset\")",
|
||||
"after_service_exists=$(exists_flag service \"$postgres_service\")",
|
||||
"after_configmap_exists=$(exists_flag configmap \"$postgres_configmap\")",
|
||||
"printf 'namespace\\t%s\\n' \"$namespace\"",
|
||||
"printf 'secret\\t%s\\n' \"$postgres_secret\"",
|
||||
"printf 'statefulSet\\t%s\\n' \"$postgres_statefulset\"",
|
||||
"printf 'service\\t%s\\n' \"$postgres_service\"",
|
||||
"printf 'configMap\\t%s\\n' \"$postgres_configmap\"",
|
||||
"printf 'pvc\\t%s\\n' \"$pvc\"",
|
||||
"printf 'preset\\t%s\\n' \"$preset\"",
|
||||
"printf 'action\\t%s\\n' \"$action\"",
|
||||
@@ -423,21 +662,119 @@ function ownedPostgresCleanupScript(options: NodeSecretOptions, spec: RuntimeSec
|
||||
"printf 'beforePvcPhase\\t%s\\n' \"$before_pvc_phase\"",
|
||||
"printf 'beforePersistentVolume\\t%s\\n' \"$before_pv\"",
|
||||
"printf 'beforeStatefulSetExists\\t%s\\n' \"$before_statefulset_exists\"",
|
||||
"printf 'beforeServiceExists\\t%s\\n' \"$before_service_exists\"",
|
||||
"printf 'beforeConfigMapExists\\t%s\\n' \"$before_configmap_exists\"",
|
||||
"printf 'platformServiceExists\\t%s\\n' \"$platform_service_exists\"",
|
||||
"printf 'afterSecretExists\\t%s\\n' \"$after_secret_exists\"",
|
||||
"printf 'afterPvcExists\\t%s\\n' \"$after_pvc_exists\"",
|
||||
"printf 'afterPvcPhase\\t%s\\n' \"$after_pvc_phase\"",
|
||||
"printf 'afterPersistentVolume\\t%s\\n' \"$after_pv\"",
|
||||
"printf 'afterStatefulSetExists\\t%s\\n' \"$after_statefulset_exists\"",
|
||||
"printf 'afterServiceExists\\t%s\\n' \"$after_service_exists\"",
|
||||
"printf 'afterConfigMapExists\\t%s\\n' \"$after_configmap_exists\"",
|
||||
"printf 'deleteStatefulSetExitCode\\t%s\\n' \"$delete_statefulset_exit\"",
|
||||
"printf 'deleteServiceExitCode\\t%s\\n' \"$delete_service_exit\"",
|
||||
"printf 'deleteConfigMapExitCode\\t%s\\n' \"$delete_configmap_exit\"",
|
||||
"printf 'deleteSecretExitCode\\t%s\\n' \"$delete_secret_exit\"",
|
||||
"printf 'deletePvcExitCode\\t%s\\n' \"$delete_pvc_exit\"",
|
||||
"if [ \"$platform_service_exists\" != yes ]; then exit 44; fi",
|
||||
"if [ \"$before_statefulset_exists\" = yes ] || [ \"$after_statefulset_exists\" = yes ]; then exit 45; fi",
|
||||
"if [ \"$after_statefulset_exists\" = yes ] || [ \"$after_service_exists\" = yes ] || [ \"$after_configmap_exists\" = yes ] || [ \"$after_secret_exists\" = yes ] || [ \"$after_pvc_exists\" = yes ]; then exit 45; fi",
|
||||
"if [ -n \"$delete_statefulset_exit\" ] && [ \"$delete_statefulset_exit\" != 0 ]; then exit \"$delete_statefulset_exit\"; fi",
|
||||
"if [ -n \"$delete_service_exit\" ] && [ \"$delete_service_exit\" != 0 ]; then exit \"$delete_service_exit\"; fi",
|
||||
"if [ -n \"$delete_configmap_exit\" ] && [ \"$delete_configmap_exit\" != 0 ]; then exit \"$delete_configmap_exit\"; fi",
|
||||
"if [ -n \"$delete_secret_exit\" ] && [ \"$delete_secret_exit\" != 0 ]; then exit \"$delete_secret_exit\"; fi",
|
||||
"if [ -n \"$delete_pvc_exit\" ] && [ \"$delete_pvc_exit\" != 0 ]; then exit \"$delete_pvc_exit\"; fi",
|
||||
].join("\n");
|
||||
}
|
||||
|
||||
function platformDbSecretStatusScript(options: NodeSecretOptions, spec: RuntimeSecretSpec): string {
|
||||
const isOpenFga = options.preset === "openfga";
|
||||
return [
|
||||
"set +e",
|
||||
`namespace=${shellQuote(spec.namespace)}`,
|
||||
`name=${shellQuote(isOpenFga ? spec.openFgaSecret : spec.cloudApiDbSecret)}`,
|
||||
`database_url_key=${shellQuote(isOpenFga ? OPENFGA_DATASTORE_URI_KEY : spec.cloudApiDbKey)}`,
|
||||
`authn_key=${shellQuote(OPENFGA_AUTHN_KEY)}`,
|
||||
`postgres_password_key=${shellQuote(OPENFGA_POSTGRES_PASSWORD_KEY)}`,
|
||||
`legacy_postgres_secret=${shellQuote(spec.postgresSecret)}`,
|
||||
`platform_service=${shellQuote(spec.platformPostgresService)}`,
|
||||
`platform_host=${shellQuote(spec.platformPostgresService)}`,
|
||||
`platform_host_fqdn=${shellQuote(spec.openFgaDbHost)}`,
|
||||
`db_name=${shellQuote(isOpenFga ? spec.openFgaDbName : spec.cloudApiDbName)}`,
|
||||
`db_user=${shellQuote(isOpenFga ? spec.openFgaDbUser : spec.cloudApiDbUser)}`,
|
||||
`db_host=${shellQuote(isOpenFga ? spec.openFgaDbHost : spec.cloudApiDbHost)}`,
|
||||
`selected_key=${shellQuote(options.key ?? "")}`,
|
||||
`preset=${shellQuote(options.preset)}`,
|
||||
"dry_run=true",
|
||||
"secret_exists_flag() { kubectl -n \"$namespace\" get secret \"$1\" >/dev/null 2>&1 && printf yes || printf no; }",
|
||||
"resource_exists_flag() { kubectl -n \"$namespace\" get \"$1\" \"$2\" >/dev/null 2>&1 && printf yes || printf no; }",
|
||||
"endpointslice_exists_flag() { kubectl -n \"$namespace\" get endpointslice -l \"kubernetes.io/service-name=$1\" -o name 2>/dev/null | grep -q . && printf yes || printf no; }",
|
||||
"secret_b64_key() { kubectl -n \"$namespace\" get secret \"$1\" -o \"go-template={{ index .data \\\"$2\\\" }}\" 2>/dev/null || true; }",
|
||||
"decoded_value() { if [ -n \"$1\" ]; then printf '%s' \"$1\" | base64 -d 2>/dev/null || true; fi; }",
|
||||
"decoded_length() { if [ -n \"$1\" ]; then printf '%s' \"$1\" | base64 -d 2>/dev/null | wc -c | tr -d ' '; else printf '0'; fi; }",
|
||||
"uri_has_platform_host=no",
|
||||
"uri_has_db_name=no",
|
||||
"uri_has_db_user=no",
|
||||
"uri_matches_expected() {",
|
||||
" uri=$1",
|
||||
" uri_has_platform_host=no",
|
||||
" uri_has_db_name=no",
|
||||
" uri_has_db_user=no",
|
||||
" case \"$uri\" in *\"@$platform_host:\"*|*\"@$platform_host/\"*|*\"@$platform_host_fqdn:\"*|*\"@$platform_host_fqdn/\"*) uri_has_platform_host=yes ;; esac",
|
||||
" case \"$uri\" in *\"/$db_name\"|*\"/$db_name?\"*|*\"/$db_name?\"*) uri_has_db_name=yes ;; esac",
|
||||
" case \"$uri\" in postgres://$db_user:*|postgresql://$db_user:*) uri_has_db_user=yes ;; esac",
|
||||
"}",
|
||||
"exists=$(secret_exists_flag \"$name\")",
|
||||
"legacy_postgres_exists=$(secret_exists_flag \"$legacy_postgres_secret\")",
|
||||
"uri_b64=$(secret_b64_key \"$name\" \"$database_url_key\")",
|
||||
"uri_present=$([ -n \"$uri_b64\" ] && printf yes || printf no)",
|
||||
"uri_bytes=$(decoded_length \"$uri_b64\")",
|
||||
"uri_value=$(decoded_value \"$uri_b64\")",
|
||||
"authn_b64=$(secret_b64_key \"$name\" \"$authn_key\")",
|
||||
"authn_present=$([ -n \"$authn_b64\" ] && printf yes || printf no)",
|
||||
"authn_bytes=$(decoded_length \"$authn_b64\")",
|
||||
"pg_password_b64=$(secret_b64_key \"$name\" \"$postgres_password_key\")",
|
||||
"pg_password_present=$([ -n \"$pg_password_b64\" ] && printf yes || printf no)",
|
||||
"pg_password_bytes=$(decoded_length \"$pg_password_b64\")",
|
||||
"platform_service_exists=$(resource_exists_flag service \"$platform_service\")",
|
||||
"platform_endpoints_exists=$(resource_exists_flag endpoints \"$platform_service\")",
|
||||
"platform_endpointslice_exists=$(endpointslice_exists_flag \"$platform_service\")",
|
||||
"uri_matches_expected \"$uri_value\"",
|
||||
"printf 'namespace\\t%s\\n' \"$namespace\"",
|
||||
"printf 'secret\\t%s\\n' \"$name\"",
|
||||
"printf 'key\\t%s\\n' \"$database_url_key\"",
|
||||
"printf 'preset\\t%s\\n' \"$preset\"",
|
||||
"printf 'action\\tobserved\\n'",
|
||||
"printf 'dryRun\\t%s\\n' \"$dry_run\"",
|
||||
"printf 'mutation\\tfalse\\n'",
|
||||
"printf 'platformDbMode\\ttrue\\n'",
|
||||
"printf 'afterExists\\t%s\\n' \"$exists\"",
|
||||
"printf 'afterDatabaseUrlPresent\\t%s\\n' \"$uri_present\"",
|
||||
"printf 'afterDatabaseUrlBytes\\t%s\\n' \"$uri_bytes\"",
|
||||
"printf 'afterDatastoreUriPresent\\t%s\\n' \"$uri_present\"",
|
||||
"printf 'afterDatastoreUriBytes\\t%s\\n' \"$uri_bytes\"",
|
||||
"printf 'afterAuthnPresent\\t%s\\n' \"$authn_present\"",
|
||||
"printf 'afterAuthnBytes\\t%s\\n' \"$authn_bytes\"",
|
||||
"printf 'afterPostgresPasswordPresent\\t%s\\n' \"$pg_password_present\"",
|
||||
"printf 'afterPostgresPasswordBytes\\t%s\\n' \"$pg_password_bytes\"",
|
||||
"printf 'legacyPostgresSecret\\t%s\\n' \"$legacy_postgres_secret\"",
|
||||
"printf 'legacyPostgresSecretExists\\t%s\\n' \"$legacy_postgres_exists\"",
|
||||
"printf 'afterPostgresSecretExists\\t%s\\n' \"$legacy_postgres_exists\"",
|
||||
"printf 'platformService\\t%s\\n' \"$platform_service\"",
|
||||
"printf 'platformServiceExists\\t%s\\n' \"$platform_service_exists\"",
|
||||
"printf 'platformEndpointsExists\\t%s\\n' \"$platform_endpoints_exists\"",
|
||||
"printf 'platformEndpointSliceExists\\t%s\\n' \"$platform_endpointslice_exists\"",
|
||||
"printf 'dbName\\t%s\\n' \"$db_name\"",
|
||||
"printf 'dbUser\\t%s\\n' \"$db_user\"",
|
||||
"printf 'dbHost\\t%s\\n' \"$db_host\"",
|
||||
"printf 'dbHostMatchesPlatform\\t%s\\n' \"$uri_has_platform_host\"",
|
||||
"printf 'dbNameMatchesExpected\\t%s\\n' \"$uri_has_db_name\"",
|
||||
"printf 'dbUserMatchesExpected\\t%s\\n' \"$uri_has_db_user\"",
|
||||
"uri_value=",
|
||||
"if [ \"$platform_service_exists\" != yes ]; then exit 44; fi",
|
||||
].join("\n");
|
||||
}
|
||||
|
||||
function openFgaSecretScript(options: NodeSecretOptions, spec: RuntimeSecretSpec): string {
|
||||
return [
|
||||
"set +e",
|
||||
@@ -896,44 +1233,57 @@ function cloudApiDbSecretScript(options: NodeSecretOptions, spec: RuntimeSecretS
|
||||
function secretStatusFromText(text: string, commandOk: boolean, exitCode: number | null, stderr: string, spec: RuntimeSecretSpec): Record<string, unknown> {
|
||||
const fields = keyValueLinesFromText(text);
|
||||
if (fields.preset === "owned-postgres-cleanup") {
|
||||
const absent = fields.afterSecretExists !== "yes" && fields.afterPvcExists !== "yes";
|
||||
const oldWorkloadAbsent = fields.afterStatefulSetExists !== "yes";
|
||||
const absent = fields.afterStatefulSetExists !== "yes" &&
|
||||
fields.afterServiceExists !== "yes" &&
|
||||
fields.afterConfigMapExists !== "yes" &&
|
||||
fields.afterSecretExists !== "yes" &&
|
||||
fields.afterPvcExists !== "yes";
|
||||
const platformServiceReady = fields.platformServiceExists === "yes";
|
||||
return {
|
||||
ok: commandOk && absent && oldWorkloadAbsent && platformServiceReady,
|
||||
ok: commandOk && absent && platformServiceReady,
|
||||
namespace: fields.namespace || spec.namespace,
|
||||
secret: fields.secret || spec.postgresSecret,
|
||||
statefulSet: fields.statefulSet || spec.postgresStatefulSet,
|
||||
service: fields.service || spec.postgresSecret,
|
||||
configMap: fields.configMap || `${spec.postgresSecret}-init`,
|
||||
pvc: fields.pvc || `data-${spec.postgresSecret}-0`,
|
||||
preset: "owned-postgres-cleanup",
|
||||
action: fields.action || null,
|
||||
dryRun: fields.dryRun === "true",
|
||||
mutation: fields.mutation === "true",
|
||||
before: {
|
||||
statefulSetExists: fields.beforeStatefulSetExists === "yes",
|
||||
serviceExists: fields.beforeServiceExists === "yes",
|
||||
configMapExists: fields.beforeConfigMapExists === "yes",
|
||||
secretExists: fields.beforeSecretExists === "yes",
|
||||
pvcExists: fields.beforePvcExists === "yes",
|
||||
pvcPhase: fields.beforePvcPhase || null,
|
||||
persistentVolume: fields.beforePersistentVolume || null,
|
||||
statefulSetExists: fields.beforeStatefulSetExists === "yes",
|
||||
},
|
||||
after: {
|
||||
statefulSetExists: fields.afterStatefulSetExists === "yes",
|
||||
serviceExists: fields.afterServiceExists === "yes",
|
||||
configMapExists: fields.afterConfigMapExists === "yes",
|
||||
secretExists: fields.afterSecretExists === "yes",
|
||||
pvcExists: fields.afterPvcExists === "yes",
|
||||
pvcPhase: fields.afterPvcPhase || null,
|
||||
persistentVolume: fields.afterPersistentVolume || null,
|
||||
statefulSetExists: fields.afterStatefulSetExists === "yes",
|
||||
},
|
||||
platformService: {
|
||||
name: "g14-platform-postgres",
|
||||
exists: platformServiceReady,
|
||||
},
|
||||
deleteStatefulSetExitCode: numericField(fields.deleteStatefulSetExitCode),
|
||||
deleteServiceExitCode: numericField(fields.deleteServiceExitCode),
|
||||
deleteConfigMapExitCode: numericField(fields.deleteConfigMapExitCode),
|
||||
deleteSecretExitCode: numericField(fields.deleteSecretExitCode),
|
||||
deletePvcExitCode: numericField(fields.deletePvcExitCode),
|
||||
exitCode,
|
||||
stderr: commandOk ? "" : stderr.trim().slice(0, 2000),
|
||||
valuesRedacted: true,
|
||||
summary: absent
|
||||
? `${fields.secret || spec.postgresSecret} and ${fields.pvc || `data-${spec.postgresSecret}-0`} absent`
|
||||
: `${fields.secret || spec.postgresSecret} or ${fields.pvc || `data-${spec.postgresSecret}-0`} still exists`,
|
||||
? `${fields.statefulSet || spec.postgresStatefulSet}, ${fields.service || spec.postgresSecret}, ${fields.configMap || `${spec.postgresSecret}-init`}, ${fields.secret || spec.postgresSecret}, and ${fields.pvc || `data-${spec.postgresSecret}-0`} absent`
|
||||
: `owned Postgres resources still exist in ${fields.namespace || spec.namespace}`,
|
||||
};
|
||||
}
|
||||
if (fields.preset === "master-server-admin-api-key") {
|
||||
@@ -1002,6 +1352,54 @@ function secretStatusFromText(text: string, commandOk: boolean, exitCode: number
|
||||
if (fields.preset === "cloud-api-db") {
|
||||
const beforeUrlBytes = numericField(fields.beforeDatabaseUrlBytes);
|
||||
const afterUrlBytes = numericField(fields.afterDatabaseUrlBytes);
|
||||
if (fields.platformDbMode === "true") {
|
||||
const keysHealthy = fields.afterExists === "yes" &&
|
||||
fields.afterDatabaseUrlPresent === "yes" &&
|
||||
typeof afterUrlBytes === "number" && afterUrlBytes > 0;
|
||||
const platformBridgeHealthy = fields.platformServiceExists === "yes" &&
|
||||
(fields.platformEndpointsExists === "yes" || fields.platformEndpointSliceExists === "yes");
|
||||
const uriHealthy = fields.dbHostMatchesPlatform === "yes" &&
|
||||
fields.dbNameMatchesExpected === "yes" &&
|
||||
fields.dbUserMatchesExpected === "yes";
|
||||
const healthy = keysHealthy && platformBridgeHealthy && uriHealthy;
|
||||
return {
|
||||
ok: commandOk && healthy,
|
||||
namespace: fields.namespace || spec.namespace,
|
||||
secret: fields.secret || spec.cloudApiDbSecret,
|
||||
key: fields.key || spec.cloudApiDbKey,
|
||||
preset: "cloud-api-db",
|
||||
action: fields.action || null,
|
||||
dryRun: fields.dryRun === "true",
|
||||
mutation: fields.mutation === "true",
|
||||
platformDbMode: true,
|
||||
after: {
|
||||
exists: fields.afterExists === "yes",
|
||||
databaseUrl: { keyPresent: fields.afterDatabaseUrlPresent === "yes", valueBytes: afterUrlBytes },
|
||||
},
|
||||
legacyPostgresSecret: {
|
||||
name: fields.legacyPostgresSecret || spec.postgresSecret,
|
||||
exists: fields.legacyPostgresSecretExists === "yes",
|
||||
},
|
||||
platformService: {
|
||||
name: fields.platformService || spec.platformPostgresService,
|
||||
exists: fields.platformServiceExists === "yes",
|
||||
endpointsExist: fields.platformEndpointsExists === "yes",
|
||||
endpointSliceExists: fields.platformEndpointSliceExists === "yes",
|
||||
},
|
||||
dbName: fields.dbName || spec.cloudApiDbName,
|
||||
dbUser: fields.dbUser || spec.cloudApiDbUser,
|
||||
dbHost: fields.dbHost || spec.cloudApiDbHost,
|
||||
dbHostMatchesPlatform: fields.dbHostMatchesPlatform === "yes",
|
||||
dbNameMatchesExpected: fields.dbNameMatchesExpected === "yes",
|
||||
dbUserMatchesExpected: fields.dbUserMatchesExpected === "yes",
|
||||
exitCode,
|
||||
stderr: commandOk ? "" : stderr.trim().slice(0, 2000),
|
||||
valuesRedacted: true,
|
||||
summary: healthy
|
||||
? `${fields.secret || spec.cloudApiDbSecret}/${fields.key || spec.cloudApiDbKey} points to ${fields.platformService || spec.platformPostgresService}`
|
||||
: `${fields.secret || spec.cloudApiDbSecret}/${fields.key || spec.cloudApiDbKey} is not aligned to platform DB`,
|
||||
};
|
||||
}
|
||||
const keysHealthy = fields.afterExists === "yes" &&
|
||||
fields.afterDatabaseUrlPresent === "yes" &&
|
||||
typeof afterUrlBytes === "number" && afterUrlBytes > 0;
|
||||
@@ -1054,6 +1452,57 @@ function secretStatusFromText(text: string, commandOk: boolean, exitCode: number
|
||||
const afterAuthnBytes = numericField(fields.afterAuthnBytes);
|
||||
const afterUriBytes = numericField(fields.afterDatastoreUriBytes);
|
||||
const afterPasswordBytes = numericField(fields.afterPostgresPasswordBytes);
|
||||
if (fields.platformDbMode === "true") {
|
||||
const keysHealthy = fields.afterExists === "yes" &&
|
||||
fields.afterAuthnPresent === "yes" &&
|
||||
fields.afterDatastoreUriPresent === "yes" &&
|
||||
typeof afterAuthnBytes === "number" && afterAuthnBytes > 0 &&
|
||||
typeof afterUriBytes === "number" && afterUriBytes > 0;
|
||||
const platformBridgeHealthy = fields.platformServiceExists === "yes" &&
|
||||
(fields.platformEndpointsExists === "yes" || fields.platformEndpointSliceExists === "yes");
|
||||
const uriHealthy = fields.dbHostMatchesPlatform === "yes" &&
|
||||
fields.dbNameMatchesExpected === "yes" &&
|
||||
fields.dbUserMatchesExpected === "yes";
|
||||
const healthy = keysHealthy && platformBridgeHealthy && uriHealthy;
|
||||
return {
|
||||
ok: commandOk && healthy,
|
||||
namespace: fields.namespace || spec.namespace,
|
||||
secret: fields.secret || spec.openFgaSecret,
|
||||
preset: fields.preset || "openfga",
|
||||
action: fields.action || null,
|
||||
dryRun: fields.dryRun === "true",
|
||||
mutation: fields.mutation === "true",
|
||||
platformDbMode: true,
|
||||
after: {
|
||||
exists: fields.afterExists === "yes",
|
||||
authnPresharedKey: { keyPresent: fields.afterAuthnPresent === "yes", valueBytes: afterAuthnBytes },
|
||||
datastoreUri: { keyPresent: fields.afterDatastoreUriPresent === "yes", valueBytes: afterUriBytes },
|
||||
postgresPassword: { keyPresent: fields.afterPostgresPasswordPresent === "yes", valueBytes: afterPasswordBytes },
|
||||
},
|
||||
legacyPostgresSecret: {
|
||||
name: fields.legacyPostgresSecret || spec.postgresSecret,
|
||||
exists: fields.legacyPostgresSecretExists === "yes",
|
||||
},
|
||||
platformService: {
|
||||
name: fields.platformService || spec.platformPostgresService,
|
||||
exists: fields.platformServiceExists === "yes",
|
||||
endpointsExist: fields.platformEndpointsExists === "yes",
|
||||
endpointSliceExists: fields.platformEndpointSliceExists === "yes",
|
||||
},
|
||||
dbName: fields.dbName || spec.openFgaDbName,
|
||||
dbUser: fields.dbUser || spec.openFgaDbUser,
|
||||
dbHost: fields.dbHost || spec.openFgaDbHost,
|
||||
dbHostMatchesPlatform: fields.dbHostMatchesPlatform === "yes",
|
||||
dbNameMatchesExpected: fields.dbNameMatchesExpected === "yes",
|
||||
dbUserMatchesExpected: fields.dbUserMatchesExpected === "yes",
|
||||
exitCode,
|
||||
stderr: commandOk ? "" : stderr.trim().slice(0, 2000),
|
||||
valuesRedacted: true,
|
||||
summary: healthy
|
||||
? `${fields.secret || spec.openFgaSecret} datastore-uri points to ${fields.platformService || spec.platformPostgresService}`
|
||||
: `${fields.secret || spec.openFgaSecret} datastore-uri is not aligned to platform DB`,
|
||||
};
|
||||
}
|
||||
const keysHealthy = fields.afterExists === "yes" &&
|
||||
fields.afterPostgresSecretExists === "yes" &&
|
||||
fields.afterAuthnPresent === "yes" &&
|
||||
|
||||
Reference in New Issue
Block a user