From 7306000ac1e7b3689ba60d0062a791fb29a5d009 Mon Sep 17 00:00:00 2001 From: Codex Date: Tue, 9 Jun 2026 01:11:04 +0000 Subject: [PATCH] fix: bootstrap v03 cloud api db secret --- docs/reference/cli.md | 4 +- scripts/hwlab-g14-contract-test.ts | 51 ++++++- scripts/src/hwlab-g14.ts | 4 +- scripts/src/hwlab-node.ts | 235 ++++++++++++++++++++++++++++- 4 files changed, 286 insertions(+), 8 deletions(-) diff --git a/docs/reference/cli.md b/docs/reference/cli.md index 9e282dc1..ab4fe109 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -16,6 +16,8 @@ CI/CD、GitOps、rollout、artifact 发布、PR 合并后的 runtime lane 滚动 当现有 CLI 对某个 CI/CD 操作缺字段、缺动作、缺状态或缺权限时，处理顺序是先补 CLI，再执行发布或治理动作。临时低层 route 写操作只允许用于一次性止血，并且必须随后把稳定能力补进 CLI 与本参考文档；不能把手工 `kubectl apply/delete/annotate`、原生 GitHub CLI、手写 REST 请求或 registry shell 脚本沉淀成长期流程。长时观察仍遵守 60 秒短查询和 submit-and-poll 语义，不用单个 `trans`/`tran` 等待完整 PipelineRun 或 Argo rollout 结束。 +`hwlab nodes secret status|ensure --node G14 --lane v03 --name hwlab-cloud-api-v03-db` 是 v03 Cloud API Postgres runtime SecretRef 的受控 bootstrap 入口；`ensure` 从 lane-local Postgres Secret 派生 `database-url`、确保同名 runtime role/database 存在，并且只在 Secret/DB 有实际补齐时滚动 `hwlab-cloud-api` 以重新注入 env。输出仅披露 Secret 名、key presence、decoded byte count、DB role/database presence、rollout exit code、mutation 和后续命令，禁止打印 base64、database URL、password 或可复用凭据。 + `hwlab nodes secret status|ensure --node G14 --lane v03 --name hwlab-v03-code-agent-provider` 是 v03 Code Agent / MoonBridge provider SecretRef 的受控 bootstrap 入口；`ensure` 只从集群内既有 `hwlab-v02/hwlab-v02-code-agent-provider` 复制 `openai-api-key`、`opencode-api-key` 到 lane-local Secret，输出仅披露 source/target Secret 名、key presence、decoded byte count、mutation 和后续命令，禁止打印 base64、解码值、完整 API key 或可复用凭据。OpenFGA 和 master admin API key 继续使用同一命名空间下的 `hwlab nodes secret ... --name hwlab-v03-openfga|hwlab-v03-master-server-admin-api-key`。 ## Command Model @@ -59,7 +61,7 @@ CI/CD、GitOps、rollout、artifact 发布、PR 合并后的 runtime lane 滚动 创建 PipelineRun 前会读取 `devops-infra` mirror refs，若 `localV02` 未等于当前 source commit，则自动执行一次受控 manual `git-mirror sync` Job 并复核 ref，复核失败时停止触发，避免 Tekton `prepare-source` 已知失败；services 参数只包含 v02 runtime service matrix，`hwlab-cli` 是固定 repo 短连接源码工具，不进入 PipelineRun service build。 `--dry-run` 只报告是否会 pre-sync，不创建 Job；confirmed trigger 默认创建 `.state/jobs/` 异步 job 并立刻返回 `job.id`、`statusCommand`、stdout/stderr 路径，避免 git mirror pre-sync 或 PipelineRun 创建期间长时间阻塞；`--wait` 路径也必须向 stderr 输出 `hwlab.v02.trigger.progress` JSON 事件，覆盖 `control-plane-refresh`、`git-mirror-pre-sync` 和 `create-pipelinerun`，避免异步 job 长时间只有启动命令而无法判断卡点；默认 JSON 必须对 `manifest_b64`、长脚本和远端 stdout/stderr 做有界摘要，保留长度与 hash，最终 trigger 结果只返回阶段摘要和关键 tail，完整内容通过 job stdout/stderr 文件渐进披露；只有现场同步调试才显式加 `--wait`；旧 `rerun-current` 只作为输入别名保留。PipelineRun `Completed`、Argo `Synced/Healthy` 和 `webAssets.ok=true` 只证明 G14 runtime 已更新；交付收口还必须用 `hwlab g14 git-mirror status` 查看 `cache.summary.pendingFlush`，若为 true，继续执行受控 `hwlab g14 git-mirror flush --confirm` 并用 job status 轮询到 `pendingFlush=false`。 - `hwlab g14 control-plane runtime-migration --lane v02 [--dry-run|--allow-live-db-read --dry-run|--confirm]` 只通过 `hwlab-v02` namespace 当前 `deployment/hwlab-cloud-api -c hwlab-cloud-api` 内 repo-owned migration CLI 执行；不读取或打印 Secret 值、不触碰 PROD、不绕到手工 `psql`。 -- `hwlab g14 secret status|ensure --lane v02 --name hwlab-v02-openfga|hwlab-v02-master-server-admin-api-key [--dry-run|--confirm]` 和 `hwlab nodes secret status|ensure --node G14 --lane v03 --name hwlab-v03-openfga|hwlab-v03-master-server-admin-api-key [--dry-run|--confirm]` 是 HWLAB runtime lane SecretRef bootstrap 的标准入口：OpenFGA preset 确保 preshared token、datastore URI 和 Postgres password 存在；v02 继续确保独立 `hwlab_openfga` 数据库/角色存在，v03+ 默认从 lane spec 派生 namespace、Postgres Secret 和主 DB 用户，避免复制 v02 专用硬编码。master server admin API key preset 确保本机 `/root/.config/hwlab-v0x/master-server-admin-api-key.env` 以 0600 保存 `HWLAB_API_KEY`，并同步到对应 lane 的 `*-master-server-admin-api-key/api-key`。`status` 只返回 key 是否存在、解码后字节数、key prefix 和 DB 对象存在性，永远不读取、不打印、不回传 secret 明文。`hwlab g14 secret delete --lane v02 --name [--dry-run|--confirm]` 只用于删除确认已不被 workload 引用的 v0.2 废弃 Secret，默认 dry-run，拒绝删除 OpenFGA/Postgres/master admin API key 等必需 Secret；共享 device-pod API key 已退出当前授权路径，不再提供 ensure/bootstrap 入口。 +- `hwlab g14 secret status|ensure --lane v02 --name hwlab-v02-openfga|hwlab-v02-master-server-admin-api-key [--dry-run|--confirm]` 和 `hwlab nodes secret status|ensure --node G14 --lane v03 --name hwlab-v03-openfga|hwlab-cloud-api-v03-db|hwlab-v03-master-server-admin-api-key [--dry-run|--confirm]` 是 HWLAB runtime lane SecretRef bootstrap 的标准入口：OpenFGA preset 确保 preshared token、datastore URI 和 Postgres password 存在；Cloud API DB preset 确保 lane-local `database-url` SecretRef、runtime role/database 和必要的 Cloud API env 重新注入；v02 继续确保独立 `hwlab_openfga` 数据库/角色存在，v03+ 默认从 lane spec 派生 namespace、Postgres Secret 和主 DB 用户，避免复制 v02 专用硬编码。master server admin API key preset 确保本机 `/root/.config/hwlab-v0x/master-server-admin-api-key.env` 以 0600 保存 `HWLAB_API_KEY`，并同步到对应 lane 的 `*-master-server-admin-api-key/api-key`。`status` 只返回 key 是否存在、解码后字节数、key prefix、DB 对象存在性和 rollout exit code，永远不读取、不打印、不回传 secret 明文。`hwlab g14 secret delete --lane v02 --name [--dry-run|--confirm]` 只用于删除确认已不被 workload 引用的 v0.2 废弃 Secret，默认 dry-run，拒绝删除 OpenFGA/Postgres/master admin API key 等必需 Secret；共享 device-pod API key 已退出当前授权路径，不再提供 ensure/bootstrap 入口。 - `hwlab g14 control-plane cleanup-runs --lane v02|v03|g14|all [--min-age-minutes N] [--limit N] [--dry-run|--confirm]` 是完成态 PipelineRun 工作区 retention 入口；真实清理只删除已完成 PipelineRun，让 Tekton/local-path 回收临时 PVC，不触碰 registry storage、业务 PVC、Secret、runtime workload 或 GitOps desired state。带 `--pipeline-run ` 或 `--source-commit ` 的定点清理必须先直接查询目标 PipelineRun，而不是只从全量列表过滤；不存在的目标返回 `target-pipelinerun-not-found`，未完成目标返回 `target-pipelinerun-not-terminal`，空查询和读取失败分别返回 `target-pipelinerun-query-empty` / `target-pipelinerun-query-failed`，年龄保护仍返回 `below-min-age`。`hwlab nodes control-plane cleanup-runs --node G14 --lane v03 --pipeline-run ` 是 v0.3 failed run 受控重试前的清理入口。 - `hwlab g14 control-plane cleanup-released-pvs --lane all [--limit N] [--dry-run|--confirm]` 是 local-path 未自动回收后的补充 retention 入口；只列并删除 `Released`、`local-path`、`Delete`、`claimNamespace=hwlab-ci` 且 claim 名称形如 Tekton 临时 `pvc-*` 的 PV。 - `hwlab g14 git-mirror status|apply|sync|flush [--dry-run|--confirm]` 是 `devops-infra` git mirror/relay 的受控维护入口：`apply` 渲染并 server-side apply `devops-infra/git-mirror.yaml`，同时删除遗留 `git-mirror-hwlab-sync` CronJob；`sync` 创建一次性 manual Job，把 GitHub allowlist refs 拉入本地 mirror；`flush` 创建一次性 manual Job，把本地 `v0.2-gitops` 快进推回 GitHub。 diff --git a/scripts/hwlab-g14-contract-test.ts b/scripts/hwlab-g14-contract-test.ts index f911d87f..9e4ccad4 100644 --- a/scripts/hwlab-g14-contract-test.ts +++ b/scripts/hwlab-g14-contract-test.ts @@ -153,11 +153,58 @@ assertCondition( assertCondition( hwlabHelpUsage.some((line) => line.includes("hwlab nodes secret status --node G14 --lane v03 --name hwlab-v03-code-agent-provider")) && hwlabHelpUsage.some((line) => line.includes("hwlab nodes secret ensure --node G14 --lane v03 --name hwlab-v03-code-agent-provider --confirm")) + && hwlabHelpUsage.some((line) => line.includes("hwlab nodes secret status --node G14 --lane v03 --name hwlab-cloud-api-v03-db")) + && hwlabHelpUsage.some((line) => line.includes("hwlab nodes secret ensure --node G14 --lane v03 --name hwlab-cloud-api-v03-db --confirm")) && hwlabNodeHelpJson.includes("hwlab nodes secret status --node G14 --lane v03 --name hwlab-v03-code-agent-provider") - && hwlabNodeHelpJson.includes("hwlab nodes secret ensure --node G14 --lane v03 --name hwlab-v03-code-agent-provider --confirm"), - "v0.3 node-scoped secret help must expose the controlled code-agent provider SecretRef bootstrap path", + && hwlabNodeHelpJson.includes("hwlab nodes secret ensure --node G14 --lane v03 --name hwlab-v03-code-agent-provider --confirm") + && hwlabNodeHelpJson.includes("hwlab nodes secret status --node G14 --lane v03 --name hwlab-cloud-api-v03-db") + && hwlabNodeHelpJson.includes("hwlab nodes secret ensure --node G14 --lane v03 --name hwlab-cloud-api-v03-db --confirm"), + "v0.3 node-scoped secret help must expose the controlled code-agent provider and cloud-api DB SecretRef bootstrap paths", { hwlabHelpUsage, hwlabNodeHelp: hwlabNodeHelp() }, ); +const cloudApiDbStatus = nodeSecretStatusFromTextForTest([ + "namespace\thwlab-v03", + "secret\thwlab-cloud-api-v03-db", + "key\tdatabase-url", + "preset\tcloud-api-db", + "action\tensured", + "dryRun\tfalse", + "mutation\ttrue", + "beforeExists\tno", + "beforePostgresSecretExists\tyes", + "beforeDatabaseUrlPresent\tno", + "beforeDatabaseUrlBytes\t0", + "afterExists\tyes", + "afterDatabaseUrlPresent\tyes", + "afterDatabaseUrlBytes\t112", + "postgresAdminSecretPresent\tyes", + "postgresSecret\thwlab-v03-postgres", + "dbName\thwlab_v03", + "dbUser\thwlab_v03", + "dbHost\thwlab-v03-postgres.hwlab-v03.svc.cluster.local", + "dbRoleExistsBefore\tt", + "dbDatabaseExistsBefore\tt", + "dbProbeExitCodeBefore\t0", + "dbRoleExistsAfter\tt", + "dbDatabaseExistsAfter\tt", + "dbProbeExitCodeAfter\t0", + "cloudApiDeployment\thwlab-cloud-api", + "applyExitCode\t0", + "dbEnsureExitCode\t0", + "rolloutRestartExitCode\t0", + "rolloutStatusExitCode\t0", +].join("\n"), true, 0, ""); +assertCondition( + record(cloudApiDbStatus).ok === true + && record(cloudApiDbStatus).valuesRedacted === true + && record(record(cloudApiDbStatus).after).exists === true + && record(record(record(cloudApiDbStatus).after).databaseUrl).valueBytes === 112 + && record(cloudApiDbStatus).cloudApiDeployment === "hwlab-cloud-api" + && !JSON.stringify(cloudApiDbStatus).includes("postgres://") + && !JSON.stringify(cloudApiDbStatus).includes("password"), + "cloud-api DB Secret status must be redacted while proving DB SecretRef and runtime database are present", + cloudApiDbStatus, +); const codeAgentProviderStatus = nodeSecretStatusFromTextForTest([ "namespace\thwlab-v03", "secret\thwlab-v03-code-agent-provider", diff --git a/scripts/src/hwlab-g14.ts b/scripts/src/hwlab-g14.ts index 6ca02638..9e362844 100644 --- a/scripts/src/hwlab-g14.ts +++ b/scripts/src/hwlab-g14.ts @@ -9293,6 +9293,8 @@ export function hwlabG14Help(): Record { "bun scripts/cli.ts hwlab g14 secret ensure --lane v02 --name hwlab-v02-master-server-admin-api-key --confirm", "bun scripts/cli.ts hwlab nodes secret status --node G14 --lane v03 --name hwlab-v03-master-server-admin-api-key", "bun scripts/cli.ts hwlab nodes secret ensure --node G14 --lane v03 --name hwlab-v03-master-server-admin-api-key --confirm", + "bun scripts/cli.ts hwlab nodes secret status --node G14 --lane v03 --name hwlab-cloud-api-v03-db", + "bun scripts/cli.ts hwlab nodes secret ensure --node G14 --lane v03 --name hwlab-cloud-api-v03-db --confirm", "bun scripts/cli.ts hwlab nodes secret status --node G14 --lane v03 --name hwlab-v03-code-agent-provider", "bun scripts/cli.ts hwlab nodes secret ensure --node G14 --lane v03 --name hwlab-v03-code-agent-provider --confirm", "bun scripts/cli.ts hwlab g14 secret delete --lane v02 --name --dry-run", @@ -9320,7 +9322,7 @@ export function hwlabG14Help(): Record { "bun scripts/cli.ts hwlab g14 upstream-image ensure --name openfga --tag v1.17.0 --confirm", "bun scripts/cli.ts job status --tail-bytes 30000", ], - description: "G14 HWLAB PR monitor, legacy DEV/PROD retirement status/plan/execute command, bounded v0.2 control-plane bootstrap/cleanup/runtime-migration helper, node-scoped runtime lane v03 control-plane apply/status/refresh/trigger entry, runtime lane SecretRef bootstrap, devops-infra git mirror and observability maintenance, controlled CI tools image build/status entry, and allowlisted upstream image mirroring. The legacy base=G14 monitor is blocked by the retirement contract; the local retirement marker records live execution evidence. `--lane v02` monitors base=v0.2 PRs, waits for GitHub preflight/CI readiness, automatically merges ready PRs without waiting for other active v0.2 PipelineRuns, triggers v0.2 CD with latest-only GitOps writeback, flushes the git mirror when needed, and posts deduplicated PR comments for pending, blocked/conflict, success, superseded, failure, or timeout states. confirmed control-plane trigger-current and git-mirror sync/flush also return async jobs by default, with --wait reserved for explicit synchronous debugging. control-plane v02 keeps the full closeout/cleanup/runtime-migration verdict path; v03+ is advertised through `hwlab nodes ... --node --lane vNN` so node identity remains configuration data instead of a command family. secret status/ensure is the standard runtime lane SecretRef bootstrap path for OpenFGA and master admin API key; it never reads or prints secret values. upstream-image status/ensure only mirrors allowlisted upstream runtime images into the G14 local registry. git-mirror status/apply/sync/flush is the manual devops-infra mirror/relay control path and does not install a CronJob. observability status/apply/query/targets/boundary/closeout owns the shared Prometheus Operator and Prometheus instance in devops-infra, adds bounded PromQL assertions and semantic closeout summaries, while HWLAB lane manifests own only ServiceMonitor and PrometheusRule objects.", + description: "G14 HWLAB PR monitor, legacy DEV/PROD retirement status/plan/execute command, bounded v0.2 control-plane bootstrap/cleanup/runtime-migration helper, node-scoped runtime lane v03 control-plane apply/status/refresh/trigger entry, runtime lane SecretRef bootstrap, devops-infra git mirror and observability maintenance, controlled CI tools image build/status entry, and allowlisted upstream image mirroring. The legacy base=G14 monitor is blocked by the retirement contract; the local retirement marker records live execution evidence. `--lane v02` monitors base=v0.2 PRs, waits for GitHub preflight/CI readiness, automatically merges ready PRs without waiting for other active v0.2 PipelineRuns, triggers v0.2 CD with latest-only GitOps writeback, flushes the git mirror when needed, and posts deduplicated PR comments for pending, blocked/conflict, success, superseded, failure, or timeout states. confirmed control-plane trigger-current and git-mirror sync/flush also return async jobs by default, with --wait reserved for explicit synchronous debugging. control-plane v02 keeps the full closeout/cleanup/runtime-migration verdict path; v03+ is advertised through `hwlab nodes ... --node --lane vNN` so node identity remains configuration data instead of a command family. secret status/ensure is the standard runtime lane SecretRef bootstrap path for OpenFGA, cloud-api DB, master admin API key, and code-agent provider refs; it never reads or prints secret values. upstream-image status/ensure only mirrors allowlisted upstream runtime images into the G14 local registry. git-mirror status/apply/sync/flush is the manual devops-infra mirror/relay control path and does not install a CronJob. observability status/apply/query/targets/boundary/closeout owns the shared Prometheus Operator and Prometheus instance in devops-infra, adds bounded PromQL assertions and semantic closeout summaries, while HWLAB lane manifests own only ServiceMonitor and PrometheusRule objects.", defaults: { repo: HWLAB_REPO, legacyBase: G14_SOURCE_BRANCH, diff --git a/scripts/src/hwlab-node.ts b/scripts/src/hwlab-node.ts index 461b1b7e..5fdb7fe8 100644 --- a/scripts/src/hwlab-node.ts +++ b/scripts/src/hwlab-node.ts @@ -6,7 +6,7 @@ import { runHwlabG14Command } from "./hwlab-g14"; import { hwlabRuntimeLaneConfigPath, hwlabRuntimeLaneSpec, isHwlabRuntimeLane, type HwlabRuntimeLane } from "./hwlab-node-lanes"; type SecretAction = "status" | "ensure"; -type SecretPreset = "openfga" | "master-server-admin-api-key" | "code-agent-provider"; +type SecretPreset = "openfga" | "master-server-admin-api-key" | "code-agent-provider" | "cloud-api-db"; type DelegatedNodeDomain = "control-plane" | "git-mirror"; interface NodeSecretOptions { @@ -33,6 +33,12 @@ interface RuntimeSecretSpec { openFgaDbUser: string; openFgaDbHost: string; masterAdminApiKeySecret: string; + cloudApiDbSecret: string; + cloudApiDbKey: string; + cloudApiDbName: string; + cloudApiDbUser: string; + cloudApiDbHost: string; + cloudApiDeployment: string; codeAgentProviderSecret: string; codeAgentProviderSourceNamespace: string; codeAgentProviderSourceSecret: string; @@ -44,6 +50,7 @@ const MASTER_ADMIN_API_KEY_KEY = "api-key"; const OPENFGA_AUTHN_KEY = "authn-preshared-key"; const OPENFGA_DATASTORE_URI_KEY = "datastore-uri"; const OPENFGA_POSTGRES_PASSWORD_KEY = "postgres-password"; +const CLOUD_API_DB_KEY = "database-url"; const CODE_AGENT_PROVIDER_OPENAI_KEY = "openai-api-key"; const CODE_AGENT_PROVIDER_OPENCODE_KEY = "opencode-api-key"; const CODE_AGENT_PROVIDER_SOURCE_NAMESPACE = "hwlab-v02"; @@ -77,6 +84,8 @@ export function hwlabNodeHelp(): Record { "bun scripts/cli.ts hwlab nodes secret status --node G14 --lane v03 --name hwlab-v03-openfga", "bun scripts/cli.ts hwlab nodes secret ensure --node G14 --lane v03 --name hwlab-v03-openfga --confirm", "bun scripts/cli.ts hwlab nodes secret ensure --node G14 --lane v03 --name hwlab-v03-master-server-admin-api-key --confirm", + "bun scripts/cli.ts hwlab nodes secret status --node G14 --lane v03 --name hwlab-cloud-api-v03-db", + "bun scripts/cli.ts hwlab nodes secret ensure --node G14 --lane v03 --name hwlab-cloud-api-v03-db --confirm", "bun scripts/cli.ts hwlab nodes secret status --node G14 --lane v03 --name hwlab-v03-code-agent-provider", "bun scripts/cli.ts hwlab nodes secret ensure --node G14 --lane v03 --name hwlab-v03-code-agent-provider --confirm", ], @@ -195,7 +204,7 @@ function rewriteDelegatedNodeString(value: string, scoped: ReturnType { ? openFgaSecretScript(options, spec) : options.preset === "master-server-admin-api-key" ? masterAdminApiKeySecretScript(options, spec) - : codeAgentProviderSecretScript(options, spec); + : options.preset === "cloud-api-db" + ? cloudApiDbSecretScript(options, spec) + : codeAgentProviderSecretScript(options, spec); const result = runTransScript(options.node, script, input, options.timeoutSeconds); const status = secretStatusFromText(statusText(result), result.exitCode === 0, result.exitCode, result.stderr, spec); const dryRunOk = options.action === "ensure" && options.dryRun && result.exitCode === 0; @@ -626,6 +657,150 @@ function codeAgentProviderSecretScript(options: NodeSecretOptions, spec: Runtime ].join("\n"); } +function cloudApiDbSecretScript(options: NodeSecretOptions, spec: RuntimeSecretSpec): string { + return [ + "set +e", + `namespace=${shellQuote(spec.namespace)}`, + `name=${shellQuote(spec.cloudApiDbSecret)}`, + `database_url_key=${shellQuote(spec.cloudApiDbKey)}`, + `postgres_secret=${shellQuote(spec.postgresSecret)}`, + `postgres_statefulset=${shellQuote(spec.postgresStatefulSet)}`, + `postgres_admin_user=${shellQuote(spec.postgresAdminUser)}`, + `db_name=${shellQuote(spec.cloudApiDbName)}`, + `db_user=${shellQuote(spec.cloudApiDbUser)}`, + `db_host=${shellQuote(spec.cloudApiDbHost)}`, + `cloud_api_deployment=${shellQuote(spec.cloudApiDeployment)}`, + `action_request=${shellQuote(options.action)}`, + `dry_run=${shellQuote(options.dryRun ? "true" : "false")}`, + `field_manager=${shellQuote(spec.fieldManager)}`, + "preset=cloud-api-db", + "secret_exists_flag() { kubectl -n \"$namespace\" get secret \"$1\" >/dev/null 2>&1 && printf yes || printf no; }", + "secret_b64_key() { kubectl -n \"$namespace\" get secret \"$1\" -o \"go-template={{ index .data \\\"$2\\\" }}\" 2>/dev/null || true; }", + "decoded_value() { if [ -n \"$1\" ]; then printf '%s' \"$1\" | base64 -d 2>/dev/null || true; fi; }", + "decoded_length() { if [ -n \"$1\" ]; then printf '%s' \"$1\" | base64 -d 2>/dev/null | wc -c | tr -d ' '; else printf '0'; fi; }", + "psql_scalar() { kubectl -n \"$namespace\" exec \"statefulset/$postgres_statefulset\" -c postgres -- env PGPASSWORD=\"$postgres_admin_password\" psql -U \"$postgres_admin_user\" -d postgres -tAc \"$1\" 2>/dev/null | tr -d '[:space:]'; }", + "probe_db() {", + " role_result=unknown", + " database_result=unknown", + " probe_exit=missing-postgres-admin-secret", + " if [ -n \"$postgres_admin_password\" ]; then", + " role_result=$(psql_scalar \"select exists(select 1 from pg_roles where rolname='$db_user');\")", + " role_exit=$?", + " database_result=$(psql_scalar \"select exists(select 1 from pg_database where datname='$db_name');\")", + " database_exit=$?", + " if [ \"$role_exit\" -eq 0 ] && [ \"$database_exit\" -eq 0 ]; then probe_exit=0; else probe_exit=$role_exit/$database_exit; fi", + " fi", + "}", + "before_exists=$(secret_exists_flag \"$name\")", + "before_postgres_exists=$(secret_exists_flag \"$postgres_secret\")", + "before_url_b64=$(secret_b64_key \"$name\" \"$database_url_key\")", + "before_url_present=$([ -n \"$before_url_b64\" ] && printf yes || printf no)", + "before_url_bytes=$(decoded_length \"$before_url_b64\")", + "database_url=$(decoded_value \"$before_url_b64\")", + "postgres_admin_b64=$(secret_b64_key \"$postgres_secret\" POSTGRES_PASSWORD)", + "postgres_admin_present=$([ -n \"$postgres_admin_b64\" ] && printf yes || printf no)", + "postgres_admin_password=$(decoded_value \"$postgres_admin_b64\")", + "probe_db", + "db_role_exists_before=$role_result", + "db_database_exists_before=$database_result", + "db_probe_exit_before=$probe_exit", + "action=observed", + "mutation=false", + "apply_exit=", + "db_ensure_exit=", + "rollout_restart_exit=", + "rollout_status_exit=", + "if [ \"$action_request\" = ensure ]; then", + " missing_secret=false", + " [ \"$before_url_present\" = yes ] && [ \"$before_url_bytes\" -gt 0 ] || missing_secret=true", + " missing_db=false", + " [ \"$db_role_exists_before\" = t ] || missing_db=true", + " [ \"$db_database_exists_before\" = t ] || missing_db=true", + " if [ \"$dry_run\" = true ]; then", + " if [ \"$before_postgres_exists\" != yes ] || [ \"$postgres_admin_present\" != yes ] || [ \"$missing_secret\" = true ] || [ \"$missing_db\" = true ]; then action=would-ensure; else action=kept; fi", + " elif [ \"$before_postgres_exists\" != yes ] || [ \"$postgres_admin_present\" != yes ] || [ -z \"$postgres_admin_password\" ]; then", + " action=postgres-admin-secret-missing", + " apply_exit=44", + " elif [ \"$missing_secret\" = false ] && [ \"$missing_db\" = false ]; then", + " action=kept", + " else", + " [ -n \"$database_url\" ] || database_url=\"postgres://$db_user:$postgres_admin_password@$db_host:5432/$db_name?sslmode=disable\"", + " kubectl -n \"$namespace\" create secret generic \"$name\" --from-literal=\"$database_url_key=$database_url\" --dry-run=client -o yaml | kubectl apply --server-side --force-conflicts --field-manager=\"$field_manager\" -f -", + " apply_exit=$?", + " if [ \"$apply_exit\" -eq 0 ]; then", + " kubectl -n \"$namespace\" exec -i \"statefulset/$postgres_statefulset\" -c postgres -- env PGPASSWORD=\"$postgres_admin_password\" psql -v ON_ERROR_STOP=1 -U \"$postgres_admin_user\" -d postgres -v db_name=\"$db_name\" -v db_user=\"$db_user\" -v db_pass=\"$postgres_admin_password\" >/tmp/hwlab-cloud-api-db-psql.out 2>/tmp/hwlab-cloud-api-db-psql.err <<'SQL'", + "SELECT format('CREATE ROLE %I LOGIN PASSWORD %L', :'db_user', :'db_pass')", + "WHERE NOT EXISTS (SELECT 1 FROM pg_roles WHERE rolname = :'db_user')", + "\\gexec", + "ALTER ROLE :\"db_user\" LOGIN PASSWORD :'db_pass';", + "SELECT format('CREATE DATABASE %I OWNER %I', :'db_name', :'db_user')", + "WHERE NOT EXISTS (SELECT 1 FROM pg_database WHERE datname = :'db_name')", + "\\gexec", + "ALTER DATABASE :\"db_name\" OWNER TO :\"db_user\";", + "SQL", + " db_ensure_exit=$?", + " if [ \"$db_ensure_exit\" -eq 0 ]; then", + " if [ \"$missing_secret\" = true ] || [ \"$missing_db\" = true ]; then", + " kubectl -n \"$namespace\" rollout restart \"deployment/$cloud_api_deployment\" >/tmp/hwlab-cloud-api-rollout-restart.out 2>/tmp/hwlab-cloud-api-rollout-restart.err", + " rollout_restart_exit=$?", + " if [ \"$rollout_restart_exit\" -eq 0 ]; then", + " kubectl -n \"$namespace\" rollout status \"deployment/$cloud_api_deployment\" --timeout=180s >/tmp/hwlab-cloud-api-rollout-status.out 2>/tmp/hwlab-cloud-api-rollout-status.err", + " rollout_status_exit=$?", + " fi", + " fi", + " if [ -n \"$rollout_restart_exit\" ] && [ \"$rollout_restart_exit\" != 0 ]; then action=rollout-restart-failed", + " elif [ -n \"$rollout_status_exit\" ] && [ \"$rollout_status_exit\" != 0 ]; then action=rollout-status-failed", + " else action=ensured; mutation=true; fi", + " else action=db-ensure-failed; fi", + " else action=apply-failed; fi", + " fi", + "fi", + "after_exists=$(secret_exists_flag \"$name\")", + "after_url_b64=$(secret_b64_key \"$name\" \"$database_url_key\")", + "after_url_present=$([ -n \"$after_url_b64\" ] && printf yes || printf no)", + "after_url_bytes=$(decoded_length \"$after_url_b64\")", + "probe_db", + "db_role_exists_after=$role_result", + "db_database_exists_after=$database_result", + "db_probe_exit_after=$probe_exit", + "printf 'namespace\\t%s\\n' \"$namespace\"", + "printf 'secret\\t%s\\n' \"$name\"", + "printf 'key\\t%s\\n' \"$database_url_key\"", + "printf 'preset\\t%s\\n' \"$preset\"", + "printf 'action\\t%s\\n' \"$action\"", + "printf 'dryRun\\t%s\\n' \"$dry_run\"", + "printf 'mutation\\t%s\\n' \"$mutation\"", + "printf 'beforeExists\\t%s\\n' \"$before_exists\"", + "printf 'beforePostgresSecretExists\\t%s\\n' \"$before_postgres_exists\"", + "printf 'beforeDatabaseUrlPresent\\t%s\\n' \"$before_url_present\"", + "printf 'beforeDatabaseUrlBytes\\t%s\\n' \"$before_url_bytes\"", + "printf 'afterExists\\t%s\\n' \"$after_exists\"", + "printf 'afterDatabaseUrlPresent\\t%s\\n' \"$after_url_present\"", + "printf 'afterDatabaseUrlBytes\\t%s\\n' \"$after_url_bytes\"", + "printf 'postgresAdminSecretPresent\\t%s\\n' \"$postgres_admin_present\"", + "printf 'postgresSecret\\t%s\\n' \"$postgres_secret\"", + "printf 'dbName\\t%s\\n' \"$db_name\"", + "printf 'dbUser\\t%s\\n' \"$db_user\"", + "printf 'dbHost\\t%s\\n' \"$db_host\"", + "printf 'dbRoleExistsBefore\\t%s\\n' \"$db_role_exists_before\"", + "printf 'dbDatabaseExistsBefore\\t%s\\n' \"$db_database_exists_before\"", + "printf 'dbProbeExitCodeBefore\\t%s\\n' \"$db_probe_exit_before\"", + "printf 'dbRoleExistsAfter\\t%s\\n' \"$db_role_exists_after\"", + "printf 'dbDatabaseExistsAfter\\t%s\\n' \"$db_database_exists_after\"", + "printf 'dbProbeExitCodeAfter\\t%s\\n' \"$db_probe_exit_after\"", + "printf 'cloudApiDeployment\\t%s\\n' \"$cloud_api_deployment\"", + "printf 'applyExitCode\\t%s\\n' \"$apply_exit\"", + "printf 'dbEnsureExitCode\\t%s\\n' \"$db_ensure_exit\"", + "printf 'rolloutRestartExitCode\\t%s\\n' \"$rollout_restart_exit\"", + "printf 'rolloutStatusExitCode\\t%s\\n' \"$rollout_status_exit\"", + "database_url= postgres_admin_password=", + "if [ -n \"$apply_exit\" ] && [ \"$apply_exit\" != 0 ]; then exit \"$apply_exit\"; fi", + "if [ -n \"$db_ensure_exit\" ] && [ \"$db_ensure_exit\" != 0 ]; then exit \"$db_ensure_exit\"; fi", + "if [ -n \"$rollout_restart_exit\" ] && [ \"$rollout_restart_exit\" != 0 ]; then exit \"$rollout_restart_exit\"; fi", + "if [ -n \"$rollout_status_exit\" ] && [ \"$rollout_status_exit\" != 0 ]; then exit \"$rollout_status_exit\"; fi", + ].join("\n"); +} + function secretStatusFromText(text: string, commandOk: boolean, exitCode: number | null, stderr: string, spec: RuntimeSecretSpec): Record { const fields = keyValueLinesFromText(text); if (fields.preset === "master-server-admin-api-key") { @@ -691,6 +866,58 @@ function secretStatusFromText(text: string, commandOk: boolean, exitCode: number summary: healthy ? `${fields.secret || spec.codeAgentProviderSecret} has a usable provider key` : `${fields.secret || spec.codeAgentProviderSecret} missing provider keys`, }; } + if (fields.preset === "cloud-api-db") { + const beforeUrlBytes = numericField(fields.beforeDatabaseUrlBytes); + const afterUrlBytes = numericField(fields.afterDatabaseUrlBytes); + const keysHealthy = fields.afterExists === "yes" && + fields.afterDatabaseUrlPresent === "yes" && + typeof afterUrlBytes === "number" && afterUrlBytes > 0; + const databaseHealthy = fields.dbRoleExistsAfter === "t" && fields.dbDatabaseExistsAfter === "t"; + const healthy = keysHealthy && databaseHealthy; + return { + ok: commandOk && healthy, + namespace: fields.namespace || spec.namespace, + secret: fields.secret || spec.cloudApiDbSecret, + key: fields.key || spec.cloudApiDbKey, + preset: "cloud-api-db", + action: fields.action || null, + dryRun: fields.dryRun === "true", + mutation: fields.mutation === "true", + before: { + exists: fields.beforeExists === "yes", + postgresSecretExists: fields.beforePostgresSecretExists === "yes", + databaseUrl: { keyPresent: fields.beforeDatabaseUrlPresent === "yes", valueBytes: beforeUrlBytes }, + database: { + roleExists: fields.dbRoleExistsBefore || "unknown", + databaseExists: fields.dbDatabaseExistsBefore || "unknown", + probeExitCode: fields.dbProbeExitCodeBefore || null, + }, + }, + after: { + exists: fields.afterExists === "yes", + databaseUrl: { keyPresent: fields.afterDatabaseUrlPresent === "yes", valueBytes: afterUrlBytes }, + database: { + roleExists: fields.dbRoleExistsAfter || "unknown", + databaseExists: fields.dbDatabaseExistsAfter || "unknown", + probeExitCode: fields.dbProbeExitCodeAfter || null, + }, + }, + postgresAdminSecretPresent: fields.postgresAdminSecretPresent === "yes", + postgresSecret: fields.postgresSecret || spec.postgresSecret, + dbName: fields.dbName || spec.cloudApiDbName, + dbUser: fields.dbUser || spec.cloudApiDbUser, + dbHost: fields.dbHost || spec.cloudApiDbHost, + cloudApiDeployment: fields.cloudApiDeployment || spec.cloudApiDeployment, + applyExitCode: numericField(fields.applyExitCode), + dbEnsureExitCode: numericField(fields.dbEnsureExitCode), + rolloutRestartExitCode: numericField(fields.rolloutRestartExitCode), + rolloutStatusExitCode: numericField(fields.rolloutStatusExitCode), + exitCode, + stderr: commandOk ? "" : stderr.trim().slice(0, 2000), + valuesRedacted: true, + summary: healthy ? `${fields.secret || spec.cloudApiDbSecret}/${fields.key || spec.cloudApiDbKey} exists and runtime database is present` : `${fields.secret || spec.cloudApiDbSecret}/${fields.key || spec.cloudApiDbKey} or runtime database missing`, + }; + } const afterAuthnBytes = numericField(fields.afterAuthnBytes); const afterUriBytes = numericField(fields.afterDatastoreUriBytes); const afterPasswordBytes = numericField(fields.afterPostgresPasswordBytes);