fix: bootstrap v03 cloud api db secret
This commit is contained in:
@@ -16,6 +16,8 @@ CI/CD、GitOps、rollout、artifact 发布、PR 合并后的 runtime lane 滚动
|
||||
|
||||
当现有 CLI 对某个 CI/CD 操作缺字段、缺动作、缺状态或缺权限时,处理顺序是先补 CLI,再执行发布或治理动作。临时低层 route 写操作只允许用于一次性止血,并且必须随后把稳定能力补进 CLI 与本参考文档;不能把手工 `kubectl apply/delete/annotate`、原生 GitHub CLI、手写 REST 请求或 registry shell 脚本沉淀成长期流程。长时观察仍遵守 60 秒短查询和 submit-and-poll 语义,不用单个 `trans`/`tran` 等待完整 PipelineRun 或 Argo rollout 结束。
|
||||
|
||||
`hwlab nodes secret status|ensure --node G14 --lane v03 --name hwlab-cloud-api-v03-db` 是 v03 Cloud API Postgres runtime SecretRef 的受控 bootstrap 入口;`ensure` 从 lane-local Postgres Secret 派生 `database-url`、确保同名 runtime role/database 存在,并且只在 Secret/DB 有实际补齐时滚动 `hwlab-cloud-api` 以重新注入 env。输出仅披露 Secret 名、key presence、decoded byte count、DB role/database presence、rollout exit code、mutation 和后续命令,禁止打印 base64、database URL、password 或可复用凭据。
|
||||
|
||||
`hwlab nodes secret status|ensure --node G14 --lane v03 --name hwlab-v03-code-agent-provider` 是 v03 Code Agent / MoonBridge provider SecretRef 的受控 bootstrap 入口;`ensure` 只从集群内既有 `hwlab-v02/hwlab-v02-code-agent-provider` 复制 `openai-api-key`、`opencode-api-key` 到 lane-local Secret,输出仅披露 source/target Secret 名、key presence、decoded byte count、mutation 和后续命令,禁止打印 base64、解码值、完整 API key 或可复用凭据。OpenFGA 和 master admin API key 继续使用同一命名空间下的 `hwlab nodes secret ... --name hwlab-v03-openfga|hwlab-v03-master-server-admin-api-key`。
|
||||
|
||||
## Command Model
|
||||
@@ -59,7 +61,7 @@ CI/CD、GitOps、rollout、artifact 发布、PR 合并后的 runtime lane 滚动
|
||||
创建 PipelineRun 前会读取 `devops-infra` mirror refs,若 `localV02` 未等于当前 source commit,则自动执行一次受控 manual `git-mirror sync` Job 并复核 ref,复核失败时停止触发,避免 Tekton `prepare-source` 已知失败;services 参数只包含 v02 runtime service matrix,`hwlab-cli` 是固定 repo 短连接源码工具,不进入 PipelineRun service build。
|
||||
`--dry-run` 只报告是否会 pre-sync,不创建 Job;confirmed trigger 默认创建 `.state/jobs/` 异步 job 并立刻返回 `job.id`、`statusCommand`、stdout/stderr 路径,避免 git mirror pre-sync 或 PipelineRun 创建期间长时间阻塞;`--wait` 路径也必须向 stderr 输出 `hwlab.v02.trigger.progress` JSON 事件,覆盖 `control-plane-refresh`、`git-mirror-pre-sync` 和 `create-pipelinerun`,避免异步 job 长时间只有启动命令而无法判断卡点;默认 JSON 必须对 `manifest_b64`、长脚本和远端 stdout/stderr 做有界摘要,保留长度与 hash,最终 trigger 结果只返回阶段摘要和关键 tail,完整内容通过 job stdout/stderr 文件渐进披露;只有现场同步调试才显式加 `--wait`;旧 `rerun-current` 只作为输入别名保留。PipelineRun `Completed`、Argo `Synced/Healthy` 和 `webAssets.ok=true` 只证明 G14 runtime 已更新;交付收口还必须用 `hwlab g14 git-mirror status` 查看 `cache.summary.pendingFlush`,若为 true,继续执行受控 `hwlab g14 git-mirror flush --confirm` 并用 job status 轮询到 `pendingFlush=false`。
|
||||
- `hwlab g14 control-plane runtime-migration --lane v02 [--dry-run|--allow-live-db-read --dry-run|--confirm]` 只通过 `hwlab-v02` namespace 当前 `deployment/hwlab-cloud-api -c hwlab-cloud-api` 内 repo-owned migration CLI 执行;不读取或打印 Secret 值、不触碰 PROD、不绕到手工 `psql`。
|
||||
- `hwlab g14 secret status|ensure --lane v02 --name hwlab-v02-openfga|hwlab-v02-master-server-admin-api-key [--dry-run|--confirm]` 和 `hwlab nodes secret status|ensure --node G14 --lane v03 --name hwlab-v03-openfga|hwlab-v03-master-server-admin-api-key [--dry-run|--confirm]` 是 HWLAB runtime lane SecretRef bootstrap 的标准入口:OpenFGA preset 确保 preshared token、datastore URI 和 Postgres password 存在;v02 继续确保独立 `hwlab_openfga` 数据库/角色存在,v03+ 默认从 lane spec 派生 namespace、Postgres Secret 和主 DB 用户,避免复制 v02 专用硬编码。master server admin API key preset 确保本机 `/root/.config/hwlab-v0x/master-server-admin-api-key.env` 以 0600 保存 `HWLAB_API_KEY`,并同步到对应 lane 的 `*-master-server-admin-api-key/api-key`。`status` 只返回 key 是否存在、解码后字节数、key prefix 和 DB 对象存在性,永远不读取、不打印、不回传 secret 明文。`hwlab g14 secret delete --lane v02 --name <obsolete-hwlab-v02-secret> [--dry-run|--confirm]` 只用于删除确认已不被 workload 引用的 v0.2 废弃 Secret,默认 dry-run,拒绝删除 OpenFGA/Postgres/master admin API key 等必需 Secret;共享 device-pod API key 已退出当前授权路径,不再提供 ensure/bootstrap 入口。
|
||||
- `hwlab g14 secret status|ensure --lane v02 --name hwlab-v02-openfga|hwlab-v02-master-server-admin-api-key [--dry-run|--confirm]` 和 `hwlab nodes secret status|ensure --node G14 --lane v03 --name hwlab-v03-openfga|hwlab-cloud-api-v03-db|hwlab-v03-master-server-admin-api-key [--dry-run|--confirm]` 是 HWLAB runtime lane SecretRef bootstrap 的标准入口:OpenFGA preset 确保 preshared token、datastore URI 和 Postgres password 存在;Cloud API DB preset 确保 lane-local `database-url` SecretRef、runtime role/database 和必要的 Cloud API env 重新注入;v02 继续确保独立 `hwlab_openfga` 数据库/角色存在,v03+ 默认从 lane spec 派生 namespace、Postgres Secret 和主 DB 用户,避免复制 v02 专用硬编码。master server admin API key preset 确保本机 `/root/.config/hwlab-v0x/master-server-admin-api-key.env` 以 0600 保存 `HWLAB_API_KEY`,并同步到对应 lane 的 `*-master-server-admin-api-key/api-key`。`status` 只返回 key 是否存在、解码后字节数、key prefix、DB 对象存在性和 rollout exit code,永远不读取、不打印、不回传 secret 明文。`hwlab g14 secret delete --lane v02 --name <obsolete-hwlab-v02-secret> [--dry-run|--confirm]` 只用于删除确认已不被 workload 引用的 v0.2 废弃 Secret,默认 dry-run,拒绝删除 OpenFGA/Postgres/master admin API key 等必需 Secret;共享 device-pod API key 已退出当前授权路径,不再提供 ensure/bootstrap 入口。
|
||||
- `hwlab g14 control-plane cleanup-runs --lane v02|v03|g14|all [--min-age-minutes N] [--limit N] [--dry-run|--confirm]` 是完成态 PipelineRun 工作区 retention 入口;真实清理只删除已完成 PipelineRun,让 Tekton/local-path 回收临时 PVC,不触碰 registry storage、业务 PVC、Secret、runtime workload 或 GitOps desired state。带 `--pipeline-run <name>` 或 `--source-commit <full-sha>` 的定点清理必须先直接查询目标 PipelineRun,而不是只从全量列表过滤;不存在的目标返回 `target-pipelinerun-not-found`,未完成目标返回 `target-pipelinerun-not-terminal`,空查询和读取失败分别返回 `target-pipelinerun-query-empty` / `target-pipelinerun-query-failed`,年龄保护仍返回 `below-min-age`。`hwlab nodes control-plane cleanup-runs --node G14 --lane v03 --pipeline-run <name>` 是 v0.3 failed run 受控重试前的清理入口。
|
||||
- `hwlab g14 control-plane cleanup-released-pvs --lane all [--limit N] [--dry-run|--confirm]` 是 local-path 未自动回收后的补充 retention 入口;只列并删除 `Released`、`local-path`、`Delete`、`claimNamespace=hwlab-ci` 且 claim 名称形如 Tekton 临时 `pvc-*` 的 PV。
|
||||
- `hwlab g14 git-mirror status|apply|sync|flush [--dry-run|--confirm]` 是 `devops-infra` git mirror/relay 的受控维护入口:`apply` 渲染并 server-side apply `devops-infra/git-mirror.yaml`,同时删除遗留 `git-mirror-hwlab-sync` CronJob;`sync` 创建一次性 manual Job,把 GitHub allowlist refs 拉入本地 mirror;`flush` 创建一次性 manual Job,把本地 `v0.2-gitops` 快进推回 GitHub。
|
||||
|
||||
@@ -153,11 +153,58 @@ assertCondition(
|
||||
assertCondition(
|
||||
hwlabHelpUsage.some((line) => line.includes("hwlab nodes secret status --node G14 --lane v03 --name hwlab-v03-code-agent-provider"))
|
||||
&& hwlabHelpUsage.some((line) => line.includes("hwlab nodes secret ensure --node G14 --lane v03 --name hwlab-v03-code-agent-provider --confirm"))
|
||||
&& hwlabHelpUsage.some((line) => line.includes("hwlab nodes secret status --node G14 --lane v03 --name hwlab-cloud-api-v03-db"))
|
||||
&& hwlabHelpUsage.some((line) => line.includes("hwlab nodes secret ensure --node G14 --lane v03 --name hwlab-cloud-api-v03-db --confirm"))
|
||||
&& hwlabNodeHelpJson.includes("hwlab nodes secret status --node G14 --lane v03 --name hwlab-v03-code-agent-provider")
|
||||
&& hwlabNodeHelpJson.includes("hwlab nodes secret ensure --node G14 --lane v03 --name hwlab-v03-code-agent-provider --confirm"),
|
||||
"v0.3 node-scoped secret help must expose the controlled code-agent provider SecretRef bootstrap path",
|
||||
&& hwlabNodeHelpJson.includes("hwlab nodes secret ensure --node G14 --lane v03 --name hwlab-v03-code-agent-provider --confirm")
|
||||
&& hwlabNodeHelpJson.includes("hwlab nodes secret status --node G14 --lane v03 --name hwlab-cloud-api-v03-db")
|
||||
&& hwlabNodeHelpJson.includes("hwlab nodes secret ensure --node G14 --lane v03 --name hwlab-cloud-api-v03-db --confirm"),
|
||||
"v0.3 node-scoped secret help must expose the controlled code-agent provider and cloud-api DB SecretRef bootstrap paths",
|
||||
{ hwlabHelpUsage, hwlabNodeHelp: hwlabNodeHelp() },
|
||||
);
|
||||
const cloudApiDbStatus = nodeSecretStatusFromTextForTest([
|
||||
"namespace\thwlab-v03",
|
||||
"secret\thwlab-cloud-api-v03-db",
|
||||
"key\tdatabase-url",
|
||||
"preset\tcloud-api-db",
|
||||
"action\tensured",
|
||||
"dryRun\tfalse",
|
||||
"mutation\ttrue",
|
||||
"beforeExists\tno",
|
||||
"beforePostgresSecretExists\tyes",
|
||||
"beforeDatabaseUrlPresent\tno",
|
||||
"beforeDatabaseUrlBytes\t0",
|
||||
"afterExists\tyes",
|
||||
"afterDatabaseUrlPresent\tyes",
|
||||
"afterDatabaseUrlBytes\t112",
|
||||
"postgresAdminSecretPresent\tyes",
|
||||
"postgresSecret\thwlab-v03-postgres",
|
||||
"dbName\thwlab_v03",
|
||||
"dbUser\thwlab_v03",
|
||||
"dbHost\thwlab-v03-postgres.hwlab-v03.svc.cluster.local",
|
||||
"dbRoleExistsBefore\tt",
|
||||
"dbDatabaseExistsBefore\tt",
|
||||
"dbProbeExitCodeBefore\t0",
|
||||
"dbRoleExistsAfter\tt",
|
||||
"dbDatabaseExistsAfter\tt",
|
||||
"dbProbeExitCodeAfter\t0",
|
||||
"cloudApiDeployment\thwlab-cloud-api",
|
||||
"applyExitCode\t0",
|
||||
"dbEnsureExitCode\t0",
|
||||
"rolloutRestartExitCode\t0",
|
||||
"rolloutStatusExitCode\t0",
|
||||
].join("\n"), true, 0, "");
|
||||
assertCondition(
|
||||
record(cloudApiDbStatus).ok === true
|
||||
&& record(cloudApiDbStatus).valuesRedacted === true
|
||||
&& record(record(cloudApiDbStatus).after).exists === true
|
||||
&& record(record(record(cloudApiDbStatus).after).databaseUrl).valueBytes === 112
|
||||
&& record(cloudApiDbStatus).cloudApiDeployment === "hwlab-cloud-api"
|
||||
&& !JSON.stringify(cloudApiDbStatus).includes("postgres://")
|
||||
&& !JSON.stringify(cloudApiDbStatus).includes("password"),
|
||||
"cloud-api DB Secret status must be redacted while proving DB SecretRef and runtime database are present",
|
||||
cloudApiDbStatus,
|
||||
);
|
||||
const codeAgentProviderStatus = nodeSecretStatusFromTextForTest([
|
||||
"namespace\thwlab-v03",
|
||||
"secret\thwlab-v03-code-agent-provider",
|
||||
|
||||
@@ -9293,6 +9293,8 @@ export function hwlabG14Help(): Record<string, unknown> {
|
||||
"bun scripts/cli.ts hwlab g14 secret ensure --lane v02 --name hwlab-v02-master-server-admin-api-key --confirm",
|
||||
"bun scripts/cli.ts hwlab nodes secret status --node G14 --lane v03 --name hwlab-v03-master-server-admin-api-key",
|
||||
"bun scripts/cli.ts hwlab nodes secret ensure --node G14 --lane v03 --name hwlab-v03-master-server-admin-api-key --confirm",
|
||||
"bun scripts/cli.ts hwlab nodes secret status --node G14 --lane v03 --name hwlab-cloud-api-v03-db",
|
||||
"bun scripts/cli.ts hwlab nodes secret ensure --node G14 --lane v03 --name hwlab-cloud-api-v03-db --confirm",
|
||||
"bun scripts/cli.ts hwlab nodes secret status --node G14 --lane v03 --name hwlab-v03-code-agent-provider",
|
||||
"bun scripts/cli.ts hwlab nodes secret ensure --node G14 --lane v03 --name hwlab-v03-code-agent-provider --confirm",
|
||||
"bun scripts/cli.ts hwlab g14 secret delete --lane v02 --name <obsolete-hwlab-v02-secret> --dry-run",
|
||||
@@ -9320,7 +9322,7 @@ export function hwlabG14Help(): Record<string, unknown> {
|
||||
"bun scripts/cli.ts hwlab g14 upstream-image ensure --name openfga --tag v1.17.0 --confirm",
|
||||
"bun scripts/cli.ts job status <jobId> --tail-bytes 30000",
|
||||
],
|
||||
description: "G14 HWLAB PR monitor, legacy DEV/PROD retirement status/plan/execute command, bounded v0.2 control-plane bootstrap/cleanup/runtime-migration helper, node-scoped runtime lane v03 control-plane apply/status/refresh/trigger entry, runtime lane SecretRef bootstrap, devops-infra git mirror and observability maintenance, controlled CI tools image build/status entry, and allowlisted upstream image mirroring. The legacy base=G14 monitor is blocked by the retirement contract; the local retirement marker records live execution evidence. `--lane v02` monitors base=v0.2 PRs, waits for GitHub preflight/CI readiness, automatically merges ready PRs without waiting for other active v0.2 PipelineRuns, triggers v0.2 CD with latest-only GitOps writeback, flushes the git mirror when needed, and posts deduplicated PR comments for pending, blocked/conflict, success, superseded, failure, or timeout states. confirmed control-plane trigger-current and git-mirror sync/flush also return async jobs by default, with --wait reserved for explicit synchronous debugging. control-plane v02 keeps the full closeout/cleanup/runtime-migration verdict path; v03+ is advertised through `hwlab nodes ... --node <node-id> --lane vNN` so node identity remains configuration data instead of a command family. secret status/ensure is the standard runtime lane SecretRef bootstrap path for OpenFGA and master admin API key; it never reads or prints secret values. upstream-image status/ensure only mirrors allowlisted upstream runtime images into the G14 local registry. git-mirror status/apply/sync/flush is the manual devops-infra mirror/relay control path and does not install a CronJob. observability status/apply/query/targets/boundary/closeout owns the shared Prometheus Operator and Prometheus instance in devops-infra, adds bounded PromQL assertions and semantic closeout summaries, while HWLAB lane manifests own only ServiceMonitor and PrometheusRule objects.",
|
||||
description: "G14 HWLAB PR monitor, legacy DEV/PROD retirement status/plan/execute command, bounded v0.2 control-plane bootstrap/cleanup/runtime-migration helper, node-scoped runtime lane v03 control-plane apply/status/refresh/trigger entry, runtime lane SecretRef bootstrap, devops-infra git mirror and observability maintenance, controlled CI tools image build/status entry, and allowlisted upstream image mirroring. The legacy base=G14 monitor is blocked by the retirement contract; the local retirement marker records live execution evidence. `--lane v02` monitors base=v0.2 PRs, waits for GitHub preflight/CI readiness, automatically merges ready PRs without waiting for other active v0.2 PipelineRuns, triggers v0.2 CD with latest-only GitOps writeback, flushes the git mirror when needed, and posts deduplicated PR comments for pending, blocked/conflict, success, superseded, failure, or timeout states. confirmed control-plane trigger-current and git-mirror sync/flush also return async jobs by default, with --wait reserved for explicit synchronous debugging. control-plane v02 keeps the full closeout/cleanup/runtime-migration verdict path; v03+ is advertised through `hwlab nodes ... --node <node-id> --lane vNN` so node identity remains configuration data instead of a command family. secret status/ensure is the standard runtime lane SecretRef bootstrap path for OpenFGA, cloud-api DB, master admin API key, and code-agent provider refs; it never reads or prints secret values. upstream-image status/ensure only mirrors allowlisted upstream runtime images into the G14 local registry. git-mirror status/apply/sync/flush is the manual devops-infra mirror/relay control path and does not install a CronJob. observability status/apply/query/targets/boundary/closeout owns the shared Prometheus Operator and Prometheus instance in devops-infra, adds bounded PromQL assertions and semantic closeout summaries, while HWLAB lane manifests own only ServiceMonitor and PrometheusRule objects.",
|
||||
defaults: {
|
||||
repo: HWLAB_REPO,
|
||||
legacyBase: G14_SOURCE_BRANCH,
|
||||
|
||||
+231
-4
@@ -6,7 +6,7 @@ import { runHwlabG14Command } from "./hwlab-g14";
|
||||
import { hwlabRuntimeLaneConfigPath, hwlabRuntimeLaneSpec, isHwlabRuntimeLane, type HwlabRuntimeLane } from "./hwlab-node-lanes";
|
||||
|
||||
type SecretAction = "status" | "ensure";
|
||||
type SecretPreset = "openfga" | "master-server-admin-api-key" | "code-agent-provider";
|
||||
type SecretPreset = "openfga" | "master-server-admin-api-key" | "code-agent-provider" | "cloud-api-db";
|
||||
type DelegatedNodeDomain = "control-plane" | "git-mirror";
|
||||
|
||||
interface NodeSecretOptions {
|
||||
@@ -33,6 +33,12 @@ interface RuntimeSecretSpec {
|
||||
openFgaDbUser: string;
|
||||
openFgaDbHost: string;
|
||||
masterAdminApiKeySecret: string;
|
||||
cloudApiDbSecret: string;
|
||||
cloudApiDbKey: string;
|
||||
cloudApiDbName: string;
|
||||
cloudApiDbUser: string;
|
||||
cloudApiDbHost: string;
|
||||
cloudApiDeployment: string;
|
||||
codeAgentProviderSecret: string;
|
||||
codeAgentProviderSourceNamespace: string;
|
||||
codeAgentProviderSourceSecret: string;
|
||||
@@ -44,6 +50,7 @@ const MASTER_ADMIN_API_KEY_KEY = "api-key";
|
||||
const OPENFGA_AUTHN_KEY = "authn-preshared-key";
|
||||
const OPENFGA_DATASTORE_URI_KEY = "datastore-uri";
|
||||
const OPENFGA_POSTGRES_PASSWORD_KEY = "postgres-password";
|
||||
const CLOUD_API_DB_KEY = "database-url";
|
||||
const CODE_AGENT_PROVIDER_OPENAI_KEY = "openai-api-key";
|
||||
const CODE_AGENT_PROVIDER_OPENCODE_KEY = "opencode-api-key";
|
||||
const CODE_AGENT_PROVIDER_SOURCE_NAMESPACE = "hwlab-v02";
|
||||
@@ -77,6 +84,8 @@ export function hwlabNodeHelp(): Record<string, unknown> {
|
||||
"bun scripts/cli.ts hwlab nodes secret status --node G14 --lane v03 --name hwlab-v03-openfga",
|
||||
"bun scripts/cli.ts hwlab nodes secret ensure --node G14 --lane v03 --name hwlab-v03-openfga --confirm",
|
||||
"bun scripts/cli.ts hwlab nodes secret ensure --node G14 --lane v03 --name hwlab-v03-master-server-admin-api-key --confirm",
|
||||
"bun scripts/cli.ts hwlab nodes secret status --node G14 --lane v03 --name hwlab-cloud-api-v03-db",
|
||||
"bun scripts/cli.ts hwlab nodes secret ensure --node G14 --lane v03 --name hwlab-cloud-api-v03-db --confirm",
|
||||
"bun scripts/cli.ts hwlab nodes secret status --node G14 --lane v03 --name hwlab-v03-code-agent-provider",
|
||||
"bun scripts/cli.ts hwlab nodes secret ensure --node G14 --lane v03 --name hwlab-v03-code-agent-provider --confirm",
|
||||
],
|
||||
@@ -195,7 +204,7 @@ function rewriteDelegatedNodeString(value: string, scoped: ReturnType<typeof par
|
||||
function parseSecretOptions(args: string[]): NodeSecretOptions {
|
||||
const [actionRaw] = args;
|
||||
if (actionRaw !== "status" && actionRaw !== "ensure") {
|
||||
throw new Error("secret usage: status|ensure --node NODE --lane vNN --name hwlab-vNN-openfga|hwlab-vNN-master-server-admin-api-key|hwlab-vNN-code-agent-provider [--dry-run|--confirm]");
|
||||
throw new Error("secret usage: status|ensure --node NODE --lane vNN --name hwlab-vNN-openfga|hwlab-vNN-master-server-admin-api-key|hwlab-cloud-api-vNN-db|hwlab-vNN-code-agent-provider [--dry-run|--confirm]");
|
||||
}
|
||||
const node = requiredOption(args, "--node");
|
||||
assertNodeId(node);
|
||||
@@ -237,8 +246,22 @@ function parseSecretOptions(args: string[]): NodeSecretOptions {
|
||||
timeoutSeconds: positiveIntegerOption(args, "--timeout-seconds", 180, 900),
|
||||
};
|
||||
}
|
||||
if (name === spec.cloudApiDbSecret) {
|
||||
if (key !== undefined && key !== spec.cloudApiDbKey) throw new Error(`secret ${name} supports only key ${spec.cloudApiDbKey}`);
|
||||
return {
|
||||
action: actionRaw,
|
||||
node,
|
||||
lane,
|
||||
name,
|
||||
key: key ?? spec.cloudApiDbKey,
|
||||
preset: "cloud-api-db",
|
||||
confirm,
|
||||
dryRun: actionRaw === "status" ? true : explicitDryRun || !confirm,
|
||||
timeoutSeconds: positiveIntegerOption(args, "--timeout-seconds", 240, 900),
|
||||
};
|
||||
}
|
||||
if (name !== spec.openFgaSecret) {
|
||||
throw new Error(`secret status/ensure supports --name ${spec.openFgaSecret}, ${spec.masterAdminApiKeySecret}, or ${spec.codeAgentProviderSecret} for --lane ${lane}`);
|
||||
throw new Error(`secret status/ensure supports --name ${spec.openFgaSecret}, ${spec.masterAdminApiKeySecret}, ${spec.cloudApiDbSecret}, or ${spec.codeAgentProviderSecret} for --lane ${lane}`);
|
||||
}
|
||||
if (key !== undefined && key !== OPENFGA_AUTHN_KEY && key !== OPENFGA_DATASTORE_URI_KEY && key !== OPENFGA_POSTGRES_PASSWORD_KEY) {
|
||||
throw new Error(`secret ${name} supports keys ${OPENFGA_AUTHN_KEY}, ${OPENFGA_DATASTORE_URI_KEY}, and ${OPENFGA_POSTGRES_PASSWORD_KEY}`);
|
||||
@@ -270,6 +293,12 @@ function runtimeSecretSpec(input: { node: string; lane: string }): RuntimeSecret
|
||||
openFgaDbUser: "hwlab_openfga",
|
||||
openFgaDbHost: `${namespace}-postgres.${namespace}.svc.cluster.local`,
|
||||
masterAdminApiKeySecret: `${namespace}-master-server-admin-api-key`,
|
||||
cloudApiDbSecret: `hwlab-cloud-api-${input.lane}-db`,
|
||||
cloudApiDbKey: CLOUD_API_DB_KEY,
|
||||
cloudApiDbName: `hwlab_${input.lane}`,
|
||||
cloudApiDbUser: `hwlab_${input.lane}`,
|
||||
cloudApiDbHost: `${namespace}-postgres.${namespace}.svc.cluster.local`,
|
||||
cloudApiDeployment: "hwlab-cloud-api",
|
||||
codeAgentProviderSecret: `${namespace}-code-agent-provider`,
|
||||
codeAgentProviderSourceNamespace: CODE_AGENT_PROVIDER_SOURCE_NAMESPACE,
|
||||
codeAgentProviderSourceSecret: CODE_AGENT_PROVIDER_SOURCE_SECRET,
|
||||
@@ -286,7 +315,9 @@ function runNodeSecret(options: NodeSecretOptions): Record<string, unknown> {
|
||||
? openFgaSecretScript(options, spec)
|
||||
: options.preset === "master-server-admin-api-key"
|
||||
? masterAdminApiKeySecretScript(options, spec)
|
||||
: codeAgentProviderSecretScript(options, spec);
|
||||
: options.preset === "cloud-api-db"
|
||||
? cloudApiDbSecretScript(options, spec)
|
||||
: codeAgentProviderSecretScript(options, spec);
|
||||
const result = runTransScript(options.node, script, input, options.timeoutSeconds);
|
||||
const status = secretStatusFromText(statusText(result), result.exitCode === 0, result.exitCode, result.stderr, spec);
|
||||
const dryRunOk = options.action === "ensure" && options.dryRun && result.exitCode === 0;
|
||||
@@ -626,6 +657,150 @@ function codeAgentProviderSecretScript(options: NodeSecretOptions, spec: Runtime
|
||||
].join("\n");
|
||||
}
|
||||
|
||||
function cloudApiDbSecretScript(options: NodeSecretOptions, spec: RuntimeSecretSpec): string {
|
||||
return [
|
||||
"set +e",
|
||||
`namespace=${shellQuote(spec.namespace)}`,
|
||||
`name=${shellQuote(spec.cloudApiDbSecret)}`,
|
||||
`database_url_key=${shellQuote(spec.cloudApiDbKey)}`,
|
||||
`postgres_secret=${shellQuote(spec.postgresSecret)}`,
|
||||
`postgres_statefulset=${shellQuote(spec.postgresStatefulSet)}`,
|
||||
`postgres_admin_user=${shellQuote(spec.postgresAdminUser)}`,
|
||||
`db_name=${shellQuote(spec.cloudApiDbName)}`,
|
||||
`db_user=${shellQuote(spec.cloudApiDbUser)}`,
|
||||
`db_host=${shellQuote(spec.cloudApiDbHost)}`,
|
||||
`cloud_api_deployment=${shellQuote(spec.cloudApiDeployment)}`,
|
||||
`action_request=${shellQuote(options.action)}`,
|
||||
`dry_run=${shellQuote(options.dryRun ? "true" : "false")}`,
|
||||
`field_manager=${shellQuote(spec.fieldManager)}`,
|
||||
"preset=cloud-api-db",
|
||||
"secret_exists_flag() { kubectl -n \"$namespace\" get secret \"$1\" >/dev/null 2>&1 && printf yes || printf no; }",
|
||||
"secret_b64_key() { kubectl -n \"$namespace\" get secret \"$1\" -o \"go-template={{ index .data \\\"$2\\\" }}\" 2>/dev/null || true; }",
|
||||
"decoded_value() { if [ -n \"$1\" ]; then printf '%s' \"$1\" | base64 -d 2>/dev/null || true; fi; }",
|
||||
"decoded_length() { if [ -n \"$1\" ]; then printf '%s' \"$1\" | base64 -d 2>/dev/null | wc -c | tr -d ' '; else printf '0'; fi; }",
|
||||
"psql_scalar() { kubectl -n \"$namespace\" exec \"statefulset/$postgres_statefulset\" -c postgres -- env PGPASSWORD=\"$postgres_admin_password\" psql -U \"$postgres_admin_user\" -d postgres -tAc \"$1\" 2>/dev/null | tr -d '[:space:]'; }",
|
||||
"probe_db() {",
|
||||
" role_result=unknown",
|
||||
" database_result=unknown",
|
||||
" probe_exit=missing-postgres-admin-secret",
|
||||
" if [ -n \"$postgres_admin_password\" ]; then",
|
||||
" role_result=$(psql_scalar \"select exists(select 1 from pg_roles where rolname='$db_user');\")",
|
||||
" role_exit=$?",
|
||||
" database_result=$(psql_scalar \"select exists(select 1 from pg_database where datname='$db_name');\")",
|
||||
" database_exit=$?",
|
||||
" if [ \"$role_exit\" -eq 0 ] && [ \"$database_exit\" -eq 0 ]; then probe_exit=0; else probe_exit=$role_exit/$database_exit; fi",
|
||||
" fi",
|
||||
"}",
|
||||
"before_exists=$(secret_exists_flag \"$name\")",
|
||||
"before_postgres_exists=$(secret_exists_flag \"$postgres_secret\")",
|
||||
"before_url_b64=$(secret_b64_key \"$name\" \"$database_url_key\")",
|
||||
"before_url_present=$([ -n \"$before_url_b64\" ] && printf yes || printf no)",
|
||||
"before_url_bytes=$(decoded_length \"$before_url_b64\")",
|
||||
"database_url=$(decoded_value \"$before_url_b64\")",
|
||||
"postgres_admin_b64=$(secret_b64_key \"$postgres_secret\" POSTGRES_PASSWORD)",
|
||||
"postgres_admin_present=$([ -n \"$postgres_admin_b64\" ] && printf yes || printf no)",
|
||||
"postgres_admin_password=$(decoded_value \"$postgres_admin_b64\")",
|
||||
"probe_db",
|
||||
"db_role_exists_before=$role_result",
|
||||
"db_database_exists_before=$database_result",
|
||||
"db_probe_exit_before=$probe_exit",
|
||||
"action=observed",
|
||||
"mutation=false",
|
||||
"apply_exit=",
|
||||
"db_ensure_exit=",
|
||||
"rollout_restart_exit=",
|
||||
"rollout_status_exit=",
|
||||
"if [ \"$action_request\" = ensure ]; then",
|
||||
" missing_secret=false",
|
||||
" [ \"$before_url_present\" = yes ] && [ \"$before_url_bytes\" -gt 0 ] || missing_secret=true",
|
||||
" missing_db=false",
|
||||
" [ \"$db_role_exists_before\" = t ] || missing_db=true",
|
||||
" [ \"$db_database_exists_before\" = t ] || missing_db=true",
|
||||
" if [ \"$dry_run\" = true ]; then",
|
||||
" if [ \"$before_postgres_exists\" != yes ] || [ \"$postgres_admin_present\" != yes ] || [ \"$missing_secret\" = true ] || [ \"$missing_db\" = true ]; then action=would-ensure; else action=kept; fi",
|
||||
" elif [ \"$before_postgres_exists\" != yes ] || [ \"$postgres_admin_present\" != yes ] || [ -z \"$postgres_admin_password\" ]; then",
|
||||
" action=postgres-admin-secret-missing",
|
||||
" apply_exit=44",
|
||||
" elif [ \"$missing_secret\" = false ] && [ \"$missing_db\" = false ]; then",
|
||||
" action=kept",
|
||||
" else",
|
||||
" [ -n \"$database_url\" ] || database_url=\"postgres://$db_user:$postgres_admin_password@$db_host:5432/$db_name?sslmode=disable\"",
|
||||
" kubectl -n \"$namespace\" create secret generic \"$name\" --from-literal=\"$database_url_key=$database_url\" --dry-run=client -o yaml | kubectl apply --server-side --force-conflicts --field-manager=\"$field_manager\" -f -",
|
||||
" apply_exit=$?",
|
||||
" if [ \"$apply_exit\" -eq 0 ]; then",
|
||||
" kubectl -n \"$namespace\" exec -i \"statefulset/$postgres_statefulset\" -c postgres -- env PGPASSWORD=\"$postgres_admin_password\" psql -v ON_ERROR_STOP=1 -U \"$postgres_admin_user\" -d postgres -v db_name=\"$db_name\" -v db_user=\"$db_user\" -v db_pass=\"$postgres_admin_password\" >/tmp/hwlab-cloud-api-db-psql.out 2>/tmp/hwlab-cloud-api-db-psql.err <<'SQL'",
|
||||
"SELECT format('CREATE ROLE %I LOGIN PASSWORD %L', :'db_user', :'db_pass')",
|
||||
"WHERE NOT EXISTS (SELECT 1 FROM pg_roles WHERE rolname = :'db_user')",
|
||||
"\\gexec",
|
||||
"ALTER ROLE :\"db_user\" LOGIN PASSWORD :'db_pass';",
|
||||
"SELECT format('CREATE DATABASE %I OWNER %I', :'db_name', :'db_user')",
|
||||
"WHERE NOT EXISTS (SELECT 1 FROM pg_database WHERE datname = :'db_name')",
|
||||
"\\gexec",
|
||||
"ALTER DATABASE :\"db_name\" OWNER TO :\"db_user\";",
|
||||
"SQL",
|
||||
" db_ensure_exit=$?",
|
||||
" if [ \"$db_ensure_exit\" -eq 0 ]; then",
|
||||
" if [ \"$missing_secret\" = true ] || [ \"$missing_db\" = true ]; then",
|
||||
" kubectl -n \"$namespace\" rollout restart \"deployment/$cloud_api_deployment\" >/tmp/hwlab-cloud-api-rollout-restart.out 2>/tmp/hwlab-cloud-api-rollout-restart.err",
|
||||
" rollout_restart_exit=$?",
|
||||
" if [ \"$rollout_restart_exit\" -eq 0 ]; then",
|
||||
" kubectl -n \"$namespace\" rollout status \"deployment/$cloud_api_deployment\" --timeout=180s >/tmp/hwlab-cloud-api-rollout-status.out 2>/tmp/hwlab-cloud-api-rollout-status.err",
|
||||
" rollout_status_exit=$?",
|
||||
" fi",
|
||||
" fi",
|
||||
" if [ -n \"$rollout_restart_exit\" ] && [ \"$rollout_restart_exit\" != 0 ]; then action=rollout-restart-failed",
|
||||
" elif [ -n \"$rollout_status_exit\" ] && [ \"$rollout_status_exit\" != 0 ]; then action=rollout-status-failed",
|
||||
" else action=ensured; mutation=true; fi",
|
||||
" else action=db-ensure-failed; fi",
|
||||
" else action=apply-failed; fi",
|
||||
" fi",
|
||||
"fi",
|
||||
"after_exists=$(secret_exists_flag \"$name\")",
|
||||
"after_url_b64=$(secret_b64_key \"$name\" \"$database_url_key\")",
|
||||
"after_url_present=$([ -n \"$after_url_b64\" ] && printf yes || printf no)",
|
||||
"after_url_bytes=$(decoded_length \"$after_url_b64\")",
|
||||
"probe_db",
|
||||
"db_role_exists_after=$role_result",
|
||||
"db_database_exists_after=$database_result",
|
||||
"db_probe_exit_after=$probe_exit",
|
||||
"printf 'namespace\\t%s\\n' \"$namespace\"",
|
||||
"printf 'secret\\t%s\\n' \"$name\"",
|
||||
"printf 'key\\t%s\\n' \"$database_url_key\"",
|
||||
"printf 'preset\\t%s\\n' \"$preset\"",
|
||||
"printf 'action\\t%s\\n' \"$action\"",
|
||||
"printf 'dryRun\\t%s\\n' \"$dry_run\"",
|
||||
"printf 'mutation\\t%s\\n' \"$mutation\"",
|
||||
"printf 'beforeExists\\t%s\\n' \"$before_exists\"",
|
||||
"printf 'beforePostgresSecretExists\\t%s\\n' \"$before_postgres_exists\"",
|
||||
"printf 'beforeDatabaseUrlPresent\\t%s\\n' \"$before_url_present\"",
|
||||
"printf 'beforeDatabaseUrlBytes\\t%s\\n' \"$before_url_bytes\"",
|
||||
"printf 'afterExists\\t%s\\n' \"$after_exists\"",
|
||||
"printf 'afterDatabaseUrlPresent\\t%s\\n' \"$after_url_present\"",
|
||||
"printf 'afterDatabaseUrlBytes\\t%s\\n' \"$after_url_bytes\"",
|
||||
"printf 'postgresAdminSecretPresent\\t%s\\n' \"$postgres_admin_present\"",
|
||||
"printf 'postgresSecret\\t%s\\n' \"$postgres_secret\"",
|
||||
"printf 'dbName\\t%s\\n' \"$db_name\"",
|
||||
"printf 'dbUser\\t%s\\n' \"$db_user\"",
|
||||
"printf 'dbHost\\t%s\\n' \"$db_host\"",
|
||||
"printf 'dbRoleExistsBefore\\t%s\\n' \"$db_role_exists_before\"",
|
||||
"printf 'dbDatabaseExistsBefore\\t%s\\n' \"$db_database_exists_before\"",
|
||||
"printf 'dbProbeExitCodeBefore\\t%s\\n' \"$db_probe_exit_before\"",
|
||||
"printf 'dbRoleExistsAfter\\t%s\\n' \"$db_role_exists_after\"",
|
||||
"printf 'dbDatabaseExistsAfter\\t%s\\n' \"$db_database_exists_after\"",
|
||||
"printf 'dbProbeExitCodeAfter\\t%s\\n' \"$db_probe_exit_after\"",
|
||||
"printf 'cloudApiDeployment\\t%s\\n' \"$cloud_api_deployment\"",
|
||||
"printf 'applyExitCode\\t%s\\n' \"$apply_exit\"",
|
||||
"printf 'dbEnsureExitCode\\t%s\\n' \"$db_ensure_exit\"",
|
||||
"printf 'rolloutRestartExitCode\\t%s\\n' \"$rollout_restart_exit\"",
|
||||
"printf 'rolloutStatusExitCode\\t%s\\n' \"$rollout_status_exit\"",
|
||||
"database_url= postgres_admin_password=",
|
||||
"if [ -n \"$apply_exit\" ] && [ \"$apply_exit\" != 0 ]; then exit \"$apply_exit\"; fi",
|
||||
"if [ -n \"$db_ensure_exit\" ] && [ \"$db_ensure_exit\" != 0 ]; then exit \"$db_ensure_exit\"; fi",
|
||||
"if [ -n \"$rollout_restart_exit\" ] && [ \"$rollout_restart_exit\" != 0 ]; then exit \"$rollout_restart_exit\"; fi",
|
||||
"if [ -n \"$rollout_status_exit\" ] && [ \"$rollout_status_exit\" != 0 ]; then exit \"$rollout_status_exit\"; fi",
|
||||
].join("\n");
|
||||
}
|
||||
|
||||
function secretStatusFromText(text: string, commandOk: boolean, exitCode: number | null, stderr: string, spec: RuntimeSecretSpec): Record<string, unknown> {
|
||||
const fields = keyValueLinesFromText(text);
|
||||
if (fields.preset === "master-server-admin-api-key") {
|
||||
@@ -691,6 +866,58 @@ function secretStatusFromText(text: string, commandOk: boolean, exitCode: number
|
||||
summary: healthy ? `${fields.secret || spec.codeAgentProviderSecret} has a usable provider key` : `${fields.secret || spec.codeAgentProviderSecret} missing provider keys`,
|
||||
};
|
||||
}
|
||||
if (fields.preset === "cloud-api-db") {
|
||||
const beforeUrlBytes = numericField(fields.beforeDatabaseUrlBytes);
|
||||
const afterUrlBytes = numericField(fields.afterDatabaseUrlBytes);
|
||||
const keysHealthy = fields.afterExists === "yes" &&
|
||||
fields.afterDatabaseUrlPresent === "yes" &&
|
||||
typeof afterUrlBytes === "number" && afterUrlBytes > 0;
|
||||
const databaseHealthy = fields.dbRoleExistsAfter === "t" && fields.dbDatabaseExistsAfter === "t";
|
||||
const healthy = keysHealthy && databaseHealthy;
|
||||
return {
|
||||
ok: commandOk && healthy,
|
||||
namespace: fields.namespace || spec.namespace,
|
||||
secret: fields.secret || spec.cloudApiDbSecret,
|
||||
key: fields.key || spec.cloudApiDbKey,
|
||||
preset: "cloud-api-db",
|
||||
action: fields.action || null,
|
||||
dryRun: fields.dryRun === "true",
|
||||
mutation: fields.mutation === "true",
|
||||
before: {
|
||||
exists: fields.beforeExists === "yes",
|
||||
postgresSecretExists: fields.beforePostgresSecretExists === "yes",
|
||||
databaseUrl: { keyPresent: fields.beforeDatabaseUrlPresent === "yes", valueBytes: beforeUrlBytes },
|
||||
database: {
|
||||
roleExists: fields.dbRoleExistsBefore || "unknown",
|
||||
databaseExists: fields.dbDatabaseExistsBefore || "unknown",
|
||||
probeExitCode: fields.dbProbeExitCodeBefore || null,
|
||||
},
|
||||
},
|
||||
after: {
|
||||
exists: fields.afterExists === "yes",
|
||||
databaseUrl: { keyPresent: fields.afterDatabaseUrlPresent === "yes", valueBytes: afterUrlBytes },
|
||||
database: {
|
||||
roleExists: fields.dbRoleExistsAfter || "unknown",
|
||||
databaseExists: fields.dbDatabaseExistsAfter || "unknown",
|
||||
probeExitCode: fields.dbProbeExitCodeAfter || null,
|
||||
},
|
||||
},
|
||||
postgresAdminSecretPresent: fields.postgresAdminSecretPresent === "yes",
|
||||
postgresSecret: fields.postgresSecret || spec.postgresSecret,
|
||||
dbName: fields.dbName || spec.cloudApiDbName,
|
||||
dbUser: fields.dbUser || spec.cloudApiDbUser,
|
||||
dbHost: fields.dbHost || spec.cloudApiDbHost,
|
||||
cloudApiDeployment: fields.cloudApiDeployment || spec.cloudApiDeployment,
|
||||
applyExitCode: numericField(fields.applyExitCode),
|
||||
dbEnsureExitCode: numericField(fields.dbEnsureExitCode),
|
||||
rolloutRestartExitCode: numericField(fields.rolloutRestartExitCode),
|
||||
rolloutStatusExitCode: numericField(fields.rolloutStatusExitCode),
|
||||
exitCode,
|
||||
stderr: commandOk ? "" : stderr.trim().slice(0, 2000),
|
||||
valuesRedacted: true,
|
||||
summary: healthy ? `${fields.secret || spec.cloudApiDbSecret}/${fields.key || spec.cloudApiDbKey} exists and runtime database is present` : `${fields.secret || spec.cloudApiDbSecret}/${fields.key || spec.cloudApiDbKey} or runtime database missing`,
|
||||
};
|
||||
}
|
||||
const afterAuthnBytes = numericField(fields.afterAuthnBytes);
|
||||
const afterUriBytes = numericField(fields.afterDatastoreUriBytes);
|
||||
const afterPasswordBytes = numericField(fields.afterPostgresPasswordBytes);
|
||||
|
||||
Reference in New Issue
Block a user