diff --git a/AGENTS.md b/AGENTS.md index 73d03bc2..c08d1903 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -65,7 +65,7 @@ UniDesk 是一个以主 server 为统一入口的分布式工作平台；本文 - `bun scripts/cli.ts auth-broker contract|health --dry-run|credential-request --dry-run|pr-preflight --dry-run`：查看 Auth Broker P0 Rust skeleton 与 CLI adapter contract，runner 无 `GH_TOKEN`/`GITHUB_TOKEN` 时返回结构化 `auth-missing`/`broker-needed`，不读取或打印 token 值，规则见 `docs/reference/auth-broker.md`。 - `bun scripts/cli.ts gh preflight|auth status|issue ...|pr list|files|diff --stat|read|view|preflight|closeout|create|edit|update|comment` / `bun scripts/code-queue-pr-preflight-example.ts`：通过 REST 执行安全 GitHub issue 读写、脱敏 auth/status 诊断、body-file Markdown 写入、当日滚动简报时间线 ClaudeQQ 通知、escape 扫描、只读 cleanup-plan 和 #20 board-audit、PR changed-file/stat summary、PR 创建/评论 dry-run、REST-only 低噪声 PR title/body 编辑、PR 收口元数据观察（含 merged/closed 区分与 merge commit）、低噪声 PR 收口 preflight 与 runner PR preflight；`gh issue/pr read|view` 支持 `owner/repo#number` shorthand，`--raw|--full` 是显式完整披露别名，`gh pr diff` 仅支持 `--stat` 紧凑 JSON，`gh pr merge` 当前仍结构化拒绝但普通 PR 可按任务边界用 repo-owned GitHub 路径收口，规则见 `docs/reference/cli.md` 和 `docs/reference/code-queue-supervision.md`。 - `bun scripts/cli.ts commander contract|plan --dry-run|smoke --dry-run|approval request --dry-run|prompt-lint --kind gpt55-pr`：查看 host Codex 指挥官直管微服务 skeleton 的 source/contract、无 daemon smoke 验证计划、.state/commander/ 状态模型、trace summary 聚合、ClaudeQQ 高风险请示草案和 GPT-5.5 PR prompt 边界辅助 lint；当前只返回 dry-run 计划和 backend-core `microservice proxy claudeqq` 授权后候选命令，不接 live bridge、不接管人工指挥官，不发送消息，`prompt-lint` 不作为业务 PR 门禁也不改变 `codex submit` 默认行为，规则见 `docs/reference/host-codex-commander.md`。 -- `bun scripts/cli.ts hwlab cd status --env dev` / `bun scripts/cli.ts hwlab cd apply --env dev --dry-run`：HWLAB DEV CD 指挥侧 wrapper，调用 HWLAB repo-owned 发布脚本并强制 D601 原生 k3s kubeconfig；真实 apply 只暴露 host-commander-only 命令形状，规则见 `docs/reference/hwlab.md`。 +- `bun scripts/cli.ts hwlab cd status --env dev` / `bun scripts/cli.ts hwlab cd apply --env dev --dry-run`：HWLAB DEV CD 指挥侧 wrapper，通过 D601 provider SSH 调用 HWLAB repo-owned `scripts/dev-cd-apply.mjs` 并强制原生 k3s kubeconfig；真实 apply 只暴露 host-commander-only 命令形状，规则见 `docs/reference/hwlab.md`。 - `bun scripts/cli.ts ci install/status/run/publish-backend-core/publish-user-service/run-dev-e2e/logs`：在 D601 原生 k3s 上安装和运行 Tekton CI，支持每 commit 检查、Code Queue 只读性能门禁、`CI.json` catalog 驱动的 backend-core 与 user-service commit-pinned 镜像发布和手动触发的 `origin/master:deploy.json#environments.dev` 临时 namespace e2e；catalog/producer/consumer 分工见 `docs/reference/cicd-standardization.md`，`run-dev-e2e` 的 Git 控制 runner、短 launcher 和 no-CD 边界见 `docs/reference/dev-ci-runner.md`，Tekton 规则见 `docs/reference/ci.md`。 - `bun scripts/cli.ts codex deploy `：旧 Code Queue 兼容部署入口已禁用，原因是它会绕过受控部署边界直连 D601 部署 Code Queue；规则见 `docs/reference/codex-deploy.md`。 - `bun scripts/cli.ts codex prompt-lint [prompt|--prompt-file path|--prompt-stdin]` / `codex submit [prompt] [--prompt-file path|--prompt-stdin] [--queue ]` / `codex pr-preflight [--remote]`：`prompt-lint` 在派发/steer 前 dry-run 检查 runner prompt 的 DEV 测试授权分级（`read-only`/`live-read`/`live-mutating`）且不回显 prompt；`submit --dry-run` 同时给出 MiniMax/GPT/人工路由建议、该 lint 结果和 requested/effective execution mode；真实提交成功只返回写入确认、task id、服务级 runnerPermissions 和后续查看命令，不回显 prompt；`pr-preflight` 只读检查 D601 scheduler/runner 的 GitHub token、egress 和 PR 能力，PR 型派单前必须使用，规则见 `docs/reference/cli.md` 和 `docs/reference/code-queue-supervision.md`。 diff --git a/docs/reference/cli.md b/docs/reference/cli.md index ef02e098..72f090d9 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -31,7 +31,7 @@ CLI 可以从 `master` 快速演进，但必须兼容 `deploy.json` 固定的 CI - `dev-env prewarm-images [--image image] [--provider-id D601] [--no-pull] [--proxy-url URL] [--pull-timeout-ms N] [--dry-run]` 创建异步 job，通过 UniDesk SSH 维护桥在 D601 上把开发底座依赖镜像从 Docker 缓存导入原生 k3s containerd。默认镜像是 `postgres:16-alpine` 和 `rancher/mirrored-library-busybox:1.36.1`，用于避免 `postgres-dev` 与 local-path helper pod 卡在外部 registry 拉取。该命令固定验证 `/etc/rancher/k3s/k3s.yaml` 指向的 native k3s 上下文，并输出 `dev_env_containerd_image_ready=...` 作为成功判据；它不 apply manifest、不修改生产 `unidesk` namespace。 - `artifact-registry plan|render|status|health|install|deploy-backend-core|deploy-service` 管理 D601 host-managed CNCF Distribution registry 的声明、安装、只读检查和 pull-only artifact CD。该 registry 固定为 D601 loopback `127.0.0.1:5000`，由 systemd + Docker Compose 管理，位于 native k3s 故障域外；`deploy-service` 只拉取 CI 已发布的 commit-pinned 镜像、retag/recreate 或导入 native k3s，并做 live commit 验证，不构建 runtime source。`deploy-backend-core` 是 deprecated 兼容名，标准 backend-core prod CD 入口是 `deploy apply --env prod --service backend-core`。长期规则见 `docs/reference/artifact-registry.md`。 - `commander contract|plan --dry-run|smoke --dry-run|approval request --dry-run|prompt-lint --kind gpt55-pr` 是 host Codex 指挥官直管微服务 skeleton 入口。当前命令返回 `phase=source-contract`、service/API/state/bridge/prompt/trace/#20/#46/ClaudeQQ 审批边界、.state/commander/ 状态模型、dev 无 daemon smoke contract、dry-run 计划和 GPT-5.5 PR prompt 边界辅助 lint，不接 live bridge、不注入 prompt、不发送 ClaudeQQ。`approval request --dry-run` 会生成 200 字以内中文纯文本 ClaudeQQ 审批草案、`notification-path-unavailable` blocker 和授权后唯一可用的 `bun scripts/cli.ts microservice proxy claudeqq /api/push/text --method POST --body-json '' --raw` 命令；不得提示使用本机 ClaudeQQ skill、powershell 或本地 server。`prompt-lint` 支持 `--prompt-file` 与 `--stdin`，输出 `ok`、`missingClauses`、`riskLevel`、`suggestedPatchSnippet` 且不回显完整 prompt；它是 commander 辅助检查，不是业务 PR 门禁，也不改变 `codex submit` 默认行为。`plan`、`smoke` 与 `approval request` 必须带 `--dry-run`；缺少时返回 `error=dry-run-required`。长期规则见 `docs/reference/host-codex-commander.md`。 -- `hwlab cd status|preflight|apply --env dev [--dry-run]` 是 HWLAB DEV CD 指挥侧 wrapper。它只调用 HWLAB repo-owned 受控入口，不内嵌发布 kubectl 逻辑：`status` 汇总固定 CD mirror、Git clean/main/origin-main、`deploy/deploy.json`/artifact catalog/workloads 同源收敛、D601 native k3s guard、CD Lease lock、16666/16667 live revision 和当前 live workload image；`preflight` 进一步检查必需 SecretRef 对象/键存在性并运行 HWLAB `scripts/dev-cd-apply.mjs --dry-run` 受控事务摘要；完整 stdout/stderr 写入 `.state/hwlab-cd//`，stdout 只返回有界摘要。默认 HWLAB CD repo 是 `/home/ubuntu/hwlab_cd`，`/home/ubuntu/hwlab` runner 历史目录不得作为发布真相。wrapper 强制 `KUBECONFIG=/etc/rancher/k3s/k3s.yaml` 并只以这个显式目标作为 gate；显式目标出现 `docker-desktop`、`desktop-control-plane` 或 `127.0.0.1:11700` 信号会结构化拒绝，写操作计划还必须观察到 node `d601`。`apply --dry-run` 只调用 HWLAB `scripts/dev-cd-apply.mjs --dry-run --kubeconfig /etc/rancher/k3s/k3s.yaml`；真实 apply 只暴露 `scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report` 命令形状并标注 host-commander-only，本 runner 不执行 live apply。长期规则见 `docs/reference/hwlab.md`。 +- `hwlab cd status|preflight|apply --env dev [--dry-run]` 是 HWLAB DEV CD 指挥侧 wrapper。默认通过 UniDesk provider `host.ssh` 进入 D601，再调用 HWLAB repo-owned `scripts/dev-cd-apply.mjs`，不内嵌发布 kubectl 逻辑：`status` 汇总固定 CD mirror、Git clean/main/origin-main、`deploy/deploy.json`/artifact catalog/report、D601 native k3s guard 和 CD Lease lock，并用 `scripts/dev-cd-apply.mjs --status --skip-live-verify` 取得 target/promotion 摘要；`preflight` 进一步检查必需 SecretRef 对象/键存在性并运行 HWLAB `scripts/dev-cd-apply.mjs --dry-run --skip-live-verify` 受控事务摘要。完整远端 stdout/stderr 写入 D601 `~/.state/unidesk-hwlab-cd//` 和本地 `.state/hwlab-cd//` task dump，stdout 只返回有界摘要。默认 HWLAB CD repo 是 `/home/ubuntu/hwlab_cd`，`/home/ubuntu/hwlab` runner 历史目录不得作为发布真相。wrapper 强制 `KUBECONFIG=/etc/rancher/k3s/k3s.yaml` 并只以这个显式目标作为 gate；显式目标出现 `docker-desktop`、`desktop-control-plane` 或 `127.0.0.1:11700` 信号会结构化拒绝，写操作计划还必须观察到 node `d601`。真实 apply 只暴露 `scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report` 命令形状并标注 host-commander-only，本 runner 不执行 live apply、rollout、Lease mutation 或 16666/16667 live verification。长期规则见 `docs/reference/hwlab.md`。 - `gh auth status [--repo owner/name]` 探测 GitHub 操作前置条件并输出脱敏 JSON：是否存在 `gh` binary、是否存在 `GH_TOKEN`/`GITHUB_TOKEN` 或可用 `gh auth token` fallback、REST API 是否可达、目标 repo 是否可见、issue 是否可读。degraded reason 必须归类为 `missing-binary`、`missing-token`、`auth-failed`、`github-transient`、`network-proxy-failed`、`permission-denied`、`repo-not-found`、`repo-forbidden`、`issue-not-found`、`pr-not-found`、`scope-insufficient`、`validation-failed`、`invalid-response` 或 `unsupported-command`，不得打印 token；失败对象必须包含 `runnerDisposition=infra-blocked|business-failed`，runner 应优先用该字段分流。`github-transient` 表示 GitHub DNS/API 连接在收到 HTTP 状态前失败，输出应带 `retryable=true` 或等价 commander action；这不是缺 token、认证失败、权限不足或 PR 语义失败。 - `codex prompt-lint [prompt|--prompt-file path|--prompt-stdin]` 是派发/steer 前的本地 dry-run prompt lint。它只读取 prompt 文本，返回 `dryRun=true`、`mutation=false`、`declaredClass`、`effectiveClass`、`requiredClass`、`dispatchDisposition`、缺失或矛盾项和有界 evidence，不访问 live service、不提交任务、不打印完整 prompt。分级固定为 `read-only`、`live-read`、`live-mutating`；未声明时按 `read-only` 处理。`codex submit --dry-run` 与 `codex steer --dry-run` 会嵌入同一 `promptLint` 结果，帮助指挥官在 dispatch/steer 前发现缺失或矛盾的 live mutation 授权。长期规则见 `docs/reference/code-queue-supervision.md` 的 DEV 测试授权分级。 - `gh issue list [--state open|closed|all] [--limit N] [--repo owner/name] [--json number,title,state,url,updatedAt,createdAt,author,labels]` 通过 GitHub REST 列出 issue，默认 `state=open`、`limit=30`，输出稳定 JSON 且不依赖系统 `gh` binary。`--limit` 会映射到 GitHub `per_page` 并限制返回数量，避免一次拉爆上下文；未知 state 或未知 `--json` 字段必须结构化失败并带 `runnerDisposition=business-failed`。GitHub issues API 可能混入 PR，CLI 会从 `.data.issues` 中过滤 pull request。 diff --git a/docs/reference/hwlab.md b/docs/reference/hwlab.md index 1872a179..44056c18 100644 --- a/docs/reference/hwlab.md +++ b/docs/reference/hwlab.md @@ -50,14 +50,14 @@ bun scripts/cli.ts hwlab cd preflight --env dev bun scripts/cli.ts hwlab cd apply --env dev --dry-run ``` -wrapper 的职责是把 host commander 常用的 HWLAB DEV rollout 查看/准备动作收敛到单一入口。它只调用 HWLAB repo-owned 受控脚本，不在 UniDesk 内重写发布流程或拼接 ad hoc `kubectl apply`： +wrapper 的职责是把 host commander 常用的 HWLAB DEV rollout 查看/准备动作收敛到单一入口。默认路径通过 UniDesk main-server frontend/backend-core 的 provider `host.ssh` 能力进入 D601，再在 D601 上执行有界检查脚本；它只调用 HWLAB repo-owned 受控脚本，不在 UniDesk 内重写发布流程或拼接 ad hoc `kubectl apply`： - 默认 HWLAB CD repo 是 D601 固定干净 mirror `/home/ubuntu/hwlab_cd`，也可用 `--hwlab-repo` 显式指定同等干净 clone。wrapper 必须检查 `git status --short --branch`、origin remote、当前 branch `main`、本地 `origin/main`、`FETCH_HEAD` 和 worktree 权限；任何 dirty worktree、错误 remote、非 main、HEAD 未跟上本地 `origin/main` 或权限异常都返回结构化 blocker。`/home/ubuntu/hwlab` 是 runner 历史目录，不得作为发布真相。 - `deploy/deploy.json` 是唯一 desired-state。wrapper 只把 `deploy/artifact-catalog.dev.json`、`deploy/k8s/base/workloads.yaml` 和 `reports/dev-gate/dev-artifacts.json` 当作派生/证据读数；`status`/`preflight` 必须显示 target commit/ref、deploy.json、artifact catalog、workloads 和 live workload image 是否同源/收敛，不引入第二套 desired state。 -- `status` 只读汇总 HWLAB repo path、Git clean/main/origin-main、desired-state 收敛、D601 native k3s guard、`Lease/hwlab-dev/hwlab-dev-cd-lock`、公网 `16666/16667` live revision 和当前 live workload image。 +- `status` 只读汇总 HWLAB repo path、Git clean/main/origin-main、desired-state 收敛、D601 native k3s guard 和 `Lease/hwlab-dev/hwlab-dev-cd-lock`；同时调用 HWLAB `scripts/dev-cd-apply.mjs --status --skip-live-verify` 取得 repo-owned target/promotion/deploy.json/artifact 摘要。16666/16667 live verification 不由本 runner 执行。 - `preflight` 在 `status` 的基础上检查 apply 前 SecretRef：`hwlab-cloud-api-dev-db/database-url`、`hwlab-cloud-api-dev-db-admin/admin-url`、`hwlab-code-agent-provider/openai-api-key`。只验证 Secret 对象和 key 元数据存在性，缺失时返回 blocker、影响范围和修复 runbook；禁止读取或打印 Secret value。 -- `apply --dry-run` 调用 HWLAB `scripts/dev-cd-apply.mjs --dry-run --kubeconfig /etc/rancher/k3s/k3s.yaml`，只生成受控事务准备/阻塞摘要，不做真实 apply、rollout 或 live verification。历史 `scripts/dev-deploy-apply.mjs` 可作为 HWLAB 内部支持脚本出现，但 UniDesk wrapper 不能把它当成平行 CD 入口。 -- 完整下游 stdout/stderr、HTTP body 和 kubectl 读命令输出写入 UniDesk `.state/hwlab-cd//` dump 目录；CLI stdout 只显示有界摘要和 dump path。 +- `apply --dry-run` 调用 HWLAB `scripts/dev-cd-apply.mjs --dry-run --kubeconfig /etc/rancher/k3s/k3s.yaml --skip-live-verify`，只生成受控事务准备/阻塞摘要，不做真实 apply、rollout、Lease mutation 或 live verification。历史 `scripts/dev-deploy-apply.mjs` 可作为 HWLAB 内部支持脚本出现，但 UniDesk wrapper 不能把它当成平行 CD 入口。 +- 完整下游 stdout/stderr 和 kubectl 读命令输出写入 D601 `~/.state/unidesk-hwlab-cd//`，UniDesk 本地仅保存 provider task stdout/stderr dump；CLI stdout 只显示有界摘要和 dump path。 - wrapper 显式注入 `KUBECONFIG=/etc/rancher/k3s/k3s.yaml` 并以这个显式目标作为唯一 gate：目标 context/server/nodes 若出现 `docker-desktop`、`desktop-control-plane` 或 `127.0.0.1:11700` 必须拒绝继续，写操作计划或 `apply --dry-run` 前目标 nodes 必须包含 `d601`。裸 `kubectl` 默认 context 只作为诊断输出；即使默认 kubeconfig 仍残留 Docker Desktop，只要显式 D601 kubeconfig 通过，也不能把默认 context 当成 CD blocker。 真实 DEV apply 只允许 host commander 在明确授权后执行。UniDesk wrapper 可以展示受控命令形状： diff --git a/scripts/hwlab-cd-wrapper-contract-test.ts b/scripts/hwlab-cd-wrapper-contract-test.ts index 1ad816b1..c67e6837 100644 --- a/scripts/hwlab-cd-wrapper-contract-test.ts +++ b/scripts/hwlab-cd-wrapper-contract-test.ts @@ -1,30 +1,43 @@ +import assert from "node:assert/strict"; import { spawnSync } from "node:child_process"; import { mkdirSync, writeFileSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; -import assert from "node:assert/strict"; +import { buildHwlabCdRemoteCommandForTest } from "./src/hwlab-cd"; type JsonRecord = Record; +function dataOf(payload: JsonRecord): JsonRecord { + return payload.data as JsonRecord; +} + function runCli(args: string[], env: NodeJS.ProcessEnv = {}): JsonRecord { const result = spawnSync("bun", ["scripts/cli.ts", ...args], { cwd: process.cwd(), - env: { ...process.env, ...env }, + env: { + ...process.env, + UNIDESK_HWLAB_CD_TRANSPORT: "local", + ...env, + }, encoding: "utf8", - timeout: 20_000, + timeout: 25_000, }); assert.equal(result.stderr, "", `stderr should be empty: ${result.stderr}`); assert.notEqual(result.stdout.trim(), "", "CLI must not produce empty output"); const parsed = JSON.parse(result.stdout) as JsonRecord; - if (result.status !== 0) { - assert.equal(parsed.ok, false, `nonzero CLI should return ok=false: ${result.stdout}`); - } + if (result.status !== 0) assert.equal(parsed.ok, false, `nonzero CLI should return ok=false: ${result.stdout}`); return parsed; } +function writeJson(path: string, value: unknown): void { + writeFileSync(path, `${JSON.stringify(value, null, 2)}\n`); +} + function makeFakeHwlabRepo(): string { - const root = join(tmpdir(), `unidesk-hwlab-cd-wrapper-${process.pid}-${Date.now()}`); + const root = join(tmpdir(), `unidesk-hwlab-cd-wrapper-${process.pid}-${Date.now()}-${Math.random().toString(16).slice(2)}`); mkdirSync(join(root, "scripts"), { recursive: true }); + mkdirSync(join(root, "deploy/k8s/base"), { recursive: true }); + mkdirSync(join(root, "reports/dev-gate"), { recursive: true }); writeFileSync(join(root, "scripts/dev-cd-apply.mjs"), [ "const kubeconfigIndex = process.argv.indexOf('--kubeconfig');", "process.stdout.write(JSON.stringify({", @@ -38,7 +51,7 @@ function makeFakeHwlabRepo(): string { " ref: 'origin/main',", " promotionCommit: 'abc1234567890abcdef',", " shortCommitId: 'abc1234',", - " promotionSource: 'deploy-json',", + " promotionSource: 'deploy/deploy.json',", " publishRequired: false,", " headCommitId: 'abc1234567890abcdef',", " headMatchesTarget: true,", @@ -50,40 +63,14 @@ function makeFakeHwlabRepo(): string { " artifactCatalog: { path: 'deploy/artifact-catalog.dev.json', commitId: 'abc1234', artifactState: 'published', ciPublished: true, registryVerified: true },", " artifactReport: { path: 'reports/dev-gate/dev-artifacts.json', commitId: 'abc1234' },", " lock: { status: 'absent' },", - " liveDelta: { status: 'unknown' },", + " liveDelta: { status: 'not_run', reason: '--skip-live-verify' },", " kubeconfig: kubeconfigIndex >= 0 ? process.argv[kubeconfigIndex + 1] : null", "}, null, 2));", ].join("\n")); - writeFileSync(join(root, "scripts/dev-deploy-apply.mjs"), [ - "const dryRun = process.argv.includes('--dry-run');", - "const kubeconfigIndex = process.argv.indexOf('--kubeconfig');", - "process.stdout.write(JSON.stringify({", - " reportVersion: 'v1',", - " status: dryRun ? 'pass' : 'blocked',", - " commitId: 'abc1234',", - " namespace: 'hwlab-dev',", - " endpoint: 'http://74.48.78.17:16667',", - " blockers: [],", - " devDeployApply: {", - " conclusion: { status: 'ready', blockerCount: 0 },", - " artifactPlan: { expectedArtifactCommit: 'abc1234', deployCommitId: 'abc1234', catalogCommitId: 'abc1234', published: true, registryVerified: true, imageCount: 13, requiredServiceCount: 13, unpublishedServices: [] },", - " applyBoundary: { currentMode: 'dry-run', defaultNoWrite: true, mutationAttempted: false, mutationAllowed: false, kubeconfigSource: 'flag:--kubeconfig', writeScope: 'KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl apply -k deploy/k8s/dev', noWriteScope: 'server-side dry-run only', forbiddenActions: ['prod-deploy'] },", - " applyStep: { status: 'pass', command: 'KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl apply --dry-run=server -k deploy/k8s/dev', mutationAttempted: false },", - " manualCommands: { status: 'ready' }", - " },", - " kubeconfig: kubeconfigIndex >= 0 ? process.argv[kubeconfigIndex + 1] : null", - "}, null, 2));", - ].join("\n")); - writeFileSync(join(root, "scripts/deploy-desired-state-plan.mjs"), [ - "process.stdout.write(JSON.stringify({", - " kind: 'hwlab-deploy-desired-state-plan',", - " mode: 'read-only-plan',", - " status: 'pass',", - " source: { deploy: 'deploy/deploy.json', artifactCatalog: 'deploy/artifact-catalog.dev.json', workloads: 'deploy/k8s/base/workloads.yaml', optionalReport: 'reports/dev-gate/dev-artifacts.json' },", - " promotionBoundary: { authoritativeDesiredState: ['deploy/deploy.json', 'deploy/artifact-catalog.dev.json', 'deploy/k8s/base/workloads.yaml'], nonAuthoritativeEvidence: ['reports/dev-gate/dev-artifacts.json'] },", - " summary: { desiredCommitId: 'abc1234', desiredImageTag: 'abc1234', artifactState: 'published', ciPublished: true, registryVerified: true, services: 13, workloadContainers: 13, diagnostics: 0, blockers: 0, targetConvergence: 'not_requested' }", - "}, null, 2));", - ].join("\n")); + writeJson(join(root, "deploy/deploy.json"), { commitId: "abc1234", environment: "dev", namespace: "hwlab-dev", endpoint: "http://74.48.78.17:16667" }); + writeJson(join(root, "deploy/artifact-catalog.dev.json"), { commitId: "abc1234", artifactState: "published", publish: { ciPublished: true, registryVerified: true }, services: [{ id: "hwlab-cloud-api", digest: "sha256:" + "a".repeat(64) }] }); + writeJson(join(root, "reports/dev-gate/dev-artifacts.json"), { commitId: "abc1234", artifactPublish: { sourceCommitId: "abc1234", serviceCount: 1, ciPublished: true, registryVerified: true } }); + writeFileSync(join(root, "deploy/k8s/base/workloads.yaml"), "apiVersion: v1\nkind: List\nitems: []\n"); spawnSync("git", ["init", "-b", "main"], { cwd: root, encoding: "utf8" }); spawnSync("git", ["config", "user.email", "test@example.invalid"], { cwd: root, encoding: "utf8" }); spawnSync("git", ["config", "user.name", "HWLAB CD Test"], { cwd: root, encoding: "utf8" }); @@ -95,19 +82,19 @@ function makeFakeHwlabRepo(): string { return root; } -function makeFakeBin(mode: "native" | "desktop" | "stale-default" | "wrong-node" | "missing-secret"): string { - const bin = join(tmpdir(), `unidesk-hwlab-cd-bin-${process.pid}-${Date.now()}-${mode}`); +function makeFakeBin(mode: "native" | "desktop" | "stale-default" | "wrong-node" | "missing-secret" | "second-plane"): string { + const bin = join(tmpdir(), `unidesk-hwlab-cd-bin-${process.pid}-${Date.now()}-${mode}-${Math.random().toString(16).slice(2)}`); mkdirSync(bin, { recursive: true }); const explicitContext = mode === "desktop" ? "docker-desktop" : "default"; const explicitServer = mode === "desktop" ? "https://127.0.0.1:11700" : "https://127.0.0.1:6443"; const explicitNodes = mode === "desktop" ? "desktop-control-plane" : mode === "wrong-node" ? "d602" : "d601"; - const defaultContext = mode === "stale-default" ? "docker-desktop" : explicitContext; - const defaultServer = mode === "stale-default" ? "https://127.0.0.1:11700" : explicitServer; + const defaultContext = mode === "stale-default" ? "docker-desktop" : mode === "second-plane" ? "other-k3s" : explicitContext; + const defaultServer = mode === "stale-default" ? "https://127.0.0.1:11700" : mode === "second-plane" ? "https://10.0.0.2:6443" : explicitServer; const defaultNodes = mode === "stale-default" ? "desktop-control-plane" : explicitNodes; + const defaultNamespaceOk = mode === "second-plane"; writeFileSync(join(bin, "kubectl"), [ "#!/usr/bin/env bash", "set -euo pipefail", - "printf 'KUBECONFIG=%s\\n' \"${KUBECONFIG:-}\" >&2", "context=" + JSON.stringify(explicitContext), "server=" + JSON.stringify(explicitServer), "nodes=" + JSON.stringify(explicitNodes), @@ -119,33 +106,56 @@ function makeFakeBin(mode: "native" | "desktop" | "stale-default" | "wrong-node" "if [[ \"$*\" == 'config current-context' ]]; then printf '%s\\n' \"$context\"; exit 0; fi", "if [[ \"$*\" == 'config view --minify -o jsonpath={.clusters[0].cluster.server}' ]]; then printf '%s' \"$server\"; exit 0; fi", "if [[ \"$*\" == 'get nodes -o jsonpath={range .items[*]}{.metadata.name}{\"\\n\"}{end}' ]]; then printf '%s\\n' \"$nodes\"; exit 0; fi", - "if [[ \"$*\" == '-n hwlab-dev get lease hwlab-dev-cd-lock -o json' ]]; then printf 'Error from server (NotFound): leases.coordination.k8s.io \"hwlab-dev-cd-lock\" not found\\n' >&2; exit 1; fi", - "if [[ \"$*\" == '-n hwlab-dev get deploy -o jsonpath={range .items[*]}{.metadata.name}{\"\\t\"}{range .spec.template.spec.containers[*]}{.name}{\"=\"}{.image}{\",\"}{end}{\"\\n\"}{end}' ]]; then printf 'hwlab-cloud-api\\thwlab-cloud-api=127.0.0.1:5000/hwlab/hwlab-cloud-api:abc1234,\\n'; exit 0; fi", + "if [[ \"$*\" == 'get namespace hwlab-dev -o name' ]]; then", + " if [[ \"${KUBECONFIG:-}\" == '' && " + JSON.stringify(defaultNamespaceOk ? "yes" : "no") + " == 'yes' ]]; then printf 'namespace/hwlab-dev\\n'; exit 0; fi", + " if [[ \"${KUBECONFIG:-}\" != '' ]]; then printf 'namespace/hwlab-dev\\n'; exit 0; fi", + " printf 'Error from server (NotFound): namespaces \"hwlab-dev\" not found\\n' >&2; exit 1", + "fi", + "if [[ \"$*\" =~ ^-n[[:space:]]+hwlab-dev[[:space:]]+get[[:space:]]+lease[[:space:]]+hwlab-dev-cd-lock[[:space:]]+-o[[:space:]]+json$ ]]; then printf 'Error from server (NotFound): leases.coordination.k8s.io \"hwlab-dev-cd-lock\" not found\\n' >&2; exit 1; fi", "if [[ \"$*\" == '-n hwlab-dev get secret hwlab-code-agent-provider -o name' && " + JSON.stringify(mode) + " == 'missing-secret' ]]; then printf 'Error from server (NotFound): secrets \"hwlab-code-agent-provider\" not found\\n' >&2; exit 1; fi", "if [[ \"$*\" =~ ^-n\\ hwlab-dev\\ get\\ secret\\ ([^[:space:]]+)\\ -o\\ name$ ]]; then printf 'secret/%s\\n' \"${BASH_REMATCH[1]}\"; exit 0; fi", "if [[ \"$*\" == '-n hwlab-dev describe secret hwlab-cloud-api-dev-db' ]]; then printf 'Name: hwlab-cloud-api-dev-db\\nData\\n====\\ndatabase-url: 48 bytes\\n'; exit 0; fi", "if [[ \"$*\" == '-n hwlab-dev describe secret hwlab-cloud-api-dev-db-admin' ]]; then printf 'Name: hwlab-cloud-api-dev-db-admin\\nData\\n====\\nadmin-url: 48 bytes\\n'; exit 0; fi", "if [[ \"$*\" == '-n hwlab-dev describe secret hwlab-code-agent-provider' ]]; then printf 'Name: hwlab-code-agent-provider\\nData\\n====\\nopenai-api-key: 48 bytes\\n'; exit 0; fi", - "printf '{}\\n'", + "printf 'unexpected kubectl args: %s\\n' \"$*\" >&2; exit 99", ].join("\n")); spawnSync("chmod", ["+x", join(bin, "kubectl")]); return bin; } +function scopes(data: JsonRecord): string[] { + return ((data.blockers ?? []) as JsonRecord[]).map((blocker) => String(blocker.scope ?? "")); +} + +function withLocalTransport(args: string[]): string[] { + return [...args, "--transport", "local"]; +} + const fakeRepo = makeFakeHwlabRepo(); const nativeBin = makeFakeBin("native"); const desktopBin = makeFakeBin("desktop"); const staleDefaultBin = makeFakeBin("stale-default"); const wrongNodeBin = makeFakeBin("wrong-node"); const missingSecretBin = makeFakeBin("missing-secret"); -const liveBody = "data:application/json,%7B%22serviceId%22%3A%22hwlab-cloud-web%22%2C%22environment%22%3A%22dev%22%2C%22status%22%3A%22ok%22%2C%22revision%22%3A%22abc1234%22%7D"; -const apiBody = "data:application/json,%7B%22serviceId%22%3A%22hwlab-cloud-api%22%2C%22environment%22%3A%22dev%22%2C%22status%22%3A%22ok%22%2C%22revision%22%3A%22abc1234%22%7D"; +const secondPlaneBin = makeFakeBin("second-plane"); const help = runCli(["hwlab", "help"]); assert.equal(help.ok, true); assert.equal((help.data as JsonRecord).command, "hwlab cd"); -const runnerHistoryRepo = runCli([ +const remoteCommand = buildHwlabCdRemoteCommandForTest(withLocalTransport(["cd", "apply", "--env", "dev", "--dry-run"])); +assert.equal(remoteCommand.includes("scripts/dev-cd-apply.mjs"), true); +assert.equal(remoteCommand.includes("/etc/rancher/k3s/k3s.yaml"), true); +assert.equal(remoteCommand.includes("kubectl rollout"), false); + +const realApply = runCli(["hwlab", "cd", "apply", "--env", "dev", "--hwlab-repo", fakeRepo], { + PATH: `${nativeBin}:${process.env.PATH ?? ""}`, +}); +assert.equal(realApply.ok, false); +assert.equal(dataOf(realApply).error, "host-commander-only-real-apply"); +assert.equal((dataOf(realApply).safety as JsonRecord).rolloutExecuted, false); + +const runnerHistoryRepo = runCli(withLocalTransport([ "hwlab", "cd", "status", @@ -153,15 +163,13 @@ const runnerHistoryRepo = runCli([ "dev", "--hwlab-repo", "/home/ubuntu/hwlab", -], { +]), { PATH: `${nativeBin}:${process.env.PATH ?? ""}`, }); assert.equal(runnerHistoryRepo.ok, false); -const runnerHistoryCandidates = ((runnerHistoryRepo.data as JsonRecord).repo as JsonRecord).candidates as JsonRecord[]; -assert.equal(runnerHistoryCandidates[0]?.rejected, true); -assert.equal(runnerHistoryCandidates[0]?.rejectionReason, "runner-history-directory-is-not-hwlab-cd-release-truth"); +assert.equal((dataOf(runnerHistoryRepo).repo as JsonRecord).rejected, true); -const applyDryRun = runCli([ +const applyDryRun = runCli(withLocalTransport([ "hwlab", "cd", "apply", @@ -170,55 +178,30 @@ const applyDryRun = runCli([ "--dry-run", "--hwlab-repo", fakeRepo, -], { +]), { PATH: `${nativeBin}:${process.env.PATH ?? ""}`, }); assert.equal(applyDryRun.ok, true); -const dryRunData = applyDryRun.data as JsonRecord; +const dryRunData = dataOf(applyDryRun); assert.equal(dryRunData.dryRun, true); assert.equal(dryRunData.mutation, false); -assert.equal(((dryRunData.d601NativeK3sGuard as JsonRecord).injectedEnv as JsonRecord).KUBECONFIG, "/etc/rancher/k3s/k3s.yaml"); -assert.equal((dryRunData.d601NativeK3sGuard as JsonRecord).requiredNodePresent, true); -assert.equal((dryRunData.controlledDryRun as JsonRecord).commandOk, true); -assert.equal((dryRunData.secretRefPreflight as JsonRecord).status, "pass"); -assert.equal(((dryRunData.controlledDryRun as JsonRecord).controlledEntrypoint), "scripts/dev-cd-apply.mjs"); -assert.equal(((dryRunData.hostCommanderOnlyLiveApply as JsonRecord).commandShape as unknown[]).includes("scripts/dev-cd-apply.mjs"), true); +assert.equal((dryRunData.safety as JsonRecord).kubectlApplyExecuted, false); +assert.equal((dryRunData.safety as JsonRecord).rolloutExecuted, false); +assert.equal((dryRunData.safety as JsonRecord).liveVerifyExecuted, false); +assert.equal((dryRunData.safety as JsonRecord).cdLockMutated, false); +assert.equal((dryRunData.safety as JsonRecord).secretValuesPrinted, false); +assert.equal((dryRunData.remote as JsonRecord).providerId, "D601"); +assert.equal(((dryRunData.remote as JsonRecord).commandCalls as unknown[]).includes("scripts/dev-cd-apply.mjs"), true); +assert.equal((dryRunData.kubeconfig as JsonRecord).path, "/etc/rancher/k3s/k3s.yaml"); +assert.equal((dryRunData.nodeGuard as JsonRecord).requiredNodePresent, true); +assert.equal((dryRunData.worktreeGuard as JsonRecord).clean, true); +assert.equal((dryRunData.secretPreflight as JsonRecord).status, "pass"); +assert.equal((dryRunData.controlledDevCd as JsonRecord).controlledEntrypoint, "scripts/dev-cd-apply.mjs"); +assert.equal(((dryRunData.target as JsonRecord).promotionCommit), "abc1234567890abcdef"); +assert.equal(((dryRunData.promotion as JsonRecord).source), "deploy/deploy.json"); assert.equal(JSON.stringify(dryRunData).includes("sk-secret"), false); -const preflight = runCli([ - "hwlab", - "cd", - "preflight", - "--env", - "dev", - "--hwlab-repo", - fakeRepo, -], { - PATH: `${nativeBin}:${process.env.PATH ?? ""}`, - UNIDESK_HWLAB_CD_TEST_FRONTEND_LIVE_URL: liveBody, - UNIDESK_HWLAB_CD_TEST_API_LIVE_URL: apiBody, -}); -assert.equal(preflight.ok, true); -const preflightData = preflight.data as JsonRecord; -assert.equal(preflightData.mutation, false); -assert.equal((preflightData.secretRefPreflight as JsonRecord).status, "pass"); -assert.equal((preflightData.liveWorkloads as JsonRecord).status, "observed"); - -const realApply = runCli([ - "hwlab", - "cd", - "apply", - "--env", - "dev", - "--hwlab-repo", - fakeRepo, -], { - PATH: `${nativeBin}:${process.env.PATH ?? ""}`, -}); -assert.equal(realApply.ok, false); -assert.equal((realApply.data as JsonRecord).error, "host-commander-only-real-apply"); - -const status = runCli([ +const status = runCli(withLocalTransport([ "hwlab", "cd", "status", @@ -226,18 +209,16 @@ const status = runCli([ "dev", "--hwlab-repo", fakeRepo, -], { +]), { PATH: `${nativeBin}:${process.env.PATH ?? ""}`, - UNIDESK_HWLAB_CD_TEST_FRONTEND_LIVE_URL: liveBody, - UNIDESK_HWLAB_CD_TEST_API_LIVE_URL: apiBody, }); assert.equal(status.ok, true); -const statusData = status.data as JsonRecord; -assert.equal(((statusData.d601NativeK3sGuard as JsonRecord).injectedEnv as JsonRecord).KUBECONFIG, "/etc/rancher/k3s/k3s.yaml"); -assert.equal((statusData.liveRevisions as JsonRecord).status, "observed"); -assert.ok(typeof statusData.dumpDir === "string" && String(statusData.dumpDir).includes(".state/hwlab-cd")); +const statusData = dataOf(status); +assert.equal((statusData.secretPreflight as JsonRecord).status, "skipped"); +assert.equal((statusData.lockState as JsonRecord).status, "absent"); +assert.ok(typeof statusData.reportDumpPath === "string"); -const staleDefaultOk = runCli([ +const staleDefaultOk = runCli(withLocalTransport([ "hwlab", "cd", "apply", @@ -246,18 +227,17 @@ const staleDefaultOk = runCli([ "--dry-run", "--hwlab-repo", fakeRepo, -], { +]), { PATH: `${staleDefaultBin}:${process.env.PATH ?? ""}`, KUBECONFIG: "", }); assert.equal(staleDefaultOk.ok, true); -const staleDefaultGuard = (staleDefaultOk.data as JsonRecord).d601NativeK3sGuard as JsonRecord; +const staleDefaultGuard = dataOf(staleDefaultOk).nodeGuard as JsonRecord; assert.equal(staleDefaultGuard.status, "pass"); assert.equal(staleDefaultGuard.refusal, false); assert.equal((staleDefaultGuard.defaultKubectlDiagnostic as JsonRecord).status, "stale-forbidden-default"); -assert.deepEqual((staleDefaultGuard.defaultKubectlDiagnostic as JsonRecord).refusalSignals, ["docker-desktop", "desktop-control-plane", "127.0.0.1:11700"]); -const desktopRefusal = runCli([ +const desktopRefusal = runCli(withLocalTransport([ "hwlab", "cd", "apply", @@ -266,14 +246,15 @@ const desktopRefusal = runCli([ "--dry-run", "--hwlab-repo", fakeRepo, -], { +]), { PATH: `${desktopBin}:${process.env.PATH ?? ""}`, }); assert.equal(desktopRefusal.ok, false); -assert.equal((desktopRefusal.data as JsonRecord).error, "native-k3s-guard-refused"); -assert.deepEqual((desktopRefusal.data as JsonRecord).d601NativeK3sGuard && ((desktopRefusal.data as JsonRecord).d601NativeK3sGuard as JsonRecord).refusalSignals, ["docker-desktop", "desktop-control-plane", "127.0.0.1:11700"]); +const desktopData = dataOf(desktopRefusal); +assert.equal(desktopData.status, "refused"); +assert.deepEqual((desktopData.nodeGuard as JsonRecord).refusalSignals, ["docker-desktop", "desktop-control-plane", "127.0.0.1:11700"]); -const wrongNodeBlocked = runCli([ +const wrongNodeBlocked = runCli(withLocalTransport([ "hwlab", "cd", "apply", @@ -282,16 +263,15 @@ const wrongNodeBlocked = runCli([ "--dry-run", "--hwlab-repo", fakeRepo, -], { +]), { PATH: `${wrongNodeBin}:${process.env.PATH ?? ""}`, }); assert.equal(wrongNodeBlocked.ok, false); -const wrongNodeGuard = (wrongNodeBlocked.data as JsonRecord).d601NativeK3sGuard as JsonRecord; -assert.equal(wrongNodeGuard.status, "blocked"); -assert.equal(wrongNodeGuard.requiredNodePresent, false); -assert.equal(((wrongNodeBlocked.data as JsonRecord).blockers as JsonRecord[]).some((blocker) => blocker.scope === "d601-native-k3s-guard"), true); +const wrongNodeData = dataOf(wrongNodeBlocked); +assert.equal((wrongNodeData.nodeGuard as JsonRecord).requiredNodePresent, false); +assert.equal(scopes(wrongNodeData).includes("d601-native-k3s-guard"), true); -const missingSecretBlocked = runCli([ +const missingSecretBlocked = runCli(withLocalTransport([ "hwlab", "cd", "apply", @@ -300,14 +280,46 @@ const missingSecretBlocked = runCli([ "--dry-run", "--hwlab-repo", fakeRepo, -], { +]), { PATH: `${missingSecretBin}:${process.env.PATH ?? ""}`, }); assert.equal(missingSecretBlocked.ok, false); -const missingSecretData = missingSecretBlocked.data as JsonRecord; -assert.equal((missingSecretData.secretRefPreflight as JsonRecord).status, "blocked"); -assert.equal((missingSecretData.controlledDryRun as JsonRecord).status, "skipped"); -assert.equal((missingSecretData.blockers as JsonRecord[]).some((blocker) => blocker.scope === "secretref:hwlab-code-agent-provider/openai-api-key"), true); +const missingSecretData = dataOf(missingSecretBlocked); +assert.equal((missingSecretData.secretPreflight as JsonRecord).status, "blocked"); +assert.equal((missingSecretData.controlledDevCd as JsonRecord).status, "skipped"); +assert.equal(scopes(missingSecretData).includes("secretref:hwlab-code-agent-provider/openai-api-key"), true); assert.equal(JSON.stringify(missingSecretData).includes("sk-secret"), false); +const secondPlaneBlocked = runCli(withLocalTransport([ + "hwlab", + "cd", + "apply", + "--env", + "dev", + "--dry-run", + "--hwlab-repo", + fakeRepo, +]), { + PATH: `${secondPlaneBin}:${process.env.PATH ?? ""}`, +}); +assert.equal(secondPlaneBlocked.ok, false); +assert.equal(((dataOf(secondPlaneBlocked).nodeGuard as JsonRecord).defaultKubectlDiagnostic as JsonRecord).secondControlPlaneRisk, true); +assert.equal(scopes(dataOf(secondPlaneBlocked)).includes("second-hwlab-dev-control-plane"), true); + +writeFileSync(join(fakeRepo, "dirty.txt"), "dirty\n"); +const dirtyBlocked = runCli(withLocalTransport([ + "hwlab", + "cd", + "apply", + "--env", + "dev", + "--dry-run", + "--hwlab-repo", + fakeRepo, +]), { + PATH: `${nativeBin}:${process.env.PATH ?? ""}`, +}); +assert.equal(dirtyBlocked.ok, false); +assert.equal(scopes(dataOf(dirtyBlocked)).includes("hwlab-git-clean"), true); + console.log(JSON.stringify({ ok: true, checked: "hwlab-cd-wrapper-contract" })); diff --git a/scripts/src/hwlab-cd-remote-runner.cjs b/scripts/src/hwlab-cd-remote-runner.cjs new file mode 100644 index 00000000..7ae7204a --- /dev/null +++ b/scripts/src/hwlab-cd-remote-runner.cjs @@ -0,0 +1,676 @@ +const { spawn } = require("node:child_process"); +const crypto = require("node:crypto"); +const fs = require("node:fs"); +const fsp = require("node:fs/promises"); +const os = require("node:os"); +const path = require("node:path"); + +const namespace = "hwlab-dev"; +const lockName = "hwlab-dev-cd-lock"; +const nativeKubeconfig = "/etc/rancher/k3s/k3s.yaml"; +const defaultRepo = "/home/ubuntu/hwlab_cd"; +const rejectedRepo = "/home/ubuntu/hwlab"; +const requiredNodeName = "d601"; +const tailChars = 1400; +const parseCaptureLimitBytes = 1024 * 1024; +const requiredSecretRefs = [ + { secretName: "hwlab-cloud-api-dev-db", secretKey: "database-url", consumers: ["hwlab-cloud-api"] }, + { secretName: "hwlab-cloud-api-dev-db-admin", secretKey: "admin-url", consumers: ["runtime provisioning", "runtime migration"] }, + { secretName: "hwlab-code-agent-provider", secretKey: "openai-api-key", consumers: ["hwlab-cloud-api", "code agent provider"] }, +]; + +function decodeOptions() { + const raw = process.env.UNIDESK_HWLAB_CD_OPTIONS_B64 || ""; + if (!raw) throw new Error("UNIDESK_HWLAB_CD_OPTIONS_B64 is required"); + return JSON.parse(Buffer.from(raw, "base64").toString("utf8")); +} + +function redact(value) { + return String(value || "") + .replace(/\b(Bearer\s+)[A-Za-z0-9._~+/=-]+/giu, "$1") + .replace(/\b(token|password|api[_-]?key|secret|client-key-data|client-certificate-data|certificate-authority-data)\b(\s*[:=]\s*)(["']?)([^"'\s,}]+)/giu, "$1$2$3") + .replace(/\b(postgres(?:ql)?:\/\/[^:\s/@]+:)([^@\s]+)(@)/giu, "$1$3"); +} + +function boundedTail(value) { + const text = redact(value); + return text.slice(Math.max(0, text.length - tailChars)); +} + +function readFileTail(filePath) { + try { + return boundedTail(fs.readFileSync(filePath, "utf8")); + } catch { + return ""; + } +} + +function commandView(result) { + return { + id: result.id, + command: result.command, + cwd: result.cwd, + ok: result.ok, + exitCode: result.exitCode, + signal: result.signal, + timedOut: result.timedOut, + durationMs: result.durationMs, + dump: { + stdout: result.dump.stdout, + stderr: result.dump.stderr, + stdoutBytes: result.dump.stdoutBytes, + stderrBytes: result.dump.stderrBytes, + }, + }; +} + +async function runCaptured(command, cwd, dumpDir, id, options = {}) { + await fsp.mkdir(dumpDir, { recursive: true, mode: 0o700 }); + const stdoutPath = path.join(dumpDir, `${id}.stdout.txt`); + const stderrPath = path.join(dumpDir, `${id}.stderr.txt`); + fs.writeFileSync(stdoutPath, "", { mode: 0o600 }); + fs.writeFileSync(stderrPath, "", { mode: 0o600 }); + + const startedAt = Date.now(); + let timedOut = false; + let spawnErrorMessage = ""; + let stdoutBytes = 0; + let stderrBytes = 0; + const stdoutChunks = []; + const stderrChunks = []; + const stdout = fs.createWriteStream(stdoutPath, { flags: "a" }); + const stderr = fs.createWriteStream(stderrPath, { flags: "a" }); + const child = spawn(command[0], command.slice(1), { + cwd, + env: options.env || process.env, + stdio: ["ignore", "pipe", "pipe"], + }); + const timeout = options.timeoutMs + ? setTimeout(() => { + timedOut = true; + child.kill("SIGTERM"); + setTimeout(() => child.kill("SIGKILL"), 1000).unref(); + }, options.timeoutMs) + : null; + + child.stdout.on("data", (chunk) => { + stdoutBytes += chunk.byteLength; + stdout.write(chunk); + if (Buffer.concat(stdoutChunks).byteLength < parseCaptureLimitBytes) stdoutChunks.push(Buffer.from(chunk)); + }); + child.stderr.on("data", (chunk) => { + stderrBytes += chunk.byteLength; + stderr.write(chunk); + if (Buffer.concat(stderrChunks).byteLength < parseCaptureLimitBytes) stderrChunks.push(Buffer.from(chunk)); + }); + + const closed = await new Promise((resolve) => { + child.on("error", (error) => { + spawnErrorMessage = error.message; + }); + child.on("close", (exitCode, signal) => resolve({ exitCode, signal })); + }); + if (timeout !== null) clearTimeout(timeout); + await new Promise((resolve) => stdout.end(resolve)); + await new Promise((resolve) => stderr.end(resolve)); + + const stdoutText = stdoutBytes <= parseCaptureLimitBytes ? Buffer.concat(stdoutChunks).toString("utf8") : ""; + const stderrText = stderrBytes <= parseCaptureLimitBytes ? Buffer.concat(stderrChunks).toString("utf8") : spawnErrorMessage; + const exitCode = spawnErrorMessage && closed.exitCode === null ? 127 : closed.exitCode; + return { + id, + command, + cwd, + ok: exitCode === 0 && !timedOut, + exitCode, + signal: closed.signal, + timedOut, + durationMs: Date.now() - startedAt, + stdoutText, + stderrText, + dump: { + stdout: stdoutPath, + stderr: stderrPath, + stdoutBytes, + stderrBytes, + stdoutTail: boundedTail(stdoutText || readFileTail(stdoutPath)), + stderrTail: boundedTail(stderrText || readFileTail(stderrPath)), + }, + }; +} + +function parseJson(text) { + if (!String(text || "").trim()) return null; + try { + return JSON.parse(text); + } catch { + return null; + } +} + +function asRecord(value) { + return value && typeof value === "object" && !Array.isArray(value) ? value : null; +} + +function stringValue(value) { + return typeof value === "string" && value.length > 0 ? value : null; +} + +function sha256(text) { + return `sha256:${crypto.createHash("sha256").update(text).digest("hex")}`; +} + +function accessCheck(targetPath, mode) { + try { + fs.accessSync(targetPath, mode); + return { ok: true, path: targetPath }; + } catch (error) { + return { ok: false, path: targetPath, error: error instanceof Error ? error.message : String(error) }; + } +} + +function resolveRepo(provided) { + const rawPath = provided || process.env.UNIDESK_HWLAB_REPO || defaultRepo; + const absolutePath = path.resolve(rawPath); + const rejected = absolutePath === rejectedRepo; + const devCdApply = path.join(absolutePath, "scripts/dev-cd-apply.mjs"); + const deployJson = path.join(absolutePath, "deploy/deploy.json"); + return { + status: !rejected && fs.existsSync(absolutePath) && fs.existsSync(devCdApply) && fs.existsSync(deployJson) ? "selected" : "blocked", + source: provided ? "option" : process.env.UNIDESK_HWLAB_REPO ? "env:UNIDESK_HWLAB_REPO" : "default:d601-clean-mirror", + path: absolutePath, + defaultPath: defaultRepo, + rejected, + rejectionReason: rejected ? "runner-history-directory-is-not-hwlab-cd-release-truth" : null, + exists: fs.existsSync(absolutePath), + hasDevCdApply: fs.existsSync(devCdApply), + hasDeployJson: fs.existsSync(deployJson), + }; +} + +async function gitSummary(repoPath, dumpDir, timeoutMs) { + const [branch, head, originMain, remote, gitDir, gitCommonDir, statusShort, statusPorcelain] = await Promise.all([ + runCaptured(["git", "rev-parse", "--abbrev-ref", "HEAD"], repoPath, dumpDir, "git-branch", { timeoutMs }), + runCaptured(["git", "rev-parse", "HEAD"], repoPath, dumpDir, "git-head", { timeoutMs }), + runCaptured(["git", "rev-parse", "--verify", "origin/main^{commit}"], repoPath, dumpDir, "git-origin-main", { timeoutMs }), + runCaptured(["git", "remote", "get-url", "origin"], repoPath, dumpDir, "git-remote-origin", { timeoutMs }), + runCaptured(["git", "rev-parse", "--path-format=absolute", "--git-dir"], repoPath, dumpDir, "git-dir", { timeoutMs }), + runCaptured(["git", "rev-parse", "--path-format=absolute", "--git-common-dir"], repoPath, dumpDir, "git-common-dir", { timeoutMs }), + runCaptured(["git", "status", "--short", "--branch"], repoPath, dumpDir, "git-status-short", { timeoutMs }), + runCaptured(["git", "status", "--porcelain=v1"], repoPath, dumpDir, "git-status-porcelain", { timeoutMs }), + ]); + const branchName = branch.stdoutText.trim(); + const headCommit = head.stdoutText.trim(); + const originMainCommit = originMain.stdoutText.trim(); + const remoteUrl = remote.stdoutText.trim(); + const gitDirPath = gitDir.stdoutText.trim(); + const gitCommonDirPath = gitCommonDir.stdoutText.trim() || gitDirPath; + const fetchHeadPath = gitCommonDirPath ? path.join(gitCommonDirPath, "FETCH_HEAD") : path.join(repoPath, ".git", "FETCH_HEAD"); + const worktreesPath = gitCommonDirPath ? path.join(gitCommonDirPath, "worktrees") : path.join(repoPath, ".git", "worktrees"); + const porcelain = statusPorcelain.stdoutText.trim(); + const dirtyLines = porcelain ? porcelain.split("\n").filter(Boolean) : []; + const worktreeAccess = accessCheck(repoPath, fs.constants.R_OK | fs.constants.W_OK | fs.constants.X_OK); + const fetchHeadAccess = accessCheck(fetchHeadPath, fs.constants.R_OK | fs.constants.W_OK); + const worktreesAccess = fs.existsSync(worktreesPath) + ? accessCheck(worktreesPath, fs.constants.R_OK | fs.constants.W_OK | fs.constants.X_OK) + : { ok: true, path: worktreesPath, exists: false }; + const remoteMatches = /github\.com[:/]pikasTech\/HWLAB(?:\.git)?$/u.test(remoteUrl); + const blockers = []; + if (!branch.ok || !head.ok || !remote.ok || !gitDir.ok || !gitCommonDir.ok || !statusShort.ok || !statusPorcelain.ok) blockers.push({ scope: "hwlab-git-command", summary: "One or more HWLAB git guard commands failed." }); + if (dirtyLines.length > 0) blockers.push({ scope: "hwlab-git-clean", summary: "HWLAB CD worktree has local modifications; deploy/deploy.json cannot be treated as release truth." }); + if (branchName !== "main") blockers.push({ scope: "hwlab-git-main", summary: `HWLAB CD worktree branch is ${branchName || ""}, expected main.` }); + if (!remoteMatches) blockers.push({ scope: "hwlab-git-remote", summary: "HWLAB CD worktree origin is not pikasTech/HWLAB." }); + if (originMain.ok && headCommit && originMainCommit && headCommit !== originMainCommit) blockers.push({ scope: "hwlab-git-origin-main", summary: "HWLAB CD worktree HEAD does not match local origin/main." }); + if (!worktreeAccess.ok) blockers.push({ scope: "hwlab-worktree-permission", summary: "HWLAB CD worktree is not readable/writable/executable by the wrapper." }); + if (!fetchHeadAccess.ok) blockers.push({ scope: "hwlab-fetch-head-permission", summary: "HWLAB CD FETCH_HEAD is missing or not writable by the wrapper." }); + if (!worktreesAccess.ok) blockers.push({ scope: "hwlab-worktrees-permission", summary: "HWLAB .git/worktrees permission guard failed." }); + return { + status: blockers.length === 0 ? "pass" : "blocked", + clean: dirtyLines.length === 0, + branch: branchName || null, + onMain: branchName === "main", + remote: remoteUrl || null, + remoteMatches, + expectedRemote: "git@github.com:pikasTech/HWLAB.git or https://github.com/pikasTech/HWLAB.git", + headCommit: headCommit || null, + originMainCommit: originMain.ok ? originMainCommit || null : null, + headMatchesOriginMain: originMain.ok && headCommit.length > 0 && headCommit === originMainCommit, + worktreeAccess, + fetchHead: { path: fetchHeadPath, exists: fs.existsSync(fetchHeadPath), writable: fetchHeadAccess.ok, error: fetchHeadAccess.error || null }, + worktrees: worktreesAccess, + statusShort: statusShort.stdoutText.trim().split("\n").filter(Boolean).slice(0, 30), + dirtyCount: dirtyLines.length, + dirtyPreview: dirtyLines.slice(0, 20), + blockers, + commands: [branch, head, originMain, remote, gitDir, gitCommonDir, statusShort, statusPorcelain].map(commandView), + }; +} + +function readJsonSummary(repoPath, relativePath, fallback = {}) { + const absolutePath = path.join(repoPath, relativePath); + try { + const raw = fs.readFileSync(absolutePath, "utf8"); + const parsed = JSON.parse(raw); + return { + path: relativePath, + exists: true, + hash: sha256(raw), + commitId: parsed.commitId || parsed.sourceCommitId || null, + environment: parsed.environment || null, + namespace: parsed.namespace || namespace, + endpoint: parsed.endpoint || null, + artifactState: parsed.artifactState || parsed.publish?.artifactState || null, + ciPublished: parsed.publish?.ciPublished ?? parsed.artifactPublish?.ciPublished ?? null, + registryVerified: parsed.publish?.registryVerified ?? parsed.artifactPublish?.registryVerified ?? null, + serviceCount: Array.isArray(parsed.services) ? parsed.services.length : parsed.artifactPublish?.serviceCount ?? null, + ...fallback, + }; + } catch (error) { + return { + path: relativePath, + exists: false, + hash: null, + error: error instanceof Error ? error.message : String(error), + ...fallback, + }; + } +} + +function forbiddenKubeSignals(text) { + const signals = []; + if (/docker-desktop/iu.test(text)) signals.push("docker-desktop"); + if (/desktop-control-plane/iu.test(text)) signals.push("desktop-control-plane"); + if (/127\.0\.0\.1:11700/u.test(text)) signals.push("127.0.0.1:11700"); + return [...new Set(signals)]; +} + +async function observeKube(kubeconfig, dumpDir, timeoutMs, withExplicitEnv) { + const env = withExplicitEnv ? { ...process.env, KUBECONFIG: kubeconfig } : { ...process.env }; + if (!withExplicitEnv) { + delete env.KUBECONFIG; + delete env.HWLAB_DEV_KUBECONFIG; + } + const [context, server, nodes, namespaceCheck] = await Promise.all([ + runCaptured(["kubectl", "config", "current-context"], process.cwd(), dumpDir, withExplicitEnv ? "k3s-current-context" : "default-current-context", { env, timeoutMs }), + runCaptured(["kubectl", "config", "view", "--minify", "-o", "jsonpath={.clusters[0].cluster.server}"], process.cwd(), dumpDir, withExplicitEnv ? "k3s-server" : "default-server", { env, timeoutMs }), + runCaptured(["kubectl", "get", "nodes", "-o", "jsonpath={range .items[*]}{.metadata.name}{\"\\n\"}{end}"], process.cwd(), dumpDir, withExplicitEnv ? "k3s-nodes" : "default-nodes", { env, timeoutMs }), + runCaptured(["kubectl", "get", "namespace", namespace, "-o", "name"], process.cwd(), dumpDir, withExplicitEnv ? "k3s-namespace" : "default-namespace", { env, timeoutMs }), + ]); + const combinedText = [context.stdoutText, context.stderrText, server.stdoutText, server.stderrText, nodes.stdoutText, nodes.stderrText, namespaceCheck.stdoutText, namespaceCheck.stderrText].join("\n"); + return { + checked: true, + explicit: withExplicitEnv, + currentContext: context.stdoutText.trim() || null, + apiServer: server.stdoutText.trim() || null, + nodeNames: nodes.stdoutText.split(/\r?\n/u).map((line) => line.trim()).filter(Boolean), + namespaceObserved: namespaceCheck.ok, + commandsOk: context.ok && server.ok && nodes.ok, + refusalSignals: forbiddenKubeSignals(combinedText), + commands: [context, server, nodes, namespaceCheck].map(commandView), + }; +} + +async function nodeGuard(kubeconfig, dumpDir, timeoutMs) { + const [explicit, bare] = await Promise.all([ + observeKube(kubeconfig, dumpDir, timeoutMs, true), + observeKube(kubeconfig, dumpDir, timeoutMs, false), + ]); + const wrongKubeconfig = kubeconfig !== nativeKubeconfig; + const requiredNodePresent = explicit.nodeNames.includes(requiredNodeName); + const explicitRefusal = explicit.refusalSignals.length > 0; + const secondControlPlaneRisk = Boolean( + bare.namespaceObserved && + explicit.apiServer && + bare.apiServer && + bare.apiServer !== explicit.apiServer + ); + const status = explicitRefusal + ? "refused" + : wrongKubeconfig || !explicit.commandsOk || !requiredNodePresent + ? "blocked" + : "pass"; + return { + status, + refusal: explicitRefusal, + refusalSignals: explicit.refusalSignals, + kubeconfig, + expectedKubeconfig: nativeKubeconfig, + currentContext: explicit.currentContext, + apiServer: explicit.apiServer, + nodeNames: explicit.nodeNames, + nodeCount: explicit.nodeNames.length, + requiredNodeName, + requiredNodePresent, + commandsOk: explicit.commandsOk, + secondControlPlaneRisk, + defaultKubectlDiagnostic: { + checked: true, + currentContext: bare.currentContext, + apiServer: bare.apiServer, + nodeNames: bare.nodeNames, + namespaceObserved: bare.namespaceObserved, + refusalSignals: bare.refusalSignals, + status: bare.refusalSignals.length > 0 ? "stale-forbidden-default" : bare.commandsOk ? "clean" : "unavailable", + secondControlPlaneRisk, + summary: secondControlPlaneRisk + ? "Bare kubectl can observe hwlab-dev through a different control plane; write paths must stop." + : bare.refusalSignals.length > 0 + ? "Bare kubectl shows a forbidden local control-plane signal; explicit D601 KUBECONFIG remains the target." + : "Bare kubectl did not prove a second HWLAB DEV control plane.", + }, + summary: explicitRefusal + ? "Refusing because explicit D601 kubeconfig resolved to a forbidden Docker Desktop control-plane signal." + : wrongKubeconfig + ? `Expected KUBECONFIG=${nativeKubeconfig}.` + : !explicit.commandsOk + ? "Explicit D601 kubeconfig could not read context, server, and nodes." + : !requiredNodePresent + ? "Explicit D601 kubeconfig did not report node d601." + : "D601 native k3s guard passed with explicit KUBECONFIG.", + commands: [...explicit.commands, ...bare.commands], + }; +} + +async function lockStatus(kubeconfig, guard, dumpDir, timeoutMs) { + if (guard.status !== "pass") { + return { status: "skipped", reason: "d601-native-k3s-guard-not-pass", namespace, lockName, present: null }; + } + const env = { ...process.env, KUBECONFIG: kubeconfig }; + const result = await runCaptured(["kubectl", "-n", namespace, "get", "lease", lockName, "-o", "json"], process.cwd(), dumpDir, "cd-lock", { env, timeoutMs }); + const text = `${result.stdoutText}\n${result.stderrText}`; + if (!result.ok && /not\s*found|notfound/iu.test(text)) { + return { status: "absent", namespace, lockName, present: false, held: false, stale: false, command: commandView(result) }; + } + const parsed = asRecord(parseJson(result.stdoutText)); + const metadata = asRecord(parsed?.metadata) || {}; + const spec = asRecord(parsed?.spec) || {}; + const annotations = asRecord(metadata.annotations) || {}; + const annotation = (name) => stringValue(annotations[`hwlab.pikastech.local/${name}`]); + const lock = parsed === null ? null : { + lockBackend: "Lease", + lockName: stringValue(metadata.name) || lockName, + namespace: stringValue(metadata.namespace) || namespace, + holderIdentity: stringValue(spec.holderIdentity), + resourceVersion: stringValue(metadata.resourceVersion), + ownerTaskId: annotation("ownerTaskId"), + transactionId: annotation("transactionId"), + phase: annotation("phase") || "unknown", + promotionCommit: annotation("promotionCommit"), + deployJsonHash: annotation("deployJsonHash"), + targetRef: annotation("targetRef"), + targetNamespace: annotation("targetNamespace"), + updatedAt: annotation("updatedAt") || stringValue(spec.renewTime), + ttlSeconds: Number(annotation("ttlSeconds") || spec.leaseDurationSeconds || 3600), + releaseStatus: annotation("releaseStatus"), + releasedAt: annotation("releasedAt"), + }; + const updatedAt = stringValue(lock?.updatedAt) || ""; + const ttlSeconds = Number(lock?.ttlSeconds || 3600); + const expiresAtMs = Number.isFinite(Date.parse(updatedAt)) ? Date.parse(updatedAt) + ttlSeconds * 1000 : 0; + const retryAfterSeconds = Math.max(0, Math.ceil((expiresAtMs - Date.now()) / 1000)); + return { + status: result.ok && lock !== null ? "observed" : "blocked", + present: result.ok && lock !== null, + ...(lock || {}), + held: lock !== null && lock.phase !== "released" && retryAfterSeconds > 0, + stale: lock !== null && lock.phase !== "released" && retryAfterSeconds <= 0, + retryAfterSeconds, + expiresAt: expiresAtMs > 0 ? new Date(expiresAtMs).toISOString() : null, + command: commandView(result), + }; +} + +function parseSecretDescribeKeys(stdout) { + const keys = new Set(); + let inData = false; + for (const rawLine of String(stdout || "").split(/\r?\n/u)) { + const line = rawLine.trim(); + if (/^Data\b/iu.test(line)) { + inData = true; + continue; + } + if (!inData || line === "" || /^=+$/u.test(line)) continue; + const match = line.match(/^([A-Za-z0-9_.-]+):\s+\d+\s+bytes\b/iu); + if (match) keys.add(match[1]); + } + return [...keys].sort(); +} + +async function secretPreflight(kubeconfig, guard, dumpDir, timeoutMs) { + if (guard.status !== "pass") { + return { + status: "skipped", + reason: "d601-native-k3s-guard-not-pass", + namespace, + mutation: false, + safety: { secretValuesRead: false, secretValuesPrinted: false, secretKeyNamesOnly: true }, + secretRefs: requiredSecretRefs.map((ref) => ({ ...ref, status: "not_checked" })), + blockers: [], + }; + } + const env = { ...process.env, KUBECONFIG: kubeconfig }; + const observations = []; + for (const ref of requiredSecretRefs) { + const idBase = `secretref-${ref.secretName}-${ref.secretKey}`.replace(/[^A-Za-z0-9_.-]/gu, "-"); + const exists = await runCaptured(["kubectl", "-n", namespace, "get", "secret", ref.secretName, "-o", "name"], process.cwd(), dumpDir, `${idBase}-exists`, { env, timeoutMs }); + if (!exists.ok) { + observations.push({ ...ref, status: "missing-secret", exists: false, keyPresent: false, command: commandView(exists) }); + continue; + } + const describe = await runCaptured(["kubectl", "-n", namespace, "describe", "secret", ref.secretName], process.cwd(), dumpDir, `${idBase}-describe`, { env, timeoutMs }); + const keysObserved = parseSecretDescribeKeys(describe.stdoutText); + const keyPresent = keysObserved.includes(ref.secretKey); + observations.push({ + ...ref, + status: keyPresent ? "present" : describe.ok ? "missing-key" : "key-observation-blocked", + exists: true, + keyPresent, + keysObserved, + command: commandView(describe), + }); + } + const blockers = observations + .filter((observation) => observation.status !== "present") + .map((observation) => ({ + scope: `secretref:${observation.secretName}/${observation.secretKey}`, + summary: observation.status === "missing-secret" + ? `Required Secret ${observation.secretName} is missing in ${namespace}.` + : `Required SecretRef ${observation.secretName}/${observation.secretKey} is not present as key metadata in ${namespace}.`, + impact: `${observation.consumers.join(", ")} would fail after a DEV CD apply or runtime preflight Job.`, + runbook: `Create or repair Secret ${observation.secretName} with key ${observation.secretKey} in namespace ${namespace}; verify key presence only and do not print the Secret value.`, + })); + return { + status: blockers.length === 0 ? "pass" : "blocked", + namespace, + mutation: false, + safety: { secretValuesRead: false, secretValuesPrinted: false, secretKeyNamesOnly: true }, + requiredSecretRefs: requiredSecretRefs.map((ref) => `${ref.secretName}/${ref.secretKey}`), + secretRefs: observations, + blockers, + }; +} + +function summarizeControlled(parsed, command) { + const record = asRecord(parsed); + const target = asRecord(record?.target); + const desiredStateCheck = asRecord(target?.desiredStateCheck); + const artifactBoundary = asRecord(target?.artifactBoundary); + return { + status: stringValue(record?.status) || (command.ok ? "unknown" : "blocked"), + commandOk: command.ok, + mode: stringValue(record?.mode), + mutationAttempted: record?.mutationAttempted ?? false, + prodTouched: record?.prodTouched ?? false, + controlledEntrypoint: "scripts/dev-cd-apply.mjs", + target: target === null ? null : { + ref: stringValue(target.ref), + promotionCommit: stringValue(target.promotionCommit), + shortCommitId: stringValue(target.shortCommitId), + promotionSource: stringValue(target.promotionSource), + publishRequired: target.publishRequired ?? null, + headCommitId: stringValue(target.headCommitId), + headMatchesTarget: target.headMatchesTarget ?? null, + desiredStateCheck: desiredStateCheck === null ? null : { + status: stringValue(desiredStateCheck.status), + summary: desiredStateCheck.summary ?? null, + diagnostics: Array.isArray(desiredStateCheck.diagnostics) ? desiredStateCheck.diagnostics.slice(0, 5) : [], + }, + artifactBoundary: artifactBoundary === null ? null : { + status: stringValue(artifactBoundary.status), + desiredState: artifactBoundary.desiredState ?? null, + }, + namespace: stringValue(target.namespace) || namespace, + }, + deployJson: asRecord(record?.deployJson), + artifactCatalog: asRecord(record?.artifactCatalog), + artifactReport: asRecord(record?.artifactReport), + lock: asRecord(record?.lock), + liveDelta: asRecord(record?.liveDelta), + blockers: Array.isArray(record?.blockers) ? record.blockers.slice(0, 12) : [], + nextActions: Array.isArray(record?.nextActions) ? record.nextActions.slice(0, 12) : [], + parsed: record !== null, + command: commandView(command), + }; +} + +async function runDevCdApply(repoPath, kubeconfig, action, dumpDir, timeoutMs) { + const modeArg = action === "status" ? "--status" : "--dry-run"; + const command = ["node", "scripts/dev-cd-apply.mjs", modeArg, "--kubeconfig", kubeconfig, "--skip-live-verify"]; + const result = await runCaptured(command, repoPath, dumpDir, `controlled-dev-cd-apply-${action === "status" ? "status" : "dry-run"}`, { + env: { ...process.env, KUBECONFIG: kubeconfig }, + timeoutMs, + }); + return summarizeControlled(parseJson(result.stdoutText), result); +} + +function collectBlockers({ repo, git, guard, lock, secretRefs, controlled, action }) { + const blockers = []; + if (repo.status !== "selected") { + blockers.push({ scope: "hwlab-repo", summary: repo.rejected ? "Rejected runner history HWLAB directory." : "No eligible HWLAB CD repo with scripts/dev-cd-apply.mjs and deploy/deploy.json was found." }); + } + if (git?.blockers) blockers.push(...git.blockers); + if (guard) { + if (guard.refusal) blockers.push({ scope: "d601-native-k3s-guard", summary: guard.summary, refusal: true }); + else if (guard.status !== "pass") blockers.push({ scope: "d601-native-k3s-guard", summary: guard.summary }); + if ((action === "preflight" || action === "apply") && guard.secondControlPlaneRisk) { + blockers.push({ scope: "second-hwlab-dev-control-plane", summary: "Bare kubectl can observe hwlab-dev through a different control plane; refusing write-path planning." }); + } + } + if ((action === "preflight" || action === "apply") && lock?.held === true) { + blockers.push({ scope: "cd-lock", summary: `HWLAB DEV CD lock is held by ${lock.ownerTaskId || lock.holderIdentity || "unknown"}.` }); + } + if (secretRefs?.blockers) blockers.push(...secretRefs.blockers); + if (controlled) { + if (controlled.commandOk === false) blockers.push({ scope: "controlled-dev-cd", summary: "HWLAB scripts/dev-cd-apply.mjs did not complete successfully." }); + if (controlled.status === "blocked") blockers.push({ scope: "controlled-dev-cd-status", summary: "HWLAB scripts/dev-cd-apply.mjs reported blocked status." }); + } + return blockers; +} + +function nextSafeCommand(action, blockers) { + if (blockers.length > 0) return "resolve the structured blockers, then rerun bun scripts/cli.ts hwlab cd status --env dev"; + if (action === "status") return "bun scripts/cli.ts hwlab cd apply --env dev --dry-run"; + return "host commander or the unique CD runner may decide whether to run node scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report on D601; this wrapper did not apply or rollout"; +} + +async function main() { + const options = decodeOptions(); + const timeoutMs = Math.min(Number(options.timeoutMs || 45000), 60000); + const runId = options.runId || `${new Date().toISOString().replace(/[:.]/g, "-")}-${process.pid}`; + const dumpDir = path.join(os.homedir(), ".state", "unidesk-hwlab-cd", runId); + await fsp.mkdir(dumpDir, { recursive: true, mode: 0o700 }); + + const repo = resolveRepo(options.repoPath || null); + const action = options.action; + const base = { + ok: false, + env: "dev", + environment: "dev", + action, + dryRun: action === "apply" ? Boolean(options.dryRun) : action !== "status", + mutation: false, + remoteHost: "D601", + workspace: repo.status === "selected" ? { path: repo.path, source: repo.source } : null, + kubeconfig: { source: "explicit", path: options.kubeconfig || nativeKubeconfig, required: nativeKubeconfig }, + dumpPath: dumpDir, + safety: { + wrapperCallsOnlyRepoOwnedDevCdApply: true, + deployJsonAuthoritative: "deploy/deploy.json", + kubectlApplyExecuted: false, + rolloutExecuted: false, + liveVerifyExecuted: false, + secretValuesRead: false, + secretValuesPrinted: false, + cdLockMutated: false, + prodTouched: false, + }, + }; + + if (repo.status !== "selected") { + const blockers = collectBlockers({ repo, action }); + console.log(JSON.stringify({ ...base, status: "blocked", error: "hwlab-repo-not-found", repo, blockers, nextSafeCommand: nextSafeCommand(action, blockers) }, null, 2)); + process.exitCode = 1; + return; + } + + const repoPath = repo.path; + process.chdir(repoPath); + const [git, guard] = await Promise.all([ + gitSummary(repoPath, dumpDir, Math.min(timeoutMs, 15000)), + nodeGuard(options.kubeconfig || nativeKubeconfig, dumpDir, Math.min(timeoutMs, 15000)), + ]); + const desiredState = { + deployJson: readJsonSummary(repoPath, "deploy/deploy.json"), + artifactCatalog: readJsonSummary(repoPath, "deploy/artifact-catalog.dev.json"), + artifactReport: readJsonSummary(repoPath, "reports/dev-gate/dev-artifacts.json"), + }; + const lock = await lockStatus(options.kubeconfig || nativeKubeconfig, guard, dumpDir, Math.min(timeoutMs, 15000)); + const needsSecretPreflight = action === "preflight" || action === "apply"; + const secretRefs = needsSecretPreflight + ? await secretPreflight(options.kubeconfig || nativeKubeconfig, guard, dumpDir, Math.min(timeoutMs, 15000)) + : { status: "skipped", reason: "status-does-not-run-secret-preflight", namespace, mutation: false, blockers: [] }; + const preCommandBlockers = collectBlockers({ repo, git, guard, lock, secretRefs, action }); + const canRunControlled = preCommandBlockers.length === 0 || action === "status"; + const controlled = canRunControlled && git.status === "pass" && guard.status === "pass" + ? await runDevCdApply(repoPath, options.kubeconfig || nativeKubeconfig, action === "status" ? "status" : "apply", dumpDir, Math.max(1000, timeoutMs - 5000)) + : { status: "skipped", commandOk: true, reason: "preflight-blockers-before-controlled-dev-cd-apply" }; + const blockers = collectBlockers({ repo, git, guard, lock, secretRefs, controlled, action }); + const ok = blockers.length === 0; + const target = controlled.target || { + ref: "deploy/deploy.json", + promotionCommit: desiredState.deployJson.commitId || git.headCommit, + shortCommitId: desiredState.deployJson.commitId || null, + promotionSource: "deploy/deploy.json", + namespace, + }; + console.log(JSON.stringify({ + ...base, + ok, + status: ok ? action === "status" ? "ready" : "prepared" : guard.refusal ? "refused" : "blocked", + repo, + workspace: { path: repoPath, source: repo.source }, + worktreeGuard: git, + nodeGuard: guard, + secretPreflight: secretRefs, + lockState: lock, + desiredState, + target, + promotion: { + source: "deploy/deploy.json", + deployJson: desiredState.deployJson, + artifactCatalog: desiredState.artifactCatalog, + artifactReport: desiredState.artifactReport, + }, + controlledDevCd: controlled, + reportDumpPath: controlled.command?.dump?.stdout || dumpDir, + blockers, + nextSafeCommand: nextSafeCommand(action, blockers), + }, null, 2)); + process.exitCode = ok ? 0 : 1; +} + +main().catch((error) => { + const message = error instanceof Error ? error.message : String(error); + console.log(JSON.stringify({ ok: false, env: "dev", status: "blocked", error: "remote-wrapper-exception", message: redact(message) }, null, 2)); + process.exitCode = 1; +}); diff --git a/scripts/src/hwlab-cd.ts b/scripts/src/hwlab-cd.ts index 5dce42bd..585f5eb9 100644 --- a/scripts/src/hwlab-cd.ts +++ b/scripts/src/hwlab-cd.ts @@ -1,17 +1,14 @@ import { spawn } from "node:child_process"; import { randomBytes } from "node:crypto"; -import { accessSync, constants as fsConstants, createWriteStream, existsSync, mkdirSync, openSync, readSync, statSync, writeFileSync, closeSync } from "node:fs"; -import { mkdir, writeFile } from "node:fs/promises"; -import { join, resolve } from "node:path"; -import { repoRoot, rootPath } from "./config"; -import { - classifyD601DefaultKubectlDiagnostic, - classifyD601K3sTarget, - d601NativeKubeconfig, -} from "./d601-k3s-guard"; +import { createWriteStream, mkdirSync, readFileSync } from "node:fs"; +import { mkdir } from "node:fs/promises"; +import { join } from "node:path"; +import { readConfig, repoRoot, rootPath, type UniDeskConfig } from "./config"; +import { d601NativeKubeconfig } from "./d601-k3s-guard"; type HwlabCdAction = "status" | "preflight" | "apply"; type HwlabCdEnvironment = "dev"; +type HwlabCdTransport = "frontend" | "local"; interface HwlabCdOptions { action: HwlabCdAction; @@ -19,9 +16,10 @@ interface HwlabCdOptions { dryRun: boolean; repoPath: string | null; kubeconfig: string; + providerId: string; + transport: HwlabCdTransport; + mainServerHost: string | null; timeoutMs: number; - frontendLiveUrl: string; - apiLiveUrl: string; } interface CapturedCommand { @@ -40,41 +38,29 @@ interface CapturedCommand { stderr: string; stdoutBytes: number; stderrBytes: number; - stdoutTail: string; - stderrTail: string; }; } -interface CommandView { - id: string; - command: string[]; - cwd: string; +interface FrontendSession { + baseUrl: string; + cookie: string; +} + +interface FetchJsonResult { ok: boolean; - exitCode: number | null; - signal: NodeJS.Signals | null; - timedOut: boolean; - durationMs: number; - dump: { - stdout: string; - stderr: string; - stdoutBytes: number; - stderrBytes: number; - }; + status?: number; + body?: unknown; + error?: string; } +const defaultProviderId = "D601"; +const defaultHwlabCdRepoPath = "/home/ubuntu/hwlab_cd"; +const rejectedRunnerHistoryRepoPath = "/home/ubuntu/hwlab"; const namespace = "hwlab-dev"; const lockName = "hwlab-dev-cd-lock"; -const nativeKubeconfig = d601NativeKubeconfig; -const defaultHwlabCdRepoPath = "/home/ubuntu/hwlab_cd"; -const defaultFrontendLiveUrl = "http://74.48.78.17:16666/health/live"; -const defaultApiLiveUrl = "http://74.48.78.17:16667/health/live"; -const parseCaptureLimitBytes = 4 * 1024 * 1024; -const tailChars = 1000; -const requiredSecretRefs = [ - { secretName: "hwlab-cloud-api-dev-db", secretKey: "database-url", consumers: ["hwlab-cloud-api"] }, - { secretName: "hwlab-cloud-api-dev-db-admin", secretKey: "admin-url", consumers: ["runtime provisioning", "runtime migration"] }, - { secretName: "hwlab-code-agent-provider", secretKey: "openai-api-key", consumers: ["hwlab-cloud-api", "code agent provider"] }, -]; +const maxTimeoutMs = 60_000; +const defaultTimeoutMs = 45_000; +const remoteScriptSource = readFileSync(rootPath("scripts", "src", "hwlab-cd-remote-runner.cjs"), "utf8"); function isHelpArg(value: string | undefined): boolean { return value === "help" || value === "--help" || value === "-h"; @@ -89,7 +75,16 @@ function readOption(argv: string[], index: number, option: string): string { function parsePositiveInteger(value: string, option: string): number { const parsed = Number(value); if (!Number.isInteger(parsed) || parsed <= 0) throw new Error(`${option} must be a positive integer`); - return parsed; + return Math.min(parsed, maxTimeoutMs); +} + +function parseTransport(value: string, option: string): HwlabCdTransport { + if (value === "frontend" || value === "local") return value; + throw new Error(`${option} must be one of: frontend, local`); +} + +function envTransport(): HwlabCdTransport { + return process.env.UNIDESK_HWLAB_CD_TRANSPORT === "local" ? "local" : "frontend"; } function parseOptions(args: string[]): HwlabCdOptions { @@ -102,10 +97,11 @@ function parseOptions(args: string[]): HwlabCdOptions { environment: "dev", dryRun: false, repoPath: null, - kubeconfig: nativeKubeconfig, - timeoutMs: 60_000, - frontendLiveUrl: process.env.UNIDESK_HWLAB_CD_TEST_FRONTEND_LIVE_URL ?? defaultFrontendLiveUrl, - apiLiveUrl: process.env.UNIDESK_HWLAB_CD_TEST_API_LIVE_URL ?? defaultApiLiveUrl, + kubeconfig: d601NativeKubeconfig, + providerId: defaultProviderId, + transport: envTransport(), + mainServerHost: process.env.UNIDESK_MAIN_SERVER_IP ?? process.env.UNIDESK_MAIN_SERVER_HOST ?? null, + timeoutMs: defaultTimeoutMs, }; let envSeen = false; @@ -124,26 +120,30 @@ function parseOptions(args: string[]): HwlabCdOptions { index += 1; } else if (arg === "--kubeconfig") { const value = readOption(args, index + 1, arg); - if (value !== nativeKubeconfig) { - throw new Error(`hwlab cd requires the D601 native k3s kubeconfig: ${nativeKubeconfig}`); - } + if (value !== d601NativeKubeconfig) throw new Error(`hwlab cd requires the D601 native k3s kubeconfig: ${d601NativeKubeconfig}`); options.kubeconfig = value; index += 1; + } else if (arg === "--provider-id") { + options.providerId = readOption(args, index + 1, arg); + index += 1; + } else if (arg === "--main-server-ip" || arg === "--main-server-host") { + options.mainServerHost = readOption(args, index + 1, arg); + index += 1; + } else if (arg === "--transport") { + options.transport = parseTransport(readOption(args, index + 1, arg), arg); + index += 1; } else if (arg === "--timeout-ms") { options.timeoutMs = parsePositiveInteger(readOption(args, index + 1, arg), arg); index += 1; - } else if (arg === "--frontend-live-url") { - options.frontendLiveUrl = readOption(args, index + 1, arg); - index += 1; - } else if (arg === "--api-live-url") { - options.apiLiveUrl = readOption(args, index + 1, arg); - index += 1; } else if (!isHelpArg(arg)) { throw new Error(`unknown hwlab cd option: ${arg}`); } } if (!envSeen) throw new Error("hwlab cd requires --env dev"); + if (options.action === "apply" && !options.dryRun) { + return options; + } return options; } @@ -152,73 +152,143 @@ function makeRunId(): string { return `${timestamp}-${process.pid}-${randomBytes(4).toString("hex")}`; } -function redact(value: string): string { - return value - .replace(/\b(Bearer\s+)[A-Za-z0-9._~+/=-]+/giu, "$1") - .replace(/\b(token|password|client-key-data|client-certificate-data|certificate-authority-data)\b(\s*[:=]\s*)(["']?)([^"'\s,}]+)/giu, "$1$2$3") - .replace(/\b(postgres(?:ql)?:\/\/[^:\s/@]+:)([^@\s]+)(@)/giu, "$1$3"); +function shellQuote(value: string): string { + return `'${value.replace(/'/gu, "'\\''")}'`; } -function boundedTail(value: string): string { - const redacted = redact(value); - return redacted.slice(Math.max(0, redacted.length - tailChars)); +function b64Json(value: unknown): string { + return Buffer.from(JSON.stringify(value), "utf8").toString("base64"); } -async function runCaptured(command: string[], cwd: string, dumpDir: string, id: string, options: { env?: NodeJS.ProcessEnv; timeoutMs?: number } = {}): Promise { +function buildRemoteCommand(options: HwlabCdOptions, runId: string): string { + const remoteOptions = { + action: options.action, + environment: options.environment, + dryRun: options.dryRun, + repoPath: options.repoPath, + kubeconfig: options.kubeconfig, + timeoutMs: options.timeoutMs, + runId, + }; + return [ + `UNIDESK_HWLAB_CD_OPTIONS_B64=${shellQuote(b64Json(remoteOptions))} node <<'UNIDESK_HWLAB_CD_JS'`, + remoteScriptSource, + "UNIDESK_HWLAB_CD_JS", + ].join("\n"); +} + +export function buildHwlabCdRemoteCommandForTest(args: string[]): string { + return buildRemoteCommand(parseOptions(args), "test-run-id"); +} + +function frontendBaseUrl(host: string, config: UniDeskConfig): string { + if (host.startsWith("http://") || host.startsWith("https://")) return host.replace(/\/+$/u, ""); + if (/:\d+$/u.test(host)) return `http://${host}`; + return `http://${host}:${config.network.frontend.port}`; +} + +async function readJson(url: string, init?: RequestInit, timeoutMs = 8000): Promise { + const controller = new AbortController(); + const timer = setTimeout(() => controller.abort(), timeoutMs); + try { + const response = await fetch(url, { ...init, signal: controller.signal }); + const text = await response.text(); + let body: unknown = text; + try { + body = text.length > 0 ? JSON.parse(text) : null; + } catch { + body = { text: text.slice(0, 2000) }; + } + return { ok: response.ok, status: response.status, body }; + } catch (error) { + return { ok: false, error: error instanceof Error ? error.message : String(error) }; + } finally { + clearTimeout(timer); + } +} + +async function loginFrontend(host: string, config: UniDeskConfig): Promise { + const baseUrl = frontendBaseUrl(host, config); + const controller = new AbortController(); + const timer = setTimeout(() => controller.abort(), 8000); + try { + const raw = await fetch(`${baseUrl}/login`, { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ username: config.auth.username, password: config.auth.password }), + signal: controller.signal, + }); + if (!raw.ok) throw new Error(`frontend login failed via ${baseUrl}: status=${raw.status}`); + const cookie = raw.headers.get("set-cookie")?.split(";")[0] ?? ""; + if (cookie.length === 0) throw new Error(`frontend login via ${baseUrl} did not return a session cookie`); + return { baseUrl, cookie }; + } finally { + clearTimeout(timer); + } +} + +async function frontendJson(session: FrontendSession, path: string, init?: RequestInit, timeoutMs = 8000): Promise { + const headers = new Headers(init?.headers); + headers.set("cookie", session.cookie); + if (init?.body !== undefined && !headers.has("content-type")) headers.set("content-type", "application/json"); + return readJson(`${session.baseUrl}${path}`, { ...init, headers }, timeoutMs); +} + +function asRecord(value: unknown): Record | null { + return typeof value === "object" && value !== null && !Array.isArray(value) ? value as Record : null; +} + +async function waitForFrontendTask(session: FrontendSession, taskId: string, timeoutMs: number): Promise { + const startedAt = Date.now(); + let latest: unknown = null; + while (Date.now() - startedAt < timeoutMs) { + latest = await frontendJson(session, `/api/tasks/${encodeURIComponent(taskId)}`, undefined, 8000); + const task = asRecord(asRecord((latest as { body?: unknown }).body)?.task); + const status = typeof task?.status === "string" ? task.status : null; + if (status === "succeeded" || status === "failed") return { ok: true, task }; + await Bun.sleep(500); + } + return { ok: false, timeoutMs, latest }; +} + +async function runCaptured(command: string[], cwd: string, dumpDir: string, id: string, timeoutMs: number, env: NodeJS.ProcessEnv = process.env): Promise { await mkdir(dumpDir, { recursive: true }); const stdoutPath = join(dumpDir, `${id}.stdout.txt`); const stderrPath = join(dumpDir, `${id}.stderr.txt`); - writeFileSync(stdoutPath, "", { mode: 0o600 }); - writeFileSync(stderrPath, "", { mode: 0o600 }); - + const stdout = createWriteStream(stdoutPath, { flags: "w", mode: 0o600 }); + const stderr = createWriteStream(stderrPath, { flags: "w", mode: 0o600 }); const startedAt = Date.now(); - let timedOut = false; + let stdoutText = ""; + let stderrText = ""; let stdoutBytes = 0; let stderrBytes = 0; + let timedOut = false; let spawnErrorMessage = ""; - const stdoutChunks: Buffer[] = []; - const stderrChunks: Buffer[] = []; - const stdout = createWriteStream(stdoutPath, { flags: "a" }); - const stderr = createWriteStream(stderrPath, { flags: "a" }); - - const child = spawn(command[0] ?? "", command.slice(1), { - cwd, - env: options.env ?? process.env, - stdio: ["ignore", "pipe", "pipe"], - }); - const timeout = options.timeoutMs === undefined - ? null - : setTimeout(() => { - timedOut = true; - child.kill("SIGTERM"); - setTimeout(() => child.kill("SIGKILL"), 1000).unref(); - }, options.timeoutMs); - + const child = spawn(command[0] ?? "", command.slice(1), { cwd, env, stdio: ["ignore", "pipe", "pipe"] }); + const timer = setTimeout(() => { + timedOut = true; + child.kill("SIGTERM"); + setTimeout(() => child.kill("SIGKILL"), 1000).unref(); + }, timeoutMs); child.stdout.on("data", (chunk: Buffer) => { stdoutBytes += chunk.byteLength; stdout.write(chunk); - if (stdoutBytes <= parseCaptureLimitBytes) stdoutChunks.push(Buffer.from(chunk)); + if (stdoutText.length < 1_000_000) stdoutText += chunk.toString("utf8"); }); child.stderr.on("data", (chunk: Buffer) => { stderrBytes += chunk.byteLength; stderr.write(chunk); - if (stderrBytes <= parseCaptureLimitBytes) stderrChunks.push(Buffer.from(chunk)); + if (stderrText.length < 1_000_000) stderrText += chunk.toString("utf8"); }); - const closed = await new Promise<{ exitCode: number | null; signal: NodeJS.Signals | null }>((resolveClose) => { child.on("error", (error) => { spawnErrorMessage = error.message; }); - child.on("close", (exitCode, signal) => { - resolveClose({ exitCode, signal }); - }); + child.on("close", (exitCode, signal) => resolveClose({ exitCode, signal })); }); - if (timeout !== null) clearTimeout(timeout); + clearTimeout(timer); await new Promise((resolveEnd) => stdout.end(resolveEnd)); await new Promise((resolveEnd) => stderr.end(resolveEnd)); - - const stdoutText = stdoutBytes <= parseCaptureLimitBytes ? Buffer.concat(stdoutChunks).toString("utf8") : ""; - const stderrText = stderrBytes <= parseCaptureLimitBytes ? Buffer.concat(stderrChunks).toString("utf8") : spawnErrorMessage; const exitCode = spawnErrorMessage.length > 0 && closed.exitCode === null ? 127 : closed.exitCode; return { id, @@ -230,884 +300,194 @@ async function runCaptured(command: string[], cwd: string, dumpDir: string, id: timedOut, durationMs: Date.now() - startedAt, stdoutText, - stderrText, - dump: { - stdout: stdoutPath, - stderr: stderrPath, - stdoutBytes, - stderrBytes, - stdoutTail: boundedTail(stdoutText || readFileTail(stdoutPath)), - stderrTail: boundedTail(stderrText || readFileTail(stderrPath)), - }, + stderrText: stderrText || spawnErrorMessage, + dump: { stdout: stdoutPath, stderr: stderrPath, stdoutBytes, stderrBytes }, }; } -function readFileTail(path: string): string { +function parseRemoteStdout(stdout: string): Record | null { + const trimmed = stdout.trim(); + if (trimmed.length === 0) return null; try { - const size = statSync(path).size; - const bytesToRead = Math.min(size, tailChars * 4); - const buffer = Buffer.alloc(bytesToRead); - const fd = openSync(path, "r"); - try { - readSync(fd, buffer, 0, bytesToRead, size - bytesToRead); - } finally { - closeSync(fd); + return JSON.parse(trimmed) as Record; + } catch { + const start = trimmed.indexOf("{"); + const end = trimmed.lastIndexOf("}"); + if (start >= 0 && end > start) { + try { + return JSON.parse(trimmed.slice(start, end + 1)) as Record; + } catch { + return null; + } } - return buffer.toString("utf8").slice(-tailChars); - } catch { - return ""; - } -} - -function commandView(result: CapturedCommand): CommandView { - return { - id: result.id, - command: result.command, - cwd: result.cwd, - ok: result.ok, - exitCode: result.exitCode, - signal: result.signal, - timedOut: result.timedOut, - durationMs: result.durationMs, - dump: { - stdout: result.dump.stdout, - stderr: result.dump.stderr, - stdoutBytes: result.dump.stdoutBytes, - stderrBytes: result.dump.stderrBytes, - }, - }; -} - -function parseJson(text: string): unknown | null { - if (text.trim().length === 0) return null; - try { - return JSON.parse(text); - } catch { return null; } } -function asRecord(value: unknown): Record | null { - return typeof value === "object" && value !== null && !Array.isArray(value) ? value as Record : null; +function commandShape(providerId = defaultProviderId): string[] { + return ["frontend", "/api/dispatch", providerId, "host.ssh", "UNIDESK_HWLAB_CD_OPTIONS_B64=... node <<'UNIDESK_HWLAB_CD_JS'"]; } -function stringValue(value: unknown): string | null { - return typeof value === "string" && value.length > 0 ? value : null; -} - -function accessCheck(path: string, mode: number): Record { - try { - accessSync(path, mode); - return { ok: true, path }; - } catch (error) { - return { ok: false, path, error: error instanceof Error ? error.message : String(error) }; - } -} - -function defaultRepoCandidates(provided: string | null): { source: string; path: string }[] { - if (provided !== null) return [{ source: "option", path: provided }]; - if (process.env.UNIDESK_HWLAB_REPO !== undefined) return [{ source: "env:UNIDESK_HWLAB_REPO", path: process.env.UNIDESK_HWLAB_REPO }]; - return [ - { source: "default:d601-clean-mirror", path: defaultHwlabCdRepoPath }, - { source: "rejected:runner-history", path: "/home/ubuntu/hwlab" }, - ]; -} - -function resolveHwlabRepo(provided: string | null): Record { - const candidates = defaultRepoCandidates(provided).map((candidate) => { - const absolutePath = resolve(candidate.path); - const devCdApply = join(absolutePath, "scripts/dev-cd-apply.mjs"); - const desiredStatePlan = join(absolutePath, "scripts/deploy-desired-state-plan.mjs"); - const rejected = candidate.source.startsWith("rejected:") || absolutePath === "/home/ubuntu/hwlab"; - return { - ...candidate, - path: absolutePath, - exists: existsSync(absolutePath), - eligible: !rejected, - rejected, - rejectionReason: rejected ? "runner-history-directory-is-not-hwlab-cd-release-truth" : null, - hasDevCdApply: existsSync(devCdApply), - hasDesiredStatePlan: existsSync(desiredStatePlan), - }; - }); - const selected = candidates.find((candidate) => candidate.eligible && candidate.exists && candidate.hasDevCdApply && candidate.hasDesiredStatePlan) ?? null; - return { - ok: selected !== null, - defaultPath: defaultHwlabCdRepoPath, - selected, - candidates, - }; -} - -async function gitSummary(repoPath: string, dumpDir: string, timeoutMs: number): Promise> { - const [branch, head, originMain, remote, gitDir, statusShort, statusPorcelain] = await Promise.all([ - runCaptured(["git", "rev-parse", "--abbrev-ref", "HEAD"], repoPath, dumpDir, "git-branch", { timeoutMs }), - runCaptured(["git", "rev-parse", "HEAD"], repoPath, dumpDir, "git-head", { timeoutMs }), - runCaptured(["git", "rev-parse", "--verify", "origin/main^{commit}"], repoPath, dumpDir, "git-origin-main", { timeoutMs }), - runCaptured(["git", "remote", "get-url", "origin"], repoPath, dumpDir, "git-remote-origin", { timeoutMs }), - runCaptured(["git", "rev-parse", "--path-format=absolute", "--git-dir"], repoPath, dumpDir, "git-dir", { timeoutMs }), - runCaptured(["git", "status", "--short", "--branch"], repoPath, dumpDir, "git-status-short", { timeoutMs }), - runCaptured(["git", "status", "--porcelain=v1"], repoPath, dumpDir, "git-status-porcelain", { timeoutMs }), - ]); - const branchName = branch.stdoutText.trim(); - const headCommit = head.stdoutText.trim(); - const originMainCommit = originMain.stdoutText.trim(); - const remoteUrl = remote.stdoutText.trim(); - const gitDirPath = gitDir.stdoutText.trim(); - const fetchHeadPath = gitDirPath.length > 0 ? join(gitDirPath, "FETCH_HEAD") : join(repoPath, ".git", "FETCH_HEAD"); - const porcelain = statusPorcelain.stdoutText.trim(); - const statusLines = statusShort.stdoutText.trim().split("\n").filter((line) => line.length > 0); - const dirtyLines = porcelain.length === 0 ? [] : porcelain.split("\n").filter((line) => line.length > 0); - const worktreeAccess = accessCheck(repoPath, fsConstants.R_OK | fsConstants.W_OK | fsConstants.X_OK); - const fetchHeadAccess = accessCheck(fetchHeadPath, fsConstants.R_OK); - const remoteMatches = /github\.com[:/]pikasTech\/HWLAB(?:\.git)?$/u.test(remoteUrl); - return { - ok: branch.ok && head.ok && remote.ok && gitDir.ok && statusShort.ok && statusPorcelain.ok && worktreeAccess.ok && fetchHeadAccess.ok, - clean: dirtyLines.length === 0, - branch: branchName || null, - onMain: branchName === "main", - remote: remoteUrl || null, - remoteMatches, - expectedRemote: "git@github.com:pikasTech/HWLAB.git or https://github.com/pikasTech/HWLAB.git", - headCommit: headCommit || null, - originMainCommit: originMain.ok ? originMainCommit || null : null, - headMatchesOriginMain: originMain.ok && headCommit.length > 0 && headCommit === originMainCommit, - worktreeAccess, - fetchHead: { - path: fetchHeadPath, - exists: existsSync(fetchHeadPath), - readable: fetchHeadAccess.ok, - error: fetchHeadAccess.error, - }, - statusShort: statusLines.slice(0, 40), - dirtyCount: dirtyLines.length, - dirtyPreview: dirtyLines.slice(0, 30), - commands: [branch, head, originMain, remote, gitDir, statusShort, statusPorcelain].map(commandView), - }; -} - -async function nativeK3sGuard(kubeconfig: string, dumpDir: string, timeoutMs: number): Promise> { - const env = { ...process.env, KUBECONFIG: kubeconfig }; - const [context, server, nodes, defaultContext, defaultServer, defaultNodes] = await Promise.all([ - runCaptured(["kubectl", "config", "current-context"], repoRoot, dumpDir, "k3s-current-context", { env, timeoutMs }), - runCaptured(["kubectl", "config", "view", "--minify", "-o", "jsonpath={.clusters[0].cluster.server}"], repoRoot, dumpDir, "k3s-server", { env, timeoutMs }), - runCaptured(["kubectl", "get", "nodes", "-o", "jsonpath={range .items[*]}{.metadata.name}{\"\\n\"}{end}"], repoRoot, dumpDir, "k3s-nodes", { env, timeoutMs }), - runCaptured(["kubectl", "config", "current-context"], repoRoot, dumpDir, "default-kubectl-current-context", { timeoutMs }), - runCaptured(["kubectl", "config", "view", "--minify", "-o", "jsonpath={.clusters[0].cluster.server}"], repoRoot, dumpDir, "default-kubectl-server", { timeoutMs }), - runCaptured(["kubectl", "get", "nodes", "-o", "jsonpath={range .items[*]}{.metadata.name}{\"\\n\"}{end}"], repoRoot, dumpDir, "default-kubectl-nodes", { timeoutMs }), - ]); - const contextText = context.stdoutText.trim(); - const serverText = server.stdoutText.trim(); - const nodeNames = nodes.stdoutText.split("\n").map((line) => line.trim()).filter((line) => line.length > 0); - const defaultDiagnostic = classifyD601DefaultKubectlDiagnostic({ - currentContext: defaultContext.stdoutText.trim() || null, - apiServer: defaultServer.stdoutText.trim() || null, - combinedText: `${defaultContext.stdoutText}\n${defaultContext.stderrText}\n${defaultServer.stdoutText}\n${defaultServer.stderrText}\n${defaultNodes.stdoutText}\n${defaultNodes.stderrText}`, - commandsOk: defaultContext.ok && defaultServer.ok && defaultNodes.ok, - }); - const guard = classifyD601K3sTarget({ - kubeconfig, - expectedKubeconfig: nativeKubeconfig, - currentContext: contextText || null, - apiServer: serverText || null, - nodeNames, - commandsOk: context.ok && server.ok && nodes.ok, - combinedText: `${context.stdoutText}\n${context.stderrText}\n${server.stdoutText}\n${server.stderrText}\n${nodes.stdoutText}\n${nodes.stderrText}`, - }, defaultDiagnostic); - return { - ...guard, - injectedEnv: { KUBECONFIG: kubeconfig }, - commands: [context, server, nodes, defaultContext, defaultServer, defaultNodes].map(commandView), - }; -} - -async function liveWorkloadStatus(kubeconfig: string, guard: Record, dumpDir: string, timeoutMs: number): Promise> { - if (guard.refusal === true || guard.status !== "pass") { - return { - status: "skipped", - reason: "d601-native-k3s-guard-not-pass", - mutation: false, - namespace, - }; - } - const env = { ...process.env, KUBECONFIG: kubeconfig }; - const jsonpath = "{range .items[*]}{.metadata.name}{\"\\t\"}{range .spec.template.spec.containers[*]}{.name}{\"=\"}{.image}{\",\"}{end}{\"\\n\"}{end}"; - const result = await runCaptured(["kubectl", "-n", namespace, "get", "deploy", "-o", `jsonpath=${jsonpath}`], repoRoot, dumpDir, "live-workload-images", { env, timeoutMs }); - const workloads = result.stdoutText - .split("\n") - .map((line) => line.trim()) - .filter((line) => line.length > 0) - .slice(0, 80) - .map((line) => { - const [deployment, rawContainers = ""] = line.split("\t"); - const containers = rawContainers - .split(",") - .map((item) => item.trim()) - .filter((item) => item.length > 0) - .slice(0, 20) - .map((item) => { - const separator = item.indexOf("="); - const name = separator === -1 ? item : item.slice(0, separator); - const image = separator === -1 ? null : item.slice(separator + 1); - return { name, image, tag: image === null ? null : image.split(":").at(-1) ?? null }; - }); - return { deployment, containers }; - }); - return { - status: result.ok ? "observed" : "blocked", - namespace, - mutation: false, - workloadCount: workloads.length, - workloads, - command: commandView(result), - }; -} - -function annotation(annotations: Record, name: string): string | null { - return stringValue(annotations[`hwlab.pikastech.local/${name}`]); -} - -function classifyLock(lock: Record | null): Record { - if (lock === null || lock.phase === "released") return { held: false, stale: false, retryAfterSeconds: 0, expiresAt: null }; - const updatedAt = stringValue(lock.updatedAt) ?? ""; - const ttlSeconds = Number(lock.ttlSeconds ?? 3600); - const updatedMs = Number.isFinite(Date.parse(updatedAt)) ? Date.parse(updatedAt) : 0; - const expiresAtMs = updatedMs + (Number.isFinite(ttlSeconds) ? ttlSeconds : 3600) * 1000; - const retryAfterSeconds = Math.max(0, Math.ceil((expiresAtMs - Date.now()) / 1000)); - return { - held: retryAfterSeconds > 0, - stale: retryAfterSeconds <= 0, - retryAfterSeconds, - expiresAt: Number.isFinite(expiresAtMs) ? new Date(expiresAtMs).toISOString() : null, - }; -} - -async function cdLockStatus(kubeconfig: string, guard: Record, dumpDir: string, timeoutMs: number): Promise> { - if (guard.refusal === true) { - return { - status: "skipped", - reason: "native-k3s-guard-refused", - present: null, - lockName, - namespace, - }; - } - const env = { ...process.env, KUBECONFIG: kubeconfig }; - const result = await runCaptured(["kubectl", "-n", namespace, "get", "lease", lockName, "-o", "json"], repoRoot, dumpDir, "cd-lock", { env, timeoutMs }); - const text = `${result.stderrText}\n${result.stdoutText}`; - if (!result.ok && /not\s*found|notfound/iu.test(text)) { - return { - status: "absent", - present: false, - held: false, - stale: false, - lockName, - namespace, - command: commandView(result), - }; - } - const parsed = asRecord(parseJson(result.stdoutText)); - const annotations = asRecord(asRecord(parsed?.metadata)?.annotations) ?? {}; - const lock = parsed === null ? null : { - lockBackend: "Lease", - lockName: stringValue(asRecord(parsed.metadata)?.name) ?? lockName, - namespace: stringValue(asRecord(parsed.metadata)?.namespace) ?? namespace, - holderIdentity: stringValue(asRecord(parsed.spec)?.holderIdentity), - resourceVersion: stringValue(asRecord(parsed.metadata)?.resourceVersion), - ownerTaskId: annotation(annotations, "ownerTaskId"), - transactionId: annotation(annotations, "transactionId"), - phase: annotation(annotations, "phase") ?? "unknown", - promotionCommit: annotation(annotations, "promotionCommit"), - deployJsonHash: annotation(annotations, "deployJsonHash"), - targetRef: annotation(annotations, "targetRef"), - targetNamespace: annotation(annotations, "targetNamespace"), - updatedAt: annotation(annotations, "updatedAt") ?? stringValue(asRecord(parsed.spec)?.renewTime), - ttlSeconds: Number(annotation(annotations, "ttlSeconds") ?? asRecord(parsed.spec)?.leaseDurationSeconds ?? 3600), - releaseStatus: annotation(annotations, "releaseStatus"), - releasedAt: annotation(annotations, "releasedAt"), - }; - return { - status: result.ok && lock !== null ? "observed" : "blocked", - present: result.ok && lock !== null, - ...(lock ?? {}), - ...classifyLock(lock), - command: commandView(result), - }; -} - -function summarizeDesiredState(parsed: unknown, command: CapturedCommand): Record { - const record = asRecord(parsed); - const summary = asRecord(record?.summary); - const source = asRecord(record?.source); - const promotionBoundary = asRecord(record?.promotionBoundary); - return { - status: stringValue(record?.status) ?? (command.ok ? "unknown" : "blocked"), - kind: stringValue(record?.kind), - mode: stringValue(record?.mode), - source: { - deployJson: stringValue(source?.deploy) ?? "deploy/deploy.json", - artifactCatalog: stringValue(source?.artifactCatalog) ?? "deploy/artifact-catalog.dev.json", - workloads: stringValue(source?.workloads) ?? "deploy/k8s/base/workloads.yaml", - optionalReport: stringValue(source?.optionalReport), - }, - authoritativeDesiredState: Array.isArray(promotionBoundary?.authoritativeDesiredState) - ? promotionBoundary.authoritativeDesiredState.slice(0, 10) - : ["deploy/deploy.json", "deploy/artifact-catalog.dev.json", "deploy/k8s/base/workloads.yaml"], - nonAuthoritativeEvidence: Array.isArray(promotionBoundary?.nonAuthoritativeEvidence) - ? promotionBoundary.nonAuthoritativeEvidence.slice(0, 10) - : [], - desiredCommitId: stringValue(summary?.desiredCommitId), - desiredImageTag: stringValue(summary?.desiredImageTag), - artifactState: stringValue(summary?.artifactState), - ciPublished: summary?.ciPublished ?? null, - registryVerified: summary?.registryVerified ?? null, - services: summary?.services ?? null, - workloadContainers: summary?.workloadContainers ?? null, - diagnostics: summary?.diagnostics ?? null, - blockers: summary?.blockers ?? null, - targetConvergence: summary?.targetConvergence ?? null, - command: commandView(command), - parsed: record !== null, - }; -} - -async function desiredStateStatus(repoPath: string, dumpDir: string, timeoutMs: number): Promise> { - const script = join(repoPath, "scripts/deploy-desired-state-plan.mjs"); - if (!existsSync(script)) { - return { status: "skipped", reason: "scripts/deploy-desired-state-plan.mjs not found" }; - } - const result = await runCaptured(["node", "scripts/deploy-desired-state-plan.mjs", "--pretty"], repoPath, dumpDir, "desired-state-plan", { timeoutMs }); - return summarizeDesiredState(parseJson(result.stdoutText), result); -} - -async function controlledObservability(repoPath: string, dumpDir: string, timeoutMs: number): Promise> { - const script = join(repoPath, "scripts/d601-k3s-readonly-observability.mjs"); - if (!existsSync(script)) { - return { status: "skipped", reason: "scripts/d601-k3s-readonly-observability.mjs not found" }; - } - const result = await runCaptured(["node", "scripts/d601-k3s-readonly-observability.mjs", "--no-write", "--timeout-ms", String(Math.min(timeoutMs, 10_000))], repoPath, dumpDir, "d601-readonly-observability", { - env: { ...process.env, KUBECONFIG: nativeKubeconfig }, - timeoutMs: Math.max(timeoutMs, 15_000), - }); - const parsed = asRecord(parseJson(result.stdoutText)); - return { - status: stringValue(parsed?.conclusion) ?? (result.ok ? "unknown" : "blocked"), - reportKind: stringValue(parsed?.reportKind), - readable: parsed?.readable ?? null, - runnerKubeconfigReadable: parsed?.runnerKubeconfigReadable ?? null, - d601PublicEndpointsReachable: parsed?.d601PublicEndpointsReachable ?? null, - d601K3sUnavailable: parsed?.d601K3sUnavailable ?? null, - attemptedExecutors: parsed?.attemptedExecutors ?? [], - blockers: parsed?.blockers ?? [], - command: commandView(result), - parsed: parsed !== null, - }; -} - -async function controlledDevCdDryRun(repoPath: string, kubeconfig: string, dumpDir: string, timeoutMs: number): Promise> { - const result = await runCaptured(["node", "scripts/dev-cd-apply.mjs", "--dry-run", "--kubeconfig", kubeconfig, "--skip-live-verify"], repoPath, dumpDir, "controlled-dev-cd-apply-dry-run", { - env: { ...process.env, KUBECONFIG: kubeconfig }, - timeoutMs, - }); - const parsed = asRecord(parseJson(result.stdoutText)); - const target = asRecord(parsed?.target); - const desiredStateCheck = asRecord(target?.desiredStateCheck); - const artifactBoundary = asRecord(target?.artifactBoundary); - return { - status: stringValue(parsed?.status) ?? (result.ok ? "unknown" : "blocked"), - commandOk: result.ok, - mode: stringValue(parsed?.mode), - mutationAttempted: parsed?.mutationAttempted ?? false, - prodTouched: parsed?.prodTouched ?? false, - target: target === null ? null : { - ref: stringValue(target.ref), - promotionCommit: stringValue(target.promotionCommit), - shortCommitId: stringValue(target.shortCommitId), - promotionSource: stringValue(target.promotionSource), - publishRequired: target.publishRequired ?? null, - headCommitId: stringValue(target.headCommitId), - headMatchesTarget: target.headMatchesTarget ?? null, - desiredStateCheck, - artifactBoundary: artifactBoundary === null ? null : { - status: stringValue(artifactBoundary.status), - desiredState: artifactBoundary.desiredState ?? null, - }, - namespace: stringValue(target.namespace) ?? namespace, - }, - deployJson: asRecord(parsed?.deployJson), - artifactCatalog: asRecord(parsed?.artifactCatalog), - artifactReport: asRecord(parsed?.artifactReport), - lock: asRecord(parsed?.lock), - liveDelta: asRecord(parsed?.liveDelta), - nextActions: Array.isArray(parsed?.nextActions) ? parsed.nextActions.slice(0, 20) : [], - controlledEntrypoint: "scripts/dev-cd-apply.mjs", - command: commandView(result), - parsed: parsed !== null, - }; -} - -function escapeRegExp(value: string): string { - return value.replace(/[.*+?^${}()|[\]\\]/gu, "\\$&"); -} - -async function requiredSecretRefPreflight(kubeconfig: string, guard: Record, dumpDir: string, timeoutMs: number): Promise> { - if (guard.refusal === true || guard.status !== "pass") { - return { - status: "skipped", - reason: "d601-native-k3s-guard-not-pass", - namespace, - mutation: false, - safety: { - secretValuesRead: false, - secretValuesPrinted: false, - secretKeyNamesOnly: true, - }, - secretRefs: requiredSecretRefs.map((ref) => ({ ...ref, status: "not_checked" })), - blockers: [], - }; - } - const env = { ...process.env, KUBECONFIG: kubeconfig }; - const observations = await Promise.all(requiredSecretRefs.map(async (ref) => { - const idBase = `secretref-${ref.secretName}-${ref.secretKey}`.replace(/[^A-Za-z0-9_.-]/gu, "-"); - const exists = await runCaptured(["kubectl", "-n", namespace, "get", "secret", ref.secretName, "-o", "name"], repoRoot, dumpDir, `${idBase}-exists`, { env, timeoutMs }); - if (!exists.ok) { - return { - ...ref, - status: "missing-secret", - exists: false, - keyPresent: false, - command: commandView(exists), - }; - } - const describe = await runCaptured(["kubectl", "-n", namespace, "describe", "secret", ref.secretName], repoRoot, dumpDir, `${idBase}-describe`, { env, timeoutMs }); - const keyPattern = new RegExp(`(?:^|\\s)${escapeRegExp(ref.secretKey)}(?:\\s|:|$)`, "mu"); - const keyPresent = describe.ok && keyPattern.test(describe.stdoutText); - return { - ...ref, - status: keyPresent ? "present" : describe.ok ? "missing-key" : "key-observation-blocked", - exists: true, - keyPresent, - command: commandView(describe), - }; - })); - const blockers = observations - .filter((observation) => observation.status !== "present") - .map((observation) => ({ - scope: `secretref:${observation.secretName}/${observation.secretKey}`, - summary: observation.status === "missing-secret" - ? `Required Secret ${observation.secretName} is missing in ${namespace}.` - : `Required SecretRef ${observation.secretName}/${observation.secretKey} is not present as key metadata in ${namespace}.`, - impact: `${observation.consumers.join(", ")} would fail after a DEV CD apply or runtime preflight Job.`, - runbook: `Create or repair Secret ${observation.secretName} with key ${observation.secretKey} in namespace ${namespace}; verify key presence only and do not print the Secret value.`, - })); - return { - status: blockers.length === 0 ? "pass" : "blocked", - namespace, - mutation: false, - safety: { - secretValuesRead: false, - secretValuesPrinted: false, - secretKeyNamesOnly: true, - }, - requiredSecretRefs: requiredSecretRefs.map((ref) => `${ref.secretName}/${ref.secretKey}`), - secretRefs: observations, - blockers, - }; -} - -function revisionFromHealth(json: Record | null): string | null { - const commit = asRecord(json?.commit); - const image = asRecord(json?.image); - return stringValue(json?.revision) - ?? stringValue(json?.commitId) - ?? stringValue(commit?.id) - ?? stringValue(image?.tag) - ?? null; -} - -function imageFromHealth(json: Record | null): string | null { - const image = json?.image; - if (typeof image === "string") return image; - return stringValue(asRecord(image)?.reference); -} - -async function fetchLive(url: string, dumpDir: string, id: string, timeoutMs: number): Promise> { - const startedAt = Date.now(); - await mkdir(dumpDir, { recursive: true }); - const bodyPath = join(dumpDir, `${id}.body.txt`); - const errorPath = join(dumpDir, `${id}.error.txt`); - const controller = new AbortController(); - const timeout = setTimeout(() => controller.abort(), timeoutMs); - try { - const response = await fetch(url, { method: "GET", signal: controller.signal }); - const body = await response.text(); - await writeFile(bodyPath, body, { mode: 0o600 }); - await writeFile(errorPath, "", { mode: 0o600 }); - const json = asRecord(parseJson(body)); - return { - id, - url, - ok: response.ok && json !== null, - httpStatus: response.status, - durationMs: Date.now() - startedAt, - serviceId: stringValue(json?.serviceId) ?? stringValue(asRecord(json?.service)?.id), - environment: stringValue(json?.environment), - applicationStatus: stringValue(json?.status), - ready: json?.ready ?? null, - revision: revisionFromHealth(json), - image: imageFromHealth(json), - blockerCodes: Array.isArray(json?.blockerCodes) ? json.blockerCodes.slice(0, 20) : [], - dump: { - body: bodyPath, - error: errorPath, - bodyBytes: Buffer.byteLength(body, "utf8"), - bodyTail: boundedTail(body), - }, - }; - } catch (error) { - const message = error instanceof Error ? error.message : String(error); - await writeFile(bodyPath, "", { mode: 0o600 }); - await writeFile(errorPath, message, { mode: 0o600 }); - return { - id, - url, - ok: false, - httpStatus: null, - durationMs: Date.now() - startedAt, - error: message, - dump: { - body: bodyPath, - error: errorPath, - bodyBytes: 0, - bodyTail: "", - }, - }; - } finally { - clearTimeout(timeout); - } -} - -async function liveRevisionStatus(options: HwlabCdOptions, dumpDir: string): Promise> { - const [frontend, api] = await Promise.all([ - fetchLive(options.frontendLiveUrl, dumpDir, "live-16666", Math.min(options.timeoutMs, 10_000)), - fetchLive(options.apiLiveUrl, dumpDir, "live-16667", Math.min(options.timeoutMs, 10_000)), - ]); - return { - status: frontend.ok === true && api.ok === true ? "observed" : "blocked", - endpoints: [frontend, api], - }; -} - -function collectBlockers(parts: { - git?: Record; - desired?: Record; - guard?: Record; - lock?: Record; - live?: Record; - secretRefs?: Record; - controlled?: Record; - dryRun?: Record; -}): Record[] { - const blockers: Record[] = []; - if (parts.git !== undefined) { - if (parts.git.clean === false) blockers.push({ scope: "hwlab-git-clean", summary: "HWLAB repo has local modifications." }); - if (parts.git.onMain === false) blockers.push({ scope: "hwlab-git-main", summary: "HWLAB repo is not on main." }); - if (parts.git.remoteMatches === false) blockers.push({ scope: "hwlab-git-remote", summary: "HWLAB repo origin remote is not pikasTech/HWLAB." }); - if (parts.git.headMatchesOriginMain === false) blockers.push({ scope: "hwlab-git-origin-main", summary: "HWLAB HEAD does not match local origin/main." }); - const worktreeAccess = asRecord(parts.git.worktreeAccess); - if (worktreeAccess?.ok === false) blockers.push({ scope: "hwlab-worktree-permission", summary: "HWLAB CD worktree is not readable/writable by the wrapper." }); - const fetchHead = asRecord(parts.git.fetchHead); - if (fetchHead?.readable === false) blockers.push({ scope: "hwlab-fetch-head-permission", summary: "HWLAB CD FETCH_HEAD is missing or unreadable." }); - } - if (parts.desired !== undefined && parts.desired.status !== "pass") { - blockers.push({ scope: "desired-state", summary: `HWLAB desired-state status is ${String(parts.desired.status)}` }); - } - if (parts.guard !== undefined && parts.guard.status !== "pass") { - blockers.push({ scope: "d601-native-k3s-guard", summary: String(parts.guard.summary ?? "D601 native k3s guard is not pass."), refusal: parts.guard.refusal === true }); - } - if (parts.lock !== undefined && parts.lock.held === true) { - blockers.push({ scope: "cd-lock", summary: `HWLAB DEV CD lock is held by ${String(parts.lock.ownerTaskId ?? parts.lock.holderIdentity ?? "unknown")}.` }); - } - if (parts.live !== undefined && parts.live.status !== "observed") { - blockers.push({ scope: "live-revision", summary: "16666/16667 live revision summary is not fully observable." }); - } - if (parts.secretRefs !== undefined && parts.secretRefs.status === "blocked") { - const secretBlockers = Array.isArray(parts.secretRefs.blockers) ? parts.secretRefs.blockers : []; - blockers.push(...secretBlockers.filter((item): item is Record => typeof item === "object" && item !== null && !Array.isArray(item))); - } - if (parts.controlled !== undefined) { - if (parts.controlled.commandOk === false) blockers.push({ scope: "controlled-dev-cd-dry-run", summary: "HWLAB scripts/dev-cd-apply.mjs --dry-run did not complete successfully." }); - const target = asRecord(parts.controlled.target); - const desiredStateCheck = asRecord(target?.desiredStateCheck); - if (desiredStateCheck !== null && desiredStateCheck.status !== "pass") { - blockers.push({ scope: "controlled-dev-cd-desired-state", summary: `HWLAB dev-cd-apply desired-state check is ${String(desiredStateCheck.status)}.` }); - } - const artifactBoundary = asRecord(target?.artifactBoundary); - if (artifactBoundary !== null && artifactBoundary.status !== "pass") { - blockers.push({ scope: "controlled-dev-cd-artifact-boundary", summary: `HWLAB dev-cd-apply artifact boundary is ${String(artifactBoundary.status)}.` }); - } - } - if (parts.dryRun !== undefined && parts.dryRun.commandOk === false) { - blockers.push({ scope: "apply-dry-run-command", summary: "HWLAB controlled dry-run command failed." }); - } - if (parts.dryRun !== undefined && parts.dryRun.status === "blocked") { - blockers.push({ scope: "apply-dry-run-blockers", summary: "HWLAB controlled dry-run reported open blockers." }); - } - return blockers; -} - -async function status(options: HwlabCdOptions): Promise> { - const dumpDir = rootPath(".state", "hwlab-cd", makeRunId()); +async function runLocalTransport(options: HwlabCdOptions, remoteCommand: string, runId: string): Promise> { + const dumpDir = rootPath(".state", "hwlab-cd", runId); mkdirSync(dumpDir, { recursive: true, mode: 0o700 }); - const repo = resolveHwlabRepo(options.repoPath); - const selected = asRecord(repo.selected); - if (selected === null) { + const result = await runCaptured(["bash", "-c", remoteCommand], repoRoot, dumpDir, "local-remote-wrapper", options.timeoutMs, process.env); + const parsed = parseRemoteStdout(result.stdoutText); + if (parsed === null) { return { ok: false, + env: options.environment, status: "blocked", - environment: options.environment, - dumpDir, - repo, - error: "hwlab-repo-not-found", + error: "remote-empty-or-invalid-json", + remote: { + host: "local-test", + providerId: options.providerId, + transport: "local", + commandShape: commandShape(options.providerId), + commandCalls: ["scripts/dev-cd-apply.mjs"], + stdoutDump: result.dump.stdout, + stderrDump: result.dump.stderr, + exitCode: result.exitCode, + timedOut: result.timedOut, + }, + dumpPath: dumpDir, }; } - const repoPath = String(selected.path); - const [git, desired, guard, controlled, controlledCd, live] = await Promise.all([ - gitSummary(repoPath, dumpDir, Math.min(options.timeoutMs, 15_000)), - desiredStateStatus(repoPath, dumpDir, options.timeoutMs), - nativeK3sGuard(options.kubeconfig, dumpDir, Math.min(options.timeoutMs, 15_000)), - controlledObservability(repoPath, dumpDir, Math.min(options.timeoutMs, 15_000)), - controlledDevCdDryRun(repoPath, options.kubeconfig, dumpDir, Math.min(options.timeoutMs, 30_000)), - liveRevisionStatus(options, dumpDir), - ]); - const [lock, liveWorkloads] = await Promise.all([ - cdLockStatus(options.kubeconfig, guard, dumpDir, Math.min(options.timeoutMs, 15_000)), - liveWorkloadStatus(options.kubeconfig, guard, dumpDir, Math.min(options.timeoutMs, 15_000)), - ]); - const blockers = collectBlockers({ git, desired, guard, lock, live, controlled: controlledCd }); return { - ok: guard.refusal !== true, - status: blockers.length === 0 ? "ready" : "blocked", - environment: options.environment, - dryRun: false, - mutation: false, - dumpDir, - hwlabRepo: { - path: repoPath, - source: selected.source, - controlledEntrypoints: { - devCdApply: join(repoPath, "scripts/dev-cd-apply.mjs"), - desiredStatePlan: join(repoPath, "scripts/deploy-desired-state-plan.mjs"), - }, - }, - git, - desiredState: desired, - controlledDevCdDryRun: controlledCd, - d601NativeK3sGuard: guard, - controlledObservability: controlled, - cdLock: lock, - liveRevisions: live, - liveWorkloads, - blockers, - nextCommands: { - preflight: "bun scripts/cli.ts hwlab cd preflight --env dev", - dryRunApply: "bun scripts/cli.ts hwlab cd apply --env dev --dry-run", - fullDump: `find ${JSON.stringify(dumpDir)} -type f -maxdepth 1 -print`, + ...parsed, + remote: { + host: "local-test", + providerId: options.providerId, + transport: "local", + commandShape: commandShape(options.providerId), + commandCalls: ["scripts/dev-cd-apply.mjs"], + stdoutDump: result.dump.stdout, + stderrDump: result.dump.stderr, + exitCode: result.exitCode, + timedOut: result.timedOut, }, + localDumpPath: dumpDir, }; } -function realApplyRefusal(options: HwlabCdOptions, dumpDir: string, repoPath: string): Record { - const controlledCommand = [ - "node", - "scripts/dev-cd-apply.mjs", - "--apply", - "--confirm-dev", - "--confirmed-non-production", - "--write-report", - "--kubeconfig", - options.kubeconfig, - ]; +async function runFrontendTransport(options: HwlabCdOptions, remoteCommand: string, config: UniDeskConfig, runId: string): Promise> { + const host = options.mainServerHost ?? config.network.publicHost; + const dumpDir = rootPath(".state", "hwlab-cd", runId); + mkdirSync(dumpDir, { recursive: true, mode: 0o700 }); + const session = await loginFrontend(host, config); + const dispatch = await frontendJson(session, "/api/dispatch", { + method: "POST", + body: JSON.stringify({ + providerId: options.providerId, + command: "host.ssh", + payload: { source: "cli-hwlab-cd", mode: "exec", command: remoteCommand, timeoutMs: options.timeoutMs }, + }), + }, 12_000); + const taskId = String(asRecord(dispatch.body)?.taskId ?? ""); + if (!dispatch.ok || taskId.length === 0) { + return { + ok: false, + env: options.environment, + status: "blocked", + error: "provider-dispatch-failed", + remote: { + host: session.baseUrl, + providerId: options.providerId, + transport: "frontend", + commandShape: commandShape(options.providerId), + commandCalls: ["scripts/dev-cd-apply.mjs"], + dispatch, + }, + dumpPath: dumpDir, + nextSafeCommand: "restore UniDesk frontend/backend-core provider dispatch, then rerun bun scripts/cli.ts hwlab cd status --env dev", + }; + } + const wait = await waitForFrontendTask(session, taskId, Math.min(options.timeoutMs + 5000, maxTimeoutMs)); + const task = asRecord((wait as { task?: unknown }).task); + const result = asRecord(task?.result) ?? {}; + const stdout = typeof result.stdout === "string" ? result.stdout : ""; + const stderr = typeof result.stderr === "string" ? result.stderr : ""; + const stdoutDump = join(dumpDir, "frontend-task.stdout.txt"); + const stderrDump = join(dumpDir, "frontend-task.stderr.txt"); + Bun.write(stdoutDump, stdout); + Bun.write(stderrDump, stderr); + const parsed = parseRemoteStdout(stdout); + if (parsed === null) { + return { + ok: false, + env: options.environment, + status: "blocked", + error: "remote-empty-or-invalid-json", + remote: { + host: session.baseUrl, + providerId: options.providerId, + transport: "frontend", + taskId, + taskStatus: task?.status ?? null, + commandShape: commandShape(options.providerId), + commandCalls: ["scripts/dev-cd-apply.mjs"], + stdoutDump, + stderrDump, + exitCode: typeof result.exitCode === "number" ? result.exitCode : null, + }, + dumpPath: dumpDir, + }; + } + return { + ...parsed, + remote: { + host: session.baseUrl, + providerId: options.providerId, + transport: "frontend", + taskId, + taskStatus: task?.status ?? null, + commandShape: commandShape(options.providerId), + commandCalls: ["scripts/dev-cd-apply.mjs"], + stdoutDump, + stderrDump, + exitCode: typeof result.exitCode === "number" ? result.exitCode : null, + }, + localDumpPath: dumpDir, + }; +} + +function realApplyRefusal(options: HwlabCdOptions): Record { return { ok: false, + env: options.environment, status: "refused", - environment: options.environment, + action: "apply", dryRun: false, mutation: false, - dumpDir, - hwlabRepo: { path: repoPath }, + remote: { + host: options.mainServerHost ?? "configured-main-server", + providerId: options.providerId, + transport: options.transport, + commandShape: commandShape(options.providerId), + commandCalls: ["scripts/dev-cd-apply.mjs"], + }, error: "host-commander-only-real-apply", - summary: "UniDesk HWLAB CD wrapper exposes the real DEV apply shape but does not execute it from this runner path. Use dry-run here; host commander must run live apply only after explicit DEV CD authorization.", - hostCommanderOnly: true, - requiredDefaultCommand: "bun scripts/cli.ts hwlab cd apply --env dev --dry-run", + summary: "UniDesk HWLAB CD wrapper does not execute live DEV apply, rollout, live verification, or CD lock mutation from this runner path.", controlledEntrypoint: "scripts/dev-cd-apply.mjs", - controlledCommandShape: controlledCommand, + controlledCommandShape: [ + "node", + "scripts/dev-cd-apply.mjs", + "--apply", + "--confirm-dev", + "--confirmed-non-production", + "--write-report", + "--kubeconfig", + options.kubeconfig, + ], + nextSafeCommand: "bun scripts/cli.ts hwlab cd apply --env dev --dry-run", safety: { - prodTouched: false, kubectlApplyExecuted: false, rolloutExecuted: false, + liveVerifyExecuted: false, + cdLockMutated: false, + prodTouched: false, secretValuesRead: false, - }, - }; -} - -async function apply(options: HwlabCdOptions): Promise> { - const dumpDir = rootPath(".state", "hwlab-cd", makeRunId()); - mkdirSync(dumpDir, { recursive: true, mode: 0o700 }); - const repo = resolveHwlabRepo(options.repoPath); - const selected = asRecord(repo.selected); - if (selected === null) { - return { - ok: false, - status: "blocked", - environment: options.environment, - dryRun: options.dryRun, - mutation: false, - dumpDir, - repo, - error: "hwlab-repo-not-found", - }; - } - const repoPath = String(selected.path); - if (!options.dryRun) return realApplyRefusal(options, dumpDir, repoPath); - - const [git, desired, guard] = await Promise.all([ - gitSummary(repoPath, dumpDir, Math.min(options.timeoutMs, 15_000)), - desiredStateStatus(repoPath, dumpDir, options.timeoutMs), - nativeK3sGuard(options.kubeconfig, dumpDir, Math.min(options.timeoutMs, 15_000)), - ]); - if (guard.refusal === true) { - return { - ok: false, - status: "refused", - environment: options.environment, - dryRun: true, - mutation: false, - dumpDir, - hwlabRepo: { path: repoPath, source: selected.source }, - d601NativeK3sGuard: guard, - error: "native-k3s-guard-refused", - summary: "Refusing HWLAB DEV CD dry-run because kubectl resolved to a forbidden Docker Desktop control plane signal.", - }; - } - - const secretRefs = await requiredSecretRefPreflight(options.kubeconfig, guard, dumpDir, Math.min(options.timeoutMs, 15_000)); - const preCommandBlockers = collectBlockers({ git, desired, guard, secretRefs }); - const dryRun = preCommandBlockers.length === 0 - ? await controlledDevCdDryRun(repoPath, options.kubeconfig, dumpDir, options.timeoutMs) - : { status: "skipped", commandOk: true, reason: "preflight-blockers-before-controlled-dev-cd-dry-run" }; - const blockers = collectBlockers({ git, desired, guard, secretRefs, controlled: dryRun }); - return { - ok: blockers.length === 0, - status: blockers.length === 0 ? "prepared" : "blocked", - environment: options.environment, - dryRun: true, - mutation: false, - dumpDir, - hwlabRepo: { - path: repoPath, - source: selected.source, - controlledEntrypoint: join(repoPath, "scripts/dev-cd-apply.mjs"), - desiredStatePlan: join(repoPath, "scripts/deploy-desired-state-plan.mjs"), - }, - git, - desiredState: desired, - d601NativeK3sGuard: guard, - secretRefPreflight: secretRefs, - controlledDryRun: dryRun, - blockers, - hostCommanderOnlyLiveApply: { - supportedByShapeOnly: true, - notExecuted: true, - commandShape: [ - "node", - "scripts/dev-cd-apply.mjs", - "--apply", - "--confirm-dev", - "--confirmed-non-production", - "--write-report", - "--kubeconfig", - options.kubeconfig, - ], - }, - }; -} - -async function preflight(options: HwlabCdOptions): Promise> { - const dumpDir = rootPath(".state", "hwlab-cd", makeRunId()); - mkdirSync(dumpDir, { recursive: true, mode: 0o700 }); - const repo = resolveHwlabRepo(options.repoPath); - const selected = asRecord(repo.selected); - if (selected === null) { - return { - ok: false, - status: "blocked", - environment: options.environment, - dryRun: true, - mutation: false, - dumpDir, - repo, - error: "hwlab-repo-not-found", - }; - } - const repoPath = String(selected.path); - const [git, desired, guard, live] = await Promise.all([ - gitSummary(repoPath, dumpDir, Math.min(options.timeoutMs, 15_000)), - desiredStateStatus(repoPath, dumpDir, options.timeoutMs), - nativeK3sGuard(options.kubeconfig, dumpDir, Math.min(options.timeoutMs, 15_000)), - liveRevisionStatus(options, dumpDir), - ]); - if (guard.refusal === true) { - return { - ok: false, - status: "refused", - environment: options.environment, - dryRun: true, - mutation: false, - dumpDir, - hwlabRepo: { path: repoPath, source: selected.source }, - git, - desiredState: desired, - d601NativeK3sGuard: guard, - error: "native-k3s-guard-refused", - summary: "Refusing HWLAB DEV CD preflight because kubectl resolved to a forbidden Docker Desktop control plane signal.", - }; - } - const [lock, liveWorkloads, secretRefs] = await Promise.all([ - cdLockStatus(options.kubeconfig, guard, dumpDir, Math.min(options.timeoutMs, 15_000)), - liveWorkloadStatus(options.kubeconfig, guard, dumpDir, Math.min(options.timeoutMs, 15_000)), - requiredSecretRefPreflight(options.kubeconfig, guard, dumpDir, Math.min(options.timeoutMs, 15_000)), - ]); - const preCommandBlockers = collectBlockers({ git, desired, guard, lock, live, secretRefs }); - const controlled = preCommandBlockers.length === 0 - ? await controlledDevCdDryRun(repoPath, options.kubeconfig, dumpDir, Math.min(options.timeoutMs, 30_000)) - : { status: "skipped", commandOk: true, reason: "preflight-blockers-before-controlled-dev-cd-dry-run" }; - const blockers = collectBlockers({ git, desired, guard, lock, live, secretRefs, controlled }); - return { - ok: blockers.length === 0, - status: blockers.length === 0 ? "pass" : "blocked", - environment: options.environment, - dryRun: true, - mutation: false, - dumpDir, - hwlabRepo: { - path: repoPath, - source: selected.source, - controlledEntrypoint: join(repoPath, "scripts/dev-cd-apply.mjs"), - desiredStatePlan: join(repoPath, "scripts/deploy-desired-state-plan.mjs"), - }, - git, - desiredState: desired, - d601NativeK3sGuard: guard, - cdLock: lock, - liveRevisions: live, - liveWorkloads, - secretRefPreflight: secretRefs, - controlledDevCdDryRun: controlled, - blockers, - nextCommands: { - dryRunApply: "bun scripts/cli.ts hwlab cd apply --env dev --dry-run", - fullDump: `find ${JSON.stringify(dumpDir)} -type f -maxdepth 1 -print`, + secretValuesPrinted: false, }, }; } @@ -1121,14 +501,15 @@ export function hwlabHelp(): Record { "bun scripts/cli.ts hwlab cd preflight --env dev", "bun scripts/cli.ts hwlab cd apply --env dev --dry-run", ], - description: "Inspect and prepare HWLAB DEV CD from UniDesk without embedding release kubectl logic. The wrapper calls HWLAB repo-owned scripts and writes full stdout/stderr dumps under .state/hwlab-cd/.", + description: "Bounded UniDesk wrapper for the HWLAB DEV CD path. The default transport dispatches a D601 provider host.ssh command and the D601 side calls HWLAB repo-owned scripts/dev-cd-apply.mjs.", boundary: [ - `KUBECONFIG is forced to ${nativeKubeconfig}`, - "docker-desktop, desktop-control-plane, and 127.0.0.1:11700 are refusal signals", - `default HWLAB CD repo is ${defaultHwlabCdRepoPath}; /home/ubuntu/hwlab is rejected as runner history`, - "status and preflight are read-only and bounded; live apply is host-commander-only and not executed by the dry-run path", - "preflight checks required SecretRef object/key presence without reading or printing Secret values", - "dry-run apply calls HWLAB scripts/dev-cd-apply.mjs --dry-run; live apply shape points at scripts/dev-cd-apply.mjs --apply", + `KUBECONFIG is forced to ${d601NativeKubeconfig}`, + "docker-desktop, desktop-control-plane, and 127.0.0.1:11700 are refusal signals for the explicit D601 target", + `default HWLAB CD repo is ${defaultHwlabCdRepoPath}; ${rejectedRunnerHistoryRepoPath} is rejected as runner history`, + "deploy/deploy.json remains the authoritative desired-state source", + "preflight/apply --dry-run check required SecretRef object/key metadata without reading or printing Secret values", + "status/preflight/apply --dry-run call only HWLAB scripts/dev-cd-apply.mjs with --skip-live-verify; no apply, rollout, lock mutation, live verification, DB write, or secret read is executed", + "real apply is structured refused and must remain with the host commander or unique CD runner", ], }; } @@ -1136,7 +517,10 @@ export function hwlabHelp(): Record { export async function runHwlabCdCommand(args: string[]): Promise> { if (args.length === 0 || args.some(isHelpArg)) return { ok: true, ...hwlabHelp() }; const options = parseOptions(args); - if (options.action === "status") return status(options); - if (options.action === "preflight") return preflight(options); - return apply(options); + if (options.action === "apply" && !options.dryRun) return realApplyRefusal(options); + const runId = makeRunId(); + const remoteCommand = buildRemoteCommand(options, runId); + if (options.transport === "local") return runLocalTransport(options, remoteCommand, runId); + const config = readConfig(); + return runFrontendTransport(options, remoteCommand, config, runId); }