feat: automate HWLAB v02 PR CD monitor
This commit is contained in:
Codex
@@ -215,7 +215,7 @@ UniDesk 是一个以主 server 为统一入口的分布式工作平台；本文
|
||||
- `bun scripts/cli.ts auth-broker contract|health --dry-run|credential-request --dry-run|pr-preflight --dry-run`：查看 Auth Broker P0 Rust skeleton 与 CLI adapter contract，runner 无 `GH_TOKEN`/`GITHUB_TOKEN` 时返回结构化 `auth-missing`/`broker-needed`，不读取或打印 token 值，规则见 `docs/reference/auth-broker.md`。
|
||||
- `bun scripts/cli.ts gh preflight|auth status|issue ...|pr list|files|diff --stat|read|view|preflight|closeout|create|edit|update|comment|merge` / `bun scripts/code-queue-pr-preflight-example.ts`：通过 REST 执行安全 GitHub issue 读写、分页 issue list、inactive issue stale-close、脱敏 auth/status 诊断、body-file Markdown 写入、当日滚动简报时间线 ClaudeQQ 通知、escape 扫描、只读 cleanup-plan 和 #20 board-audit、PR changed-file/stat summary、PR 创建/评论 dry-run、REST-only 低噪声 PR title/body 编辑、PR 收口元数据观察（含 merged/closed 区分与 merge commit）、低噪声 PR 收口 preflight、guarded PR merge 与 runner PR preflight；`gh issue/pr read|view` 支持 `owner/repo#number` shorthand，`--raw|--full` 是显式完整披露别名，`gh pr diff` 仅支持 `--stat` 紧凑 JSON，`gh pr merge` 会先执行 closeout 预检并拒绝非 open、draft、冲突、非 CLEAN、失败或 pending checks 的 PR，规则见 `docs/reference/cli.md` 和 `docs/reference/code-queue-supervision.md`。
|
||||
- `bun scripts/cli.ts commander contract|plan --dry-run|smoke --dry-run|approval request --dry-run|prompt-lint --kind gpt55-pr`：查看 host Codex 指挥官直管微服务 skeleton 的 source/contract、无 daemon smoke 验证计划、.state/commander/ 状态模型、trace summary 聚合、ClaudeQQ 高风险请示草案和 GPT-5.5 PR prompt 边界辅助 lint；当前只返回 dry-run 计划和 backend-core `microservice proxy claudeqq` 授权后候选命令，不接 live bridge、不接管人工指挥官，不发送消息，`prompt-lint` 不作为业务 PR 门禁也不改变 `codex submit` 默认行为，规则见 `docs/reference/host-codex-commander.md`。
|
||||
- `bun scripts/cli.ts hwlab g14 monitor-prs`：一行启动异步监控 HWLAB base=G14 的未合并 PR；可合并时走 UniDesk `gh pr merge` 合并、监控 G14 Tekton/GitOps/Argo DEV rollout，并向 #7 索引的北京日期每日简报追加 CI/CD 耗时与上线 changelog，规则见 `docs/reference/g14.md` 与 `docs/reference/cli.md`。
|
||||
- `bun scripts/cli.ts hwlab g14 monitor-prs [--lane g14|v02]`：一行启动异步监控 HWLAB PR；默认 base=G14 合并后滚动 G14 DEV 并写每日简报，`--lane v02` 监控 base=v0.2，CI/preflight 通过后自动合并、触发 v0.2 CD，并在 PR 下评论 pending/冲突/成功/失败/超时状态，规则见 `docs/reference/g14.md` 与 `docs/reference/cli.md`。
|
||||
- `bun scripts/cli.ts agentrun v01 control-plane status|trigger-current [--dry-run|--confirm]`：通过 G14 route 只读观察或手动触发 AgentRun `v0.1` commit-pinned Tekton/Argo PipelineRun，规则见 `docs/reference/agentrun.md` 与 `docs/reference/cli.md`。
|
||||
- `bun scripts/cli.ts hwlab cd audit --env dev` / `status|preflight|apply --dry-run`：旧 D601 HWLAB DEV CD 指挥侧 wrapper，仅用于显式 legacy 诊断和迁移对照；当前 HWLAB DEV/PROD source/runtime truth 已迁到 G14 `/root/hwlab` 与 G14 k3s/GitOps，规则见 `docs/reference/hwlab.md`。
|
||||
- `bun scripts/cli.ts ci install/status/run/publish-backend-core/publish-user-service/run-dev-e2e/logs`：在 D601 原生 k3s 上安装和运行 Tekton CI，支持每 commit 检查、Code Queue 只读性能门禁、`CI.json` catalog 驱动的 backend-core 与 user-service commit-pinned 镜像发布和手动触发的 `origin/master:deploy.json#environments.dev` 临时 namespace e2e；catalog/producer/consumer 分工见 `docs/reference/cicd-standardization.md`，`run-dev-e2e` 的 Git 控制 runner、短 launcher 和 no-CD 边界见 `docs/reference/dev-ci-runner.md`，Tekton 规则见 `docs/reference/ci.md`。
|
||||
|
||||
@@ -46,7 +46,8 @@ CI/CD、GitOps、rollout、artifact 发布、PR 合并后的 DEV/PROD 滚动、P
|
||||
- `dev-env prewarm-images [--image image] [--provider-id D601] [--no-pull] [--proxy-url URL] [--pull-timeout-ms N] [--dry-run]` 创建异步 job，通过 UniDesk SSH 维护桥在 D601 上把开发底座依赖镜像从 Docker 缓存导入原生 k3s containerd。默认镜像是 `postgres:16-alpine` 和 `rancher/mirrored-library-busybox:1.36.1`，用于避免 `postgres-dev` 与 local-path helper pod 卡在外部 registry 拉取。该命令固定验证 `/etc/rancher/k3s/k3s.yaml` 指向的 native k3s 上下文，并输出 `dev_env_containerd_image_ready=...` 作为成功判据；它不 apply manifest、不修改生产 `unidesk` namespace。
|
||||
- `artifact-registry plan|render|status|health|install|deploy-backend-core|deploy-service` 管理 D601 host-managed CNCF Distribution registry 的声明、安装、只读检查和 pull-only artifact CD。该 registry 固定为 D601 loopback `127.0.0.1:5000`，由 systemd + Docker Compose 管理，位于 native k3s 故障域外；`deploy-service` 只拉取 CI 已发布的 commit-pinned 镜像、retag/recreate 或导入 native k3s，并做 live commit 验证，不构建 runtime source。`deploy-backend-core` 是 deprecated 兼容名，标准 backend-core prod CD 入口是 `deploy apply --env prod --service backend-core`。长期规则见 `docs/reference/artifact-registry.md`。
|
||||
- `commander contract|plan --dry-run|smoke --dry-run|approval request --dry-run|prompt-lint --kind gpt55-pr` 是 host Codex 指挥官直管微服务 skeleton 入口。当前命令返回 `phase=source-contract`、service/API/state/bridge/prompt/trace/#20/#46/ClaudeQQ 审批边界、.state/commander/ 状态模型、dev 无 daemon smoke contract、dry-run 计划和 GPT-5.5 PR prompt 边界辅助 lint，不接 live bridge、不注入 prompt、不发送 ClaudeQQ。`approval request --dry-run` 会生成 200 字以内中文纯文本 ClaudeQQ 审批草案、`notification-path-unavailable` blocker 和授权后唯一可用的 `bun scripts/cli.ts microservice proxy claudeqq /api/push/text --method POST --body-json '<payload>' --raw` 命令；不得提示使用本机 ClaudeQQ skill、powershell 或本地 server。`prompt-lint` 支持 `--prompt-file` 与 `--stdin`，输出 `ok`、`missingClauses`、`riskLevel`、`suggestedPatchSnippet` 且不回显完整 prompt；它是 commander 辅助检查，不是业务 PR 门禁，也不改变 `codex submit` 默认行为。`plan`、`smoke` 与 `approval request` 必须带 `--dry-run`；缺少时返回 `error=dry-run-required`。长期规则见 `docs/reference/host-codex-commander.md`。
|
||||
- `hwlab g14 monitor-prs [--once] [--dry-run] [--interval-seconds N] [--max-cycles N] [--timeout-seconds N]` 是当前 HWLAB G14 PR -> CI/CD -> DEV rollout 的一行式入口。普通调用创建 `.state/jobs/` 异步 job 并立刻返回 `job.id`、`statusCommand` 和 stdout/stderr 路径；后台 worker 每轮通过 UniDesk `gh pr list/preflight/merge` 监控 `pikasTech/HWLAB` base=`G14` 的 open PR，ready 时合并，然后通过 UniDesk `trans G14:k3s` 观察 `hwlab-g14-ci-poll-<short>`、Argo `hwlab-g14-dev` 和 DEV `/health/live`，直到 DEV `Synced/Healthy` 且 Deployment/StatefulSet ready；历史 `Completed` smoke/debug pod 不作为 rollout blocker。每次成功 DEV rollout 后，worker 会定位或创建 #7“指挥简报索引”中的北京日期每日简报 issue，并追加 CI/CD 耗时、CI/CD 关键指标、语义化上线 changelog、自动 diff 摘要、PipelineRun、GitOps revision 和 DEV 验证摘要；关键指标来自 G14 Tekton TaskRun results，固定包含 `lazy build reused: x/y`、reused services、rebuild services 和每个 service 的独立耗时/状态/backend，用于观察 lazy build 机制效果。语义化 changelog 优先从 PR body 的 `## 修改`/`## 变更`/`## Changelog` 等段落提取，diff 摘要只作为文件和统计证据保留，不替代 changelog。也可用 `hwlab g14 record-rollout --pr <number> --source-commit <sha>` 手动补记，手动补记同样会按 PipelineRun 采集 TaskRun 指标。状态指针按用途分离：长期监控只写 `.state/hwlab-g14/latest-monitor-job.json`，`--once` 写 `latest-once-job.json`，`--dry-run` 写 `latest-dry-run-job.json`，`--once --dry-run` 写 `latest-once-dry-run-job.json`，避免一次性收口覆盖持续监控入口。`--once --dry-run` 只做单轮监控和 merge plan，不写 GitHub、不等待 rollout。该命令禁止使用原生 `gh` 或手拼 GitHub 请求；如果 UniDesk `gh` 子命令字段或行为不够，必须先改进 `scripts/src/gh.ts` 后再使用。
|
||||
- `hwlab g14 monitor-prs [--lane g14|v02] [--once] [--dry-run] [--interval-seconds N] [--max-cycles N] [--timeout-seconds N]` 是当前 HWLAB G14 PR -> CI/CD -> DEV rollout 的一行式入口。普通调用创建 `.state/jobs/` 异步 job 并立刻返回 `job.id`、`statusCommand` 和 stdout/stderr 路径；后台 worker 每轮通过 UniDesk `gh pr list/preflight/merge` 监控 `pikasTech/HWLAB` base=`G14` 的 open PR，ready 时合并，然后通过 UniDesk `trans G14:k3s` 观察 `hwlab-g14-ci-poll-<short>`、Argo `hwlab-g14-dev` 和 DEV `/health/live`，直到 DEV `Synced/Healthy` 且 Deployment/StatefulSet ready；历史 `Completed` smoke/debug pod 不作为 rollout blocker。每次成功 DEV rollout 后，worker 会定位或创建 #7“指挥简报索引”中的北京日期每日简报 issue，并追加 CI/CD 耗时、CI/CD 关键指标、语义化上线 changelog、自动 diff 摘要、PipelineRun、GitOps revision 和 DEV 验证摘要；关键指标来自 G14 Tekton TaskRun results，固定包含 `lazy build reused: x/y`、reused services、rebuild services 和每个 service 的独立耗时/状态/backend，用于观察 lazy build 机制效果。语义化 changelog 优先从 PR body 的 `## 修改`/`## 变更`/`## Changelog` 等段落提取，diff 摘要只作为文件和统计证据保留，不替代 changelog。也可用 `hwlab g14 record-rollout --pr <number> --source-commit <sha>` 手动补记，手动补记同样会按 PipelineRun 采集 TaskRun 指标。G14 状态指针按用途分离：长期监控只写 `.state/hwlab-g14/latest-monitor-job.json`，`--once` 写 `latest-once-job.json`，`--dry-run` 写 `latest-dry-run-job.json`，`--once --dry-run` 写 `latest-once-dry-run-job.json`，避免一次性收口覆盖持续监控入口。`--once --dry-run` 只做单轮监控和 merge plan，不写 GitHub、不等待 rollout。该命令禁止使用原生 `gh` 或手拼 GitHub 请求；如果 UniDesk `gh` 子命令字段或行为不够，必须先改进 `scripts/src/gh.ts` 后再使用。
|
||||
- `hwlab g14 monitor-prs --lane v02` 是 HWLAB `v0.2` 的 PR -> CI -> CD 自动化入口。它只监控 base=`v0.2` 的 open PR：每轮先用 UniDesk `gh pr preflight` 读取 GitHub CI/checks、mergeability 和冲突状态；pending 时在 PR 下写等待评论，blocked/conflict 时写阻塞评论；ready 时先确认 v0.2 lane 没有运行中的 PipelineRun，再用 UniDesk `gh pr merge` 合并，随后执行受控 `control-plane trigger-current --lane v02 --confirm --wait`、轮询定点 `control-plane status --lane v02 --source-commit <merge-sha>`，必要时执行 `git-mirror flush --confirm --wait`。不管 CD 成功、失败或超时，都在原 PR 下用 `gh pr comment create --body-file` 追加语义化状态，正文固定包含起止时间、总耗时、冲突状态、CI/preflight conclusion、source commit、PipelineRun、targetValidation、Argo/webAssets 和 git mirror pendingFlush/githubInSync。评论去重状态写入 `.state/hwlab-g14/v02-pr-comment-signatures.json`，同一状态签名不会重复刷评论；v0.2 monitor 指针使用 `.state/hwlab-g14/latest-v02-monitor-job.json`、`latest-v02-once-job.json`、`latest-v02-dry-run-job.json` 和 `latest-v02-once-dry-run-job.json`，不会覆盖默认 G14 monitor 指针。`--lane v02 --once --dry-run` 只做单轮 preflight/merge/CD/comment plan，不写 GitHub、不触发 CD。
|
||||
- `agentrun v01 control-plane status|trigger-current|refresh [--dry-run|--confirm]` 是 AgentRun `v0.1` 在 G14 k3s 的受控 Tekton/Argo 入口。`status` 只读汇总固定 source worktree commit、对应 commit-pinned PipelineRun、GitOps latest、Argo Application、`agentrun-v01` manager source commit、`planArtifacts.summary`、env image result 和 git mirror 摘要，并报告 manager/Argo/GitOps 是否对齐当前 source commit。默认输出是 compact commander 视图：`summary` 给出 source、PipelineRun、Argo、manager image、git mirror 和 `aligned` 结论；`timings` 给出 `sourceMs`、`runtimeMs`、`gitMirrorMs` 和 `totalMs`；远端 stdout/stderr tail 默认省略，失败时仍展开必要 tail，完整 tail 用 `--full`，原始 git mirror cache 用 `--raw`。`status` 聚合 source 后会并行读取 runtime 和 git mirror，并向 stderr 输出 `agentrun.control-plane.status.progress` JSON 事件，覆盖 `source`、`runtime`、`git-mirror` 的 started/succeeded/failed 和 elapsedMs，避免 10s 以上状态聚合期间无可见进展；`trigger-current` 先快进 `G14:/root/agentrun-v01` 到 `origin/v0.1`，检查 `devops-infra` mirror 的 `localV01` 是否等于目标 source commit，必要时先执行受控 mirror sync，再创建 `agentrun-v01-ci-<short12>` PipelineRun。confirmed trigger 只提交 CI/CD 工作并返回后续 `status` 命令，不等待完整 PipelineRun；同名 PipelineRun 运行中或已成功时拒绝重复触发，只允许失败态重建或首次创建。`refresh` 只对 `argocd/agentrun-g14-v01` 执行 hard refresh，用于 GitOps promotion 已完成但 Argo 仍停留旧 revision 时的受控同步入口；它不直接 patch runtime workload。AgentRun 运行时和 SPEC 事实来源仍在 AgentRun 仓库，UniDesk 只维护受控运维入口。
|
||||
- `agentrun v01 git-mirror status|sync|flush [--dry-run|--confirm]` 是 AgentRun `v0.1` 使用 `devops-infra` git mirror/relay 的受控维护入口。`status` 默认返回 read/write URL、`localV01`、`githubV01`、`localGitops`、`githubGitops`、`pendingFlush`、`githubInSync` 和 exact full-SHA shallow fetch 摘要，不默认展开完整 cache stdout；需要探测 tail 时用 `--full`，需要原始 cache 输出时用 `--raw`。`sync` 创建 manual Job，把 GitHub `v0.1` 和 `v0.1-gitops` refs 拉入 `/cache/pikasTech/agentrun.git`；`flush` 把本地 `v0.1-gitops` 快进推回 GitHub。confirmed `sync`/`flush` 默认创建 `.state/jobs/` 异步 job 并立刻返回 `job.id`、`statusCommand` 和日志路径；只有现场同步调试才显式加 `--wait`。该入口与 HWLAB v0.2 mirror 共用 `devops-infra` 服务和 cache PVC，但 repo path、refs、status 文件和 CLI 命令彼此独立。
|
||||
- `hwlab g14 control-plane status|apply --lane v02 [--dry-run|--confirm]` 是 HWLAB `v0.2` 加法 lane 的受控 Tekton/Argo 控制面维护入口，source commit 只来自 G14 专用 bare repo `/root/hwlab-v02-cicd.git` 的 `refs/remotes/origin/v0.2`；`/root/hwlab-v02` 只作为人工开发和短连接源码工具 workspace 被观测，dirty/stale 状态必须输出为 isolated warning 而不能阻塞 CI/CD。该入口面向 branch `v0.2`、namespace `hwlab-ci` 和 Argo application `hwlab-g14-v02`；默认 `status` 只读汇总最新 source head 的 pipeline、RBAC/ServiceAccount、Argo、当前 commit PipelineRun、当前 PipelineRun 的 TaskRun 条件摘要、最近 PipelineRun 摘要、活跃 PipelineRun、遗留 v02 CronJob 清理状态、commit alignment，以及 19666/19667 的 Cloud Web 静态资源和 API live 探针。分支被后续提交推进后，要复查已完成 run 时使用 `status --lane v02 --pipeline-run hwlab-v02-ci-poll-<short-sha>`；已知完整 source SHA 但不想依赖最新 head 时使用 `status --lane v02 --source-commit <full-sha>`。定点 `status` 输出 `statusTarget.mode` 和 `targetValidation`，只检查指定 PipelineRun/source commit 的证据；`targetValidation.state=passed` 表示该目标已满足 PipelineRun succeeded、Argo `Synced/Healthy`、19666/19667 探针、Git mirror flushed，并且该 run 的 `planArtifacts.rolloutServices` 运行时 source commit 对齐；`planArtifacts.reusedServices` 作为 runtime/provenance 证据呈现，但不能被强制要求等于目标 source commit。`targetValidation.state=superseded` 表示该目标已成功且 runtime 已被同一分支后续成功 PipelineRun 取代，`falseGreenGuard` 在该状态下应标为 superseded/not-applicable。两种状态都不得因为 `origin/v0.2` 后续推进而把历史 run 判为失败；默认不带定点参数时仍严格判定最新 source head alignment。TaskRun 摘要的 `performance` 字段会把超过 120s 的 build TaskRun 标为慢任务、超过 180s 标为 critical warning，用于暴露 env reuse/git mirror 命中率回归，但不作为阻断门禁；CI/CD 性能验收应同时看 `planArtifacts.summary`、`taskRuns.performance.warningCount` 和 PipelineRun duration，纯 CLI/文档或无 runtime 重建需求的后续提交应稳定表现为 `build=0 reuse=<service-count>` 且无 build TaskRun warning，首次引入或切换 env image 时允许只构建必要 env image 一次。`webAssets` 必须直接给出 `readonly-rpc` 删除、sidebar/workspace/event panel 关键 CSS、`/app.js` 是否可读取和字节数、`/health/live` 与 API revision；`apiRevision` 是 cloud-api 服务自身 revision，Cloud Web 静态资源变更时允许它与 source commit 不同，不能把这种差异误判成 Cloud Web 未发布。默认只读取必要字段，禁止把完整 PipelineRun spec、Tekton 内联脚本、历史大对象或整份 CSS/HTML/JS 展开到默认输出；`apply` 先自动 fetch `/root/hwlab-v02-cicd.git` 并从 commit-pinned detached worktree 执行 render check，再经 `G14:k3s` server-side apply `tekton-v02/rbac.yaml`、`pipeline.yaml`、`argocd/project.yaml` 和 `argocd/application-v02.yaml`，confirmed apply 会删除遗留 v02 CronJob，但不会应用 runtime-v02 workload、Secret 或数据迁移。
|
||||
@@ -99,7 +100,7 @@ CI/CD、GitOps、rollout、artifact 发布、PR 合并后的 DEV/PROD 滚动、P
|
||||
- `codex interrupt|cancel <taskId>` 通过 Code Queue 私有代理请求中断；running/judging 任务会请求 D601 当前 agent run 停止，queued/retry_wait 任务的取消也必须保持与 WebUI 相同代理路径，返回有界 task 摘要和后续查询命令。任何需要接触 active run 的动作仍属于 D601 执行面。
|
||||
- Code Queue 多队列 lane 由 `codex` 命令命名空间管理：`queues [--full|--all] [--limit N] [--page N|--offset N]` 列表、`queue create <queueId>` 创建、`queue merge <sourceQueueId> --into <targetQueueId>` 合并、`move <taskId> --queue <queueId>` 迁移；这些队列管理入口默认由主 server `code-queue-mgr` 直管 PostgreSQL，仍通过稳定 `code-queue` 用户服务代理路径访问。`codex queues` 默认只返回 active/nonempty/unread/runnable queue 摘要、activity、commanderConcurrency、全局 counts 和 execution diagnostics；`--full` 或 `--all` 只切换为完整队列行视图的一页，仍受 `--limit`/`--page`/`--offset` 分页约束，不再默认携带 deprecated full array。summary 和 full 的稳定机读路径都是 `.data.queues.items[]`，全局元数据固定在 `.data.queues.commanderConcurrency`、`.data.queues.activity`、`.data.queues.counts`、`.data.queues.executionDiagnostics`、`.data.queues.activeTaskIds` 和 `.data.queues.queuedTaskIds`；需要完整 upstream 时使用输出中的 raw command。`commanderConcurrency.activeRunnerCount` / `activity.effectiveActiveTaskCount` 是指挥官并发判断的有效活跃数，`schedulerLocalActiveQueueCount`/`activeQueueIds` 只描述本地 scheduler active-run slots，不能覆盖数据库 running 计数或 heartbeat-fresh runner 计数。旧 full 顶层数组语义已作为 deprecated 兼容信息记录，不再作为 `.data.queues` 主形态。同一个 queue 内部串行执行，不同 queue 之间并行执行。迁移只允许尚未被 scheduler claim 的 `queued`/`retry_wait` 任务，必须满足 `startedAt=null`、`currentAttempt=0` 且没有 active thread/turn；已进入 `running`/`judging` 或已有 claim 标记的任务返回 409，不得被 move/merge 回写成 queued。合并会移动可迁移任务归属并自动删除源 queue 记录，只保留合并后的目标 queue；若 source 或 target queue 存在 active/claimed 任务，合并整体返回 409。合并后的目标 queue 按任务原 `queueEnteredAt`/`createdAt` 时间顺序串行，成功迁移 queued/retry_wait 任务后由 D601 scheduler 轮询推进。
|
||||
- 所有 `codex` 查询和管理命令必须走与 WebUI 相同的 backend-core 私有代理路径 `/api/microservices/code-queue/proxy/...`；CLI 不得为了提交、移动、中断、取消或队列管理直接调用 D601 内部 Service、数据库、pod curl 或 k3sctl scheduler 子服务。若该路径失败，应先修复 CLI/backend/provider tunnel 链路，而不是绕过控制面。
|
||||
- `job list [--limit N] [--include-command]` 与 `job status <jobId|latest> [--tail-bytes N]` 查询 `.state/jobs/` 文件系统状态，是异步命令的可观测入口。`job list` 默认只返回最新 50 条摘要，并为已知异步工作流返回轻量 `progress.summary` 与后续查询命令；`job status` 默认返回结构化 `progress`、stdout/stderr 末尾 12000 字节、`tailPolicy` 与完整日志路径。已知工作流应从有界日志尾部抽取阶段、关键对象名和下一步命令，避免为了判断当前阶段而手工打开完整 stdout/stderr。`hwlab_g14_v02_trigger_current` 的 progress 必须暴露 trigger 阶段、source commit 和 PipelineRun；`hwlab_g14_git_mirror_sync|flush` 与 `agentrun_v01_git_mirror_sync|flush` 的 progress 必须暴露 sync/flush 状态、Job 名、pendingFlush 与 fetch/push/total/SSH timing，并给出对应 repo 的 mirror status 命令。
|
||||
- `job list [--limit N] [--include-command]` 与 `job status <jobId|latest> [--tail-bytes N]` 查询 `.state/jobs/` 文件系统状态，是异步命令的可观测入口。`job list` 默认只返回最新 50 条摘要，并为已知异步工作流返回轻量 `progress.summary` 与后续查询命令；`job status` 默认返回结构化 `progress`、stdout/stderr 末尾 12000 字节、`tailPolicy` 与完整日志路径。已知工作流应从有界日志尾部抽取阶段、关键对象名和下一步命令，避免为了判断当前阶段而手工打开完整 stdout/stderr。`hwlab_g14_v02_trigger_current` 的 progress 必须暴露 trigger 阶段、source commit 和 PipelineRun；`hwlab_g14_v02_pr_monitor` 的 progress 必须暴露 preflight、merge、source-head、lane-idle、cd-trigger、cd-status、git-mirror-flush 和 pr-comment 阶段，以及 PR、source commit、PipelineRun、targetValidation/pendingFlush 摘要；`hwlab_g14_git_mirror_sync|flush` 与 `agentrun_v01_git_mirror_sync|flush` 的 progress 必须暴露 sync/flush 状态、Job 名、pendingFlush 与 fetch/push/total/SSH timing，并给出对应 repo 的 mirror status 命令。
|
||||
- `debug health`、`debug dispatch` 与 `debug task` 走真实内部 core、WebSocket、数据库、provider、系统指标、Docker 状态和 Host SSH 维护桥流程，只用于开发调试，不写入 `TEST.md` 的正式验收步骤。
|
||||
- `e2e run [--only pattern[,pattern...]] [--skip pattern[,pattern...]]` 使用 publicHost 派生的公开 production frontend/dev frontend/provider ingress URL，并通过 Docker 内网验证 core API、PostgreSQL、provider self-connection、系统指标曲线、Docker 状态快照、provider.upgrade 预检和 Playwright 前端页面，是交付前的自动化 E2E 门禁；CLI 默认输出 check 状态摘要，完整诊断写入 `resultPath`，日常迭代应优先用 `--only` / `--skip` 跑最小必要集合。
|
||||
|
||||
|
||||
@@ -67,9 +67,9 @@ Master-side FRP server maintenance for HWLAB public ports is documented in `docs
|
||||
|
||||
The `v0.2` CI/CD integration must be additive: add a manual UniDesk trigger, dedicated CI/CD source repo, `devops-infra` git mirror/relay, GitOps desired-state lane, Argo CD Application, namespace resources, artifact catalog, and `deploy.json` environment only when they target `v0.2`/`hwlab-v02` explicitly. Do not add a `v0.2` branch poller or retarget the existing `G14` poller, DEV/PROD Argo Applications, DEV/PROD runtime paths, or existing namespace resources to bootstrap `v0.2`.
|
||||
|
||||
The `devops-infra` git mirror/relay remains manual and CLI-controlled, not CronJob-driven. The standard `v0.2` CI/CD trigger is `bun scripts/cli.ts hwlab g14 control-plane trigger-current --lane v02 --confirm`; this command must fetch `/root/hwlab-v02-cicd.git`, resolve the current `origin/v0.2` source commit, check the mirror's `localV02` ref before creating the PipelineRun, run one bounded manual `git-mirror sync` Job when the mirror is stale, and only continue after the mirror ref matches the current source commit. Use `hwlab g14 git-mirror sync --confirm` directly only for explicit mirror maintenance or diagnosis.
|
||||
The `devops-infra` git mirror/relay remains manual and CLI-controlled, not CronJob-driven. The standard `v0.2` delivery trigger is `bun scripts/cli.ts hwlab g14 monitor-prs --lane v02`: it watches base=`v0.2` PRs, waits for GitHub preflight/CI readiness, auto-merges only ready and non-conflicting PRs, then drives the same controlled CD path and comments pending/blocked/succeeded/failed/timeout state back to the PR. The lower-level `bun scripts/cli.ts hwlab g14 control-plane trigger-current --lane v02 --confirm` remains the manual recovery or diagnosis entry; it must fetch `/root/hwlab-v02-cicd.git`, resolve the current `origin/v0.2` source commit, check the mirror's `localV02` ref before creating the PipelineRun, run one bounded manual `git-mirror sync` Job when the mirror is stale, and only continue after the mirror ref matches the current source commit. Use `hwlab g14 git-mirror sync --confirm` directly only for explicit mirror maintenance or diagnosis.
|
||||
|
||||
After a `v0.2` PipelineRun completes, treat runtime rollout and remote GitOps persistence as two separate checks. `hwlab g14 control-plane status --lane v02` is the runtime check: it must show the expected source commit, PipelineRun completed, Argo `Synced/Healthy`, public 19666/19667 probes passing, and Cloud Web asset probes such as `/app.js` readable. `hwlab g14 git-mirror status` is the persistence check: `cache.summary.pendingFlush` must be false and `cache.summary.githubInSync` true before declaring GitOps fully flushed back to GitHub. If runtime is healthy but `pendingFlush=true`, run `bun scripts/cli.ts hwlab g14 git-mirror flush --confirm` and poll the returned job with `bun scripts/cli.ts job status <jobId> --tail-bytes 12000`; do not replace this with raw `kubectl`, native `git push`, or a long SSH wait.
|
||||
After a `v0.2` PipelineRun completes, treat runtime rollout and remote GitOps persistence as two separate checks. `hwlab g14 control-plane status --lane v02` is the runtime check: it must show the expected source commit, PipelineRun completed, Argo `Synced/Healthy`, public 19666/19667 probes passing, and Cloud Web asset probes such as `/app.js` readable. `hwlab g14 git-mirror status` is the persistence check: `cache.summary.pendingFlush` must be false and `cache.summary.githubInSync` true before declaring GitOps fully flushed back to GitHub. The PR monitor performs this flush automatically for its own merged PRs and records the result in the PR comment. Manual operators should run `bun scripts/cli.ts hwlab g14 git-mirror flush --confirm` and poll the returned job with `bun scripts/cli.ts job status <jobId> --tail-bytes 12000` only when they used lower-level manual trigger/status paths or when the monitor reports a flush failure; do not replace this with raw `kubectl`, native `git push`, or a long SSH wait.
|
||||
|
||||
When closing an issue against a specific completed `v0.2` PipelineRun, use targeted status instead of the latest-head status if `origin/v0.2` has already advanced through a parallel task:
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import { gitMirrorFlushJobManifest, gitMirrorStatusSummary, gitMirrorSyncJobManifest, gitMirrorV02SyncRequirement, hwlabG14Help, hwlabG14MonitorStateFileName, parseGitMirrorStatusRefs, parsePipelineTaskRunMetrics, rolloutRecordBody, semanticChangelogBullets, v02CommitAlignment, v02ControlPlaneRenderScript, v02FalseGreenGuard, v02PipelineServiceIds, v02TaskRunPerformanceSummary } from "./src/hwlab-g14";
|
||||
import { gitMirrorFlushJobManifest, gitMirrorStatusSummary, gitMirrorSyncJobManifest, gitMirrorV02SyncRequirement, hwlabG14Help, hwlabG14MonitorStateFileName, parseGitMirrorStatusRefs, parsePipelineTaskRunMetrics, rolloutRecordBody, semanticChangelogBullets, v02CommitAlignment, v02ControlPlaneRenderScript, v02FalseGreenGuard, v02PipelineServiceIds, v02PrAutomationCommentBody, v02TaskRunPerformanceSummary } from "./src/hwlab-g14";
|
||||
|
||||
function assertCondition(condition: unknown, message: string, detail: unknown = {}): void {
|
||||
if (!condition) throw new Error(`${message}: ${JSON.stringify(detail)}`);
|
||||
@@ -24,6 +24,13 @@ assertCondition(
|
||||
hwlabG14MonitorStateFileName({ once: true, dryRun: true }) === "latest-once-dry-run-job.json",
|
||||
"once dry-runs need a distinct pointer for low-noise diagnostics",
|
||||
);
|
||||
assertCondition(
|
||||
hwlabG14MonitorStateFileName({ lane: "v02", once: false, dryRun: false }) === "latest-v02-monitor-job.json"
|
||||
&& hwlabG14MonitorStateFileName({ lane: "v02", once: true, dryRun: false }) === "latest-v02-once-job.json"
|
||||
&& hwlabG14MonitorStateFileName({ lane: "v02", once: false, dryRun: true }) === "latest-v02-dry-run-job.json"
|
||||
&& hwlabG14MonitorStateFileName({ lane: "v02", once: true, dryRun: true }) === "latest-v02-once-dry-run-job.json",
|
||||
"v0.2 PR monitor state pointers must not overwrite the legacy G14 monitor pointers",
|
||||
);
|
||||
const hwlabHelpUsage = Array.isArray(hwlabG14Help().usage) ? hwlabG14Help().usage.map(String) : [];
|
||||
assertCondition(
|
||||
hwlabHelpUsage.some((line) => line.includes("control-plane status --lane v02 --pipeline-run"))
|
||||
@@ -31,6 +38,60 @@ assertCondition(
|
||||
"v0.2 control-plane status help must expose targeted PipelineRun/source-commit inspection",
|
||||
hwlabHelpUsage,
|
||||
);
|
||||
assertCondition(
|
||||
hwlabHelpUsage.some((line) => line.includes("monitor-prs --lane v02"))
|
||||
&& JSON.stringify(hwlabG14Help()).includes("v02-pr-comment-signatures.json"),
|
||||
"v0.2 PR monitor help must expose the auto CI/CD lane and dedupe comment state",
|
||||
hwlabG14Help(),
|
||||
);
|
||||
|
||||
const v02CommentBody = v02PrAutomationCommentBody({
|
||||
pr: {
|
||||
number: 848,
|
||||
title: "迁移：v0.2 PR 合并后自动触发 CD",
|
||||
url: "https://github.com/pikasTech/HWLAB/pull/848",
|
||||
baseRefName: "v0.2",
|
||||
headRefName: "fix/v02-auto-cd",
|
||||
},
|
||||
phase: "cd-passed",
|
||||
state: "cd-succeeded",
|
||||
startedAt: "2026-06-04T00:00:00.000Z",
|
||||
observedAt: "2026-06-04T00:07:30.000Z",
|
||||
elapsedSeconds: 450,
|
||||
preflight: {
|
||||
conclusion: "ready",
|
||||
readyForCommanderMerge: true,
|
||||
mergeable: "MERGEABLE",
|
||||
mergeStateStatus: "CLEAN",
|
||||
blockers: [],
|
||||
pending: [],
|
||||
},
|
||||
sourceCommit: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
|
||||
pipelineRun: "hwlab-v02-ci-poll-aaaaaaaaaaaa",
|
||||
cd: {
|
||||
pipelineStatus: "True",
|
||||
pipelineReason: "Succeeded",
|
||||
targetValidationState: "passed",
|
||||
argoSync: "Synced",
|
||||
argoHealth: "Healthy",
|
||||
webAssetsOk: true,
|
||||
pendingFlush: false,
|
||||
githubInSync: true,
|
||||
rolloutServices: ["hwlab-cloud-api"],
|
||||
buildServices: [],
|
||||
reusedServices: ["hwlab-cloud-web"],
|
||||
},
|
||||
flush: { ok: true },
|
||||
});
|
||||
assertCondition(
|
||||
v02CommentBody.includes("elapsed: 7m30s")
|
||||
&& v02CommentBody.includes("conflict: `clear-or-unknown`")
|
||||
&& v02CommentBody.includes("hwlab-v02-ci-poll-aaaaaaaaaaaa")
|
||||
&& v02CommentBody.includes("pendingFlush=`false`")
|
||||
&& v02CommentBody.includes("targetValidation: `passed`"),
|
||||
"v0.2 PR automation comments must include duration, conflict state, PipelineRun, target validation, and git mirror flush status",
|
||||
v02CommentBody,
|
||||
);
|
||||
|
||||
const gitMirrorJob = gitMirrorSyncJobManifest("git-mirror-hwlab-sync-manual-test");
|
||||
assertCondition(gitMirrorJob.kind === "Job", "git mirror sync must be a manual Job, not a CronJob", gitMirrorJob);
|
||||
|
||||
+714
-23
@@ -1,5 +1,5 @@
|
||||
import { existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { dirname, join } from "node:path";
|
||||
import { createHash } from "node:crypto";
|
||||
import { repoRoot, rootPath, type Config } from "./config";
|
||||
import { runCommand } from "./command";
|
||||
@@ -65,7 +65,10 @@ const BEIJING_OFFSET_MS = 8 * 60 * 60 * 1000;
|
||||
const V02_BUILD_TASKRUN_WARNING_SECONDS = 120;
|
||||
const V02_BUILD_TASKRUN_CRITICAL_SECONDS = 180;
|
||||
|
||||
export type G14MonitorLane = "g14" | "v02";
|
||||
|
||||
interface G14MonitorOptions {
|
||||
lane: G14MonitorLane;
|
||||
intervalSeconds: number;
|
||||
maxCycles: number;
|
||||
once: boolean;
|
||||
@@ -149,7 +152,7 @@ interface ShellSection {
|
||||
exitCode: number | null;
|
||||
}
|
||||
|
||||
interface OpenPullRequest {
|
||||
export interface OpenPullRequest {
|
||||
number: number;
|
||||
title?: string;
|
||||
url?: string;
|
||||
@@ -184,22 +187,49 @@ interface CiPipelineMetrics {
|
||||
rawText?: string;
|
||||
}
|
||||
|
||||
export function hwlabG14MonitorStateFileName(options: Pick<G14MonitorOptions, "once" | "dryRun">): string {
|
||||
if (options.once && options.dryRun) return "latest-once-dry-run-job.json";
|
||||
if (options.once) return "latest-once-job.json";
|
||||
if (options.dryRun) return "latest-dry-run-job.json";
|
||||
return "latest-monitor-job.json";
|
||||
export interface V02PrCommentInput {
|
||||
pr: OpenPullRequest;
|
||||
phase: string;
|
||||
state: "waiting-ci" | "blocked" | "merge-failed" | "cd-started" | "cd-succeeded" | "cd-failed" | "cd-timeout" | "cd-blocked";
|
||||
startedAt: string;
|
||||
observedAt: string;
|
||||
elapsedSeconds: number | null;
|
||||
preflight?: Record<string, unknown>;
|
||||
merge?: Record<string, unknown>;
|
||||
sourceCommit?: string | null;
|
||||
pipelineRun?: string | null;
|
||||
cd?: Record<string, unknown>;
|
||||
flush?: Record<string, unknown>;
|
||||
dryRun?: boolean;
|
||||
message?: string;
|
||||
}
|
||||
|
||||
function hwlabG14MonitorStateRole(options: Pick<G14MonitorOptions, "once" | "dryRun">): string {
|
||||
if (options.once && options.dryRun) return "once-dry-run";
|
||||
if (options.once) return "once";
|
||||
if (options.dryRun) return "dry-run-monitor";
|
||||
return "monitor";
|
||||
export function hwlabG14MonitorStateFileName(options: Pick<G14MonitorOptions, "once" | "dryRun"> & { lane?: G14MonitorLane }): string {
|
||||
const prefix = options.lane === "v02" ? "latest-v02-" : "latest-";
|
||||
if (options.once && options.dryRun) return options.lane === "v02" ? `${prefix}once-dry-run-job.json` : "latest-once-dry-run-job.json";
|
||||
if (options.once && options.lane === "v02") return `${prefix}once-job.json`;
|
||||
if (options.once) return "latest-once-job.json";
|
||||
if (options.dryRun) return `${prefix}dry-run-job.json`;
|
||||
return `${prefix}monitor-job.json`;
|
||||
}
|
||||
|
||||
function hwlabG14MonitorStateRole(options: Pick<G14MonitorOptions, "once" | "dryRun"> & { lane?: G14MonitorLane }): string {
|
||||
const lanePrefix = options.lane === "v02" ? "v02-" : "";
|
||||
if (options.once && options.dryRun) return `${lanePrefix}once-dry-run`;
|
||||
if (options.once) return `${lanePrefix}once`;
|
||||
if (options.dryRun) return `${lanePrefix}dry-run-monitor`;
|
||||
return `${lanePrefix}monitor`;
|
||||
}
|
||||
|
||||
function parseMonitorLane(args: string[]): G14MonitorLane {
|
||||
const lane = optionValue(args, "--lane") ?? "g14";
|
||||
if (lane !== "g14" && lane !== "v02") throw new Error("monitor-prs --lane must be g14 or v02");
|
||||
return lane;
|
||||
}
|
||||
|
||||
function parseOptions(args: string[]): G14MonitorOptions {
|
||||
return {
|
||||
lane: parseMonitorLane(args),
|
||||
intervalSeconds: positiveIntegerOption(args, "--interval-seconds", DEFAULT_INTERVAL_SECONDS, 86400),
|
||||
maxCycles: positiveIntegerOption(args, "--max-cycles", DEFAULT_MAX_CYCLES, 100000),
|
||||
once: args.includes("--once"),
|
||||
@@ -519,12 +549,16 @@ function isCommandSuccess(result: CommandJsonResult): boolean {
|
||||
return dataOk !== false;
|
||||
}
|
||||
|
||||
function extractPullRequests(result: CommandJsonResult): OpenPullRequest[] {
|
||||
function monitorBaseBranch(lane: G14MonitorLane): string {
|
||||
return lane === "v02" ? V02_SOURCE_BRANCH : G14_SOURCE_BRANCH;
|
||||
}
|
||||
|
||||
function extractPullRequests(result: CommandJsonResult, baseBranch = G14_SOURCE_BRANCH): OpenPullRequest[] {
|
||||
const prs = nested(result.parsed, ["data", "pullRequests"]);
|
||||
if (!Array.isArray(prs)) return [];
|
||||
return prs
|
||||
.map((item) => record(item))
|
||||
.filter((item) => item.baseRefName === G14_SOURCE_BRANCH || record(item.base).ref === G14_SOURCE_BRANCH)
|
||||
.filter((item) => item.baseRefName === baseBranch || record(item.base).ref === baseBranch)
|
||||
.map((item) => ({
|
||||
number: Number(item.number),
|
||||
title: typeof item.title === "string" ? item.title : undefined,
|
||||
@@ -550,6 +584,10 @@ function printProgressEvent(event: string, data: Record<string, unknown> = {}):
|
||||
process.stderr.write(`${JSON.stringify({ event, at: new Date().toISOString(), ...data })}\n`);
|
||||
}
|
||||
|
||||
function printV02PrMonitorProgress(data: Record<string, unknown> = {}): void {
|
||||
printProgressEvent("hwlab.v02.pr-monitor.progress", data);
|
||||
}
|
||||
|
||||
function shortSha(sha: string): string {
|
||||
return sha.slice(0, 12);
|
||||
}
|
||||
@@ -3753,12 +3791,36 @@ function summarizePrFiles(filesResult: CommandJsonResult): Record<string, unknow
|
||||
|
||||
function mergeCommitFromPr(prData: Record<string, unknown>): string | null {
|
||||
const mergeCommit = record(record(prData.json).mergeCommit);
|
||||
const fromJson = mergeCommit.oid;
|
||||
const fromJson = mergeCommit.oid ?? mergeCommit.sha;
|
||||
if (typeof fromJson === "string") return fromJson;
|
||||
const fromSummary = record(record(prData.pullRequest).mergeCommit).oid;
|
||||
const fromSummary = record(record(prData.pullRequest).mergeCommit).oid ?? record(record(prData.pullRequest).mergeCommit).sha;
|
||||
return typeof fromSummary === "string" ? fromSummary : null;
|
||||
}
|
||||
|
||||
function sourceCommitFromMergeResult(merge: CommandJsonResult, prNumber: number): string | null {
|
||||
const fromMerge = mergeCommitFromPr(commandData(merge));
|
||||
if (fromMerge !== null) return fromMerge;
|
||||
const prRead = readPullRequest(prNumber);
|
||||
if (!isCommandSuccess(prRead)) return null;
|
||||
return mergeCommitFromPr(commandData(prRead));
|
||||
}
|
||||
|
||||
async function waitForV02Head(expectedCommit: string | null, timeoutSeconds: number): Promise<Record<string, unknown>> {
|
||||
const started = Date.now();
|
||||
let head: string | null = null;
|
||||
while (Date.now() - started < timeoutSeconds * 1000) {
|
||||
head = getV02Head();
|
||||
if (expectedCommit === null) {
|
||||
if (head !== null) return { ok: true, sourceCommit: head, expectedCommit, matched: true, waitedSeconds: Math.round((Date.now() - started) / 1000) };
|
||||
} else if (head === expectedCommit) {
|
||||
return { ok: true, sourceCommit: head, expectedCommit, matched: true, waitedSeconds: Math.round((Date.now() - started) / 1000) };
|
||||
}
|
||||
printEvent("v02.source.wait", { expectedCommit, head, waitedSeconds: Math.round((Date.now() - started) / 1000) });
|
||||
await sleep(5_000);
|
||||
}
|
||||
return { ok: false, sourceCommit: head, expectedCommit, matched: false, waitedSeconds: Math.round((Date.now() - started) / 1000), degradedReason: "v02-source-head-not-aligned-after-merge" };
|
||||
}
|
||||
|
||||
function currentGitopsRevision(): string | null {
|
||||
const argo = getArgoStatus();
|
||||
if (!isCommandSuccess(argo)) return null;
|
||||
@@ -3868,6 +3930,540 @@ function appendRolloutBrief(options: G14RecordRolloutOptions, rollout: Record<st
|
||||
return { ok: true, date: now.date, brief, sourceCommit, pipelineRun, gitopsRevision, ciMetrics, bodyPath, update: commandData(update), ensured };
|
||||
}
|
||||
|
||||
function v02PrCommentStatePath(): string {
|
||||
return rootPath(".state", "hwlab-g14", "v02-pr-comment-signatures.json");
|
||||
}
|
||||
|
||||
function readV02PrCommentState(): Record<string, string> {
|
||||
const path = v02PrCommentStatePath();
|
||||
if (!existsSync(path)) return {};
|
||||
try {
|
||||
const parsed = JSON.parse(readFileSync(path, "utf8")) as unknown;
|
||||
const state: Record<string, string> = {};
|
||||
for (const [key, value] of Object.entries(record(parsed))) {
|
||||
if (typeof value === "string") state[key] = value;
|
||||
}
|
||||
return state;
|
||||
} catch {
|
||||
return {};
|
||||
}
|
||||
}
|
||||
|
||||
function writeV02PrCommentState(state: Record<string, string>): string {
|
||||
const path = v02PrCommentStatePath();
|
||||
mkdirSync(dirname(path), { recursive: true });
|
||||
writeFileSync(path, `${JSON.stringify(state, null, 2)}\n`, "utf8");
|
||||
return path;
|
||||
}
|
||||
|
||||
function preflightSummary(preflight: CommandJsonResult): Record<string, unknown> {
|
||||
const data = commandData(preflight);
|
||||
const mergeability = record(data.mergeability);
|
||||
return {
|
||||
ok: isCommandSuccess(preflight),
|
||||
conclusion: stringOrNull(mergeability.conclusion) ?? (isCommandSuccess(preflight) ? "unknown" : "preflight-command-failed"),
|
||||
readyForCommanderMerge: mergeability.readyForCommanderMerge === true,
|
||||
blockers: Array.isArray(mergeability.blockers) ? mergeability.blockers.map(String).slice(0, 20) : [],
|
||||
pending: Array.isArray(mergeability.pending) ? mergeability.pending.map(String).slice(0, 20) : [],
|
||||
mergeable: mergeability.mergeable ?? nested(data, ["pullRequest", "mergeable"]) ?? null,
|
||||
mergeStateStatus: mergeability.mergeStateStatus ?? nested(data, ["pullRequest", "mergeStateStatus"]) ?? null,
|
||||
headRefName: nested(data, ["pullRequest", "headRefName"]) ?? null,
|
||||
baseRefName: nested(data, ["pullRequest", "baseRefName"]) ?? null,
|
||||
statusChecks: record(data.statusChecks).summary ?? data.statusChecks ?? null,
|
||||
};
|
||||
}
|
||||
|
||||
function v02PrConflictState(summary: Record<string, unknown>): string {
|
||||
const mergeable = String(summary.mergeable ?? "").toUpperCase();
|
||||
const mergeStateStatus = String(summary.mergeStateStatus ?? "").toUpperCase();
|
||||
const blockers = Array.isArray(summary.blockers) ? summary.blockers.map(String).join("\n") : "";
|
||||
if (mergeable.includes("CONFLICT") || mergeStateStatus.includes("DIRTY") || /conflict/i.test(blockers)) return "conflict";
|
||||
return "clear-or-unknown";
|
||||
}
|
||||
|
||||
function summarizeV02CdStatus(status: Record<string, unknown>): Record<string, unknown> {
|
||||
const pipelineRun = record(status.pipelineRun);
|
||||
const targetValidation = record(status.targetValidation);
|
||||
const gitMirror = record(status.gitMirror);
|
||||
const gitMirrorSummary = record(gitMirror.summary);
|
||||
const planArtifacts = record(status.planArtifacts);
|
||||
const planSummary = record(planArtifacts.summary);
|
||||
const argo = record(status.argo);
|
||||
const webAssets = record(status.webAssets);
|
||||
return {
|
||||
ok: status.ok === true,
|
||||
sourceCommit: status.sourceCommit ?? nested(status, ["statusTarget", "sourceCommit"]) ?? null,
|
||||
pipelineRun: pipelineRun.pipelineRun ?? pipelineRun.name ?? nested(status, ["statusTarget", "pipelineRun"]) ?? null,
|
||||
pipelineStatus: pipelineRun.status ?? null,
|
||||
pipelineReason: pipelineRun.reason ?? null,
|
||||
targetValidationState: targetValidation.state ?? null,
|
||||
targetValidationOk: targetValidation.ok ?? null,
|
||||
targetValidationFailures: Array.isArray(targetValidation.failures) ? targetValidation.failures.slice(0, 10) : [],
|
||||
targetValidationWarnings: Array.isArray(targetValidation.warnings) ? targetValidation.warnings.slice(0, 10) : [],
|
||||
argoSync: argo.syncStatus ?? nested(argo, ["summary", "syncStatus"]) ?? null,
|
||||
argoHealth: argo.healthStatus ?? nested(argo, ["summary", "healthStatus"]) ?? null,
|
||||
webAssetsOk: webAssets.ok ?? null,
|
||||
pendingFlush: gitMirrorSummary.pendingFlush ?? null,
|
||||
githubInSync: gitMirrorSummary.githubInSync ?? null,
|
||||
buildServices: planSummary.buildServices ?? planArtifacts.buildServices ?? null,
|
||||
reusedServices: planSummary.reusedServices ?? planArtifacts.reusedServices ?? null,
|
||||
rolloutServices: planSummary.rolloutServices ?? planArtifacts.rolloutServices ?? null,
|
||||
};
|
||||
}
|
||||
|
||||
function v02CdPassed(status: Record<string, unknown>): boolean {
|
||||
const state = String(nested(status, ["targetValidation", "state"]) ?? "");
|
||||
return state === "passed" || state === "superseded";
|
||||
}
|
||||
|
||||
function v02PipelineSucceeded(status: Record<string, unknown>): boolean {
|
||||
return String(nested(status, ["pipelineRun", "status"]) ?? "") === "True";
|
||||
}
|
||||
|
||||
function v02CdFailed(status: Record<string, unknown>): boolean {
|
||||
const pipelineStatus = String(nested(status, ["pipelineRun", "status"]) ?? "");
|
||||
return pipelineStatus === "False";
|
||||
}
|
||||
|
||||
function activeV02PipelineRuns(status: Record<string, unknown>): Record<string, unknown>[] {
|
||||
const active = record(status.activePipelineRuns);
|
||||
const items = Array.isArray(active.items) ? active.items : [];
|
||||
return items.map((item) => record(item)).filter((item) => String(item.status ?? "") === "Unknown");
|
||||
}
|
||||
|
||||
function v02PrCommentSignature(input: V02PrCommentInput): string {
|
||||
const summary = input.preflight ?? {};
|
||||
const cd = input.cd ?? {};
|
||||
const flush = input.flush ?? {};
|
||||
return textHash(JSON.stringify({
|
||||
pr: input.pr.number,
|
||||
state: input.state,
|
||||
phase: input.phase,
|
||||
conclusion: summary.conclusion ?? null,
|
||||
readyForCommanderMerge: summary.readyForCommanderMerge ?? null,
|
||||
conflict: v02PrConflictState(summary),
|
||||
blockers: summary.blockers ?? [],
|
||||
pending: summary.pending ?? [],
|
||||
sourceCommit: input.sourceCommit ?? null,
|
||||
pipelineRun: input.pipelineRun ?? null,
|
||||
pipelineStatus: cd.pipelineStatus ?? null,
|
||||
pipelineReason: cd.pipelineReason ?? null,
|
||||
targetValidationState: cd.targetValidationState ?? null,
|
||||
pendingFlush: cd.pendingFlush ?? null,
|
||||
flushOk: flush.ok ?? null,
|
||||
dryRun: input.dryRun === true,
|
||||
}));
|
||||
}
|
||||
|
||||
function listValue(value: unknown): string {
|
||||
if (!Array.isArray(value) || value.length === 0) return "none";
|
||||
return value.map(String).slice(0, 8).join("; ");
|
||||
}
|
||||
|
||||
function scalarValue(value: unknown): string {
|
||||
if (value === null || value === undefined || value === "") return "n/a";
|
||||
if (typeof value === "object") return JSON.stringify(value).slice(0, 240);
|
||||
return String(value);
|
||||
}
|
||||
|
||||
export function v02PrAutomationCommentBody(input: V02PrCommentInput): string {
|
||||
const preflight = input.preflight ?? {};
|
||||
const cd = input.cd ?? {};
|
||||
const flush = input.flush ?? {};
|
||||
const title = input.pr.title ?? `PR #${input.pr.number}`;
|
||||
const url = input.pr.url ?? `https://github.com/${HWLAB_REPO}/pull/${input.pr.number}`;
|
||||
const elapsed = formatDuration(input.elapsedSeconds);
|
||||
const dryRunPrefix = input.dryRun === true ? "[dry-run] " : "";
|
||||
return [
|
||||
`### ${dryRunPrefix}v0.2 自动 CI/CD 状态`,
|
||||
"",
|
||||
`${input.message ?? "UniDesk monitor 已记录本轮 PR 自动化状态。"}`,
|
||||
"",
|
||||
"- PR:",
|
||||
` - [#${input.pr.number} ${title}](${url})`,
|
||||
` - base: \`${input.pr.baseRefName ?? V02_SOURCE_BRANCH}\`; head: \`${input.pr.headRefName ?? "unknown"}\``,
|
||||
"- 自动化状态:",
|
||||
` - state: \`${input.state}\`; phase: \`${input.phase}\``,
|
||||
` - startedAt: \`${input.startedAt}\`; observedAt: \`${input.observedAt}\`; elapsed: ${elapsed}`,
|
||||
"- CI / mergeability:",
|
||||
` - conclusion: \`${scalarValue(preflight.conclusion)}\`; readyForCommanderMerge: \`${String(preflight.readyForCommanderMerge ?? "n/a")}\`; conflict: \`${v02PrConflictState(preflight)}\``,
|
||||
` - blockers: ${listValue(preflight.blockers)}`,
|
||||
` - pending: ${listValue(preflight.pending)}`,
|
||||
"- CD / runtime:",
|
||||
` - sourceCommit: \`${input.sourceCommit ?? "n/a"}\``,
|
||||
` - PipelineRun: \`${input.pipelineRun ?? scalarValue(cd.pipelineRun)}\``,
|
||||
` - pipeline: \`${scalarValue(cd.pipelineStatus)} / ${scalarValue(cd.pipelineReason)}\`; targetValidation: \`${scalarValue(cd.targetValidationState)}\``,
|
||||
` - Argo: \`${scalarValue(cd.argoSync)} / ${scalarValue(cd.argoHealth)}\`; webAssets.ok: \`${String(cd.webAssetsOk ?? "n/a")}\``,
|
||||
` - Git mirror: pendingFlush=\`${String(cd.pendingFlush ?? "n/a")}\`, githubInSync=\`${String(cd.githubInSync ?? "n/a")}\`, flush.ok=\`${String(flush.ok ?? "n/a")}\``,
|
||||
` - rolloutServices: ${listValue(cd.rolloutServices)}; buildServices: ${listValue(cd.buildServices)}; reusedServices: ${listValue(cd.reusedServices)}`,
|
||||
"",
|
||||
"后续动作由 `bun scripts/cli.ts hwlab g14 monitor-prs --lane v02` 继续驱动；同一状态签名不会重复刷评论。",
|
||||
].join("\n");
|
||||
}
|
||||
|
||||
function commentV02PullRequest(input: V02PrCommentInput): Record<string, unknown> {
|
||||
const body = v02PrAutomationCommentBody(input);
|
||||
const signature = v02PrCommentSignature(input);
|
||||
const cacheKey = `pr-${input.pr.number}`;
|
||||
if (input.dryRun === true) {
|
||||
return { ok: true, dryRun: true, skipped: true, signature, wouldComment: { bodyPreview: body.slice(0, 1200), bodyChars: body.length } };
|
||||
}
|
||||
const state = readV02PrCommentState();
|
||||
if (state[cacheKey] === signature) {
|
||||
return { ok: true, skipped: true, reason: "same-pr-comment-signature", signature, stateFile: v02PrCommentStatePath() };
|
||||
}
|
||||
const bodyPath = writeStateFile(`v02-pr-${input.pr.number}-${input.state}-${signature}.md`, `${body}\n`);
|
||||
const create = cliJson(["gh", "pr", "comment", "create", String(input.pr.number), "--repo", HWLAB_REPO, "--body-file", bodyPath], 100_000);
|
||||
if (!isCommandSuccess(create)) return { ok: false, phase: "pr-comment", signature, bodyPath, create };
|
||||
state[cacheKey] = signature;
|
||||
const stateFile = writeV02PrCommentState(state);
|
||||
return { ok: true, signature, bodyPath, stateFile, create: commandData(create) };
|
||||
}
|
||||
|
||||
function triggerV02Current(timeoutSeconds: number): Record<string, unknown> {
|
||||
return runV02ControlPlane({
|
||||
action: "trigger-current",
|
||||
lane: "v02",
|
||||
dryRun: false,
|
||||
confirm: true,
|
||||
wait: true,
|
||||
allowLiveDbRead: false,
|
||||
timeoutSeconds,
|
||||
minAgeMinutes: 30,
|
||||
limit: 20,
|
||||
});
|
||||
}
|
||||
|
||||
function flushV02GitMirrorIfNeeded(status: Record<string, unknown>, timeoutSeconds: number): Record<string, unknown> {
|
||||
const pendingFlush = nested(status, ["gitMirror", "summary", "pendingFlush"]);
|
||||
if (pendingFlush !== true) return { ok: true, skipped: true, reason: "git-mirror-already-flushed" };
|
||||
return runG14GitMirror({
|
||||
action: "flush",
|
||||
confirm: true,
|
||||
dryRun: false,
|
||||
wait: true,
|
||||
timeoutSeconds,
|
||||
});
|
||||
}
|
||||
|
||||
function pullRequestFromMergeCommand(merge: CommandJsonResult): Record<string, unknown> {
|
||||
const data = commandData(merge);
|
||||
const direct = record(data.pullRequest);
|
||||
if (Object.keys(direct).length > 0) return direct;
|
||||
return record(nested(data, ["details", "details", "pullRequest"]));
|
||||
}
|
||||
|
||||
function mergeCommandRaceState(merge: CommandJsonResult): "merged" | "closed" | null {
|
||||
const pullRequest = pullRequestFromMergeCommand(merge);
|
||||
if (pullRequest.merged === true || pullRequest.stateDetail === "merged") return "merged";
|
||||
if (pullRequest.closed === true || pullRequest.state === "closed") return "closed";
|
||||
return null;
|
||||
}
|
||||
|
||||
async function waitForV02Cd(sourceCommit: string, timeoutSeconds: number): Promise<Record<string, unknown>> {
|
||||
const started = Date.now();
|
||||
const startedAt = new Date(started).toISOString();
|
||||
const pipelineRun = v02PipelineRunName(sourceCommit);
|
||||
let lastStatus: Record<string, unknown> = {};
|
||||
let firstPassedAt: string | null = null;
|
||||
while (Date.now() - started < timeoutSeconds * 1000) {
|
||||
lastStatus = v02ControlPlaneStatus({ sourceCommit, mode: "source-commit" });
|
||||
const summary = summarizeV02CdStatus(lastStatus);
|
||||
printEvent("v02.cd.status", { sourceCommit, pipelineRun, summary });
|
||||
printV02PrMonitorProgress({ stage: "cd-status", status: "running", sourceCommit, pipelineRun, targetValidationState: summary.targetValidationState ?? null, pipelineStatus: summary.pipelineStatus ?? null, pendingFlush: summary.pendingFlush ?? null });
|
||||
if (v02PipelineSucceeded(lastStatus) && summary.pendingFlush === true) {
|
||||
const flush = flushV02GitMirrorIfNeeded(lastStatus, Math.min(timeoutSeconds, 600));
|
||||
if (record(flush).ok !== true) {
|
||||
return {
|
||||
ok: false,
|
||||
phase: "git-mirror-flush",
|
||||
startedAt,
|
||||
finishedAt: new Date().toISOString(),
|
||||
sourceCommit,
|
||||
pipelineRun,
|
||||
status: summary,
|
||||
rawStatus: lastStatus,
|
||||
flush,
|
||||
};
|
||||
}
|
||||
const finalStatus = v02ControlPlaneStatus({ sourceCommit, mode: "source-commit" });
|
||||
const finalSummary = summarizeV02CdStatus(finalStatus);
|
||||
if (v02CdPassed(finalStatus)) {
|
||||
printV02PrMonitorProgress({ stage: "git-mirror-flush", status: "succeeded", sourceCommit, pipelineRun, pendingFlush: finalSummary.pendingFlush ?? null });
|
||||
return {
|
||||
ok: true,
|
||||
phase: "cd-passed",
|
||||
startedAt,
|
||||
finishedAt: new Date().toISOString(),
|
||||
sourceCommit,
|
||||
pipelineRun,
|
||||
status: finalSummary,
|
||||
rawStatus: finalStatus,
|
||||
flush,
|
||||
};
|
||||
}
|
||||
lastStatus = finalStatus;
|
||||
printEvent("v02.cd.after-flush", { sourceCommit, pipelineRun, flushOk: record(flush).ok, summary: finalSummary });
|
||||
printV02PrMonitorProgress({ stage: "git-mirror-flush", status: record(flush).ok === true ? "succeeded" : "failed", sourceCommit, pipelineRun, pendingFlush: finalSummary.pendingFlush ?? null });
|
||||
}
|
||||
if (v02CdPassed(lastStatus)) {
|
||||
firstPassedAt ??= new Date().toISOString();
|
||||
const flush = flushV02GitMirrorIfNeeded(lastStatus, Math.min(timeoutSeconds, 600));
|
||||
const finalStatus = v02ControlPlaneStatus({ sourceCommit, mode: "source-commit" });
|
||||
printV02PrMonitorProgress({ stage: "cd-status", status: v02CdPassed(finalStatus) ? "succeeded" : "failed", sourceCommit, pipelineRun, targetValidationState: nested(finalStatus, ["targetValidation", "state"]) ?? null });
|
||||
return {
|
||||
ok: v02CdPassed(finalStatus),
|
||||
phase: v02CdPassed(finalStatus) ? "cd-passed" : "git-mirror-flush",
|
||||
startedAt,
|
||||
finishedAt: new Date().toISOString(),
|
||||
sourceCommit,
|
||||
pipelineRun,
|
||||
status: summarizeV02CdStatus(finalStatus),
|
||||
rawStatus: finalStatus,
|
||||
flush,
|
||||
};
|
||||
}
|
||||
if (v02CdFailed(lastStatus)) {
|
||||
printV02PrMonitorProgress({ stage: "cd-status", status: "failed", sourceCommit, pipelineRun, pipelineStatus: summary.pipelineStatus ?? null, pipelineReason: summary.pipelineReason ?? null });
|
||||
return {
|
||||
ok: false,
|
||||
phase: "cd-failed",
|
||||
startedAt,
|
||||
finishedAt: new Date().toISOString(),
|
||||
sourceCommit,
|
||||
pipelineRun,
|
||||
status: summary,
|
||||
rawStatus: lastStatus,
|
||||
};
|
||||
}
|
||||
await sleep(30_000);
|
||||
}
|
||||
return {
|
||||
ok: false,
|
||||
phase: "timeout",
|
||||
startedAt,
|
||||
finishedAt: new Date().toISOString(),
|
||||
sourceCommit,
|
||||
pipelineRun,
|
||||
timeoutSeconds,
|
||||
firstPassedAt,
|
||||
status: summarizeV02CdStatus(lastStatus),
|
||||
rawStatus: lastStatus,
|
||||
};
|
||||
}
|
||||
|
||||
async function waitForV02LaneIdle(timeoutSeconds: number): Promise<Record<string, unknown>> {
|
||||
const started = Date.now();
|
||||
let lastStatus: Record<string, unknown> = {};
|
||||
let activeRuns: Record<string, unknown>[] = [];
|
||||
while (Date.now() - started < timeoutSeconds * 1000) {
|
||||
lastStatus = v02ControlPlaneStatus();
|
||||
activeRuns = activeV02PipelineRuns(lastStatus);
|
||||
printEvent("v02.cd.lane-idle", { activeCount: activeRuns.length, activeRuns: activeRuns.slice(0, 5) });
|
||||
printV02PrMonitorProgress({ stage: "lane-idle", status: activeRuns.length === 0 ? "succeeded" : "running", activeCount: activeRuns.length });
|
||||
if (activeRuns.length === 0) {
|
||||
return {
|
||||
ok: true,
|
||||
waitedSeconds: Math.round((Date.now() - started) / 1000),
|
||||
status: summarizeV02CdStatus(lastStatus),
|
||||
activeRuns,
|
||||
};
|
||||
}
|
||||
await sleep(30_000);
|
||||
}
|
||||
return {
|
||||
ok: false,
|
||||
phase: "active-run-timeout",
|
||||
timeoutSeconds,
|
||||
waitedSeconds: Math.round((Date.now() - started) / 1000),
|
||||
status: summarizeV02CdStatus(lastStatus),
|
||||
activeRuns,
|
||||
};
|
||||
}
|
||||
|
||||
async function runV02PrAutoCd(pr: OpenPullRequest, preflight: Record<string, unknown>, merge: CommandJsonResult, options: G14MonitorOptions, startedAt: string): Promise<Record<string, unknown>> {
|
||||
const mergeRaceState = isCommandSuccess(merge) ? null : mergeCommandRaceState(merge);
|
||||
if (!isCommandSuccess(merge) && mergeRaceState !== "merged") {
|
||||
printV02PrMonitorProgress({ stage: "merge", status: "failed", pr: pr.number, mergeRaceState });
|
||||
const comment = commentV02PullRequest({
|
||||
pr,
|
||||
phase: "merge",
|
||||
state: "merge-failed",
|
||||
startedAt,
|
||||
observedAt: new Date().toISOString(),
|
||||
elapsedSeconds: durationSeconds(startedAt, new Date().toISOString()),
|
||||
preflight,
|
||||
merge: compactCommandResult(merge),
|
||||
dryRun: options.dryRun,
|
||||
message: mergeRaceState === "closed"
|
||||
? "PR preflight 曾通过，但自动合并前 PR 已被关闭且未确认 merged；本轮不会触发 CD。"
|
||||
: "PR preflight 已通过，但自动合并失败；需要处理 GitHub 返回的 merge blocker 后重试。",
|
||||
});
|
||||
return { ok: mergeRaceState === "closed", phase: "merge", action: "merge-race-closed", pr, merge, comment };
|
||||
}
|
||||
const expectedCommit = options.dryRun ? null : sourceCommitFromMergeResult(merge, pr.number);
|
||||
if (!options.dryRun && expectedCommit === null) {
|
||||
printV02PrMonitorProgress({ stage: "source-head", status: "failed", pr: pr.number, sourceCommit: null, pipelineRun: null, degradedReason: "merge-commit-unresolved" });
|
||||
const comment = commentV02PullRequest({
|
||||
pr,
|
||||
phase: "merge-commit",
|
||||
state: "cd-failed",
|
||||
startedAt,
|
||||
observedAt: new Date().toISOString(),
|
||||
elapsedSeconds: durationSeconds(startedAt, new Date().toISOString()),
|
||||
preflight,
|
||||
merge: commandData(merge),
|
||||
sourceCommit: null,
|
||||
pipelineRun: null,
|
||||
dryRun: false,
|
||||
message: "PR 已合并，但 UniDesk 未能从 GitHub merge/read 结果解析 merge commit；为避免触发错误 source head，本轮未启动 CD。",
|
||||
});
|
||||
return { ok: false, phase: "merge-commit", pr, merge: commandData(merge), comment, degradedReason: "merge-commit-unresolved" };
|
||||
}
|
||||
const headWait = options.dryRun ? { ok: true, sourceCommit: null, expectedCommit, matched: true } : await waitForV02Head(expectedCommit, 120);
|
||||
const sourceCommit = typeof record(headWait).sourceCommit === "string" ? String(record(headWait).sourceCommit) : null;
|
||||
const pipelineRun = sourceCommit === null ? null : v02PipelineRunName(sourceCommit);
|
||||
printV02PrMonitorProgress({ stage: "merge", status: "succeeded", pr: pr.number, sourceCommit, pipelineRun, mergeRaceState });
|
||||
const startedComment = commentV02PullRequest({
|
||||
pr,
|
||||
phase: options.dryRun ? "dry-run" : "cd-trigger",
|
||||
state: "cd-started",
|
||||
startedAt,
|
||||
observedAt: new Date().toISOString(),
|
||||
elapsedSeconds: durationSeconds(startedAt, new Date().toISOString()),
|
||||
preflight,
|
||||
merge: commandData(merge),
|
||||
sourceCommit,
|
||||
pipelineRun,
|
||||
dryRun: options.dryRun,
|
||||
message: options.dryRun
|
||||
? "dry-run：PR 已达到自动合并条件；本轮不会写 GitHub merge 或 CD。"
|
||||
: mergeRaceState === "merged"
|
||||
? "PR 在本轮 merge 前已被合并；worker 继续对齐 merge commit 并驱动 v0.2 CD。后续成功、失败或超时会继续在本 PR 下回复。"
|
||||
: "PR 已自动合并，v0.2 CD 准备开始。后续成功、失败或超时会继续在本 PR 下回复。",
|
||||
});
|
||||
if (!options.dryRun && record(startedComment).ok !== true) {
|
||||
return { ok: false, phase: "pr-comment", pr, sourceCommit, pipelineRun, comment: startedComment };
|
||||
}
|
||||
if (options.dryRun) return { ok: true, action: "dry-run-merge", pr, merge: commandData(merge), comment: startedComment };
|
||||
if (sourceCommit === null || record(headWait).ok !== true) {
|
||||
printV02PrMonitorProgress({ stage: "source-head", status: "failed", pr: pr.number, sourceCommit, pipelineRun, expectedCommit });
|
||||
const comment = commentV02PullRequest({
|
||||
pr,
|
||||
phase: "v02-source-head",
|
||||
state: "cd-timeout",
|
||||
startedAt,
|
||||
observedAt: new Date().toISOString(),
|
||||
elapsedSeconds: durationSeconds(startedAt, new Date().toISOString()),
|
||||
preflight,
|
||||
merge: commandData(merge),
|
||||
sourceCommit,
|
||||
pipelineRun,
|
||||
dryRun: false,
|
||||
message: "PR 已合并，但 G14 v0.2 CI/CD source repo 没有在等待窗口内对齐 merge commit；为避免触发旧 head，本轮未启动 CD。",
|
||||
});
|
||||
return { ok: false, phase: "v02-head", pr, merge: commandData(merge), expectedCommit, headWait, comment: record(comment).ok === true ? comment : startedComment, degradedReason: "v02-head-unresolved-after-merge" };
|
||||
}
|
||||
const before = v02ControlPlaneStatus({ sourceCommit, mode: "source-commit" });
|
||||
const beforeSummary = summarizeV02CdStatus(before);
|
||||
const activeRuns = activeV02PipelineRuns(before);
|
||||
if (activeRuns.length > 0 && !v02CdPassed(before)) {
|
||||
printV02PrMonitorProgress({ stage: "lane-idle", status: "running", pr: pr.number, sourceCommit, pipelineRun, activeCount: activeRuns.length });
|
||||
const comment = commentV02PullRequest({
|
||||
pr,
|
||||
phase: "cd-active-run",
|
||||
state: "cd-blocked",
|
||||
startedAt,
|
||||
observedAt: new Date().toISOString(),
|
||||
elapsedSeconds: durationSeconds(startedAt, new Date().toISOString()),
|
||||
preflight,
|
||||
merge: commandData(merge),
|
||||
sourceCommit,
|
||||
pipelineRun,
|
||||
cd: beforeSummary,
|
||||
dryRun: false,
|
||||
message: "PR 已合并，但 v0.2 当前已有运行中的 PipelineRun；worker 会等待 lane 空闲后继续触发当前 merge commit 的 CD。",
|
||||
});
|
||||
if (record(comment).ok !== true) return { ok: false, phase: "pr-comment", pr, sourceCommit, pipelineRun, activeRuns, before: beforeSummary, comment };
|
||||
const idle = await waitForV02LaneIdle(options.timeoutSeconds);
|
||||
if (record(idle).ok !== true) {
|
||||
const timeoutComment = commentV02PullRequest({
|
||||
pr,
|
||||
phase: "cd-active-run-timeout",
|
||||
state: "cd-timeout",
|
||||
startedAt,
|
||||
observedAt: new Date().toISOString(),
|
||||
elapsedSeconds: durationSeconds(startedAt, new Date().toISOString()),
|
||||
preflight,
|
||||
merge: commandData(merge),
|
||||
sourceCommit,
|
||||
pipelineRun,
|
||||
cd: record(idle.status),
|
||||
dryRun: false,
|
||||
message: "PR 已合并，但 v0.2 lane 长时间被已有 PipelineRun 占用，当前 merge commit 尚未触发 CD；本评论保留 active run 状态用于接续排障。",
|
||||
});
|
||||
return { ok: false, phase: "cd-active-run-timeout", pr, sourceCommit, pipelineRun, activeRuns, before: beforeSummary, idle, comment, timeoutComment };
|
||||
}
|
||||
}
|
||||
const trigger = v02CdPassed(before) ? { ok: true, skipped: true, reason: "source-commit-already-deployed" } : triggerV02Current(Math.min(options.timeoutSeconds, 600));
|
||||
printEvent("v02.cd.trigger", { pr: pr.number, sourceCommit, pipelineRun, ok: record(trigger).ok, skipped: record(trigger).skipped ?? false, degradedReason: record(trigger).degradedReason ?? null });
|
||||
printV02PrMonitorProgress({ stage: "cd-trigger", status: record(trigger).ok === true || record(trigger).degradedReason === "refuse-active-or-successful-pipelinerun" ? "succeeded" : "failed", pr: pr.number, sourceCommit, pipelineRun, skipped: record(trigger).skipped ?? false, degradedReason: record(trigger).degradedReason ?? null });
|
||||
if (record(trigger).ok !== true && record(trigger).degradedReason !== "refuse-active-or-successful-pipelinerun") {
|
||||
const comment = commentV02PullRequest({
|
||||
pr,
|
||||
phase: "cd-trigger",
|
||||
state: "cd-failed",
|
||||
startedAt,
|
||||
observedAt: new Date().toISOString(),
|
||||
elapsedSeconds: durationSeconds(startedAt, new Date().toISOString()),
|
||||
preflight,
|
||||
merge: commandData(merge),
|
||||
sourceCommit,
|
||||
pipelineRun,
|
||||
cd: beforeSummary,
|
||||
dryRun: false,
|
||||
message: "PR 已合并，但 v0.2 CD 触发失败；需要查看 trigger 阶段的 degradedReason。",
|
||||
});
|
||||
return { ok: false, phase: "cd-trigger", pr, sourceCommit, pipelineRun, trigger, comment };
|
||||
}
|
||||
const cd = await waitForV02Cd(sourceCommit, options.timeoutSeconds);
|
||||
const cdStatus = record(cd.status);
|
||||
const flush = record(cd.flush);
|
||||
const cdOk = cd.ok === true;
|
||||
const cdPhase = String(cd.phase ?? "");
|
||||
const finalComment = commentV02PullRequest({
|
||||
pr,
|
||||
phase: cdPhase,
|
||||
state: cdOk ? "cd-succeeded" : cdPhase === "timeout" ? "cd-timeout" : "cd-failed",
|
||||
startedAt,
|
||||
observedAt: new Date().toISOString(),
|
||||
elapsedSeconds: durationSeconds(startedAt, new Date().toISOString()),
|
||||
preflight,
|
||||
merge: commandData(merge),
|
||||
sourceCommit,
|
||||
pipelineRun,
|
||||
cd: cdStatus,
|
||||
flush,
|
||||
dryRun: false,
|
||||
message: cdOk
|
||||
? "v0.2 自动 CD 已完成：PipelineRun、Argo/runtime、公开探针和 Git mirror flush 收口均已检查。"
|
||||
: cdPhase === "timeout"
|
||||
? "v0.2 自动 CD 超时未收口；本评论保留最后一次 targetValidation / PipelineRun / Git mirror 状态用于接续排障。"
|
||||
: "v0.2 自动 CD 失败；本评论保留失败阶段和最后一次状态用于接续排障。",
|
||||
});
|
||||
return {
|
||||
ok: cdOk && record(finalComment).ok === true,
|
||||
action: cdOk ? "merged-and-rolled-v02" : "v02-cd-failed",
|
||||
phase: cdOk ? "cd-passed" : cdPhase,
|
||||
pr,
|
||||
sourceCommit,
|
||||
pipelineRun,
|
||||
trigger,
|
||||
cd,
|
||||
comment: finalComment,
|
||||
};
|
||||
}
|
||||
|
||||
async function waitForG14Dev(sourceCommit: string, timeoutSeconds: number): Promise<Record<string, unknown>> {
|
||||
const started = Date.now();
|
||||
const startedAt = new Date(started).toISOString();
|
||||
@@ -3910,7 +4506,88 @@ async function waitForG14Dev(sourceCommit: string, timeoutSeconds: number): Prom
|
||||
return { ok: false, phase: "timeout", sourceCommit, pipelineText, argoText, startedAt, pipelineSucceededAt, workloadsReady, healthOk, timeoutSeconds };
|
||||
}
|
||||
|
||||
async function monitorV02Cycle(options: G14MonitorOptions, cycle: number): Promise<Record<string, unknown>> {
|
||||
printEvent("v02.monitor.cycle.start", { cycle, dryRun: options.dryRun });
|
||||
const listed = listOpenG14PullRequests();
|
||||
if (!isCommandSuccess(listed)) return { ok: false, cycle, phase: "list-prs", listed };
|
||||
const prs = extractPullRequests(listed, V02_SOURCE_BRANCH);
|
||||
printEvent("v02.monitor.prs", { cycle, count: prs.length, pullRequests: prs });
|
||||
if (prs.length === 0) return { ok: true, cycle, action: "none", lane: "v02", pullRequests: [] };
|
||||
const observations: unknown[] = [];
|
||||
for (const pr of prs) {
|
||||
const startedAt = new Date().toISOString();
|
||||
const preflightResult = preflightPullRequest(pr.number);
|
||||
const preflight = preflightSummary(preflightResult);
|
||||
printEvent("v02.pr.preflight", {
|
||||
cycle,
|
||||
number: pr.number,
|
||||
ok: isCommandSuccess(preflightResult),
|
||||
conclusion: preflight.conclusion,
|
||||
readyForCommanderMerge: preflight.readyForCommanderMerge,
|
||||
conflict: v02PrConflictState(preflight),
|
||||
});
|
||||
printV02PrMonitorProgress({
|
||||
stage: "preflight",
|
||||
status: preflight.readyForCommanderMerge === true ? "succeeded" : "running",
|
||||
pr: pr.number,
|
||||
conclusion: preflight.conclusion,
|
||||
conflict: v02PrConflictState(preflight),
|
||||
});
|
||||
if (!isCommandSuccess(preflightResult) || preflight.readyForCommanderMerge !== true) {
|
||||
const conclusion = String(preflight.conclusion ?? "unknown");
|
||||
const blocked = conclusion === "blocked" || v02PrConflictState(preflight) === "conflict" || !isCommandSuccess(preflightResult);
|
||||
const comment = commentV02PullRequest({
|
||||
pr,
|
||||
phase: "preflight",
|
||||
state: blocked ? "blocked" : "waiting-ci",
|
||||
startedAt,
|
||||
observedAt: new Date().toISOString(),
|
||||
elapsedSeconds: durationSeconds(startedAt, new Date().toISOString()),
|
||||
preflight,
|
||||
dryRun: options.dryRun,
|
||||
message: blocked
|
||||
? "v0.2 自动化已暂停在 PR preflight：存在 blocker 或冲突，需要先修复后再继续。"
|
||||
: "v0.2 自动化正在等待 GitHub CI / mergeability 收敛；CI 通过且无冲突后会自动合并并触发 CD。",
|
||||
});
|
||||
observations.push({ pullRequest: pr, preflight, comment });
|
||||
printV02PrMonitorProgress({ stage: "pr-comment", status: record(comment).ok === true ? "succeeded" : "failed", pr: pr.number, conclusion: preflight.conclusion });
|
||||
if (record(comment).ok !== true) return { ok: false, cycle, lane: "v02", phase: "pr-comment", pullRequest: pr, preflight, comment, observations };
|
||||
continue;
|
||||
}
|
||||
const currentStatus = v02ControlPlaneStatus();
|
||||
const activeRuns = activeV02PipelineRuns(currentStatus);
|
||||
if (activeRuns.length > 0) {
|
||||
const cd = summarizeV02CdStatus(currentStatus);
|
||||
const comment = commentV02PullRequest({
|
||||
pr,
|
||||
phase: "cd-active-before-merge",
|
||||
state: "cd-blocked",
|
||||
startedAt,
|
||||
observedAt: new Date().toISOString(),
|
||||
elapsedSeconds: durationSeconds(startedAt, new Date().toISOString()),
|
||||
preflight,
|
||||
cd,
|
||||
dryRun: options.dryRun,
|
||||
message: "PR 已通过 CI / mergeability，但 v0.2 lane 当前已有运行中的 PipelineRun；为保持 PR merge commit 与 CD 目标一一对应，本轮暂不合并，待 lane 空闲后自动继续。",
|
||||
});
|
||||
observations.push({ pullRequest: pr, preflight, activeRuns, cd, comment });
|
||||
printV02PrMonitorProgress({ stage: "pr-comment", status: record(comment).ok === true ? "succeeded" : "failed", pr: pr.number, activeCount: activeRuns.length });
|
||||
if (record(comment).ok !== true) return { ok: false, cycle, lane: "v02", phase: "pr-comment", pullRequest: pr, preflight, comment, observations };
|
||||
continue;
|
||||
}
|
||||
const merge = mergePullRequest(pr.number, options.dryRun);
|
||||
printEvent("v02.pr.merge", { cycle, number: pr.number, dryRun: options.dryRun, ok: isCommandSuccess(merge) });
|
||||
printV02PrMonitorProgress({ stage: "merge", status: isCommandSuccess(merge) ? "succeeded" : "running", pr: pr.number, dryRun: options.dryRun });
|
||||
const result = await runV02PrAutoCd(pr, preflight, merge, options, startedAt);
|
||||
observations.push(result);
|
||||
if (record(result).ok !== true) return { ok: false, cycle, lane: "v02", phase: record(result).phase ?? "v02-auto-cd", pullRequest: pr, result, observations };
|
||||
return { ok: true, cycle, lane: "v02", action: record(result).action ?? (options.dryRun ? "dry-run-merge" : "merged-and-rolled-v02"), result, observations };
|
||||
}
|
||||
return { ok: true, cycle, lane: "v02", action: "none", observations };
|
||||
}
|
||||
|
||||
async function monitorCycle(options: G14MonitorOptions, cycle: number): Promise<Record<string, unknown>> {
|
||||
if (options.lane === "v02") return monitorV02Cycle(options, cycle);
|
||||
printEvent("g14.monitor.cycle.start", { cycle, dryRun: options.dryRun });
|
||||
const precheck = precheckWorkspace();
|
||||
if (!isCommandSuccess(precheck)) return { ok: false, cycle, phase: "workspace-precheck", precheck };
|
||||
@@ -3950,10 +4627,10 @@ async function runMonitorWorker(options: G14MonitorOptions): Promise<Record<stri
|
||||
cycle += 1;
|
||||
const result = await monitorCycle(options, cycle);
|
||||
results.push(result);
|
||||
printEvent("g14.monitor.cycle.done", { cycle, ok: record(result).ok, action: record(result).action ?? null, phase: record(result).phase ?? null });
|
||||
printEvent(`${options.lane}.monitor.cycle.done`, { cycle, lane: options.lane, ok: record(result).ok, action: record(result).action ?? null, phase: record(result).phase ?? null });
|
||||
if (record(result).ok !== true) return { ok: false, cycles: cycle, lastResult: result, results };
|
||||
if (options.once || record(result).action !== "none") return { ok: true, cycles: cycle, lastResult: result, results };
|
||||
printEvent("g14.monitor.sleep", { cycle, intervalSeconds: options.intervalSeconds });
|
||||
if (options.once || (options.lane === "g14" && record(result).action !== "none")) return { ok: true, cycles: cycle, lastResult: result, results };
|
||||
printEvent(`${options.lane}.monitor.sleep`, { cycle, lane: options.lane, intervalSeconds: options.intervalSeconds });
|
||||
await sleep(options.intervalSeconds * 1000);
|
||||
}
|
||||
return { ok: true, cycles: cycle, results };
|
||||
@@ -3965,7 +4642,9 @@ export function hwlabG14Help(): Record<string, unknown> {
|
||||
output: "json",
|
||||
usage: [
|
||||
"bun scripts/cli.ts hwlab g14 monitor-prs",
|
||||
"bun scripts/cli.ts hwlab g14 monitor-prs --lane v02",
|
||||
"bun scripts/cli.ts hwlab g14 monitor-prs --once --dry-run",
|
||||
"bun scripts/cli.ts hwlab g14 monitor-prs --lane v02 --once --dry-run",
|
||||
"bun scripts/cli.ts hwlab g14 record-rollout --pr <number> [--source-commit sha]",
|
||||
"bun scripts/cli.ts hwlab g14 control-plane status --lane v02",
|
||||
"bun scripts/cli.ts hwlab g14 control-plane status --lane v02 --pipeline-run hwlab-v02-ci-poll-<short-sha>",
|
||||
@@ -3994,10 +4673,11 @@ export function hwlabG14Help(): Record<string, unknown> {
|
||||
"bun scripts/cli.ts hwlab g14 tools-image build --name ci-node-tools --tag node22-alpine-bun-v1 --confirm",
|
||||
"bun scripts/cli.ts job status <jobId> --tail-bytes 30000",
|
||||
],
|
||||
description: "G14 HWLAB PR monitor, DEV rollout command, bounded v0.2 control-plane bootstrap/cleanup/runtime-migration helper, v0.2 runtime SecretRef bootstrap, devops-infra git mirror maintenance, and controlled CI tools image build/status entry. The public monitor starts a fire-and-forget job; confirmed control-plane trigger-current and git-mirror sync/flush also return async jobs by default, with --wait reserved for explicit synchronous debugging. control-plane status/apply/cleanup-runs/cleanup-released-pvs/runtime-migration uses UniDesk G14:k3s routes for v0.2 Tekton/Argo control resources, runtime migration, and completed CI workspace retention only. secret status/ensure is the standard v0.2 runtime SecretRef bootstrap path; it never reads or prints secret values. git-mirror status/apply/sync/flush is the manual devops-infra mirror/relay control path and does not install a CronJob.",
|
||||
description: "G14 HWLAB PR monitor, DEV rollout command, bounded v0.2 control-plane bootstrap/cleanup/runtime-migration helper, v0.2 runtime SecretRef bootstrap, devops-infra git mirror maintenance, and controlled CI tools image build/status entry. The public monitor starts a fire-and-forget job. Default monitor lane is base=G14; --lane v02 monitors base=v0.2 PRs, waits for GitHub preflight/CI readiness, automatically merges ready PRs, triggers v0.2 CD, flushes the git mirror when needed, and posts deduplicated PR comments for pending, blocked/conflict, success, failure, or timeout states. confirmed control-plane trigger-current and git-mirror sync/flush also return async jobs by default, with --wait reserved for explicit synchronous debugging. control-plane status/apply/cleanup-runs/cleanup-released-pvs/runtime-migration uses UniDesk G14:k3s routes for v0.2 Tekton/Argo control resources, runtime migration, and completed CI workspace retention only. secret status/ensure is the standard v0.2 runtime SecretRef bootstrap path; it never reads or prints secret values. git-mirror status/apply/sync/flush is the manual devops-infra mirror/relay control path and does not install a CronJob.",
|
||||
defaults: {
|
||||
repo: HWLAB_REPO,
|
||||
base: G14_SOURCE_BRANCH,
|
||||
v02Base: V02_SOURCE_BRANCH,
|
||||
provider: G14_PROVIDER,
|
||||
workspace: G14_WORKSPACE,
|
||||
v02Workspace: V02_WORKSPACE,
|
||||
@@ -4012,6 +4692,11 @@ export function hwlabG14Help(): Record<string, unknown> {
|
||||
once: ".state/hwlab-g14/latest-once-job.json",
|
||||
dryRun: ".state/hwlab-g14/latest-dry-run-job.json",
|
||||
onceDryRun: ".state/hwlab-g14/latest-once-dry-run-job.json",
|
||||
v02Monitor: ".state/hwlab-g14/latest-v02-monitor-job.json",
|
||||
v02Once: ".state/hwlab-g14/latest-v02-once-job.json",
|
||||
v02DryRun: ".state/hwlab-g14/latest-v02-dry-run-job.json",
|
||||
v02OnceDryRun: ".state/hwlab-g14/latest-v02-once-dry-run-job.json",
|
||||
v02PrCommentSignatures: ".state/hwlab-g14/v02-pr-comment-signatures.json",
|
||||
},
|
||||
};
|
||||
}
|
||||
@@ -4050,8 +4735,12 @@ export async function runHwlabG14Command(_config: Config, args: string[]): Promi
|
||||
}
|
||||
const options = parseOptions(args.slice(1));
|
||||
if (options.worker) return runMonitorWorker(options);
|
||||
const command = ["bun", "scripts/cli.ts", "hwlab", "g14", "monitor-prs", "--worker", "--interval-seconds", String(options.intervalSeconds), "--timeout-seconds", String(options.timeoutSeconds), ...(options.once ? ["--once"] : []), ...(options.dryRun ? ["--dry-run"] : []), ...(options.maxCycles > 0 ? ["--max-cycles", String(options.maxCycles)] : [])];
|
||||
const job = startJob("hwlab_g14_pr_monitor", command, `Monitor ${HWLAB_REPO} PRs targeting ${G14_SOURCE_BRANCH} and roll merged changes to G14 DEV`);
|
||||
const command = ["bun", "scripts/cli.ts", "hwlab", "g14", "monitor-prs", "--worker", "--lane", options.lane, "--interval-seconds", String(options.intervalSeconds), "--timeout-seconds", String(options.timeoutSeconds), ...(options.once ? ["--once"] : []), ...(options.dryRun ? ["--dry-run"] : []), ...(options.maxCycles > 0 ? ["--max-cycles", String(options.maxCycles)] : [])];
|
||||
const jobName = options.lane === "v02" ? "hwlab_g14_v02_pr_monitor" : "hwlab_g14_pr_monitor";
|
||||
const jobNote = options.lane === "v02"
|
||||
? `Monitor ${HWLAB_REPO} PRs targeting ${V02_SOURCE_BRANCH}, merge ready PRs, trigger v0.2 CD, and comment PR progress`
|
||||
: `Monitor ${HWLAB_REPO} PRs targeting ${G14_SOURCE_BRANCH} and roll merged changes to G14 DEV`;
|
||||
const job = startJob(jobName, command, jobNote);
|
||||
const statusCommand = `bun scripts/cli.ts job status ${job.id} --tail-bytes 30000`;
|
||||
const stateDir = rootPath(".state", "hwlab-g14");
|
||||
mkdirSync(stateDir, { recursive: true });
|
||||
@@ -4059,10 +4748,12 @@ export async function runHwlabG14Command(_config: Config, args: string[]): Promi
|
||||
const stateFileRole = hwlabG14MonitorStateRole(options);
|
||||
const latestPath = join(stateDir, stateFileName);
|
||||
const previousLatest = existsSync(latestPath) ? readFileSync(latestPath, "utf8") : null;
|
||||
writeFileSync(latestPath, `${JSON.stringify({ jobId: job.id, createdAt: job.createdAt, statusCommand, role: stateFileRole }, null, 2)}\n`, "utf8");
|
||||
writeFileSync(latestPath, `${JSON.stringify({ jobId: job.id, createdAt: job.createdAt, statusCommand, role: stateFileRole, lane: options.lane, baseBranch: monitorBaseBranch(options.lane) }, null, 2)}\n`, "utf8");
|
||||
return {
|
||||
ok: true,
|
||||
command: "hwlab g14 monitor-prs",
|
||||
lane: options.lane,
|
||||
baseBranch: monitorBaseBranch(options.lane),
|
||||
mode: "async-job",
|
||||
job,
|
||||
statusCommand,
|
||||
|
||||
+9
-3
@@ -224,14 +224,20 @@ export function jobWithTail(job: JobRecord, maxBytes = 12000): JobRecord & {
|
||||
|
||||
function summarizeJobProgress(job: JobRecord, maxBytes = 96_000, tails?: { stdoutTail: string; stderrTail: string }): JobProgressSummary {
|
||||
const knownWorkflow = job.name === "hwlab_g14_v02_trigger_current";
|
||||
const v02PrMonitorWorkflow = job.name === "hwlab_g14_v02_pr_monitor";
|
||||
const gitMirrorWorkflow = job.name === "hwlab_g14_git_mirror_sync" || job.name === "hwlab_g14_git_mirror_flush" || job.name === "agentrun_v01_git_mirror_sync" || job.name === "agentrun_v01_git_mirror_flush";
|
||||
if (!knownWorkflow && !gitMirrorWorkflow) return genericJobProgress(job, tails?.stderrTail);
|
||||
if (!knownWorkflow && !v02PrMonitorWorkflow && !gitMirrorWorkflow) return genericJobProgress(job, tails?.stderrTail);
|
||||
const nowMs = Date.now();
|
||||
const progressTailBytes = Math.max(4096, Math.floor(maxBytes));
|
||||
const stderrTail = tails?.stderrTail ?? tailFile(job.stderrFile, progressTailBytes);
|
||||
const stdoutTail = tails?.stdoutTail ?? tailFile(job.stdoutFile, progressTailBytes);
|
||||
if (gitMirrorWorkflow) return summarizeGitMirrorJobProgress(job, stdoutTail, stderrTail, nowMs);
|
||||
const events = parseJsonLineEvents(stderrTail, "hwlab.v02.trigger.progress");
|
||||
const events = v02PrMonitorWorkflow
|
||||
? [
|
||||
...parseJsonLineEvents(stderrTail, "hwlab.v02.pr-monitor.progress"),
|
||||
...parseJsonLineEvents(stderrTail, "hwlab.v02.trigger.progress"),
|
||||
].sort((left, right) => String(left.at ?? "").localeCompare(String(right.at ?? "")))
|
||||
: parseJsonLineEvents(stderrTail, "hwlab.v02.trigger.progress");
|
||||
const lastEvent = events.at(-1) ?? {};
|
||||
const stage = stringField(lastEvent.stage);
|
||||
const stageStatus = stringField(lastEvent.status);
|
||||
@@ -243,7 +249,7 @@ function summarizeJobProgress(job: JobRecord, maxBytes = 96_000, tails?: { stdou
|
||||
? false
|
||||
: null;
|
||||
const lastEventAt = stringField(lastEvent.at);
|
||||
const kind = events.length > 0 || knownWorkflow ? "hwlab-v02-trigger" : "generic";
|
||||
const kind = events.length > 0 || knownWorkflow || v02PrMonitorWorkflow ? "hwlab-v02-trigger" : "generic";
|
||||
const elapsedSeconds = jobElapsedSeconds(job, nowMs);
|
||||
const stageElapsedSeconds = currentStageElapsedSeconds(events, stage, stageStatus, job, nowMs);
|
||||
const lastEventAgeSeconds = lastEventAt === null ? null : secondsSince(lastEventAt, job.finishedAt ?? nowMs);
|
||||
|
||||
Reference in New Issue
Block a user