fix: formalize pac closeout guidance
Pipelines as Code CI / hwlab-web-probe-sentinel-jd01- Success

This commit is contained in:
Codex
2026-07-06 17:58:43 +00:00
parent 888066ddd8
commit 51058f4635
10 changed files with 273 additions and 35 deletions
+3 -1
View File
@@ -20,6 +20,7 @@ bun scripts/cli.ts platform-infra gitea mirror webhook status --target JD01
bun scripts/cli.ts platform-infra gitea mirror webhook test --target JD01 --repo unidesk-master --confirm
bun scripts/cli.ts platform-infra pipelines-as-code status --target JD01
bun scripts/cli.ts platform-infra pipelines-as-code status --target JD01 --consumer hwlab-jd01-v03
bun scripts/cli.ts platform-infra pipelines-as-code closeout --target JD01 --consumer <consumer> --source-commit <sha> --wait
bun scripts/cli.ts platform-infra pipelines-as-code history --target JD01 --limit 10
bun scripts/cli.ts platform-infra pipelines-as-code history --target JD01 --id <pipelinerun>
bun scripts/cli.ts cicd branch-follower status
@@ -41,7 +42,7 @@ bun scripts/cli.ts cicd branch-follower debug-step --follower web-probe-sentinel
- CI/CD、GitOps、rollout、PipelineRun、Argo、git-mirror 和 AgentRun 部署必须走受控 CLI;不要用裸 `kubectl``argo``tkn``curl` 当正式控制入口。
- CI/CD source authority 只能来自 YAML 声明的 Kubernetes 托管 source authoritylegacy lane 使用 k8s git-mirror snapshotJD01 migrated lane 使用 Gitea controlled mirror + immutable snapshot ref + Pipelines-as-Code。受控命令先在 k8s 内同步/创建不可变 `refs/unidesk/snapshots/.../<commit>` stage refbuild/status/publish 只消费该 snapshothost worktree、本地 `git fetch/pull`、可变 branch ref 或 Pipeline 内直连 GitHub 都不能作为 authoritative source。
- JD01 `agentrun-jd01-v02``sentinel-jd01-v03``hwlab-jd01-v03` 的正式 CI/CD closeout 入口是 `platform-infra pipelines-as-code status|history --target JD01``cicd branch-follower``cicd gitea-actions-poc` 对这三者只保留历史/迁移只读用途,不得作为当前交付判断入口。
- JD01 `agentrun-jd01-v02``sentinel-jd01-v03``hwlab-jd01-v03` 的正式 CI/CD closeout 入口是 `platform-infra pipelines-as-code closeout|status|history --target JD01``closeout` 是通用 consumer 引导入口,只读取/等待 PaC、Tekton、GitOps、Argo 和 runtime 对齐,不触发旧手动发布。`cicd branch-follower``cicd gitea-actions-poc` 对这三者只保留历史/迁移只读用途,不得作为当前交付判断入口。
- GitHub/Git 相关 egress 必须走 YAML-first host proxy/sourceRefbranch-follower controller 读 `config/cicd-branch-followers.yaml#controller.source.githubSsh`runtime legacy git-mirror 读 owning lane/control-plane YAML 的 host proxy 和 `githubTransport`Gitea/PaC 迁移 lane 读 `config/platform-infra/gitea.yaml``config/platform-infra/pipelines-as-code.yaml`;禁止依赖未声明 host env、trans proxy、裸直连 GitHub 或 CLI 输出解析。
- Gitea/PaC lane 的 GitHub -> Gitea 自动同步必须用 `platform-infra gitea mirror webhook apply|status|test`。该路径是单向同步,只更新 Gitea branch 和 immutable snapshot ref;不得引入 Gitea -> GitHub 回写、轮询 fallback、Gitea Actions/act_runner fallback 或第二套状态存储。
- Gitea webhook/FRPC/Caddy 变更的 closeout 必须看受控 CLI 的 rollout/status/test 证据;Secret 或 ConfigMap apply 成功不等于 connector Pod 已加载新 proxy/path。
@@ -62,6 +63,7 @@ bun scripts/cli.ts cicd branch-follower debug-step --follower web-probe-sentinel
- CI/CD 验证、测试和性能度量必须在目标 NODE/k8s 内执行,尤其是 branch-follower、Tekton/Argo、runtime reuse/env reuse、git mirror 和 runtime-ready 相关改动;不要在 master/local host 跑 test 或用本地验证结果替代目标运行面证据。本机只用于源码阅读、编辑和必要静态语法检查,正式收敛结论必须来自目标 NODE 计算出的短摘要。
- in-cluster/controller/native helper 不能假设镜像内存在 `kubectl` binary。目标 Pod/Job 内读取或写入 Kubernetes 对象必须走 serviceaccount token + Kubernetes HTTPS API 或已封装的 native helper`kubectl` 只允许在 operator 侧受控 CLI/trans 边界作为 transport/debug 包装,不得进入正式 controller 状态读写链路。
- 一旦发现 CI/CD CLI 被误用且可能写入错误状态、产生伪证据或绕过目标运行面,必须立刻先把用法改成更符合直觉的公开入口并更新本 skill/reference,再继续验证或交付;不要只靠口头记忆、隐藏 flag、手动约定或后续小心来避免复发。内部 in-cluster 模式必须只由目标 k8s Job/Pod 调用,操作者从本机只能用公开入口提交目标侧 Job 或读取目标侧摘要。
- JD01 migrated consumer 的旧手动 publish/debug 命令不得作为正式 closeout 引导;正式 CLI 输出默认必须指向通用 `platform-infra pipelines-as-code closeout|status|history`。需要保留的旧命令只能标为 debug/test/recovery 单步,且 manual recovery 必须显式说明原因。
- Secret 只通过 YAML sourceRef/targetKey 和受控 CLI 下发;输出只披露 presence/fingerprint。
- 长命令用异步 job 或短轮询;不要长时间挂住 trans/ssh。
@@ -81,4 +81,4 @@ PipelineRun 失败或长时间未完成时,先按定点 `control-plane status
小范围 PR 触发 120s 时必须看 plan artifacts 的 `affectedServices/buildServices/reusedServices`:如果 source diff 很小却出现所有 envreuse 服务都在 `buildServices``reusedServices=[]`,优先怀疑 current GitOps artifact catalog 没有 hydrate 到 source plan 阶段,而不是继续盲目重跑 PipelineRun。
Web 哨兵 control-plane 出现 registry half-state、GitOps 超预算或异步 git-mirror flush 时,先看默认 `DRILLDOWN` 表。PipelineRun/TaskRun 细节走 `platform-infra pipelines-as-code history --id`registry tag 走 `web-probe sentinel image status`GitOps flush 走受控 `hwlab nodes git-mirror status|flush` 或 async job status;不要回退到裸 `kubectl/tkn/argo` 或解析原生命令输出。
JD01 migrated consumer 出现 registry half-state、GitOps 超预算或 runtime 未对齐时,先走通用 `platform-infra pipelines-as-code closeout --target JD01 --consumer <id> --source-commit <sha> --wait`,再用 `status`/`history` drill down。Web 哨兵 control-plane 的默认 `DRILLDOWN` 只作为 debug/status 辅助;PipelineRun/TaskRun 细节走 `platform-infra pipelines-as-code history --id`registry tag 走 `web-probe sentinel image status`GitOps flush 走受控 `hwlab nodes git-mirror status|flush` 或 async job status;不要回退到裸 `kubectl/tkn/argo`、consumer 专用手动 publish 或解析原生命令输出。
@@ -18,9 +18,9 @@ Do not add Gitea Actions, `act_runner`, branch-follower, host worktree, direct G
| Consumer | Source | PaC namespace | Pipeline | Argo application | Status |
|---|---|---|---|---|---|
| `agentrun-jd01-v02` | `pikasTech/agentrun@v0.2` | `agentrun-ci` | `agentrun-jd01-v02-ci-image-publish` | `agentrun-jd01-v02` | `platform-infra pipelines-as-code status --target JD01 --consumer agentrun-jd01-v02` |
| `sentinel-jd01-v03` | `pikasTech/unidesk@master` | `devops-infra` | `hwlab-web-probe-sentinel-jd01-pac` | `hwlab-web-probe-sentinel-jd01` | `platform-infra pipelines-as-code status --target JD01 --consumer sentinel-jd01-v03` |
| `hwlab-jd01-v03` | `pikasTech/HWLAB@v0.3` | `hwlab-ci` | `hwlab-jd01-v03-ci-image-publish` | `hwlab-node-v03` | `platform-infra pipelines-as-code status --target JD01 --consumer hwlab-jd01-v03` |
| `agentrun-jd01-v02` | `pikasTech/agentrun@v0.2` | `agentrun-ci` | `agentrun-jd01-v02-ci-image-publish` | `agentrun-jd01-v02` | `platform-infra pipelines-as-code closeout --target JD01 --consumer agentrun-jd01-v02 --source-commit <sha> --wait` |
| `sentinel-jd01-v03` | `pikasTech/unidesk@master` | `devops-infra` | `hwlab-web-probe-sentinel-jd01-pac` | `hwlab-web-probe-sentinel-jd01` | `platform-infra pipelines-as-code closeout --target JD01 --consumer sentinel-jd01-v03 --source-commit <sha> --wait` |
| `hwlab-jd01-v03` | `pikasTech/HWLAB@v0.3` | `hwlab-ci` | `hwlab-jd01-v03-ci-image-publish` | `hwlab-node-v03` | `platform-infra pipelines-as-code closeout --target JD01 --consumer hwlab-jd01-v03 --source-commit <sha> --wait` |
Use `bun scripts/cli.ts platform-infra pipelines-as-code history --target JD01 --limit 10` for all-consumer trigger, timing and reuse audit. It must read live Gitea Repository CR plus Tekton PipelineRun/TaskRun objects on JD01, aggregate on the target side, and print `READ_ERRORS`.
@@ -28,9 +28,12 @@ Use `bun scripts/cli.ts platform-infra pipelines-as-code history --target JD01 -
1. `bun scripts/cli.ts platform-infra gitea mirror status --target JD01`
2. `bun scripts/cli.ts platform-infra gitea mirror webhook status --target JD01`
3. `bun scripts/cli.ts platform-infra pipelines-as-code status --target JD01 --consumer <id>`
4. `bun scripts/cli.ts platform-infra pipelines-as-code history --target JD01 --consumer <id> --limit 10`
5. For ambiguous rows, drill into the PipelineRun with `bun scripts/cli.ts platform-infra pipelines-as-code history --target JD01 --id <pipelinerun>`.
3. `bun scripts/cli.ts platform-infra pipelines-as-code closeout --target JD01 --consumer <id> --source-commit <sha> --wait`
4. `bun scripts/cli.ts platform-infra pipelines-as-code status --target JD01 --consumer <id>`
5. `bun scripts/cli.ts platform-infra pipelines-as-code history --target JD01 --consumer <id> --limit 10`
6. For ambiguous rows, drill into the PipelineRun with `bun scripts/cli.ts platform-infra pipelines-as-code history --target JD01 --id <pipelinerun>`.
`closeout` is the common formal entry for all migrated JD01 consumers. It waits only for PaC/Tekton/GitOps/Argo/runtime observation from live target-side summaries and does not trigger legacy publish paths. Consumer-specific debug commands such as Web sentinel `publish-current` are recovery/test steps only and must not be shown as the normal next action for formal delivery.
Do not use `cicd branch-follower status` to decide whether the three migrated JD01 consumers are current. It is historical/migration-only and may contain stale state from before PaC cutover.
@@ -41,12 +41,12 @@ bun scripts/cli.ts hwlab g14 observability status|apply|query|targets|boundary|c
bun scripts/cli.ts platform-infra sub2api plan|apply|status|validate
bun scripts/cli.ts platform-infra sub2api codex-pool plan|sync|validate|expose|configure-local
bun scripts/cli.ts platform-infra gitea plan|apply|status|validate|mirror --target JD01
bun scripts/cli.ts platform-infra pipelines-as-code plan|apply|status|history|webhook-test --target JD01 [--consumer <id>]
bun scripts/cli.ts platform-infra pipelines-as-code plan|apply|status|closeout|history|webhook-test --target JD01 [--consumer <id>]
bun scripts/cli.ts platform-infra wechat-archive plan|apply|status|validate|pull
bun scripts/cli.ts platform-infra wechat-archive wcf-host-status|collector-plan|collector-apply|collector-status
```
`platform-infra` 是 UniDesk 运维的平台基础设施控制面;新增平台服务优先进入该命名空间或对应 YAML 声明目标,旧 `devops-infra` 只作为渐进迁移来源。Sub2API 日常部署、Codex pool、FRP 暴露、master `~/.codex` 配置、验收和排障统一使用 `$unidesk-sub2api`。Gitea mirror 和 Pipelines-as-Code 是 JD01 migrated CI source/trigger 平台服务,source-of-truth 分别是 `config/platform-infra/gitea.yaml``config/platform-infra/pipelines-as-code.yaml`;正式架构和三 consumer 矩阵见 [gitea-pac.md](gitea-pac.md)。PaC status 是 migrated lane closeout 入口,PaC history 是触发/耗时/env reuse 审计入口,默认输出必须包含 `READ_ERRORS` 并在目标 node 聚合,不能把读取失败或 namespace 过大误报为空表成功。不用 Gitea Actions、act_runner、branch-follower自维护脚本兜底。`agentrun-jd01-v02` 是默认 consumer;哨兵使用 `--consumer sentinel-jd01-v03`HWLAB 使用 `--consumer hwlab-jd01-v03` 查看 PaC/Tekton/Argo/env reuse 证据。WeChat archive 是 platform-infra 的 YAML-first 工作流入口;只读 collector 的副本、镜像、WCF host、端口和版本 pin 都以 YAML 为准。
`platform-infra` 是 UniDesk 运维的平台基础设施控制面;新增平台服务优先进入该命名空间或对应 YAML 声明目标,旧 `devops-infra` 只作为渐进迁移来源。Sub2API 日常部署、Codex pool、FRP 暴露、master `~/.codex` 配置、验收和排障统一使用 `$unidesk-sub2api`。Gitea mirror 和 Pipelines-as-Code 是 JD01 migrated CI source/trigger 平台服务,source-of-truth 分别是 `config/platform-infra/gitea.yaml``config/platform-infra/pipelines-as-code.yaml`;正式架构和三 consumer 矩阵见 [gitea-pac.md](gitea-pac.md)。PaC closeout 是 migrated lane 正式交付入口,PaC status 是当前状态入口,PaC history 是触发/耗时/env reuse 审计入口,默认输出必须包含 `READ_ERRORS` 并在目标 node 聚合,不能把读取失败或 namespace 过大误报为空表成功。不用 Gitea Actions、act_runner、branch-follower自维护脚本或 consumer 专用手动 publish 兜底。`agentrun-jd01-v02` 是默认 consumer;哨兵使用 `--consumer sentinel-jd01-v03`HWLAB 使用 `--consumer hwlab-jd01-v03` 查看 PaC/Tekton/Argo/env reuse 证据。WeChat archive 是 platform-infra 的 YAML-first 工作流入口;只读 collector 的副本、镜像、WCF host、端口和版本 pin 都以 YAML 为准。
## CI Tools Image
+3 -2
View File
@@ -78,7 +78,8 @@ export function hwlabNodeWebProbeHelp(): Record<string, unknown> {
"bun scripts/cli.ts web-probe observe gc --node JD01 --lane v03 --keep-hours 24 --confirm",
"bun scripts/cli.ts web-probe sentinel plan --node D601 --lane v03 --dry-run",
"bun scripts/cli.ts web-probe sentinel plan --node D601 --lane v03 --sentinel workbench-auth-session-switch-2users",
"bun scripts/cli.ts web-probe sentinel publish-current --node JD01 --lane v03 --sentinel jd01-web-probe-sentinel --confirm --wait",
"bun scripts/cli.ts platform-infra pipelines-as-code closeout --target JD01 --consumer sentinel-jd01-v03 --source-commit <sha> --wait",
"bun scripts/cli.ts platform-infra pipelines-as-code status --target JD01 --consumer sentinel-jd01-v03",
"bun scripts/cli.ts web-probe sentinel dashboard verify --node D601 --lane v03 --sentinel workbench-dsflash-go-tool-call-10x",
"bun scripts/cli.ts web-probe sentinel dashboard screenshot --node D601 --lane v03 --sentinel workbench-auth-session-switch-2users",
"bun scripts/cli.ts web-probe sentinel dashboard trigger --node JD01 --lane v03 --sentinel jd01-web-probe-sentinel",
@@ -92,7 +93,7 @@ export function hwlabNodeWebProbeHelp(): Record<string, unknown> {
script: "Run caller-provided Playwright JS after CLI-managed /auth/login; scripts must not handle secrets themselves.",
screenshot: "Capture a no-auth or public page through the selected node/lane remote browser and download PNG artifacts to the caller /tmp by default.",
observe: "Start, inspect, control, stop, collect, analyze, and garbage-collect raw artifacts for long-running observers.",
sentinel: "Render and operate the YAML-first web-probe sentinel wrapper, one-click publish, image, GitOps, dashboard verification, maintenance and report views.",
sentinel: "Render and operate the YAML-first web-probe sentinel wrapper, image, GitOps, dashboard verification, maintenance and report views; migrated JD01 formal closeout uses platform-infra pipelines-as-code, while publish-current is debug/recovery only.",
},
notes: [
"Default URL, browser proxy mode, observe/analyze thresholds, and project-management command allowlist come from config/hwlab-node-lanes.yaml webProbe.",
@@ -57,6 +57,8 @@ export type WebProbeSentinelOptions =
readonly wait: boolean;
readonly timeoutSeconds: number;
readonly rerun: boolean;
readonly manualRecovery: boolean;
readonly reason: string | null;
readonly sourceOverride: WebProbeSentinelSourceOverrideOptions;
}
| {
@@ -451,11 +453,11 @@ export function renderPublishResult(publish: Record<string, unknown>): string {
"PUBLISH_DRILLDOWN",
` status: ${commands.cliStatus ?? "-"}`,
` logs: ${commands.logs ?? "-"}`,
` describe: ${commands.describe ?? "-"}`,
` publish-current: ${commands.publishCurrent ?? "-"}`,
` git-mirror: ${commands.gitMirrorStatus ?? "-"}`,
` sync: ${commands.gitMirrorSync ?? "-"}`,
` flush: ${commands.gitMirrorFlush ?? "-"}`,
` describe: ${commands.describe ?? "-"}`,
` manual-recovery: ${commands.publishCurrent ?? "-"}`,
` git-mirror: ${commands.gitMirrorStatus ?? "-"}`,
` sync: ${commands.gitMirrorSync ?? "-"}`,
` flush: ${commands.gitMirrorFlush ?? "-"}`,
` apply: ${commands.controlPlaneApply ?? "-"}`,
);
}
@@ -591,7 +593,8 @@ export function renderPublishCurrentResult(result: Record<string, unknown>): str
Object.keys(recoveryNext).length === 0 ? "RECOVERY_NEXT\n-" : [
"RECOVERY_NEXT",
table(["REASON", "PIPELINERUN", "DIGEST", "GITOPS"], [[recoveryNext.reason, recoveryNext.pipelineRun ?? "-", short(recoveryNext.digestRef), short(recoveryNext.gitopsCommit)]]),
` publish-current: ${recoveryNext.publishCurrent ?? "-"}`,
` pac-closeout: ${recoveryNext.pacCloseout ?? "-"}`,
` manual-recovery: ${recoveryNext.publishCurrent ?? "-"}`,
` status: ${recoveryNext.nextStatus ?? "-"}`,
` git-mirror: ${recoveryNext.gitMirrorStatus ?? "-"}`,
` sync: ${recoveryNext.gitMirrorSync ?? "-"}`,
@@ -600,7 +603,9 @@ export function renderPublishCurrentResult(result: Record<string, unknown>): str
].join("\n"),
"",
"NEXT",
` publish-current: ${next.publishCurrent ?? "-"}`,
` pac-closeout: ${next.pacCloseout ?? "-"}`,
` pac-status: ${next.pacStatus ?? "-"}`,
` pac-history: ${next.pacHistory ?? "-"}`,
` status: ${next.controlPlaneStatus ?? "-"}`,
` post-deploy-dashboard: ${next.dashboardVerify ?? "-"}`,
` git-mirror: ${next.gitMirrorStatus ?? "-"}`,
@@ -608,6 +613,8 @@ export function renderPublishCurrentResult(result: Record<string, unknown>): str
` flush: ${next.gitMirrorFlush ?? "-"}`,
"",
"DISCLOSURE",
" formal closeout for migrated JD01 sentinel CI/CD is platform-infra pipelines-as-code closeout/status/history.",
" publish-current is a debug/test/recovery step only; operator-side recovery must be explicit.",
` end-to-end and stage budgets are read from ${Object.keys(validationPlan).length > 0 ? "publishCurrent YAML and runtime.healthPath" : "YAML-required publishCurrent fields"}.`,
" CI/CD validation only checks the configured health endpoint; web-probe, Playwright and browser dashboard checks are post-deploy evidence, not this gate.",
" image build uses Tekton PipelineRun and BuildKit; this command does not require Docker daemon/socket/build.",
@@ -766,7 +773,8 @@ export function renderControlPlaneResult(result: Record<string, unknown>): strin
Object.keys(recoveryNext).length === 0 ? "RECOVERY_NEXT\n-" : [
"RECOVERY_NEXT",
table(["REASON", "PIPELINERUN", "DIGEST", "GITOPS"], [[recoveryNext.reason, recoveryNext.pipelineRun ?? "-", short(recoveryNext.digestRef), short(recoveryNext.gitopsCommit)]]),
` publish-current: ${recoveryNext.publishCurrent ?? "-"}`,
` pac-closeout: ${recoveryNext.pacCloseout ?? "-"}`,
` manual-recovery: ${recoveryNext.publishCurrent ?? "-"}`,
` status: ${recoveryNext.nextStatus ?? "-"}`,
` git-mirror: ${recoveryNext.gitMirrorStatus ?? "-"}`,
` sync: ${recoveryNext.gitMirrorSync ?? "-"}`,
@@ -775,6 +783,9 @@ export function renderControlPlaneResult(result: Record<string, unknown>): strin
].join("\n"),
"",
"NEXT",
` pac-closeout: ${next.pacCloseout ?? "-"}`,
` pac-status: ${next.pacStatus ?? "-"}`,
` pac-history: ${next.pacHistory ?? "-"}`,
` plan: ${next.plan ?? "-"}`,
` status: ${next.status ?? "-"}`,
` image: ${next.image ?? "-"}`,
+73 -10
View File
@@ -356,6 +356,52 @@ function runSentinelPublishCurrentConfirmedInner(state: SentinelCicdState, optio
const deadline = startedAt + budgetSeconds * 1000;
const remainingBudgetSeconds = () => strictRemainingSeconds(deadline, budgetSeconds);
let controlResult: Record<string, unknown> | null = null;
const inCluster = Boolean(process.env.KUBERNETES_SERVICE_HOST && process.env.KUBERNETES_SERVICE_PORT);
if (!inCluster && !options.manualRecovery) {
const next = publishCurrentNext(state);
const result = {
ok: false,
command,
node: state.spec.nodeId,
lane: state.spec.lane,
sentinelId: state.sentinelId,
mode: "debug-step-only-blocked",
mutation: false,
deliverySource: "auto-pac-required",
specRef: SPEC_REF,
source: state.sourceHead,
image: state.image,
pipelineRun: sentinelPipelineRunName(state, options.rerun),
budget,
validationPlan: publishCurrentHealthValidationPlan(state),
stageBudgets: publishCurrentStageBudgets(state),
warnings: [
"operator-side publish-current is debug/test/recovery only; formal delivery must use platform-infra pipelines-as-code closeout/status/history.",
],
blocker: {
code: "sentinel-publish-current-debug-step-only",
reason: "this operator-side command no longer performs formal Web sentinel delivery; use PaC closeout for automatic branch following",
valuesRedacted: true,
},
recoveryNext: {
reason: "manual recovery is only allowed after PaC is terminally failed, half-applied, or explicitly over budget",
pipelineRun: null,
digestRef: null,
gitopsCommit: null,
pacCloseout: next.pacCloseout,
publishCurrent: next.manualRecovery,
nextStatus: next.pacStatus,
gitMirrorStatus: next.gitMirrorStatus,
gitMirrorSync: null,
gitMirrorFlush: null,
controlPlaneApply: null,
valuesRedacted: true,
},
next,
valuesRedacted: true,
};
return rendered(false, command, renderPublishCurrentResult(result));
}
sentinelProgressEvent("sentinel.publish.progress", {
phase: "publish-current-start",
status: "running",
@@ -365,7 +411,7 @@ function runSentinelPublishCurrentConfirmedInner(state: SentinelCicdState, optio
lane: state.spec.lane,
sentinelId: state.sentinelId,
});
if (process.env.KUBERNETES_SERVICE_HOST && process.env.KUBERNETES_SERVICE_PORT) {
if (inCluster) {
const publish = runSentinelPublishJob(state, true, Math.max(1, remainingBudgetSeconds()), options.rerun);
const elapsedMs = Date.now() - startedAt;
const ok = record(publish).ok === true;
@@ -1049,8 +1095,13 @@ function publishCurrentNext(state: SentinelCicdState): Record<string, string> {
const node = state.spec.nodeId;
const lane = state.spec.lane;
const suffix = sentinelCliSuffix(state);
const consumer = sentinelPacConsumerId(state);
const sourceCommit = state.sourceHead.commit === null ? "" : ` --source-commit ${state.sourceHead.commit}`;
return {
publishCurrent: `bun scripts/cli.ts web-probe sentinel publish-current --node ${node} --lane ${lane}${suffix} --confirm --wait`,
pacCloseout: `bun scripts/cli.ts platform-infra pipelines-as-code closeout --target ${node} --consumer ${consumer}${sourceCommit} --wait`,
pacStatus: `bun scripts/cli.ts platform-infra pipelines-as-code status --target ${node} --consumer ${consumer}`,
pacHistory: `bun scripts/cli.ts platform-infra pipelines-as-code history --target ${node} --consumer ${consumer} --limit 10`,
manualRecovery: `bun scripts/cli.ts web-probe sentinel publish-current --node ${node} --lane ${lane}${suffix} --confirm --wait --manual-recovery --reason <reason>`,
controlPlaneStatus: `bun scripts/cli.ts web-probe sentinel control-plane status --node ${node} --lane ${lane}${suffix}`,
dashboardVerify: `bun scripts/cli.ts web-probe sentinel dashboard verify --node ${node} --lane ${lane}${suffix}`,
gitMirrorStatus: `bun scripts/cli.ts hwlab nodes git-mirror status --node ${node} --lane ${lane}`,
@@ -1059,6 +1110,10 @@ function publishCurrentNext(state: SentinelCicdState): Record<string, string> {
};
}
function sentinelPacConsumerId(state: SentinelCicdState): string {
return `sentinel-${state.spec.nodeId.toLowerCase()}-${state.spec.lane}`;
}
function renderSentinelManifests(
spec: HwlabRuntimeLaneSpec,
sentinelId: string,
@@ -1963,7 +2018,7 @@ function sentinelObservedStatusDiagnosis(state: SentinelCicdState, value: unknow
? "sentinel-cadence-not-ready"
: "sentinel-control-plane-observed-not-ready";
const reason = code === "sentinel-publish-half-state-registry-missing"
? "source mirror contains the selected commit, but the expected registry tag is missing; retry publish-current so Tekton builds/pushes the image and advances GitOps/runtime"
? "source mirror contains the selected commit, but the expected registry tag is missing; inspect PaC closeout/history before considering explicit manual recovery"
: code === "sentinel-gitops-manifest-not-updated"
? "registry contains the selected image, but GitOps manifest is not yet updated to a digest-pinned runtime image"
: code === "sentinel-runtime-not-aligned"
@@ -1998,7 +2053,7 @@ function sentinelObservedStatusDiagnosis(state: SentinelCicdState, value: unknow
runtime: `ready=${runtimeDeployment.readyReplicas ?? "-"}/${runtimeDeployment.desiredReplicas ?? "-"} image=${short(runtimeDeployment.image)} expected=${short(runtimeDeployment.expectedImage)}`,
pipelineRun,
warning: code === "sentinel-publish-half-state-registry-missing"
? `source mirror already exposes ${short(state.sourceHead.commit)} but registry tag ${state.image.tag} is missing; rerun ${next.publishCurrent} and then recheck ${next.controlPlaneStatus}.`
? `source mirror already exposes ${short(state.sourceHead.commit)} but registry tag ${state.image.tag} is missing; run ${next.pacCloseout} and inspect ${next.pacHistory}.`
: code === "sentinel-git-mirror-not-in-sync"
? `runtime is aligned but git-mirror is not in sync; run ${next.gitMirrorSync} and then recheck ${next.controlPlaneStatus}.`
: null,
@@ -2008,8 +2063,9 @@ function sentinelObservedStatusDiagnosis(state: SentinelCicdState, value: unknow
pipelineRun,
digestRef: registryPresent ? expectedRuntimeImageFromRegistry(state, record(observed.registry)) : null,
gitopsCommit: gitops.revision ?? null,
publishCurrent: shouldRetryPublish ? next.publishCurrent : null,
nextStatus: next.controlPlaneStatus,
pacCloseout: next.pacCloseout,
publishCurrent: shouldRetryPublish ? next.manualRecovery : null,
nextStatus: next.pacStatus,
gitMirrorStatus: next.gitMirrorStatus,
gitMirrorSync: !gitMirrorReady ? next.gitMirrorSync : null,
gitMirrorFlush: null,
@@ -2389,8 +2445,9 @@ function installSentinelPublishInterruptHandler(state: SentinelCicdState, contex
lane: state.spec.lane,
sentinelId: state.sentinelId,
recoveryNext: {
status: next.controlPlaneStatus,
retry: next.publishCurrent,
pacCloseout: next.pacCloseout,
pacStatus: next.pacStatus,
retry: next.manualRecovery,
gitMirrorStatus: next.gitMirrorStatus,
gitMirrorSync: next.gitMirrorSync,
gitMirrorFlush: next.gitMirrorFlush,
@@ -2428,7 +2485,12 @@ function controlPlaneNext(state: SentinelCicdState, action: WebProbeSentinelCont
const node = state.spec.nodeId;
const lane = state.spec.lane;
const suffix = sentinelCliSuffix(state);
const consumer = sentinelPacConsumerId(state);
const sourceCommit = state.sourceHead.commit === null ? "" : ` --source-commit ${state.sourceHead.commit}`;
return {
pacCloseout: `bun scripts/cli.ts platform-infra pipelines-as-code closeout --target ${node} --consumer ${consumer}${sourceCommit} --wait`,
pacStatus: `bun scripts/cli.ts platform-infra pipelines-as-code status --target ${node} --consumer ${consumer}`,
pacHistory: `bun scripts/cli.ts platform-infra pipelines-as-code history --target ${node} --consumer ${consumer} --limit 10`,
plan: `bun scripts/cli.ts web-probe sentinel control-plane plan --node ${node} --lane ${lane}${suffix} --dry-run`,
status: `bun scripts/cli.ts web-probe sentinel control-plane status --node ${node} --lane ${lane}${suffix}`,
image: `bun scripts/cli.ts web-probe sentinel image status --node ${node} --lane ${lane}${suffix}`,
@@ -2457,8 +2519,9 @@ function controlPlaneRecoveryNext(state: SentinelCicdState, ok: boolean, publish
gitopsCommit: payload.gitopsCommit ?? null,
flushMode: flushRecord.mode ?? null,
observedReady: sentinelObservedReady(observedRecord),
publishCurrent: `bun scripts/cli.ts web-probe sentinel publish-current --node ${state.spec.nodeId} --lane ${state.spec.lane}${sentinelCliSuffix(state)} --confirm --wait`,
nextStatus: next.status,
pacCloseout: next.pacCloseout,
publishCurrent: `bun scripts/cli.ts web-probe sentinel publish-current --node ${state.spec.nodeId} --lane ${state.spec.lane}${sentinelCliSuffix(state)} --confirm --wait --manual-recovery --reason <reason>`,
nextStatus: next.pacStatus,
gitMirrorStatus: next.gitMirrorStatus,
gitMirrorSync: next.gitMirrorSync,
gitMirrorFlush: next.gitMirrorFlush,
+5 -2
View File
@@ -81,7 +81,7 @@ export function parseNodeWebProbeSentinelOptions(args: string[]): NodeWebProbeSe
"--source-stage-ref",
"--source-mirror-commit",
"--source-authority",
]), new Set(["--dry-run", "--confirm", "--wait", "--rerun", "--quick-verify", "--raw", "--full", "--latest", "--full-page", "--no-full-page"]));
]), new Set(["--dry-run", "--confirm", "--wait", "--rerun", "--manual-recovery", "--recovery", "--quick-verify", "--raw", "--full", "--latest", "--full-page", "--no-full-page"]));
const node = requiredOption(args, "--node");
assertNodeId(node);
const lane = requiredOption(args, "--lane");
@@ -108,7 +108,10 @@ export function parseNodeWebProbeSentinelOptions(args: string[]): NodeWebProbeSe
}
sentinel = { kind: "control-plane", action: controlPlaneAction, node, lane, sentinelId, dryRun: controlPlaneAction === "apply" || controlPlaneAction === "trigger-current" ? dryRun || !confirm : dryRun, confirm, wait: args.includes("--wait"), timeoutSeconds, rerun: args.includes("--rerun"), sourceOverride };
} else if (sentinelActionRaw === "publish-current") {
sentinel = { kind: "publish", action: "publish-current", node, lane, sentinelId, dryRun: dryRun || !confirm, confirm, wait: args.includes("--wait"), timeoutSeconds, rerun: args.includes("--rerun"), sourceOverride };
const manualRecovery = args.includes("--manual-recovery") || args.includes("--recovery");
const reason = optionValue(args, "--reason") ?? null;
if (manualRecovery && (reason === null || reason.trim().length < 8)) throw new Error("web-probe sentinel publish-current manual recovery requires --reason with at least 8 characters");
sentinel = { kind: "publish", action: "publish-current", node, lane, sentinelId, dryRun: dryRun || !confirm, confirm, wait: args.includes("--wait"), timeoutSeconds, rerun: args.includes("--rerun"), manualRecovery, reason, sourceOverride };
} else if (sentinelActionRaw === "maintenance") {
const maintenanceAction = args[1];
if (maintenanceAction !== "status" && maintenanceAction !== "start" && maintenanceAction !== "stop") {
+153 -1
View File
@@ -124,6 +124,11 @@ interface HistoryOptions extends CommonOptions {
detailId: string | null;
}
interface CloseoutOptions extends CommonOptions {
sourceCommit: string | null;
wait: boolean;
}
interface ApplyOptions extends CommonOptions {
confirm: boolean;
dryRun: boolean;
@@ -161,6 +166,11 @@ export async function runPlatformInfraPipelinesAsCodeCommand(config: UniDeskConf
const result = await status(config, options);
return options.full || options.raw ? result : renderStatus(result);
}
if (action === "closeout") {
const options = parseCloseoutOptions(args.slice(1));
const result = await closeout(config, options);
return options.full || options.raw ? result : renderCloseout(result);
}
if (action === "history" || action === "runs") {
const options = parseHistoryOptions(args.slice(1));
const result = await history(config, options);
@@ -176,13 +186,14 @@ export async function runPlatformInfraPipelinesAsCodeCommand(config: UniDeskConf
function help(): Record<string, unknown> {
return {
command: "platform-infra pipelines-as-code plan|apply|status|history|webhook-test",
command: "platform-infra pipelines-as-code plan|apply|status|closeout|history|webhook-test",
configTruth: configLabel,
usage: [
"bun scripts/cli.ts platform-infra pipelines-as-code plan --target JD01",
"bun scripts/cli.ts platform-infra pipelines-as-code apply --target JD01 --dry-run",
"bun scripts/cli.ts platform-infra pipelines-as-code apply --target JD01 --confirm",
"bun scripts/cli.ts platform-infra pipelines-as-code status --target JD01 [--full|--raw]",
"bun scripts/cli.ts platform-infra pipelines-as-code closeout --target JD01 --consumer <consumer> --source-commit <sha> --wait",
"bun scripts/cli.ts platform-infra pipelines-as-code history --target JD01 [--consumer hwlab-jd01-v03] [--limit 10]",
"bun scripts/cli.ts platform-infra pipelines-as-code history --target JD01 --id <pipelinerun>",
"bun scripts/cli.ts platform-infra pipelines-as-code webhook-test --target JD01 --confirm",
@@ -413,6 +424,52 @@ async function status(config: UniDeskConfig, options: CommonOptions): Promise<Re
};
}
async function closeout(config: UniDeskConfig, options: CloseoutOptions): Promise<Record<string, unknown>> {
const pac = readPacConfig();
const target = resolveTarget(pac, options.targetId);
const consumer = resolveConsumer(pac, options.consumerId);
const startedAt = Date.now();
const waitMs = Math.max(1, pac.release.waitTimeoutSeconds) * 1000;
let attempts = 0;
let current = await status(config, options);
attempts += 1;
while (options.wait && !closeoutReady(current, options.sourceCommit) && Date.now() - startedAt < waitMs) {
await sleep(3000);
current = await status(config, options);
attempts += 1;
}
const summary = record(current.summary);
const latest = record(summary.latestPipelineRun);
const diagnostics = record(summary.diagnostics);
const observedSourceCommit = stringValue(latest.sourceCommit) !== "-" ? stringValue(latest.sourceCommit) : stringValue(diagnostics.sourceCommit);
const sourceMatched = options.sourceCommit === null || observedSourceCommit === options.sourceCommit;
const ready = current.ok === true && sourceMatched;
return {
ok: ready,
action: "platform-infra-pipelines-as-code-closeout",
mutation: false,
target: targetSummary(target),
consumer,
sourceCommit: options.sourceCommit,
observedSourceCommit,
sourceMatched,
ready,
wait: {
requested: options.wait,
attempts,
elapsedMs: Date.now() - startedAt,
budgetSeconds: pac.release.waitTimeoutSeconds,
source: `${configLabel}.release.waitTimeoutSeconds`,
valuesPrinted: false,
},
summary,
status: current,
blocker: ready ? null : closeoutBlocker(current, options.sourceCommit, observedSourceCommit, sourceMatched),
next: nextCommands(target.id, consumer.id, pac.defaults.consumerId),
valuesPrinted: false,
};
}
async function history(config: UniDeskConfig, options: HistoryOptions): Promise<Record<string, unknown>> {
const pac = readPacConfig();
const target = resolveTarget(pac, options.targetId);
@@ -721,6 +778,7 @@ function nextCommands(targetId: string, consumerId: string, defaultConsumerId: s
const suffix = consumerSuffix(consumerId, defaultConsumerId);
return {
apply: `bun scripts/cli.ts platform-infra pipelines-as-code apply --target ${targetId}${suffix} --confirm`,
closeout: `bun scripts/cli.ts platform-infra pipelines-as-code closeout --target ${targetId}${suffix} --wait`,
status: `bun scripts/cli.ts platform-infra pipelines-as-code status --target ${targetId}${suffix}`,
history: `bun scripts/cli.ts platform-infra pipelines-as-code history --target ${targetId}${suffix}`,
webhookTest: `bun scripts/cli.ts platform-infra pipelines-as-code webhook-test --target ${targetId}${suffix} --confirm`,
@@ -819,12 +877,50 @@ function renderStatus(result: Record<string, unknown>): RenderedCliResult {
` pipeline-run: ${latest.name === undefined ? "-" : `bun scripts/cli.ts platform-infra pipelines-as-code history --target ${stringValue(record(result.target).id)} --id ${stringValue(latest.name)}`}`,
"",
"NEXT",
` closeout: ${stringValue(record(result.next).closeout)}`,
` full: ${stringValue(record(result.next).status)} --full`,
` all-history: ${stringValue(record(result.next).history).replace(/ --consumer [^ ]+/u, "")} --limit 10`,
];
return rendered(result, "platform-infra pipelines-as-code status", lines);
}
function renderCloseout(result: Record<string, unknown>): RenderedCliResult {
const target = record(result.target);
const consumer = record(result.consumer);
const wait = record(result.wait);
const summary = record(result.summary);
const latest = record(summary.latestPipelineRun);
const artifact = record(summary.artifact);
const argo = record(summary.argo);
const runtime = record(summary.runtime);
const diagnostics = record(summary.diagnostics);
const blocker = record(result.blocker);
const lines = [
"PLATFORM-INFRA PIPELINES-AS-CODE CLOSEOUT",
...table(["TARGET", "CONSUMER", "READY", "SOURCE_MATCHED", "MUTATION"], [[stringValue(target.id), stringValue(consumer.id), boolText(result.ready), boolText(result.sourceMatched), boolText(result.mutation)]]),
"",
"WAIT",
...table(["REQUESTED", "ATTEMPTS", "ELAPSED_MS", "BUDGET_S", "SOURCE"], [[stringValue(wait.requested), stringValue(wait.attempts), stringValue(wait.elapsedMs), stringValue(wait.budgetSeconds), stringValue(wait.source)]]),
"",
"SOURCE / PIPELINERUN",
...table(["EXPECTED", "OBSERVED", "PIPELINERUN", "STATUS", "DURATION_S"], [[short(stringValue(result.sourceCommit), 16), short(stringValue(result.observedSourceCommit), 16), stringValue(latest.name), stringValue(latest.reason ?? latest.status), stringValue(latest.durationSeconds)]]),
"",
"RUNTIME",
...table(["IMAGE_STATUS", "ENV_REUSE", "DIGEST", "GITOPS", "ARGO", "RUNTIME"], [[stringValue(artifact.imageStatus), stringValue(artifact.envReuse), short(stringValue(artifact.digest), 18), short(stringValue(artifact.gitopsCommit), 12), `${stringValue(argo.sync)}/${stringValue(argo.health)}`, runtimeText(runtime)]]),
"",
"CICD DIAGNOSIS",
...table(["OK", "CODE", "PHASE", "HINT"], [[boolText(diagnostics.ok !== false), stringValue(diagnostics.code), stringValue(diagnostics.phase), compactLine(stringValue(diagnostics.hint))]]),
"",
Object.keys(blocker).length === 0 ? "BLOCKER\n-" : ["BLOCKER", table(["CODE", "REASON"], [[stringValue(blocker.code), compactLine(stringValue(blocker.reason))]])].join("\n"),
"",
"NEXT",
` status: ${stringValue(record(result.next).status)}`,
` history: ${stringValue(record(result.next).history)} --limit 10`,
latest.name === undefined ? " pipeline-run: -" : ` pipeline-run: ${stringValue(record(result.next).history)} --id ${stringValue(latest.name)}`,
];
return rendered(result, "platform-infra pipelines-as-code closeout", lines);
}
function renderHistory(result: Record<string, unknown>): RenderedCliResult {
const rows = arrayRecords(result.rows);
const config = record(result.config);
@@ -912,6 +1008,31 @@ function parseWebhookTestOptions(args: string[]): WebhookTestOptions {
return { ...parseCommonOptions(commonArgs), confirm };
}
function parseCloseoutOptions(args: string[]): CloseoutOptions {
const commonArgs: string[] = [];
let sourceCommit: string | null = null;
let wait = false;
for (let index = 0; index < args.length; index += 1) {
const arg = args[index];
if (arg === "--source-commit" || arg === "--commit") {
const value = args[index + 1];
if (value === undefined || value.startsWith("--")) throw new Error(`${arg} requires a value`);
if (!/^[0-9a-f]{40}$/iu.test(value)) throw new Error(`${arg} must be a full git sha`);
sourceCommit = value;
index += 1;
} else if (arg === "--wait") {
wait = true;
} else {
commonArgs.push(arg);
if (arg === "--target" || arg === "--node" || arg === "--consumer") {
commonArgs.push(args[index + 1] ?? "");
index += 1;
}
}
}
return { ...parseCommonOptions(commonArgs), sourceCommit, wait };
}
function parseHistoryOptions(args: string[]): HistoryOptions {
const commonArgs: string[] = [];
let limit = 5;
@@ -968,6 +1089,37 @@ function parseCommonOptions(args: string[]): CommonOptions {
return { targetId, consumerId, full, raw };
}
function closeoutReady(value: Record<string, unknown>, sourceCommit: string | null): boolean {
if (value.ok !== true) return false;
if (sourceCommit === null) return true;
const summary = record(value.summary);
const latest = record(summary.latestPipelineRun);
const diagnostics = record(summary.diagnostics);
const observedSourceCommit = stringValue(latest.sourceCommit) !== "-" ? stringValue(latest.sourceCommit) : stringValue(diagnostics.sourceCommit);
return observedSourceCommit === sourceCommit;
}
function closeoutBlocker(value: Record<string, unknown>, expected: string | null, observed: string, sourceMatched: boolean): Record<string, unknown> {
if (!sourceMatched) {
return {
code: "pac-closeout-source-mismatch",
reason: `expected source commit ${expected ?? "-"} but latest observed source commit is ${observed}`,
valuesPrinted: false,
};
}
const summary = record(value.summary);
const diagnostics = record(summary.diagnostics);
return {
code: stringValue(diagnostics.code, "pac-closeout-not-ready"),
reason: stringValue(diagnostics.hint, "PaC status is not ready; inspect status/history for the selected consumer"),
valuesPrinted: false,
};
}
function sleep(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
}
function positiveInteger(obj: Record<string, unknown>, key: string, path: string): number {
const value = y.integerField(obj, key, path);
if (value < 1) throw new Error(`${configLabel}.${path}.${key} must be positive`);
+5 -2
View File
@@ -369,7 +369,7 @@ export interface ManagedResourceCleanupPlan {
export function platformInfraHelp(): unknown {
const target = sub2ApiHelpTargetSummary();
return {
command: "platform-infra sub2api|langbot|n8n|wechat-archive|observability|secret-plane|kafka|gitea ...",
command: "platform-infra sub2api|langbot|n8n|wechat-archive|observability|secret-plane|kafka|gitea|pipelines-as-code ...",
output: "json",
usage: [
"bun scripts/cli.ts platform-infra sub2api plan [--target G14|D601]",
@@ -439,8 +439,11 @@ export function platformInfraHelp(): unknown {
"bun scripts/cli.ts platform-infra gitea mirror bootstrap --target JD01 --confirm",
"bun scripts/cli.ts platform-infra gitea mirror sync --target JD01 --confirm",
"bun scripts/cli.ts platform-infra gitea mirror status --target JD01",
"bun scripts/cli.ts platform-infra pipelines-as-code closeout --target JD01 --consumer <consumer> --source-commit <sha> --wait",
"bun scripts/cli.ts platform-infra pipelines-as-code status --target JD01 --consumer <consumer>",
"bun scripts/cli.ts platform-infra pipelines-as-code history --target JD01 --consumer <consumer> --limit 10",
],
description: "Operate YAML-controlled platform-infra services such as Sub2API, LangBot, n8n, WeChat archive workflows, OpenTelemetry tracing, the independent target-scoped secret plane, the D518 Kafka event bus, and the internal Gitea source-authority service for GH-1548/GH-1549. Public services use PK01 Caddy+FRP rather than Kubernetes Ingress, NodePort, or LoadBalancer.",
description: "Operate YAML-controlled platform-infra services such as Sub2API, LangBot, n8n, WeChat archive workflows, OpenTelemetry tracing, the independent target-scoped secret plane, the D518 Kafka event bus, internal Gitea source authority, and Gitea + Pipelines-as-Code closeout for JD01 migrated consumers. Public services use PK01 Caddy+FRP rather than Kubernetes Ingress, NodePort, or LoadBalancer.",
target,
codexPool: {
usage: [