From 0aab8d35ade7ac5c4bd6669c73c403e743a673a4 Mon Sep 17 00:00:00 2001 From: Codex Date: Sat, 11 Jul 2026 06:10:20 +0200 Subject: [PATCH 1/5] feat(cicd): generate PaC source artifacts from YAML --- .agents/skills/unidesk-cicd/SKILL.md | 4 + .../unidesk-cicd/references/gitea-pac.md | 22 + config/platform-infra/pipelines-as-code.yaml | 16 + .../R1.1_Task_Report.md | 22 + .../R1.2_Task_Report.md | 23 + .../MDTODO/yaml-owned-pac-source-artifacts.md | 22 + .../cicd/pac-source-artifact-runtime.mjs | 238 +++++ scripts/src/agentrun-manifests.ts | 11 +- scripts/src/help.ts | 1 + scripts/src/hwlab-node/deploy-overlay.ts | 66 ++ scripts/src/hwlab-node/render.ts | 45 +- scripts/src/hwlab-node/web-probe.ts | 7 + ...-pipelines-as-code-source-artifact.test.ts | 358 +++++++ ...infra-pipelines-as-code-source-artifact.ts | 909 ++++++++++++++++++ .../src/platform-infra-pipelines-as-code.ts | 187 +++- scripts/src/stable-json.ts | 12 + 16 files changed, 1901 insertions(+), 42 deletions(-) create mode 100644 docs/MDTODO/details/yaml-owned-pac-source-artifacts/R1.1_Task_Report.md create mode 100644 docs/MDTODO/details/yaml-owned-pac-source-artifacts/R1.2_Task_Report.md create mode 100644 docs/MDTODO/yaml-owned-pac-source-artifacts.md create mode 100644 scripts/native/cicd/pac-source-artifact-runtime.mjs create mode 100644 scripts/src/hwlab-node/deploy-overlay.ts create mode 100644 scripts/src/platform-infra-pipelines-as-code-source-artifact.test.ts create mode 100644 scripts/src/platform-infra-pipelines-as-code-source-artifact.ts create mode 100644 scripts/src/stable-json.ts diff --git a/.agents/skills/unidesk-cicd/SKILL.md b/.agents/skills/unidesk-cicd/SKILL.md index 54cec7ba..88c210a7 100644 --- a/.agents/skills/unidesk-cicd/SKILL.md +++ b/.agents/skills/unidesk-cicd/SKILL.md @@ -27,6 +27,9 @@ bun scripts/cli.ts platform-infra pipelines-as-code history --target JD01 --limi bun scripts/cli.ts platform-infra pipelines-as-code history --target JD01 --id bun scripts/cli.ts platform-infra pipelines-as-code debug-step --target JD01 --consumer bun scripts/cli.ts platform-infra pipelines-as-code debug-step --target JD01 --consumer --id +bun scripts/cli.ts platform-infra pipelines-as-code source-artifact check --target NC01 --consumer --source-worktree +bun scripts/cli.ts platform-infra pipelines-as-code source-artifact write --target NC01 --consumer --source-worktree --confirm +bun scripts/cli.ts platform-infra pipelines-as-code source-artifact verify-runtime --target NC01 --consumer --source-worktree --source-commit ``` 节点级只读状态必须优先用 `cicd status --node `。它从 `config/platform-infra/pipelines-as-code.yaml` 找到该 node 的所有当前 PaC consumer，一次性汇总 PipelineRun、Argo/GitOps、runtime readiness 和诊断；不要再靠阅读源码或手动拼三条 consumer 命令来回答 “NC01 的 CI/CD 流水线情况”。`platform-infra pipelines-as-code status --target --consumer ` 只作为单 consumer drill-down。 @@ -48,6 +51,7 @@ bun scripts/cli.ts platform-infra pipelines-as-code debug-step --target JD01 --c - CI/CD source authority 只能来自 YAML 声明的 Kubernetes 托管 source authority：legacy lane 使用 k8s git-mirror snapshot，Gitea/PaC migrated lane 使用 GitHub PR merge -> GitHub webhook bridge -> Gitea controlled mirror + immutable snapshot ref -> Pipelines-as-Code。受控命令先在 k8s 内同步/创建不可变 `refs/unidesk/snapshots/.../` stage ref；build/status/publish 只消费该 snapshot，host worktree、本地 `git fetch/pull`、可变 branch ref 或 Pipeline 内直连 GitHub 都不能作为 authoritative source。 - JD01/NC01 `agentrun--v02`、`sentinel--v03`、`hwlab--v03` 的正式 CI/CD closeout 入口是 `cicd status --node ` 和 `platform-infra pipelines-as-code closeout|status|history --target --consumer `。`closeout` 是通用 consumer 引导入口，只读取/等待 PaC、Tekton、GitOps、Argo 和 runtime 对齐，不触发旧手动发布。`cicd branch-follower` 和 `cicd gitea-actions-poc` 对这些 consumer 只保留历史/迁移只读用途，不得作为当前交付判断入口。 - PaC `.tekton` 文件必须用 Repository CR 的 target/node 参数隔离 JD01/NC01，避免同一个 Gitea push 在一个 target cluster 内额外创建另一个 target 的 PipelineRun；history/detail 必须按实际 PipelineRun prefix/pipeline 归属 consumer。 +- PaC source artifact 必须由 `config/platform-infra/pipelines-as-code.yaml#consumers[].sourceArtifact` 显式声明，并通过 `source-artifact plan|check|write|status|verify-runtime` 管理；desired 只来自 owning YAML 与共享 renderer，禁止从 live Pipeline/PipelineRun 反向导出。`verify-runtime` 必须绑定完整 source commit，未声明 owner 或证据冲突时 fail-closed。 - PaC source-only closeout 必须使用目标侧结构化证据： - `g14-ci-plan` 的 source commit、affected/build/rollout 计数必须完整； - `collect-artifacts` 与 `gitops-promote` 必须同时明确 `no-build-no-rollout-plan`； diff --git a/.agents/skills/unidesk-cicd/references/gitea-pac.md b/.agents/skills/unidesk-cicd/references/gitea-pac.md index a2acc7ad..51132ba6 100644 --- a/.agents/skills/unidesk-cicd/references/gitea-pac.md +++ b/.agents/skills/unidesk-cicd/references/gitea-pac.md @@ -18,6 +18,28 @@ GitHub remains the upstream write authority. A delivery should be triggered by m - PaC Repository CR `spec.url` matches the URL in Gitea webhook payloads. Keep `config/platform-infra/pipelines-as-code.yaml#repositories[].url` on the public Gitea repository URL, and keep k8s-internal service URLs only in `cloneUrl` or `params.git_read_url`; otherwise PaC logs `cannot find a repository match` and no PipelineRun is created. - Repositories shared by JD01 and NC01 must pass a target-specific `node` Repository param, and each `.tekton` PipelineRun must filter on that param with `pipelinesascode.tekton.dev/on-cel-expression`. A push for one target must not create the other target's PipelineRun in the same target cluster. +## PaC 源码侧制品同步 + +当 owning Pipeline 与消费仓 `.tekton` 内联 `pipelineSpec` 或远程 Pipeline 文件存在漂移时，统一使用以下入口： + +```bash +bun scripts/cli.ts platform-infra pipelines-as-code source-artifact plan --target --consumer --source-worktree +bun scripts/cli.ts platform-infra pipelines-as-code source-artifact check --target --consumer --source-worktree +bun scripts/cli.ts platform-infra pipelines-as-code source-artifact write --target --consumer --source-worktree --confirm +bun scripts/cli.ts platform-infra pipelines-as-code source-artifact status --target --consumer --source-worktree [--source-commit ] +bun scripts/cli.ts platform-infra pipelines-as-code source-artifact verify-runtime --target --consumer --source-worktree --source-commit +``` + +- `config/platform-infra/pipelines-as-code.yaml#consumers[].sourceArtifact` 是生成职责的声明入口；未声明的 consumer 必须稳定拒绝，模板不得替它发明 owner。 +- desired 只来自 owning YAML 与共享 domain renderer；live Pipeline 和 PipelineRun 只能作为只读诊断证据，禁止从运行面反向导出 desired 或源码制品。 +- `plan`、`check`、`write` 只比较或更新显式消费仓 worktree，不访问运行面；worktree 必须是绝对路径、Git 根目录，并与声明仓库的精确 remote identity 一致。 +- `write` 先完成全部路径、内容和 renderer 合同校验，再通过同目录临时文件与 rename 更新；连续执行必须无差异。 +- `status` 是非门禁诊断，允许不传 commit；`verify-runtime` 必须传完整四十位 `--source-commit`，并按 PaC/source-commit label 或 annotation 精确筛选 PipelineRun，不得按 prefix 直接取全局 latest。 +- 同一 source commit 因 webhook 重投或 PaC retry 产生多个 PipelineRun 时，只在完整内联 spec 与 provenance 全部一致时确定性选择最新一条并披露候选数；存在冲突必须 fail-closed。 +- runtime observer 必须区分对象 `NotFound`、transport/RBAC/命令不可用和 spec drift；完整大 spec 通过受控临时文件传入目标侧原生 observer，不得放入 shell argv 或环境变量。 +- 完整 spec 比较只移除六类已验证的 Tekton admission default，并限定在裸 Pipeline spec、完整 Pipeline 或 PipelineRun `pipelineSpec` 三个明确根；其他同名字段不得过滤。 +- HWLAB source artifact 必须保留正式 postprocess、verify 和六个 Kafka refresh 环境合同；AgentRun source artifact 必须保留 owning Pipeline 的完整 RBAC、参数和 manager 环境合同。 + ## Coverage Matrix | Consumer | Source | PaC namespace | Pipeline | Argo application | Status | diff --git a/config/platform-infra/pipelines-as-code.yaml b/config/platform-infra/pipelines-as-code.yaml index 74206148..3f537f05 100644 --- a/config/platform-infra/pipelines-as-code.yaml +++ b/config/platform-infra/pipelines-as-code.yaml @@ -104,6 +104,12 @@ consumers: variables: NODE: NC01 LANE: v02 + sourceArtifact: + mode: embedded-pipeline-spec + renderer: agentrun-control-plane + configRef: config/agentrun.yaml#controlPlane.lanes.nc01-v02 + pipelineRunPath: .tekton/agentrun-nc01-v02.yaml + maxKeepRuns: 8 - extends: templates.consumers.sentinelV03 variables: NODE: JD01 @@ -138,6 +144,13 @@ consumers: variables: NODE: NC01 LANE: v03 + sourceArtifact: + mode: remote-pipeline-annotation + renderer: hwlab-runtime-lane + configRef: config/hwlab-node-lanes.yaml#lanes.v03.targets.NC01 + pipelinePath: ci/pipelines/hwlab-nc01-v03-ci-image-publish.yaml + pipelineRunPath: .tekton/hwlab-nc01-v03-pac.yaml + maxKeepRuns: 8 templates: repositories: agentrunV02: @@ -212,6 +225,9 @@ templates: pipeline_name: "hwlab-${nodeLower}-v03-ci-image-publish" pipeline_run_prefix: "hwlab-${nodeLower}-v03-ci-poll" service_account: "hwlab-${nodeLower}-v03-tekton-runner" + workspace_pvc_size: 8Gi + git_ssh_secret: hwlab-git-ssh + pipeline_timeout: 1h0m0s node: "${NODE}" lane: v03 runtime_namespace: hwlab-v03 diff --git a/docs/MDTODO/details/yaml-owned-pac-source-artifacts/R1.1_Task_Report.md b/docs/MDTODO/details/yaml-owned-pac-source-artifacts/R1.1_Task_Report.md new file mode 100644 index 00000000..06c3d4a0 --- /dev/null +++ b/docs/MDTODO/details/yaml-owned-pac-source-artifacts/R1.1_Task_Report.md @@ -0,0 +1,22 @@ +# R1.1 任务报告 + +关联问题：[UniDesk #1745](https://github.com/pikasTech/unidesk/issues/1745)。 + +## 完成内容 + +- 在 `config/platform-infra/pipelines-as-code.yaml` 为 NC01 AgentRun 与 HWLAB consumer 声明独立 `sourceArtifact` owner；未声明的 JD01 consumer 保持 fail-closed。 +- 实现 `plan`、`check`、`write`、`status`、`verify-runtime` 五个受控动作，要求显式 target、consumer 和绝对 worktree。 +- AgentRun 复用正式 control-plane Pipeline renderer 生成内联 `pipelineSpec`；HWLAB 复用正式 runtime lane renderer、deploy overlay、postprocess 和 verify 生成远程 Pipeline。 +- owning Pipeline、源码侧制品和运行面共用 config ref、effective config hash、renderer 与 mode provenance。 +- runtime verify 强制绑定完整 source commit，并精确筛选同 commit PipelineRun；重复交付仅在 spec 与 provenance 一致时确定性选取，冲突 fail-closed。 +- 远端 observer 已抽到原生 `.mjs`；完整大 spec 通过目标侧临时文件传递，避免 shell argv/env 的 `ARG_MAX`。 + +## 验证证据 + +- AgentRun `plan` 成功识别旧内联 spec 漂移。 +- HWLAB `plan` 与 `check` 稳定得到 desired `3fb6e00ba833`、source `7d51de639afa`，且不写消费仓。 +- HWLAB e76 source commit 的真实 `status` 精确返回 live `3fb6e00ba833`、embedded `7d51de639afa` 和单一 PipelineRun；`verify-runtime` 对旧 embedded drift 以失败退出。 + +## 未包含范围 + +- 本项不更新消费仓 artifact，不触发 rollout，也不从 live 反向导出 desired。 diff --git a/docs/MDTODO/details/yaml-owned-pac-source-artifacts/R1.2_Task_Report.md b/docs/MDTODO/details/yaml-owned-pac-source-artifacts/R1.2_Task_Report.md new file mode 100644 index 00000000..f3881f0f --- /dev/null +++ b/docs/MDTODO/details/yaml-owned-pac-source-artifacts/R1.2_Task_Report.md @@ -0,0 +1,23 @@ +# R1.2 任务报告 + +关联问题：[UniDesk #1745](https://github.com/pikasTech/unidesk/issues/1745)。 + +## 完成内容 + +- 完整比较 Pipeline spec，并输出 canonical hash、provenance 与首个漂移路径，不输出完整 spec 或 Secret。 +- canonicalizer 只移除六类已验证 Tekton admission default，并严格限定三个合法 spec 根。 +- source worktree 校验绝对路径、Git 根、精确 remote identity 和 symlink 边界；写入采用同目录临时文件与 rename，重复写入无变化。 +- 默认 validation error 为有界三行文本；显式 JSON/full 才保留结构化诊断。 +- 外层 CLI 的 `source-artifact --help` 与 ` --help` 均返回专项帮助。 +- `$unidesk-cicd` 的 PaC reference 已记录 desired authority、五动作、exact-commit 门禁、大 spec transport 和 fail-closed 规则。 + +## 测试证据 + +- 新增定向测试共十三项、七十二个断言，全部通过。 +- 覆盖六类 admission default 正向与同名字段负向、本地/远端 canonicalizer 一致、同 commit retry/conflict、NotFound/transport 区分、未声明 JD01 owner、错误脱敏、共享 HWLAB overlay 等价性。 +- 使用大于当前约 203 KiB HWLAB Pipeline 的 payload 实跑原生 observer 临时文件 transport，live 与 embedded 均完成完整 hash 比较。 +- HWLAB postprocess、verify 与六个 Kafka refresh marker 逐项删除均能触发 fail-closed。 + +## 基线说明 + +- 额外运行历史 `scripts/src/agentrun.test.ts` 时出现五个既有失败，集中在旧测试 fixture 的 `version` 与已迁移路由断言；本分支未修改这些测试或 AgentRun client parser。 diff --git a/docs/MDTODO/yaml-owned-pac-source-artifacts.md b/docs/MDTODO/yaml-owned-pac-source-artifacts.md new file mode 100644 index 00000000..ea27e0a7 --- /dev/null +++ b/docs/MDTODO/yaml-owned-pac-source-artifacts.md @@ -0,0 +1,22 @@ +# YAML owning Pipeline 到 PaC source artifact 正规生成 + +- 来源：[UniDesk #1745](https://github.com/pikasTech/unidesk/issues/1745) +- 关联故障：[UniDesk #1726](https://github.com/pikasTech/unidesk/issues/1726) +- 范围：统一连接 owning YAML/domain renderer 与 consumer 声明的 source artifact；同时支持 PipelineRun inline `pipelineSpec` 和 `.tekton` annotation 引用 remote Pipeline 两种载体，禁止一次性手工 patch。 + + +## R1 [in_progress] + +落实 [UniDesk #1745](https://github.com/pikasTech/unidesk/issues/1745)：建立 owning YAML/domain renderer 到 consumer-declared PaC source artifact 的生成、校验、部署和验收闭环，完成任务后将详细报告写入[任务报告](./details/yaml-owned-pac-source-artifacts/R1_Task_Report.md)。 +### R1.1 [completed] + +基于 [UniDesk #1745](https://github.com/pikasTech/unidesk/issues/1745) 实现 target/consumer 隔离的 source artifact schema、inline pipelineSpec/remote Pipeline renderer 与 plan/check/write/status/verify-runtime CLI，完成任务后将详细报告写入[任务报告](./details/yaml-owned-pac-source-artifacts/R1.1_Task_Report.md)。 +### R1.2 [completed] + +为 [UniDesk #1745](https://github.com/pikasTech/unidesk/issues/1745) 补齐完整 spec、provenance、canonical hash、first-drift、身份边界、脱敏输出测试和长期文档，完成任务后将详细报告写入[任务报告](./details/yaml-owned-pac-source-artifacts/R1.2_Task_Report.md)。 +### R1.3 [in_progress] + +落实 [UniDesk #1745](https://github.com/pikasTech/unidesk/issues/1745)：使用生成器更新 AgentRun source artifact，并向 HWLAB 消费侧交付 remote Pipeline 接口与 fixture；经受控 PR/preflight 交付，不做手工 env/base64 patch，完成任务后将详细报告写入[任务报告](./details/yaml-owned-pac-source-artifacts/R1.3_Task_Report.md)。 +### R1.4 + +协同 rollout owner 在 NC01 验证 runtime embedded spec、manager Healthy、stale-pending 周期回收及 dsflash Secret 未越权，并回写 [UniDesk #1726](https://github.com/pikasTech/unidesk/issues/1726) 与 #1745，完成任务后将详细报告写入[任务报告](./details/yaml-owned-pac-source-artifacts/R1.4_Task_Report.md)。 diff --git a/scripts/native/cicd/pac-source-artifact-runtime.mjs b/scripts/native/cicd/pac-source-artifact-runtime.mjs new file mode 100644 index 00000000..6fbf92bf --- /dev/null +++ b/scripts/native/cicd/pac-source-artifact-runtime.mjs @@ -0,0 +1,238 @@ +import { createHash } from "node:crypto"; +import { execFileSync } from "node:child_process"; +import { readFileSync } from "node:fs"; + +const provenanceKeys = { + configRef: "unidesk.ai/owning-config-ref", + effectiveConfigSha256: "unidesk.ai/effective-config-sha256", + renderer: "unidesk.ai/source-artifact-renderer", +}; + +const sourceCommitKeys = [ + "pipelinesascode.tekton.dev/sha", + "pipelinesascode.tekton.dev/commit", + "agentrun.pikastech.local/source-commit", + "unidesk.ai/source-commit", + "hwlab.pikastech.local/source-commit", +]; + +function record(value) { + return value !== null && typeof value === "object" && !Array.isArray(value) ? value : {}; +} + +function compact(value) { + const raw = JSON.stringify(value); + if (raw === undefined) return String(value); + return raw.length <= 160 ? raw : `${raw.slice(0, 157)}...`; +} + +function unavailable(reason, sourceCommit = null, candidateCount = 0) { + return { + status: "unavailable", + name: null, + canonicalSha256: null, + firstDrift: null, + provenanceAligned: false, + configRef: null, + effectiveConfigSha256: null, + sourceCommit, + reason, + candidateCount, + }; +} + +function missing(reason, sourceCommit = null) { + return { ...unavailable(reason, sourceCommit, 0), status: "missing" }; +} + +function isAdmissionDefault(path, value) { + const normalized = path.map((segment) => typeof segment === "number" ? "*" : segment).join("."); + const emptyRecord = value !== null && typeof value === "object" && !Array.isArray(value) && Object.keys(value).length === 0; + const atTaskSpec = (suffix) => [ + `tasks.*.taskSpec.${suffix}`, + `spec.tasks.*.taskSpec.${suffix}`, + `spec.pipelineSpec.tasks.*.taskSpec.${suffix}`, + ].includes(normalized); + if (atTaskSpec("metadata")) return emptyRecord; + if (atTaskSpec("spec")) return value === null; + if (atTaskSpec("params.*.type")) return value === "string"; + if (atTaskSpec("results.*.type")) return value === "string"; + if (atTaskSpec("steps.*.computeResources")) return emptyRecord; + if (atTaskSpec("sidecars.*.computeResources")) return emptyRecord; + return false; +} + +export function canonicalizePacPipelineSpec(value, path = []) { + if (Array.isArray(value)) return value.map((item, index) => canonicalizePacPipelineSpec(item, [...path, index])); + if (value === null || typeof value !== "object") return value; + const output = {}; + for (const key of Object.keys(value).sort()) { + const child = value[key]; + const childPath = [...path, key]; + if (isAdmissionDefault(childPath, child)) continue; + output[key] = canonicalizePacPipelineSpec(child, childPath); + } + return output; +} + +export function canonicalPacPipelineSha256(value) { + return `sha256:${createHash("sha256").update(JSON.stringify(canonicalizePacPipelineSpec(value))).digest("hex")}`; +} + +function firstCanonicalDrift(expected, actual, path = "$") { + if (Object.is(expected, actual)) return null; + if (Array.isArray(expected) || Array.isArray(actual)) { + if (!Array.isArray(expected) || !Array.isArray(actual)) return { path, expected: compact(expected), actual: compact(actual) }; + if (expected.length !== actual.length) return { path: `${path}.length`, expected: String(expected.length), actual: String(actual.length) }; + for (let index = 0; index < expected.length; index += 1) { + const drift = firstCanonicalDrift(expected[index], actual[index], `${path}[${index}]`); + if (drift !== null) return drift; + } + return null; + } + const expectedRecord = expected !== null && typeof expected === "object"; + const actualRecord = actual !== null && typeof actual === "object"; + if (expectedRecord || actualRecord) { + if (!expectedRecord || !actualRecord) return { path, expected: compact(expected), actual: compact(actual) }; + const keys = [...new Set([...Object.keys(expected), ...Object.keys(actual)])].sort(); + for (const key of keys) { + if (!Object.prototype.hasOwnProperty.call(expected, key) || !Object.prototype.hasOwnProperty.call(actual, key)) { + return { path: `${path}.${key}`, expected: compact(expected[key]), actual: compact(actual[key]) }; + } + const drift = firstCanonicalDrift(expected[key], actual[key], `${path}.${key}`); + if (drift !== null) return drift; + } + return null; + } + return { path, expected: compact(expected), actual: compact(actual) }; +} + +function firstDrift(expected, actual) { + return firstCanonicalDrift(canonicalizePacPipelineSpec(expected), canonicalizePacPipelineSpec(actual)); +} + +function kubectlJson(args, label) { + try { + return { + kind: "ok", + value: JSON.parse(execFileSync("kubectl", args, { encoding: "utf8", timeout: 30_000, maxBuffer: 64 * 1024 * 1024 })), + }; + } catch (error) { + const stderr = typeof error?.stderr === "string" ? error.stderr : Buffer.isBuffer(error?.stderr) ? error.stderr.toString("utf8") : ""; + const message = stderr.trim().split(/\r?\n/u)[0] || String(error?.message ?? "command failed"); + if (/\bNotFound\b|\bnot found\b/u.test(message)) return { kind: "not-found", reason: `${label}-not-found` }; + const status = Number.isInteger(error?.status) ? error.status : "unknown"; + return { kind: "unavailable", reason: `${label}-command-failed:${status}:${compact(message)}` }; + } +} + +function manifestProvenance(manifest) { + const annotations = record(record(manifest?.metadata).annotations); + return { + configRef: typeof annotations[provenanceKeys.configRef] === "string" ? annotations[provenanceKeys.configRef] : null, + effectiveConfigSha256: typeof annotations[provenanceKeys.effectiveConfigSha256] === "string" ? annotations[provenanceKeys.effectiveConfigSha256] : null, + renderer: typeof annotations[provenanceKeys.renderer] === "string" ? annotations[provenanceKeys.renderer] : null, + }; +} + +function manifestSourceCommits(manifest) { + const metadata = record(manifest?.metadata); + const labels = record(metadata.labels); + const annotations = record(metadata.annotations); + return [...new Set(sourceCommitKeys.flatMap((key) => [labels[key], annotations[key]]) + .filter((value) => typeof value === "string" && /^[0-9a-f]{40}$/iu.test(value)) + .map((value) => value.toLowerCase()))]; +} + +function observedItem(input, manifest, spec, sourceCommit = null, candidateCount = 0) { + const observedProvenance = manifestProvenance(manifest); + const provenanceAligned = observedProvenance.configRef === input.provenance.configRef + && observedProvenance.effectiveConfigSha256 === input.provenance.effectiveConfigSha256 + && observedProvenance.renderer === input.provenance.renderer; + const drift = firstDrift(input.desiredSpec, spec); + return { + status: drift === null ? "aligned" : "drifted", + name: typeof record(manifest.metadata).name === "string" ? record(manifest.metadata).name : null, + canonicalSha256: canonicalPacPipelineSha256(spec), + firstDrift: drift, + provenanceAligned, + configRef: observedProvenance.configRef, + effectiveConfigSha256: observedProvenance.effectiveConfigSha256, + sourceCommit, + reason: null, + candidateCount, + }; +} + +export function selectPipelineRunBySourceCommit(items, pipelineRunPrefix, sourceCommit) { + if (sourceCommit === null) return { kind: "unavailable", reason: "source-commit-not-specified", run: null, candidates: [] }; + const candidates = items.filter((run) => { + const name = String(record(run?.metadata).name ?? ""); + return name.startsWith(pipelineRunPrefix) && manifestSourceCommits(run).includes(sourceCommit); + }); + if (candidates.length === 0) return { kind: "missing", reason: "source-commit-run-not-found", run: null, candidates: [] }; + if (candidates.some((run) => { + const commits = manifestSourceCommits(run); + return commits.length !== 1 || commits[0] !== sourceCommit; + })) return { kind: "unavailable", reason: "source-commit-identity-conflict", run: null, candidates }; + const ordered = [...candidates].sort((left, right) => { + const time = String(record(right?.metadata).creationTimestamp ?? "").localeCompare(String(record(left?.metadata).creationTimestamp ?? "")); + return time !== 0 ? time : String(record(right?.metadata).name ?? "").localeCompare(String(record(left?.metadata).name ?? "")); + }); + return { kind: "ok", reason: null, run: ordered[0], candidates: ordered }; +} + +export function observePacSourceArtifactRuntime(input, readKubectlJson = kubectlJson) { + const liveResult = readKubectlJson(["get", "pipeline", input.pipeline, "-n", input.namespace, "-o", "json"], "pipeline-get"); + const live = liveResult.kind === "ok" + ? observedItem(input, liveResult.value, record(liveResult.value.spec)) + : liveResult.kind === "not-found" + ? missing(liveResult.reason) + : unavailable(liveResult.reason); + + let embedded; + if (input.sourceCommit === null) { + embedded = unavailable("source-commit-not-specified"); + } else { + const runsResult = readKubectlJson(["get", "pipelinerun", "-n", input.namespace, "-o", "json"], "pipelinerun-list"); + if (runsResult.kind !== "ok") { + embedded = runsResult.kind === "not-found" ? missing(runsResult.reason, input.sourceCommit) : unavailable(runsResult.reason, input.sourceCommit); + } else { + const items = Array.isArray(runsResult.value?.items) ? runsResult.value.items : null; + if (items === null) { + embedded = unavailable("pipelinerun-list-invalid-shape", input.sourceCommit); + } else { + const selected = selectPipelineRunBySourceCommit(items, input.pipelineRunPrefix, input.sourceCommit); + if (selected.kind === "ok") { + const signatures = selected.candidates.map((run) => JSON.stringify({ + canonicalSha256: canonicalPacPipelineSha256(record(record(run.spec).pipelineSpec)), + provenance: manifestProvenance(run), + })); + embedded = new Set(signatures).size === 1 + ? observedItem(input, selected.run, record(record(selected.run.spec).pipelineSpec), input.sourceCommit, selected.candidates.length) + : unavailable(`source-commit-run-conflict:${selected.candidates.length}`, input.sourceCommit, selected.candidates.length); + } else { + embedded = selected.kind === "missing" + ? missing(selected.reason, input.sourceCommit) + : unavailable(selected.reason, input.sourceCommit, selected.candidates.length); + } + } + } + } + return { live, embedded }; +} + +if (typeof process.env.PAC_SOURCE_ARTIFACT_INPUT_FILE === "string" || typeof process.env.PAC_SOURCE_ARTIFACT_INPUT === "string") { + try { + const inputText = typeof process.env.PAC_SOURCE_ARTIFACT_INPUT_FILE === "string" + ? readFileSync(process.env.PAC_SOURCE_ARTIFACT_INPUT_FILE, "utf8") + : Buffer.from(process.env.PAC_SOURCE_ARTIFACT_INPUT, "base64").toString("utf8"); + const input = JSON.parse(inputText); + process.stdout.write(JSON.stringify(observePacSourceArtifactRuntime(input))); + } catch (error) { + process.stdout.write(JSON.stringify({ + live: unavailable(`runtime-observer-input-error:${compact(String(error?.message ?? error))}`), + embedded: unavailable(`runtime-observer-input-error:${compact(String(error?.message ?? error))}`), + })); + } +} diff --git a/scripts/src/agentrun-manifests.ts b/scripts/src/agentrun-manifests.ts index 37d1770b..c1833aa5 100644 --- a/scripts/src/agentrun-manifests.ts +++ b/scripts/src/agentrun-manifests.ts @@ -2,6 +2,7 @@ // Renders AgentRun YAML lane policy into runtime manager environment. import { createHash } from "node:crypto"; import type { AgentRunLaneSpec } from "./agentrun-lanes"; +import { stableJsonSha256 } from "./stable-json"; export interface AgentRunArtifactService { readonly serviceId: string; @@ -75,7 +76,7 @@ export function renderAgentRunControlPlaneManifests(spec: AgentRunLaneSpec): rea subjects: [{ kind: "ServiceAccount", name: spec.ci.serviceAccountName }], roleRef: { apiGroup: "rbac.authorization.k8s.io", kind: "Role", name: spec.ci.serviceAccountName }, }, - agentRunPipelineManifest(spec), + renderAgentRunPipelineManifest(spec), agentRunArgoProjectManifest(spec), agentRunArgoApplicationManifest(spec), ]; @@ -172,7 +173,7 @@ export function renderedObjectsDigest(objects: readonly Record[ return `sha256:${createHash("sha256").update(yamlAll(objects)).digest("hex")}`; } -function agentRunPipelineManifest(spec: AgentRunLaneSpec): Record { +export function renderAgentRunPipelineManifest(spec: AgentRunLaneSpec): Record { const build = spec.deployment.manager.imageBuild; if (spec.ci.buildkitImage === null) throw new Error(`config/agentrun.yaml controlPlane.lanes.${spec.lane}.ci.buildkitImage is required for AgentRun Tekton image build`); return { @@ -182,6 +183,12 @@ function agentRunPipelineManifest(spec: AgentRunLaneSpec): Record (await import("./cicd")).cicdHelp(), cicdHelpSummary()); if (top === "agentrun") return loadHelp(async () => (await import("./agentrun")).agentRunHelp(), agentRunHelpSummary()); + if (top === "platform-infra" && (sub === "pipelines-as-code" || sub === "pac") && args[2] === "source-artifact") return null; if (top === "platform-infra") return loadHelp(async () => (await import("./platform-infra")).platformInfraHelp(), platformInfraHelpSummary()); if (top === "platform-db") return platformDbHelp(); if (top === "secrets") return secretsHelp(); diff --git a/scripts/src/hwlab-node/deploy-overlay.ts b/scripts/src/hwlab-node/deploy-overlay.ts new file mode 100644 index 00000000..877e5758 --- /dev/null +++ b/scripts/src/hwlab-node/deploy-overlay.ts @@ -0,0 +1,66 @@ +export function applyNodeRuntimeDeployYamlOverlay(document: Record, overlay: Record): Record { + const doc = structuredClone(document); + const nodeId = requiredString(overlay.nodeId, "overlay.nodeId"); + const laneId = requiredString(overlay.lane, "overlay.lane"); + const nodes = record(doc.nodes); + const node = record(nodes[nodeId]); + nodes[nodeId] = { ...node, gitopsRoot: overlay.gitopsRoot, sourceRepo: overlay.gitUrl }; + doc.nodes = nodes; + const lanes = record(doc.lanes); + const lane = record(lanes[laneId]); + const envRecipe = record(lane.envRecipe); + const downloadStack = { + ...record(envRecipe.downloadStack), + httpProxy: overlay.dockerProxyHttp, + httpsProxy: overlay.dockerProxyHttps, + noProxy: overlay.dockerNoProxyList, + }; + const nextLane: Record = { + ...lane, + node: nodeId, + sourceBranch: overlay.sourceBranch, + gitopsBranch: overlay.gitopsBranch, + namespace: overlay.runtimeNamespace, + endpoint: overlay.publicApiUrl, + publicEndpoints: { frontend: overlay.publicWebUrl, api: overlay.publicApiUrl }, + artifactCatalog: overlay.catalogPath, + runtimePath: overlay.runtimePath, + imageTagMode: "full", + sourceRepo: overlay.gitUrl, + observability: overlay.observability, + envRecipe: { ...envRecipe, downloadStack }, + }; + if (overlay.externalPostgres === undefined || overlay.externalPostgres === null) delete nextLane.externalPostgres; + else nextLane.externalPostgres = overlay.externalPostgres; + if (overlay.runtimeStore !== undefined) nextLane.runtimeStore = overlay.runtimeStore; + if (overlay.codeAgentRuntime !== undefined) nextLane.codeAgentRuntime = overlay.codeAgentRuntime; + if (overlay.deployYamlGitMirror !== undefined) nextLane.gitMirror = overlay.deployYamlGitMirror; + lanes[laneId] = nextLane; + doc.lanes = lanes; + return doc; +} + +export function nodeRuntimeDeployYamlOverlayShellScript(): string[] { + return [ + "node - \"$overlay_b64\" <<'NODE_UNIDESK_DEPLOY_OVERLAY'", + "const fs = require('fs');", + "const YAML = require('yaml');", + `const applyNodeRuntimeDeployYamlOverlay = ${applyNodeRuntimeDeployYamlOverlay.toString()};`, + `const record = ${record.toString()};`, + `const requiredString = ${requiredString.toString()};`, + "const overlay = JSON.parse(Buffer.from(process.argv[2], 'base64').toString('utf8'));", + "const path = 'deploy/deploy.yaml';", + "const doc = YAML.parse(fs.readFileSync(path, 'utf8'));", + "fs.writeFileSync(path, YAML.stringify(applyNodeRuntimeDeployYamlOverlay(doc, overlay)));", + "NODE_UNIDESK_DEPLOY_OVERLAY", + ]; +} + +function record(value: unknown): Record { + return typeof value === "object" && value !== null && !Array.isArray(value) ? value as Record : {}; +} + +function requiredString(value: unknown, label: string): string { + if (typeof value !== "string" || value.length === 0) throw new Error(`${label} must be a non-empty string`); + return value; +} diff --git a/scripts/src/hwlab-node/render.ts b/scripts/src/hwlab-node/render.ts index 2a49b5f8..8de169af 100644 --- a/scripts/src/hwlab-node/render.ts +++ b/scripts/src/hwlab-node/render.ts @@ -40,6 +40,7 @@ import { externalPostgresBridgeStatus, externalPostgresSecretStatus, getNodeRunt import { webObserveShort, webObserveText } from "./web-probe-observe"; import { readNodeRuntimeStatusPolicy, runNodeRuntimeStatusProbe, skippedNodeRuntimeStatusCommand, type NodeRuntimeStatusPolicy, type NodeRuntimeStatusWorkerEvent } from "./control-plane-status-observed"; import { hwlabRuntimeActiveExternalPostgres } from "../hwlab-node-lanes"; +import { nodeRuntimeDeployYamlOverlayShellScript } from "./deploy-overlay"; const runtimeGitopsObservabilityNativeScript = readFileSync(rootPath("scripts/native/hwlab/runtime-gitops-observability.mjs"), "utf8").trimEnd(); const runtimeGitopsPipelineGuardNativeScript = readFileSync(rootPath("scripts/native/hwlab/runtime-gitops-pipeline-guard.mjs"), "utf8").trimEnd(); @@ -1468,44 +1469,7 @@ export function renderNodeRuntimeControlPlaneOnNode(spec: HwlabRuntimeLaneSpec, "git -C \"$worktree_dir\" checkout --detach \"$source_commit\"", "cd \"$worktree_dir\"", ...yamlDependencyInstallScript(spec.downloadProfile.npm.registry, spec.downloadProfile.npm.fetchTimeoutSeconds, spec.downloadProfile.npm.retries, "control-plane-render"), - "node - \"$overlay_b64\" <<'NODE'", - "const fs = require('fs');", - "const YAML = require('yaml');", - "const overlay = JSON.parse(Buffer.from(process.argv[2], 'base64').toString('utf8'));", - "const path = 'deploy/deploy.yaml';", - "const doc = YAML.parse(fs.readFileSync(path, 'utf8'));", - "doc.nodes = doc.nodes || {};", - "doc.nodes[overlay.nodeId] = { ...(doc.nodes[overlay.nodeId] || {}), gitopsRoot: overlay.gitopsRoot, sourceRepo: overlay.gitUrl };", - "doc.lanes = doc.lanes || {};", - "const lane = doc.lanes[overlay.lane] || {};", - "const downloadStack = {", - " ...(lane.envRecipe?.downloadStack || {}),", - " httpProxy: overlay.dockerProxyHttp,", - " httpsProxy: overlay.dockerProxyHttps,", - " noProxy: overlay.dockerNoProxyList,", - "};", - "doc.lanes[overlay.lane] = {", - " ...lane,", - " node: overlay.nodeId,", - " sourceBranch: overlay.sourceBranch,", - " gitopsBranch: overlay.gitopsBranch,", - " namespace: overlay.runtimeNamespace,", - " endpoint: overlay.publicApiUrl,", - " publicEndpoints: { frontend: overlay.publicWebUrl, api: overlay.publicApiUrl },", - " artifactCatalog: overlay.catalogPath,", - " runtimePath: overlay.runtimePath,", - " imageTagMode: 'full',", - " sourceRepo: overlay.gitUrl,", - " observability: overlay.observability,", - " envRecipe: { ...(lane.envRecipe || {}), downloadStack },", - "};", - "if (overlay.externalPostgres === undefined || overlay.externalPostgres === null) delete doc.lanes[overlay.lane].externalPostgres;", - "else doc.lanes[overlay.lane].externalPostgres = overlay.externalPostgres;", - "if (overlay.runtimeStore !== undefined) doc.lanes[overlay.lane].runtimeStore = overlay.runtimeStore;", - "if (overlay.codeAgentRuntime !== undefined) doc.lanes[overlay.lane].codeAgentRuntime = overlay.codeAgentRuntime;", - "if (overlay.deployYamlGitMirror !== undefined) doc.lanes[overlay.lane].gitMirror = overlay.deployYamlGitMirror;", - "fs.writeFileSync(path, YAML.stringify(doc));", - "NODE", + ...nodeRuntimeDeployYamlOverlayShellScript(), "if [ -f scripts/gitops-render.mjs ]; then render_script=scripts/gitops-render.mjs; else echo 'render script missing: scripts/gitops-render.mjs' >&2; exit 43; fi", [ "node scripts/run-bun.mjs \"$render_script\"", @@ -2557,6 +2521,11 @@ export function nodeRuntimePipelinePostprocessScript(): string[] { " doc.metadata = doc.metadata || {};", " if (typeof overlay.pipelineName === 'string' && overlay.pipelineName.length > 0) doc.metadata.name = overlay.pipelineName;", " doc.metadata.annotations = doc.metadata.annotations || {};", + " const provenance = overlay.sourceArtifactProvenance || {};", + " if (provenance.configRef) doc.metadata.annotations['unidesk.ai/owning-config-ref'] = provenance.configRef;", + " if (provenance.effectiveConfigSha256) doc.metadata.annotations['unidesk.ai/effective-config-sha256'] = provenance.effectiveConfigSha256;", + " if (provenance.renderer) doc.metadata.annotations['unidesk.ai/source-artifact-renderer'] = provenance.renderer;", + " if (provenance.mode) doc.metadata.annotations['unidesk.ai/source-artifact-mode'] = provenance.mode;", " doc.metadata.annotations['hwlab.pikastech.local/download-profile'] = overlay.downloadProfileId;", " doc.metadata.annotations['hwlab.pikastech.local/network-profile'] = overlay.networkProfileId;", " for (const task of doc.spec?.tasks || []) {", diff --git a/scripts/src/hwlab-node/web-probe.ts b/scripts/src/hwlab-node/web-probe.ts index bd12830d..ff37f7bc 100644 --- a/scripts/src/hwlab-node/web-probe.ts +++ b/scripts/src/hwlab-node/web-probe.ts @@ -40,6 +40,7 @@ import { assertKnownOptions, nodeWebProbeAutoCommandTimeoutSeconds, normalizeNod import { resolveNodeWebProbeCliOrigin } from "./web-probe-origin"; import { hwlabRuntimeActiveExternalPostgres } from "../hwlab-node-lanes"; import { resolveEgressProxySourceRef } from "../egress-proxy-sources"; +import { stableJsonSha256 } from "../stable-json"; export function nodeRuntimeRenderOverlay(spec: HwlabRuntimeLaneSpec): Record { const gitSshProxy = httpProxyEndpoint(spec.networkProfile.proxy.http); @@ -74,6 +75,12 @@ export function nodeRuntimeRenderOverlay(spec: HwlabRuntimeLaneSpec): Record { + return { + "unidesk.ai/owning-config-ref": provenance.configRef, + "unidesk.ai/effective-config-sha256": provenance.effectiveConfigSha256, + "unidesk.ai/source-artifact-renderer": provenance.renderer, + }; +} + +function pipeline(): Record { + return { + metadata: { name: "agentrun-nc01-v02-ci-image-publish", annotations: manifestAnnotations() }, + spec: desiredSpec, + }; +} + +function pipelineRun(name: string, creationTimestamp: string, spec: Record = desiredSpec): Record { + return { + metadata: { + name, + creationTimestamp, + labels: { "pipelinesascode.tekton.dev/sha": sourceCommit }, + annotations: manifestAnnotations(), + }, + spec: { pipelineSpec: spec }, + }; +} + +function runtimeInput(commit: string | null = sourceCommit): Record { + return { + namespace: "agentrun-ci", + pipeline: "agentrun-nc01-v02-ci-image-publish", + pipelineRunPrefix: "agentrun-nc01-v02-ci", + desiredSpec, + provenance, + sourceCommit: commit, + }; +} + +describe("PaC source artifact CLI contract", () => { + test("verify-runtime requires and normalizes a full source commit", () => { + expect(() => parsePacSourceArtifactOptions([ + "verify-runtime", "--target", "NC01", "--consumer", "agentrun-nc01-v02", "--source-worktree", "/tmp/repo", + ])).toThrow("requires explicit --source-commit"); + const parsed = parsePacSourceArtifactOptions([ + "verify-runtime", "--target", "NC01", "--consumer", "agentrun-nc01-v02", "--source-worktree", "/tmp/repo", "--source-commit", sourceCommit.toUpperCase(), + ]); + expect(parsed.sourceCommit).toBe(sourceCommit); + expect(() => parsePacSourceArtifactOptions([ + "plan", "--target", "NC01", "--consumer", "agentrun-nc01-v02", "--source-worktree", "/tmp/repo", "--source-commit", sourceCommit, + ])).toThrow("accepted only"); + }); + + test("real outer CLI returns bounded scoped help for both help positions", () => { + for (const tail of [["--help"], ["check", "--help"]]) { + const result = spawnSync("bun", ["scripts/cli.ts", "platform-infra", "pipelines-as-code", "source-artifact", ...tail], { + cwd: rootPath(), encoding: "utf8", timeout: 30_000, + }); + expect(result.status).toBe(0); + expect(result.stdout).toContain("PAC SOURCE ARTIFACT"); + expect(result.stdout).toContain("verify-runtime"); + expect(result.stdout).toContain("--source-commit "); + expect(result.stdout.split("\n").length).toBeLessThan(25); + expect(result.stdout).not.toContain("platform-infra sub2api plan"); + } + }); + + test("predictable validation failures omit stack and rendered specs", () => { + const result = spawnSync("bun", [ + "scripts/cli.ts", "platform-infra", "pipelines-as-code", "source-artifact", "plan", + "--target", "NC01", "--consumer", "agentrun-nc01-v02", "--source-worktree", rootPath(), + ], { cwd: rootPath(), encoding: "utf8", timeout: 30_000 }); + expect(result.status).toBe(1); + expect(result.stdout).toContain("origin does not match PaC repository pikasTech-agentrun"); + expect(result.stdout).toContain("pikasTech/unidesk.git"); + expect(result.stdout).not.toContain("stack"); + expect(result.stdout).not.toContain("taskSpec"); + expect(result.stdout.length).toBeLessThan(1500); + }); + + test("undeclared JD01 source artifact owner fails closed", () => { + const result = spawnSync("bun", [ + "scripts/cli.ts", "platform-infra", "pipelines-as-code", "source-artifact", "plan", + "--target", "JD01", "--consumer", "agentrun-jd01-v02", "--source-worktree", rootPath(), + ], { cwd: rootPath(), encoding: "utf8", timeout: 30_000 }); + expect(result.status).toBe(1); + expect(result.stdout).toContain("has no sourceArtifact owner"); + }); +}); + +describe("Tekton admission canonicalization", () => { + const admitted = { + tasks: [{ + name: "build", + taskSpec: { + metadata: {}, + spec: null, + params: [{ name: "input", type: "string" }], + results: [{ name: "output", type: "string" }], + steps: [{ name: "step", computeResources: {} }], + sidecars: [{ name: "sidecar", computeResources: {} }], + }, + }], + }; + const desired = { + tasks: [{ + name: "build", + taskSpec: { + params: [{ name: "input" }], + results: [{ name: "output" }], + steps: [{ name: "step" }], + sidecars: [{ name: "sidecar" }], + }, + }], + }; + + test("removes exactly the six proven defaults and keeps local/native hashes equal", () => { + expect(canonicalizePacPipelineSpec(admitted)).toEqual(desired); + expect(nativeCanonicalize(admitted)).toEqual(desired); + expect(canonicalSha256(admitted)).toBe(canonicalSha256(desired)); + expect(nativeCanonicalSha256(admitted)).toBe(canonicalSha256(admitted)); + }); + + test("does not strip same-name fields outside the three Pipeline spec roots", () => { + const negative = { + unrelated: admitted, + tasks: [{ taskSpec: { + metadata: { labels: {} }, + spec: {}, + params: [{ type: "array" }], + results: [{ type: "object" }], + steps: [{ computeResources: { requests: { cpu: "1" } } }], + sidecars: [{ computeResources: { limits: { memory: "1Gi" } } }], + } }], + }; + expect(canonicalizePacPipelineSpec(negative)).toEqual(negative); + expect(nativeCanonicalize(negative)).toEqual(negative); + }); +}); + +describe("runtime source-commit selection", () => { + test("same-commit retries select the newest only when spec and provenance agree", () => { + const oldRun = pipelineRun("agentrun-nc01-v02-ci-old", "2026-07-10T00:00:00Z"); + const newRun = pipelineRun("agentrun-nc01-v02-ci-new", "2026-07-11T00:00:00Z"); + const selected = selectPipelineRunBySourceCommit([oldRun, newRun], "agentrun-nc01-v02-ci", sourceCommit); + expect(selected.kind).toBe("ok"); + expect(selected.run.metadata.name).toBe("agentrun-nc01-v02-ci-new"); + const observed = observePacSourceArtifactRuntime(runtimeInput(), (args: string[], label: string) => label === "pipeline-get" + ? { kind: "ok", value: pipeline() } + : { kind: "ok", value: { items: [oldRun, newRun] } }); + expect(observed.embedded.status).toBe("aligned"); + expect(observed.embedded.name).toBe("agentrun-nc01-v02-ci-new"); + expect(observed.embedded.candidateCount).toBe(2); + expect(observed.embedded.sourceCommit).toBe(sourceCommit); + }); + + test("same-commit conflicting embedded specs fail closed", () => { + const observed = observePacSourceArtifactRuntime(runtimeInput(), (_args: string[], label: string) => label === "pipeline-get" + ? { kind: "ok", value: pipeline() } + : { kind: "ok", value: { items: [ + pipelineRun("agentrun-nc01-v02-ci-old", "2026-07-10T00:00:00Z"), + pipelineRun("agentrun-nc01-v02-ci-new", "2026-07-11T00:00:00Z", { params: [], tasks: [{ name: "changed" }] }), + ] } }); + expect(observed.embedded.status).toBe("unavailable"); + expect(observed.embedded.reason).toBe("source-commit-run-conflict:2"); + expect(observed.embedded.candidateCount).toBe(2); + }); + + test("NotFound, missing commit, and transport failures remain distinct", () => { + const noCommit = observePacSourceArtifactRuntime(runtimeInput(null), () => ({ kind: "not-found", reason: "pipeline-get-not-found" })); + expect(noCommit.live.status).toBe("missing"); + expect(noCommit.live.reason).toBe("pipeline-get-not-found"); + expect(noCommit.embedded.status).toBe("unavailable"); + expect(noCommit.embedded.reason).toBe("source-commit-not-specified"); + + const transport = observePacSourceArtifactRuntime(runtimeInput(), (_args: string[], label: string) => ({ kind: "unavailable", reason: `${label}-command-failed:126:permission denied` })); + expect(transport.live.status).toBe("unavailable"); + expect(transport.embedded.status).toBe("unavailable"); + expect(transport.live.reason).toContain("permission denied"); + expect(transport.embedded.reason).toContain("permission denied"); + }); + + test("temp-file transport handles a payload larger than the current 203 KiB HWLAB Pipeline", () => { + const temporary = mkdtempSync(resolve(tmpdir(), "unidesk-pac-runtime-test-")); + try { + const bin = resolve(temporary, "bin"); + mkdirSync(bin); + const hugeSpec = { params: [], tasks: [{ name: "render", taskSpec: { steps: [{ name: "render", script: "x".repeat(220 * 1024) }] } }] }; + const input = { + namespace: "hwlab-ci", + pipeline: "hwlab-nc01-v03-ci-image-publish", + pipelineRunPrefix: "hwlab-nc01-v03-ci-poll", + desiredSpec: hugeSpec, + provenance, + sourceCommit, + }; + expect(Buffer.byteLength(JSON.stringify(input), "utf8")).toBeGreaterThan(203 * 1024); + const pipelineFile = resolve(temporary, "pipeline.json"); + const runsFile = resolve(temporary, "runs.json"); + writeFileSync(pipelineFile, JSON.stringify({ metadata: { name: input.pipeline, annotations: manifestAnnotations() }, spec: hugeSpec })); + writeFileSync(runsFile, JSON.stringify({ items: [pipelineRun("hwlab-nc01-v03-ci-poll-large", "2026-07-11T00:00:00Z", hugeSpec)] })); + const kubectl = resolve(bin, "kubectl"); + writeFileSync(kubectl, "#!/bin/sh\nset -eu\ncase \"$2\" in pipeline) cat \"$PAC_TEST_PIPELINE\" ;; pipelinerun) cat \"$PAC_TEST_RUNS\" ;; *) exit 2 ;; esac\n"); + chmodSync(kubectl, 0o755); + const shell = renderPacSourceArtifactRuntimeObserverShell(input); + expect(shell).toContain("observer_input=$(mktemp)"); + expect(shell).toContain("PAC_SOURCE_ARTIFACT_INPUT_FILE=\"$observer_input\""); + expect(shell).not.toContain("PAC_SOURCE_ARTIFACT_INPUT="); + const result = spawnSync("sh", [], { + input: shell, + encoding: "utf8", + timeout: 30_000, + maxBuffer: 8 * 1024 * 1024, + env: { + ...process.env, + PATH: `${bin}:${process.env.PATH ?? ""}`, + PAC_TEST_PIPELINE: pipelineFile, + PAC_TEST_RUNS: runsFile, + }, + }); + expect(result.status).toBe(0); + const observed = JSON.parse(result.stdout) as Record; + expect(observed.live.status).toBe("aligned"); + expect(observed.embedded.status).toBe("aligned"); + expect(observed.embedded.sourceCommit).toBe(sourceCommit); + } finally { + rmSync(temporary, { recursive: true, force: true }); + } + }); +}); + +describe("HWLAB domain Pipeline contract", () => { + const requiredMarkers = [ + "unidesk-runtime-gitops-postprocess", + "unidesk-runtime-gitops-verify", + "HWLAB_WORKBENCH_KAFKA_REFRESH_REPLAY_ENABLED", + "HWLAB_WORKBENCH_KAFKA_REFRESH_REPLAY_GROUP_PREFIX", + "HWLAB_WORKBENCH_KAFKA_REFRESH_REPLAY_TIMEOUT_MS", + "HWLAB_WORKBENCH_KAFKA_REFRESH_REPLAY_SCAN_LIMIT", + "HWLAB_WORKBENCH_KAFKA_REFRESH_REPLAY_MATCHED_EVENT_LIMIT", + "HWLAB_WORKBENCH_KAFKA_REFRESH_REPLAY_LIVE_BUFFER_LIMIT", + ]; + + test("accepts the complete renderer contract and rejects every missing critical marker", () => { + const complete = { tasks: [{ taskSpec: { steps: [{ script: requiredMarkers.join("\n") }] } }] }; + expect(() => assertHwlabPipelineContracts(complete)).not.toThrow(); + for (const marker of requiredMarkers) { + const missing = { tasks: [{ taskSpec: { steps: [{ script: requiredMarkers.filter((item) => item !== marker).join("\n") }] } }] }; + expect(() => assertHwlabPipelineContracts(missing)).toThrow(marker); + } + }); +}); + +describe("shared HWLAB deploy overlay", () => { + test("preserves exact undefined/null semantics without mutating its input", () => { + const document = { + nodes: { NC01: { keep: "node" } }, + lanes: { v03: { + keep: "lane", + externalPostgres: { old: true }, + runtimeStore: { old: true }, + codeAgentRuntime: { old: true }, + gitMirror: { old: true }, + envRecipe: { keep: "recipe", downloadStack: { keep: "download" } }, + } }, + }; + const overlay = { + nodeId: "NC01", + lane: "v03", + gitopsRoot: "deploy/gitops/node/nc01", + gitUrl: "git@example/HWLAB.git", + sourceBranch: "v0.3", + gitopsBranch: "v0.3-gitops", + runtimeNamespace: "hwlab-v03", + publicApiUrl: "https://api.example", + publicWebUrl: "https://web.example", + catalogPath: "deploy/artifacts/nc01.yaml", + runtimePath: "deploy/gitops/node/nc01/hwlab-v03", + observability: { enabled: true }, + dockerProxyHttp: null, + dockerProxyHttps: null, + dockerNoProxyList: ["localhost"], + externalPostgres: null, + runtimeStore: null, + codeAgentRuntime: null, + deployYamlGitMirror: null, + }; + const rendered = applyNodeRuntimeDeployYamlOverlay(document, overlay); + expect(document.lanes.v03.externalPostgres).toEqual({ old: true }); + expect(rendered.lanes.v03).not.toHaveProperty("externalPostgres"); + expect(rendered.lanes.v03.runtimeStore).toBeNull(); + expect(rendered.lanes.v03.codeAgentRuntime).toBeNull(); + expect(rendered.lanes.v03.gitMirror).toBeNull(); + expect(rendered.lanes.v03.keep).toBe("lane"); + expect(nodeRuntimeDeployYamlOverlayShellScript().join("\n")).not.toContain(": Record { + const temporary = mkdtempSync(resolve(tmpdir(), "unidesk-pac-overlay-test-")); + try { + mkdirSync(resolve(temporary, "deploy")); + const document = { nodes: { NC01: {} }, lanes: { v03: { envRecipe: { downloadStack: {} } } } }; + const overlay = { + nodeId: "NC01", lane: "v03", gitopsRoot: "deploy/gitops/node/nc01", gitUrl: "git@example/HWLAB.git", + sourceBranch: "v0.3", gitopsBranch: "v0.3-gitops", runtimeNamespace: "hwlab-v03", + publicApiUrl: "https://api.example", publicWebUrl: "https://web.example", catalogPath: "deploy/artifacts/nc01.yaml", + runtimePath: "deploy/gitops/node/nc01/hwlab-v03", observability: { enabled: true }, + dockerProxyHttp: "http://proxy", dockerProxyHttps: "http://proxy", dockerNoProxyList: ["localhost"], + externalPostgres: { host: "postgres" }, runtimeStore: { mode: "postgres" }, codeAgentRuntime: { enabled: true }, + deployYamlGitMirror: { readUrl: "http://mirror" }, + }; + writeFileSync(resolve(temporary, "deploy", "deploy.yaml"), Bun.YAML.stringify(document)); + const overlayBase64 = Buffer.from(JSON.stringify(overlay), "utf8").toString("base64"); + const shell = [`overlay_b64='${overlayBase64}'`, ...nodeRuntimeDeployYamlOverlayShellScript()].join("\n"); + const result = spawnSync("sh", [], { + cwd: temporary, + input: shell, + encoding: "utf8", + timeout: 30_000, + env: { ...process.env, NODE_PATH: resolve(rootPath(), "node_modules") }, + }); + expect(result.status).toBe(0); + const shellRendered = Bun.YAML.parse(readFileSync(resolve(temporary, "deploy", "deploy.yaml"), "utf8")) as Record; + expect(shellRendered).toEqual(applyNodeRuntimeDeployYamlOverlay(document, overlay)); + } finally { + rmSync(temporary, { recursive: true, force: true }); + } + }); +}); diff --git a/scripts/src/platform-infra-pipelines-as-code-source-artifact.ts b/scripts/src/platform-infra-pipelines-as-code-source-artifact.ts new file mode 100644 index 00000000..9cca8f96 --- /dev/null +++ b/scripts/src/platform-infra-pipelines-as-code-source-artifact.ts @@ -0,0 +1,909 @@ +import { createHash } from "node:crypto"; +import { cpSync, existsSync, lstatSync, mkdirSync, mkdtempSync, readFileSync, realpathSync, renameSync, rmSync, statSync, symlinkSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { dirname, isAbsolute, relative, resolve, sep } from "node:path"; +import { spawnSync } from "node:child_process"; + +import { rootPath } from "./config"; +import { AGENTRUN_CONFIG_PATH, resolveAgentRunLaneTarget } from "./agentrun-lanes"; +import { renderAgentRunPipelineManifest } from "./agentrun-manifests"; +import { hwlabRuntimeLaneSpecForNode, isHwlabRuntimeLane, type HwlabRuntimeLaneSpec } from "./hwlab-node-lanes"; +import { nodeRuntimeGitopsRoot } from "./hwlab-node/cleanup"; +import { nodeRuntimePipelinePostprocessScript } from "./hwlab-node/render"; +import { nodeRuntimeRenderOverlay } from "./hwlab-node/web-probe"; +import { applyNodeRuntimeDeployYamlOverlay } from "./hwlab-node/deploy-overlay"; +import type { RenderedCliResult } from "./output"; +import { renderedCliResult } from "./agentrun/render"; +import { stableJsonSha256 } from "./stable-json"; + +export type PacSourceArtifactMode = "embedded-pipeline-spec" | "remote-pipeline-annotation"; +export type PacSourceArtifactRenderer = "agentrun-control-plane" | "hwlab-runtime-lane"; +export type PacSourceArtifactAction = "plan" | "check" | "write" | "status" | "verify-runtime"; + +export interface PacSourceArtifactSpec { + readonly mode: PacSourceArtifactMode; + readonly renderer: PacSourceArtifactRenderer; + readonly configRef: string; + readonly pipelineRunPath: string; + readonly pipelinePath: string | null; + readonly maxKeepRuns: number; +} + +export interface PacSourceArtifactBinding { + readonly target: { readonly id: string }; + readonly consumer: { + readonly id: string; + readonly node: string; + readonly lane: string; + readonly namespace: string; + readonly pipeline: string; + readonly pipelineRunPrefix: string; + readonly sourceArtifact: PacSourceArtifactSpec; + }; + readonly repository: { + readonly id: string; + readonly cloneUrl: string; + readonly owner: string; + readonly repo: string; + readonly params: Readonly>; + }; +} + +export interface PacSourceArtifactOptions { + readonly action: PacSourceArtifactAction; + readonly targetId: string; + readonly consumerId: string; + readonly sourceWorktree: string; + readonly sourceCommit: string | null; + readonly confirm: boolean; + readonly json: boolean; + readonly full: boolean; +} + +export interface PacSourceArtifactRuntimeObservation { + readonly live: PacSourceArtifactRuntimeItem; + readonly embedded: PacSourceArtifactRuntimeItem; +} + +export interface PacSourceArtifactRuntimeItem { + readonly status: "aligned" | "drifted" | "missing" | "unavailable"; + readonly name: string | null; + readonly canonicalSha256: string | null; + readonly firstDrift: PacSourceArtifactDrift | null; + readonly provenanceAligned: boolean; + readonly configRef: string | null; + readonly effectiveConfigSha256: string | null; + readonly sourceCommit: string | null; + readonly reason: string | null; + readonly candidateCount: number; +} + +export interface PacSourceArtifactDrift { + readonly path: string; + readonly expected: string; + readonly actual: string; +} + +export type PacSourceArtifactRuntimeObserver = (input: { + readonly binding: PacSourceArtifactBinding; + readonly desiredSpec: Record; + readonly provenance: PacSourceArtifactProvenance; + readonly sourceCommit: string | null; +}) => Promise; + +interface PacSourceArtifactProvenance { + readonly configRef: string; + readonly effectiveConfigSha256: string; + readonly renderer: PacSourceArtifactRenderer; +} + +interface DesiredArtifact { + readonly pipeline: Record; + readonly pipelineRun: Record; + readonly provenance: PacSourceArtifactProvenance; + readonly desiredSpec: Record; + readonly files: readonly { readonly path: string; readonly content: string }[]; +} + +interface SourceInspection { + readonly status: "aligned" | "drifted" | "missing"; + readonly aligned: boolean; + readonly pipelineCanonicalSha256: string | null; + readonly pipelineRunCanonicalSha256: string | null; + readonly firstDrift: PacSourceArtifactDrift | null; + readonly provenanceAligned: boolean; +} + +const provenanceKeys = { + configRef: "unidesk.ai/owning-config-ref", + effectiveConfigSha256: "unidesk.ai/effective-config-sha256", + renderer: "unidesk.ai/source-artifact-renderer", + mode: "unidesk.ai/source-artifact-mode", +} as const; + +export function parsePacSourceArtifactOptions(args: readonly string[]): PacSourceArtifactOptions { + const action = args[0]; + if (!isSourceArtifactAction(action)) { + throw new Error("source-artifact requires one of plan, check, write, status, verify-runtime"); + } + let targetId: string | null = null; + let consumerId: string | null = null; + let sourceWorktree: string | null = null; + let sourceCommit: string | null = null; + let confirm = false; + let json = false; + let full = false; + for (let index = 1; index < args.length; index += 1) { + const arg = args[index]; + if (arg === "--confirm") { + confirm = true; + continue; + } + if (arg === "--json") { + json = true; + continue; + } + if (arg === "--full") { + full = true; + continue; + } + if (arg === "--target" || arg === "--consumer" || arg === "--source-worktree" || arg === "--source-commit") { + const value = args[index + 1]; + if (value === undefined || value.startsWith("--")) throw new Error(`${arg} requires a value`); + if (arg === "--target") targetId = value; + else if (arg === "--consumer") consumerId = value; + else if (arg === "--source-worktree") sourceWorktree = value; + else sourceCommit = value.toLowerCase(); + index += 1; + continue; + } + throw new Error(`unsupported source-artifact option: ${arg}`); + } + if (targetId === null) throw new Error("source-artifact requires explicit --target"); + if (consumerId === null) throw new Error("source-artifact requires explicit --consumer"); + if (sourceWorktree === null) throw new Error("source-artifact requires explicit --source-worktree"); + if (!isAbsolute(sourceWorktree)) throw new Error("--source-worktree must be an absolute path"); + if (sourceCommit !== null && !/^[0-9a-f]{40}$/u.test(sourceCommit)) throw new Error("--source-commit must be a full 40-character git SHA"); + if (sourceCommit !== null && action !== "status" && action !== "verify-runtime") throw new Error("--source-commit is accepted only by source-artifact status or verify-runtime"); + if (action === "verify-runtime" && sourceCommit === null) throw new Error("source-artifact verify-runtime requires explicit --source-commit"); + if (action === "write" && !confirm) throw new Error("source-artifact write requires --confirm"); + if (action !== "write" && confirm) throw new Error("--confirm is accepted only by source-artifact write"); + return { action, targetId, consumerId, sourceWorktree, sourceCommit, confirm, json, full }; +} + +export async function runPacSourceArtifact( + binding: PacSourceArtifactBinding, + options: PacSourceArtifactOptions, + observeRuntime?: PacSourceArtifactRuntimeObserver, +): Promise> { + if (binding.target.id.toLowerCase() !== options.targetId.toLowerCase()) { + throw new Error(`source-artifact target ${options.targetId} does not match resolved target ${binding.target.id}`); + } + if (binding.consumer.id.toLowerCase() !== options.consumerId.toLowerCase()) { + throw new Error(`source-artifact consumer ${options.consumerId} does not match resolved consumer ${binding.consumer.id}`); + } + const sourceWorktree = validateSourceWorktree(options.sourceWorktree, binding); + const desired = renderDesiredArtifact(binding, sourceWorktree); + let source = inspectSourceArtifact(sourceWorktree, binding, desired); + const written: string[] = []; + if (options.action === "write") { + written.push(...writeArtifactFilesAtomically(sourceWorktree, desired.files)); + source = inspectSourceArtifact(sourceWorktree, binding, desired); + } + const needsRuntime = options.action === "status" || options.action === "verify-runtime"; + const runtime = needsRuntime + ? observeRuntime === undefined + ? unavailableRuntimeObservation() + : await observeRuntime({ binding, desiredSpec: desired.desiredSpec, provenance: desired.provenance, sourceCommit: options.sourceCommit }) + : null; + const sourceOk = source.aligned && source.provenanceAligned; + const runtimeOk = runtime !== null + && runtime.live.status === "aligned" + && runtime.live.provenanceAligned + && runtime.embedded.status === "aligned" + && runtime.embedded.provenanceAligned; + const ok = options.action === "check" + ? sourceOk + : options.action === "verify-runtime" + ? sourceOk && runtimeOk + : options.action === "write" + ? sourceOk + : true; + const desiredSpecSha256 = canonicalSha256(desired.desiredSpec); + return { + ok, + action: `platform-infra-pipelines-as-code-source-artifact-${options.action}`, + mutation: options.action === "write" && written.length > 0, + target: binding.target.id, + consumer: binding.consumer.id, + node: binding.consumer.node, + lane: binding.consumer.lane, + mode: binding.consumer.sourceArtifact.mode, + renderer: binding.consumer.sourceArtifact.renderer, + sourceWorktree, + files: { + pipelineRun: binding.consumer.sourceArtifact.pipelineRunPath, + pipeline: binding.consumer.sourceArtifact.pipelinePath, + written, + }, + provenance: desired.provenance, + desired: { + canonicalSha256: desiredSpecSha256, + pipeline: binding.consumer.pipeline, + namespace: binding.consumer.namespace, + }, + source, + runtime, + runtimeTarget: { sourceCommit: options.sourceCommit }, + runtimeAlignment: runtime === null + ? "not-requested" + : runtimeOk + ? "aligned" + : runtime.live.status === "missing" || runtime.embedded.status === "missing" + ? "missing" + : runtime.live.status === "unavailable" || runtime.embedded.status === "unavailable" + ? "unavailable" + : "pending", + boundary: { + desiredAuthority: "owning YAML plus domain renderer", + liveExportAllowed: false, + sourceWorktreeExplicit: true, + canonicalAdmissionDefaultsRemoved: [ + "tasks[].taskSpec.metadata={}", + "tasks[].taskSpec.spec=null", + "tasks[].taskSpec.params[].type=string", + "tasks[].taskSpec.results[].type=string", + "tasks[].taskSpec.steps[].computeResources={}", + "tasks[].taskSpec.sidecars[].computeResources={}", + ], + valuesPrinted: false, + }, + next: sourceOk + ? options.action === "verify-runtime" + ? null + : `After the consumer PR rolls out, run source-artifact verify-runtime for ${binding.consumer.id} with --source-commit .` + : options.action === "write" + ? "Generated files are still inconsistent; inspect firstDrift and do not publish." + : `Run source-artifact write --confirm against the explicit ${sourceWorktree} worktree.`, + }; +} + +export function renderPacSourceArtifactResult(result: Record): RenderedCliResult { + const files = record(result.files); + const desired = record(result.desired); + const source = record(result.source); + const runtime = result.runtime === null ? null : record(result.runtime); + const live = runtime === null ? null : record(runtime.live); + const embedded = runtime === null ? null : record(runtime.embedded); + const firstDrift = source.firstDrift === null ? null : record(source.firstDrift); + const lines = [ + "PAC SOURCE ARTIFACT", + `target=${text(result.target)} consumer=${text(result.consumer)} node=${text(result.node)} lane=${text(result.lane)}`, + `mode=${text(result.mode)} renderer=${text(result.renderer)} mutation=${text(result.mutation)}`, + `pipelineRun=${text(files.pipelineRun)} pipeline=${text(files.pipeline ?? "-")}`, + `desired=${shortSha(desired.canonicalSha256)} source=${shortSha(source.pipelineCanonicalSha256)} aligned=${text(source.aligned)} provenance=${text(source.provenanceAligned)}`, + `firstDrift=${firstDrift === null ? "-" : `${text(firstDrift.path)} expected=${text(firstDrift.expected)} actual=${text(firstDrift.actual)}`}`, + `runtime=${text(result.runtimeAlignment)} live=${text(live?.name)}@${shortSha(live?.canonicalSha256)} embedded=${text(embedded?.name)}@${shortSha(embedded?.canonicalSha256)} commit=${text(embedded?.sourceCommit ?? record(result.runtimeTarget).sourceCommit)} candidates=${text(embedded?.candidateCount)}`, + `runtimeReason=live:${text(live?.reason)} embedded:${text(embedded?.reason)}`, + `written=${Array.isArray(files.written) && files.written.length > 0 ? files.written.join(",") : "-"}`, + ]; + if (typeof result.next === "string") lines.push(`next=${result.next}`); + return renderedCliResult(result.ok !== false, String(result.action), `${lines.join("\n")}\n`); +} + +export function canonicalizePacPipelineSpec(value: unknown, path: readonly (string | number)[] = []): unknown { + if (Array.isArray(value)) return value.map((item, index) => canonicalizePacPipelineSpec(item, [...path, index])); + if (!isRecord(value)) return value; + const output: Record = {}; + for (const key of Object.keys(value).sort()) { + const child = value[key]; + const childPath = [...path, key]; + if (isProvenTektonAdmissionDefault(childPath, child)) continue; + output[key] = canonicalizePacPipelineSpec(child, childPath); + } + return output; +} + +export function canonicalSha256(value: unknown): string { + return `sha256:${createHash("sha256").update(JSON.stringify(canonicalizePacPipelineSpec(value))).digest("hex")}`; +} + +export function firstPacSourceArtifactDrift(expected: unknown, actual: unknown, path = "$"): PacSourceArtifactDrift | null { + const left = canonicalizePacPipelineSpec(expected); + const right = canonicalizePacPipelineSpec(actual); + return firstCanonicalDrift(left, right, path); +} + +function renderDesiredArtifact(binding: PacSourceArtifactBinding, sourceWorktree: string): DesiredArtifact { + const sourceArtifact = binding.consumer.sourceArtifact; + const rendered = sourceArtifact.renderer === "agentrun-control-plane" + ? renderAgentRunDesiredPipeline(binding) + : renderHwlabDesiredPipeline(binding, sourceWorktree); + const pipeline = withPipelineProvenance(rendered.pipeline, rendered.provenance, sourceArtifact.mode); + const desiredSpec = requiredRecord(pipeline.spec, "rendered Pipeline.spec"); + assertPipelineIdentity(pipeline, binding); + if (sourceArtifact.renderer === "hwlab-runtime-lane") assertHwlabPipelineContracts(desiredSpec); + const pipelineRun = sourceArtifact.mode === "embedded-pipeline-spec" + ? embeddedPipelineRun(binding, desiredSpec, rendered.provenance) + : remotePipelineRun(binding, pipeline, rendered.provenance); + const files = sourceArtifact.mode === "embedded-pipeline-spec" + ? [{ path: sourceArtifact.pipelineRunPath, content: yaml(pipelineRun) }] + : [ + { path: requiredString(sourceArtifact.pipelinePath, "sourceArtifact.pipelinePath"), content: `${JSON.stringify(pipeline)}\n` }, + { path: sourceArtifact.pipelineRunPath, content: yaml(pipelineRun) }, + ]; + return { pipeline, pipelineRun, provenance: rendered.provenance, desiredSpec, files }; +} + +function renderAgentRunDesiredPipeline(binding: PacSourceArtifactBinding): { pipeline: Record; provenance: PacSourceArtifactProvenance } { + const { spec } = resolveAgentRunLaneTarget({ node: binding.consumer.node, lane: binding.consumer.lane }); + const expectedConfigRef = `${AGENTRUN_CONFIG_PATH}#controlPlane.lanes.${spec.lane}`; + assertConfigRef(binding.consumer.sourceArtifact.configRef, expectedConfigRef); + assertBindingIdentity(binding, { + node: spec.nodeId, + lane: spec.lane, + namespace: spec.ci.namespace, + pipeline: spec.ci.pipeline, + pipelineRunPrefix: spec.ci.pipelineRunPrefix, + serviceAccount: spec.ci.serviceAccountName, + sourceBranch: spec.source.branch, + gitopsBranch: spec.gitops.branch, + }); + return { + pipeline: renderAgentRunPipelineManifest(spec), + provenance: { + configRef: expectedConfigRef, + effectiveConfigSha256: stableJsonSha256(spec), + renderer: "agentrun-control-plane", + }, + }; +} + +function renderHwlabDesiredPipeline(binding: PacSourceArtifactBinding, sourceWorktree: string): { pipeline: Record; provenance: PacSourceArtifactProvenance } { + if (!isHwlabRuntimeLane(binding.consumer.lane)) throw new Error(`HWLAB source artifact lane ${binding.consumer.lane} is not declared`); + const spec = hwlabRuntimeLaneSpecForNode(binding.consumer.lane, binding.consumer.node); + const expectedConfigRef = `config/hwlab-node-lanes.yaml#lanes.${spec.lane}.targets.${spec.nodeId}`; + assertConfigRef(binding.consumer.sourceArtifact.configRef, expectedConfigRef); + assertBindingIdentity(binding, { + node: spec.nodeId, + lane: spec.lane, + namespace: "hwlab-ci", + pipeline: spec.pipeline, + pipelineRunPrefix: spec.pipelineRunPrefix, + serviceAccount: spec.serviceAccountName, + sourceBranch: spec.sourceBranch, + gitopsBranch: spec.gitopsBranch, + }); + const pipeline = renderHwlabPipelineFromOwningYaml(spec, sourceWorktree); + return { + pipeline, + provenance: { + configRef: expectedConfigRef, + effectiveConfigSha256: stableJsonSha256(spec), + renderer: "hwlab-runtime-lane", + }, + }; +} + +function renderHwlabPipelineFromOwningYaml(spec: HwlabRuntimeLaneSpec, sourceWorktree: string): Record { + const temporaryRoot = mkdtempSync(resolve(tmpdir(), "unidesk-pac-source-artifact-")); + const temporarySource = resolve(temporaryRoot, "source"); + const output = resolve(temporaryRoot, "rendered"); + const sourceCommit = git(sourceWorktree, ["rev-parse", "HEAD"]); + try { + copySourceWorktreeForRender(sourceWorktree, temporarySource); + const overlayRecord = nodeRuntimeRenderOverlay(spec); + const deployPath = resolve(temporarySource, "deploy", "deploy.yaml"); + const deploy = parseYamlRecord(readFileSync(deployPath, "utf8"), deployPath); + writeFileSync(deployPath, `${Bun.YAML.stringify(applyNodeRuntimeDeployYamlOverlay(deploy, overlayRecord)).trim()}\n`); + const renderArgs = [ + "scripts/run-bun.mjs", + "scripts/gitops-render.mjs", + "--lane", spec.lane, + "--node", spec.nodeId, + "--gitops-root", nodeRuntimeGitopsRoot(spec), + "--catalog-path", spec.catalogPath, + "--image-tag-mode", "full", + "--source-revision", sourceCommit, + "--source-repo", spec.gitUrl, + "--source-branch", spec.sourceBranch, + "--gitops-branch", spec.gitopsBranch, + "--git-read-url", spec.gitReadUrl, + "--git-write-url", spec.gitWriteUrl, + "--registry-prefix", spec.registryPrefix, + "--runtime-endpoint", spec.publicApiUrl, + "--web-endpoint", spec.publicWebUrl, + "--out", output, + ]; + runChecked("node", renderArgs, temporarySource, 120_000, "HWLAB owning YAML renderer"); + const overlay = Buffer.from(JSON.stringify(overlayRecord), "utf8").toString("base64"); + const postprocess = [ + "set -eu", + `render_dir=${shellQuote(output)}`, + `overlay_b64=${shellQuote(overlay)}`, + ...nodeRuntimePipelinePostprocessScript(), + ].join("\n"); + runChecked("sh", ["-c", postprocess], temporarySource, 120_000, "HWLAB UniDesk domain postprocess/verify renderer"); + const path = resolve(output, spec.tektonDir, "pipeline.yaml"); + if (!existsSync(path)) throw new Error(`HWLAB domain renderer did not produce ${path}`); + return parseYamlRecord(readFileSync(path, "utf8"), path); + } finally { + rmSync(temporaryRoot, { recursive: true, force: true }); + } +} + +function copySourceWorktreeForRender(sourceWorktree: string, destination: string): void { + const excluded = new Set([".git", ".worktree", ".state", "node_modules", "coverage", "tmp", ".tmp"]); + cpSync(sourceWorktree, destination, { + recursive: true, + filter: (source) => { + const rel = relative(sourceWorktree, source); + if (rel === "") return true; + return !excluded.has(rel.split(sep)[0] ?? ""); + }, + }); + const nodeModules = resolve(sourceWorktree, "node_modules"); + if (existsSync(nodeModules)) symlinkSync(nodeModules, resolve(destination, "node_modules"), "dir"); +} + +function embeddedPipelineRun(binding: PacSourceArtifactBinding, desiredSpec: Record, provenance: PacSourceArtifactProvenance): Record { + return { + apiVersion: "tekton.dev/v1", + kind: "PipelineRun", + metadata: { + name: `${binding.consumer.pipelineRunPrefix}-{{ revision }}`, + namespace: binding.consumer.namespace, + annotations: pipelineRunAnnotations(binding, provenance), + labels: pipelineRunLabels(binding, "agentrun"), + }, + spec: { + pipelineSpec: desiredSpec, + taskRunTemplate: taskRunTemplate(binding), + params: pipelineRunParams(binding, desiredSpec), + workspaces: pipelineRunWorkspaces(binding, desiredSpec), + }, + }; +} + +function remotePipelineRun(binding: PacSourceArtifactBinding, pipeline: Record, provenance: PacSourceArtifactProvenance): Record { + const desiredSpec = requiredRecord(pipeline.spec, "rendered Pipeline.spec"); + const pipelinePath = requiredString(binding.consumer.sourceArtifact.pipelinePath, "sourceArtifact.pipelinePath"); + const annotations = { + ...pipelineRunAnnotations(binding, provenance), + "pipelinesascode.tekton.dev/pipeline": pipelinePath, + }; + if (provenance.renderer === "hwlab-runtime-lane") { + if (!isHwlabRuntimeLane(binding.consumer.lane)) throw new Error(`HWLAB source artifact lane ${binding.consumer.lane} is not declared`); + const spec = hwlabRuntimeLaneSpecForNode(binding.consumer.lane, binding.consumer.node); + Object.assign(annotations, { + "hwlab.pikastech.local/ci-contract": "tekton-native-primitive-tasks", + "hwlab.pikastech.local/download-profile": spec.downloadProfileId, + "hwlab.pikastech.local/gitops-branch": spec.gitopsBranch, + "hwlab.pikastech.local/network-profile": spec.networkProfileId, + "hwlab.pikastech.local/node": spec.nodeId, + "hwlab.pikastech.local/policy": "native-per-service-taskrun-image-publish", + "hwlab.pikastech.local/runtime-path": spec.runtimePath, + "hwlab.pikastech.local/source-branch": spec.sourceBranch, + "hwlab.pikastech.local/source-config": binding.consumer.sourceArtifact.pipelineRunPath, + }); + } + return { + apiVersion: "tekton.dev/v1", + kind: "PipelineRun", + metadata: { + generateName: `${binding.consumer.pipelineRunPrefix}-`, + namespace: binding.consumer.namespace, + annotations, + labels: pipelineRunLabels(binding, "hwlab"), + }, + spec: { + timeouts: { pipeline: binding.repository.params.pipeline_timeout }, + taskRunTemplate: taskRunTemplate(binding), + pipelineRef: { name: binding.consumer.pipeline }, + workspaces: pipelineRunWorkspaces(binding, desiredSpec), + params: pipelineRunParams(binding, desiredSpec), + }, + }; +} + +function pipelineRunAnnotations(binding: PacSourceArtifactBinding, provenance: PacSourceArtifactProvenance): Record { + const branch = requiredParam(binding, "source_branch"); + return { + "pipelinesascode.tekton.dev/on-event": "[push]", + "pipelinesascode.tekton.dev/on-target-branch": `[${branch}]`, + "pipelinesascode.tekton.dev/on-cel-expression": `event == 'push' && target_branch == '${branch}' && node == '${binding.consumer.node}'`, + "pipelinesascode.tekton.dev/max-keep-runs": String(binding.consumer.sourceArtifact.maxKeepRuns), + [provenanceKeys.configRef]: provenance.configRef, + [provenanceKeys.effectiveConfigSha256]: provenance.effectiveConfigSha256, + [provenanceKeys.renderer]: provenance.renderer, + [provenanceKeys.mode]: binding.consumer.sourceArtifact.mode, + }; +} + +function pipelineRunLabels(binding: PacSourceArtifactBinding, partOf: "agentrun" | "hwlab"): Record { + return partOf === "agentrun" + ? { + "app.kubernetes.io/part-of": "agentrun", + "agentrun.pikastech.local/lane": "v0.2", + "agentrun.pikastech.local/node": binding.consumer.node, + "agentrun.pikastech.local/source-commit": "{{ revision }}", + "agentrun.pikastech.local/trigger": "pipelines-as-code", + } + : { + "app.kubernetes.io/name": `${binding.consumer.id}-pac`, + "app.kubernetes.io/part-of": "hwlab", + "unidesk.ai/source-commit": "{{ revision }}", + "hwlab.pikastech.local/gitops-target": binding.consumer.lane, + "hwlab.pikastech.local/source-commit": "{{ revision }}", + "hwlab.pikastech.local/trigger": "pipelines-as-code", + }; +} + +function taskRunTemplate(binding: PacSourceArtifactBinding): Record { + return { + serviceAccountName: `{{ service_account }}`, + podTemplate: { + hostNetwork: true, + dnsPolicy: "ClusterFirstWithHostNet", + securityContext: { fsGroup: 1000 }, + }, + }; +} + +function pipelineRunParams(binding: PacSourceArtifactBinding, desiredSpec: Record): Record[] { + const params = arrayRecords(desiredSpec.params, "Pipeline.spec.params"); + return params.map((param) => { + const name = requiredString(param.name, "Pipeline.spec.params[].name"); + if (name === "revision" || name === "source-commit") return { name, value: "{{ revision }}" }; + if (name === "source-stage-ref") return { name, value: "{{ source_snapshot_prefix }}/{{ revision }}" }; + if (name === "git-url") return { name, value: "{{ repo_url }}" }; + const repositoryParam = name.replaceAll("-", "_"); + if (Object.prototype.hasOwnProperty.call(binding.repository.params, repositoryParam)) { + return { name, value: `{{ ${repositoryParam} }}` }; + } + if (Object.prototype.hasOwnProperty.call(param, "default")) return { name, value: param.default }; + throw new Error(`Pipeline param ${name} has no default or PaC repository parameter`); + }); +} + +function pipelineRunWorkspaces(binding: PacSourceArtifactBinding, desiredSpec: Record): Record[] { + return arrayRecords(desiredSpec.workspaces, "Pipeline.spec.workspaces").map((workspace) => { + const name = requiredString(workspace.name, "Pipeline.spec.workspaces[].name"); + if (name === "source") { + return { + name, + volumeClaimTemplate: { + spec: { + accessModes: ["ReadWriteOnce"], + resources: { requests: { storage: requiredParam(binding, "workspace_pvc_size") } }, + }, + }, + }; + } + if (name === "git-ssh") return { name, secret: { secretName: requiredParam(binding, "git_ssh_secret") } }; + throw new Error(`unsupported Pipeline workspace ${name}; declare a renderer mapping before generating the source artifact`); + }); +} + +function withPipelineProvenance(pipeline: Record, provenance: PacSourceArtifactProvenance, mode: PacSourceArtifactMode): Record { + const next = structuredClone(pipeline); + const metadata = requiredRecord(next.metadata, "rendered Pipeline.metadata"); + const annotations = isRecord(metadata.annotations) ? metadata.annotations : {}; + metadata.annotations = { + ...annotations, + [provenanceKeys.configRef]: provenance.configRef, + [provenanceKeys.effectiveConfigSha256]: provenance.effectiveConfigSha256, + [provenanceKeys.renderer]: provenance.renderer, + [provenanceKeys.mode]: mode, + }; + return next; +} + +function inspectSourceArtifact(sourceWorktree: string, binding: PacSourceArtifactBinding, desired: DesiredArtifact): SourceInspection { + const sourceArtifact = binding.consumer.sourceArtifact; + const pipelineRunPath = safeArtifactPath(sourceWorktree, sourceArtifact.pipelineRunPath); + const pipelinePath = sourceArtifact.pipelinePath === null ? null : safeArtifactPath(sourceWorktree, sourceArtifact.pipelinePath); + if (!existsSync(pipelineRunPath) || (pipelinePath !== null && !existsSync(pipelinePath))) { + return { status: "missing", aligned: false, pipelineCanonicalSha256: null, pipelineRunCanonicalSha256: null, firstDrift: { path: "$", expected: "generated artifact", actual: "missing" }, provenanceAligned: false }; + } + const actualPipelineRun = parseYamlRecord(readFileSync(pipelineRunPath, "utf8"), sourceArtifact.pipelineRunPath); + const actualPipeline = sourceArtifact.mode === "remote-pipeline-annotation" + ? parseYamlRecord(readFileSync(requiredString(pipelinePath, "pipelinePath"), "utf8"), requiredString(sourceArtifact.pipelinePath, "pipelinePath")) + : { + apiVersion: "tekton.dev/v1", + kind: "Pipeline", + metadata: desired.pipeline.metadata, + spec: requiredRecord(requiredRecord(actualPipelineRun.spec, "PipelineRun.spec").pipelineSpec, "PipelineRun.spec.pipelineSpec"), + }; + const pipelineDrift = firstPacSourceArtifactDrift(desired.pipeline, actualPipeline); + const pipelineRunDrift = firstPacSourceArtifactDrift(desired.pipelineRun, actualPipelineRun); + const actualProvenance = sourceArtifact.mode === "remote-pipeline-annotation" + ? provenanceFromManifest(actualPipeline) + : provenanceFromManifest(actualPipelineRun); + const wrapperProvenance = provenanceFromManifest(actualPipelineRun); + const provenanceAligned = provenanceEquals(actualProvenance, desired.provenance) && provenanceEquals(wrapperProvenance, desired.provenance); + const aligned = pipelineDrift === null && pipelineRunDrift === null && provenanceAligned; + return { + status: aligned ? "aligned" : "drifted", + aligned, + pipelineCanonicalSha256: canonicalSha256(requiredRecord(actualPipeline.spec, "source Pipeline.spec")), + pipelineRunCanonicalSha256: canonicalSha256(actualPipelineRun), + firstDrift: pipelineDrift ?? pipelineRunDrift ?? (provenanceAligned ? null : { path: "$.metadata.annotations.provenance", expected: JSON.stringify(desired.provenance), actual: JSON.stringify(actualProvenance) }), + provenanceAligned, + }; +} + +function validateSourceWorktree(input: string, binding: PacSourceArtifactBinding): string { + if (!existsSync(input) || !statSync(input).isDirectory()) throw new Error(`source worktree does not exist or is not a directory: ${input}`); + const worktree = realpathSync(input); + const top = realpathSync(git(worktree, ["rev-parse", "--show-toplevel"])); + if (top !== worktree) throw new Error(`--source-worktree must be the git worktree root; resolved ${top}`); + const remote = git(worktree, ["remote", "get-url", "origin"]); + const repoIdentity = binding.repository.repo.toLowerCase().replace(/\.git$/u, ""); + const canonicalIdentity = repoIdentity.replace(/^pikastech-/u, "pikastech/"); + const allowedIdentities = new Set([ + canonicalIdentity, + `${binding.repository.owner}/${repoIdentity}`.toLowerCase(), + remoteRepositoryIdentity(binding.repository.cloneUrl), + ]); + const observedIdentity = remoteRepositoryIdentity(remote); + if (!allowedIdentities.has(observedIdentity)) { + throw new Error(`source worktree origin does not match PaC repository ${binding.repository.repo}; observed ${remote}`); + } + safeArtifactPath(worktree, binding.consumer.sourceArtifact.pipelineRunPath); + if (binding.consumer.sourceArtifact.pipelinePath !== null) safeArtifactPath(worktree, binding.consumer.sourceArtifact.pipelinePath); + return worktree; +} + +function remoteRepositoryIdentity(remote: string): string { + const trimmed = remote.trim(); + const scp = /^[^/@\s]+@[^:\s]+:(.+)$/u.exec(trimmed); + let path = scp?.[1] ?? ""; + if (path.length === 0) { + try { + path = new URL(trimmed).pathname; + } catch { + throw new Error(`git remote must be an SSH or URL repository identity; observed ${remote}`); + } + } + return path.replace(/^\/+|\/+$/gu, "").replace(/\.git$/iu, "").toLowerCase(); +} + +function writeArtifactFilesAtomically(sourceWorktree: string, files: readonly { readonly path: string; readonly content: string }[]): string[] { + const pending = files.map((file) => ({ ...file, output: safeArtifactPath(sourceWorktree, file.path) })) + .filter((file) => !existsSync(file.output) || readFileSync(file.output, "utf8") !== file.content); + if (pending.length === 0) return []; + for (const file of pending) { + mkdirSync(dirname(file.output), { recursive: true }); + safeArtifactPath(sourceWorktree, file.path); + } + const token = `${process.pid}-${Date.now()}`; + const staged = pending.map((file, index) => ({ ...file, temporary: `${file.output}.unidesk-${token}-${index}.tmp` })); + try { + for (const file of staged) writeFileSync(file.temporary, file.content, { flag: "wx" }); + for (const file of staged) renameSync(file.temporary, file.output); + } finally { + for (const file of staged) rmSync(file.temporary, { force: true }); + } + return pending.map((file) => file.path); +} + +function safeArtifactPath(worktree: string, relativePath: string): string { + if (relativePath.length === 0 || isAbsolute(relativePath) || relativePath.split(/[\\/]/u).includes("..")) { + throw new Error(`source artifact path must be a non-empty worktree-relative path without ..: ${relativePath}`); + } + const root = realpathSync(worktree); + const candidate = resolve(root, relativePath); + if (candidate !== root && !candidate.startsWith(`${root}${sep}`)) throw new Error(`source artifact path escapes worktree: ${relativePath}`); + let current = root; + for (const segment of relativePath.split(/[\\/]/u).filter(Boolean)) { + current = resolve(current, segment); + if (!existsSync(current)) continue; + if (lstatSync(current).isSymbolicLink()) throw new Error(`source artifact path crosses symlink: ${relativePath}`); + const resolved = realpathSync(current); + if (resolved !== root && !resolved.startsWith(`${root}${sep}`)) throw new Error(`source artifact path resolves outside worktree: ${relativePath}`); + } + return candidate; +} + +function assertConfigRef(actual: string, expected: string): void { + if (actual !== expected) throw new Error(`sourceArtifact.configRef must equal resolved owning selector ${expected}; observed ${actual}`); + const [file, selector] = actual.split("#", 2); + if (file === undefined || selector === undefined || selector.length === 0) throw new Error(`invalid sourceArtifact.configRef: ${actual}`); + const parsed = Bun.YAML.parse(readFileSync(rootPath(...file.split("/")), "utf8")) as unknown; + let value = parsed; + for (const segment of selector.split(".")) { + if (!isRecord(value) || !Object.prototype.hasOwnProperty.call(value, segment)) throw new Error(`sourceArtifact.configRef selector does not exist: ${actual}`); + value = value[segment]; + } + if (!isRecord(value)) throw new Error(`sourceArtifact.configRef must resolve to an owning YAML object: ${actual}`); +} + +function assertBindingIdentity(binding: PacSourceArtifactBinding, expected: { + node: string; + lane: string; + namespace: string; + pipeline: string; + pipelineRunPrefix: string; + serviceAccount: string; + sourceBranch: string; + gitopsBranch: string; +}): void { + const observed = { + node: binding.consumer.node, + lane: binding.consumer.lane, + namespace: binding.consumer.namespace, + pipeline: binding.consumer.pipeline, + pipelineRunPrefix: binding.consumer.pipelineRunPrefix, + serviceAccount: requiredParam(binding, "service_account"), + sourceBranch: requiredParam(binding, "source_branch"), + gitopsBranch: requiredParam(binding, "gitops_branch"), + }; + for (const key of Object.keys(expected) as Array) { + if (observed[key] !== expected[key]) throw new Error(`PaC ${binding.consumer.id} ${key}=${observed[key]} does not match owning renderer ${expected[key]}`); + } + if (requiredParam(binding, "pipeline_name") !== expected.pipeline) throw new Error(`PaC ${binding.consumer.id} repository pipeline_name does not match ${expected.pipeline}`); + if (requiredParam(binding, "pipeline_run_prefix") !== expected.pipelineRunPrefix) throw new Error(`PaC ${binding.consumer.id} repository pipeline_run_prefix does not match ${expected.pipelineRunPrefix}`); + if (requiredParam(binding, "node") !== expected.node) throw new Error(`PaC ${binding.consumer.id} repository node does not match ${expected.node}`); +} + +function assertPipelineIdentity(pipeline: Record, binding: PacSourceArtifactBinding): void { + if (pipeline.apiVersion !== "tekton.dev/v1" || pipeline.kind !== "Pipeline") throw new Error("domain renderer must return tekton.dev/v1 Pipeline"); + const metadata = requiredRecord(pipeline.metadata, "rendered Pipeline.metadata"); + if (metadata.name !== binding.consumer.pipeline) throw new Error(`rendered Pipeline name ${String(metadata.name)} does not match consumer ${binding.consumer.pipeline}`); + if (metadata.namespace !== binding.consumer.namespace) throw new Error(`rendered Pipeline namespace ${String(metadata.namespace)} does not match consumer ${binding.consumer.namespace}`); +} + +export function assertHwlabPipelineContracts(spec: Record): void { + const text = JSON.stringify(spec); + const required = [ + "unidesk-runtime-gitops-postprocess", + "unidesk-runtime-gitops-verify", + "HWLAB_WORKBENCH_KAFKA_REFRESH_REPLAY_ENABLED", + "HWLAB_WORKBENCH_KAFKA_REFRESH_REPLAY_GROUP_PREFIX", + "HWLAB_WORKBENCH_KAFKA_REFRESH_REPLAY_TIMEOUT_MS", + "HWLAB_WORKBENCH_KAFKA_REFRESH_REPLAY_SCAN_LIMIT", + "HWLAB_WORKBENCH_KAFKA_REFRESH_REPLAY_MATCHED_EVENT_LIMIT", + "HWLAB_WORKBENCH_KAFKA_REFRESH_REPLAY_LIVE_BUFFER_LIMIT", + ]; + const missing = required.filter((value) => !text.includes(value)); + if (missing.length > 0) throw new Error(`HWLAB domain renderer missing required postprocess/verify refresh contract: ${missing.join(", ")}`); +} + +function provenanceFromManifest(manifest: Record): PacSourceArtifactProvenance | null { + const metadata = isRecord(manifest.metadata) ? manifest.metadata : {}; + const annotations = isRecord(metadata.annotations) ? metadata.annotations : {}; + const configRef = annotations[provenanceKeys.configRef]; + const effectiveConfigSha256 = annotations[provenanceKeys.effectiveConfigSha256]; + const renderer = annotations[provenanceKeys.renderer]; + if (typeof configRef !== "string" || typeof effectiveConfigSha256 !== "string" || (renderer !== "agentrun-control-plane" && renderer !== "hwlab-runtime-lane")) return null; + return { configRef, effectiveConfigSha256, renderer }; +} + +function provenanceEquals(actual: PacSourceArtifactProvenance | null, expected: PacSourceArtifactProvenance): boolean { + return actual !== null + && actual.configRef === expected.configRef + && actual.effectiveConfigSha256 === expected.effectiveConfigSha256 + && actual.renderer === expected.renderer; +} + +function firstCanonicalDrift(expected: unknown, actual: unknown, path: string): PacSourceArtifactDrift | null { + if (Object.is(expected, actual)) return null; + if (Array.isArray(expected) || Array.isArray(actual)) { + if (!Array.isArray(expected) || !Array.isArray(actual)) return drift(path, expected, actual); + if (expected.length !== actual.length) return drift(`${path}.length`, expected.length, actual.length); + for (let index = 0; index < expected.length; index += 1) { + const child = firstCanonicalDrift(expected[index], actual[index], `${path}[${index}]`); + if (child !== null) return child; + } + return null; + } + if (isRecord(expected) || isRecord(actual)) { + if (!isRecord(expected) || !isRecord(actual)) return drift(path, expected, actual); + const keys = [...new Set([...Object.keys(expected), ...Object.keys(actual)])].sort(); + for (const key of keys) { + if (!Object.prototype.hasOwnProperty.call(expected, key) || !Object.prototype.hasOwnProperty.call(actual, key)) return drift(`${path}.${key}`, expected[key], actual[key]); + const child = firstCanonicalDrift(expected[key], actual[key], `${path}.${key}`); + if (child !== null) return child; + } + return null; + } + return drift(path, expected, actual); +} + +function isProvenTektonAdmissionDefault(path: readonly (string | number)[], value: unknown): boolean { + const normalized = path.map((segment) => typeof segment === "number" ? "*" : segment).join("."); + const emptyRecord = isRecord(value) && Object.keys(value).length === 0; + const atTaskSpec = (suffix: string): boolean => [ + `tasks.*.taskSpec.${suffix}`, + `spec.tasks.*.taskSpec.${suffix}`, + `spec.pipelineSpec.tasks.*.taskSpec.${suffix}`, + ].includes(normalized); + if (atTaskSpec("metadata")) return emptyRecord; + if (atTaskSpec("spec")) return value === null; + if (atTaskSpec("params.*.type")) return value === "string"; + if (atTaskSpec("results.*.type")) return value === "string"; + if (atTaskSpec("steps.*.computeResources")) return emptyRecord; + if (atTaskSpec("sidecars.*.computeResources")) return emptyRecord; + return false; +} + +function drift(path: string, expected: unknown, actual: unknown): PacSourceArtifactDrift { + return { path, expected: compactValue(expected), actual: compactValue(actual) }; +} + +function compactValue(value: unknown): string { + const raw = JSON.stringify(value); + if (raw === undefined) return String(value); + return raw.length <= 160 ? raw : `${raw.slice(0, 157)}...`; +} + +function unavailableRuntimeObservation(): PacSourceArtifactRuntimeObservation { + const item: PacSourceArtifactRuntimeItem = { status: "unavailable", name: null, canonicalSha256: null, firstDrift: null, provenanceAligned: false, configRef: null, effectiveConfigSha256: null, sourceCommit: null, reason: "runtime-observer-not-configured", candidateCount: 0 }; + return { live: item, embedded: item }; +} + +function parseYamlRecord(text: string, label: string): Record { + const parsed = Bun.YAML.parse(text) as unknown; + if (!isRecord(parsed)) throw new Error(`${label} must contain one YAML object`); + return parsed; +} + +function yaml(value: Record): string { + return `${Bun.YAML.stringify(value).trim()}\n`; +} + +function requiredParam(binding: PacSourceArtifactBinding, key: string): string { + return requiredString(binding.repository.params[key], `PaC repository ${binding.repository.id} params.${key}`); +} + +function requiredString(value: unknown, label: string): string { + if (typeof value !== "string" || value.length === 0) throw new Error(`${label} must be a non-empty string`); + return value; +} + +function requiredRecord(value: unknown, label: string): Record { + if (!isRecord(value)) throw new Error(`${label} must be an object`); + return value; +} + +function arrayRecords(value: unknown, label: string): Record[] { + if (!Array.isArray(value) || value.some((item) => !isRecord(item))) throw new Error(`${label} must be an array of objects`); + return value as Record[]; +} + +function isRecord(value: unknown): value is Record { + return typeof value === "object" && value !== null && !Array.isArray(value); +} + +function record(value: unknown): Record { + return isRecord(value) ? value : {}; +} + +function isSourceArtifactAction(value: string | undefined): value is PacSourceArtifactAction { + return value === "plan" || value === "check" || value === "write" || value === "status" || value === "verify-runtime"; +} + +function git(worktree: string, args: readonly string[]): string { + const result = spawnSync("git", ["-C", worktree, ...args], { encoding: "utf8", timeout: 15_000, maxBuffer: 1024 * 1024 }); + if (result.status !== 0) throw new Error(`git ${args.join(" ")} failed for ${worktree}: ${(result.stderr || result.stdout).trim().slice(0, 1000)}`); + return result.stdout.trim(); +} + +function runChecked(command: string, args: readonly string[], cwd: string, timeout: number, label: string): void { + const result = spawnSync(command, [...args], { cwd, encoding: "utf8", timeout, maxBuffer: 32 * 1024 * 1024 }); + if (result.status !== 0) throw new Error(`${label} failed: ${(result.stderr || result.stdout).trim().slice(-4000)}`); +} + +function shellQuote(value: string): string { + return `'${value.replaceAll("'", `'"'"'`)}'`; +} + +function text(value: unknown): string { + if (value === null || value === undefined || value === "") return "-"; + return String(value); +} + +function shortSha(value: unknown): string { + return typeof value === "string" && value.length > 0 ? value.replace(/^sha256:/u, "").slice(0, 12) : "-"; +} diff --git a/scripts/src/platform-infra-pipelines-as-code.ts b/scripts/src/platform-infra-pipelines-as-code.ts index 841363c1..68e3a79a 100644 --- a/scripts/src/platform-infra-pipelines-as-code.ts +++ b/scripts/src/platform-infra-pipelines-as-code.ts @@ -1,4 +1,4 @@ -import { createHash, randomBytes } from "node:crypto"; +import { randomBytes } from "node:crypto"; import { chmodSync, existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs"; import { dirname, join } from "node:path"; import type { UniDeskConfig } from "./config"; @@ -17,11 +17,21 @@ import { sha256Fingerprint, } from "./platform-infra-ops-library"; import { materializeYamlComposition } from "./yaml-composition"; +import { + canonicalizePacPipelineSpec, + parsePacSourceArtifactOptions, + renderPacSourceArtifactResult, + runPacSourceArtifact, + type PacSourceArtifactBinding, + type PacSourceArtifactRuntimeObservation, + type PacSourceArtifactSpec, +} from "./platform-infra-pipelines-as-code-source-artifact"; const configFile = rootPath("config", "platform-infra", "pipelines-as-code.yaml"); const configLabel = "config/platform-infra/pipelines-as-code.yaml"; const remoteScriptFile = rootPath("scripts", "src", "platform-infra-pipelines-as-code-remote.sh"); const evaluatorFile = rootPath("scripts", "native", "cicd", "pac-status-evaluator.cjs"); +const sourceArtifactRuntimeObserverFile = rootPath("scripts", "native", "cicd", "pac-source-artifact-runtime.mjs"); const fieldManager = "unidesk-platform-infra-pipelines-as-code"; const y = createYamlFieldReader(configLabel); @@ -132,6 +142,7 @@ interface PacConsumer { repositoryRef: string; closeoutGitOpsMirrorFlush: boolean; closeoutGitOpsMirrorLane: "v02" | "v03" | null; + sourceArtifact: PacSourceArtifactSpec | null; } interface CommonOptions { @@ -216,12 +227,78 @@ export async function runPlatformInfraPipelinesAsCodeCommand(config: UniDeskConf const result = await webhookTest(config, options); return options.json || options.full || options.raw ? result : renderWebhookTest(result); } + if (action === "source-artifact") { + if (args.length === 1 || args.slice(1).includes("--help") || args.slice(1).includes("-h")) return sourceArtifactHelp(); + const structuredError = args.includes("--json") || args.includes("--full"); + try { + const options = parsePacSourceArtifactOptions(args.slice(1)); + const pac = readPacConfig(); + const target = resolveTarget(pac, options.targetId); + const consumer = resolveConsumer(pac, options.consumerId); + if (consumer.node.toLowerCase() !== target.id.toLowerCase()) throw new Error(`Pipelines-as-Code consumer ${consumer.id} belongs to ${consumer.node}, not target ${target.id}`); + if (consumer.sourceArtifact === null) throw new Error(`Pipelines-as-Code consumer ${consumer.id} has no sourceArtifact owner in ${configLabel}`); + const repository = resolveRepository(pac, consumer.repositoryRef); + const binding: PacSourceArtifactBinding = { + target: { id: target.id }, + consumer: { + id: consumer.id, + node: consumer.node, + lane: consumer.lane, + namespace: consumer.namespace, + pipeline: consumer.pipeline, + pipelineRunPrefix: consumer.pipelineRunPrefix, + sourceArtifact: consumer.sourceArtifact, + }, + repository: { + id: repository.id, + cloneUrl: repository.cloneUrl, + owner: repository.owner, + repo: repository.repo, + params: repository.params, + }, + }; + const result = await runPacSourceArtifact(binding, options, async ({ desiredSpec, provenance, sourceCommit }) => await observeSourceArtifactRuntime(config, target, consumer, desiredSpec, provenance, sourceCommit)); + return options.json || options.full ? result : renderPacSourceArtifactResult(result); + } catch (error) { + if (structuredError) throw error; + return sourceArtifactValidationError(error); + } + } return { ok: false, error: "unsupported-platform-infra-pipelines-as-code-command", args, help: help() }; } +function sourceArtifactHelp(): RenderedCliResult { + const lines = [ + "PAC SOURCE ARTIFACT", + "Generate and verify consumer-declared PaC source artifacts from owning YAML plus the shared domain renderer.", + "", + "Usage:", + " ... source-artifact plan --target --consumer --source-worktree ", + " ... source-artifact check --target --consumer --source-worktree ", + " ... source-artifact write --target --consumer --source-worktree --confirm", + " ... source-artifact status --target --consumer --source-worktree [--source-commit ]", + " ... source-artifact verify-runtime --target --consumer --source-worktree --source-commit ", + "", + "plan/check/write compare desired YAML-rendered state with source files and never access runtime.", + "status is non-gating runtime diagnostics; verify-runtime is fail-closed and binds the PipelineRun to the exact source commit.", + "Use --json or --full for explicit structured output; source specs and Secret values are never printed.", + ]; + return { ok: true, command: "platform-infra-pipelines-as-code-source-artifact-help", renderedText: `${lines.join("\n")}\n`, contentType: "text/plain" }; +} + +function sourceArtifactValidationError(error: unknown): RenderedCliResult { + const reason = String(error instanceof Error ? error.message : error).replace(/[\r\n]+/gu, " ").slice(0, 1000); + return { + ok: false, + command: "platform-infra-pipelines-as-code-source-artifact-validation", + renderedText: `PAC SOURCE ARTIFACT ERROR\nreason=${reason}\nnext=Fix the explicit target, consumer, source worktree, or owning YAML declaration; rerun plan before write.\n`, + contentType: "text/plain", + }; +} + function help(): Record { return { - command: "platform-infra pipelines-as-code plan|apply|status|closeout|history|debug-step", + command: "platform-infra pipelines-as-code plan|apply|status|closeout|history|debug-step|source-artifact", configTruth: configLabel, usage: [ "bun scripts/cli.ts platform-infra pipelines-as-code plan --target JD01", @@ -232,6 +309,9 @@ function help(): Record { "bun scripts/cli.ts platform-infra pipelines-as-code history --target JD01 [--consumer hwlab-jd01-v03] [--limit 10]", "bun scripts/cli.ts platform-infra pipelines-as-code history --target JD01 --id ", "bun scripts/cli.ts platform-infra pipelines-as-code debug-step --target JD01 [--consumer ] [--id ] [--json]", + "bun scripts/cli.ts platform-infra pipelines-as-code source-artifact check --target NC01 --consumer agentrun-nc01-v02 --source-worktree /abs/worktree", + "bun scripts/cli.ts platform-infra pipelines-as-code source-artifact write --target NC01 --consumer agentrun-nc01-v02 --source-worktree /abs/worktree --confirm", + "bun scripts/cli.ts platform-infra pipelines-as-code source-artifact verify-runtime --target NC01 --consumer agentrun-nc01-v02 --source-worktree /abs/worktree --source-commit ", ], diagnostics: "webhook-test exists only for bounded connectivity diagnosis and must not be used as delivery evidence.", boundary: "Sole CI trigger path for GH-1552/GH-1607: GitHub PR merge -> GitHub webhook bridge -> Gitea mirror/snapshot -> Pipelines-as-Code -> Tekton -> GitOps/Argo/k8s runtime.", @@ -353,9 +433,112 @@ function parseConsumer(consumer: Record, path: string, defaultR repositoryRef: typeof consumer.repositoryRef === "string" && consumer.repositoryRef.length > 0 ? consumer.repositoryRef : defaultRepositoryRef, closeoutGitOpsMirrorFlush: y.booleanField(consumer, "closeoutGitOpsMirrorFlush", path), closeoutGitOpsMirrorLane: consumer.closeoutGitOpsMirrorLane === undefined ? null : y.enumField(consumer, "closeoutGitOpsMirrorLane", path, ["v02", "v03"] as const), + sourceArtifact: consumer.sourceArtifact === undefined ? null : parseSourceArtifact(y.objectField(consumer, "sourceArtifact", path), `${path}.sourceArtifact`), }; } +function parseSourceArtifact(value: Record, path: string): PacSourceArtifactSpec { + const mode = y.enumField(value, "mode", path, ["embedded-pipeline-spec", "remote-pipeline-annotation"] as const); + const renderer = y.enumField(value, "renderer", path, ["agentrun-control-plane", "hwlab-runtime-lane"] as const); + const pipelineRunPath = sourceArtifactPath(y.stringField(value, "pipelineRunPath", path), `${path}.pipelineRunPath`); + const pipelinePath = value.pipelinePath === undefined ? null : sourceArtifactPath(y.stringField(value, "pipelinePath", path), `${path}.pipelinePath`); + if (mode === "embedded-pipeline-spec" && pipelinePath !== null) throw new Error(`${path}.pipelinePath is forbidden for embedded-pipeline-spec`); + if (mode === "remote-pipeline-annotation" && pipelinePath === null) throw new Error(`${path}.pipelinePath is required for remote-pipeline-annotation`); + if (renderer === "agentrun-control-plane" && mode !== "embedded-pipeline-spec") throw new Error(`${path}.renderer agentrun-control-plane requires embedded-pipeline-spec`); + if (renderer === "hwlab-runtime-lane" && mode !== "remote-pipeline-annotation") throw new Error(`${path}.renderer hwlab-runtime-lane requires remote-pipeline-annotation`); + return { + mode, + renderer, + configRef: y.stringField(value, "configRef", path), + pipelineRunPath, + pipelinePath, + maxKeepRuns: positiveInteger(value, "maxKeepRuns", path), + }; +} + +function sourceArtifactPath(value: string, path: string): string { + if (value.startsWith("/") || value.split(/[\\/]/u).includes("..") || !value.endsWith(".yaml")) throw new Error(`${path} must be a worktree-relative .yaml path without ..`); + return value; +} + +async function observeSourceArtifactRuntime( + config: UniDeskConfig, + target: PacTarget, + consumer: PacConsumer, + desiredSpec: Record, + provenance: { readonly configRef: string; readonly effectiveConfigSha256: string; readonly renderer: string }, + sourceCommit: string | null, +): Promise { + const script = renderPacSourceArtifactRuntimeObserverShell({ + namespace: consumer.namespace, + pipeline: consumer.pipeline, + pipelineRunPrefix: consumer.pipelineRunPrefix, + desiredSpec: canonicalizePacPipelineSpec(desiredSpec), + provenance, + sourceCommit, + }); + + const captured = await capture(config, target.route, ["sh"], script); + if (captured.exitCode !== 0) { + const compact = compactCapture(captured); + const detail = String(compact.stderrTail ?? compact.stdoutTail ?? "").trim().split(/\r?\n/u).filter(Boolean)[0]?.replace(/\s+/gu, " ").slice(0, 240); + return unavailableSourceArtifactRuntime(`runtime-transport-exit-${captured.exitCode}${detail ? `:${detail}` : ""}`, sourceCommit); + } + const parsed = parseJsonOutput(captured.stdout); + if (parsed === null) return unavailableSourceArtifactRuntime("runtime-observer-invalid-json", sourceCommit); + return { + live: runtimeSourceArtifactItem(parsed.live), + embedded: runtimeSourceArtifactItem(parsed.embedded), + }; +} + +export function renderPacSourceArtifactRuntimeObserverShell(inputValue: Record): string { + const input = JSON.stringify(inputValue); + const observer = readFileSync(sourceArtifactRuntimeObserverFile, "utf8").trimEnd(); + if (observer.includes("NODE_PAC_SOURCE_ARTIFACT_STATUS") || input.includes("JSON_PAC_SOURCE_ARTIFACT_INPUT")) throw new Error("PaC source artifact observer input contains a reserved heredoc delimiter"); + return `set -eu +observer_input=$(mktemp) +trap 'rm -f "$observer_input"' EXIT HUP INT TERM +cat >"$observer_input" <<'JSON_PAC_SOURCE_ARTIFACT_INPUT' +${input} +JSON_PAC_SOURCE_ARTIFACT_INPUT +PAC_SOURCE_ARTIFACT_INPUT_FILE="$observer_input" node --input-type=module <<'NODE_PAC_SOURCE_ARTIFACT_STATUS' +${observer} +NODE_PAC_SOURCE_ARTIFACT_STATUS +`; +} + +function runtimeSourceArtifactItem(value: unknown): PacSourceArtifactRuntimeObservation["live"] { + if (typeof value !== "object" || value === null || Array.isArray(value)) return unavailableSourceArtifactRuntime("runtime-observer-invalid-item", null).live; + const item = value as Record; + const status = item.status; + if (status !== "aligned" && status !== "drifted" && status !== "missing" && status !== "unavailable") return unavailableSourceArtifactRuntime("runtime-observer-invalid-status", null).live; + const drift = item.firstDrift; + return { + status, + name: typeof item.name === "string" ? item.name : null, + canonicalSha256: typeof item.canonicalSha256 === "string" ? item.canonicalSha256 : null, + firstDrift: typeof drift === "object" && drift !== null && !Array.isArray(drift) + ? { + path: String((drift as Record).path ?? "$"), + expected: String((drift as Record).expected ?? ""), + actual: String((drift as Record).actual ?? ""), + } + : null, + provenanceAligned: item.provenanceAligned === true, + configRef: typeof item.configRef === "string" ? item.configRef : null, + effectiveConfigSha256: typeof item.effectiveConfigSha256 === "string" ? item.effectiveConfigSha256 : null, + sourceCommit: typeof item.sourceCommit === "string" ? item.sourceCommit : null, + reason: typeof item.reason === "string" ? item.reason : null, + candidateCount: typeof item.candidateCount === "number" && Number.isInteger(item.candidateCount) && item.candidateCount >= 0 ? item.candidateCount : 0, + }; +} + +function unavailableSourceArtifactRuntime(reason: string, sourceCommit: string | null): PacSourceArtifactRuntimeObservation { + const item = { status: "unavailable" as const, name: null, canonicalSha256: null, firstDrift: null, provenanceAligned: false, configRef: null, effectiveConfigSha256: null, sourceCommit, reason, candidateCount: 0 }; + return { live: item, embedded: item }; +} + function parseTarget(record: Record, index: number): PacTarget { const path = `targets[${index}]`; return { diff --git a/scripts/src/stable-json.ts b/scripts/src/stable-json.ts new file mode 100644 index 00000000..3217f1b4 --- /dev/null +++ b/scripts/src/stable-json.ts @@ -0,0 +1,12 @@ +import { createHash } from "node:crypto"; + +export function stableJsonValue(value: unknown): unknown { + if (Array.isArray(value)) return value.map(stableJsonValue); + if (typeof value !== "object" || value === null) return value; + const input = value as Record; + return Object.fromEntries(Object.keys(input).sort().map((key) => [key, stableJsonValue(input[key])])); +} + +export function stableJsonSha256(value: unknown): string { + return `sha256:${createHash("sha256").update(JSON.stringify(stableJsonValue(value))).digest("hex")}`; +} From 0fa47e9295e4bb602a11d48df0f2dd451ece1773 Mon Sep 17 00:00:00 2001 From: Codex Date: Sat, 11 Jul 2026 06:59:43 +0200 Subject: [PATCH 2/5] fix: validate Kafka refresh delivery in web probe --- .agents/skills/unidesk-webdev/SKILL.md | 7 +- config/hwlab-node-lanes.yaml | 10 ++ .../R1.1_Task_Report.md | 7 + .../R1.2_Task_Report.md | 6 + .../R1.3_Task_Report.md | 7 + .../web-probe-kafka-refresh-delivery.md | 22 +++ scripts/src/hwlab-node-lanes.ts | 28 +++- ...web-observe-runner-realtime-source.test.ts | 158 +++++++++++++++++- ...node-web-observe-runner-realtime-source.ts | 133 ++++++++++++--- .../web-probe-observe-options.test.ts | 11 ++ 10 files changed, 360 insertions(+), 29 deletions(-) create mode 100644 docs/MDTODO/details/web-probe-kafka-refresh-delivery/R1.1_Task_Report.md create mode 100644 docs/MDTODO/details/web-probe-kafka-refresh-delivery/R1.2_Task_Report.md create mode 100644 docs/MDTODO/details/web-probe-kafka-refresh-delivery/R1.3_Task_Report.md create mode 100644 docs/MDTODO/web-probe-kafka-refresh-delivery.md diff --git a/.agents/skills/unidesk-webdev/SKILL.md b/.agents/skills/unidesk-webdev/SKILL.md index 50225cde..98e38116 100644 --- a/.agents/skills/unidesk-webdev/SKILL.md +++ b/.agents/skills/unidesk-webdev/SKILL.md @@ -39,7 +39,12 @@ description: UniDesk Web 开发与浏览器验证技能。用户处理 UniDesk/H - Project Management/MDTODO closeout 必须区分 `control` 页和被动 `observer` 页：显式 `observe command` 的 command result、control URL 和对应截图是用户动作证据；observer 周期刷新或 stop 后根路由空态只能作为对照信号，不能覆盖 command result。涉及报告的验收要同时记录 `reportPreviewVisible`、`reportFullscreenVisible`、报告 deep link 和截图 SHA。 - MDTODO → Workbench launch 验收必须引用 `launchWorkbenchFromMdtodo` 的 command result，确认文件/任务选择、Workbench session、launch/chat 状态和 OTel trace header；不要只凭页面最终停在 Workbench URL 判断通过。 - Workbench provider profile 验收使用 `observe command --type sendPrompt --provider ` 时，command result 必须显示 `requestedProvider`、`providerSelection.selected`、`/v1/agent/chat` 202、traceId 和最终 `turn-summary`；只有 prompt 成功不够，必须确认浏览器控件实际选中目标 profile。 -- Workbench 纯实时 fanout 的重复验收使用 `observe command --type validateRealtimeFanout --profile --provider --text --second-text `；它在 observer 后台维持两个独立 browser context 和三条 debug SSE，命令提交后按 `status --command-id ` 轮询，不得回退为长连接 `web-probe script`。 +- Workbench Kafka 实时 fanout 的重复验收： + - 使用 `observe command --type validateRealtimeFanout --profile --provider --text --second-text `； + - connected、refresh/reconnect 和 capability 期望来自 owning YAML，可独立组合，禁止在命令中硬编码 live-only 或 refresh mode； + - refresh 启用时必须证明 retention 到 live handoff、正式 `user` event、`hwlab.event.v1` 同 envelope，以及主页面单一 EventSource 和同一 UI reducer 终态； + - observer 后台维持两个独立 product subscriber 和三条 debug SSE； + - 命令提交后按 `status --command-id ` 轮询，不得回退为长连接 `web-probe script`。 - Workbench 隔离 Kafka debug 重放使用 `observe command --type validateWorkbenchKafkaDebugReplay`： - 复用 observer 当前 session 和 semantic origin； - 按 owning YAML 校验 topic、独立 group prefix、请求静默窗口和产品/调试 SSE 路径； diff --git a/config/hwlab-node-lanes.yaml b/config/hwlab-node-lanes.yaml index 2ea379fc..737053f1 100644 --- a/config/hwlab-node-lanes.yaml +++ b/config/hwlab-node-lanes.yaml @@ -579,6 +579,14 @@ templates: navigationMaxAttempts: 4 navigationRetryDelayMs: 1000 reconnectQuietMs: 3000 + expectedProductSse: + deliverySemantics: kafka-retention-then-live + liveOnly: false + replay: true + replaySupported: true + lossPossible: false + refreshHandoff: true + replayPriorTraceOnReconnect: true forbiddenRequestPaths: - /v1/workbench/sync - /v1/agent/chat/result/ @@ -589,11 +597,13 @@ templates: - /finalizer requiredEventTypes: agentrun: + - user_message - tool_call - command_output - assistant_message - terminal_status hwlab: + - user - tool - status - assistant diff --git a/docs/MDTODO/details/web-probe-kafka-refresh-delivery/R1.1_Task_Report.md b/docs/MDTODO/details/web-probe-kafka-refresh-delivery/R1.1_Task_Report.md new file mode 100644 index 00000000..53973d09 --- /dev/null +++ b/docs/MDTODO/details/web-probe-kafka-refresh-delivery/R1.1_Task_Report.md @@ -0,0 +1,7 @@ +# R1.1 任务报告 + +- 来源：https://github.com/pikasTech/unidesk/issues/1751 +- 已冻结 YAML-owned 可组合合同：`directPublish`、`liveKafkaSse`、`kafkaRefreshReplay` 独立校验。 +- refresh profile 要求 `workbench.connected` 披露 `kafka-retention-then-live`，并验证 barrier、retention 计数和 live handoff。 +- 验收禁止 HTTP history、cursor、projector、snapshot、polling、gap-fill 和 fallback。 +- 正式用户输入必须以稳定 `userMessageId` 一对一贯穿 AgentRun、HWLAB Kafka 和 product SSE。 diff --git a/docs/MDTODO/details/web-probe-kafka-refresh-delivery/R1.2_Task_Report.md b/docs/MDTODO/details/web-probe-kafka-refresh-delivery/R1.2_Task_Report.md new file mode 100644 index 00000000..15f66793 --- /dev/null +++ b/docs/MDTODO/details/web-probe-kafka-refresh-delivery/R1.2_Task_Report.md @@ -0,0 +1,6 @@ +# R1.2 任务报告 + +- owning YAML 新增 `expectedProductSse`，connected/reconnect 期望不再由 runner 硬编码 live-only。 +- typed `validateRealtimeFanout` 同时支持 YAML 显式声明的 live-only 与 refresh+live 合同。 +- reconnect 时验证 retained trace 原 envelope 重放，随后第二轮增量继续从 live fanout 到达。 +- report 输出 bounded connected、handoff、正式 user event、request ledger 和同一 Workbench UI reducer 证据。 diff --git a/docs/MDTODO/details/web-probe-kafka-refresh-delivery/R1.3_Task_Report.md b/docs/MDTODO/details/web-probe-kafka-refresh-delivery/R1.3_Task_Report.md new file mode 100644 index 00000000..849688d8 --- /dev/null +++ b/docs/MDTODO/details/web-probe-kafka-refresh-delivery/R1.3_Task_Report.md @@ -0,0 +1,7 @@ +# R1.3 任务报告 + +- `bun test` 定向执行 17 个测试，全部通过。 +- 覆盖 YAML refresh/live connected 合同、legacy live-only profile、handoff 计数一致性和正式 user event 三段 lineage。 +- 覆盖 semantic origin 继承、URL 选择歧义拒绝和既有 collect 行为。 +- `bun scripts/cli.ts check --syntax-only` 通过 11 项检查。 +- `git diff --check` 通过。 diff --git a/docs/MDTODO/web-probe-kafka-refresh-delivery.md b/docs/MDTODO/web-probe-kafka-refresh-delivery.md new file mode 100644 index 00000000..9b57fc3a --- /dev/null +++ b/docs/MDTODO/web-probe-kafka-refresh-delivery.md @@ -0,0 +1,22 @@ +# Web Probe Kafka 刷新重放验收 + +来源：https://github.com/pikasTech/unidesk/issues/1751 + +边界：只增强 YAML-first typed Web Probe；不部署、不合并，不引入临时脚本、URL/IP、HTTP history、projector 或 fallback。 + + +## R1 [in_progress] + +完成 [UniDesk #1751](https://github.com/pikasTech/unidesk/issues/1751) 的 YAML-first Kafka refresh→live typed Web Probe 验收，完成任务后将详细报告写入[任务报告](./details/web-probe-kafka-refresh-delivery/R1_Task_Report.md)。 +### R1.1 [completed] + +冻结 [#1751](https://github.com/pikasTech/unidesk/issues/1751) 可组合 connected、handoff、user event 与同源 reducer 合同，完成任务后将详细报告写入[任务报告](./details/web-probe-kafka-refresh-delivery/R1.1_Task_Report.md)。 +### R1.2 [completed] + +实现 YAML profile 与 typed realtime validator，不保留 hardcoded live-only，完成任务后将详细报告写入[任务报告](./details/web-probe-kafka-refresh-delivery/R1.2_Task_Report.md)。 +### R1.3 [completed] + +补齐 connected、retention→live、正式 user event、同 EventSource/envelope 的定向测试，完成任务后将详细报告写入[任务报告](./details/web-probe-kafka-refresh-delivery/R1.3_Task_Report.md)。 +### R1.4 [in_progress] + +完成帮助/skill、报告、提交、非 draft PR 与 preflight，保持不部署不合并，完成任务后将详细报告写入[任务报告](./details/web-probe-kafka-refresh-delivery/R1.4_Task_Report.md)。 diff --git a/scripts/src/hwlab-node-lanes.ts b/scripts/src/hwlab-node-lanes.ts index 27edb30b..685a0516 100644 --- a/scripts/src/hwlab-node-lanes.ts +++ b/scripts/src/hwlab-node-lanes.ts @@ -192,6 +192,15 @@ export interface HwlabRuntimeWebProbeRealtimeFanoutProfileSpec { readonly navigationMaxAttempts: number; readonly navigationRetryDelayMs: number; readonly reconnectQuietMs: number; + readonly expectedProductSse: Readonly<{ + deliverySemantics: string; + liveOnly: boolean; + replay: boolean; + replaySupported: boolean; + lossPossible: boolean; + refreshHandoff: boolean; + replayPriorTraceOnReconnect: boolean; + }>; readonly forbiddenRequestPaths: readonly string[]; readonly requiredEventTypes: Readonly<{ agentrun: readonly string[]; @@ -1435,7 +1444,23 @@ function webProbeRealtimeFanoutProfileConfig(value: unknown, path: string): Hwla throw new Error(`${path}.debugStreams must contain stdio, agentrun, and hwlab exactly once`); } const fromBeginning = booleanField(raw, "fromBeginning", path); - if (fromBeginning !== false) throw new Error(`${path}.fromBeginning must be false for live-only validation`); + if (fromBeginning !== false) throw new Error(`${path}.fromBeginning must be false for turn-scoped debug streams`); + const expectedProductSseRaw = asRecord(raw.expectedProductSse, `${path}.expectedProductSse`); + const expectedProductSse = { + deliverySemantics: stringField(expectedProductSseRaw, "deliverySemantics", `${path}.expectedProductSse`), + liveOnly: booleanField(expectedProductSseRaw, "liveOnly", `${path}.expectedProductSse`), + replay: booleanField(expectedProductSseRaw, "replay", `${path}.expectedProductSse`), + replaySupported: booleanField(expectedProductSseRaw, "replaySupported", `${path}.expectedProductSse`), + lossPossible: booleanField(expectedProductSseRaw, "lossPossible", `${path}.expectedProductSse`), + refreshHandoff: booleanField(expectedProductSseRaw, "refreshHandoff", `${path}.expectedProductSse`), + replayPriorTraceOnReconnect: booleanField(expectedProductSseRaw, "replayPriorTraceOnReconnect", `${path}.expectedProductSse`), + }; + if (expectedProductSse.refreshHandoff && (!expectedProductSse.replay || !expectedProductSse.replaySupported || expectedProductSse.liveOnly)) { + throw new Error(`${path}.expectedProductSse refresh handoff requires replay=true, replaySupported=true, and liveOnly=false`); + } + if (expectedProductSse.replayPriorTraceOnReconnect && !expectedProductSse.replay) { + throw new Error(`${path}.expectedProductSse replayPriorTraceOnReconnect requires replay=true`); + } const requiredEventTypes = asRecord(raw.requiredEventTypes, `${path}.requiredEventTypes`); const forbiddenRequestPaths = nonEmptyStringArrayField(raw, "forbiddenRequestPaths", path); if (new Set(forbiddenRequestPaths).size !== forbiddenRequestPaths.length) throw new Error(`${path}.forbiddenRequestPaths must not contain duplicates`); @@ -1466,6 +1491,7 @@ function webProbeRealtimeFanoutProfileConfig(value: unknown, path: string): Hwla navigationMaxAttempts: boundedIntegerField(raw, "navigationMaxAttempts", path, 1, 10), navigationRetryDelayMs: boundedIntegerField(raw, "navigationRetryDelayMs", path, 100, 10_000), reconnectQuietMs: boundedIntegerField(raw, "reconnectQuietMs", path, 250, 30_000), + expectedProductSse, forbiddenRequestPaths, requiredEventTypes: { agentrun: agentrunEventTypes, diff --git a/scripts/src/hwlab-node-web-observe-runner-realtime-source.test.ts b/scripts/src/hwlab-node-web-observe-runner-realtime-source.test.ts index a2c79b70..88203959 100644 --- a/scripts/src/hwlab-node-web-observe-runner-realtime-source.test.ts +++ b/scripts/src/hwlab-node-web-observe-runner-realtime-source.test.ts @@ -25,12 +25,21 @@ test("realtime fanout keeps the YAML terminal timeout when durationMs is absent" terminalTimeoutMs: 480_000, terminalQuietMs: 2_000, reconnectQuietMs: 3_000, + expectedProductSse: { + deliverySemantics: "kafka-retention-then-live", + liveOnly: false, + replay: true, + replaySupported: true, + lossPossible: false, + refreshHandoff: true, + replayPriorTraceOnReconnect: true, + }, forbiddenRequestPaths: [], requiredEventTypes: { agentrun: ["terminal_status"], hwlab: ["terminal"] }, expectedKafka: { topics: { stdio: "codex-stdio.raw.v1", agentrun: "agentrun.event.v1", hwlab: "hwlab.event.v1" }, groups: { directPublish: "direct", transactionalProjector: "projector", liveSse: "live" }, - capabilities: { directPublish: true, liveKafkaSse: true }, + capabilities: { directPublish: true, liveKafkaSse: true, kafkaRefreshReplay: true }, }, }; @@ -83,3 +92,150 @@ test("realtime fanout validates stdio against the scoped AgentRun session lineag assert.equal(expectedStreamSessionId("agentrun", hwlabSessionId), hwlabSessionId); assert.equal(expectedStreamSessionId("hwlab", hwlabSessionId), hwlabSessionId); }); + +test("realtime fanout validates YAML-owned refresh and legacy live-only connected contracts", () => { + const source = nodeWebObserveRunnerRealtimeSource(); + const build = new Function(`${source}\nreturn realtimeAssertConnectedContract;`) as () => ( + connected: Record, + sessionId: string, + label: string, + profile: Record, + options?: Record, + ) => void; + const assertConnected = build(); + const sessionId = "ses_refresh_contract"; + const expectedKafka = { + topics: { hwlab: "hwlab.event.v1" }, + capabilities: { directPublish: true, liveKafkaSse: true, kafkaRefreshReplay: true }, + }; + const refreshProfile = { + expectedKafka, + expectedProductSse: { + deliverySemantics: "kafka-retention-then-live", + liveOnly: false, + replay: true, + replaySupported: true, + lossPossible: false, + refreshHandoff: true, + replayPriorTraceOnReconnect: true, + }, + }; + const refreshConnected = { + realtimeSource: "hwlab.event.v1", + deliverySemantics: "kafka-retention-then-live", + liveOnly: false, + replay: true, + replaySupported: true, + lossPossible: false, + filters: { sessionId, traceId: null }, + capabilities: expectedKafka.capabilities, + refreshReplay: { + phase: "live", + topic: "hwlab.event.v1", + topicPartitions: [0], + barrier: [{ partition: 0, endOffset: "18" }], + counts: { matched: 8, replayed: 8, buffered: 1, bufferedDelivered: 1, liveDelivered: 0, deduplicated: 0 }, + }, + }; + + assert.doesNotThrow(() => assertConnected(refreshConnected, sessionId, "refresh", refreshProfile, { requireRetainedEvents: true })); + assert.throws( + () => assertConnected({ ...refreshConnected, deliverySemantics: "live-only" }, sessionId, "refresh", refreshProfile), + /differs from the owning YAML/u, + ); + assert.throws( + () => assertConnected({ ...refreshConnected, refreshReplay: { ...refreshConnected.refreshReplay, counts: { ...refreshConnected.refreshReplay.counts, replayed: 9 } } }, sessionId, "refresh", refreshProfile), + /replayed count exceeds matched count/u, + ); + + const liveProfile = { + expectedKafka: { topics: { hwlab: "hwlab.event.v1" }, capabilities: { directPublish: true, liveKafkaSse: true, kafkaRefreshReplay: false } }, + expectedProductSse: { + deliverySemantics: "live-only", + liveOnly: true, + replay: false, + replaySupported: false, + lossPossible: true, + refreshHandoff: false, + replayPriorTraceOnReconnect: false, + }, + }; + assert.doesNotThrow(() => assertConnected({ + realtimeSource: "hwlab.event.v1", + deliverySemantics: "live-only", + liveOnly: true, + replay: false, + replaySupported: false, + lossPossible: true, + filters: { sessionId, traceId: null }, + capabilities: liveProfile.expectedKafka.capabilities, + }, sessionId, "live", liveProfile)); +}); + +test("realtime fanout proves one formal user event through AgentRun, HWLAB Kafka, and product SSE", () => { + const source = nodeWebObserveRunnerRealtimeSource(); + const build = new Function(`${source}\nreturn realtimeValidateTurnEvidence;`) as () => (input: Record) => void; + const validate = build(); + const traceId = "trc_user_passthrough"; + const sessionId = "ses_user_passthrough"; + const runId = "run_user_passthrough"; + const commandId = "cmd_user_passthrough"; + const userMessageId = "msg_user_passthrough"; + const base = { traceId, runId, commandId, terminal: false }; + const userHwlab = { + ...base, + partition: 0, + offset: "0", + hwlabSessionId: sessionId, + eventId: "hwlab:evt_user", + sourceEventId: "evt_user", + sourceSeq: 1, + eventType: "user", + userMessageId, + envelopeFingerprint: "fingerprint:user", + }; + const terminalHwlab = { + ...base, + partition: 0, + offset: "1", + hwlabSessionId: sessionId, + eventId: "hwlab:evt_terminal", + sourceEventId: "evt_terminal", + sourceSeq: 2, + eventType: "terminal", + userMessageId: null, + terminal: true, + envelopeFingerprint: "fingerprint:terminal", + }; + const debug = { + connected: { + stdio: { consumerReady: true, topic: "codex-stdio.raw.v1" }, + agentrun: { consumerReady: true, topic: "agentrun.event.v1" }, + hwlab: { consumerReady: true, topic: "hwlab.event.v1" }, + }, + openCount: { stdio: 1, agentrun: 1, hwlab: 1 }, + connectedEventCount: { stdio: 1, agentrun: 1, hwlab: 1 }, + errors: { stdio: [], agentrun: [], hwlab: [] }, + records: { + stdio: [{ ...base, partition: 0, offset: "0", frameSeq: 1, hwlabSessionId: "ses_agentrun_user_passthrough" }], + agentrun: [ + { ...base, partition: 0, offset: "0", hwlabSessionId: sessionId, eventId: "evt_user", eventType: "user_message", userMessageId }, + { ...base, partition: 0, offset: "1", hwlabSessionId: sessionId, eventId: "evt_terminal", eventType: "terminal_status", terminal: true }, + ], + hwlab: [userHwlab, terminalHwlab], + }, + }; + const product = { openCount: 1, connectedEventCount: 1, errors: [], records: [{ ...userHwlab }, { ...terminalHwlab }] }; + const profile = { + debugStreams: ["stdio", "agentrun", "hwlab"], + requiredEventTypes: { agentrun: ["user_message", "terminal_status"], hwlab: ["user", "terminal"] }, + expectedKafka: { topics: { stdio: "codex-stdio.raw.v1", agentrun: "agentrun.event.v1", hwlab: "hwlab.event.v1" } }, + }; + + assert.doesNotThrow(() => validate({ turn: 1, traceId, sessionId, runId, commandId, debug, terminalSnapshots: [product], preTerminalSnapshots: [], profile })); + const missingUserProduct = { ...product, records: [terminalHwlab] }; + assert.throws( + () => validate({ turn: 1, traceId, sessionId, runId, commandId, debug, terminalSnapshots: [missingUserProduct], preTerminalSnapshots: [], profile }), + /product subscriber lacks pre-terminal or terminal event|product SSE lacks the formal user event/u, + ); +}); diff --git a/scripts/src/hwlab-node-web-observe-runner-realtime-source.ts b/scripts/src/hwlab-node-web-observe-runner-realtime-source.ts index 56f31e0e..ab6265aa 100644 --- a/scripts/src/hwlab-node-web-observe-runner-realtime-source.ts +++ b/scripts/src/hwlab-node-web-observe-runner-realtime-source.ts @@ -75,14 +75,13 @@ async function validateRealtimeFanout(command) { debug = await realtimeNewDebugContext(storageState, effective); const connectedA = await realtimeWaitProductConnected(subscriberA, effective.barrierTimeoutMs); const connectedB = await realtimeWaitProductConnected(subscriberB, effective.barrierTimeoutMs); - realtimeAssertLiveConnected(connectedA, sessionId, "subscriber-a", effective); - realtimeAssertLiveConnected(connectedB, sessionId, "subscriber-b", effective); + realtimeAssertConnectedContract(connectedA, sessionId, "subscriber-a", effective); + realtimeAssertConnectedContract(connectedB, sessionId, "subscriber-b", effective); await appendJsonl(files.control, eventRecord("realtime-fanout-product-ready", { profile: profileId, sessionId, subscriberCount: 2, - liveOnly: true, - replay: false, + expectedProductSse: effective.expectedProductSse, valuesRedacted: true, })); @@ -132,12 +131,14 @@ async function validateRealtimeFanout(command) { subscriberB2 = await realtimeNewSubscriber(storageState, sessionId, "subscriber-b-reconnected", effective); const bReconnected = await realtimeWaitProductConnected(subscriberB2, effective.barrierTimeoutMs); - realtimeAssertLiveConnected(bReconnected, sessionId, "subscriber-b-reconnected", effective); + realtimeAssertConnectedContract(bReconnected, sessionId, "subscriber-b-reconnected", effective, { requireRetainedEvents: effective.expectedProductSse.replayPriorTraceOnReconnect }); await sleep(effective.reconnectQuietMs); const reconnectBeforeSecond = await realtimeProductSnapshot(subscriberB2); realtimeAssertStableProductTransport(reconnectBeforeSecond, "subscriber-b-reconnected-before-second-turn"); - if (reconnectBeforeSecond.records.some((item) => item.traceId === first.traceId)) { - throw new Error("reconnected subscriber replayed first-turn events"); + if (effective.expectedProductSse.replayPriorTraceOnReconnect) { + realtimeAssertReplayedTrace(firstDebug, reconnectBeforeSecond, first.traceId, "subscriber-b-reconnected-before-second-turn"); + } else if (reconnectBeforeSecond.records.some((item) => item.traceId === first.traceId)) { + throw new Error("reconnected subscriber replayed first-turn events contrary to the owning YAML"); } const second = await realtimeRunTurn({ @@ -159,8 +160,10 @@ async function validateRealtimeFanout(command) { const [secondProductA, secondProductB] = secondStable.products; realtimeAssertStableProductTransport(secondProductA, "subscriber-a-second-terminal"); realtimeAssertStableProductTransport(secondProductB, "subscriber-b-reconnected-second-terminal"); - if (secondProductB.records.some((item) => item.traceId === first.traceId)) { - throw new Error("reconnected subscriber received first-turn events after the second submit"); + if (effective.expectedProductSse.replayPriorTraceOnReconnect) { + realtimeAssertReplayedTrace(firstDebug, secondProductB, first.traceId, "subscriber-b-reconnected-after-second-turn"); + } else if (secondProductB.records.some((item) => item.traceId === first.traceId)) { + throw new Error("reconnected subscriber received first-turn events contrary to the owning YAML"); } realtimeValidateTurnEvidence({ turn: 2, @@ -179,6 +182,9 @@ async function validateRealtimeFanout(command) { const productB2 = await realtimeProductSnapshot(subscriberB2); realtimeAssertStableProductTransport(productA, "subscriber-a-final"); realtimeAssertStableProductTransport(productB2, "subscriber-b-reconnected-final"); + if (effective.expectedProductSse.replayPriorTraceOnReconnect) { + realtimeAssertReplayedTrace(firstDebug, productB2, first.traceId, "subscriber-b-reconnected-final"); + } const mainStreams = network.filter((item) => item.path === effective.productEventsPath && item.sessionId === sessionId); const forbidden = network.filter((item) => realtimeForbiddenRequest(item, effective.forbiddenRequestPaths)); if (mainStreams.length !== 1) throw new Error("main Workbench session SSE connection count is " + mainStreams.length + ", expected 1"); @@ -187,14 +193,14 @@ async function validateRealtimeFanout(command) { if (forbidden.length > 0) throw new Error("forbidden recovery or polling requests observed: " + forbidden.map((item) => item.path).join(",")); const summaryScreenshot = await realtimeCaptureSubscriberScreenshot(debug.page, reportDir, "realtime-fanout-summary.png", { - title: "Pure Kafka live fanout validated", + title: "Pure Kafka retention and live fanout validated", profile: profileId, sessionId, traceIds: [first.traceId, second.traceId], productSubscriberCount: 2, debugStreams: effective.debugStreams, forbiddenRequestCount: forbidden.length, - replayedEventCount: 0, + replayedEventCount: bReconnected?.refreshReplay?.counts?.replayed || 0, valuesRedacted: true, }); const subscriberAScreenshot = await realtimeCaptureSubscriberScreenshot(subscriberA.page, reportDir, "subscriber-a-final.png", { @@ -206,9 +212,9 @@ async function validateRealtimeFanout(command) { valuesRedacted: true, }); const subscriberB2Screenshot = await realtimeCaptureSubscriberScreenshot(subscriberB2.page, reportDir, "subscriber-b-reconnected-final.png", { - title: "Subscriber B reconnected live-only", + title: "Subscriber B reconnected through Kafka retention then live", sessionId, - traceIds: [second.traceId], + traceIds: effective.expectedProductSse.replayPriorTraceOnReconnect ? [first.traceId, second.traceId] : [second.traceId], eventCount: productB2.records.length, firstTraceReplayCount: productB2.records.filter((item) => item.traceId === first.traceId).length, valuesRedacted: true, @@ -229,11 +235,20 @@ async function validateRealtimeFanout(command) { lastEventIdCount: 0, subscriberAEventCount: productA.records.length, subscriberBReconnectEventCount: productB2.records.length, - subscriberBFirstTraceReplayCount: 0, + subscriberBFirstTraceReplayCount: productB2.records.filter((item) => item.traceId === first.traceId).length, valuesRedacted: true, }, - liveContract: { + productUiReducer: { + mainSessionEventSourceCount: mainStreams.length, + sessionScoped: mainStreams[0]?.traceId === null, + firstTraceTerminalRendered: true, + secondTraceTerminalRendered: true, + evidenceSource: "same-workbench-page-eventsource-and-ui-terminal", + valuesRedacted: true, + }, + realtimeContract: { expectedKafka: effective.expectedKafka, + expectedProductSse: effective.expectedProductSse, subscriberA: realtimeConnectedSummary(connectedA), subscriberBBeforeDisconnect: realtimeConnectedSummary(connectedB), subscriberBAfterReconnect: realtimeConnectedSummary(bReconnected), @@ -268,7 +283,7 @@ async function validateRealtimeFanout(command) { commandIds: turns.map((item) => item.commandId), warmRunnerReused: report.warmRunnerReused, forbiddenRequestCount: 0, - replayedEventCount: 0, + replayedEventCount: bReconnected?.refreshReplay?.counts?.replayed || 0, reportPath: reportArtifact.path, reportSha256: reportArtifact.sha256, screenshots: report.screenshots, @@ -319,11 +334,19 @@ function realtimeFanoutEffectiveProfile(profile, durationMs) { throw new Error("realtime fanout profile debugStreams must be stdio, agentrun, and hwlab"); } if (profile.fromBeginning !== false) throw new Error("realtime fanout profile fromBeginning must be false"); + const expectedProductSse = profile.expectedProductSse && typeof profile.expectedProductSse === "object" ? profile.expectedProductSse : null; + if (!expectedProductSse || typeof expectedProductSse.deliverySemantics !== "string") throw new Error("realtime fanout profile lacks YAML-owned product SSE expectations"); const expectedKafka = profile.expectedKafka && typeof profile.expectedKafka === "object" ? profile.expectedKafka : null; if (!expectedKafka?.topics || !expectedKafka?.groups || !expectedKafka?.capabilities) throw new Error("realtime fanout profile lacks YAML-derived Kafka expectations"); if (expectedKafka.capabilities.directPublish !== true || expectedKafka.capabilities.liveKafkaSse !== true) { throw new Error("realtime fanout profile requires YAML-enabled directPublish and liveKafkaSse capabilities"); } + if (expectedProductSse.refreshHandoff !== (expectedKafka.capabilities.kafkaRefreshReplay === true)) { + throw new Error("realtime fanout product SSE refreshHandoff differs from the YAML Kafka capability"); + } + if (expectedProductSse.replayPriorTraceOnReconnect === true && expectedProductSse.replay !== true) { + throw new Error("realtime fanout replayPriorTraceOnReconnect requires replay=true"); + } const hasDurationOverride = durationMs !== null && durationMs !== undefined && durationMs !== "" && Number.isFinite(Number(durationMs)); const terminalTimeoutMs = hasDurationOverride ? Math.min(Number(profile.terminalTimeoutMs), Math.max(1000, Number(durationMs))) @@ -331,14 +354,15 @@ function realtimeFanoutEffectiveProfile(profile, durationMs) { return { ...profile, debugStreams, - barrierTimeoutMs: boundedInteger(profile.barrierTimeoutMs, 45000, 1000, 120000), - terminalTimeoutMs: boundedInteger(terminalTimeoutMs, 480000, 1000, 600000), - terminalQuietMs: boundedInteger(profile.terminalQuietMs, 2000, 250, 10000), - navigationMaxAttempts: boundedInteger(profile.navigationMaxAttempts, 4, 1, 10), - navigationRetryDelayMs: boundedInteger(profile.navigationRetryDelayMs, 1000, 100, 10000), - reconnectQuietMs: boundedInteger(profile.reconnectQuietMs, 3000, 250, 30000), + barrierTimeoutMs: Number(profile.barrierTimeoutMs), + terminalTimeoutMs: Number(terminalTimeoutMs), + terminalQuietMs: Number(profile.terminalQuietMs), + navigationMaxAttempts: Number(profile.navigationMaxAttempts), + navigationRetryDelayMs: Number(profile.navigationRetryDelayMs), + reconnectQuietMs: Number(profile.reconnectQuietMs), forbiddenRequestPaths: Array.isArray(profile.forbiddenRequestPaths) ? profile.forbiddenRequestPaths.map(String) : [], requiredEventTypes: profile.requiredEventTypes && typeof profile.requiredEventTypes === "object" ? profile.requiredEventTypes : {}, + expectedProductSse: sanitize(expectedProductSse), expectedKafka: sanitize(expectedKafka), }; } @@ -383,6 +407,7 @@ async function realtimeNewSubscriber(storageState, sessionId, label, profile) { hwlabSessionId: firstText(value.hwlabSessionId, value.sessionId, nested.hwlabSessionId, nested.sessionId, context.sessionId), runId: firstText(value.runId, nested.runId, context.runId), commandId: firstText(value.commandId, nested.commandId, context.commandId), + userMessageId: firstText(value.userMessageId, nested.userMessageId, nested.messageId), terminal: value.terminal === true || nested.terminal === true || eventType === "terminal" || eventType === "terminal_status", status: firstText(nested.status, value.status), envelopeFingerprint: fingerprint(raw), @@ -480,7 +505,7 @@ async function realtimeRunTurn(input) { } for (const subscriber of input.subscribers) { const connected = await realtimeWaitProductConnected(subscriber, input.profile.barrierTimeoutMs); - realtimeAssertLiveConnected(connected, input.sessionId, subscriber.label, input.profile); + realtimeAssertConnectedContract(connected, input.sessionId, subscriber.label, input.profile); } await appendJsonl(files.control, eventRecord("realtime-fanout-ready-barrier", { turn, @@ -587,6 +612,7 @@ async function realtimeOpenDebugStreams(debug, key, traceId, profile) { hwlabSessionId: firstText(value.hwlabSessionId, value.sessionId, run.hwlabSessionId, run.sessionId, context.sessionId, payload.hwlabSessionId, payload.sessionId, nested.hwlabSessionId, nested.sessionId), runId: firstText(value.runId, run.runId, context.runId, nested.runId, payload.runId), commandId: firstText(value.commandId, command.commandId, context.commandId, payload.commandId, nested.commandId), + userMessageId: firstText(value.userMessageId, payload.userMessageId, payload.messageId, nested.userMessageId, nested.messageId), runnerId: firstText(value.runnerId, run.runnerId, command.runnerId, context.runnerId, payload.runnerId, nested.runnerId), attemptId: firstText(value.attemptId, run.attemptId, command.attemptId, context.attemptId, payload.attemptId, nested.attemptId), terminal: value.terminal === true || nested.terminal === true || eventType === "terminal" || eventType === "terminal_status", @@ -730,15 +756,47 @@ async function realtimeCloseDebugStreams(debug, key) { }, key); } -function realtimeAssertLiveConnected(connected, sessionId, label, profile) { +function realtimeAssertConnectedContract(connected, sessionId, label, profile, options = {}) { if (connected?.realtimeSource !== profile.expectedKafka.topics.hwlab) throw new Error(label + " realtime source mismatch"); - if (connected?.deliverySemantics !== "live-only" || connected?.liveOnly !== true) throw new Error(label + " is not live-only"); - if (connected?.replay !== false || connected?.replaySupported !== false || connected?.lossPossible !== true) throw new Error(label + " replay/loss disclosure mismatch"); + const expected = profile.expectedProductSse; + for (const field of ["deliverySemantics", "liveOnly", "replay", "replaySupported", "lossPossible"]) { + if (connected?.[field] !== expected[field]) throw new Error(label + " product SSE " + field + " differs from the owning YAML"); + } if (connected?.filters?.sessionId !== sessionId) throw new Error(label + " session filter mismatch"); + if (connected?.filters?.traceId !== null) throw new Error(label + " unexpectedly used a trace-scoped product SSE filter"); const capabilities = connected?.capabilities || {}; for (const [name, expected] of Object.entries(profile.expectedKafka.capabilities)) { if (capabilities[name] !== expected) throw new Error(label + " capability " + name + " differs from the owning YAML"); } + if (expected.refreshHandoff === true) realtimeAssertRefreshHandoff(connected, label, profile, options.requireRetainedEvents === true); + else if (connected?.refreshReplay !== undefined && connected?.refreshReplay !== null) throw new Error(label + " disclosed an unexpected refresh handoff"); +} + +function realtimeAssertRefreshHandoff(connected, label, profile, requireRetainedEvents) { + const handoff = connected?.refreshReplay; + if (!handoff || handoff.phase !== "live") throw new Error(label + " refresh handoff did not reach live phase"); + if (handoff.topic !== profile.expectedKafka.topics.hwlab) throw new Error(label + " refresh handoff topic mismatch"); + if (!Array.isArray(handoff.barrier) || handoff.barrier.length === 0) throw new Error(label + " refresh handoff barrier is missing"); + if (handoff.barrier.some((entry) => !Number.isInteger(entry?.partition) || typeof entry?.endOffset !== "string" || !entry.endOffset)) { + throw new Error(label + " refresh handoff barrier identity is invalid"); + } + if (!Array.isArray(handoff.topicPartitions) || handoff.topicPartitions.some((partition) => !Number.isInteger(partition))) { + throw new Error(label + " refresh handoff topicPartitions are invalid"); + } + const counts = handoff.counts; + for (const field of ["matched", "replayed", "buffered", "bufferedDelivered", "liveDelivered", "deduplicated"]) { + if (!Number.isInteger(counts?.[field]) || counts[field] < 0) throw new Error(label + " refresh handoff count " + field + " is invalid"); + } + if (counts.replayed > counts.matched) throw new Error(label + " refresh handoff replayed count exceeds matched count"); + if (counts.bufferedDelivered > counts.buffered) throw new Error(label + " refresh handoff delivered more buffered events than observed"); + if (requireRetainedEvents && counts.replayed < 1) throw new Error(label + " refresh handoff replayed no retained events"); +} + +function realtimeAssertReplayedTrace(debug, product, traceId, label) { + const kafkaRows = (debug.records?.hwlab || []).filter((item) => item.traceId === traceId); + const productRows = (product.records || []).filter((item) => item.traceId === traceId); + if (!productRows.some((item) => item.terminal === true)) throw new Error(label + " lacks the retained terminal event"); + realtimeAssertEnvelopePassthrough(kafkaRows, productRows, true, label + " retained trace"); } function realtimeConnectedSummary(connected) { @@ -751,6 +809,7 @@ function realtimeConnectedSummary(connected) { lossPossible: connected?.lossPossible === true, filters: { sessionId: connected?.filters?.sessionId || null, traceId: connected?.filters?.traceId || null }, capabilities: sanitize(connected?.capabilities || {}), + refreshReplay: connected?.refreshReplay ? sanitize(connected.refreshReplay) : null, valuesRedacted: true, }; } @@ -802,10 +861,19 @@ function realtimeValidateTurnEvidence(input) { const commandIds = new Set(identityRows.map((item) => item.commandId).filter(Boolean)); if (!input.commandId || commandIds.size !== 1 || !commandIds.has(input.commandId)) throw new Error("turn " + input.turn + " commandId missing or mismatched across command-bound events"); const hwlabRows = input.debug.records.hwlab.filter((item) => item.traceId === input.traceId); + const agentrunUserRows = input.debug.records.agentrun.filter((item) => item.traceId === input.traceId && item.eventType === "user_message"); + const hwlabUserRows = hwlabRows.filter((item) => item.eventType === "user"); + if (agentrunUserRows.length !== 1 || hwlabUserRows.length !== 1) throw new Error("turn " + input.turn + " formal user event is not one-to-one across AgentRun and HWLAB"); + const userMessageIds = new Set([...agentrunUserRows, ...hwlabUserRows].map((item) => item.userMessageId).filter(Boolean)); + if (userMessageIds.size !== 1) throw new Error("turn " + input.turn + " formal user event lacks one stable userMessageId"); for (const snapshot of input.terminalSnapshots) { const rows = snapshot.records.filter((item) => item.traceId === input.traceId); if (!rows.some((item) => item.terminal !== true) || !rows.some((item) => item.terminal === true)) throw new Error("turn " + input.turn + " product subscriber lacks pre-terminal or terminal event"); realtimeAssertEnvelopePassthrough(hwlabRows, rows, true, "turn " + input.turn + " terminal subscriber"); + const productUserRows = rows.filter((item) => item.eventType === "user"); + if (productUserRows.length !== 1 || productUserRows[0].userMessageId !== hwlabUserRows[0].userMessageId) { + throw new Error("turn " + input.turn + " product SSE lacks the formal user event"); + } } for (const snapshot of input.preTerminalSnapshots) { const rows = snapshot.records.filter((item) => item.traceId === input.traceId); @@ -833,6 +901,7 @@ function realtimeAssertEnvelopePassthrough(kafkaRows, productRows, requireAllKaf && left.hwlabSessionId === right.hwlabSessionId && left.runId === right.runId && left.commandId === right.commandId + && left.userMessageId === right.userMessageId && left.terminal === right.terminal; for (const productRow of productRows) { if (!kafkaRows.some((kafkaRow) => sameEnvelope(kafkaRow, productRow))) throw new Error(label + " received a product SSE envelope absent from hwlab.event.v1"); @@ -890,6 +959,9 @@ function realtimeTurnSummary(turn, debug, product, terminal) { }; } const productRows = product.records.filter((item) => item.traceId === turn.traceId); + const agentrunUser = debug.records.agentrun.find((item) => item.traceId === turn.traceId && item.eventType === "user_message") || null; + const hwlabUser = debug.records.hwlab.find((item) => item.traceId === turn.traceId && item.eventType === "user") || null; + const productUser = productRows.find((item) => item.eventType === "user") || null; return { turn: turn.turn, traceId: turn.traceId, @@ -901,6 +973,15 @@ function realtimeTurnSummary(turn, debug, product, terminal) { admissionStatus: turn.admissionStatus, terminalStatus: terminal?.status || null, streams, + formalUserEvent: { + userMessageId: hwlabUser?.userMessageId || agentrunUser?.userMessageId || null, + agentrunEventId: agentrunUser?.eventId || null, + hwlabEventId: hwlabUser?.eventId || null, + hwlabSourceEventId: hwlabUser?.sourceEventId || null, + kafkaEnvelopeFingerprint: hwlabUser?.envelopeFingerprint || null, + productEnvelopeFingerprint: productUser?.envelopeFingerprint || null, + valuesRedacted: true, + }, product: { count: productRows.length, openCount: product.openCount, connectedEventCount: product.connectedEventCount, errorCount: product.errors.length, preTerminalSeen: productRows.some((item) => item.terminal !== true), terminalSeen: productRows.some((item) => item.terminal === true), valuesRedacted: true }, valuesRedacted: true, }; diff --git a/scripts/src/hwlab-node/web-probe-observe-options.test.ts b/scripts/src/hwlab-node/web-probe-observe-options.test.ts index 7b475375..a3864091 100644 --- a/scripts/src/hwlab-node/web-probe-observe-options.test.ts +++ b/scripts/src/hwlab-node/web-probe-observe-options.test.ts @@ -32,6 +32,17 @@ test("validateRealtimeFanout parses the YAML profile and two independent prompts assert.equal(options.commandSecondText, "second turn"); assert.deepEqual(spec.webProbe?.realtimeFanoutProfiles?.["pure-kafka-live"]?.debugStreams, ["stdio", "agentrun", "hwlab"]); assert.equal(spec.webProbe?.realtimeFanoutProfiles?.["pure-kafka-live"]?.fromBeginning, false); + assert.deepEqual(spec.webProbe?.realtimeFanoutProfiles?.["pure-kafka-live"]?.expectedProductSse, { + deliverySemantics: "kafka-retention-then-live", + liveOnly: false, + replay: true, + replaySupported: true, + lossPossible: false, + refreshHandoff: true, + replayPriorTraceOnReconnect: true, + }); + assert.ok(spec.webProbe?.realtimeFanoutProfiles?.["pure-kafka-live"]?.requiredEventTypes.agentrun.includes("user_message")); + assert.ok(spec.webProbe?.realtimeFanoutProfiles?.["pure-kafka-live"]?.requiredEventTypes.hwlab.includes("user")); }); test("validateRealtimeFanout refuses body as a substitute for the second prompt", () => { From 74ac2aca4f1a2601c315927472aa828ddc1545d2 Mon Sep 17 00:00:00 2001 From: Codex Date: Sat, 11 Jul 2026 07:01:44 +0200 Subject: [PATCH 3/5] docs: close Kafka refresh web probe task --- .../R1.4_Task_Report.md | 7 +++++++ .../web-probe-kafka-refresh-delivery/R1_Task_Report.md | 10 ++++++++++ docs/MDTODO/web-probe-kafka-refresh-delivery.md | 4 ++-- 3 files changed, 19 insertions(+), 2 deletions(-) create mode 100644 docs/MDTODO/details/web-probe-kafka-refresh-delivery/R1.4_Task_Report.md create mode 100644 docs/MDTODO/details/web-probe-kafka-refresh-delivery/R1_Task_Report.md diff --git a/docs/MDTODO/details/web-probe-kafka-refresh-delivery/R1.4_Task_Report.md b/docs/MDTODO/details/web-probe-kafka-refresh-delivery/R1.4_Task_Report.md new file mode 100644 index 00000000..5bef4a18 --- /dev/null +++ b/docs/MDTODO/details/web-probe-kafka-refresh-delivery/R1.4_Task_Report.md @@ -0,0 +1,7 @@ +# R1.4 任务报告 + +- 更新 `$unidesk-webdev`，明确该 typed command 由 YAML 组合 connected、refresh 和 capability 合同。 +- 提交 `0fa47e9295e4bb602a11d48df0f2dd451ece1773` 已推送到 `fix/web-probe-kafka-refresh-delivery`。 +- 非 draft PR：https://github.com/pikasTech/unidesk/pull/1752 +- 受控 `gh pr preflight` 返回 `MERGEABLE/CLEAN`，无代码侧 blocker 或 pending check。 +- 未部署、未合并。 diff --git a/docs/MDTODO/details/web-probe-kafka-refresh-delivery/R1_Task_Report.md b/docs/MDTODO/details/web-probe-kafka-refresh-delivery/R1_Task_Report.md new file mode 100644 index 00000000..f2101caf --- /dev/null +++ b/docs/MDTODO/details/web-probe-kafka-refresh-delivery/R1_Task_Report.md @@ -0,0 +1,10 @@ +# R1 任务报告 + +- 来源：https://github.com/pikasTech/unidesk/issues/1751 +- 交付 PR：https://github.com/pikasTech/unidesk/pull/1752 +- `validateRealtimeFanout` 已从 owning YAML 读取可组合 product SSE 合同，同时支持显式 live-only 与 refresh+live profile。 +- refresh 验收覆盖 connected、barrier、retention→live handoff、重连 retained trace、第二轮 live 增量和正式 user event 三段 lineage。 +- product evidence 同时证明 `hwlab.event.v1` envelope 原样透传，以及主 Workbench 单一 session-scoped EventSource 进入同一 UI reducer 并渲染终态。 +- 定向测试 17/17、syntax-only 11/11、PR preflight `MERGEABLE/CLEAN`。 +- 未引入 HTTP history、projector、snapshot、polling、gap-fill、fallback、临时 script 或 URL/IP 运行面选择。 +- 按任务边界未部署、未合并。 diff --git a/docs/MDTODO/web-probe-kafka-refresh-delivery.md b/docs/MDTODO/web-probe-kafka-refresh-delivery.md index 9b57fc3a..9a6410af 100644 --- a/docs/MDTODO/web-probe-kafka-refresh-delivery.md +++ b/docs/MDTODO/web-probe-kafka-refresh-delivery.md @@ -5,7 +5,7 @@ 边界：只增强 YAML-first typed Web Probe；不部署、不合并，不引入临时脚本、URL/IP、HTTP history、projector 或 fallback。 -## R1 [in_progress] +## R1 [completed] 完成 [UniDesk #1751](https://github.com/pikasTech/unidesk/issues/1751) 的 YAML-first Kafka refresh→live typed Web Probe 验收，完成任务后将详细报告写入[任务报告](./details/web-probe-kafka-refresh-delivery/R1_Task_Report.md)。 ### R1.1 [completed] @@ -17,6 +17,6 @@ ### R1.3 [completed] 补齐 connected、retention→live、正式 user event、同 EventSource/envelope 的定向测试，完成任务后将详细报告写入[任务报告](./details/web-probe-kafka-refresh-delivery/R1.3_Task_Report.md)。 -### R1.4 [in_progress] +### R1.4 [completed] 完成帮助/skill、报告、提交、非 draft PR 与 preflight，保持不部署不合并，完成任务后将详细报告写入[任务报告](./details/web-probe-kafka-refresh-delivery/R1.4_Task_Report.md)。 From 2dd8814078e19e7d663004038e5b3c1488ee97f2 Mon Sep 17 00:00:00 2001 From: Codex Date: Sat, 11 Jul 2026 07:03:12 +0200 Subject: [PATCH 4/5] fix(cicd): harden PaC source artifact verification --- .agents/skills/unidesk-cicd/SKILL.md | 10 +- .../unidesk-cicd/references/gitea-pac.md | 20 +- config/agentrun.yaml | 1 + config/platform-infra/pipelines-as-code.yaml | 8 + .../cicd/pac-source-artifact-runtime.mjs | 168 +++++++-- scripts/src/agentrun-lanes.ts | 2 + ...-pipelines-as-code-source-artifact.test.ts | 349 ++++++++++++++++-- ...infra-pipelines-as-code-source-artifact.ts | 346 +++++++++++++---- .../src/platform-infra-pipelines-as-code.ts | 148 ++++++-- 9 files changed, 882 insertions(+), 170 deletions(-) diff --git a/.agents/skills/unidesk-cicd/SKILL.md b/.agents/skills/unidesk-cicd/SKILL.md index 88c210a7..27503802 100644 --- a/.agents/skills/unidesk-cicd/SKILL.md +++ b/.agents/skills/unidesk-cicd/SKILL.md @@ -51,7 +51,15 @@ bun scripts/cli.ts platform-infra pipelines-as-code source-artifact verify-runti - CI/CD source authority 只能来自 YAML 声明的 Kubernetes 托管 source authority：legacy lane 使用 k8s git-mirror snapshot，Gitea/PaC migrated lane 使用 GitHub PR merge -> GitHub webhook bridge -> Gitea controlled mirror + immutable snapshot ref -> Pipelines-as-Code。受控命令先在 k8s 内同步/创建不可变 `refs/unidesk/snapshots/.../` stage ref；build/status/publish 只消费该 snapshot，host worktree、本地 `git fetch/pull`、可变 branch ref 或 Pipeline 内直连 GitHub 都不能作为 authoritative source。 - JD01/NC01 `agentrun--v02`、`sentinel--v03`、`hwlab--v03` 的正式 CI/CD closeout 入口是 `cicd status --node ` 和 `platform-infra pipelines-as-code closeout|status|history --target --consumer `。`closeout` 是通用 consumer 引导入口，只读取/等待 PaC、Tekton、GitOps、Argo 和 runtime 对齐，不触发旧手动发布。`cicd branch-follower` 和 `cicd gitea-actions-poc` 对这些 consumer 只保留历史/迁移只读用途，不得作为当前交付判断入口。 - PaC `.tekton` 文件必须用 Repository CR 的 target/node 参数隔离 JD01/NC01，避免同一个 Gitea push 在一个 target cluster 内额外创建另一个 target 的 PipelineRun；history/detail 必须按实际 PipelineRun prefix/pipeline 归属 consumer。 -- PaC source artifact 必须由 `config/platform-infra/pipelines-as-code.yaml#consumers[].sourceArtifact` 显式声明，并通过 `source-artifact plan|check|write|status|verify-runtime` 管理；desired 只来自 owning YAML 与共享 renderer，禁止从 live Pipeline/PipelineRun 反向导出。`verify-runtime` 必须绑定完整 source commit，未声明 owner 或证据冲突时 fail-closed。 +- PaC source artifact 必须遵循以下边界： + - 由 `config/platform-infra/pipelines-as-code.yaml#consumers[].sourceArtifact` 显式声明； + - 通过 `source-artifact plan|check|write|status|verify-runtime` 管理； + - `taskRunTemplate` 等运行事实归属 owning YAML，代码只负责校验和渲染； + - desired 只来自 owning YAML 与共享 renderer，禁止从 live Pipeline/PipelineRun 反向导出； + - `verify-runtime` 使用干净且 `HEAD` 等于完整 source commit 的独立 worktree； + - 未声明 owner、worktree 证据不成立或运行面证据冲突时 fail-closed； + - 错误和 drift 在默认、`--json`、`--full` 中都只能披露路径、类型、长度、SHA-256 或 URL fingerprint； + - 禁止回显脚本、URL 凭据、查询参数、Authorization、Secret 和远端 stderr。 - PaC source-only closeout 必须使用目标侧结构化证据： - `g14-ci-plan` 的 source commit、affected/build/rollout 计数必须完整； - `collect-artifacts` 与 `gitops-promote` 必须同时明确 `no-build-no-rollout-plan`； diff --git a/.agents/skills/unidesk-cicd/references/gitea-pac.md b/.agents/skills/unidesk-cicd/references/gitea-pac.md index 51132ba6..ef091b84 100644 --- a/.agents/skills/unidesk-cicd/references/gitea-pac.md +++ b/.agents/skills/unidesk-cicd/references/gitea-pac.md @@ -34,11 +34,25 @@ bun scripts/cli.ts platform-infra pipelines-as-code source-artifact verify-runti - desired 只来自 owning YAML 与共享 domain renderer；live Pipeline 和 PipelineRun 只能作为只读诊断证据，禁止从运行面反向导出 desired 或源码制品。 - `plan`、`check`、`write` 只比较或更新显式消费仓 worktree，不访问运行面；worktree 必须是绝对路径、Git 根目录，并与声明仓库的精确 remote identity 一致。 - `write` 先完成全部路径、内容和 renderer 合同校验，再通过同目录临时文件与 rename 更新；连续执行必须无差异。 -- `status` 是非门禁诊断，允许不传 commit；`verify-runtime` 必须传完整四十位 `--source-commit`，并按 PaC/source-commit label 或 annotation 精确筛选 PipelineRun，不得按 prefix 直接取全局 latest。 +- `status` 是非门禁诊断，允许不传 commit。 +- `verify-runtime` 必须满足以下条件： + - 传入完整四十位 `--source-commit`； + - `--source-worktree` 干净； + - worktree `HEAD` 与该 commit 完全相同； + - 合并提交的验收新建精确提交 worktree，不使用带未提交修改或停在父提交的开发 worktree； + - PipelineRun 按 PaC/source-commit label 或 annotation 精确筛选，不按 prefix 直接取全局 latest。 - 同一 source commit 因 webhook 重投或 PaC retry 产生多个 PipelineRun 时，只在完整内联 spec 与 provenance 全部一致时确定性选择最新一条并披露候选数；存在冲突必须 fail-closed。 -- runtime observer 必须区分对象 `NotFound`、transport/RBAC/命令不可用和 spec drift；完整大 spec 通过受控临时文件传入目标侧原生 observer，不得放入 shell argv 或环境变量。 +- runtime observer 必须满足以下约束： + - 区分对象 `NotFound`、transport/RBAC/命令不可用和 spec drift； + - 完整大 spec 通过受控临时文件传入目标侧原生 observer，不放入 shell argv 或环境变量； + - 默认、`--json` 和 `--full` 的错误与 drift 只输出结构路径、类型、长度、SHA-256 或 URL fingerprint； + - 不回显脚本、URL 凭据、查询参数、Authorization、Secret、stdout 或 stderr 原文。 - 完整 spec 比较只移除六类已验证的 Tekton admission default，并限定在裸 Pipeline spec、完整 Pipeline 或 PipelineRun `pipelineSpec` 三个明确根；其他同名字段不得过滤。 -- HWLAB source artifact 必须保留正式 postprocess、verify 和六个 Kafka refresh 环境合同；AgentRun source artifact 必须保留 owning Pipeline 的完整 RBAC、参数和 manager 环境合同。 +- `sourceArtifact.taskRunTemplate` 是 `hostNetwork`、`dnsPolicy` 和 `fsGroup` 的唯一所有者；生成器不得另设硬编码默认值。 +- HWLAB source artifact 必须经过完整 domain renderer、正式 postprocess/verify 和完整 spec 比较。 + - Kafka refresh 等业务合同由 owning renderer 与消费者结构化负例测试保障； + - 不得在通用生成器中用字符串 marker 扫描建立第二真相。 +- AgentRun source artifact 通过完整 owning Pipeline 与完整 spec 比较保留 RBAC、参数和 manager 环境合同。 ## Coverage Matrix diff --git a/config/agentrun.yaml b/config/agentrun.yaml index 7ead6dec..ae60ce2a 100644 --- a/config/agentrun.yaml +++ b/config/agentrun.yaml @@ -454,6 +454,7 @@ controlPlane: source: statusMode: k3s-git-mirror repository: pikasTech/agentrun + worktreeRemote: git@github.com:pikasTech/agentrun.git branch: v0.2 sourceAuthority: mode: gitMirrorSnapshot diff --git a/config/platform-infra/pipelines-as-code.yaml b/config/platform-infra/pipelines-as-code.yaml index 3f537f05..3ad13012 100644 --- a/config/platform-infra/pipelines-as-code.yaml +++ b/config/platform-infra/pipelines-as-code.yaml @@ -110,6 +110,10 @@ consumers: configRef: config/agentrun.yaml#controlPlane.lanes.nc01-v02 pipelineRunPath: .tekton/agentrun-nc01-v02.yaml maxKeepRuns: 8 + taskRunTemplate: + hostNetwork: true + dnsPolicy: ClusterFirstWithHostNet + fsGroup: 1000 - extends: templates.consumers.sentinelV03 variables: NODE: JD01 @@ -151,6 +155,10 @@ consumers: pipelinePath: ci/pipelines/hwlab-nc01-v03-ci-image-publish.yaml pipelineRunPath: .tekton/hwlab-nc01-v03-pac.yaml maxKeepRuns: 8 + taskRunTemplate: + hostNetwork: true + dnsPolicy: ClusterFirstWithHostNet + fsGroup: 1000 templates: repositories: agentrunV02: diff --git a/scripts/native/cicd/pac-source-artifact-runtime.mjs b/scripts/native/cicd/pac-source-artifact-runtime.mjs index 6fbf92bf..50e003a0 100644 --- a/scripts/native/cicd/pac-source-artifact-runtime.mjs +++ b/scripts/native/cicd/pac-source-artifact-runtime.mjs @@ -6,6 +6,7 @@ const provenanceKeys = { configRef: "unidesk.ai/owning-config-ref", effectiveConfigSha256: "unidesk.ai/effective-config-sha256", renderer: "unidesk.ai/source-artifact-renderer", + mode: "unidesk.ai/source-artifact-mode", }; const sourceCommitKeys = [ @@ -16,14 +17,59 @@ const sourceCommitKeys = [ "hwlab.pikastech.local/source-commit", ]; +const knownDriftPathKeys = new Set([ + "accessModes", "annotations", "apiVersion", "args", "command", "computeResources", "default", "description", "dnsPolicy", "env", "envFrom", + "executionMode", "finally", "fsGroup", "generateName", "hostNetwork", "image", "kind", "labels", "length", "metadata", "mode", "mountPath", "name", "namespace", "onError", "operator", + "optional", "params", "pipeline", "pipelineRef", "pipelineSpec", "podTemplate", "readinessProbe", "readOnly", "requests", "resources", + "results", "retries", "runAfter", "script", "secret", "secretName", "securityContext", "serviceAccountName", "sidecars", "spec", "steps", + "storage", "subPath", "taskRunTemplate", "taskSpec", "tasks", "timeout", "timeouts", "type", "value", "values", "volumeClaimTemplate", + "volumeMounts", "volumes", "when", "workspaces", "workingDir", "wrapper", +]); + function record(value) { return value !== null && typeof value === "object" && !Array.isArray(value) ? value : {}; } -function compact(value) { - const raw = JSON.stringify(value); - if (raw === undefined) return String(value); - return raw.length <= 160 ? raw : `${raw.slice(0, 157)}...`; +function stableValue(value) { + if (Array.isArray(value)) return value.map(stableValue); + if (value !== null && typeof value === "object") { + return Object.fromEntries(Object.keys(value).sort().map((key) => [key, stableValue(value[key])])); + } + if (value === undefined) return { __type: "undefined" }; + if (typeof value === "bigint") return { __type: "bigint", value: String(value) }; + return value; +} + +function valueEvidence(value) { + const type = value === null ? "null" : Array.isArray(value) ? "array" : typeof value; + const serialized = JSON.stringify(stableValue(value)); + const text = serialized === undefined ? String(value) : serialized; + const length = typeof value === "string" + ? Buffer.byteLength(value, "utf8") + : Array.isArray(value) + ? value.length + : value !== null && typeof value === "object" + ? Object.keys(value).length + : Buffer.byteLength(text, "utf8"); + const sha256 = createHash("sha256").update(text).digest("hex"); + return `type=${type},length=${length},sha256=${sha256}`; +} + +function safeRuntimeReason(reason) { + const text = typeof reason === "string" ? reason : String(reason); + const fixed = new Set([ + "source-commit-not-specified", + "source-commit-run-not-found", + "source-commit-identity-conflict", + "pipeline-get-not-found", + "pipelinerun-list-not-found", + "pipelinerun-list-invalid-shape", + ]); + if (fixed.has(text)) return text; + if (/^source-commit-run-conflict:\d+$/u.test(text)) return text; + if (/^(?:pipeline-get|pipelinerun-list)-command-failed:(?:unknown|\d+):type=string,length=\d+,sha256=[0-9a-f]{64}$/u.test(text)) return text; + if (/^runtime-observer-input-error:type=string,length=\d+,sha256=[0-9a-f]{64}$/u.test(text)) return text; + return `runtime-observer-reason:${valueEvidence(text)}`; } function unavailable(reason, sourceCommit = null, candidateCount = 0) { @@ -36,7 +82,7 @@ function unavailable(reason, sourceCommit = null, candidateCount = 0) { configRef: null, effectiveConfigSha256: null, sourceCommit, - reason, + reason: safeRuntimeReason(reason), candidateCount, }; } @@ -82,8 +128,8 @@ export function canonicalPacPipelineSha256(value) { function firstCanonicalDrift(expected, actual, path = "$") { if (Object.is(expected, actual)) return null; if (Array.isArray(expected) || Array.isArray(actual)) { - if (!Array.isArray(expected) || !Array.isArray(actual)) return { path, expected: compact(expected), actual: compact(actual) }; - if (expected.length !== actual.length) return { path: `${path}.length`, expected: String(expected.length), actual: String(actual.length) }; + if (!Array.isArray(expected) || !Array.isArray(actual)) return { path, expected: valueEvidence(expected), actual: valueEvidence(actual) }; + if (expected.length !== actual.length) return { path: `${path}.length`, expected: valueEvidence(expected.length), actual: valueEvidence(actual.length) }; for (let index = 0; index < expected.length; index += 1) { const drift = firstCanonicalDrift(expected[index], actual[index], `${path}[${index}]`); if (drift !== null) return drift; @@ -93,18 +139,21 @@ function firstCanonicalDrift(expected, actual, path = "$") { const expectedRecord = expected !== null && typeof expected === "object"; const actualRecord = actual !== null && typeof actual === "object"; if (expectedRecord || actualRecord) { - if (!expectedRecord || !actualRecord) return { path, expected: compact(expected), actual: compact(actual) }; + if (!expectedRecord || !actualRecord) return { path, expected: valueEvidence(expected), actual: valueEvidence(actual) }; const keys = [...new Set([...Object.keys(expected), ...Object.keys(actual)])].sort(); for (const key of keys) { - if (!Object.prototype.hasOwnProperty.call(expected, key) || !Object.prototype.hasOwnProperty.call(actual, key)) { - return { path: `${path}.${key}`, expected: compact(expected[key]), actual: compact(actual[key]) }; + const expectedOwnsKey = Object.prototype.hasOwnProperty.call(expected, key); + const actualOwnsKey = Object.prototype.hasOwnProperty.call(actual, key); + const childPath = knownDriftPathKeys.has(key) ? `${path}.${key}` : `${path}.[key:${valueEvidence(key)}]`; + if (!expectedOwnsKey || !actualOwnsKey) { + return { path: childPath, expected: valueEvidence(expected[key]), actual: valueEvidence(actual[key]) }; } - const drift = firstCanonicalDrift(expected[key], actual[key], `${path}.${key}`); + const drift = firstCanonicalDrift(expected[key], actual[key], childPath); if (drift !== null) return drift; } return null; } - return { path, expected: compact(expected), actual: compact(actual) }; + return { path, expected: valueEvidence(expected), actual: valueEvidence(actual) }; } function firstDrift(expected, actual) { @@ -119,10 +168,11 @@ function kubectlJson(args, label) { }; } catch (error) { const stderr = typeof error?.stderr === "string" ? error.stderr : Buffer.isBuffer(error?.stderr) ? error.stderr.toString("utf8") : ""; - const message = stderr.trim().split(/\r?\n/u)[0] || String(error?.message ?? "command failed"); + const stdout = typeof error?.stdout === "string" ? error.stdout : Buffer.isBuffer(error?.stdout) ? error.stdout.toString("utf8") : ""; + const message = stderr.trim().split(/\r?\n/u)[0] || stdout.trim().split(/\r?\n/u)[0] || String(error?.message ?? "command failed"); if (/\bNotFound\b|\bnot found\b/u.test(message)) return { kind: "not-found", reason: `${label}-not-found` }; const status = Number.isInteger(error?.status) ? error.status : "unknown"; - return { kind: "unavailable", reason: `${label}-command-failed:${status}:${compact(message)}` }; + return { kind: "unavailable", reason: `${label}-command-failed:${status}:${valueEvidence(message)}` }; } } @@ -132,6 +182,7 @@ function manifestProvenance(manifest) { configRef: typeof annotations[provenanceKeys.configRef] === "string" ? annotations[provenanceKeys.configRef] : null, effectiveConfigSha256: typeof annotations[provenanceKeys.effectiveConfigSha256] === "string" ? annotations[provenanceKeys.effectiveConfigSha256] : null, renderer: typeof annotations[provenanceKeys.renderer] === "string" ? annotations[provenanceKeys.renderer] : null, + mode: typeof annotations[provenanceKeys.mode] === "string" ? annotations[provenanceKeys.mode] : null, }; } @@ -144,20 +195,68 @@ function manifestSourceCommits(manifest) { .map((value) => value.toLowerCase()))]; } -function observedItem(input, manifest, spec, sourceCommit = null, candidateCount = 0) { +function safeManifestName(value) { + return typeof value === "string" + && value.length <= 253 + && /^[a-z0-9](?:[-a-z0-9.]*[a-z0-9])?$/u.test(value) + ? value + : null; +} + +function safeAlignedConfigRef(observed, expected) { + return observed === expected + && typeof expected === "string" + && expected.length <= 512 + && /^[A-Za-z0-9_./-]+#[A-Za-z0-9_.-]+$/u.test(expected) + ? expected + : null; +} + +function safeAlignedSha256(observed, expected) { + return observed === expected && typeof expected === "string" && /^sha256:[0-9a-f]{64}$/u.test(expected) ? expected : null; +} + +function pipelineRunWrapper(manifest) { + const spec = record(manifest?.spec); + const hasPipelineSpec = Object.prototype.hasOwnProperty.call(spec, "pipelineSpec"); + const hasPipelineRef = Object.prototype.hasOwnProperty.call(spec, "pipelineRef"); + const observedMode = manifestProvenance(manifest).mode; + const mode = observedMode === "embedded-pipeline-spec" || observedMode === "remote-pipeline-annotation" ? observedMode : "invalid"; + const executionMode = hasPipelineSpec && !hasPipelineRef ? "pipeline-spec" : "invalid"; + const wrapperSpec = { ...spec }; + delete wrapperSpec.pipelineSpec; + delete wrapperSpec.pipelineRef; + return { mode, executionMode, spec: wrapperSpec }; +} + +function observedItem(input, manifest, spec, sourceCommit = null, candidateCount = 0, wrapper = null) { const observedProvenance = manifestProvenance(manifest); const provenanceAligned = observedProvenance.configRef === input.provenance.configRef && observedProvenance.effectiveConfigSha256 === input.provenance.effectiveConfigSha256 - && observedProvenance.renderer === input.provenance.renderer; - const drift = firstDrift(input.desiredSpec, spec); + && observedProvenance.renderer === input.provenance.renderer + && observedProvenance.mode === input.provenance.mode; + const comparePipelineSpec = wrapper === null || wrapper.executionMode === "pipeline-spec"; + const specDrift = comparePipelineSpec ? firstDrift(input.desiredSpec, spec) : null; + const wrapperDrift = wrapper === null || input.desiredWrapper === null + ? null + : firstDrift(input.desiredWrapper, wrapper); + const drift = specDrift ?? wrapperDrift; return { status: drift === null ? "aligned" : "drifted", - name: typeof record(manifest.metadata).name === "string" ? record(manifest.metadata).name : null, - canonicalSha256: canonicalPacPipelineSha256(spec), + name: safeManifestName(record(manifest.metadata).name), + canonicalSha256: canonicalPacPipelineSha256( + wrapper === null + ? spec + : wrapper.executionMode === "pipeline-spec" + ? { pipelineSpec: spec, wrapper } + : { wrapper }, + ), firstDrift: drift, provenanceAligned, - configRef: observedProvenance.configRef, - effectiveConfigSha256: observedProvenance.effectiveConfigSha256, + configRef: safeAlignedConfigRef(observedProvenance.configRef, input.provenance.configRef), + effectiveConfigSha256: safeAlignedSha256(observedProvenance.effectiveConfigSha256, input.provenance.effectiveConfigSha256), + renderer: observedProvenance.renderer === input.provenance.renderer ? input.provenance.renderer : null, + mode: observedProvenance.mode === input.provenance.mode ? input.provenance.mode : null, sourceCommit, reason: null, candidateCount, @@ -205,11 +304,23 @@ export function observePacSourceArtifactRuntime(input, readKubectlJson = kubectl const selected = selectPipelineRunBySourceCommit(items, input.pipelineRunPrefix, input.sourceCommit); if (selected.kind === "ok") { const signatures = selected.candidates.map((run) => JSON.stringify({ - canonicalSha256: canonicalPacPipelineSha256(record(record(run.spec).pipelineSpec)), + canonicalSha256: (() => { + const wrapper = pipelineRunWrapper(run); + return canonicalPacPipelineSha256(wrapper.executionMode === "pipeline-spec" + ? { pipelineSpec: record(record(run.spec).pipelineSpec), wrapper } + : { wrapper }); + })(), provenance: manifestProvenance(run), })); embedded = new Set(signatures).size === 1 - ? observedItem(input, selected.run, record(record(selected.run.spec).pipelineSpec), input.sourceCommit, selected.candidates.length) + ? observedItem( + input, + selected.run, + record(record(selected.run.spec).pipelineSpec), + input.sourceCommit, + selected.candidates.length, + pipelineRunWrapper(selected.run), + ) : unavailable(`source-commit-run-conflict:${selected.candidates.length}`, input.sourceCommit, selected.candidates.length); } else { embedded = selected.kind === "missing" @@ -222,17 +333,16 @@ export function observePacSourceArtifactRuntime(input, readKubectlJson = kubectl return { live, embedded }; } -if (typeof process.env.PAC_SOURCE_ARTIFACT_INPUT_FILE === "string" || typeof process.env.PAC_SOURCE_ARTIFACT_INPUT === "string") { +if (typeof process.env.PAC_SOURCE_ARTIFACT_INPUT_FILE === "string") { try { - const inputText = typeof process.env.PAC_SOURCE_ARTIFACT_INPUT_FILE === "string" - ? readFileSync(process.env.PAC_SOURCE_ARTIFACT_INPUT_FILE, "utf8") - : Buffer.from(process.env.PAC_SOURCE_ARTIFACT_INPUT, "base64").toString("utf8"); + const inputText = readFileSync(process.env.PAC_SOURCE_ARTIFACT_INPUT_FILE, "utf8"); const input = JSON.parse(inputText); process.stdout.write(JSON.stringify(observePacSourceArtifactRuntime(input))); } catch (error) { + const evidence = valueEvidence(String(error?.message ?? error)); process.stdout.write(JSON.stringify({ - live: unavailable(`runtime-observer-input-error:${compact(String(error?.message ?? error))}`), - embedded: unavailable(`runtime-observer-input-error:${compact(String(error?.message ?? error))}`), + live: unavailable(`runtime-observer-input-error:${evidence}`), + embedded: unavailable(`runtime-observer-input-error:${evidence}`), })); } } diff --git a/scripts/src/agentrun-lanes.ts b/scripts/src/agentrun-lanes.ts index 682ee63c..05615f30 100644 --- a/scripts/src/agentrun-lanes.ts +++ b/scripts/src/agentrun-lanes.ts @@ -108,6 +108,7 @@ export interface AgentRunLaneSpec { readonly source: { readonly statusMode: "host-worktree" | "k3s-git-mirror"; readonly repository: string; + readonly worktreeRemote: string; readonly branch: string; readonly sourceAuthority: AgentRunSourceAuthoritySpec | null; readonly sourceSnapshot: AgentRunSourceSnapshotSpec | null; @@ -595,6 +596,7 @@ function parseLane(lane: string, node: AgentRunNodeSpec, input: Record { return { "unidesk.ai/owning-config-ref": provenance.configRef, "unidesk.ai/effective-config-sha256": provenance.effectiveConfigSha256, "unidesk.ai/source-artifact-renderer": provenance.renderer, + "unidesk.ai/source-artifact-mode": provenance.mode, }; } @@ -51,7 +71,7 @@ function pipelineRun(name: string, creationTimestamp: string, spec: Record { test("verify-runtime requires and normalizes a full source commit", () => { expect(() => parsePacSourceArtifactOptions([ "verify-runtime", "--target", "NC01", "--consumer", "agentrun-nc01-v02", "--source-worktree", "/tmp/repo", - ])).toThrow("requires explicit --source-commit"); + ])).toThrow("source-commit-required"); const parsed = parsePacSourceArtifactOptions([ "verify-runtime", "--target", "NC01", "--consumer", "agentrun-nc01-v02", "--source-worktree", "/tmp/repo", "--source-commit", sourceCommit.toUpperCase(), ]); @@ -100,20 +121,87 @@ describe("PaC source artifact CLI contract", () => { "--target", "NC01", "--consumer", "agentrun-nc01-v02", "--source-worktree", rootPath(), ], { cwd: rootPath(), encoding: "utf8", timeout: 30_000 }); expect(result.status).toBe(1); - expect(result.stdout).toContain("origin does not match PaC repository pikasTech-agentrun"); - expect(result.stdout).toContain("pikasTech/unidesk.git"); + expect(result.stdout).toContain("code=source-worktree-origin-mismatch"); + expect(result.stdout).toContain("observedUrlFingerprint=sha256:"); + expect(result.stdout).not.toContain("observedOwnerRepo"); expect(result.stdout).not.toContain("stack"); expect(result.stdout).not.toContain("taskSpec"); expect(result.stdout.length).toBeLessThan(1500); }); + test("default, JSON, and full failures never disclose a malicious origin", () => { + const temporary = mkdtempSync(resolve(tmpdir(), "unidesk-pac-origin-test-")); + const secrets = [ + "review-user", + "REVIEW_SUPER_SECRET", + "QUERY_SUPER_SECRET", + "Authorization", + "https://review-user:REVIEW_SUPER_SECRET@example.invalid/pikasTech/agentrun.git?token=QUERY_SUPER_SECRET", + ]; + try { + for (const args of [ + ["init"], + ["config", "user.email", "pac-test@example.invalid"], + ["config", "user.name", "PaC Test"], + ]) expect(spawnSync("git", args, { cwd: temporary, encoding: "utf8" }).status).toBe(0); + writeFileSync(resolve(temporary, "README.md"), "fixture\n"); + expect(spawnSync("git", ["add", "README.md"], { cwd: temporary, encoding: "utf8" }).status).toBe(0); + expect(spawnSync("git", ["commit", "-m", "fixture"], { cwd: temporary, encoding: "utf8" }).status).toBe(0); + expect(spawnSync("git", ["remote", "add", "origin", secrets[4] as string], { cwd: temporary, encoding: "utf8" }).status).toBe(0); + const origins = [ + secrets[4] as string, + "audit@example.invalid:pikasTech/agentrun.git?token=SCP_QUERY_SUPER_SECRET", + "https://review-user:REVIEW_SUPER_SECRET@github.com/pikasTech/agentrun.git", + "https://github.com/pikasTech/agentrun.git?token=QUERY_SUPER_SECRET", + "https://github.com/pikasTech/agentrun.git#QUERY_SUPER_SECRET", + ]; + for (const origin of origins) { + expect(spawnSync("git", ["remote", "set-url", "origin", origin], { cwd: temporary, encoding: "utf8" }).status).toBe(0); + for (const outputFlag of [null, "--json", "--full"] as const) { + const args = [ + "scripts/cli.ts", "platform-infra", "pipelines-as-code", "source-artifact", "plan", + "--target", "NC01", "--consumer", "agentrun-nc01-v02", "--source-worktree", temporary, + ]; + if (outputFlag !== null) args.push(outputFlag); + const result = spawnSync("bun", args, { cwd: rootPath(), encoding: "utf8", timeout: 30_000 }); + const output = `${result.stdout}\n${result.stderr}`; + expect(result.status).toBe(1); + expect(output).toContain("source-worktree-origin-mismatch"); + expect(output).toContain("observedUrlFingerprint"); + expect(output).not.toContain("observedOwnerRepo"); + for (const secret of [...secrets, "SCP_QUERY_SUPER_SECRET"]) expect(output).not.toContain(secret); + } + } + } finally { + rmSync(temporary, { recursive: true, force: true }); + } + }, 15_000); + + test("verification requires a clean worktree at the exact requested commit", () => { + const exact = { head: sourceCommit, clean: true, matchesSourceCommit: true }; + expect(() => assertPacSourceArtifactVerifyWorktree("verify-runtime", sourceCommit, exact)).not.toThrow(); + expect(() => assertPacSourceArtifactVerifyWorktree("status", sourceCommit, { ...exact, clean: false })).not.toThrow(); + expect(() => assertPacSourceArtifactVerifyWorktree("verify-runtime", sourceCommit, { ...exact, clean: false })).toThrow("source-worktree-not-clean"); + expect(() => assertPacSourceArtifactVerifyWorktree("verify-runtime", sourceCommit, { ...exact, head: "b".repeat(40), matchesSourceCommit: false })).toThrow("source-worktree-commit-mismatch"); + }); + + test("taskRunTemplate operational facts are declared in owning YAML", () => { + const document = Bun.YAML.parse(readFileSync(rootPath("config", "platform-infra", "pipelines-as-code.yaml"), "utf8")) as Record; + const consumers = document.consumers as Array>; + for (const template of ["templates.consumers.agentrunV02", "templates.consumers.hwlabV03"]) { + const sourceArtifact = consumers.find((item) => item.extends === template && item.variables?.NODE === "NC01")?.sourceArtifact; + expect(sourceArtifact?.taskRunTemplate).toEqual({ hostNetwork: true, dnsPolicy: "ClusterFirstWithHostNet", fsGroup: 1000 }); + } + }); + test("undeclared JD01 source artifact owner fails closed", () => { const result = spawnSync("bun", [ "scripts/cli.ts", "platform-infra", "pipelines-as-code", "source-artifact", "plan", "--target", "JD01", "--consumer", "agentrun-jd01-v02", "--source-worktree", rootPath(), ], { cwd: rootPath(), encoding: "utf8", timeout: 30_000 }); expect(result.status).toBe(1); - expect(result.stdout).toContain("has no sourceArtifact owner"); + expect(result.stdout).toContain("code=source-artifact-operation-failed"); + expect(result.stdout).toContain("evidence=type=string"); }); }); @@ -167,6 +255,23 @@ describe("Tekton admission canonicalization", () => { }); }); +describe("source artifact serialization", () => { + test("remote Pipeline artifacts are deterministic reviewable multi-line YAML", () => { + const manifest = { + apiVersion: "tekton.dev/v1", + kind: "Pipeline", + metadata: { name: "fixture" }, + spec: { params: [{ name: "revision", type: "string" }], tasks: [{ name: "build", taskSpec: { steps: [{ name: "build", script: "set -eu\necho ok\n" }] } }] }, + }; + const first = pacSourceArtifactYaml(manifest); + const second = pacSourceArtifactYaml(manifest); + expect(first).toBe(second); + expect(first.split("\n").length).toBeGreaterThan(10); + expect(first).toContain("apiVersion: tekton.dev/v1\nkind: Pipeline\n"); + expect(Bun.YAML.parse(first)).toEqual(manifest); + }); +}); + describe("runtime source-commit selection", () => { test("same-commit retries select the newest only when spec and provenance agree", () => { const oldRun = pipelineRun("agentrun-nc01-v02-ci-old", "2026-07-10T00:00:00Z"); @@ -195,6 +300,51 @@ describe("runtime source-commit selection", () => { expect(observed.embedded.candidateCount).toBe(2); }); + test("embedded mode compares the complete PipelineRun wrapper", () => { + const run = pipelineRun("agentrun-nc01-v02-ci-wrapper", "2026-07-11T00:00:00Z"); + (run.spec as Record).taskRunTemplate.serviceAccountName = "wrong-service-account"; + const observed = observePacSourceArtifactRuntime(runtimeInput(), (_args: string[], label: string) => label === "pipeline-get" + ? { kind: "ok", value: pipeline() } + : { kind: "ok", value: { items: [run] } }); + expect(observed.embedded.status).toBe("drifted"); + expect(observed.embedded.firstDrift?.path).toBe("$.spec.taskRunTemplate.serviceAccountName"); + }); + + test("remote mode validates Pipeline spec live and wrapper on the exact-commit Run", () => { + const remoteProvenance = { ...provenance, mode: "remote-pipeline-annotation" }; + const remoteAnnotations = { + ...manifestAnnotations(), + "unidesk.ai/source-artifact-mode": "remote-pipeline-annotation", + }; + const wrapper = { + mode: "remote-pipeline-annotation", + executionMode: "pipeline-spec", + spec: { ...runtimeWrapperSpec, timeouts: { pipeline: "1h0m0s" } }, + }; + const run = { + metadata: { + name: "hwlab-nc01-v03-ci-poll-remote", + creationTimestamp: "2026-07-11T00:00:00Z", + labels: { "pipelinesascode.tekton.dev/sha": sourceCommit }, + annotations: remoteAnnotations, + }, + spec: { pipelineSpec: desiredSpec, ...wrapper.spec }, + }; + const input = { + ...runtimeInput(), + pipeline: "hwlab-nc01-v03-ci-image-publish", + pipelineRunPrefix: "hwlab-nc01-v03-ci-poll", + provenance: remoteProvenance, + desiredWrapper: wrapper, + }; + const observed = observePacSourceArtifactRuntime(input, (_args: string[], label: string) => label === "pipeline-get" + ? { kind: "ok", value: { metadata: { name: input.pipeline, annotations: remoteAnnotations }, spec: desiredSpec } } + : { kind: "ok", value: { items: [run] } }); + expect(observed.live.status).toBe("aligned"); + expect(observed.embedded.status).toBe("aligned"); + expect(observed.embedded.firstDrift).toBeNull(); + }); + test("NotFound, missing commit, and transport failures remain distinct", () => { const noCommit = observePacSourceArtifactRuntime(runtimeInput(null), () => ({ kind: "not-found", reason: "pipeline-get-not-found" })); expect(noCommit.live.status).toBe("missing"); @@ -202,14 +352,170 @@ describe("runtime source-commit selection", () => { expect(noCommit.embedded.status).toBe("unavailable"); expect(noCommit.embedded.reason).toBe("source-commit-not-specified"); - const transport = observePacSourceArtifactRuntime(runtimeInput(), (_args: string[], label: string) => ({ kind: "unavailable", reason: `${label}-command-failed:126:permission denied` })); + const secret = "Authorization: Bearer REVIEW_RUNTIME_SECRET https://review:password@example.invalid/path?token=QUERY_SECRET"; + const transport = observePacSourceArtifactRuntime(runtimeInput(), () => ({ kind: "unavailable", reason: secret })); expect(transport.live.status).toBe("unavailable"); expect(transport.embedded.status).toBe("unavailable"); - expect(transport.live.reason).toContain("permission denied"); - expect(transport.embedded.reason).toContain("permission denied"); + expect(transport.live.reason).toMatch(/^runtime-observer-reason:type=string,length=\d+,sha256=[0-9a-f]{64}$/u); + expect(transport.embedded.reason).toMatch(/^runtime-observer-reason:type=string,length=\d+,sha256=[0-9a-f]{64}$/u); + expect(JSON.stringify(transport)).not.toContain(secret); + }); + + test("first drift reports structural evidence without script or token values", () => { + const expectedSecret = "Authorization: Bearer EXPECTED_SUPER_SECRET"; + const actualSecret = "https://review-user:ACTUAL_SUPER_SECRET@example.invalid/path?token=QUERY_SUPER_SECRET"; + const input = { + ...runtimeInput(), + desiredSpec: { tasks: [{ taskSpec: { steps: [{ script: expectedSecret }] } }] }, + }; + const observed = observePacSourceArtifactRuntime(input, (_args: string[], label: string) => label === "pipeline-get" + ? { kind: "ok", value: { metadata: { name: "pipeline", annotations: manifestAnnotations() }, spec: { tasks: [{ taskSpec: { steps: [{ script: actualSecret }] } }] } } } + : { kind: "ok", value: { items: [pipelineRun("agentrun-nc01-v02-ci-secret", "2026-07-11T00:00:00Z", { tasks: [{ taskSpec: { steps: [{ script: actualSecret }] } }] })] } }); + const serialized = JSON.stringify(observed); + expect(observed.live.firstDrift?.path).toBe("$.tasks[0].taskSpec.steps[0].script"); + expect(observed.live.firstDrift?.expected).toBe(pacValueEvidence(expectedSecret)); + expect(observed.live.firstDrift?.actual).toBe(pacValueEvidence(actualSecret)); + expect(serialized).not.toContain(expectedSecret); + expect(serialized).not.toContain(actualSecret); + expect(serialized).not.toContain("Authorization"); + expect(serialized).not.toContain("QUERY_SUPER_SECRET"); + }); + + test("unknown runtime keys and forged paths are fingerprinted", () => { + const maliciousKey = "HEADER_QUERY_SUPER_SECRET"; + const local = firstPacSourceArtifactDrift({ tasks: [], [maliciousKey]: "expected" }, { tasks: [], [maliciousKey]: "actual" }); + expect(local?.path).not.toContain(maliciousKey); + expect(local?.path).toContain("type=string,length="); + + const maliciousInput = { ...runtimeInput(), desiredSpec: { ...desiredSpec, [maliciousKey]: "expected" } }; + const observed = observePacSourceArtifactRuntime(maliciousInput, (_args: string[], label: string) => label === "pipeline-get" + ? { kind: "ok", value: { ...pipeline(), spec: { ...desiredSpec, [maliciousKey]: "actual" } } } + : { kind: "ok", value: { items: [pipelineRun("agentrun-nc01-v02-ci-key", "2026-07-11T00:00:00Z", { ...desiredSpec, [maliciousKey]: "actual" })] } }); + expect(JSON.stringify(observed)).not.toContain(maliciousKey); + expect(pacRuntimeSafePath(`$.${maliciousKey}`)).toBe(`path(${pacValueEvidence(`$.${maliciousKey}`)})`); + }); + + test("untrusted runtime annotations and identities are never echoed", () => { + const secret = "Authorization: Bearer OBSERVER_SUPER_SECRET https://u:p@example.invalid/x?token=QUERY_SECRET"; + const annotations = { + ...manifestAnnotations(), + "unidesk.ai/owning-config-ref": secret, + "unidesk.ai/effective-config-sha256": secret, + "unidesk.ai/source-artifact-renderer": secret, + }; + const observed = observePacSourceArtifactRuntime(runtimeInput(), (_args: string[], label: string) => label === "pipeline-get" + ? { kind: "ok", value: { metadata: { name: secret, annotations }, spec: desiredSpec } } + : { kind: "ok", value: { items: [{ ...pipelineRun("agentrun-nc01-v02-ci-annotation", "2026-07-11T00:00:00Z"), metadata: { name: secret, creationTimestamp: "2026-07-11T00:00:00Z", labels: { "pipelinesascode.tekton.dev/sha": sourceCommit }, annotations } }] } }); + const serialized = JSON.stringify(observed); + expect(observed.live.provenanceAligned).toBe(false); + expect(observed.live.configRef).toBeNull(); + expect(observed.live.effectiveConfigSha256).toBeNull(); + expect(observed.live.name).toBeNull(); + expect(serialized).not.toContain(secret); + expect(serialized).not.toContain("OBSERVER_SUPER_SECRET"); + expect(serialized).not.toContain("QUERY_SECRET"); + + const forged = pacRuntimeSourceArtifactItem({ + status: "drifted", + name: "observer-super-secret", + canonicalSha256: `sha256:${"c".repeat(64)}`, + firstDrift: { path: "$.HEADER_QUERY_SUPER_SECRET", expected: secret, actual: secret }, + provenanceAligned: false, + configRef: "SECRET/TOKEN#QUERY", + effectiveConfigSha256: `sha256:${"d".repeat(64)}`, + renderer: provenance.renderer, + mode: provenance.mode, + sourceCommit: "e".repeat(40), + reason: secret, + candidateCount: 1, + }, { + configRef: provenance.configRef, + effectiveConfigSha256: `sha256:${"f".repeat(64)}`, + renderer: provenance.renderer, + mode: provenance.mode, + sourceCommit, + exactName: null, + namePrefix: "agentrun-nc01-v02-ci", + }); + const forgedSerialized = JSON.stringify(forged); + expect(forged.name).toBeNull(); + expect(forged.configRef).toBeNull(); + expect(forged.effectiveConfigSha256).toBeNull(); + expect(forged.provenanceAligned).toBe(false); + expect(forged.sourceCommit).toBeNull(); + expect(forged.firstDrift?.path).not.toContain("HEADER_QUERY_SUPER_SECRET"); + expect(forgedSerialized).not.toContain(secret); + expect(forgedSerialized).not.toContain("SECRET/TOKEN#QUERY"); + + const forgedAligned = pacRuntimeSourceArtifactItem({ + status: "aligned", + provenanceAligned: true, + configRef: provenance.configRef, + effectiveConfigSha256: provenance.effectiveConfigSha256, + renderer: "forged-renderer", + mode: provenance.mode, + sourceCommit, + candidateCount: 1, + }, { + configRef: provenance.configRef, + effectiveConfigSha256: provenance.effectiveConfigSha256, + renderer: provenance.renderer, + mode: provenance.mode, + sourceCommit, + exactName: null, + namePrefix: null, + }); + expect(forgedAligned.provenanceAligned).toBe(false); + }); + + test("typed runtime alignment never reports a synthetic pending state", () => { + const aligned = observePacSourceArtifactRuntime(runtimeInput(), (_args: string[], label: string) => label === "pipeline-get" + ? { kind: "ok", value: pipeline() } + : { kind: "ok", value: { items: [pipelineRun("agentrun-nc01-v02-ci-aligned", "2026-07-11T00:00:00Z")] } }); + const drifted = structuredClone(aligned); + drifted.embedded.status = "drifted"; + const exact = { head: sourceCommit, clean: true, matchesSourceCommit: true }; + expect(sourceArtifactRuntimeAlignment(null, exact, sourceCommit)).toBe("not-requested"); + expect(sourceArtifactRuntimeAlignment(aligned, exact, sourceCommit)).toBe("aligned"); + expect(sourceArtifactRuntimeAlignment(drifted, exact, sourceCommit)).toBe("drifted"); + expect(sourceArtifactRuntimeAlignment(aligned, { ...exact, clean: false }, sourceCommit)).toBe("unavailable"); + expect(JSON.stringify([aligned, drifted])).not.toContain("pending"); + }); + + test("local transport and remote observer failures retain only bounded evidence", () => { + const secret = "Authorization: Bearer TRANSPORT_SUPER_SECRET https://review:password@example.invalid/path?token=QUERY_SECRET"; + const localReason = pacRuntimeTransportFailureReason({ exitCode: 126, stdout: secret, stderr: secret }); + expect(localReason).toContain(pacValueEvidence(secret)); + expect(localReason).not.toContain(secret); + expect(pacRuntimeSafeReason(secret)).toBe(`runtime-observer-reason:${pacValueEvidence(secret)}`); + + const temporary = mkdtempSync(resolve(tmpdir(), "unidesk-pac-runtime-redaction-test-")); + try { + const bin = resolve(temporary, "bin"); + mkdirSync(bin); + const kubectl = resolve(bin, "kubectl"); + writeFileSync(kubectl, `#!/bin/sh\nprintf '%s\\n' '${secret}' >&2\nexit 126\n`); + chmodSync(kubectl, 0o755); + const shell = renderPacSourceArtifactRuntimeObserverShell(runtimeInput()); + const result = spawnSync("sh", [], { + input: shell, + encoding: "utf8", + timeout: 30_000, + env: { ...process.env, PATH: `${bin}:${process.env.PATH ?? ""}` }, + }); + expect(result.status).toBe(0); + expect(result.stdout).toContain("command-failed:126:type=string"); + expect(result.stdout).not.toContain(secret); + expect(result.stdout).not.toContain("Authorization"); + expect(result.stdout).not.toContain("QUERY_SECRET"); + } finally { + rmSync(temporary, { recursive: true, force: true }); + } }); test("temp-file transport handles a payload larger than the current 203 KiB HWLAB Pipeline", () => { + const observerSource = readFileSync(rootPath("scripts", "native", "cicd", "pac-source-artifact-runtime.mjs"), "utf8"); + expect(observerSource).not.toMatch(/process\.env\.PAC_SOURCE_ARTIFACT_INPUT(?!_FILE)/u); const temporary = mkdtempSync(resolve(tmpdir(), "unidesk-pac-runtime-test-")); try { const bin = resolve(temporary, "bin"); @@ -220,6 +526,7 @@ describe("runtime source-commit selection", () => { pipeline: "hwlab-nc01-v03-ci-image-publish", pipelineRunPrefix: "hwlab-nc01-v03-ci-poll", desiredSpec: hugeSpec, + desiredWrapper: { mode: "embedded-pipeline-spec", executionMode: "pipeline-spec", spec: runtimeWrapperSpec }, provenance, sourceCommit, }; @@ -258,28 +565,6 @@ describe("runtime source-commit selection", () => { }); }); -describe("HWLAB domain Pipeline contract", () => { - const requiredMarkers = [ - "unidesk-runtime-gitops-postprocess", - "unidesk-runtime-gitops-verify", - "HWLAB_WORKBENCH_KAFKA_REFRESH_REPLAY_ENABLED", - "HWLAB_WORKBENCH_KAFKA_REFRESH_REPLAY_GROUP_PREFIX", - "HWLAB_WORKBENCH_KAFKA_REFRESH_REPLAY_TIMEOUT_MS", - "HWLAB_WORKBENCH_KAFKA_REFRESH_REPLAY_SCAN_LIMIT", - "HWLAB_WORKBENCH_KAFKA_REFRESH_REPLAY_MATCHED_EVENT_LIMIT", - "HWLAB_WORKBENCH_KAFKA_REFRESH_REPLAY_LIVE_BUFFER_LIMIT", - ]; - - test("accepts the complete renderer contract and rejects every missing critical marker", () => { - const complete = { tasks: [{ taskSpec: { steps: [{ script: requiredMarkers.join("\n") }] } }] }; - expect(() => assertHwlabPipelineContracts(complete)).not.toThrow(); - for (const marker of requiredMarkers) { - const missing = { tasks: [{ taskSpec: { steps: [{ script: requiredMarkers.filter((item) => item !== marker).join("\n") }] } }] }; - expect(() => assertHwlabPipelineContracts(missing)).toThrow(marker); - } - }); -}); - describe("shared HWLAB deploy overlay", () => { test("preserves exact undefined/null semantics without mutating its input", () => { const document = { diff --git a/scripts/src/platform-infra-pipelines-as-code-source-artifact.ts b/scripts/src/platform-infra-pipelines-as-code-source-artifact.ts index 9cca8f96..ab099f76 100644 --- a/scripts/src/platform-infra-pipelines-as-code-source-artifact.ts +++ b/scripts/src/platform-infra-pipelines-as-code-source-artifact.ts @@ -14,11 +14,18 @@ import { nodeRuntimeRenderOverlay } from "./hwlab-node/web-probe"; import { applyNodeRuntimeDeployYamlOverlay } from "./hwlab-node/deploy-overlay"; import type { RenderedCliResult } from "./output"; import { renderedCliResult } from "./agentrun/render"; -import { stableJsonSha256 } from "./stable-json"; +import { stableJsonSha256, stableJsonValue } from "./stable-json"; export type PacSourceArtifactMode = "embedded-pipeline-spec" | "remote-pipeline-annotation"; export type PacSourceArtifactRenderer = "agentrun-control-plane" | "hwlab-runtime-lane"; export type PacSourceArtifactAction = "plan" | "check" | "write" | "status" | "verify-runtime"; +export type PacSourceArtifactRuntimeAlignment = "not-requested" | "aligned" | "drifted" | "missing" | "unavailable"; + +export interface PacSourceArtifactTaskRunTemplateSpec { + readonly hostNetwork: boolean; + readonly dnsPolicy: "ClusterFirst" | "ClusterFirstWithHostNet" | "Default" | "None"; + readonly fsGroup: number; +} export interface PacSourceArtifactSpec { readonly mode: PacSourceArtifactMode; @@ -27,6 +34,7 @@ export interface PacSourceArtifactSpec { readonly pipelineRunPath: string; readonly pipelinePath: string | null; readonly maxKeepRuns: number; + readonly taskRunTemplate: PacSourceArtifactTaskRunTemplateSpec; } export interface PacSourceArtifactBinding { @@ -42,6 +50,7 @@ export interface PacSourceArtifactBinding { }; readonly repository: { readonly id: string; + readonly url: string; readonly cloneUrl: string; readonly owner: string; readonly repo: string; @@ -87,6 +96,7 @@ export interface PacSourceArtifactDrift { export type PacSourceArtifactRuntimeObserver = (input: { readonly binding: PacSourceArtifactBinding; readonly desiredSpec: Record; + readonly desiredWrapper: Record | null; readonly provenance: PacSourceArtifactProvenance; readonly sourceCommit: string | null; }) => Promise; @@ -95,6 +105,7 @@ interface PacSourceArtifactProvenance { readonly configRef: string; readonly effectiveConfigSha256: string; readonly renderer: PacSourceArtifactRenderer; + readonly mode: PacSourceArtifactMode; } interface DesiredArtifact { @@ -114,6 +125,48 @@ interface SourceInspection { readonly provenanceAligned: boolean; } +export interface SourceWorktreeState { + readonly head: string; + readonly clean: boolean; + readonly matchesSourceCommit: boolean | null; +} + +export class PacSourceArtifactError extends Error { + readonly code: string; + readonly safeDetails: Readonly>; + + constructor(code: string, safeDetails: Readonly> = {}) { + super(code); + this.name = "PacSourceArtifactError"; + this.code = code; + this.safeDetails = safeDetails; + } +} + +export interface PacSourceArtifactSafeError { + readonly code: string; + readonly evidence: string; + readonly details: Readonly>; +} + +export function pacSourceArtifactSafeError(error: unknown): PacSourceArtifactSafeError { + const message = error instanceof Error ? error.message : String(error); + const allowedDetailKeys = new Set([ + "expectedUrlFingerprint", + "observedUrlFingerprint", + "head", + "requestedSourceCommit", + ]); + const details = error instanceof PacSourceArtifactError + ? Object.fromEntries(Object.entries(error.safeDetails).map(([key, value]) => [key, allowedDetailKeys.has(key) ? value : pacValueEvidence(value)])) + : {}; + return { + code: error instanceof PacSourceArtifactError ? error.code : "source-artifact-operation-failed", + evidence: pacValueEvidence(message), + details, + }; +} + const provenanceKeys = { configRef: "unidesk.ai/owning-config-ref", effectiveConfigSha256: "unidesk.ai/effective-config-sha256", @@ -121,6 +174,15 @@ const provenanceKeys = { mode: "unidesk.ai/source-artifact-mode", } as const; +const knownDriftPathKeys = new Set([ + "accessModes", "annotations", "apiVersion", "args", "command", "computeResources", "default", "description", "dnsPolicy", "env", "envFrom", + "executionMode", "finally", "fsGroup", "generateName", "hostNetwork", "image", "kind", "labels", "length", "metadata", "mode", "mountPath", "name", "namespace", "onError", "operator", + "optional", "params", "pipeline", "pipelineRef", "pipelineSpec", "podTemplate", "readinessProbe", "readOnly", "requests", "resources", + "results", "retries", "runAfter", "script", "secret", "secretName", "securityContext", "serviceAccountName", "sidecars", "spec", "steps", + "storage", "subPath", "taskRunTemplate", "taskSpec", "tasks", "timeout", "timeouts", "type", "value", "values", "volumeClaimTemplate", + "volumeMounts", "volumes", "when", "workspaces", "workingDir", "wrapper", +]); + export function parsePacSourceArtifactOptions(args: readonly string[]): PacSourceArtifactOptions { const action = args[0]; if (!isSourceArtifactAction(action)) { @@ -165,7 +227,7 @@ export function parsePacSourceArtifactOptions(args: readonly string[]): PacSourc if (!isAbsolute(sourceWorktree)) throw new Error("--source-worktree must be an absolute path"); if (sourceCommit !== null && !/^[0-9a-f]{40}$/u.test(sourceCommit)) throw new Error("--source-commit must be a full 40-character git SHA"); if (sourceCommit !== null && action !== "status" && action !== "verify-runtime") throw new Error("--source-commit is accepted only by source-artifact status or verify-runtime"); - if (action === "verify-runtime" && sourceCommit === null) throw new Error("source-artifact verify-runtime requires explicit --source-commit"); + if (action === "verify-runtime" && sourceCommit === null) throw new PacSourceArtifactError("source-commit-required"); if (action === "write" && !confirm) throw new Error("source-artifact write requires --confirm"); if (action !== "write" && confirm) throw new Error("--confirm is accepted only by source-artifact write"); return { action, targetId, consumerId, sourceWorktree, sourceCommit, confirm, json, full }; @@ -183,6 +245,13 @@ export async function runPacSourceArtifact( throw new Error(`source-artifact consumer ${options.consumerId} does not match resolved consumer ${binding.consumer.id}`); } const sourceWorktree = validateSourceWorktree(options.sourceWorktree, binding); + const sourceHead = git(sourceWorktree, ["rev-parse", "HEAD"]).toLowerCase(); + const sourceWorktreeState: SourceWorktreeState = { + head: sourceHead, + clean: git(sourceWorktree, ["status", "--porcelain=v1", "--untracked-files=all"]).length === 0, + matchesSourceCommit: options.sourceCommit === null ? null : sourceHead === options.sourceCommit, + }; + assertPacSourceArtifactVerifyWorktree(options.action, options.sourceCommit, sourceWorktreeState); const desired = renderDesiredArtifact(binding, sourceWorktree); let source = inspectSourceArtifact(sourceWorktree, binding, desired); const written: string[] = []; @@ -194,14 +263,17 @@ export async function runPacSourceArtifact( const runtime = needsRuntime ? observeRuntime === undefined ? unavailableRuntimeObservation() - : await observeRuntime({ binding, desiredSpec: desired.desiredSpec, provenance: desired.provenance, sourceCommit: options.sourceCommit }) + : await observeRuntime({ + binding, + desiredSpec: desired.desiredSpec, + desiredWrapper: options.sourceCommit === null ? null : desiredRuntimePipelineRunWrapper(binding, desired.pipelineRun, options.sourceCommit), + provenance: desired.provenance, + sourceCommit: options.sourceCommit, + }) : null; const sourceOk = source.aligned && source.provenanceAligned; - const runtimeOk = runtime !== null - && runtime.live.status === "aligned" - && runtime.live.provenanceAligned - && runtime.embedded.status === "aligned" - && runtime.embedded.provenanceAligned; + const runtimeAlignment = sourceArtifactRuntimeAlignment(runtime, sourceWorktreeState, options.sourceCommit); + const runtimeOk = runtimeAlignment === "aligned"; const ok = options.action === "check" ? sourceOk : options.action === "verify-runtime" @@ -233,17 +305,10 @@ export async function runPacSourceArtifact( namespace: binding.consumer.namespace, }, source, + sourceWorktreeState, runtime, runtimeTarget: { sourceCommit: options.sourceCommit }, - runtimeAlignment: runtime === null - ? "not-requested" - : runtimeOk - ? "aligned" - : runtime.live.status === "missing" || runtime.embedded.status === "missing" - ? "missing" - : runtime.live.status === "unavailable" || runtime.embedded.status === "unavailable" - ? "unavailable" - : "pending", + runtimeAlignment, boundary: { desiredAuthority: "owning YAML plus domain renderer", liveExportAllowed: false, @@ -258,32 +323,90 @@ export async function runPacSourceArtifact( ], valuesPrinted: false, }, - next: sourceOk - ? options.action === "verify-runtime" - ? null - : `After the consumer PR rolls out, run source-artifact verify-runtime for ${binding.consumer.id} with --source-commit .` - : options.action === "write" - ? "Generated files are still inconsistent; inspect firstDrift and do not publish." - : `Run source-artifact write --confirm against the explicit ${sourceWorktree} worktree.`, + next: sourceArtifactNext(binding, options, sourceOk, sourceWorktreeState, runtime, runtimeAlignment), }; } +export function assertPacSourceArtifactVerifyWorktree( + action: PacSourceArtifactAction, + sourceCommit: string | null, + worktree: SourceWorktreeState, +): void { + if (action !== "verify-runtime") return; + if (sourceCommit === null) throw new PacSourceArtifactError("source-commit-required"); + if (!worktree.clean) throw new PacSourceArtifactError("source-worktree-not-clean", { head: worktree.head }); + if (worktree.matchesSourceCommit !== true) { + throw new PacSourceArtifactError("source-worktree-commit-mismatch", { + head: worktree.head, + requestedSourceCommit: sourceCommit, + }); + } +} + +export function sourceArtifactRuntimeAlignment( + runtime: PacSourceArtifactRuntimeObservation | null, + worktree: SourceWorktreeState, + sourceCommit: string | null, +): PacSourceArtifactRuntimeAlignment { + if (runtime === null) return "not-requested"; + if (sourceCommit !== null && (!worktree.clean || worktree.matchesSourceCommit !== true)) return "unavailable"; + if (runtime.live.status === "unavailable" || runtime.embedded.status === "unavailable") return "unavailable"; + if (runtime.live.status === "missing" || runtime.embedded.status === "missing") return "missing"; + if (runtime.live.status === "drifted" || runtime.embedded.status === "drifted") return "drifted"; + if (!runtime.live.provenanceAligned || !runtime.embedded.provenanceAligned) return "drifted"; + return "aligned"; +} + +function sourceArtifactNext( + binding: PacSourceArtifactBinding, + options: PacSourceArtifactOptions, + sourceOk: boolean, + worktree: SourceWorktreeState, + runtime: PacSourceArtifactRuntimeObservation | null, + runtimeAlignment: PacSourceArtifactRuntimeAlignment, +): string { + if (!sourceOk) { + return options.action === "write" + ? "Generated files remain inconsistent; inspect the safe firstDrift evidence and do not publish." + : `Run source-artifact write --confirm for ${binding.consumer.id}, then rerun check.`; + } + if (runtime === null) return `After the consumer PR rolls out, use a clean exact-commit worktree and run verify-runtime for ${binding.consumer.id} with --source-commit .`; + if (!worktree.clean || (options.sourceCommit !== null && worktree.matchesSourceCommit !== true)) { + return "Create a clean worktree at the exact GitHub merge commit, then rerun status or verify-runtime with that full source commit."; + } + if (runtime.live.status === "unavailable") return "Repair the owning control-plane runtime observer or RBAC, then rerun status before verification."; + if (runtime.live.status === "missing") return "Apply the owning YAML control plane, confirm the live Pipeline exists, then rerun verify-runtime."; + if (runtime.live.status === "drifted" || !runtime.live.provenanceAligned) return "Apply the owning YAML control plane so the live Pipeline spec and provenance align, then rerun verify-runtime."; + if (runtime.embedded.status === "unavailable") return "Inspect the exact-commit PaC PipelineRun identity and observer access, then rerun verify-runtime."; + if (runtime.embedded.status === "missing") return "Wait for or repair the exact-commit PaC PipelineRun; do not validate a prefix-selected run."; + if (runtime.embedded.status === "drifted" || !runtime.embedded.provenanceAligned) return "Regenerate and merge the consumer source artifact, then verify the new exact-commit PipelineRun."; + return runtimeAlignment === "aligned" + ? "Exact source, live Pipeline, embedded PipelineRun, and provenance are aligned; continue consumer closeout." + : "Resolve the reported runtime layer and rerun verify-runtime."; +} + export function renderPacSourceArtifactResult(result: Record): RenderedCliResult { const files = record(result.files); const desired = record(result.desired); const source = record(result.source); + const sourceWorktreeState = record(result.sourceWorktreeState); const runtime = result.runtime === null ? null : record(result.runtime); const live = runtime === null ? null : record(runtime.live); const embedded = runtime === null ? null : record(runtime.embedded); const firstDrift = source.firstDrift === null ? null : record(source.firstDrift); + const liveDrift = live?.firstDrift === null ? null : record(live?.firstDrift); + const embeddedDrift = embedded?.firstDrift === null ? null : record(embedded?.firstDrift); const lines = [ "PAC SOURCE ARTIFACT", `target=${text(result.target)} consumer=${text(result.consumer)} node=${text(result.node)} lane=${text(result.lane)}`, `mode=${text(result.mode)} renderer=${text(result.renderer)} mutation=${text(result.mutation)}`, `pipelineRun=${text(files.pipelineRun)} pipeline=${text(files.pipeline ?? "-")}`, `desired=${shortSha(desired.canonicalSha256)} source=${shortSha(source.pipelineCanonicalSha256)} aligned=${text(source.aligned)} provenance=${text(source.provenanceAligned)}`, - `firstDrift=${firstDrift === null ? "-" : `${text(firstDrift.path)} expected=${text(firstDrift.expected)} actual=${text(firstDrift.actual)}`}`, - `runtime=${text(result.runtimeAlignment)} live=${text(live?.name)}@${shortSha(live?.canonicalSha256)} embedded=${text(embedded?.name)}@${shortSha(embedded?.canonicalSha256)} commit=${text(embedded?.sourceCommit ?? record(result.runtimeTarget).sourceCommit)} candidates=${text(embedded?.candidateCount)}`, + `sourceWorktree=head:${shortSha(sourceWorktreeState.head)} clean:${text(sourceWorktreeState.clean)} commitMatch:${text(sourceWorktreeState.matchesSourceCommit)}`, + `sourceDrift=${formatDrift(firstDrift)}`, + `runtime=${text(result.runtimeAlignment)} commit=${text(embedded?.sourceCommit ?? record(result.runtimeTarget).sourceCommit)} candidates=${text(embedded?.candidateCount)}`, + `live=status:${text(live?.status)} name:${text(live?.name)} hash:${shortSha(live?.canonicalSha256)} provenance:${text(live?.provenanceAligned)} drift:${formatDrift(liveDrift)}`, + `embedded=status:${text(embedded?.status)} name:${text(embedded?.name)} hash:${shortSha(embedded?.canonicalSha256)} provenance:${text(embedded?.provenanceAligned)} drift:${formatDrift(embeddedDrift)}`, `runtimeReason=live:${text(live?.reason)} embedded:${text(embedded?.reason)}`, `written=${Array.isArray(files.written) && files.written.length > 0 ? files.written.join(",") : "-"}`, ]; @@ -319,18 +442,17 @@ function renderDesiredArtifact(binding: PacSourceArtifactBinding, sourceWorktree const rendered = sourceArtifact.renderer === "agentrun-control-plane" ? renderAgentRunDesiredPipeline(binding) : renderHwlabDesiredPipeline(binding, sourceWorktree); - const pipeline = withPipelineProvenance(rendered.pipeline, rendered.provenance, sourceArtifact.mode); + const pipeline = withPipelineProvenance(rendered.pipeline, rendered.provenance); const desiredSpec = requiredRecord(pipeline.spec, "rendered Pipeline.spec"); assertPipelineIdentity(pipeline, binding); - if (sourceArtifact.renderer === "hwlab-runtime-lane") assertHwlabPipelineContracts(desiredSpec); const pipelineRun = sourceArtifact.mode === "embedded-pipeline-spec" ? embeddedPipelineRun(binding, desiredSpec, rendered.provenance) : remotePipelineRun(binding, pipeline, rendered.provenance); const files = sourceArtifact.mode === "embedded-pipeline-spec" - ? [{ path: sourceArtifact.pipelineRunPath, content: yaml(pipelineRun) }] + ? [{ path: sourceArtifact.pipelineRunPath, content: pacSourceArtifactYaml(pipelineRun) }] : [ - { path: requiredString(sourceArtifact.pipelinePath, "sourceArtifact.pipelinePath"), content: `${JSON.stringify(pipeline)}\n` }, - { path: sourceArtifact.pipelineRunPath, content: yaml(pipelineRun) }, + { path: requiredString(sourceArtifact.pipelinePath, "sourceArtifact.pipelinePath"), content: pacSourceArtifactYaml(pipeline) }, + { path: sourceArtifact.pipelineRunPath, content: pacSourceArtifactYaml(pipelineRun) }, ]; return { pipeline, pipelineRun, provenance: rendered.provenance, desiredSpec, files }; } @@ -355,6 +477,7 @@ function renderAgentRunDesiredPipeline(binding: PacSourceArtifactBinding): { pip configRef: expectedConfigRef, effectiveConfigSha256: stableJsonSha256(spec), renderer: "agentrun-control-plane", + mode: binding.consumer.sourceArtifact.mode, }, }; } @@ -381,6 +504,7 @@ function renderHwlabDesiredPipeline(binding: PacSourceArtifactBinding, sourceWor configRef: expectedConfigRef, effectiveConfigSha256: stableJsonSha256(spec), renderer: "hwlab-runtime-lane", + mode: binding.consumer.sourceArtifact.mode, }, }; } @@ -540,16 +664,52 @@ function pipelineRunLabels(binding: PacSourceArtifactBinding, partOf: "agentrun" } function taskRunTemplate(binding: PacSourceArtifactBinding): Record { + const declared = binding.consumer.sourceArtifact.taskRunTemplate; return { serviceAccountName: `{{ service_account }}`, podTemplate: { - hostNetwork: true, - dnsPolicy: "ClusterFirstWithHostNet", - securityContext: { fsGroup: 1000 }, + hostNetwork: declared.hostNetwork, + dnsPolicy: declared.dnsPolicy, + securityContext: { fsGroup: declared.fsGroup }, }, }; } +function desiredRuntimePipelineRunWrapper( + binding: PacSourceArtifactBinding, + pipelineRun: Record, + sourceCommit: string, +): Record { + const sourceSpec = structuredClone(requiredRecord(pipelineRun.spec, "generated PipelineRun.spec")); + delete sourceSpec.pipelineSpec; + delete sourceSpec.pipelineRef; + return { + mode: binding.consumer.sourceArtifact.mode, + executionMode: "pipeline-spec", + spec: resolvePacRuntimeTemplates(sourceSpec, binding, sourceCommit), + }; +} + +function resolvePacRuntimeTemplates(value: unknown, binding: PacSourceArtifactBinding, sourceCommit: string): unknown { + if (Array.isArray(value)) return value.map((item) => resolvePacRuntimeTemplates(item, binding, sourceCommit)); + if (isRecord(value)) { + return Object.fromEntries(Object.entries(value).map(([key, item]) => [key, resolvePacRuntimeTemplates(item, binding, sourceCommit)])); + } + if (typeof value !== "string") return value; + const templateValues: Readonly> = { + ...binding.repository.params, + revision: sourceCommit, + repo_url: binding.repository.url, + }; + const resolved = value.replace(/\{\{\s*([A-Za-z0-9_]+)\s*\}\}/gu, (_match, key: string) => { + const replacement = templateValues[key]; + if (replacement === undefined) throw new PacSourceArtifactError("source-artifact-runtime-template-unresolved", { template: pacValueEvidence(key) }); + return replacement; + }); + if (/\{\{|\}\}/u.test(resolved)) throw new PacSourceArtifactError("source-artifact-runtime-template-invalid"); + return resolved; +} + function pipelineRunParams(binding: PacSourceArtifactBinding, desiredSpec: Record): Record[] { const params = arrayRecords(desiredSpec.params, "Pipeline.spec.params"); return params.map((param) => { @@ -585,7 +745,7 @@ function pipelineRunWorkspaces(binding: PacSourceArtifactBinding, desiredSpec: R }); } -function withPipelineProvenance(pipeline: Record, provenance: PacSourceArtifactProvenance, mode: PacSourceArtifactMode): Record { +function withPipelineProvenance(pipeline: Record, provenance: PacSourceArtifactProvenance): Record { const next = structuredClone(pipeline); const metadata = requiredRecord(next.metadata, "rendered Pipeline.metadata"); const annotations = isRecord(metadata.annotations) ? metadata.annotations : {}; @@ -594,7 +754,7 @@ function withPipelineProvenance(pipeline: Record, provenance: P [provenanceKeys.configRef]: provenance.configRef, [provenanceKeys.effectiveConfigSha256]: provenance.effectiveConfigSha256, [provenanceKeys.renderer]: provenance.renderer, - [provenanceKeys.mode]: mode, + [provenanceKeys.mode]: provenance.mode, }; return next; } @@ -604,7 +764,7 @@ function inspectSourceArtifact(sourceWorktree: string, binding: PacSourceArtifac const pipelineRunPath = safeArtifactPath(sourceWorktree, sourceArtifact.pipelineRunPath); const pipelinePath = sourceArtifact.pipelinePath === null ? null : safeArtifactPath(sourceWorktree, sourceArtifact.pipelinePath); if (!existsSync(pipelineRunPath) || (pipelinePath !== null && !existsSync(pipelinePath))) { - return { status: "missing", aligned: false, pipelineCanonicalSha256: null, pipelineRunCanonicalSha256: null, firstDrift: { path: "$", expected: "generated artifact", actual: "missing" }, provenanceAligned: false }; + return { status: "missing", aligned: false, pipelineCanonicalSha256: null, pipelineRunCanonicalSha256: null, firstDrift: drift("$", { artifact: "generated" }, undefined), provenanceAligned: false }; } const actualPipelineRun = parseYamlRecord(readFileSync(pipelineRunPath, "utf8"), sourceArtifact.pipelineRunPath); const actualPipeline = sourceArtifact.mode === "remote-pipeline-annotation" @@ -628,7 +788,7 @@ function inspectSourceArtifact(sourceWorktree: string, binding: PacSourceArtifac aligned, pipelineCanonicalSha256: canonicalSha256(requiredRecord(actualPipeline.spec, "source Pipeline.spec")), pipelineRunCanonicalSha256: canonicalSha256(actualPipelineRun), - firstDrift: pipelineDrift ?? pipelineRunDrift ?? (provenanceAligned ? null : { path: "$.metadata.annotations.provenance", expected: JSON.stringify(desired.provenance), actual: JSON.stringify(actualProvenance) }), + firstDrift: pipelineDrift ?? pipelineRunDrift ?? (provenanceAligned ? null : drift("$.metadata.annotations.provenance", desired.provenance, actualProvenance)), provenanceAligned, }; } @@ -639,34 +799,51 @@ function validateSourceWorktree(input: string, binding: PacSourceArtifactBinding const top = realpathSync(git(worktree, ["rev-parse", "--show-toplevel"])); if (top !== worktree) throw new Error(`--source-worktree must be the git worktree root; resolved ${top}`); const remote = git(worktree, ["remote", "get-url", "origin"]); - const repoIdentity = binding.repository.repo.toLowerCase().replace(/\.git$/u, ""); - const canonicalIdentity = repoIdentity.replace(/^pikastech-/u, "pikastech/"); - const allowedIdentities = new Set([ - canonicalIdentity, - `${binding.repository.owner}/${repoIdentity}`.toLowerCase(), - remoteRepositoryIdentity(binding.repository.cloneUrl), - ]); + const expectedRemote = owningSourceRemote(binding); + const expectedIdentity = remoteRepositoryIdentity(expectedRemote); const observedIdentity = remoteRepositoryIdentity(remote); - if (!allowedIdentities.has(observedIdentity)) { - throw new Error(`source worktree origin does not match PaC repository ${binding.repository.repo}; observed ${remote}`); + if (expectedIdentity === null || observedIdentity === null + || expectedIdentity.host !== observedIdentity.host + || expectedIdentity.ownerRepo !== observedIdentity.ownerRepo) { + throw new PacSourceArtifactError("source-worktree-origin-mismatch", { + expectedUrlFingerprint: fingerprint(expectedRemote), + observedUrlFingerprint: fingerprint(remote), + }); } safeArtifactPath(worktree, binding.consumer.sourceArtifact.pipelineRunPath); if (binding.consumer.sourceArtifact.pipelinePath !== null) safeArtifactPath(worktree, binding.consumer.sourceArtifact.pipelinePath); return worktree; } -function remoteRepositoryIdentity(remote: string): string { +function owningSourceRemote(binding: PacSourceArtifactBinding): string { + if (binding.consumer.sourceArtifact.renderer === "agentrun-control-plane") { + return resolveAgentRunLaneTarget({ node: binding.consumer.node, lane: binding.consumer.lane }).spec.source.worktreeRemote; + } + if (!isHwlabRuntimeLane(binding.consumer.lane)) throw new PacSourceArtifactError("owning-source-lane-unresolved"); + return hwlabRuntimeLaneSpecForNode(binding.consumer.lane, binding.consumer.node).gitUrl; +} + +function remoteRepositoryIdentity(remote: string): { readonly host: string; readonly ownerRepo: string } | null { const trimmed = remote.trim(); - const scp = /^[^/@\s]+@[^:\s]+:(.+)$/u.exec(trimmed); - let path = scp?.[1] ?? ""; - if (path.length === 0) { + const scp = /^[^/@\s]+@([^:\s]+):(.+)$/u.exec(trimmed); + let host = scp?.[1] ?? ""; + let path = scp?.[2] ?? ""; + if (host.length === 0 || path.length === 0) { try { - path = new URL(trimmed).pathname; + const parsed = new URL(trimmed); + if (parsed.username.length > 0 || parsed.password.length > 0 || parsed.search.length > 0 || parsed.hash.length > 0) return null; + host = parsed.hostname; + path = parsed.pathname; } catch { - throw new Error(`git remote must be an SSH or URL repository identity; observed ${remote}`); + return null; } } - return path.replace(/^\/+|\/+$/gu, "").replace(/\.git$/iu, "").toLowerCase(); + const segments = path.replace(/^\/+|\/+$/gu, "").split("/").filter(Boolean); + if (segments.length !== 2) return null; + const owner = segments[0]?.toLowerCase(); + const repo = segments[1]?.replace(/\.git$/iu, "").toLowerCase(); + if (!host || !owner || !repo || !/^[a-z0-9_.-]+$/u.test(owner) || !/^[a-z0-9_.-]+$/u.test(repo)) return null; + return { host: host.toLowerCase(), ownerRepo: `${owner}/${repo}` }; } function writeArtifactFilesAtomically(sourceWorktree: string, files: readonly { readonly path: string; readonly content: string }[]): string[] { @@ -754,37 +931,23 @@ function assertPipelineIdentity(pipeline: Record, binding: PacS if (metadata.namespace !== binding.consumer.namespace) throw new Error(`rendered Pipeline namespace ${String(metadata.namespace)} does not match consumer ${binding.consumer.namespace}`); } -export function assertHwlabPipelineContracts(spec: Record): void { - const text = JSON.stringify(spec); - const required = [ - "unidesk-runtime-gitops-postprocess", - "unidesk-runtime-gitops-verify", - "HWLAB_WORKBENCH_KAFKA_REFRESH_REPLAY_ENABLED", - "HWLAB_WORKBENCH_KAFKA_REFRESH_REPLAY_GROUP_PREFIX", - "HWLAB_WORKBENCH_KAFKA_REFRESH_REPLAY_TIMEOUT_MS", - "HWLAB_WORKBENCH_KAFKA_REFRESH_REPLAY_SCAN_LIMIT", - "HWLAB_WORKBENCH_KAFKA_REFRESH_REPLAY_MATCHED_EVENT_LIMIT", - "HWLAB_WORKBENCH_KAFKA_REFRESH_REPLAY_LIVE_BUFFER_LIMIT", - ]; - const missing = required.filter((value) => !text.includes(value)); - if (missing.length > 0) throw new Error(`HWLAB domain renderer missing required postprocess/verify refresh contract: ${missing.join(", ")}`); -} - function provenanceFromManifest(manifest: Record): PacSourceArtifactProvenance | null { const metadata = isRecord(manifest.metadata) ? manifest.metadata : {}; const annotations = isRecord(metadata.annotations) ? metadata.annotations : {}; const configRef = annotations[provenanceKeys.configRef]; const effectiveConfigSha256 = annotations[provenanceKeys.effectiveConfigSha256]; const renderer = annotations[provenanceKeys.renderer]; - if (typeof configRef !== "string" || typeof effectiveConfigSha256 !== "string" || (renderer !== "agentrun-control-plane" && renderer !== "hwlab-runtime-lane")) return null; - return { configRef, effectiveConfigSha256, renderer }; + const mode = annotations[provenanceKeys.mode]; + if (typeof configRef !== "string" || typeof effectiveConfigSha256 !== "string" || (renderer !== "agentrun-control-plane" && renderer !== "hwlab-runtime-lane") || (mode !== "embedded-pipeline-spec" && mode !== "remote-pipeline-annotation")) return null; + return { configRef, effectiveConfigSha256, renderer, mode }; } function provenanceEquals(actual: PacSourceArtifactProvenance | null, expected: PacSourceArtifactProvenance): boolean { return actual !== null && actual.configRef === expected.configRef && actual.effectiveConfigSha256 === expected.effectiveConfigSha256 - && actual.renderer === expected.renderer; + && actual.renderer === expected.renderer + && actual.mode === expected.mode; } function firstCanonicalDrift(expected: unknown, actual: unknown, path: string): PacSourceArtifactDrift | null { @@ -802,8 +965,11 @@ function firstCanonicalDrift(expected: unknown, actual: unknown, path: string): if (!isRecord(expected) || !isRecord(actual)) return drift(path, expected, actual); const keys = [...new Set([...Object.keys(expected), ...Object.keys(actual)])].sort(); for (const key of keys) { - if (!Object.prototype.hasOwnProperty.call(expected, key) || !Object.prototype.hasOwnProperty.call(actual, key)) return drift(`${path}.${key}`, expected[key], actual[key]); - const child = firstCanonicalDrift(expected[key], actual[key], `${path}.${key}`); + const expectedOwnsKey = Object.prototype.hasOwnProperty.call(expected, key); + const actualOwnsKey = Object.prototype.hasOwnProperty.call(actual, key); + const childPath = knownDriftPathKeys.has(key) ? `${path}.${key}` : `${path}.[key:${pacValueEvidence(key)}]`; + if (!expectedOwnsKey || !actualOwnsKey) return drift(childPath, expected[key], actual[key]); + const child = firstCanonicalDrift(expected[key], actual[key], childPath); if (child !== null) return child; } return null; @@ -829,13 +995,29 @@ function isProvenTektonAdmissionDefault(path: readonly (string | number)[], valu } function drift(path: string, expected: unknown, actual: unknown): PacSourceArtifactDrift { - return { path, expected: compactValue(expected), actual: compactValue(actual) }; + return { path, expected: pacValueEvidence(expected), actual: pacValueEvidence(actual) }; } -function compactValue(value: unknown): string { - const raw = JSON.stringify(value); - if (raw === undefined) return String(value); - return raw.length <= 160 ? raw : `${raw.slice(0, 157)}...`; +export function pacValueEvidence(value: unknown): string { + const type = value === null ? "null" : Array.isArray(value) ? "array" : typeof value; + const stable = stableJsonValue(value); + const serialized = JSON.stringify(stable) ?? String(stable); + const length = typeof value === "string" + ? Buffer.byteLength(value, "utf8") + : Array.isArray(value) + ? value.length + : isRecord(value) + ? Object.keys(value).length + : Buffer.byteLength(serialized, "utf8"); + return `type=${type},length=${length},sha256=${fingerprint(serialized).replace(/^sha256:/u, "")}`; +} + +function fingerprint(value: string): string { + return `sha256:${createHash("sha256").update(value).digest("hex")}`; +} + +function formatDrift(value: Record | null): string { + return value === null ? "-" : `${text(value.path)} expected:${text(value.expected)} actual:${text(value.actual)}`; } function unavailableRuntimeObservation(): PacSourceArtifactRuntimeObservation { @@ -849,8 +1031,8 @@ function parseYamlRecord(text: string, label: string): Record { return parsed; } -function yaml(value: Record): string { - return `${Bun.YAML.stringify(value).trim()}\n`; +export function pacSourceArtifactYaml(value: Record): string { + return `${Bun.YAML.stringify(value, null, 2).trim()}\n`; } function requiredParam(binding: PacSourceArtifactBinding, key: string): string { diff --git a/scripts/src/platform-infra-pipelines-as-code.ts b/scripts/src/platform-infra-pipelines-as-code.ts index 68e3a79a..5293cd28 100644 --- a/scripts/src/platform-infra-pipelines-as-code.ts +++ b/scripts/src/platform-infra-pipelines-as-code.ts @@ -19,6 +19,8 @@ import { import { materializeYamlComposition } from "./yaml-composition"; import { canonicalizePacPipelineSpec, + pacSourceArtifactSafeError, + pacValueEvidence, parsePacSourceArtifactOptions, renderPacSourceArtifactResult, runPacSourceArtifact, @@ -251,17 +253,26 @@ export async function runPlatformInfraPipelinesAsCodeCommand(config: UniDeskConf }, repository: { id: repository.id, + url: repository.url, cloneUrl: repository.cloneUrl, owner: repository.owner, repo: repository.repo, params: repository.params, }, }; - const result = await runPacSourceArtifact(binding, options, async ({ desiredSpec, provenance, sourceCommit }) => await observeSourceArtifactRuntime(config, target, consumer, desiredSpec, provenance, sourceCommit)); + const result = await runPacSourceArtifact(binding, options, async ({ desiredSpec, desiredWrapper, provenance, sourceCommit }) => await observeSourceArtifactRuntime(config, target, consumer, desiredSpec, desiredWrapper, provenance, sourceCommit)); return options.json || options.full ? result : renderPacSourceArtifactResult(result); } catch (error) { - if (structuredError) throw error; - return sourceArtifactValidationError(error); + const safeError = pacSourceArtifactSafeError(error); + if (structuredError) { + return { + ok: false, + action: "platform-infra-pipelines-as-code-source-artifact-validation", + error: safeError, + next: "Fix the declared owner, exact source identity, worktree proof, or CLI arguments; rerun plan before write.", + }; + } + return sourceArtifactValidationError(safeError); } } return { ok: false, error: "unsupported-platform-infra-pipelines-as-code-command", args, help: help() }; @@ -286,12 +297,12 @@ function sourceArtifactHelp(): RenderedCliResult { return { ok: true, command: "platform-infra-pipelines-as-code-source-artifact-help", renderedText: `${lines.join("\n")}\n`, contentType: "text/plain" }; } -function sourceArtifactValidationError(error: unknown): RenderedCliResult { - const reason = String(error instanceof Error ? error.message : error).replace(/[\r\n]+/gu, " ").slice(0, 1000); +function sourceArtifactValidationError(error: ReturnType): RenderedCliResult { + const details = Object.entries(error.details).map(([key, value]) => `${key}=${value}`).join(" "); return { ok: false, command: "platform-infra-pipelines-as-code-source-artifact-validation", - renderedText: `PAC SOURCE ARTIFACT ERROR\nreason=${reason}\nnext=Fix the explicit target, consumer, source worktree, or owning YAML declaration; rerun plan before write.\n`, + renderedText: `PAC SOURCE ARTIFACT ERROR\ncode=${error.code} evidence=${error.evidence}${details ? ` details=${details}` : ""}\nnext=Fix the declared owner, exact source identity, worktree proof, or CLI arguments; rerun plan before write.\n`, contentType: "text/plain", }; } @@ -442,6 +453,7 @@ function parseSourceArtifact(value: Record, path: string): PacS const renderer = y.enumField(value, "renderer", path, ["agentrun-control-plane", "hwlab-runtime-lane"] as const); const pipelineRunPath = sourceArtifactPath(y.stringField(value, "pipelineRunPath", path), `${path}.pipelineRunPath`); const pipelinePath = value.pipelinePath === undefined ? null : sourceArtifactPath(y.stringField(value, "pipelinePath", path), `${path}.pipelinePath`); + const taskRunTemplate = y.objectField(value, "taskRunTemplate", path); if (mode === "embedded-pipeline-spec" && pipelinePath !== null) throw new Error(`${path}.pipelinePath is forbidden for embedded-pipeline-spec`); if (mode === "remote-pipeline-annotation" && pipelinePath === null) throw new Error(`${path}.pipelinePath is required for remote-pipeline-annotation`); if (renderer === "agentrun-control-plane" && mode !== "embedded-pipeline-spec") throw new Error(`${path}.renderer agentrun-control-plane requires embedded-pipeline-spec`); @@ -453,6 +465,11 @@ function parseSourceArtifact(value: Record, path: string): PacS pipelineRunPath, pipelinePath, maxKeepRuns: positiveInteger(value, "maxKeepRuns", path), + taskRunTemplate: { + hostNetwork: y.booleanField(taskRunTemplate, "hostNetwork", `${path}.taskRunTemplate`), + dnsPolicy: y.enumField(taskRunTemplate, "dnsPolicy", `${path}.taskRunTemplate`, ["ClusterFirst", "ClusterFirstWithHostNet", "Default", "None"] as const), + fsGroup: positiveInteger(taskRunTemplate, "fsGroup", `${path}.taskRunTemplate`), + }, }; } @@ -466,7 +483,8 @@ async function observeSourceArtifactRuntime( target: PacTarget, consumer: PacConsumer, desiredSpec: Record, - provenance: { readonly configRef: string; readonly effectiveConfigSha256: string; readonly renderer: string }, + desiredWrapper: Record | null, + provenance: { readonly configRef: string; readonly effectiveConfigSha256: string; readonly renderer: string; readonly mode: string }, sourceCommit: string | null, ): Promise { const script = renderPacSourceArtifactRuntimeObserverShell({ @@ -474,24 +492,43 @@ async function observeSourceArtifactRuntime( pipeline: consumer.pipeline, pipelineRunPrefix: consumer.pipelineRunPrefix, desiredSpec: canonicalizePacPipelineSpec(desiredSpec), + desiredWrapper: desiredWrapper === null ? null : canonicalizePacPipelineSpec(desiredWrapper), provenance, sourceCommit, }); const captured = await capture(config, target.route, ["sh"], script); if (captured.exitCode !== 0) { - const compact = compactCapture(captured); - const detail = String(compact.stderrTail ?? compact.stdoutTail ?? "").trim().split(/\r?\n/u).filter(Boolean)[0]?.replace(/\s+/gu, " ").slice(0, 240); - return unavailableSourceArtifactRuntime(`runtime-transport-exit-${captured.exitCode}${detail ? `:${detail}` : ""}`, sourceCommit); + return unavailableSourceArtifactRuntime(pacRuntimeTransportFailureReason(captured), sourceCommit); } const parsed = parseJsonOutput(captured.stdout); if (parsed === null) return unavailableSourceArtifactRuntime("runtime-observer-invalid-json", sourceCommit); return { - live: runtimeSourceArtifactItem(parsed.live), - embedded: runtimeSourceArtifactItem(parsed.embedded), + live: pacRuntimeSourceArtifactItem(parsed.live, { + configRef: provenance.configRef, + effectiveConfigSha256: provenance.effectiveConfigSha256, + renderer: provenance.renderer, + mode: provenance.mode, + sourceCommit: null, + exactName: consumer.pipeline, + namePrefix: null, + }), + embedded: pacRuntimeSourceArtifactItem(parsed.embedded, { + configRef: provenance.configRef, + effectiveConfigSha256: provenance.effectiveConfigSha256, + renderer: provenance.renderer, + mode: provenance.mode, + sourceCommit, + exactName: null, + namePrefix: consumer.pipelineRunPrefix, + }), }; } +export function pacRuntimeTransportFailureReason(captured: { readonly exitCode: number; readonly stdout: string; readonly stderr: string }): string { + return `runtime-transport-exit-${captured.exitCode}:stderr(${pacValueEvidence(captured.stderr)}):stdout(${pacValueEvidence(captured.stdout)})`; +} + export function renderPacSourceArtifactRuntimeObserverShell(inputValue: Record): string { const input = JSON.stringify(inputValue); const observer = readFileSync(sourceArtifactRuntimeObserverFile, "utf8").trimEnd(); @@ -508,32 +545,97 @@ NODE_PAC_SOURCE_ARTIFACT_STATUS `; } -function runtimeSourceArtifactItem(value: unknown): PacSourceArtifactRuntimeObservation["live"] { +export function pacRuntimeSourceArtifactItem(value: unknown, expected: { + readonly configRef: string; + readonly effectiveConfigSha256: string; + readonly renderer: string; + readonly mode: string; + readonly sourceCommit: string | null; + readonly exactName: string | null; + readonly namePrefix: string | null; +}): PacSourceArtifactRuntimeObservation["live"] { if (typeof value !== "object" || value === null || Array.isArray(value)) return unavailableSourceArtifactRuntime("runtime-observer-invalid-item", null).live; const item = value as Record; const status = item.status; if (status !== "aligned" && status !== "drifted" && status !== "missing" && status !== "unavailable") return unavailableSourceArtifactRuntime("runtime-observer-invalid-status", null).live; const drift = item.firstDrift; + const observedName = typeof item.name === "string" && item.name.length <= 253 && /^[a-z0-9](?:[-a-z0-9.]*[a-z0-9])?$/u.test(item.name) ? item.name : null; + const name = observedName !== null + && (expected.exactName === null || observedName === expected.exactName) + && (expected.namePrefix === null || observedName.startsWith(expected.namePrefix)) + ? observedName + : null; + const provenanceAligned = item.configRef === expected.configRef + && item.effectiveConfigSha256 === expected.effectiveConfigSha256 + && item.renderer === expected.renderer + && item.mode === expected.mode; return { status, - name: typeof item.name === "string" ? item.name : null, - canonicalSha256: typeof item.canonicalSha256 === "string" ? item.canonicalSha256 : null, + name, + canonicalSha256: typeof item.canonicalSha256 === "string" && /^sha256:[0-9a-f]{64}$/u.test(item.canonicalSha256) ? item.canonicalSha256 : null, firstDrift: typeof drift === "object" && drift !== null && !Array.isArray(drift) ? { - path: String((drift as Record).path ?? "$"), - expected: String((drift as Record).expected ?? ""), - actual: String((drift as Record).actual ?? ""), + path: pacRuntimeSafePath((drift as Record).path), + expected: pacRuntimeSafeEvidence((drift as Record).expected), + actual: pacRuntimeSafeEvidence((drift as Record).actual), } : null, - provenanceAligned: item.provenanceAligned === true, - configRef: typeof item.configRef === "string" ? item.configRef : null, - effectiveConfigSha256: typeof item.effectiveConfigSha256 === "string" ? item.effectiveConfigSha256 : null, - sourceCommit: typeof item.sourceCommit === "string" ? item.sourceCommit : null, - reason: typeof item.reason === "string" ? item.reason : null, + provenanceAligned, + configRef: item.configRef === expected.configRef && expected.configRef.length <= 512 && /^[A-Za-z0-9_./-]+#[A-Za-z0-9_.-]+$/u.test(expected.configRef) ? expected.configRef : null, + effectiveConfigSha256: item.effectiveConfigSha256 === expected.effectiveConfigSha256 && /^sha256:[0-9a-f]{64}$/u.test(expected.effectiveConfigSha256) ? expected.effectiveConfigSha256 : null, + sourceCommit: item.sourceCommit === expected.sourceCommit && (expected.sourceCommit === null || /^[0-9a-f]{40}$/u.test(expected.sourceCommit)) ? expected.sourceCommit : null, + reason: typeof item.reason === "string" ? pacRuntimeSafeReason(item.reason) : null, candidateCount: typeof item.candidateCount === "number" && Number.isInteger(item.candidateCount) && item.candidateCount >= 0 ? item.candidateCount : 0, }; } +export function pacRuntimeSafePath(value: unknown): string { + const path = typeof value === "string" ? value : "$"; + const allowedSegments = new Set([ + "accessModes", "annotations", "apiVersion", "args", "command", "computeResources", "default", "description", "dnsPolicy", "env", "envFrom", + "executionMode", "finally", "fsGroup", "generateName", "hostNetwork", "image", "kind", "labels", "length", "metadata", "mode", "mountPath", "name", "namespace", "onError", "operator", + "optional", "params", "pipeline", "pipelineRef", "pipelineSpec", "podTemplate", "readinessProbe", "readOnly", "requests", "resources", + "results", "retries", "runAfter", "script", "secret", "secretName", "securityContext", "serviceAccountName", "sidecars", "spec", "steps", + "storage", "subPath", "taskRunTemplate", "taskSpec", "tasks", "timeout", "timeouts", "type", "value", "values", "volumeClaimTemplate", + "volumeMounts", "volumes", "when", "workspaces", "workingDir", "wrapper", + ]); + if (!path.startsWith("$")) return `path(${pacValueEvidence(path)})`; + const token = /(?:\.([A-Za-z0-9_-]+))|(?:\[(\d+)\])/gu; + let offset = 1; + for (const match of path.slice(1).matchAll(token)) { + if (match.index !== offset - 1) return `path(${pacValueEvidence(path)})`; + if (match[1] !== undefined && !allowedSegments.has(match[1])) return `path(${pacValueEvidence(path)})`; + offset = 1 + (match.index ?? 0) + match[0].length; + } + return offset === path.length ? path : `path(${pacValueEvidence(path)})`; +} + +function pacRuntimeSafeEvidence(value: unknown): string { + return typeof value === "string" && /^type=[a-z]+,length=\d+,sha256=[0-9a-f]{64}$/u.test(value) + ? value + : pacValueEvidence(value); +} + +export function pacRuntimeSafeReason(value: string): string { + const fixed = new Set([ + "source-commit-not-specified", + "source-commit-run-not-found", + "source-commit-identity-conflict", + "pipeline-get-not-found", + "pipelinerun-list-not-found", + "pipelinerun-list-invalid-shape", + "runtime-observer-invalid-json", + "runtime-observer-invalid-item", + "runtime-observer-invalid-status", + "runtime-observer-not-configured", + ]); + if (fixed.has(value)) return value; + if (/^source-commit-run-conflict:\d+$/u.test(value)) return value; + if (/^(?:pipeline-get|pipelinerun-list)-command-failed:(?:unknown|\d+):type=string,length=\d+,sha256=[0-9a-f]{64}$/u.test(value)) return value; + if (/^(?:runtime-observer-input-error|runtime-observer-reason):type=string,length=\d+,sha256=[0-9a-f]{64}$/u.test(value)) return value; + return `runtime-observer-reason:${pacValueEvidence(value)}`; +} + function unavailableSourceArtifactRuntime(reason: string, sourceCommit: string | null): PacSourceArtifactRuntimeObservation { const item = { status: "unavailable" as const, name: null, canonicalSha256: null, firstDrift: null, provenanceAligned: false, configRef: null, effectiveConfigSha256: null, sourceCommit, reason, candidateCount: 0 }; return { live: item, embedded: item }; From 46bdccd33ac9351be58a48432ffe9d74e2356c28 Mon Sep 17 00:00:00 2001 From: Codex Date: Sat, 11 Jul 2026 07:12:14 +0200 Subject: [PATCH 5/5] docs: clarify web probe test counts --- .../web-probe-kafka-refresh-delivery/R1.3_Task_Report.md | 7 +++++-- .../web-probe-kafka-refresh-delivery/R1_Task_Report.md | 4 +++- 2 files changed, 8 insertions(+), 3 deletions(-) diff --git a/docs/MDTODO/details/web-probe-kafka-refresh-delivery/R1.3_Task_Report.md b/docs/MDTODO/details/web-probe-kafka-refresh-delivery/R1.3_Task_Report.md index 849688d8..c79ff05c 100644 --- a/docs/MDTODO/details/web-probe-kafka-refresh-delivery/R1.3_Task_Report.md +++ b/docs/MDTODO/details/web-probe-kafka-refresh-delivery/R1.3_Task_Report.md @@ -1,7 +1,10 @@ # R1.3 任务报告 -- `bun test` 定向执行 17 个测试，全部通过。 +- 精确 Bun 测试命令： + - `bun test scripts/src/hwlab-node-web-observe-runner-realtime-source.test.ts scripts/src/hwlab-node/web-probe-observe-options.test.ts scripts/src/hwlab-node/web-probe-observe-collect.test.ts` + - 三个文件合计 17/17 通过：realtime 5、options 8、collect 4。 + - 若只执行 realtime 与 options 两个核心定向文件，精确计数为 13/13。 - 覆盖 YAML refresh/live connected 合同、legacy live-only profile、handoff 计数一致性和正式 user event 三段 lineage。 - 覆盖 semantic origin 继承、URL 选择歧义拒绝和既有 collect 行为。 -- `bun scripts/cli.ts check --syntax-only` 通过 11 项检查。 +- 独立命令 `bun scripts/cli.ts check --syntax-only` 通过 11/11；该计数未并入 Bun test 的 17 个测试。 - `git diff --check` 通过。 diff --git a/docs/MDTODO/details/web-probe-kafka-refresh-delivery/R1_Task_Report.md b/docs/MDTODO/details/web-probe-kafka-refresh-delivery/R1_Task_Report.md index f2101caf..432e638d 100644 --- a/docs/MDTODO/details/web-probe-kafka-refresh-delivery/R1_Task_Report.md +++ b/docs/MDTODO/details/web-probe-kafka-refresh-delivery/R1_Task_Report.md @@ -5,6 +5,8 @@ - `validateRealtimeFanout` 已从 owning YAML 读取可组合 product SSE 合同，同时支持显式 live-only 与 refresh+live profile。 - refresh 验收覆盖 connected、barrier、retention→live handoff、重连 retained trace、第二轮 live 增量和正式 user event 三段 lineage。 - product evidence 同时证明 `hwlab.event.v1` envelope 原样透传，以及主 Workbench 单一 session-scoped EventSource 进入同一 UI reducer 并渲染终态。 -- 定向测试 17/17、syntax-only 11/11、PR preflight `MERGEABLE/CLEAN`。 +- 三文件 Bun 定向测试 17/17：realtime 5、options 8、collect 4；其中 realtime 与 options 两文件为 13/13。 +- 独立 syntax-only 检查 11/11；该计数未与 Bun test 合并。 +- PR preflight 为 `MERGEABLE/CLEAN`。 - 未引入 HTTP history、projector、snapshot、polling、gap-fill、fallback、临时 script 或 URL/IP 运行面选择。 - 按任务边界未部署、未合并。