From 2fdadce835b8d934bebca3704bcc2c069341ddf8 Mon Sep 17 00:00:00 2001 From: Codex Date: Mon, 13 Jul 2026 07:10:53 +0200 Subject: [PATCH 01/14] =?UTF-8?q?docs:=20=E6=96=B0=E5=A2=9E=E6=99=BA?= =?UTF-8?q?=E5=AA=92=E5=B7=A5=E5=8E=82=20OA=20=E8=A7=84=E6=A0=BC=E4=BD=93?= =?UTF-8?q?=E7=B3=BB?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../specs/PJ2026-02-smart-media-factory.md | 493 ++++++++++++++++++ .../specs/PJ2026-0201-information-sources.md | 338 ++++++++++++ .../specs/PJ2026-0202-editorial-planning.md | 346 ++++++++++++ .../specs/PJ2026-0203-visual-production.md | 335 ++++++++++++ .../PJ2026-0204-audio-video-production.md | 308 +++++++++++ .../specs/PJ2026-0205-pipeline-platform.md | 346 ++++++++++++ .../PJ2026-0206-publishing-operations.md | 326 ++++++++++++ .../specs/PJ2026-0207-platform-delivery.md | 451 ++++++++++++++++ 8 files changed, 2943 insertions(+) create mode 100644 project-management/PJ2026-02/specs/PJ2026-02-smart-media-factory.md create mode 100644 project-management/PJ2026-02/specs/PJ2026-0201-information-sources.md create mode 100644 project-management/PJ2026-02/specs/PJ2026-0202-editorial-planning.md create mode 100644 project-management/PJ2026-02/specs/PJ2026-0203-visual-production.md create mode 100644 project-management/PJ2026-02/specs/PJ2026-0204-audio-video-production.md create mode 100644 project-management/PJ2026-02/specs/PJ2026-0205-pipeline-platform.md create mode 100644 project-management/PJ2026-02/specs/PJ2026-0206-publishing-operations.md create mode 100644 project-management/PJ2026-02/specs/PJ2026-0207-platform-delivery.md diff --git a/project-management/PJ2026-02/specs/PJ2026-02-smart-media-factory.md b/project-management/PJ2026-02/specs/PJ2026-02-smart-media-factory.md new file mode 100644 index 00000000..5ec49738 --- /dev/null +++ b/project-management/PJ2026-02/specs/PJ2026-02-smart-media-factory.md @@ -0,0 +1,493 @@ +# PJ2026-02 智媒工厂总规格 + +## 修改历史 + +| 版本 | 对应 commit id | 更新日期 | 变更说明 | +| --- | --- | --- | --- | +| v0.1 | 待提交 | 2026-07-13 | 基于 `/root/selfmedia` 已跑通能力建立智媒工厂 L0 总规格和七个 L1 能力域。 | + +当前正文仍在规格治理草稿中；未定稿前不新增版本号，不为单次编辑追加 `待提交` 版本。 + +## 正文 + +## PJ2026-02 智媒工厂总项目需求规格 + +## 1. 文档控制 + +| 字段 | 内容 | +| --- | --- | +| 编号 | PJ2026-02 | +| 短名 | 智媒工厂 | +| 层级 | L0 总项目 | +| 状态 | 已生效 | +| 实现引用版本 | draft-2026-07-13-p0 | +| 需求规格模板 | [ISO/IEC/IEEE 29148 需求规格模板](../../templates/iso-iec-ieee-29148-requirements-spec-template.md) | +| 规格治理索引 | 本文第 7 章 | + +本文采用 ISO/IEC/IEEE 29148 需求规格模板的项目裁剪版。正文只保留稳定使命、范围、术语、系统边界、能力域分工、原子需求和必要过程控制。 + +## 2. 目的和范围 + +### 2.1 项目使命 + +项目使命包括： + +- 面向科技资讯创作者和技术团队； +- 把资讯发现、事实核验、编辑策划、动态图文、配音、字幕、成片和发布材料收敛为本地优先的生产链； +- 当前以嵌入式与 AI 交叉、纯嵌入式和 AI 为首要内容方向； +- 长期成为可配置、可单步调试的技术内容生产基础设施； +- 使每一期节目都能从来源证据追溯到口播、画面、成片和评论区引用。 + +### 2.2 当前能力基线 + +- 资讯收集： + - 通过 YAML 登记 RSS、Atom、Hacker News API 和人工来源； + - 保留一手来源、发现来源、信任层级、分类、版权提示和原始响应； + - 优先处理嵌入式与 AI、纯嵌入式、AI，并识别优惠、样片和免费额度等实用信息。 +- 编辑策划： + - 支持授权 LLM 和完全离线模板两种文稿路径； + - 生成固定开场、卡片大纲、分条口播、术语补充、总结、参考来源和评论区文案； + - 对状态、日期、数字、媒体归因、预印本和专业对象说明实施质量门。 +- 视觉制作： + - 采集来源网页或 GitHub 页面为本地视觉工件； + - 为生成内容输出暖色、圆角、可配置的 HTML 动效、PNG 预览和 PPTX； + - 以确定性时间驱动页面滚动、章节进度和阶段导航。 +- 音频成片： + - 使用本地开源中文 TTS 生成可缓存配音； + - 由 YAML 控制音色、语速、停顿、响度、字幕和编码； + - 生成 WAV、时间线、SRT、ASS、MP4、封面和自动检查报告。 +- 流水线平台： + - TypeScript CLI 支持单步、区间和完整流水线； + - 长任务采用异步作业，提供状态、进度、日志、结果、重试和取消； + - 同源 Web 编辑台展示期次、作业、日志、费用和已登记媒体工件； + - 当前工件清单是步骤输出索引，不是经过内容版本兼容性校验的完整发布包。 +- 发布运营： + - 生成与最终成片真实章节起点一致的评论区时间轴； + - 输出参考来源、公开目录价成本估算、技术验收结果和人工复核提示； + - 通过本地或公网 IP 提供候选成片和工件的受控预览。 + +### 2.3 当前差距与目标边界 + +- 当前没有持久化的人工批准状态、批准人、批准时间、退回记录或发布版本台账。 +- 当前没有把全部媒体、来源、费用和检查工件按内容版本兼容性组合成发布包。 +- 当前没有 Web 内置 Codex 终端、版本化后端容器、会话 PVC 或重启自动恢复。 +- 当前没有 YAML-first Kubernetes 部署、隔离 Secret 投影或公网 IP 部署后验收。 +- 当前没有 GitHub、mirror、PaC、Tekton、GitOps 和 Argo 自动交付链。 +- 当前 Web 编辑台仍需与生成内容主题解耦，目标控制台采用工业风和紧凑高信息密度布局。 +- 第 6 章使用“应”描述目标需求；目标要求不代表当前实现或当前上线状态。 + +### 2.4 长期愿景 + +- 形成多栏目、多主题、多语言和多平台复用的内容生产模型，同时保持每个栏目独立的来源、策划、视觉、声音和发布策略。 +- 让人工编辑能够在任何阶段插入、覆盖、重跑或批准结果，不被端到端自动化锁死。 +- 建立来源事实、内容决策、媒体工件和发布版本之间的可追溯关系，支持复核、纠错和再利用。 +- 持续扩展本地开源语音、视觉和媒体能力，使表现力提升不依赖外部付费生成服务。 +- 将内容效果、受众反馈和纠错结果回流到选题、提示词和栏目规则，但不以平台流量替代事实质量和版权边界。 +- 通过 Web 内置 Codex、隔离会话和自动交付链，把单机工具演进为可受控部署和恢复的平台。 + +### 2.5 范围内 + +- 科技资讯源的发现、采集、归一化、去重、分类、排序、证据保存和人工补录。 +- 从来源证据到结构化口播稿、镜头意图、引用清单和发布文案的编辑过程。 +- 来源页面视觉、HTML 动效、演示文稿、配音、字幕、封面和成片的本地生产。 +- YAML-first 配置、TypeScript CLI、异步作业、工件契约、日志、费用、检查和 Web 可视控制面。 +- 发布候选的技术验收、人工复核入口、时间轴、来源目录、AI 参与披露和版本留存。 +- Web 内置 Codex、隔离凭据、会话恢复、YAML-first Kubernetes 和自动交付目标能力。 +- 生成内容使用暖色圆角主题，Web 工厂、控制台和内置终端使用工业风、直角或极小倒角主题。 + +### 2.6 范围外 + +- 自动绕过登录、付费墙、访问控制、站点条款或版权限制抓取第三方内容。 +- 复制既有自媒体的原文、配音、字幕、封面、音乐、品牌视觉或受保护媒体资产。 +- 自动替代编辑者完成事实、法律、版权、平台规则和最终发布责任判断。 +- 以某个仓库、框架、CLI、模型或服务名称作为长期能力域；实现可以替换，需求边界保持稳定。 +- 社交平台账号托管、自动互动、广告投放、商业结算和未经审批的自动公开发布。 +- 向 Web 终端或 Codex Agent 提供 Kubernetes、kubeconfig、ServiceAccount 或宿主机控制权限。 +- 单次抓取故障、一次渲染、一个作业日志或阶段性样片流水账；这些属于执行证据，不属于 L0 稳定需求。 + +## 3. 术语表 + +| 术语 | 定义 | +| --- | --- | +| 智媒工厂 | 从可信科技资讯到可发布媒体候选及配套发布材料的本地优先生产系统。 | +| 期次 | 一次独立内容生产单元，具有稳定标识、标题、期数、报道日期、输入、工件和作业记录。 | +| 来源项 | 经过归一化的资讯记录，保留标题、链接、日期、摘要、来源身份、角色、信任层级和分类。 | +| 一手来源 | 由产品、项目、公司、机构或作者团队直接发布的公告、发布说明、论文或仓库记录。 | +| 发现来源 | 用于发现线索和提供编辑背景的媒体、社区或聚合入口，不替代可获得的一手事实。 | +| 来源闭环 | 口播中的可核验事实能够解析到期次来源项，并在引用与发布材料中保持同一归因。 | +| 内容单元 | 开场、大纲、单条资讯、总结或参考来源等可独立配音和映射场景的结构化文稿单元。 | +| 场景 | 具有独立画面、时间范围、章节标题和动效状态的视频组成单元。 | +| 工件 | 流水线生成并登记的来源、文稿、视觉、音频、字幕、视频、报告或发布文本文件。 | +| 发布候选 | 已通过自动技术检查但仍需人工事实、版权、听感和平台规则复核的成片及配套材料。 | +| 发布包 | 目标能力中与同一期次、内容版本和工件哈希绑定的候选媒体、引用、费用、检查和批准材料集合。 | +| 平台交付 | 将应用、内置 Codex、隔离会话和凭据通过受控 Kubernetes 与自动链交付到公网原入口的能力域。 | +| owning YAML | 对可调业务事实具有唯一所有权的 YAML 配置；代码只负责校验、调度和渲染。 | +| 实现引用版本 | 后续源码变更用于声明其遵循哪一版规格语义的稳定标识。 | + +## 4. 系统边界和接口 + +本规格把智媒工厂作为完整内容生产系统看待。本章只描述外部输入、输出和责任边界，不把内部项目治理当作产品能力。 + +| 边界项 | 内容 | +| --- | --- | +| 外部使用者 | 技术内容编辑、栏目运营者、媒体制作者、审核者和受控自动化任务。 | +| 外部输入 | 来源配置、公开 Feed/API/网页、人工来源、期次元数据、栏目规则、授权 LLM 凭据、本地模型、GitHub 提交、部署配置和发布约束。 | +| 受控资源 | 期次工件、作业记录、日志、用量、后端容器、Codex 会话、会话 PVC、GitOps 目标、K8s 工作负载和验收报告。 | +| 外部输出 | MP4、WAV、SRT/ASS、PPTX、HTML/PNG、封面、引用目录、评论区时间轴、成本估算、检查报告和可视化状态。 | +| 用户接口 | TypeScript CLI、同源 HTTP API、Web 编辑台和可直接复核的文件工件。 | +| 网络接口 | 公开 RSS/Atom/API/网页、可选 OpenAI 兼容 LLM 端点、GitHub 交付入口和受控公网 IPv4。 | +| 系统边界 | 系统负责生产、追溯、技术检查与目标应用交付；编辑者负责最终事实、版权、表达、平台合规和发布决定。 | + +## 5. 内部模块分工与规格索引 + +L1 按完成标准划分为七个稳定能力域。仓库、框架、模型、命令和运行端口只是实现引用，不构成 L1。 + +| 编号 | 内部模块 | 规格文档 | 主责边界 | 上游依赖 | 下游支撑 | +| --- | --- | --- | --- | --- | --- | +| PJ2026-0201 | 资讯源 | [PJ2026-0201 资讯源](PJ2026-0201-information-sources.md) | 来源发现、采集、证据、归一化、分类、排序和版权边界 | 公开来源、编辑者配置 | [编辑策划](PJ2026-0202-editorial-planning.md)、[视觉制作](PJ2026-0203-visual-production.md)、[发布运营](PJ2026-0206-publishing-operations.md) | +| PJ2026-0202 | 编辑策划 | [PJ2026-0202 编辑策划](PJ2026-0202-editorial-planning.md) | 选题、核验、叙事、口播、来源引用和栏目结构 | [资讯源](PJ2026-0201-information-sources.md) | [视觉制作](PJ2026-0203-visual-production.md)、[音频成片](PJ2026-0204-audio-video-production.md)、[发布运营](PJ2026-0206-publishing-operations.md) | +| PJ2026-0203 | 视觉制作 | [PJ2026-0203 视觉制作](PJ2026-0203-visual-production.md) | 来源视觉、场景设计、HTML 动效、演示文稿、进度导航和封面画面 | [资讯源](PJ2026-0201-information-sources.md)、[编辑策划](PJ2026-0202-editorial-planning.md) | [音频成片](PJ2026-0204-audio-video-production.md)、[发布运营](PJ2026-0206-publishing-operations.md) | +| PJ2026-0204 | 音频成片 | [PJ2026-0204 音频成片](PJ2026-0204-audio-video-production.md) | 本地配音、时间线、字幕、音视频合成和媒体质量 | [编辑策划](PJ2026-0202-editorial-planning.md)、[视觉制作](PJ2026-0203-visual-production.md) | [发布运营](PJ2026-0206-publishing-operations.md) | +| PJ2026-0205 | 流水线平台 | [PJ2026-0205 流水线平台](PJ2026-0205-pipeline-platform.md) | YAML 配置、CLI 调度、异步作业、可观测性、工件契约、用量计费和 Web 控制面 | 全部生产能力域的步骤契约 | 全部 L1 的可执行、可调试和可观察运行 | +| PJ2026-0206 | 发布运营 | [PJ2026-0206 发布运营](PJ2026-0206-publishing-operations.md) | 发布候选、引用与时间轴、人工复核、版本归档、预览和反馈回流 | [资讯源](PJ2026-0201-information-sources.md)、[编辑策划](PJ2026-0202-editorial-planning.md)、[视觉制作](PJ2026-0203-visual-production.md)、[音频成片](PJ2026-0204-audio-video-production.md)、[流水线平台](PJ2026-0205-pipeline-platform.md) | 内容发布者和后续栏目改进 | +| PJ2026-0207 | 平台交付 | [PJ2026-0207 平台交付](PJ2026-0207-platform-delivery.md) | Web Codex、隔离会话与凭据、K8s、公网 IP、自动交付和运行验收 | [流水线平台](PJ2026-0205-pipeline-platform.md)、[发布运营](PJ2026-0206-publishing-operations.md) | 受控运行面、编辑者和交付自动化 | + +### 5.1 目标架构图 + +```mermaid +flowchart LR + subgraph External[外部输入] + Feeds[公开 Feed / API / 网页] + Manual[人工来源与编辑决定] + Models[授权 LLM 与本地模型] + Git[GitHub / mirror / GitOps] + end + subgraph Factory[智媒工厂] + Sources[资讯源] + Editorial[编辑策划] + Visual[视觉制作] + Audio[音频成片] + Platform[流水线平台] + Publish[发布运营] + Delivery[平台交付] + end + subgraph Outputs[外部输出] + Media[视频 / 音频 / 字幕 / PPT] + Release[时间轴 / 来源 / 披露 / 验收] + Console[CLI / API / Web 编辑台] + Terminal[公网 IP / 内置 Codex] + end + Feeds --> Sources + Manual --> Sources + Manual --> Editorial + Models --> Editorial + Sources --> Editorial + Sources --> Visual + Editorial --> Visual + Editorial --> Audio + Visual --> Audio + Audio --> Publish + Sources --> Publish + Editorial --> Publish + Platform -.调度与观测.-> Sources + Platform -.调度与观测.-> Editorial + Platform -.调度与观测.-> Visual + Platform -.调度与观测.-> Audio + Platform -.调度与观测.-> Publish + Audio --> Media + Publish --> Release + Platform --> Console + Git --> Delivery + Delivery -.承载与验收.-> Platform + Delivery -.承载与验收.-> Publish + Delivery --> Terminal +``` + +目标架构保持职责分离： + +- 各生产能力域定义自身结果是否正确； +- 流水线平台负责按同一契约调度、记录和展示； +- 流水线平台不重新定义选题、视觉、音频或发布事实。 +- 平台交付负责应用运行面和自动交付，不把 Kubernetes 权限交给内容 Agent。 + +### 5.2 目标数据流图 + +```mermaid +flowchart TD + GitSource[GitHub 提交] --> DeliveryChain[mirror / PaC / Tekton / GitOps / Argo] + DeliveryChain --> Runtime[Kubernetes 应用运行面] + Runtime --> Config + Config[owning YAML 与期次元数据] --> Collect[采集与人工补录] + Network[公开来源] --> Collect + Collect --> SourceManifest[来源清单与原始快照] + SourceManifest --> Select[分类、排序、选题与事实核验] + Select --> Script[结构化文稿与来源引用] + Script --> SourceVisuals[来源页面视觉] + Script --> Voice[分段配音] + SourceManifest --> SourceVisuals + SourceVisuals --> Scenes[HTML 场景 / PNG / PPTX] + Voice --> AudioTimeline[WAV 与句级时间线] + Script --> AudioTimeline + AudioTimeline --> Subtitles[SRT / ASS] + Scenes --> Render[确定性画面渲染] + Subtitles --> Render + AudioTimeline --> Render + Render --> Candidate[MP4 / 封面 / 场景时间线] + Candidate --> Inspect[自动技术检查] + SourceManifest --> Inspect + Script --> Inspect + Inspect --> PublishPack[目标发布包组合] + Candidate --> Comment[真实时间轴与评论区来源] + Comment --> PublishPack + PublishPack --> PublicEntry[公网 IP 预览与人工复核] + Runtime --> PublicEntry +``` + +数据流中的每个正式步骤必须读取已登记上游工件并输出新工件。上游缺失或不合法时应明确失败，不得使用隐藏默认值、旧期次文件或伪造输入补成成功。 + +图中的自动交付、目标发布包和持久化人工复核属于目标能力。当前只具备主机运行、步骤工件、评论区材料和技术检查。 + +### 5.3 关键时序图 + +```mermaid +sequenceDiagram + participant G as GitHub / GitOps + participant D as 自动交付链 + participant K as Kubernetes 运行面 + participant E as 编辑者 + participant C as CLI / Web + participant P as 流水线平台 + participant S as 生产步骤 + participant A as 期次工件 + participant R as 发布复核 + G->>D: 提交受审查变更 + D->>K: 构建、协调并执行公网验收 + K-->>D: 返回版本、健康与隔离证据 + E->>C: 从公网 IP 打开 Web、CLI 或内置终端 + E->>C: 创建期次并选择单步或完整流水线 + C->>P: 提交异步作业 + P-->>C: 返回 jobId + loop 每个选中步骤 + P->>S: 传入期次、配置和取消信号 + S->>A: 读取已登记上游工件 + S->>A: 原子写入结果和清单 + S-->>P: 更新进度、日志和步骤结果 + C->>P: 查询状态、日志或结果 + P-->>C: 返回同一作业事实 + end + P-->>C: 作业终态与工件索引 + E->>R: 复核事实、版权、听感和平台规则 + R->>A: 目标能力记录批准、退回或重跑决定 + R-->>E: 发布候选或明确阻塞项 +``` + +长任务必须异步执行。CLI 和 Web 提交后立即获得作业标识，编辑者通过状态、日志和工件观察进展；重试创建新作业并保留旧证据，不能覆盖失败历史。 + +图中的自动交付、内置终端和批准记录属于目标时序；当前实现从创建期次到技术检查和候选预览的部分已经跑通。 + +### 5.4 当前实现映射 + +| 能力 | 当前实现引用 | 当前事实 | +| --- | --- | --- | +| 配置入口 | `/root/selfmedia/config/selfmedia.yaml` 及其 `configRefs` | YAML 是可调事实来源，配置缺失或不合法时失败。 | +| CLI 与作业 | `/root/selfmedia/scripts/selfmedia-cli.ts`、`scripts/src/jobs.ts`、`scripts/src/runner.ts` | 支持单步、完整流程、状态、日志、结果、重试和取消。 | +| 内容步骤 | `/root/selfmedia/scripts/src/steps/` | 已实现采集、文稿、视觉、幻灯、配音、字幕、渲染和检查。 | +| Web 编辑台 | `/root/selfmedia/scripts/src/server-app.ts`、`web/src/` | 同源展示期次、作业、用量和媒体工件。 | +| 期次事实 | `/root/selfmedia/data/editions//` | 来源、文稿、媒体、时间线、用量和检查结果按期次归档。 | +| 已验证样片 | `embedded-ai-weekly-2026-07-13-r3` | 已形成九场景、五条资讯、网页滚动、配音字幕、成片、评论区和十项技术检查。 | +| 发布状态 | 当前无持久化批准与发布台账 | 已生成候选和复核材料，但没有批准人、批准时间、退回或已发布状态。 | +| 发布包 | 当前无版本兼容性发布包 | 存在步骤工件和输出清单，但尚未按内容版本校验并组合全部发布材料。 | +| 平台交付 | 当前无正式 Kubernetes 运行面 | Web Codex、隔离会话、YAML-first K8s 和自动交付均为 [PJ2026-0207 平台交付](PJ2026-0207-platform-delivery.md) 目标。 | + +## 6. 全局原子需求 + +### 6.1 SMF-L0-REQ-001 可信资讯底座 + +| 编号 | 短名 | 主责模块 | 关联模块 | +| --- | --- | --- | --- | +| SMF-L0-REQ-001 | 可信资讯底座 | [PJ2026-0201 资讯源](PJ2026-0201-information-sources.md) | [编辑策划](PJ2026-0202-editorial-planning.md)、[视觉制作](PJ2026-0203-visual-production.md)、[发布运营](PJ2026-0206-publishing-operations.md) | + +智媒工厂应建立可追溯的科技资讯底座，使每条进入节目候选池的事实都有来源身份、时间、角色、信任层级、分类、证据摘要和原始链接。 + +职责边界如下： + +- 资讯源负责发现、归一化、去重、分类和证据边界； +- 编辑策划决定是否采用以及如何表达； +- 视觉制作只在允许范围内生成来源视觉； +- 发布运营保留最终引用； +- 任何下游模块不得把发现媒体改写为一手发布者。 + +完成标准是：从任一成片资讯可反向定位到期次来源项和公开链接；一手、媒体、社区、预印本和优惠信息的状态边界不会在下游丢失。 + +### 6.2 SMF-L0-REQ-002 有效编辑叙事 + +| 编号 | 短名 | 主责模块 | 关联模块 | +| --- | --- | --- | --- | +| SMF-L0-REQ-002 | 有效编辑叙事 | [PJ2026-0202 编辑策划](PJ2026-0202-editorial-planning.md) | [资讯源](PJ2026-0201-information-sources.md)、[视觉制作](PJ2026-0203-visual-production.md)、[音频成片](PJ2026-0204-audio-video-production.md)、[发布运营](PJ2026-0206-publishing-operations.md) | + +智媒工厂应把来源证据转化为信息密度高、口语自然、对技术受众有实际价值的栏目文稿，并保持开场、概述、故事、总结和参考来源的稳定结构。 + +编辑策划负责选题、事实边界、术语补充、媒体归因、状态限定和叙事节奏。下游模块只能消费结构化内容，不能以画面或配音修饰掩盖证据不足和文稿空泛。 + +完成标准是：每条资讯说清“发生了什么、谁发布、对受众有什么影响、目前不能据此推出什么”，并能通过来源引用复核关键事实。 + +### 6.3 SMF-L0-REQ-003 动态视觉表达 + +| 编号 | 短名 | 主责模块 | 关联模块 | +| --- | --- | --- | --- | +| SMF-L0-REQ-003 | 动态视觉表达 | [PJ2026-0203 视觉制作](PJ2026-0203-visual-production.md) | [资讯源](PJ2026-0201-information-sources.md)、[编辑策划](PJ2026-0202-editorial-planning.md)、[音频成片](PJ2026-0204-audio-video-production.md) | + +智媒工厂应提供可配置、可追溯、适合科技资讯的动态视觉表达，使信息层级、来源画面、章节状态和叙事节奏在视频中一致呈现。 + +本需求只约束生成的 HTML、PPT、PNG、封面和视频画面，不约束 Web 工厂与控制台外观。 + +视觉制作负责： + +- HTML 场景、来源页面滚动、演示文稿、预览图、章节进度和主题系统； +- 提供温暖、阳光、圆润且易读的默认视觉； +- 允许替换主题实现； +- 保持字幕安全区和内容可辨识性，不让装饰牺牲信息表达。 + +完成标准是：同一场景在指定绝对时间可确定性重现，画面来源可以追溯，九宫格式堆字、未经授权搬运和依赖在线 CDN 的生产场景不被接受。 + +### 6.4 SMF-L0-REQ-004 自然音频成片 + +| 编号 | 短名 | 主责模块 | 关联模块 | +| --- | --- | --- | --- | +| SMF-L0-REQ-004 | 自然音频成片 | [PJ2026-0204 音频成片](PJ2026-0204-audio-video-production.md) | [编辑策划](PJ2026-0202-editorial-planning.md)、[视觉制作](PJ2026-0203-visual-production.md)、[发布运营](PJ2026-0206-publishing-operations.md) | + +智媒工厂应使用本地开源媒体能力把结构化文稿和场景合成为自然、同步、可重新配置的配音字幕成片。 + +音频成片负责： + +- 语音规范化、分段推理、缓存、语速、停顿和响度； +- 字幕切分、场景时间线、编码和媒体检查； +- 在改变语速或声音参数时只重跑受影响步骤，不要求重新采集和写稿。 + +完成标准是：音频、字幕、场景和视频时间线在允许误差内一致，媒体流、分辨率、响度、峰值和可解码性通过技术检查，人工仍能识别并退回读音或表达问题。 + +### 6.5 SMF-L0-REQ-005 可调试生产平台 + +| 编号 | 短名 | 主责模块 | 关联模块 | +| --- | --- | --- | --- | +| SMF-L0-REQ-005 | 可调试生产平台 | [PJ2026-0205 流水线平台](PJ2026-0205-pipeline-platform.md) | [资讯源](PJ2026-0201-information-sources.md)、[编辑策划](PJ2026-0202-editorial-planning.md)、[视觉制作](PJ2026-0203-visual-production.md)、[音频成片](PJ2026-0204-audio-video-production.md)、[发布运营](PJ2026-0206-publishing-operations.md) | + +智媒工厂应提供 YAML-first、CLI-first 的生产平台，使每个正式步骤可独立运行、观察、重试和复现，并由同一实现组合为完整流水线和 Web 可视流程。 + +流水线平台负责： + +- 配置校验、步骤 DAG、异步作业、进度、日志、结果和取消； +- 工件索引、服务接口和可视控制面； +- 统计所有 LLM 尝试的 Token 与公开目录价估算； +- 不泄露 API key、凭据文件内容或敏感请求材料。 + +Web 工厂和控制台应采用工业风、紧凑高信息密度、直角或极小倒角的视觉语言，不复用生成内容的暖色大圆角主题。 + +完成标准是：编辑者能通过 CLI 单步定位问题，通过 Web 查看同一作业事实；失败、取消、重试和费用可见；无隐藏默认配置或付费媒体生成服务依赖。 + +### 6.6 SMF-L0-REQ-006 受控发布闭环 + +| 编号 | 短名 | 主责模块 | 关联模块 | +| --- | --- | --- | --- | +| SMF-L0-REQ-006 | 受控发布闭环 | [PJ2026-0206 发布运营](PJ2026-0206-publishing-operations.md) | [资讯源](PJ2026-0201-information-sources.md)、[编辑策划](PJ2026-0202-editorial-planning.md)、[视觉制作](PJ2026-0203-visual-production.md)、[音频成片](PJ2026-0204-audio-video-production.md)、[流水线平台](PJ2026-0205-pipeline-platform.md) | + +智媒工厂应把技术通过的成片组织为可人工批准、可退回和可追溯的发布候选，并生成与最终视频一致的时间轴、来源、AI 参与披露和复核材料。 + +发布运营负责： + +- 最终工件组合、评论区文本、技术检查结果解释和人工清单； +- 版本归档和预览入口； +- 区分自动检查通过与事实、版权、法律或平台合规批准； +- 阻止未完成必要人工复核的候选被标记为已发布。 + +完成标准是：发布者无需手工重新计算章节起点或整理来源链接，且可以明确看到批准前仍需承担的检查责任和任何阻塞项。 + +### 6.7 SMF-L0-REQ-007 隔离自动交付 + +| 编号 | 短名 | 主责模块 | 关联模块 | +| --- | --- | --- | --- | +| SMF-L0-REQ-007 | 隔离自动交付 | [PJ2026-0207 平台交付](PJ2026-0207-platform-delivery.md) | [流水线平台](PJ2026-0205-pipeline-platform.md)、[发布运营](PJ2026-0206-publishing-operations.md) | + +智媒工厂应提供可复现、最小权限和可自动验收的平台交付，使编辑者能通过公网 IP 的 Web 内置 Codex 终端进入与后端版本一致的源码、CLI 和受限 Agent 工具环境。 + +平台交付负责： + +- 使用 PVC 保存 `~/.codex/sessions` 与 supervisor 恢复元数据，不承诺持久化源码工作区； +- 通过 YAML-first `sourceRef` 分离投影后端 Token 与 Codex 模型配置； +- 确保终端没有 ServiceAccount token、kubeconfig、Kubernetes API 或宿主机权限； +- 以 YAML-first Kubernetes 暴露公网 IPv4 原入口； +- 建立 GitHub、mirror、PaC、Tekton、GitOps、Argo 和运行验收自动链。 + +完成标准包括： + +- 通过公网 IP 验证 Web、API 和内置终端； +- 验证版本一致性、会话重启恢复和权限负面边界； +- 从运行版本追溯到提交、镜像摘要、GitOps revision 和部署后验收结果。 + +## 7. 过程控制 + +### 7.1 SPEC-first 和层级规则 + +- 改变项目使命、L1 能力域、全局数据流或全局验收口径时，先更新本 L0 规格。 +- 改变某个能力域的稳定输入、输出、职责或完成标准时，先更新对应 L1 规格。 +- 单次作业、一个缺陷、一次样片、一次部署或一次文档收口不进入 L0/L1 需求正文。 +- 新增能力先按完成标准确定唯一主责 L1；跨域关系通过同目录相对链接表达，不复制同一需求。 + +### 7.2 实现引用和追溯 + +- 本规格当前实现引用版本为 `draft-2026-07-13-p0`。 +- 后续新增或修改的源码文件应在可承载注释的文件头标记： + - `SPEC: PJ2026-02 智媒工厂 draft-2026-07-13-p0`； + - 同时标记主责 L1 的编号、短名和实现引用版本。 +- 自动生成文件、第三方文件、纯配置、锁文件和二进制工件可以不加文件头： + - 其生成器、校验器或 owning YAML 入口必须能追溯到本规格； + - 期次工件必须能追溯到作业记录、步骤结果和配置引用摘要。 + +### 7.3 验收和发布控制 + +- 能力验收优先使用原入口： + - CLI 能力用 TypeScript CLI； + - Web 能力用同源 Web/API； + - 媒体能力用实际工件、播放器和媒体探测； + - 发布材料用最终场景时间线和来源清单复核。 + - 平台交付用 YAML 选中的公网 IPv4、Web 内置终端、会话重启恢复和权限负面检查。 +- 自动技术检查只决定“是否具备人工发布复核条件”，不决定“是否可以公开发布”。 +- 发布前至少复核： + - 事实、日期、数字和状态； + - 来源角色和归因； + - 版权、授权和 AI 参与披露； + - 读音、字幕、画面和时间轴； + - 平台规则、地区限制和活动期限。 + +### 7.4 配置、Secret 和成本控制 + +- 可调业务事实由 owning YAML 管理，代码不保留第二套静默默认值。 +- LLM 凭据只在运行时从用户授权位置读取： + - 日志、工件、Web 和费用摘要不得输出 API key； + - 模板模式必须可以在不读取 LLM 凭据时运行。 +- 目标运行面的凭据必须分离投影： + - 后端 API Token 使用 YAML-first `sourceRef` 投影到 `~/.env/TOKEN`； + - Codex 模型 API key 和配置从受控 `~/.codex/*.pika` 源路径投影到运行时 Codex home； + - 两类凭据使用不同 `sourceRef` 和 `targetKey`，状态只披露 presence 与 fingerprint。 +- 所有 LLM 请求尝试都应记录 Token 用量状态和费用估算状态： + - 成功、被质量门拒绝和请求失败均不能从统计中静默消失； + - 公开目录价估算必须标明来源、抓取时间、币种和“非实际账单”边界； + - 未知模型或缺失用量不得按零费用处理。 + +### 7.5 偏离判定 + +以下情况属于偏离风险，应停止扩大实现并回到对应规格重新归类： + +- 只增加来源数量，却没有改善证据、分类、选题或发布价值。 +- 只追求视觉特效、语音模型或编码参数，却没有可感知的内容完成标准。 +- 以 Web 页面、CLI 命令、仓库或模型名称替代能力域定义。 +- 绕过单步工件契约，通过临时脚本或旧期次文件拼出成功结果。 +- 自动公开发布，但没有人工批准、版权检查、平台规则检查和回退路径。 +- 通过手工主机进程、SSH、临时端口或直接 Kubernetes 权限替代目标平台交付链。 +- 增加监控或统计，却没有下游编辑、质量或成本决策使用新信号。 + +### 7.6 规格回写 + +- 稳定需求、边界或验收变化回写对应 L1；影响多个 L1 时同时更新本 L0 索引和全局数据流。 +- 实现细节、作业编号、日志、截图和长证据保留在项目运行工件，不写入稳定规格正文。 +- 规格间只使用本目录内的相对链接；不在规格正文保存 GitHub issue 或 PR 链接。 diff --git a/project-management/PJ2026-02/specs/PJ2026-0201-information-sources.md b/project-management/PJ2026-02/specs/PJ2026-0201-information-sources.md new file mode 100644 index 00000000..d71b1f4c --- /dev/null +++ b/project-management/PJ2026-02/specs/PJ2026-0201-information-sources.md @@ -0,0 +1,338 @@ +# PJ2026-0201 资讯源需求规格 + +## 修改历史 + +| 版本 | 对应 commit id | 更新日期 | 变更说明 | +| --- | --- | --- | --- | +| v0.1 | 待提交 | 2026-07-13 | 基于现有公开 Feed、Hacker News API、人工来源和期次来源清单建立资讯源规格。 | + +当前正文仍在规格治理草稿中；未定稿前不新增版本号，不为单次编辑追加 `待提交` 版本。 + +## 正文 + +## PJ2026-0201 资讯源需求规格 + +## 1. 文档控制 + +| 字段 | 内容 | +| --- | --- | +| 编号 | PJ2026-0201 | +| 短名 | 资讯源 | +| 层级 | L1 方向 | +| 状态 | 已生效 | +| 实现引用版本 | draft-2026-07-13-p0 | +| 需求规格模板 | [ISO/IEC/IEEE 29148 需求规格模板](../../templates/iso-iec-ieee-29148-requirements-spec-template.md) | +| 上级规格 | [PJ2026-02 智媒工厂总规格](PJ2026-02-smart-media-factory.md) | +| 规格治理索引 | [PJ2026-02 智媒工厂总规格](PJ2026-02-smart-media-factory.md) 第 7 章 | + +本文采用 ISO/IEC/IEEE 29148 需求规格模板的项目裁剪版。正文只保留资讯发现、采集、证据、分类、排序和使用边界的稳定需求。 + +## 2. 目的和范围 + +### 2.1 目的 + +资讯源的目标是： + +- 提供可信、及时、可追溯的候选事实； +- 主动发现嵌入式与 AI 交叉、纯嵌入式、AI 和实用优惠信息； +- 明确区分发布者原文、媒体报道、社区线索和预印本； +- 避免下游把线索误当结论或把媒体改写为一手来源。 + +### 2.2 范围内 + +- 来源目录、启停策略、抓取类型、优先级、标签、信任层级、来源角色和视觉使用提示。 +- RSS、Atom、公开 API、GitHub Releases/Changelog 和人工来源的采集与归一化。 +- 原始响应留存、规范链接、去重、分类、排序、摘要证据和抓取失败记录。 +- 嵌入式与 AI、纯嵌入式、AI 的优先级，以及硬件开发 Harness 和 AI Agent/Coding Harness 分类。 +- 官方优惠、折扣、优惠券、样片计划、开发板赠送和免费额度的条件化表达。 +- 新闻媒体、Hacker News、大学论文、官方公告和开源项目发布记录的归因边界。 +- 供下游视觉制作使用的页面策略和版权提示，不负责实际画面制作。 + +### 2.3 范围外 + +- 选题取舍、叙事角度和最终口播表达，归 [编辑策划](PJ2026-0202-editorial-planning.md)。 +- 网页截图、滚动动效、演示文稿和封面，归 [视觉制作](PJ2026-0203-visual-production.md)。 +- 自动绕过登录、验证码、付费墙、反爬机制或站点条款获取正文和媒体。 +- 保存或再发布第三方全文、原视频、原音频、大量图片或不必要的个人信息。 +- 用搜索排名、社交热度或单一媒体观点替代事实证据和编辑判断。 + +## 3. 术语表 + +| 术语 | 定义 | +| --- | --- | +| 来源目录 | 由 owning YAML 管理的允许来源、抓取方式、角色、优先级、标签和风险提示集合。 | +| 来源角色 | `primary` 表示一手事实入口，`discovery` 表示发现线索或编辑背景入口。 | +| 信任层级 | 对第一方、混合第一方、预印本、编辑媒体和社区发现等来源性质的显式标记。 | +| 规范链接 | 去除可安全忽略的跟踪参数并保持资源身份稳定的链接，用于去重和追溯。 | +| 证据摘要 | 从允许获取的元数据和短摘录中保留的事实线索，不等价于复制原文。 | +| 人工来源 | 因无稳定自动入口或需人工核验而写入期次输入的结构化来源项。 | +| 预印本 | 尚不能默认视为已经同行评审的论文发布形态。 | +| 实用优惠 | 具有明确发布者、期限、地区、资格和条件的折扣、样片、赠送或免费额度信息。 | +| 状态边界 | beta、预览、分阶段开放、预印本、临时调整、地区限定等不能在下游丢失的事实限制。 | + +## 4. 系统边界和接口 + +本规格把资讯源作为智媒工厂的事实入口看待，不负责内容表达和媒体制作。 + +| 边界项 | 内容 | +| --- | --- | +| 外部使用者 | 编辑策划、视觉制作、发布运营和人工来源维护者。 | +| 外部输入 | owning YAML 来源目录、公开 Feed/API/网页、GitHub 发布记录、人工来源条目和期次日期。 | +| 受控资源 | 原始网络响应、规范化来源项、来源清单、分类排序、抓取状态和版权提示。 | +| 外部输出 | 按优先级排序的候选池、可解析来源标识、原始链接、短证据、状态边界和来源视觉策略。 | +| 用户接口 | TypeScript CLI 的采集单步、来源清单文件、结构化日志和 Web 来源视图。 | +| 系统边界 | 资讯源负责“候选事实从哪里来、是什么角色、能否追溯”；不决定“节目最终说什么”。 | + +## 5. 内部分工与下级索引 + +| 编号 | 模块或课题 | 规格位置 | 主责边界 | 上游依赖 | 下游支撑 | +| --- | --- | --- | --- | --- | --- | +| PJ2026-020101 | 来源目录 | 本规格 6.1 | 来源身份、抓取策略、角色、信任、优先级和使用提示 | 编辑规则、公开来源 | 网络采集、分类排序 | +| PJ2026-020102 | 网络采集 | 本规格 6.2 | Feed/API 获取、超时、并发、原始响应和失败状态 | 来源目录、公开网络 | 证据归一 | +| PJ2026-020103 | 证据归一 | 本规格 6.3 | 标题、链接、日期、摘要、规范链接、去重和来源标识 | 网络采集、人工来源 | 分类排序、编辑策划 | +| PJ2026-020104 | 分类排序 | 本规格 6.4 | 主题分类、内容优先级、一手优先和候选池排序 | 证据归一、栏目规则 | 编辑策划 | +| PJ2026-020105 | 优惠信息 | 本规格 6.5 | 期限、地区、资格、条件、过期和返利边界 | 官方公告、人工核验 | 编辑策划、发布运营 | +| PJ2026-020106 | 来源治理 | 本规格 6.6 | 归因、预印本、版权、失败和过期来源处置 | 全部来源环节 | 视觉制作、发布运营 | + +本章图形描述目标数据面。当前已实现范围和未实现差距见第 7.1 节。 + +### 5.1 目标数据面架构图 + +```mermaid +flowchart LR + subgraph Inputs[外部来源] + Feed[RSS / Atom] + API[Hacker News API] + Manual[人工来源] + end + subgraph SourcePlane[资讯源数据面] + Registry[来源目录与策略] + Fetcher[受控采集器] + Normalizer[证据归一与去重] + Classifier[分类与价值排序] + Store[原始响应与来源清单] + end + Registry --> Fetcher + Feed --> Fetcher + API --> Fetcher + Manual --> Normalizer + Fetcher --> Normalizer + Fetcher --> Store + Normalizer --> Classifier + Classifier --> Store + Store --> Editorial[编辑策划] + Store --> Visual[视觉制作] + Store --> Publish[发布运营] +``` + +来源目录决定允许连接什么入口以及如何解释结果。采集器不拥有选题，来源清单不拥有最终叙事。 + +### 5.2 目标数据流图 + +```mermaid +flowchart LR + Registry[owning YAML 来源目录] --> Fetch[Feed / API 采集] + Manual[人工来源] --> Normalize[证据归一] + Fetch --> Raw[原始响应存档] + Fetch --> Normalize + Normalize --> Dedupe[规范链接与去重] + Dedupe --> Classify[分类与优先级] + Classify --> Manifest[期次来源清单] + Manifest --> Editorial[编辑策划] + Manifest --> Visual[来源视觉] + Manifest --> Publish[最终引用] +``` + +原始响应和规范化来源清单必须分开保存。下游使用来源清单中的稳定标识和链接，不直接依赖临时抓取对象或未登记页面内容。 + +### 5.3 关键时序图 + +```mermaid +sequenceDiagram + participant E as 编辑者 + participant P as 流水线平台 + participant R as 来源目录 + participant F as 公开 Feed / API + participant N as 归一与排序 + participant A as 期次来源工件 + E->>P: 提交 collect 单步 + P->>R: 读取并校验允许来源 + loop 每个启用来源 + P->>F: 按策略获取元数据 + alt 来源成功 + F-->>P: 原始响应 + P->>A: 保存响应与抓取状态 + P->>N: 归一来源项 + else 来源失败 + F-->>P: 超时或错误 + P->>A: 保存失败状态 + end + end + P->>N: 合并人工来源并去重排序 + N->>A: 写入来源清单 + P-->>E: 返回计数、分类与失败摘要 +``` + +采集失败必须在同一作业和来源清单中可见。是否继续整期由质量门判断，不能把失败来源记录为零条成功。 + +## 6. 原子需求 + +### 6.1 SOURCE-L1-REQ-001 可治理来源目录 + +| 编号 | 短名 | 主责模块 | 关联模块 | +| --- | --- | --- | --- | +| SOURCE-L1-REQ-001 | 来源目录 | PJ2026-020101 来源目录 | [编辑策划](PJ2026-0202-editorial-planning.md)、[视觉制作](PJ2026-0203-visual-production.md) | + +系统应以 owning YAML 管理允许来源。每个来源必须明确标识： + +- 名称、类型、入口和启停状态； +- 采集策略、信任层级和来源角色； +- 默认分类、分类方式、优先级和标签； +- 视觉策略、版权提示和已知问题。 + +来源目录必须支持： + +- 第一方厂商和开源项目； +- 国际嵌入式媒体、Hacker News 和 arXiv； +- 人工维护页面。 + +没有稳定公开入口的来源应声明为 `manual-only`，不能在代码中私自增加隐藏抓取路径。 + +当前基线覆盖： + +- Arm、NXP、MicroPython、LVGL、Arm-2D 和 PikaPython； +- NI、Analog Devices、ST、NVIDIA、Arduino 和嘉立创相关入口； +- AI Agent/Coding Harness 和硬件 Development/Test Harness； +- 国际嵌入式媒体和 arXiv。 + +具体名单可以演进，但字段和角色边界必须保持可校验。 + +### 6.2 SOURCE-L1-REQ-002 可审计采集 + +| 编号 | 短名 | 主责模块 | 关联模块 | +| --- | --- | --- | --- | +| SOURCE-L1-REQ-002 | 可审计采集 | PJ2026-020102 网络采集 | PJ2026-020103 证据归一、[流水线平台](PJ2026-0205-pipeline-platform.md) | + +系统应通过公开且允许使用的 Feed、API 或人工输入采集资讯，并为每个自动来源记录成功、失败、响应状态、获取时间和原始响应摘要。 + +网络采集必须遵循以下边界： + +- 明确并发、超时和失败语义； +- 单个来源失败时不得伪造成功； +- 是否允许整期继续由质量门决定； +- Hacker News 只作为发现来源； +- 不得把提交者或讨论内容改写为项目官方事实。 + +原始 Feed、Atom、API 响应应按期次存档并带哈希或等价完整性线索。人工来源必须保留登记者可复核的原始链接、日期和短证据，不得只写无法追溯的总结。 + +### 6.3 SOURCE-L1-REQ-003 稳定证据模型 + +| 编号 | 短名 | 主责模块 | 关联模块 | +| --- | --- | --- | --- | +| SOURCE-L1-REQ-003 | 证据模型 | PJ2026-020103 证据归一 | PJ2026-020102 网络采集、[编辑策划](PJ2026-0202-editorial-planning.md) | + +系统应把不同来源归一为稳定来源项。来源项至少包含： + +- 来源标识、来源名称、标题和链接； +- 发布日期、抓取时间和摘要证据； +- 分类、标签、角色和信任层级； +- 是否预印本、发现入口和视觉策略。 + +规范链接和去重应： + +- 消除安全可忽略的跟踪差异； +- 不把不同版本、不同公告或不同语言页面错误合并； +- 保留最可信、最接近一手发布者且证据更完整的候选。 + +媒体或社区发现项如果能定位到一手页面，应同时保留发现关系和一手来源；如果暂时只有媒体报道，必须在来源项和文稿中明确媒体归因。 + +### 6.4 SOURCE-L1-REQ-004 主题与价值排序 + +| 编号 | 短名 | 主责模块 | 关联模块 | +| --- | --- | --- | --- | +| SOURCE-L1-REQ-004 | 分类排序 | PJ2026-020104 分类排序 | [编辑策划](PJ2026-0202-editorial-planning.md) | + +系统应按照栏目 owning YAML 对候选资讯分类和排序，当前优先顺序为嵌入式与 AI、纯嵌入式、AI，并在同类中优先一手来源、明确状态、近期发布和可行动价值。 + +分类必须基于来源默认值和内容证据： + +- 不能因媒体栏目名称或单个宽泛关键词把普通 AI 新闻误归为嵌入式； +- 不能把汽车线束等无关 `wire harness` 混入硬件 Development/Test Harness。 + +排序只决定候选池顺序，不替代编辑选择。来源热度、Hacker News 排名和媒体标题可以作为线索，但不能覆盖证据质量、受众相关性和状态边界。 + +### 6.5 SOURCE-L1-REQ-005 有条件实用优惠 + +| 编号 | 短名 | 主责模块 | 关联模块 | +| --- | --- | --- | --- | +| SOURCE-L1-REQ-005 | 实用优惠 | PJ2026-020105 优惠信息 | [编辑策划](PJ2026-0202-editorial-planning.md)、[发布运营](PJ2026-0206-publishing-operations.md) | + +系统应识别并结构化官方优惠、折扣、优惠券、样片计划、开发板赠送和免费额度，使编辑者可以判断其是否仍然有效并适合目标受众。 + +每条优惠至少应保留： + +- 类型； +- 截止时间或有效期； +- 适用地区、资格和条件； +- 官方链接。 + +已过期、缺失日期、条件不清或使用返利链接的条目应降权或剔除，不能以“限时福利”掩盖无法核验的限制。 + +发布时必须重查活动状态。资讯源只提供采集时证据，不保证平台库存、价格、地区资格或用户最终能获得优惠。 + +### 6.6 SOURCE-L1-REQ-006 来源和版权边界 + +| 编号 | 短名 | 主责模块 | 关联模块 | +| --- | --- | --- | --- | +| SOURCE-L1-REQ-006 | 来源治理 | PJ2026-020106 来源治理 | [视觉制作](PJ2026-0203-visual-production.md)、[发布运营](PJ2026-0206-publishing-operations.md) | + +系统应在来源清单中保留可供编辑、视觉和发布环节消费的归因、版权和状态提示，并禁止把“可访问”解释为“可自由再发布”。 + +来源许可必须分层判断： + +- Feed 和 API 默认只授权其明确允许的元数据用途； +- 正文、图片、论文图表、品牌素材和页面截图应按来源条款与发布场景单独复核； +- arXiv 元数据与单篇论文内容的许可不能混为一谈。 + +来源不可访问、页面变化或视觉采集失败时，应保留失败状态并提供无图降级，不得使用无关图库或旧截图冒充当前来源。 + +## 7. 过程控制 + +### 7.1 当前实现状态与引用 + +- 当前已经实现 YAML 来源目录、Feed/API/人工来源、原始响应、归一化、去重、分类排序和来源清单。 +- 当前已经在真实 R3 期次中采集第一方、国际媒体、Hacker News 和人工来源。 +- 第 6 章仍是稳定目标口径；新增来源、许可变化和优惠复核需要持续按该口径验收。 + +- 来源 owning YAML：`/root/selfmedia/config/sources.yaml`。 +- 栏目分类和排序：`/root/selfmedia/config/editorial.yaml`。 +- 采集与归一：`/root/selfmedia/scripts/src/steps/collect.ts`。 +- 期次输入与输出： + - `inputs/manual-sources.yaml`； + - `sources/raw/`； + - `sources/manifest.json`。 +- 后续相关源码修改应标记： + - `SPEC: PJ2026-0201 资讯源 draft-2026-07-13-p0`。 + +### 7.2 原入口验收 + +- 使用 TypeScript CLI 单步运行 `collect`。 +- 核对来源总数、自动来源成功数、人工来源数、一手与发现来源数、分类分布和抓取失败。 +- 随机抽取来源项，验证规范链接、发布日期、角色、信任层级、证据摘要、原始响应和视觉策略可解析。 +- 对优惠信息单独验证截止时间、地区、资格、条件和过期处置。 +- 对媒体、Hacker News 和预印本单独验证归因和状态没有被提升。 + +### 7.3 变更和失败控制 + +- 新增或修改来源先改 owning YAML，再运行配置校验和有界采集。 +- 来源结构字段、角色语义、分类优先级或优惠接受标准变化时，先更新本规格。 +- 单个来源临时故障保留为运行证据，不直接修改稳定规格。 +- 连续失败、条款变化、入口停用或数据质量长期下降时，应在 owning YAML 中禁用或改为人工来源。 +- 来源数量不能作为完成标准；完成标准是进入下游的候选可追溯、可归因、状态清楚且对栏目有用。 + +### 7.4 回写边界 + +- 选题和文稿问题回写 [编辑策划](PJ2026-0202-editorial-planning.md)。 +- 来源页面画面和截图使用问题回写 [视觉制作](PJ2026-0203-visual-production.md)。 +- 最终引用、活动复核和平台披露问题回写 [发布运营](PJ2026-0206-publishing-operations.md)。 +- 调度、日志、作业和工件可见性问题回写 [流水线平台](PJ2026-0205-pipeline-platform.md)。 diff --git a/project-management/PJ2026-02/specs/PJ2026-0202-editorial-planning.md b/project-management/PJ2026-02/specs/PJ2026-0202-editorial-planning.md new file mode 100644 index 00000000..16d86a77 --- /dev/null +++ b/project-management/PJ2026-02/specs/PJ2026-0202-editorial-planning.md @@ -0,0 +1,346 @@ +# PJ2026-0202 编辑策划需求规格 + +## 修改历史 + +| 版本 | 对应 commit id | 更新日期 | 变更说明 | +| --- | --- | --- | --- | +| v0.1 | 待提交 | 2026-07-13 | 基于现有结构化文稿、质量门、专业对象补充和来源闭环建立编辑策划规格。 | + +当前正文仍在规格治理草稿中；未定稿前不新增版本号，不为单次编辑追加 `待提交` 版本。 + +## 正文 + +## PJ2026-0202 编辑策划需求规格 + +## 1. 文档控制 + +| 字段 | 内容 | +| --- | --- | +| 编号 | PJ2026-0202 | +| 短名 | 编辑策划 | +| 层级 | L1 方向 | +| 状态 | 已生效 | +| 实现引用版本 | draft-2026-07-13-p0 | +| 需求规格模板 | [ISO/IEC/IEEE 29148 需求规格模板](../../templates/iso-iec-ieee-29148-requirements-spec-template.md) | +| 上级规格 | [PJ2026-02 智媒工厂总规格](PJ2026-02-smart-media-factory.md) | +| 规格治理索引 | [PJ2026-02 智媒工厂总规格](PJ2026-02-smart-media-factory.md) 第 7 章 | + +本文采用 ISO/IEC/IEEE 29148 需求规格模板的项目裁剪版。正文只定义选题、事实核验、栏目结构、叙事和引用的稳定完成标准。 + +## 2. 目的和范围 + +### 2.1 目的 + +编辑策划负责： + +- 把可追溯来源转化为有判断、有信息密度、适合技术受众收听的栏目内容； +- 决定本期为什么讲这些消息； +- 决定每条消息要解释到什么深度以及哪些限制必须明说； +- 让开场、大纲、故事和总结形成连贯叙事； +- 避免把来源摘要机械拼接成“AI 味”文稿。 + +### 2.2 范围内 + +- 栏目定位、受众假设、期次题目、候选池、选题数量和内容优先级。 +- 固定开场、五十字内概述、卡片大纲、单条故事、总结和参考来源口播结构。 +- 事实锚点、日期、数字、产品状态、媒体归因、预印本和活动条件的表达边界。 +- 按受众熟悉度补充专业对象的功能、作者、公司、发布者或大学团队来源。 +- 结构化文稿、配音文本、视觉方向、来源引用和评论区来源文本的内容一致性。 +- 授权 LLM 生成、质量门重试和完全离线模板路径。 + +### 2.3 范围外 + +- 来源抓取、去重、分类和原始证据保存，归 [资讯源](PJ2026-0201-information-sources.md)。 +- HTML 画面、页面滚动、演示文稿和封面，归 [视觉制作](PJ2026-0203-visual-production.md)。 +- 音色、语速、字幕断句和视频编码，归 [音频成片](PJ2026-0204-audio-video-production.md)。 +- 发布批准、平台文案投放和效果反馈归档，归 [发布运营](PJ2026-0206-publishing-operations.md)。 +- 通过杜撰数字、空泛预测、夸张标题或隐去限制提升戏剧性。 + +## 3. 术语表 + +| 术语 | 定义 | +| --- | --- | +| 事实锚点 | 能由来源证据直接支持且决定一条资讯核心价值的日期、状态、数字、动作或对象关系。 | +| 质量门 | 在接受文稿前检查结构、长度、证据、空泛表达、状态限定和引用完整性的规则集合。 | +| 受众熟悉度 | 对目标受众是否可合理推断已经认识某个对象的编辑判断。 | +| 专业对象补充 | 对陌生库、组件、框架、协议或项目补充其功能和发布者的短句。 | +| 状态限定 | 对 beta、预览、预印本、临时调整、媒体报道、地区限制等事实边界的明确表达。 | +| 镜头意图 | 文稿为视觉制作提供的信息重点、来源页面和画面节奏建议，不是最终视觉实现。 | +| 结构化文稿 | 包含开场、大纲、故事、总结、参考来源、配音单元和来源引用关系的机器可读工件。 | +| 模板模式 | 不调用 LLM、根据来源和固定规则生成可调试文稿的离线路径。 | + +## 4. 系统边界和接口 + +本规格把编辑策划作为内容判断和结构化表达能力看待，不负责来源网络和媒体渲染。 + +| 边界项 | 内容 | +| --- | --- | +| 外部使用者 | 栏目编辑、审核者、视觉制作、音频成片和发布运营。 | +| 外部输入 | 期次元数据、来源候选池、栏目规则、受众熟悉对象、人工编辑意见和可选 LLM 响应。 | +| 受控资源 | 选题结果、结构化文稿、质量门反馈、配音单元、镜头意图、来源引用和引用目录。 | +| 外部输出 | 开场、大纲、故事、总结、参考来源口播、配音文本、视觉方向和可解析引用。 | +| 用户接口 | TypeScript CLI 文稿单步、可编辑 JSON/TXT/Markdown 工件、日志和 Web 文稿视图。 | +| 系统边界 | 编辑策划负责“说什么以及如何准确地说”；不负责来源可访问性、画面制作、声音表现和最终发布批准。 | + +## 5. 内部分工与下级索引 + +| 编号 | 模块或课题 | 规格位置 | 主责边界 | 上游依赖 | 下游支撑 | +| --- | --- | --- | --- | --- | --- | +| PJ2026-020201 | 栏目策划 | 本规格 6.1 | 栏目定位、受众、期次结构和选题组合 | 期次元数据、来源候选 | 全部编辑环节 | +| PJ2026-020202 | 事实核验 | 本规格 6.2 | 锚点、状态、数字、日期、归因和不能推出的结论 | 资讯源 | 文稿生成、发布复核 | +| PJ2026-020203 | 口播叙事 | 本规格 6.3 | 开场、大纲、故事、总结、节奏和语言质量 | 栏目策划、事实核验 | 视觉制作、音频成片 | +| PJ2026-020204 | 对象补充 | 本规格 6.4 | 专业对象功能、作者或发布者说明与熟悉对象免补充 | 事实核验、受众规则 | 口播叙事 | +| PJ2026-020205 | 文稿质量 | 本规格 6.5 | 生成、重试、质量门、模板路径和人工覆盖 | 全部编辑环节 | 流水线平台 | +| PJ2026-020206 | 引用闭环 | 本规格 6.6 | 文稿到来源、镜头、配音和发布引用的一致关系 | 资讯源、结构化文稿 | 发布运营 | + +本章图形描述目标数据面。当前已实现范围和未实现差距见第 7.1 节。 + +### 5.1 目标数据面架构图 + +```mermaid +flowchart LR + subgraph Inputs[编辑输入] + Edition[期次元数据] + Sources[来源候选与证据] + Rules[栏目与受众规则] + Human[人工编辑意见] + end + subgraph EditorialPlane[编辑策划数据面] + Planner[选题组合] + Evidence[事实锚点与状态] + Generator[LLM 或模板生成] + Gate[结构与事实质量门] + Assembler[确定性内容组装] + end + Edition --> Planner + Sources --> Planner + Rules --> Planner + Human --> Planner + Planner --> Evidence + Evidence --> Generator + Rules --> Generator + Generator --> Gate + Gate --> Assembler + Assembler --> Script[结构化文稿] + Assembler --> Citations[引用与配音文本] +``` + +确定性组装器拥有期数、日期、固定开场和引用顺序。生成模型不能越过质量门直接成为下游内容真相。 + +### 5.2 目标数据流图 + +```mermaid +flowchart TD + Pool[可追溯候选池] --> Select[按价值与组合选题] + Select --> Evidence[提取事实锚点与限制] + Evidence --> Audience[判断受众熟悉度] + Audience --> Draft[生成开场 / 大纲 / 故事 / 总结] + Draft --> Gate[结构、证据与语言质量门] + Gate -->|拒绝并反馈| Draft + Gate -->|接受| Script[结构化文稿] + Script --> Visual[镜头意图] + Script --> Audio[配音单元] + Script --> Publish[来源引用与评论区文本] +``` + +质量门拒绝必须产生具体反馈并进入下一次生成或人工修订，不能只重复同一提示。被拒绝响应仍属于可观察的 LLM 尝试和成本事实。 + +### 5.3 关键时序图 + +```mermaid +sequenceDiagram + participant E as 编辑者 + participant P as 流水线平台 + participant S as 来源清单 + participant G as 文稿生成器 + participant U as 用量记录 + participant Q as 质量门 + participant A as 文稿工件 + E->>P: 提交 script 单步 + P->>S: 读取候选与来源证据 + P->>G: 传入栏目规则和有界证据 + alt 模板模式 + G-->>P: 模板结构化草稿 + else 授权 LLM + loop 直到接受或达到上限 + G-->>P: 模型响应与 usage + P->>U: 记录本次尝试和费用状态 + P->>Q: 校验结构、事实和表达 + alt 被拒绝 + Q-->>G: 返回具体修订反馈 + else 接受 + Q-->>P: 接受结构化内容 + end + end + end + P->>A: 组装开场、正文、引用与配音单元 + P-->>E: 返回文稿、引用和质量摘要 +``` + +模板模式不得读取 LLM 凭据。模型尝试即使被拒绝，也必须先写入用量与质量结果再进入下一次尝试。 + +## 6. 原子需求 + +### 6.1 EDIT-L1-REQ-001 栏目化选题 + +| 编号 | 短名 | 主责模块 | 关联模块 | +| --- | --- | --- | --- | +| EDIT-L1-REQ-001 | 栏目选题 | PJ2026-020201 栏目策划 | [资讯源](PJ2026-0201-information-sources.md)、[发布运营](PJ2026-0206-publishing-operations.md) | + +系统应根据以下条件形成一期选题： + +- 栏目定位和报道日期； +- 目标时长和受众； +- 内容优先级和候选证据； +- 主题组合和实用价值。 + +系统不得只选择得分最高或标题最热的若干条。 + +当前栏目组合规则如下： + +- 优先嵌入式与 AI 交叉； +- 其次纯嵌入式； +- 再次 AI； +- 避免同一期内容都属于同一厂商、同一产品或同一种发布形式； +- 优惠和供应链变化只有与技术受众实际决策相关时才可入选。 + +选题完成标准包括来源足够、日期和状态明确、受众价值可以用一句话说明、节目组合有层次。候选证据不足时应减少条目或明确失败，不能用空泛评论补齐数量。 + +### 6.2 EDIT-L1-REQ-002 事实与状态核验 + +| 编号 | 短名 | 主责模块 | 关联模块 | +| --- | --- | --- | --- | +| EDIT-L1-REQ-002 | 事实核验 | PJ2026-020202 事实核验 | [资讯源](PJ2026-0201-information-sources.md)、[发布运营](PJ2026-0206-publishing-operations.md) | + +系统应在写稿前为每条入选资讯确定事实锚点、来源角色和状态边界，并要求文稿只陈述来源能够支持的内容。 + +具体事实使用规则如下： + +- 日期、版本、处理器、内存、价格、性能和截止时间只有在证据明确时才能出现； +- 来源未提供延迟、功耗、精度、兼容性或开放范围时，文稿应说明未知； +- 不能用行业常识填成该产品事实。 + +媒体报道必须说“媒体报道”或点明媒体；大学论文只有在来源明确单位时才说“来自某大学团队”，否则使用“作者团队”；预印本不得暗示已经同行评审。 + +### 6.3 EDIT-L1-REQ-003 固定栏目结构 + +| 编号 | 短名 | 主责模块 | 关联模块 | +| --- | --- | --- | --- | +| EDIT-L1-REQ-003 | 口播结构 | PJ2026-020203 口播叙事 | [视觉制作](PJ2026-0203-visual-production.md)、[音频成片](PJ2026-0204-audio-video-production.md) | + +系统应生成稳定但不僵硬的栏目结构，至少包含开场、卡片式大纲、逐条资讯、总结和参考来源口播。 + +开场应使用期次元数据确定性生成： + +- 报出期数、栏目名和日期； +- 使用“主要内容有”引出不超过五十字的概述； +- 使用“下面是详细内容”进入正文； +- 不允许 LLM 改写期数、日期或固定句式。 + +栏目叙事顺序如下： + +- 每条故事先给核心变化； +- 再解释技术或业务影响； +- 最后说明限制、验证建议或下一步； +- 总结收束共同主题，不重复逐条摘要； +- 参考来源口播固定为“以下是这期《栏目名》的参考来源，完整链接见评论区”。 + +### 6.4 EDIT-L1-REQ-004 受众适配说明 + +| 编号 | 短名 | 主责模块 | 关联模块 | +| --- | --- | --- | --- | +| EDIT-L1-REQ-004 | 对象补充 | PJ2026-020204 对象补充 | PJ2026-020202 事实核验、PJ2026-020203 口播叙事 | + +系统应根据受众熟悉度决定是否补充短介绍： + +- GPT、Claude、ESP32 等可合理推断目标受众熟悉的对象可以直接进入新消息； +- 专门的库、组件、框架、协议或项目应补充其功能和发布者。 + +补充句应说明： + +- 它是什么； +- 解决什么问题； +- 由谁发布或维护。 + +只使用来源支持的作者、公司、机构或团队信息。不能编造公司归属，也不能把仓库组织名自动当作法人或作者身份。 + +补充句服务理解，不应盖过新资讯本身；同一条口播首次介绍后不重复堆叠定义。 + +### 6.5 EDIT-L1-REQ-005 可审查文稿生成 + +| 编号 | 短名 | 主责模块 | 关联模块 | +| --- | --- | --- | --- | +| EDIT-L1-REQ-005 | 文稿质量 | PJ2026-020205 文稿质量 | [流水线平台](PJ2026-0205-pipeline-platform.md)、[音频成片](PJ2026-0204-audio-video-production.md) | + +系统应同时提供授权 LLM 和模板模式，并把两者输出为同一结构化文稿契约。LLM 只接收任务所需的来源证据和栏目规则，凭据不能进入提示词、工件或日志。 + +质量门至少检查： + +- 固定结构、概述长度和故事数量； +- 引用可解析、事实锚点和状态限定； +- 专业对象补充、空泛套话和后台术语； +- 不适合口播的格式。 + +失败时应保存拒绝原因，并把具体反馈用于下一次尝试。 + +人工编辑可以修改结构化文稿后单步重跑下游。系统不能把模型输出视为不可更改的最终稿，也不能在模板模式下读取 LLM 凭据或伪造模型元数据。 + +### 6.6 EDIT-L1-REQ-006 跨工件内容闭环 + +| 编号 | 短名 | 主责模块 | 关联模块 | +| --- | --- | --- | --- | +| EDIT-L1-REQ-006 | 引用闭环 | PJ2026-020206 引用闭环 | [资讯源](PJ2026-0201-information-sources.md)、[视觉制作](PJ2026-0203-visual-production.md)、[发布运营](PJ2026-0206-publishing-operations.md) | + +系统应为每条故事保留： + +- 来源标识、来源名称、来源角色和原始链接； +- 中文短释、镜头意图和配音单元； +- 可关联文稿、来源画面、字幕、章节时间轴和评论区引用的故事身份。 + +引用目录应只列实际采用的来源，不把候选池全部条目混入。发现来源与一手来源并存时必须保持各自角色，不能在引用阶段丢失发现关系或重复计算同一事实。 + +结构化文稿是下游内容真相源。纯文本配音稿、引用 Markdown 和评论区文本应由它和最终场景时间线生成，不应由人工复制粘贴形成互相漂移的版本。 + +## 7. 过程控制 + +### 7.1 当前实现状态与引用 + +- 当前已经实现授权 LLM、模板模式、确定性开场、质量门、专业对象补充、引用和配音单元。 +- 当前 R3 文稿已经生成开场、大纲、五条故事、总结和参考来源口播。 +- 第 6 章描述持续目标；人工批准和发布版本不由本 L1 当前实现提供。 + +- 栏目和文稿 owning YAML：`/root/selfmedia/config/editorial.yaml`。 +- 文稿生成与质量门：`/root/selfmedia/scripts/src/steps/script.ts`。 +- 评论区来源：`/root/selfmedia/scripts/src/comment-reference.ts`。 +- 期次工件： + - `script/episode.json`； + - `script/narration.txt`； + - `script/citations.md`； + - `script/comment-reference.txt`。 +- 后续相关源码修改应标记： + - `SPEC: PJ2026-0202 编辑策划 draft-2026-07-13-p0`。 + +### 7.2 原入口验收 + +- 通过 TypeScript CLI 单步运行 `script`： + - 授权 LLM 路径应记录模型、请求和用量工件； + - `--template-only` 路径应不读取 LLM 凭据。 +- 核对开场期数、栏目名、日期、五十字内概述和固定转场句。 +- 核对每条故事的事实锚点、来源角色、状态限定、专业对象补充和未知信息边界。 +- 核对引用可在期次来源清单中解析，配音单元和故事顺序一致。 +- 通过人工朗读抽检口语节奏、信息密度和“AI 味”套话。 + +### 7.3 质量门和人工覆盖 + +- 被质量门拒绝的模型响应必须保留请求级用量、拒绝原因和重试关系。 +- 达到最大重试仍失败时应输出明确阻塞，不得自动降低事实或引用要求。 +- 人工改稿后重跑下游时，应保留新的作业和工件哈希，不覆盖已发布版本的证据。 +- 提示词、受众熟悉对象、禁用表达、目标长度或状态规则变化时，先改 owning YAML；涉及稳定完成标准时同步更新本规格。 + +### 7.4 回写边界 + +- 候选来源证据不足、角色错误或分类错误回写 [资讯源](PJ2026-0201-information-sources.md)。 +- 文稿可用但画面信息层级不足回写 [视觉制作](PJ2026-0203-visual-production.md)。 +- 读音、语速、字幕或同步问题回写 [音频成片](PJ2026-0204-audio-video-production.md)。 +- 最终时间轴、来源格式、披露或批准问题回写 [发布运营](PJ2026-0206-publishing-operations.md)。 +- 模型请求、费用、作业或工件不可见问题回写 [流水线平台](PJ2026-0205-pipeline-platform.md)。 diff --git a/project-management/PJ2026-02/specs/PJ2026-0203-visual-production.md b/project-management/PJ2026-02/specs/PJ2026-0203-visual-production.md new file mode 100644 index 00000000..9fc3da94 --- /dev/null +++ b/project-management/PJ2026-02/specs/PJ2026-0203-visual-production.md @@ -0,0 +1,335 @@ +# PJ2026-0203 视觉制作需求规格 + +## 修改历史 + +| 版本 | 对应 commit id | 更新日期 | 变更说明 | +| --- | --- | --- | --- | +| v0.1 | 待提交 | 2026-07-13 | 基于来源页面视觉、暖色 HTML 动效、九阶段导航、PNG 和 PPTX 能力建立视觉制作规格。 | + +当前正文仍在规格治理草稿中；未定稿前不新增版本号，不为单次编辑追加 `待提交` 版本。 + +## 正文 + +## PJ2026-0203 视觉制作需求规格 + +## 1. 文档控制 + +| 字段 | 内容 | +| --- | --- | +| 编号 | PJ2026-0203 | +| 短名 | 视觉制作 | +| 层级 | L1 方向 | +| 状态 | 已生效 | +| 实现引用版本 | draft-2026-07-13-p0 | +| 需求规格模板 | [ISO/IEC/IEEE 29148 需求规格模板](../../templates/iso-iec-ieee-29148-requirements-spec-template.md) | +| 上级规格 | [PJ2026-02 智媒工厂总规格](PJ2026-02-smart-media-factory.md) | +| 规格治理索引 | [PJ2026-02 智媒工厂总规格](PJ2026-02-smart-media-factory.md) 第 7 章 | + +本文采用 ISO/IEC/IEEE 29148 需求规格模板的项目裁剪版。正文只定义来源视觉、场景、主题、动效、演示文稿和视觉验收的稳定需求。 + +## 2. 目的和范围 + +### 2.1 目的 + +视觉制作的目标是： + +- 把结构化文稿和来源证据转化为信息清楚、节奏连贯的动态画面； +- 形成具有栏目辨识度的视觉语言； +- 生成的 HTML、PPT、PNG、封面和视频采用温暖、阳光、圆润的科技编辑部方向； +- 避免冷冽的蓝黑仪表盘观感； +- 长期支持栏目主题切换，同时保持来源可追溯、绝对时间可复现和多工件一致。 + +### 2.2 范围内 + +- 入选资讯原始网页、GitHub 页面或论文摘要页的本地视觉采集和失败降级。 +- 开场、卡片大纲、逐条资讯、总结和参考来源等场景类型。 +- YAML 可配置的色彩、圆角、阴影、纹理、字号、留白、字幕安全区和章节导航。 +- HTML/CSS/JavaScript 动效、来源页面向下滚动、绝对时间寻址和逐帧渲染状态。 +- 顶部阶段内细进度、底部总阶段导航和总进度的对齐与跳转语义。 +- HTML、PNG、PPTX、封面候选和视觉清单等多工件输出。 +- 内容可读性、版权提示、品牌克制和离线生产边界。 + +### 2.3 范围外 + +- 资讯事实、选题和口播内容，归 [编辑策划](PJ2026-0202-editorial-planning.md)。 +- 来源身份、角色、抓取政策和版权事实，归 [资讯源](PJ2026-0201-information-sources.md)。 +- 场景时长最终分配、配音、字幕和视频编码，归 [音频成片](PJ2026-0204-audio-video-production.md)。 +- 复制已有视频的封面、截图编排、品牌包装、转场、动画或受保护素材。 +- 依赖在线 CDN、在线字体、外部脚本或付费生成服务才能渲染生产画面。 +- Web 工厂、控制台和内置终端的视觉语言： + - 控制台归 [流水线平台](PJ2026-0205-pipeline-platform.md)； + - 内置终端归 [平台交付](PJ2026-0207-platform-delivery.md)； + - 它们应使用工业风和紧凑布局，不复用本规格的暖色大圆角主题。 + +## 3. 术语表 + +| 术语 | 定义 | +| --- | --- | +| 来源视觉 | 从实际采用来源页面生成并登记的本地画面帧及其链接、滚动位置和使用策略。 | +| 场景模型 | 描述场景身份、类型、标题、正文、来源视觉、章节和 HTML 工件关系的结构化数据。 | +| 绝对时间寻址 | 通过场景内毫秒位置直接计算动效状态，使任意帧可重现而不依赖前序播放。 | +| 阶段内进度 | 位于画面顶部、表示当前场景内部播放位置的细进度条。 | +| 总阶段导航 | 位于画面底部、等宽等高展示整期短标题并与总进度 X 轴对齐的章节导航。 | +| 来源滚动 | 在若干本地来源帧之间按绝对时间确定性过渡，形成页面向下滚动的视觉效果。 | +| 内容主题令牌 | 由 YAML 管理的生成内容颜色、圆角、阴影、纹理和布局参数。 | +| 字幕安全区 | 为字幕和播放器覆盖层预留、不得放置关键信息的画面区域。 | +| 无图降级 | 来源页面不可采集或不适合使用时，以原创文字、数据卡片和来源标识替代第三方画面。 | + +## 4. 系统边界和接口 + +本规格把视觉制作作为内容画面和演示工件生产能力看待，不负责文稿真实性和音视频编码。 + +| 边界项 | 内容 | +| --- | --- | +| 外部使用者 | 编辑者、视频制作者、音频成片、发布审核者和 Web 预览用户。 | +| 外部输入 | 结构化文稿、来源清单、来源视觉策略、期次元数据、场景时长提示和主题 YAML。 | +| 受控资源 | 来源帧、视觉清单、场景模型、HTML、CSS、PNG、PPTX、封面画面和动效状态函数。 | +| 外部输出 | 可预览 HTML、确定性场景帧、演示文稿、封面候选和供成片渲染的场景清单。 | +| 用户接口 | TypeScript CLI 视觉与幻灯单步、浏览器 HTML 预览、PNG/PPTX 工件和 Web 幻灯视图。 | +| 系统边界 | 视觉制作负责“信息如何被看见和导航”；不改变来源事实、口播语义和最终媒体时间线。 | + +## 5. 内部分工与下级索引 + +| 编号 | 模块或课题 | 规格位置 | 主责边界 | 上游依赖 | 下游支撑 | +| --- | --- | --- | --- | --- | --- | +| PJ2026-020301 | 来源画面 | 本规格 6.1 | 页面采集、滚动位置、清单、失败和无图降级 | 资讯源、结构化文稿 | 场景编排 | +| PJ2026-020302 | 场景编排 | 本规格 6.2 | 开场、大纲、故事、总结、来源页和章节身份 | 编辑策划、来源画面 | 动效渲染、音频成片 | +| PJ2026-020303 | 主题系统 | 本规格 6.3 | 色彩、圆角、阴影、纹理、字号、留白和主题切换 | YAML、栏目定位 | 全部场景 | +| PJ2026-020304 | 导航进度 | 本规格 6.4 | 阶段内进度、总导航、总进度、对齐和标题高度 | 场景编排、时间线 | 成片、Web 跳转 | +| PJ2026-020305 | 动效渲染 | 本规格 6.5 | 绝对时间寻址、来源滚动、逐帧一致性和离线渲染 | 场景编排、主题系统 | 音频成片 | +| PJ2026-020306 | 视觉工件 | 本规格 6.6 | HTML、PNG、PPTX、封面和视觉质量边界 | 全部视觉环节 | 发布运营 | + +本章图形描述目标数据面。当前已实现范围和未实现差距见第 7.1 节。 + +### 5.1 目标数据面架构图 + +```mermaid +flowchart LR + subgraph Inputs[视觉输入] + Script[结构化文稿] + Sources[来源清单] + Theme[主题 owning YAML] + end + subgraph VisualPlane[视觉制作数据面] + Capture[来源页面采集器] + VisualStore[本地来源帧与清单] + SceneCompiler[场景编排器] + HtmlRenderer[HTML 动效渲染器] + DeckRenderer[PNG / PPTX 渲染器] + end + Sources --> Capture + Capture --> VisualStore + Script --> SceneCompiler + VisualStore --> SceneCompiler + Theme --> SceneCompiler + SceneCompiler --> HtmlRenderer + HtmlRenderer --> DeckRenderer + HtmlRenderer --> Motion[确定性视频帧] + DeckRenderer --> Review[人工视觉预览] +``` + +来源采集器只产生本地视觉工件。场景编排器拥有章节与画面关系，渲染器不重新解析来源事实。 + +### 5.2 目标数据流图 + +```mermaid +flowchart TD + Script[结构化文稿] --> Scenes[场景编排] + Sources[来源清单] --> Capture[来源页面采集] + Capture --> VisualManifest[来源视觉清单] + VisualManifest --> Scenes + Theme[主题 owning YAML] --> HTML[HTML 场景生成] + Scenes --> HTML + HTML --> Preview[PNG 预览] + HTML --> Motion[绝对时间动效帧] + Preview --> Deck[PPTX] + Preview --> Cover[封面候选] + Motion --> Video[音频成片] +``` + +HTML 场景是动态视觉语义的主要表达，PNG 和 PPTX 是可审查、可交换的派生工件。三者应共享场景数据和主题，不维护互相漂移的版式实现。 + +### 5.3 关键时序图 + +```mermaid +sequenceDiagram + participant E as 编辑者 + participant P as 流水线平台 + participant C as 来源页面采集器 + participant S as 场景编排器 + participant B as 本地浏览器 + participant A as 视觉工件 + E->>P: 提交 visuals 单步 + P->>C: 传入实际采用来源与视觉策略 + loop 每条故事 + C->>B: 打开公开来源并按位置截图 + alt 可以采集 + B-->>C: 本地来源帧 + C->>A: 登记链接、滚动位置与画面 + else 不可采集 + C->>A: 登记失败与无图降级 + end + end + E->>P: 提交 slides 单步 + P->>S: 传入文稿、来源视觉和主题 + S->>B: 按绝对时间生成 HTML 与预览 + B-->>A: HTML、PNG、PPTX 与场景清单 + P-->>E: 返回场景数、画面状态与工件索引 +``` + +来源视觉失败不应触发无关图片替换。幻灯步骤必须消费视觉清单中的成功或降级状态，保持 HTML、PNG 和 PPTX 同源。 + +## 6. 原子需求 + +### 6.1 VISUAL-L1-REQ-001 可追溯来源画面 + +| 编号 | 短名 | 主责模块 | 关联模块 | +| --- | --- | --- | --- | +| VISUAL-L1-REQ-001 | 来源画面 | PJ2026-020301 来源画面 | [资讯源](PJ2026-0201-information-sources.md)、[编辑策划](PJ2026-0202-editorial-planning.md) | + +系统应为每条入选资讯按来源视觉策略采集本地画面，并在视觉清单中保留故事、来源、链接、页面标题、采集状态、画面路径、滚动位置、页面高度和失败原因。 + +来源画面可以按网页、GitHub 和论文摘要使用不同样式，但必须遵守： + +- 来自实际采用的公开来源； +- 只采集必要页面范围并用于短时展示； +- 不下载整站； +- 不展示登录信息、个性化推荐、敏感资料或无关广告。 + +页面不可访问、内容不适合展示或版权边界不清时应使用无图降级。系统不得从无关图片搜索结果、旧期次或其他媒体视频中找图冒充当前来源。 + +### 6.2 VISUAL-L1-REQ-002 稳定场景结构 + +| 编号 | 短名 | 主责模块 | 关联模块 | +| --- | --- | --- | --- | +| VISUAL-L1-REQ-002 | 场景编排 | PJ2026-020302 场景编排 | [编辑策划](PJ2026-0202-editorial-planning.md)、[音频成片](PJ2026-0204-audio-video-production.md) | + +系统应把结构化文稿编排为具有稳定身份和章节语义的场景： + +- 当前栏目至少包含封面开场、卡片大纲、每条资讯、总结和参考来源； +- 故事数量变化时，场景数量应按内容生成； +- 场景数量不能写死为九页。 + +各场景职责如下： + +- 开场突出期数、日期和主题； +- 大纲使用可快速扫读的卡片； +- 故事页呈现短标题、核心变化、关键信息和来源画面； +- 总结收束主题； +- 参考来源页以类似参考文献的编号格式列出实际采用来源。 + +场景标题、章节短标题、中文短释和来源身份应来自同一结构化文稿，不允许在 HTML、PPTX 和视频中分别人工维护。 + +### 6.3 VISUAL-L1-REQ-003 暖阳可配置主题 + +| 编号 | 短名 | 主责模块 | 关联模块 | +| --- | --- | --- | --- | +| VISUAL-L1-REQ-003 | 暖阳主题 | PJ2026-020303 主题系统 | [编辑策划](PJ2026-0202-editorial-planning.md)、[发布运营](PJ2026-0206-publishing-operations.md) | + +系统应通过 owning YAML 管理生成内容主题令牌。当前默认主题使用米白、琥珀、陶土和暖棕色，大圆角、柔和阴影和轻纸纹营造温暖、阳光的编辑部感受。 + +本主题只用于生成的 HTML、PPT、PNG、封面和视频。Web 工厂、控制台与内置终端不得继承这些大圆角、卡片留白和暖色装饰规则。 + +主题必须保持： + +- 正文、标题、来源标签、进度和字幕区域的对比度与可读性； +- 通过层级、留白、大小和有限强调色区分关键信息； +- 不让暖色退化为所有元素同色； +- 不依赖颜色单独表达状态。 + +主题切换不得改变场景数据、来源关系和时间线契约。未来栏目可以有其他主题，但生产 HTML 不得依赖在线字体、脚本或样式服务。 + +### 6.4 VISUAL-L1-REQ-004 对齐章节导航 + +| 编号 | 短名 | 主责模块 | 关联模块 | +| --- | --- | --- | --- | +| VISUAL-L1-REQ-004 | 导航进度 | PJ2026-020304 导航进度 | [音频成片](PJ2026-0204-audio-video-production.md)、[发布运营](PJ2026-0206-publishing-operations.md) | + +进度布局必须满足： + +- 顶部显示当前场景内部细进度； +- 底部显示总阶段导航和总进度； +- 底部导航与总进度在 X 轴使用同一内容宽度和起止位置； +- 用户可以按进度位置理解或跳转到对应阶段。 + +阶段导航必须满足： + +- 所有阶段等宽、等高； +- 导航内只显示简短标题； +- 标题过长时允许换行，并统一增加全部阶段的 Y 轴高度； +- 不能截断、使用省略号或让单个阶段独自变高； +- 标题长度和导航最大高度由配置给出上限。 + +场景数量和标题来自当期场景模型。HTML 预览和 Web 编辑台可以点击章节跳转，成片中的导航则以高亮和进度提供同一视觉语义。 + +### 6.5 VISUAL-L1-REQ-005 确定性动态画面 + +| 编号 | 短名 | 主责模块 | 关联模块 | +| --- | --- | --- | --- | +| VISUAL-L1-REQ-005 | 动效渲染 | PJ2026-020305 动效渲染 | [音频成片](PJ2026-0204-audio-video-production.md)、[流水线平台](PJ2026-0205-pipeline-platform.md) | + +系统应以场景内绝对时间计算所有生产动效，使从任意时间点渲染的画面与从头播放到该点的画面一致。 + +来源页面滚动应使用本地采集帧和确定性插值形成向下浏览效果。进场、强调、进度和卡片动效应服务信息理解，不能造成文字晃动、阅读时间不足或字幕冲突。 + +浏览器渲染必须在禁用后台网络依赖的生产环境中运行。指定画布、捕获帧率和时间点应产生稳定尺寸与场景身份；中断重跑不得因上次浏览器状态改变结果。 + +### 6.6 VISUAL-L1-REQ-006 多工件视觉交付 + +| 编号 | 短名 | 主责模块 | 关联模块 | +| --- | --- | --- | --- | +| VISUAL-L1-REQ-006 | 视觉工件 | PJ2026-020306 视觉工件 | [音频成片](PJ2026-0204-audio-video-production.md)、[发布运营](PJ2026-0206-publishing-operations.md) | + +系统应从同一场景模型生成可直接预览的 HTML、逐场景 PNG、PPTX 和封面候选，并登记尺寸、页数、路径、字节数和哈希。 + +多工件要求如下： + +- PPTX 可以把 HTML 画面封装为高保真演示页； +- PPTX 应保持标准宽高比、页数和预览一致； +- PNG 应支持人工快速抽检和无浏览器场景预览； +- 封面候选应由栏目视觉生成，不复制来源站点封面。 + +视觉交付完成标准包括关键文字不越界、字幕安全区无关键信息、来源标识清楚、长标题可读、参考来源页可辨识、无意外网络依赖和无未登记第三方资产。 + +## 7. 过程控制 + +### 7.1 当前实现状态与引用 + +- 当前已经实现来源页面采集、无图降级、暖色 HTML 动效、PNG、PPTX 和双层进度导航。 +- 当前 R3 已生成九个场景和入选资讯的本地滚动页面帧。 +- 第 6 章描述稳定目标；多栏目主题治理和正式素材授权台账仍需后续实现。 + +- 主题与场景配置：`/root/selfmedia/config/editorial.yaml`、`config/tools.yaml`。 +- 来源视觉采集：`/root/selfmedia/scripts/src/steps/visuals.ts`。 +- 场景和演示文稿：`/root/selfmedia/scripts/src/steps/slides.ts`。 +- HTML 生成与浏览器渲染： + - `/root/selfmedia/scripts/src/slide-html.ts`； + - `/root/selfmedia/scripts/src/slide-browser.ts`。 +- 期次工件：`visuals/`、`slides/html/`、`slides/png/`、`slides/deck.pptx`、`output/cover.png`。 +- 后续相关源码修改应标记： + - `SPEC: PJ2026-0203 视觉制作 draft-2026-07-13-p0`。 + +### 7.2 原入口验收 + +- 通过 TypeScript CLI 分别单步运行来源视觉和幻灯步骤。 +- 核对每条入选资讯的视觉状态、来源链接、帧数、滚动位置和失败降级。 +- 在浏览器中检查开场、大纲、至少两条故事、总结和来源页的开始、中间、结束状态。 +- 检查顶部细进度、底部总导航、总进度 X 轴对齐、阶段等宽等高和长标题换行。 +- 核对 HTML、PNG、PPTX 的场景数、顺序、尺寸、标题和来源一致。 +- 在桌面和窄屏 Web 入口检查章节导航可点击且无横向溢出。 + +### 7.3 版权和失败控制 + +- 页面采集只使用实际采用来源和必要可见范围，并保留来源归因。 +- 来源视觉失败时允许无图降级，不允许因此阻断有充分事实证据的整期内容。 +- 视觉清单中的版权提示是人工复核输入，不构成自动授权结论。 +- 新增外部字体、图片、图标、音频或模板前必须确认许可和分发边界。 +- 视觉效果无法通过绝对时间复现、需要联网才能生产或遮挡字幕时不得进入发布候选。 + +### 7.4 回写边界 + +- 页面身份、许可提示或来源策略问题回写 [资讯源](PJ2026-0201-information-sources.md)。 +- 场景内容、标题或信息层级来源于文稿的问题回写 [编辑策划](PJ2026-0202-editorial-planning.md)。 +- 时间线、字幕安全区实测或编码问题回写 [音频成片](PJ2026-0204-audio-video-production.md)。 +- 章节跳转、工件展示或作业可见性问题回写 [流水线平台](PJ2026-0205-pipeline-platform.md)。 +- 封面选择和平台展示问题回写 [发布运营](PJ2026-0206-publishing-operations.md)。 diff --git a/project-management/PJ2026-02/specs/PJ2026-0204-audio-video-production.md b/project-management/PJ2026-02/specs/PJ2026-0204-audio-video-production.md new file mode 100644 index 00000000..24e7fd1f --- /dev/null +++ b/project-management/PJ2026-02/specs/PJ2026-0204-audio-video-production.md @@ -0,0 +1,308 @@ +# PJ2026-0204 音频成片需求规格 + +## 修改历史 + +| 版本 | 对应 commit id | 更新日期 | 变更说明 | +| --- | --- | --- | --- | +| v0.1 | 待提交 | 2026-07-13 | 基于本地 Kokoro 配音、可调语速、字幕时间线、FFmpeg 成片和自动检查建立音频成片规格。 | + +当前正文仍在规格治理草稿中；未定稿前不新增版本号，不为单次编辑追加 `待提交` 版本。 + +## 正文 + +## PJ2026-0204 音频成片需求规格 + +## 1. 文档控制 + +| 字段 | 内容 | +| --- | --- | +| 编号 | PJ2026-0204 | +| 短名 | 音频成片 | +| 层级 | L1 方向 | +| 状态 | 已生效 | +| 实现引用版本 | draft-2026-07-13-p0 | +| 需求规格模板 | [ISO/IEC/IEEE 29148 需求规格模板](../../templates/iso-iec-ieee-29148-requirements-spec-template.md) | +| 上级规格 | [PJ2026-02 智媒工厂总规格](PJ2026-02-smart-media-factory.md) | +| 规格治理索引 | [PJ2026-02 智媒工厂总规格](PJ2026-02-smart-media-factory.md) 第 7 章 | + +本文采用 ISO/IEC/IEEE 29148 需求规格模板的项目裁剪版。正文只定义本地配音、时间线、字幕、音视频合成和媒体质量的稳定需求。 + +## 2. 目的和范围 + +### 2.1 目的 + +音频成片的目标是： + +- 把已审核文稿和确定性视觉场景合成为自然、清晰、同步的视频候选； +- 使用本地开源语音、可调语速和精确时间线； +- 支持可重复生成； +- 让编辑者只重跑声音、字幕或渲染； +- 避免因媒体参数变化而重做资讯采集和编辑策划。 + +### 2.2 范围内 + +- 中文和技术术语的口播文本规范化、分段、读音检查入口和本地 TTS 推理。 +- 音色、基础速度、后处理语速、停顿、响度、采样率和缓存策略。 +- 分段 WAV、整轨 WAV、句级时间线和场景时长分配。 +- SRT、ASS 字幕的切分、换行、样式、尾部停顿和字幕安全区关系。 +- HTML 场景逐帧捕获、字幕烧录、音视频编码、封面和场景时间线。 +- 媒体流、分辨率、时长、响度、峰值、字幕尾部和完整解码检查。 + +### 2.3 范围外 + +- 文稿事实、叙事和专业对象说明，归 [编辑策划](PJ2026-0202-editorial-planning.md)。 +- 场景版式、来源画面和动效语义，归 [视觉制作](PJ2026-0203-visual-production.md)。 +- 使用外部付费配音、数字人、视频生成或云渲染服务。 +- 把自动响度、媒体探测或字幕生成结果当作最终听感和表达批准。 +- 未经授权复用第三方音乐、音效、配音或视频片段。 + +## 3. 术语表 + +| 术语 | 定义 | +| --- | --- | +| 配音单元 | 与开场、大纲、故事、总结或来源场景对应的结构化可朗读文本。 | +| 原始语音缓存 | 按文本、模型、音色和缓存版本索引的 TTS 原始推理结果。 | +| 后处理语速 | 在原始 TTS 输出后以确定倍率调整时长的音频处理参数。 | +| 句级时间线 | 每个配音分段的场景身份、开始、结束、停顿和音频路径记录。 | +| 字幕提示 | 具有开始、结束、文本和场景身份的单条 SRT/ASS 内容。 | +| 场景时间线 | 最终成片中每个视觉场景的真实起止时间和章节身份。 | +| 响度归一 | 将整轨旁白处理到目标综合响度和真峰值约束的过程。 | +| 发布候选视频 | 通过自动技术检查、等待人工复核和批准的最终编码 MP4。 | + +## 4. 系统边界和接口 + +本规格把音频成片作为媒体合成和技术质量能力看待，不负责内容事实和最终发布决定。 + +| 边界项 | 内容 | +| --- | --- | +| 外部使用者 | 编辑者、视频制作者、发布审核者和发布运营。 | +| 外部输入 | 配音单元、场景模型、HTML 动效、TTS/字幕/媒体 YAML、本地模型和可选背景音乐路径。 | +| 受控资源 | 原始语音缓存、分段音频、整轨音频、时间线、字幕、动效帧、MP4、封面和检查报告。 | +| 外部输出 | WAV、时间线 JSON、SRT、ASS、MP4、封面、场景时间线、媒体清单和检查结果。 | +| 用户接口 | TypeScript CLI 的配音、字幕、渲染和检查单步，音视频播放器及文件工件。 | +| 系统边界 | 音频成片负责“声音、字幕和画面是否技术同步并可播放”；不决定内容是否真实或是否公开发布。 | + +## 5. 内部分工与下级索引 + +| 编号 | 模块或课题 | 规格位置 | 主责边界 | 上游依赖 | 下游支撑 | +| --- | --- | --- | --- | --- | --- | +| PJ2026-020401 | 本地配音 | 本规格 6.1 | 文本规范化、TTS、声音、语速和缓存 | 编辑策划、本地模型 | 音频时间线 | +| PJ2026-020402 | 音频时间 | 本规格 6.2 | 分段、停顿、整轨、响度和句级时间线 | 本地配音 | 字幕、场景时长 | +| PJ2026-020403 | 字幕生成 | 本规格 6.3 | 提示切分、时间码、SRT/ASS 和尾部关系 | 音频时间 | 视频合成 | +| PJ2026-020404 | 视频合成 | 本规格 6.4 | 场景帧、音频、字幕、编码、原子落盘和清理 | 视觉制作、字幕生成 | 媒体检查 | +| PJ2026-020405 | 媒体检查 | 本规格 6.5 | 流、尺寸、时长、响度、峰值、解码和工件完整性 | 全部媒体环节 | 发布运营 | +| PJ2026-020406 | 媒体重跑 | 本规格 6.6 | 参数影响范围、缓存复用和单步重新生成 | 流水线平台 | 编辑调试 | + +本章图形描述目标数据面。当前已实现范围和未实现差距见第 7.1 节。 + +### 5.1 目标数据面架构图 + +```mermaid +flowchart LR + subgraph Inputs[媒体输入] + Script[配音单元] + Scenes[HTML 场景] + Config[媒体 owning YAML] + Models[本地 TTS 模型] + end + subgraph MediaPlane[音频成片数据面] + Voice[配音与缓存] + Timeline[音频时间线] + Subtitle[字幕生成] + Capture[确定性画面捕获] + Encoder[FFmpeg 合成] + Inspect[媒体技术检查] + end + Script --> Voice + Models --> Voice + Config --> Voice + Voice --> Timeline + Timeline --> Subtitle + Scenes --> Capture + Timeline --> Capture + Capture --> Encoder + Subtitle --> Encoder + Timeline --> Encoder + Encoder --> Inspect + Inspect --> Candidate[发布候选媒体] +``` + +音频时间线是字幕和场景时长的共同时间事实。编码器只消费已登记输入，不负责修改文稿或场景语义。 + +### 5.2 目标数据流图 + +```mermaid +flowchart TD + Units[配音单元] --> Normalize[口播规范化与分段] + Normalize --> Cache{原始语音缓存命中} + Cache -->|命中| Raw[原始分段 WAV] + Cache -->|未命中| TTS[本地 TTS 推理] + TTS --> Raw + Raw --> Tempo[语速、停顿与响度] + Tempo --> Audio[整轨 WAV 与句级时间线] + Audio --> Cues[SRT / ASS] + Audio --> SceneTime[场景时长] + SceneTime --> Frames[HTML 绝对时间帧] + Frames --> Encode[音视频编码] + Cues --> Encode + Audio --> Encode + Encode --> Video[MP4、封面与场景时间线] + Video --> Check[媒体检查报告] +``` + +缓存命中只复用与文本、模型、声音和缓存版本一致的原始语音。后处理结果和最终时间线必须按本次配置重新生成。 + +### 5.3 关键时序图 + +```mermaid +sequenceDiagram + participant E as 编辑者 + participant V as 配音步骤 + participant T as 本地 TTS + participant S as 字幕步骤 + participant R as 渲染步骤 + participant I as 检查步骤 + E->>V: 提交配音单步 + V->>T: 按分段查询缓存或本地推理 + T-->>V: 原始 WAV + V->>V: 调速、停顿、拼接、响度归一 + V-->>E: 整轨 WAV 与句级时间线 + E->>S: 提交字幕单步 + S-->>E: SRT / ASS + E->>R: 提交渲染单步 + R->>R: 按场景绝对时间捕获画面 + R->>R: 合成音频、字幕与视频 + R-->>E: MP4、封面与场景时间线 + E->>I: 提交检查单步 + I-->>E: 技术检查结果与人工复核提示 +``` + +每一步都应使用已登记上游工件并可独立重跑。修改语速不应自动重新调用文稿 LLM；只改字幕样式不应重新执行 TTS。 + +## 6. 原子需求 + +### 6.1 MEDIA-L1-REQ-001 本地可配置配音 + +| 编号 | 短名 | 主责模块 | 关联模块 | +| --- | --- | --- | --- | +| MEDIA-L1-REQ-001 | 本地配音 | PJ2026-020401 本地配音 | [编辑策划](PJ2026-0202-editorial-planning.md)、[流水线平台](PJ2026-0205-pipeline-platform.md) | + +系统应使用本地开源中文 TTS 生成旁白，并由 owning YAML 管理引擎、模型、声音、采样率、分段长度、语速、停顿、缓存和响度参数。 + +口播文本规范化应： + +- 处理注册符号、下划线、连接符、英文缩写和异常空白； +- 同时保留原始结构化文稿； +- 不改写事实或把技术标识变成错误词义； +- 在读音不确定时提供人工词表或改稿入口。 + +声音选择应优先自然、清晰和有轻微表现力，避免长篇机械平读。系统应允许未来替换更新的本地模型，但模型许可、文件哈希、运行资源和音色兼容性必须可复核。 + +### 6.2 MEDIA-L1-REQ-002 精确音频时间线 + +| 编号 | 短名 | 主责模块 | 关联模块 | +| --- | --- | --- | --- | +| MEDIA-L1-REQ-002 | 音频时间 | PJ2026-020402 音频时间 | PJ2026-020401 本地配音、[视觉制作](PJ2026-0203-visual-production.md) | + +系统应为每个配音分段记录场景身份、文本、音频路径、开始时间、语音结束、停顿和总结束时间，并据此拼接整轨旁白和分配场景时长。 + +后处理语速应以确定倍率改变分段和整轨时长，停顿应固化进时间线。字幕和视频必须读取时间线工件，而不是重新读取后来可能变化的 YAML 停顿值。 + +整轨音频应实施可配置响度归一，并保留目标综合响度、真峰值、响度范围和实际检测值。拼接过程不得丢段、重段或把场景身份从时间线上移除。 + +### 6.3 MEDIA-L1-REQ-003 可读同步字幕 + +| 编号 | 短名 | 主责模块 | 关联模块 | +| --- | --- | --- | --- | +| MEDIA-L1-REQ-003 | 同步字幕 | PJ2026-020403 字幕生成 | PJ2026-020402 音频时间、[视觉制作](PJ2026-0203-visual-production.md) | + +系统应从句级时间线和实际配音文本生成 SRT 与 ASS，使两种字幕的文本、顺序和场景关系一致，并在格式量化允许误差内保持时间同步。 + +字幕切分应由最大字符数、最小字符数和最大行数约束，优先按中文标点和语义边界断句。长技术标识不应被拆到难以理解的位置，短词不能孤立闪现。 + +字幕样式应与视觉主题协调并位于安全区，具备足够字号、描边和背景对比。最后一条字幕应在音频结尾前保留配置的尾部停顿，不覆盖片尾自然结束。 + +### 6.4 MEDIA-L1-REQ-004 确定性视频合成 + +| 编号 | 短名 | 主责模块 | 关联模块 | +| --- | --- | --- | --- | +| MEDIA-L1-REQ-004 | 视频合成 | PJ2026-020404 视频合成 | [视觉制作](PJ2026-0203-visual-production.md)、PJ2026-020403 字幕生成 | + +系统应按音频和场景时间线捕获确定性 HTML 动效帧，并使用本地开源媒体工具合成字幕、音频和视频。 + +视频编码参数由 owning YAML 管理，包括： + +- 帧率、编解码器、码率和像素格式； +- 质量、预设、快速起播和字体。 + +背景音乐默认为空。如启用，必须有明确授权、音量策略和人工听感复核。 + +成片应先写临时文件并在成功后原子替换正式 MP4。失败、中断或取消后应清理临时视频和可再生帧缓存，同时保留作业日志和失败证据。 + +### 6.5 MEDIA-L1-REQ-005 媒体技术验收 + +| 编号 | 短名 | 主责模块 | 关联模块 | +| --- | --- | --- | --- | +| MEDIA-L1-REQ-005 | 媒体检查 | PJ2026-020405 媒体检查 | [发布运营](PJ2026-0206-publishing-operations.md)、[流水线平台](PJ2026-0205-pipeline-platform.md) | + +系统应对发布候选执行自动技术检查，至少验证： + +- 视频流、音频流和分辨率； +- 音视频时长和场景时间线对齐； +- 声明工件存在和来源引用可解析； +- 响度、真峰值和字幕尾部停顿。 + +检查应使用实际 MP4、WAV、字幕、场景时间线、工件清单、结构化文稿和来源清单，不能只验证作业返回成功。必要时应完整解码候选视频以发现尾部损坏或编码中断。 + +所有必要检查为真只表示候选满足技术门槛。检查报告必须保留人工复核提示，明确读音、字幕观感、视觉版权、事实和平台合规仍需发布者判断。 + +### 6.6 MEDIA-L1-REQ-006 有界重跑与缓存 + +| 编号 | 短名 | 主责模块 | 关联模块 | +| --- | --- | --- | --- | +| MEDIA-L1-REQ-006 | 媒体重跑 | PJ2026-020406 媒体重跑 | [流水线平台](PJ2026-0205-pipeline-platform.md)、[编辑策划](PJ2026-0202-editorial-planning.md) | + +系统应支持配音、字幕、渲染和检查分别单步重跑，并依据参数影响范围复用安全缓存。 + +原始语音缓存键必须包含文本、模型、声音和缓存版本。只改后处理语速、停顿或响度时可复用原始 TTS；改变文本、模型、声音或规范化语义时必须失效相关缓存。 + +每次重跑应生成新的作业记录和结果摘要。正式工件可以更新为最新候选，但历史作业、成本和失败证据不得被改写为同一次成功。 + +## 7. 过程控制 + +### 7.1 当前实现状态与引用 + +- 当前已经实现本地 Kokoro 配音、缓存、可调语速、响度、句级时间线、SRT/ASS、FFmpeg 成片和十项技术检查。 +- 当前检查结果只说明媒体技术门槛通过，不提供事实、版权或发布批准状态。 +- 第 6 章描述可替换模型和持续媒体质量的目标口径。 + +- 媒体 owning YAML：`/root/selfmedia/config/tools.yaml`。 +- 配音：`/root/selfmedia/scripts/src/steps/voice.ts`、`tools/kokoro_tts.py`。 +- 字幕：`/root/selfmedia/scripts/src/steps/subtitles.ts`。 +- 渲染：`/root/selfmedia/scripts/src/steps/render.ts`、`scripts/src/slide-browser.ts`。 +- 检查：`/root/selfmedia/scripts/src/steps/inspect.ts`。 +- 期次工件：`audio/`、`subtitles/`、`output/episode.mp4`、`output/scene-timeline.json`、`output/inspection.json`。 +- 后续相关源码修改应标记： + - `SPEC: PJ2026-0204 音频成片 draft-2026-07-13-p0`。 + +### 7.2 原入口验收 + +- 通过 TypeScript CLI 分别单步运行 `voice`、`subtitles`、`render` 和 `inspect`。 +- 使用媒体探测核对 WAV 与 MP4 时长、采样率、编码、帧率、分辨率、响度和真峰值。 +- 核对分段时间线连续、场景身份完整、字幕文本一致和最后字幕尾部停顿。 +- 抽听开头、中段、技术术语密集段和结尾，检查机械感、语速、停顿、读音和削波。 +- 抽看各场景关键时间点和最终来源页，检查字幕安全区、动效同步和编码完整性。 + +### 7.3 依赖和许可控制 + +- TTS 模型、声音文件、字体和媒体组件必须在工具 YAML 或第三方清单中保留版本、来源、哈希和许可边界。 +- 依赖安装缺失时应通过受控工具安装流程恢复，不使用未登记系统路径静默替代。 +- 新模型必须先验证中文、英文术语、许可证、CPU/GPU 资源、推理速度和可复现部署，再改变默认声音。 +- 任何外部音乐、音效或素材在许可未明确前保持禁用。 + +### 7.4 回写边界 + +- 朗读内容本身冗长、含错误符号或事实表达不适合口播时回写 [编辑策划](PJ2026-0202-editorial-planning.md)。 +- 场景动效、关键信息位置或安全区问题回写 [视觉制作](PJ2026-0203-visual-production.md)。 +- 作业取消、缓存、进度、日志或工件索引问题回写 [流水线平台](PJ2026-0205-pipeline-platform.md)。 +- 技术检查通过后的事实、版权、平台和发布批准问题回写 [发布运营](PJ2026-0206-publishing-operations.md)。 diff --git a/project-management/PJ2026-02/specs/PJ2026-0205-pipeline-platform.md b/project-management/PJ2026-02/specs/PJ2026-0205-pipeline-platform.md new file mode 100644 index 00000000..b360f3ec --- /dev/null +++ b/project-management/PJ2026-02/specs/PJ2026-0205-pipeline-platform.md @@ -0,0 +1,346 @@ +# PJ2026-0205 流水线平台需求规格 + +## 修改历史 + +| 版本 | 对应 commit id | 更新日期 | 变更说明 | +| --- | --- | --- | --- | +| v0.1 | 待提交 | 2026-07-13 | 基于 YAML-first、TypeScript CLI、异步作业、用量计费和 Web 编辑台建立流水线平台规格。 | + +当前正文仍在规格治理草稿中；未定稿前不新增版本号，不为单次编辑追加 `待提交` 版本。 + +## 正文 + +## PJ2026-0205 流水线平台需求规格 + +## 1. 文档控制 + +| 字段 | 内容 | +| --- | --- | +| 编号 | PJ2026-0205 | +| 短名 | 流水线平台 | +| 层级 | L1 方向 | +| 状态 | 已生效 | +| 实现引用版本 | draft-2026-07-13-p0 | +| 需求规格模板 | [ISO/IEC/IEEE 29148 需求规格模板](../../templates/iso-iec-ieee-29148-requirements-spec-template.md) | +| 上级规格 | [PJ2026-02 智媒工厂总规格](PJ2026-02-smart-media-factory.md) | +| 规格治理索引 | [PJ2026-02 智媒工厂总规格](PJ2026-02-smart-media-factory.md) 第 7 章 | + +本文采用 ISO/IEC/IEEE 29148 需求规格模板的项目裁剪版。正文只定义配置、调度、作业、工件、计量和可视控制面的稳定运行能力。 + +## 2. 目的和范围 + +### 2.1 目的 + +流水线平台的目标是： + +- 让各生产能力能够被一致配置、单步调试、组合运行、观察、重试和复现； +- 坚持 CLI-first； +- 让每个步骤先具备稳定的 TypeScript CLI 契约； +- 让完整流水线和 Web 复用同一调度与作业事实； +- 避免可视界面成为另一套不可调试实现。 + +### 2.2 范围内 + +- owning YAML、`configRef`、模式校验、配置计划、依赖计划和失败策略。 +- 期次创建、步骤目录、单步运行、区间运行和完整流水线组合。 +- 异步作业、状态、进度、步骤状态、结构化日志、结果、取消、重试和终态。 +- 期次工件契约、原子写入、路径安全、字节数、哈希、媒体类型和清单。 +- 同源 HTTP API、Web 编辑台、媒体范围请求、服务器生命周期和公网 IP 预览。 +- LLM 请求级 Token、质量门结果、公开目录价估算、汇总和 Web/CLI 展示。 +- 本地开源依赖安装、版本与模型哈希检查，以及无付费媒体生成服务边界。 + +### 2.3 范围外 + +- 定义选题、文稿、视觉、音频或发布结果是否业务正确；这些由相应 L1 负责。 +- 把某个 CLI 命令、Web 页面、服务端口、框架或仓库本身作为能力目标。 +- 从运行环境、日志或 API 响应反解并保存 Secret。 +- 无明确用户授权的多租户公网生产服务、账号系统、自动扩容、商业账单或平台支付。 +- 为追求“全自动”而移除人工中间件、单步入口或失败证据。 + +## 3. 术语表 + +| 术语 | 定义 | +| --- | --- | +| CLI-first | 所有正式生产能力先通过可组合的 TypeScript CLI 和工件契约实现，再由其他入口复用。 | +| 单步 | 只执行一个正式生产阶段，并明确读取上游工件、输出结果和失败状态。 | +| 区间流水线 | 在步骤 DAG 中从指定起点到终点运行闭区间内步骤。 | +| 异步作业 | 提交后立即返回标识，在后台运行并通过状态、日志和结果观察的任务。 | +| 作业事实 | 作业请求、状态、进度、步骤状态、时间、日志路径、结果路径、进程和错误的权威记录。 | +| 工件清单 | 记录期次工件路径、媒体类型、字节数、哈希和生成步骤的结构化索引。 | +| 配置计划 | 只读展示有效配置、引用来源和摘要的 CLI 输出，不执行生产变更。 | +| 公开目录价估算 | 按公开模型价格和响应 Token 用量计算的参考成本，不等价于实际账单。 | +| 同源控制面 | Web 页面和 HTTP API 由同一服务 origin 提供，并读取同一作业与期次事实。 | +| 工业控制台 | 面向生产调试的紧凑高信息密度 Web，使用直角或极小倒角，不复用媒体内容主题。 | + +## 4. 系统边界和接口 + +本规格把流水线平台作为生产调度和可观察控制面看待，不接管各内容能力域的正确性定义。 + +| 边界项 | 内容 | +| --- | --- | +| 外部使用者 | 编辑者、开发调试者、自动化任务、Web 预览用户和发布审核者。 | +| 外部输入 | owning YAML、CLI/API 请求、期次标识、步骤范围、人工文件、Secret 引用和本地依赖。 | +| 受控资源 | 配置快照、作业记录、日志、结果、期次目录、工件清单、用量记录、服务状态和前端构建。 | +| 外部输出 | 机器可解析 CLI/API 响应、作业进度、日志、结果、工件链接、费用摘要和 Web 控制面。 | +| 用户接口 | TypeScript CLI、HTTP API、Web 编辑台和文件系统工件。 | +| 系统边界 | 平台负责“如何可靠运行和观察”；各 L1 负责“结果为何正确”。 | + +## 5. 内部分工与下级索引 + +| 编号 | 模块或课题 | 规格位置 | 主责边界 | 上游依赖 | 下游支撑 | +| --- | --- | --- | --- | --- | --- | +| PJ2026-020501 | 配置治理 | 本规格 6.1 | owning YAML、引用、模式、计划、Secret 边界 | 全部能力域配置 | CLI 调度、作业执行 | +| PJ2026-020502 | 步骤调度 | 本规格 6.2 | 步骤 DAG、单步、区间、完整流程和同一实现 | 各 L1 步骤契约 | 作业运行 | +| PJ2026-020503 | 作业运行 | 本规格 6.3 | 异步执行、状态、进度、日志、结果、取消和重试 | 步骤调度 | CLI、API、Web | +| PJ2026-020504 | 工件事实 | 本规格 6.4 | 期次目录、输入输出、原子落盘、清单、哈希和路径安全 | 各生产步骤 | Web、发布运营 | +| PJ2026-020505 | 用量成本 | 本规格 6.5 | LLM 请求用量、公开目录价、质量门结果和汇总 | 编辑策划、价格 YAML | CLI、Web、发布复核 | +| PJ2026-020506 | 可视控制 | 本规格 6.6 | 同源 API、Web、媒体预览、服务生命周期和公网入口 | 作业事实、工件事实 | 编辑者和审核者 | + +本章图形描述目标数据面。当前已实现范围和未实现差距见第 7.1 节。 + +### 5.1 目标数据面架构图 + +```mermaid +flowchart LR + YAML[owning YAML / configRefs] --> Config[配置校验与有效计划] + CLI[TypeScript CLI] --> Jobs[异步作业核心] + API[同源 HTTP API] --> Jobs + Web[Web 编辑台] --> API + Config --> Jobs + Jobs --> Runner[步骤调度器] + Runner --> Steps[各 L1 正式步骤] + Steps --> Artifacts[期次工件与清单] + Steps --> Logs[结构化日志与进度] + Steps --> Usage[LLM 用量与费用] + Artifacts --> API + Logs --> API + Usage --> API +``` + +CLI 和 API 只能通过同一异步作业核心运行步骤。Web 不直接执行媒体命令或修改期次文件，以免形成第二套状态和失败语义。 + +### 5.2 目标数据流图 + +```mermaid +flowchart TD + Request[CLI / API 作业请求] --> Validate[配置与期次校验] + Validate --> Job[持久化作业事实] + Job --> Worker[后台步骤运行器] + Worker --> Input[读取已登记上游工件] + Input --> Step[执行选中正式步骤] + Step --> Progress[进度与结构化日志] + Step --> Output[原子写入步骤工件] + Step --> Usage[LLM 用量与费用记录] + Output --> Result[步骤结果与工件索引] + Progress --> ReadModel[CLI / API 查询投影] + Usage --> ReadModel + Result --> ReadModel + ReadModel --> Web[Web 编辑台] +``` + +作业、日志、用量和工件可以有不同文件，但必须由同一作业标识和期次标识关联。查询投影不得凭空补造步骤成功或媒体工件。 + +### 5.3 关键时序图 + +```mermaid +sequenceDiagram + participant U as CLI / Web 用户 + participant C as CLI / 同源 API + participant J as 异步作业核心 + participant W as 后台运行器 + participant S as 正式步骤 + participant A as 作业与期次工件 + U->>C: 提交单步、区间或完整流水线 + C->>J: 校验请求并创建作业 + J->>A: 写入 queued 作业事实 + J-->>C: 立即返回 jobId + J->>W: 启动后台运行 + loop 每个选中步骤 + W->>S: 传入期次、配置、取消信号 + S->>A: 读取上游并持续写进度日志 + S->>A: 原子写入结果工件 + S-->>W: 返回步骤摘要 + end + W->>A: 写入终态与结果 + U->>C: 查询 status / logs / result + C->>A: 读取同一作业事实 + A-->>U: 返回进度、错误或工件 +``` + +取消必须传播到后台运行器。重试从旧请求创建新作业，不覆盖旧作业的终态、错误、日志和费用。 + +## 6. 原子需求 + +### 6.1 PLATFORM-L1-REQ-001 YAML-first 配置 + +| 编号 | 短名 | 主责模块 | 关联模块 | +| --- | --- | --- | --- | +| PLATFORM-L1-REQ-001 | 配置治理 | PJ2026-020501 配置治理 | [资讯源](PJ2026-0201-information-sources.md)、[编辑策划](PJ2026-0202-editorial-planning.md)、[视觉制作](PJ2026-0203-visual-production.md)、[音频成片](PJ2026-0204-audio-video-production.md) | + +系统应以一个配置入口通过显式 `configRefs` 引用各职责 owning YAML，并在运行前校验版本、类型、必填字段、枚举、数值范围、路径和引用摘要。 + +可调业务事实只能由 owning YAML 定义。代码可以提供模式和渲染逻辑，但不得在字段缺失时静默选择另一套模型、来源、端口、语速、视觉主题、质量门或价格。 + +配置计划和依赖计划必须： + +- 只读； +- 输出机器可解析结果； +- 披露引用路径和内容摘要； +- 只报告 Secret 的存在性和非敏感指纹。 + +Secret 只允许通过受控引用在运行时读取，计划输出不得显示值。 + +### 6.2 PLATFORM-L1-REQ-002 可组合单步流水线 + +| 编号 | 短名 | 主责模块 | 关联模块 | +| --- | --- | --- | --- | +| PLATFORM-L1-REQ-002 | 步骤调度 | PJ2026-020502 步骤调度 | 全部 L1 | + +系统应把以下能力定义为正式步骤： + +- 采集、文稿、来源视觉和幻灯； +- 配音、字幕、渲染和检查。 + +每个步骤必须有明确上游工件、下游工件、进度、日志、结果和失败契约，并可以通过 TypeScript CLI 独立运行。 + +流水线组合必须遵守： + +- 完整流水线和区间流水线调用同一单步实现； +- 不复制步骤逻辑； +- `from` 和 `to` 形成合法闭区间； +- 上游工件缺失或不合法时明确失败； +- 不跨期次查找旧文件补齐。 + +模板文稿模式应作为显式作业参数传递，只影响需要 LLM 的步骤。下游步骤消费统一文稿工件，不需要知道文稿来自 LLM 还是模板。 + +### 6.3 PLATFORM-L1-REQ-003 可观察异步作业 + +| 编号 | 短名 | 主责模块 | 关联模块 | +| --- | --- | --- | --- | +| PLATFORM-L1-REQ-003 | 作业运行 | PJ2026-020503 作业运行 | 全部 L1 | + +所有耗时生产命令应采用 Fire-and-Forget：提交后立即返回作业标识，后台执行，用户通过状态、日志和结果观察，不在 CLI 前台等待网络、LLM、TTS、浏览器或编码完成。 + +作业事实至少包含： + +- 请求、重试来源、排队、开始和结束时间； +- 进程、活动步骤、总进度和步骤状态； +- 日志、结果和结构化错误。 + +长步骤应持续更新进度，不允许数分钟只显示“运行中”而没有可判断位置的消息。 + +取消只作用于指定活动作业并向步骤传播信号。重试创建新的作业标识、保留旧请求和失败证据；已结束作业不能因取消或重试被改写为未发生。 + +### 6.4 PLATFORM-L1-REQ-004 稳定期次与工件契约 + +| 编号 | 短名 | 主责模块 | 关联模块 | +| --- | --- | --- | --- | +| PLATFORM-L1-REQ-004 | 工件事实 | PJ2026-020504 工件事实 | [发布运营](PJ2026-0206-publishing-operations.md)、全部生产 L1 | + +系统应以安全期次标识隔离每一期输入和输出，并保存标题、期数、报道日期和创建时间。正式工件路径应固定、可枚举、可校验，禁止路径穿越和跨期次隐式读取。 + +步骤完成时应登记： + +- 工件相对路径和媒体类型； +- 字节数和 SHA-256。 + +最终清单应覆盖来源、文稿、视觉、音频、字幕、视频、时间线、用量、引用和检查报告。Web 与发布运营不应重新扫描和猜测文件类型。 + +正式媒体和关键 JSON 应原子落盘。中断时允许保留明确标记的可恢复缓存，但不能让半写文件被登记为成功工件。 + +### 6.5 PLATFORM-L1-REQ-005 完整 LLM 用量与成本 + +| 编号 | 短名 | 主责模块 | 关联模块 | +| --- | --- | --- | --- | +| PLATFORM-L1-REQ-005 | 用量成本 | PJ2026-020505 用量成本 | [编辑策划](PJ2026-0202-editorial-planning.md)、[发布运营](PJ2026-0206-publishing-operations.md) | + +系统应为每次 LLM 请求记录期次、作业、尝试、请求标识、请求模型、响应模型、服务层、时间、HTTP 状态、用量状态、Token 明细、质量门结果和费用估算。 + +Token 明细至少区分： + +- 输入、缓存输入和缓存写入； +- 输出、推理和总量。 + +成功接受、质量门拒绝、HTTP 失败、缺失用量和无效用量都应形成记录。任何失败尝试不能从期次汇总中静默消失。 + +模型价格管理要求如下： + +- owning YAML 中的价格从公开官方页面核验； +- 保留币种、单位、服务层、长上下文阈值、抓取时间和来源； +- 未知模型、未知服务层或缺失用量标记为无法估价； +- 无法估价时给出 warning，不能按零元处理； +- CLI 和 Web 明确这是公开目录价估算，而非实际账单。 + +### 6.6 PLATFORM-L1-REQ-006 同源可视控制面 + +| 编号 | 短名 | 主责模块 | 关联模块 | +| --- | --- | --- | --- | +| PLATFORM-L1-REQ-006 | 可视控制 | PJ2026-020506 可视控制 | [发布运营](PJ2026-0206-publishing-operations.md)、全部生产 L1 | + +系统应在 CLI 完整流程可用后提供前后端可视控制面，通过同源 API 展示期次、步骤 DAG、作业、进度、日志、错误、用量和全部工件，并支持与 CLI 等价的受控动作。 + +Web 工厂和控制台应采用工业风视觉： + +- 紧凑高信息密度； +- 清晰的数据表、状态区、分栏和终端区域； +- 直角或极小倒角； +- 克制的状态色和低装饰背景； +- 不复用生成内容的暖色大圆角、纸纹和宽松卡片布局。 + +音频和视频工件应支持 HTTP Range 以便拖动预览。Web 应在桌面和窄屏可用，不以颜色单独表达状态，不依赖外部字体、分析脚本或付费生成服务。 + +服务入口要求如下： + +- 监听、端口、轮询和健康超时由 owning YAML 配置； +- 公网 IPv4 直连只能用于临时验收； +- 明文 HTTP 且无鉴权时必须明确风险； +- 长期公开服务应独立规划 TLS、访问控制和发布权限； +- 不能把临时公网入口默认升级为多用户生产服务。 + +## 7. 过程控制 + +### 7.1 当前实现状态与引用 + +- 当前已经实现 YAML 配置、TypeScript CLI、单步/区间/完整流水线、异步作业、用量费用和同源 Web。 +- 当前工件索引登记步骤输出，但尚未提供跨重跑内容版本兼容性校验和完整发布包。 +- 当前服务是主机进程和临时公网 IPv4 预览，不等价于 [平台交付](PJ2026-0207-platform-delivery.md) 目标运行面。 +- 当前 Web 样式仍需按工业控制台边界收敛；生成媒体的暖色圆角主题不应继续支配控制面。 + +- 配置入口和校验： + - `/root/selfmedia/config/selfmedia.yaml`； + - `/root/selfmedia/scripts/src/config.ts`。 +- CLI、作业和调度： + - `/root/selfmedia/scripts/selfmedia-cli.ts`； + - `/root/selfmedia/scripts/src/jobs.ts`； + - `/root/selfmedia/scripts/src/runner.ts`。 +- 用量和价格：`/root/selfmedia/scripts/src/llm-usage.ts`、`config/pricing.yaml`。 +- 服务和 Web：`/root/selfmedia/scripts/src/server-app.ts`、`scripts/src/server-lifecycle.ts`、`web/src/`。 +- 作业事实：`/root/selfmedia/.state/jobs/`、`.state/results/`、`logs/`。 +- 后续相关源码修改应标记： + - `SPEC: PJ2026-0205 流水线平台 draft-2026-07-13-p0`。 + +### 7.2 原入口验收 + +- 运行配置计划、价格计划、依赖检查和步骤列表，验证输出可解析且不泄露 Secret。 +- 创建独立期次，分别验证一个单步、一个区间流水线和一个默认完整流水线。 +- 在作业运行中查询状态和日志，确认活动步骤、分子分母进度、耗时和错误可见。 +- 验证失败、取消和重试： + - 旧作业事实保留； + - 新作业具有不同标识； + - 不产生半写正式工件。 +- 构建并启动同源 Web，检查桌面、窄屏、媒体 Range、章节跳转、用量、费用和全部工件。 + +### 7.3 安全和运行控制 + +- API key、凭据正文和敏感请求内容不得出现在 CLI、日志、工件或 Web。 +- 公网服务未配置 TLS 和访问控制时只用于有界验收，不存放不应公开的草稿或来源原始响应。 +- 服务状态、健康和日志必须可通过受控 CLI 查看；非明确要求不以手工进程和临时端口替代正式入口。 +- 依赖版本、模型文件和哈希变化先更新 owning YAML，再执行安装计划和检查。 +- 平台错误不得通过降低各 L1 质量门、跳过工件或伪造作业成功来缓解。 + +### 7.4 回写边界 + +- 某步骤结果业务错误回写拥有该结果定义的 L1，不因表面发生在 CLI 或 Web 就归平台。 +- 只有完成标准是配置、调度、状态、日志、结果、成本、工件索引或控制面行为时，才主归属本 L1。 +- 发布批准和平台投放归 [发布运营](PJ2026-0206-publishing-operations.md)。 +- 稳定步骤、作业、用量或工件契约变化时先更新本规格，再进入实现。 diff --git a/project-management/PJ2026-02/specs/PJ2026-0206-publishing-operations.md b/project-management/PJ2026-02/specs/PJ2026-0206-publishing-operations.md new file mode 100644 index 00000000..f08c02d0 --- /dev/null +++ b/project-management/PJ2026-02/specs/PJ2026-0206-publishing-operations.md @@ -0,0 +1,326 @@ +# PJ2026-0206 发布运营需求规格 + +## 修改历史 + +| 版本 | 对应 commit id | 更新日期 | 变更说明 | +| --- | --- | --- | --- | +| v0.1 | 待提交 | 2026-07-13 | 基于技术检查、真实章节时间轴、评论区来源和公网预览建立发布运营规格。 | + +当前正文仍在规格治理草稿中；未定稿前不新增版本号，不为单次编辑追加 `待提交` 版本。 + +## 正文 + +## PJ2026-0206 发布运营需求规格 + +## 1. 文档控制 + +| 字段 | 内容 | +| --- | --- | +| 编号 | PJ2026-0206 | +| 短名 | 发布运营 | +| 层级 | L1 方向 | +| 状态 | 已生效 | +| 实现引用版本 | draft-2026-07-13-p0 | +| 需求规格模板 | [ISO/IEC/IEEE 29148 需求规格模板](../../templates/iso-iec-ieee-29148-requirements-spec-template.md) | +| 上级规格 | [PJ2026-02 智媒工厂总规格](PJ2026-02-smart-media-factory.md) | +| 规格治理索引 | [PJ2026-02 智媒工厂总规格](PJ2026-02-smart-media-factory.md) 第 7 章 | + +本文采用 ISO/IEC/IEEE 29148 需求规格模板的项目裁剪版。正文只定义发布候选、时间轴、引用、人工批准、预览、归档和反馈回流的稳定需求。 + +## 2. 目的和范围 + +### 2.1 目的 + +发布运营的目标是： + +- 把技术生产完成的期次转换为可批准、可退回、可追溯的发布候选； +- 让成片、封面、字幕、评论区时间轴、来源和复核结果保持一致； +- 保留 AI 参与披露； +- 为长期栏目运营保留版本和反馈； +- 不把自动检查通过误认为已经获得事实、版权或平台批准。 + +### 2.2 范围内 + +- 发布候选视频、封面、字幕、引用、评论区文本、检查报告和工件清单的组合。 +- 基于最终场景时间线生成 `MM:SS` 章节时间轴和故事短释。 +- 最终参考来源、媒体归因、AI 参与披露、公开成本提示和人工复核清单。 +- 事实、版权、听感、视觉、优惠期限、平台规则和发布权限的人工批准或退回。 +- 本地或公网 IP 预览、媒体拖动、工件访问和临时验收风险提示。 +- 期次版本留存、发布状态、纠错、撤回、再生成和匿名化反馈回流的长期愿景。 + +### 2.3 范围外 + +- 自动代替发布者登录 B 站或其他平台并公开发布内容。 +- 自动回复评论、私信、广告、商业合作、账户结算或粉丝运营。 +- 把播放量、点赞或完播率作为事实质量和版权合规的替代指标。 +- 重新定义来源、文稿、视觉、音频和平台作业的正确性；问题应回写主责 L1。 +- 在未授权的公网入口公开草稿、原始来源响应、LLM 请求或可能含敏感信息的日志。 + +## 3. 术语表 + +| 术语 | 定义 | +| --- | --- | +| 发布包 | 某一候选版本的成片、封面、字幕、评论区文本、引用、披露和检查报告集合。 | +| 本期时间轴 | 从最终场景时间线派生、按真实故事起点输出的 `MM:SS 短标题 \| 中文短释` 列表。 | +| 本期参考来源 | 只包含实际采用来源的编号名称、标题和完整链接列表。 | +| 人工批准 | 有责任的编辑者完成事实、版权、听感、视觉、平台和权限检查后的明确决定。 | +| 退回重跑 | 指定主责 L1 和问题范围后重新执行一个或多个生产步骤，而不是覆盖旧发布候选。 | +| 发布版本 | 与期次、作业、工件哈希、检查结果和批准决定绑定的不可混淆候选或已发布状态。 | +| 纠错 | 发布前后发现事实、链接、字幕、读音或授权问题时进行记录、修正和版本更新的过程。 | +| 效果反馈 | 来自平台表现和受众反应的聚合信号，只能在去标识和明确用途后回流栏目改进。 | + +## 4. 系统边界和接口 + +本规格把发布运营作为候选组合、批准和持续运营能力看待，不负责生产步骤内部实现。 + +| 边界项 | 内容 | +| --- | --- | +| 外部使用者 | 栏目运营者、最终审核者、发布者和受控预览用户。 | +| 外部输入 | 最终 MP4、封面、字幕、场景时间线、结构化文稿、来源清单、用量、检查报告和人工决定。 | +| 受控资源 | 发布包、评论区文本、批准状态、退回原因、版本索引、纠错记录和反馈摘要。 | +| 外部输出 | 可发布候选、时间轴、参考来源、披露、审核结论、平台预览和回流建议。 | +| 用户接口 | Web 编辑台、文件工件、媒体播放器、发布复核清单和未来平台适配器。 | +| 系统边界 | 发布运营负责“是否具备发布条件以及发布什么版本”；实际平台账户操作仍由授权发布者负责。 | + +## 5. 内部分工与下级索引 + +| 编号 | 模块或课题 | 规格位置 | 主责边界 | 上游依赖 | 下游支撑 | +| --- | --- | --- | --- | --- | --- | +| PJ2026-020601 | 发布包 | 本规格 6.1 | 候选媒体、字幕、封面、来源、披露和检查组合 | 全部生产 L1 | 人工批准 | +| PJ2026-020602 | 时间轴 | 本规格 6.2 | 最终章节起点、短标题、短释和评论区顺序 | 音频成片、编辑策划 | 平台发布文案 | +| PJ2026-020603 | 来源披露 | 本规格 6.3 | 编号来源、完整链接、AI 参与和成本边界 | 资讯源、编辑策划、流水线平台 | 人工批准、受众 | +| PJ2026-020604 | 发布审核 | 本规格 6.4 | 事实、版权、听感、视觉、平台和权限批准 | 发布包、人工判断 | 发布或退回 | +| PJ2026-020605 | 预览交付 | 本规格 6.5 | 本地/公网预览、媒体访问和临时入口风险 | 流水线平台 | 审核者 | +| PJ2026-020606 | 版本反馈 | 本规格 6.6 | 版本、纠错、撤回、再生成和匿名反馈回流 | 发布决定、平台结果 | 全部 L1 改进 | + +本章图形描述目标数据面。当前实现尚未提供持久化批准状态、兼容性发布包或反馈台账；已实现范围和目标差距见第 7.1 节。 + +### 5.1 目标数据面架构图 + +```mermaid +flowchart LR + subgraph Inputs[候选输入] + Media[成片、封面与字幕] + Evidence[文稿、来源与时间线] + Quality[检查、用量与费用] + end + subgraph PublishPlane[发布运营数据面] + Composer[发布包组合器] + Review[人工审核门] + Ledger[目标批准与纠错记录] + Preview[只读预览投影] + Archive[版本与哈希归档] + end + Media --> Composer + Evidence --> Composer + Quality --> Composer + Composer --> Review + Composer --> Preview + Review --> Ledger + Ledger --> Archive + Archive --> Preview + Review --> Rerun[按主责 L1 退回] +``` + +发布包组合器不决定批准。人工审核门产生目标批准或退回决定，预览投影只展示，不得把页面点击直接变成已发布事实。 + +### 5.2 目标数据流图 + +```mermaid +flowchart TD + Media[成片 / 封面 / 字幕] --> Pack[发布包] + Timeline[最终场景时间线] --> Comment[评论区时间轴] + Sources[实际采用来源] --> Comment + Usage[LLM 用量与费用] --> Disclosure[AI 与成本披露] + Inspect[自动技术检查] --> Pack + Comment --> Pack + Disclosure --> Pack + Pack --> Review{人工复核} + Review -->|批准| Ready[可发布版本] + Review -->|退回| Route[按主责 L1 重跑] + Ready --> Archive[版本与哈希归档] + Archive --> Feedback[纠错和效果反馈] +``` + +自动检查是人工复核的输入，不是批准节点。退回必须指向资讯源、编辑策划、视觉制作、音频成片或流水线平台中的唯一主责能力域。 + +### 5.3 关键时序图 + +```mermaid +sequenceDiagram + participant I as 媒体检查 + participant C as 发布包组合器 + participant T as 场景时间线与来源 + participant E as 审核编辑者 + participant L as 目标发布台账 + participant P as 预览入口 + I->>C: 提供技术检查与候选工件 + C->>T: 读取最终章节起点和实际采用来源 + T-->>C: 返回时间轴、引用与内容版本 + C->>C: 校验兼容性并生成发布包 + C->>P: 投影只读候选 + C-->>E: 请求人工事实、版权与平台复核 + alt 人工退回 + E->>L: 记录主责 L1、原因和重跑范围 + L-->>C: 保持候选未批准 + else 人工批准 + E->>L: 记录批准人、时间和工件哈希 + L->>P: 投影可发布版本 + end +``` + +时序图描述目标能力。当前只能生成候选、时间轴、来源和检查材料，尚不能持久化批准人、批准时间或发布状态。 + +## 6. 原子需求 + +### 6.1 PUBLISH-L1-REQ-001 完整发布候选包 + +| 编号 | 短名 | 主责模块 | 关联模块 | +| --- | --- | --- | --- | +| PUBLISH-L1-REQ-001 | 发布包 | PJ2026-020601 发布包 | 全部 L1 | + +系统应为每个候选版本组合最终 MP4、封面、SRT/ASS、结构化文稿、引用目录、评论区文本、场景时间线、用量费用摘要、工件清单和自动检查报告。 + +发布包中的文件必须满足： + +- 来自同一期次； +- 来自相互兼容的最新作业链； +- 视频、字幕、时间线和评论区章节对应同一内容版本； +- 不把重跑前字幕与重跑后视频混合成候选。 + +发布包应有稳定版本标识、生成时间和关键工件哈希。缺失必要工件、技术检查失败或内容版本冲突时必须阻止进入人工批准状态。 + +### 6.2 PUBLISH-L1-REQ-002 真实评论区时间轴 + +| 编号 | 短名 | 主责模块 | 关联模块 | +| --- | --- | --- | --- | +| PUBLISH-L1-REQ-002 | 评论时间轴 | PJ2026-020602 时间轴 | [编辑策划](PJ2026-0202-editorial-planning.md)、[音频成片](PJ2026-0204-audio-video-production.md) | + +系统应从最终成片场景时间线中读取每条故事的真实开始时间，生成以“本期时间轴：”开头的评论区章节列表。 + +每行格式应为 `MM:SS 短标题 | 中文短释`，并遵守： + +- 短标题和短释来自结构化文稿； +- 时间来自最终场景； +- 不使用按字数、目标时长或草稿场景估算的值； +- 开场、大纲、总结和来源页默认不占用故事时间轴行； +- 栏目配置可以明确要求增加其他章节。 + +当语速、停顿、故事顺序或场景时长改变后，渲染或检查步骤应重新生成时间轴。无法解析最终场景时间线时不得继续沿用旧评论区文本。 + +### 6.3 PUBLISH-L1-REQ-003 可核验来源与披露 + +| 编号 | 短名 | 主责模块 | 关联模块 | +| --- | --- | --- | --- | +| PUBLISH-L1-REQ-003 | 来源披露 | PJ2026-020603 来源披露 | [资讯源](PJ2026-0201-information-sources.md)、[编辑策划](PJ2026-0202-editorial-planning.md)、[流水线平台](PJ2026-0205-pipeline-platform.md) | + +系统应在评论区时间轴之后空一行，输出“本期参考来源：”，再按故事顺序列出编号、来源名称、标题和完整链接。 + +来源披露只包含实际采用的来源，并保留媒体、社区、预印本和一手来源角色。引用顺序应与成片故事一致，链接不得被短链、返利链接或无法复核的聚合页替代。 + +发布包披露要求如下： + +- 说明 AI 是否参与文稿、配音或视频制作； +- 展示 LLM 成本时标明 Token 范围； +- 标明公开目录价来源和“非实际账单”边界； +- 不公开 Secret、完整提示词或可能受版权保护的来源全文。 + +### 6.4 PUBLISH-L1-REQ-004 人工发布门 + +| 编号 | 短名 | 主责模块 | 关联模块 | +| --- | --- | --- | --- | +| PUBLISH-L1-REQ-004 | 发布审核 | PJ2026-020604 发布审核 | 全部 L1 | + +系统应要求有责任的编辑者在公开发布前明确批准发布包，并把自动检查通过与人工批准分成不同状态。 + +人工复核至少覆盖： + +- 来源链接、日期、数字、状态和媒体归因； +- 优惠期限、地区、资格、库存和条件； +- 口播事实、专有名词、读音、语速和表达； +- 字幕断句、画面安全区、来源视觉、封面和版权； +- AI 参与披露、平台规则、发布权限和地区法律要求。 + +任何必要项失败时，审核者应记录问题、主责 L1 和重跑范围。系统不能通过隐藏 warning、手工改检查 JSON 或跳过来源页把候选标记为批准。 + +### 6.5 PUBLISH-L1-REQ-005 可控预览与交付 + +| 编号 | 短名 | 主责模块 | 关联模块 | +| --- | --- | --- | --- | +| PUBLISH-L1-REQ-005 | 预览交付 | PJ2026-020605 预览交付 | [流水线平台](PJ2026-0205-pipeline-platform.md) | + +系统应允许审核者通过本地 Web 或受控公网 IP 查看期次 DAG、最终视频、音频、字幕、幻灯、来源、费用和检查结果，并支持媒体拖动与章节跳转。 + +临时公网预览应明确协议、鉴权和内容暴露边界。当前明文 HTTP 且无登录的公网 IPv4 入口只能用于有界验收；不应公开原始响应、LLM 请求、详细日志或未批准草稿。 + +长期外部交付应规划 TLS、访问控制、只读发布包和最小暴露面，但这些能力在规格和授权明确前不得通过临时代理、共享口令或隐藏 URL 假装已经完成。 + +### 6.6 PUBLISH-L1-REQ-006 版本、纠错与反馈回流 + +| 编号 | 短名 | 主责模块 | 关联模块 | +| --- | --- | --- | --- | +| PUBLISH-L1-REQ-006 | 版本反馈 | PJ2026-020606 版本反馈 | 全部 L1 | + +系统应长期建立发布版本、候选版本、工件哈希、批准决定和平台状态之间的关系，使发布前后纠错可以定位到具体内容和生产作业。 + +发生错误时应记录： + +- 发现时间； +- 影响范围； +- 处置决定； +- 替代版本。 + +撤回或更正不能删除原有运行证据，也不能把旧候选重新标为从未发布。 + +效果反馈可以： + +- 按栏目和内容类型聚合播放、跳出、评论和人工反馈； +- 回流选题、文稿、视觉和声音改进。 + +反馈使用必须去除不必要的用户身份，不能为了追求指标自动降低事实、版权和人工批准门槛。 + +## 7. 过程控制 + +### 7.1 当前实现状态与引用 + +- 当前已经实现候选 MP4、封面、字幕、真实故事时间轴、参考来源、费用摘要、技术检查和 Web 预览。 +- 当前没有持久化批准人、批准时间、退回原因、已发布状态、发布版本台账或反馈台账。 +- 当前输出清单不是按内容版本兼容性组合的完整发布包；第 6 章相关条款均为目标需求。 + +- 最终媒体和检查： + - `/root/selfmedia/scripts/src/steps/render.ts`； + - `/root/selfmedia/scripts/src/steps/inspect.ts`。 +- 评论区和引用：`/root/selfmedia/scripts/src/comment-reference.ts`。 +- Web 预览：`/root/selfmedia/scripts/src/server-app.ts`、`web/src/`。 +- 当前期次工件： + - `output/episode.mp4`； + - `output/cover.png`； + - `output/scene-timeline.json`； + - `output/inspection.json`； + - `output/manifest.json`； + - `script/comment-reference.txt`。 +- 后续相关源码修改应标记： + - `SPEC: PJ2026-0206 发布运营 draft-2026-07-13-p0`。 + +### 7.2 原入口验收 + +- 通过检查单步确认必要技术检查全部为真，并查看人工 warning。 +- 从最终场景时间线手工抽算至少两条故事起点，与评论区 `MM:SS` 一致。 +- 核对评论区先时间轴、后参考来源，编号、标题和完整链接与成片顺序一致。 +- 通过 Web 桌面和窄屏入口播放最终视频，测试章节跳转、媒体 Range 和工件链接。 +- 人工执行事实、版权、听感、视觉、优惠期限、AI 披露和平台规则清单后再记录批准。 + +### 7.3 批准和退回控制 + +- 自动检查失败时不得进入人工批准；人工复核失败时不得标记可发布。 +- 退回必须指定唯一主责 L1、问题描述和最小重跑范围。 +- 重跑后重新生成发布包、场景时间线、评论区文本和检查结果；旧批准不自动继承。 +- 当前未实现平台自动发布和账号权限时，发布动作由人工在平台官方入口完成，系统只交付发布候选和配套文本。 + +### 7.4 回写边界 + +- 来源身份、链接、优惠条件或版权提示问题回写 [资讯源](PJ2026-0201-information-sources.md)。 +- 事实、叙事、标题、短释或披露用语问题回写 [编辑策划](PJ2026-0202-editorial-planning.md)。 +- 封面、来源画面、动效或导航问题回写 [视觉制作](PJ2026-0203-visual-production.md)。 +- 读音、字幕、同步、编码或场景时间线问题回写 [音频成片](PJ2026-0204-audio-video-production.md)。 +- 作业、用量、费用、工件索引、API 或 Web 预览问题回写 [流水线平台](PJ2026-0205-pipeline-platform.md)。 diff --git a/project-management/PJ2026-02/specs/PJ2026-0207-platform-delivery.md b/project-management/PJ2026-02/specs/PJ2026-0207-platform-delivery.md new file mode 100644 index 00000000..7fec089b --- /dev/null +++ b/project-management/PJ2026-02/specs/PJ2026-0207-platform-delivery.md @@ -0,0 +1,451 @@ +# PJ2026-0207 平台交付需求规格 + +## 修改历史 + +| 版本 | 对应 commit id | 更新日期 | 变更说明 | +| --- | --- | --- | --- | +| v0.1 | 待提交 | 2026-07-13 | 建立 Web 内置 Codex、隔离凭据、受控 K8s 部署、自动交付和运行验收目标规格。 | + +当前正文仍在规格治理草稿中；未定稿前不新增版本号，不为单次编辑追加 `待提交` 版本。 + +## 正文 + +## PJ2026-0207 平台交付需求规格 + +## 1. 文档控制 + +| 字段 | 内容 | +| --- | --- | +| 编号 | PJ2026-0207 | +| 短名 | 平台交付 | +| 层级 | L1 方向 | +| 状态 | 已生效 | +| 实现状态 | 未实现 | +| 实现引用版本 | draft-2026-07-13-p0 | +| 需求规格模板 | [ISO/IEC/IEEE 29148 需求规格模板](../../templates/iso-iec-ieee-29148-requirements-spec-template.md) | +| 上级规格 | [PJ2026-02 智媒工厂总规格](PJ2026-02-smart-media-factory.md) | +| 规格治理索引 | [PJ2026-02 智媒工厂总规格](PJ2026-02-smart-media-factory.md) 第 7 章 | + +本文采用 ISO/IEC/IEEE 29148 需求规格模板的项目裁剪版。正文定义智媒工厂从本地主机进程演进为可受控交付平台时的稳定目标，不把目标状态写成当前已实现事实。 + +## 2. 目的和范围 + +### 2.1 目的 + +平台交付的目标是： + +- 以可复现、隔离、最小权限和自动验收的方式交付到 Kubernetes 运行面； +- 让用户从 Web 内置终端进入与后端版本一致的 Codex 工作环境； +- 在容器中提供随镜像交付的源码、CLI 和受限工具； +- 通过 YAML-first 配置、GitOps 自动链和公网 IP 原入口持续交付。 + +### 2.2 当前状态 + +当前 `/root/selfmedia` 已具备本地主机上的 CLI、Web 服务、期次作业和公网 IPv4 临时预览，但尚未实现本规格要求的： + +- Web 内置 Codex 终端； +- 包含版本化源码和 CLI 的后端容器运行面； +- PVC 会话持久化、重启自动恢复和 Codex 会话续接； +- YAML-first Secret `sourceRef` 到 `~/.env/TOKEN` 的隔离投影； +- 受控 `~/.codex/*.pika` 源路径到运行时 Codex home 的模型配置与 API key 投影； +- 无 ServiceAccount token、无 kubeconfig、无 Kubernetes 或主机权限的终端沙箱； +- GitHub 到 mirror、PaC、Tekton、GitOps 和 Argo 的自动交付链； +- 由公网 IP 原入口执行的部署后运行验收。 + +### 2.3 范围内 + +- Web 内置终端、Codex 进程、会话身份、终端重连和用户可见状态。 +- 与已部署后端版本一致的容器镜像、源码快照、TypeScript CLI 和运行依赖。 +- 面向 Agent 的受限 Skill、HTTP API 和工件访问能力。 +- PVC 上的 `~/.codex/sessions`、supervisor 恢复元数据、自动恢复和滚动重启续接。 +- 后端 API `~/.env/TOKEN` 的 YAML-first `sourceRef`、运行时投影、权限和脱敏。 +- Codex 模型配置与 API key 从受控 `~/.codex/*.pika` 源路径到运行时 Codex home 的独立投影。 +- 无 ServiceAccount token、无 kubeconfig、无 Kubernetes 管理能力和无主机权限的隔离边界。 +- YAML-first Kubernetes 工作负载、服务、健康检查、资源限制和公网 IP 暴露。 +- GitHub、镜像、Pipeline as Code、Tekton、GitOps、Argo 和部署后验收的自动链。 + +### 2.4 范围外 + +- 允许 Codex 或终端用户直接运行 `kubectl`、读取集群凭据或修改 Kubernetes 对象。 +- 向终端挂载宿主机目录、容器运行时套接字、主机网络、主机进程或特权设备。 +- 在镜像、Git 仓库、日志、工件、ConfigMap、环境变量列表或 Web 响应中保存明文 Token。 +- 让流水线构建步骤直接修改生产 Deployment，或让 Argo 从未经构建验证的工作区读取源码。 +- 把临时 NodePort、手工 SSH、手工复制镜像或主机进程作为正式交付真相。 +- 多租户计费、开放注册和互联网任意用户的共享终端；这些需要独立身份与运营规格。 + +## 3. 术语表 + +| 术语 | 定义 | +| --- | --- | +| 内置终端 | 由工业风智媒工厂 Web 提供、连接到受控后端会话的紧凑浏览器终端。 | +| Agent 工具面 | Codex 在容器内可调用的源码、CLI、受限 Skill、HTTP API 和工件入口集合。 | +| 会话 PVC | 只保存 `~/.codex/sessions` 与 supervisor 恢复元数据的持久卷，不保存源码工作区或集群管理凭据。 | +| 自动恢复 | 终端或后端容器重启后，根据 PVC 中的 Codex session 与 supervisor 恢复元数据续接会话。 | +| `sourceRef` | owning YAML 对外部 Secret 对象、键和目标文件的声明式引用。 | +| `~/.env/TOKEN` | 容器用户家目录下由 Secret 运行时投影生成的 Token 文件目标，不进入镜像或源码。 | +| 运行时 Codex home | 容器内供 Codex 读取模型配置和 API key 的专用目录，只接收受控 `~/.codex/*.pika` 源路径投影。 | +| PaC | Pipeline as Code；把 Git 变更映射为受审查 Tekton 流水线运行的入口。 | +| GitOps | 以声明式目标配置仓库作为部署期望状态，由控制器协调运行面。 | +| 公网 IP 原入口 | 用户实际访问的公网 IPv4 Web、API 和终端入口，用于部署后验收。 | + +## 4. 系统边界和接口 + +本规格把平台交付作为智媒工厂的受控运行与持续交付能力看待，不重新定义内容生产步骤的业务正确性。 + +| 边界项 | 内容 | +| --- | --- | +| 外部使用者 | 智媒编辑者、受控 Codex Agent、发布审核者和交付自动化。 | +| 外部输入 | GitHub 提交、owning YAML、镜像上下文、后端 Secret `sourceRef`、Codex 配置 `sourceRef`、期次工件和终端动作。 | +| 受控资源 | 后端容器、Web 终端、会话 PVC、运行时 Codex home、受限工具、镜像摘要、GitOps 目标、K8s 工作负载和验收记录。 | +| 外部输出 | 公网 IP Web/API/终端、可恢复 Codex session、部署版本、健康状态、自动交付轨迹和验收结论。 | +| 用户接口 | Web 内置终端、TypeScript CLI、同源 HTTP API、公网 IPv4 和只读交付状态。 | +| 系统边界 | 平台拥有应用交付和会话恢复；Agent 只拥有内容工作区和受限工具，不拥有集群或主机控制权。 | + +## 5. 内部分工与下级索引 + +| 编号 | 模块或课题 | 规格位置 | 主责边界 | 上游依赖 | 下游支撑 | +| --- | --- | --- | --- | --- | --- | +| PJ2026-020701 | 内置终端 | 本规格 6.1 | Web 终端、Codex 进程、会话连接、重连和用户状态 | Web、后端容器、会话 PVC | 编辑调试 | +| PJ2026-020702 | Agent 工具 | 本规格 6.2 | 版本化源码、TypeScript CLI、受限 Skill/API 和工件访问 | 构建产物、流水线平台 | Codex Agent | +| PJ2026-020703 | 隔离凭据 | 本规格 6.3 | 后端 Token、Codex 模型配置的独立 `sourceRef`、文件权限、脱敏和负面边界 | owning YAML、外部 Secret | 内置终端、Agent 工具 | +| PJ2026-020704 | 受控部署 | 本规格 6.4 | YAML-first K8s 工作负载、资源、健康和公网 IP | 镜像摘要、配置与 Secret 引用 | 自动交付、运行验收 | +| PJ2026-020705 | 自动交付 | 本规格 6.5 | GitHub、mirror、PaC、Tekton、GitOps 和 Argo 链 | 源码与 owning YAML | 受控部署 | +| PJ2026-020706 | 运行验收 | 本规格 6.6 | 公网原入口、版本、终端、恢复、隔离和自动链证据 | 全部平台交付模块 | 发布运营、平台维护者 | + +本章图形全部描述目标数据面。当前不存在可与这些目标等价的正式 Kubernetes 运行面，当前差距见第 2.2 和第 7.1 节。 + +### 5.1 目标数据面架构图 + +```mermaid +flowchart LR + subgraph Delivery[自动交付控制面] + GitHub[GitHub 源码] + Mirror[受控镜像] + PaC[Pipeline as Code] + Tekton[Tekton 构建与验证] + Registry[摘要固定镜像] + GitOps[GitOps 目标配置] + Argo[Argo 协调] + end + subgraph Runtime[Kubernetes 应用数据面] + Public[公网 IPv4 Service] + Web[Web 与同源 API] + Terminal[终端会话代理] + Backend[后端容器、源码与 CLI] + PVC[会话 PVC] + BackendSecret[只读后端 Token 投影] + CodexSecret[只读 Codex 配置投影] + end + GitHub --> Mirror + Mirror --> PaC + PaC --> Tekton + Tekton --> Registry + Tekton --> GitOps + GitOps --> Argo + Argo --> Backend + Registry --> Backend + Public --> Web + Web --> Terminal + Terminal --> Backend + Backend --> PVC + BackendSecret --> Backend + CodexSecret --> Backend +``` + +自动交付控制面拥有构建和部署协调权限。应用数据面不获得控制面凭据，Codex 只能在后端容器的受控工作区内运行。 + +### 5.2 目标数据流图 + +```mermaid +flowchart TD + Commit[GitHub 提交] --> Sync[mirror 同步] + Sync --> Build[PaC 触发 Tekton] + Build --> Test[源码、CLI 与镜像验证] + Test --> Digest[不可变镜像摘要] + Digest --> Desired[更新 GitOps 期望状态] + Desired --> Reconcile[Argo 协调 K8s] + Config[owning YAML] --> Render[受控配置渲染] + BackendRef[后端 Token sourceRef] --> Render + CodexRef[Codex pika sourceRef] --> Render + Render --> Reconcile + Reconcile --> Pod[后端 Pod] + Pod --> Token[只读 ~/.env/TOKEN] + Pod --> CodexHome[运行时 Codex home] + Pod --> Session[PVC Codex sessions 与恢复元数据] + Browser[公网 IP 浏览器] --> Web[Web 内置终端] + Web --> Codex[容器内 Codex] + CodexHome --> Codex + Codex --> Session + Codex --> Tools[受限 Skill / API / CLI] + Pod --> Acceptance[部署后运行验收] +``` + +Secret 数据不经过 GitHub、mirror、构建日志或 GitOps 明文： + +- 后端 API Token 独立投影到 `~/.env/TOKEN`； +- Codex 模型配置和 API key 从受控 `~/.codex/*.pika` 源路径投影到运行时 Codex home； +- 两类凭据不得共享目标文件、环境变量或日志字段。 + +### 5.3 自动交付关键时序图 + +```mermaid +sequenceDiagram + participant G as GitHub + participant M as mirror / PaC + participant T as Tekton + participant R as 镜像仓库 + participant O as GitOps 目标仓库 + participant A as Argo + participant K as Kubernetes + participant V as 运行验收 + G->>M: 推送受审查提交 + M->>T: 触发 commit-pinned PipelineRun + T->>T: 校验源码、CLI、镜像和配置 + T->>R: 发布不可变镜像摘要 + T->>O: 提交摘要固定的目标配置 + O->>A: 目标 revision 可见 + A->>K: 协调 Deployment、Service 与 Secret 引用 + K-->>A: rollout 健康 + A->>V: 触发部署后原入口验收 + V->>K: 通过公网 IP 检查 Web、API、终端与恢复 + V-->>A: 返回通过或明确阻塞 +``` + +Tekton 不直接修改生产工作负载。只有经过验证并写入 GitOps 目标的镜像摘要，才由 Argo 协调到运行面。 + +### 5.4 终端恢复关键时序图 + +```mermaid +sequenceDiagram + participant U as 编辑者浏览器 + participant W as Web / 终端代理 + participant B as 后端容器 + participant P as 会话 PVC + participant C as Codex CLI + U->>W: 打开内置终端 + W->>B: 创建或连接会话 + B->>P: 读取 Codex sessions 与 supervisor 元数据 + alt 存在可续接 session + B->>C: 使用稳定 sessionId 恢复 + else 新会话 + B->>C: 创建 Codex session + C->>P: 保存 Codex session + B->>P: 保存 supervisor 恢复元数据 + end + C-->>U: 流式终端输出 + B--xW: Pod 滚动重启或连接中断 + U->>W: 自动重连 + W->>B: 连接新容器实例 + B->>P: 恢复 session 与 supervisor 状态 + B->>C: 续接既有 session + C-->>U: 返回恢复后的终端状态 +``` + +自动恢复只读取 PVC 中的业务会话事实。恢复过程不得依赖旧 Pod 内存、宿主机目录、ServiceAccount token 或 kubeconfig。 + +## 6. 原子需求 + +### 6.1 DELIVERY-L1-REQ-001 Web 内置 Codex 终端 + +| 编号 | 短名 | 主责模块 | 关联模块 | +| --- | --- | --- | --- | +| DELIVERY-L1-REQ-001 | 内置终端 | PJ2026-020701 内置终端 | [流水线平台](PJ2026-0205-pipeline-platform.md)、PJ2026-020703 隔离凭据 | + +平台应在智媒工厂 Web 内提供可访问的 Codex 终端，使编辑者能够从同一公网 IP 入口进入与后端版本一致的内容生产环境。 + +内置终端和相邻控制区应遵循控制台视觉边界： + +- 工业风； +- 紧凑高信息密度； +- 直角或极小倒角； +- 优先展示连接、session、进程、命令和错误状态； +- 不使用生成 HTML、PPT 和视频的暖色大圆角卡片主题。 + +终端应显示： + +- 连接、启动和运行状态； +- 重连、恢复、失败和关闭状态。 + +浏览器关闭或网络短暂中断不应自动终止后台 Codex session。重新连接时应回到同一业务会话，不能伪造新的空终端。 + +专用 PVC 应保存： + +- `~/.codex/sessions`； +- supervisor 恢复元数据。 + +会话 PVC 不承担源码工作区持久化。需要跨重启保存的源码变更必须进入受控变更流；未来如需持久化源码工作区，应另行定义数据生命周期、版本冲突和回收要求。 + +后端容器或 Pod 重启后，平台应自动发现并续接可恢复会话。恢复失败时输出明确原因，不以新会话冒充恢复成功。 + +### 6.2 DELIVERY-L1-REQ-002 版本一致的 Agent 工具面 + +| 编号 | 短名 | 主责模块 | 关联模块 | +| --- | --- | --- | --- | +| DELIVERY-L1-REQ-002 | Agent 工具 | PJ2026-020702 Agent 工具 | [流水线平台](PJ2026-0205-pipeline-platform.md)、[发布运营](PJ2026-0206-publishing-operations.md) | + +后端容器镜像应包含与部署 commit 一致的智媒工厂源码、TypeScript CLI、运行依赖和必要开源媒体工具，使 Codex 能在容器内执行正式单步、查看作业和复核工件。 + +Agent 工具面只暴露 owning YAML 允许的： + +- Skill； +- CLI 子命令； +- 同源 HTTP API； +- 期次工件路径。 + +工具策略应声明允许动作、只读或写入边界、超时和输出脱敏。没有声明的外部 API、集群入口和主机能力默认不可用。 + +镜像内源码是版本化实现参考，不是运行时可变部署真相： + +- Codex 不能直接改写已部署容器层或 GitOps 目标； +- 需要持久化的源码修改必须进入受控变更流； +- 会话 PVC 不保存源码工作副本。 + +### 6.3 DELIVERY-L1-REQ-003 YAML-first 隔离凭据 + +| 编号 | 短名 | 主责模块 | 关联模块 | +| --- | --- | --- | --- | +| DELIVERY-L1-REQ-003 | 隔离凭据 | PJ2026-020703 隔离凭据 | PJ2026-020701 内置终端、PJ2026-020702 Agent 工具、PJ2026-020704 受控部署 | + +凭据必须由 owning YAML 使用 `sourceRef`、`targetKey` 和目标路径声明，并分成两个互不复用的投影： + +- 后端 API Token： + - 从受控 Secret `sourceRef` 读取； + - 只读投影到容器用户的 `~/.env/TOKEN`。 +- Codex 模型配置和 API key： + - 从受控 `~/.codex/*.pika` 源路径对应的独立 `sourceRef` 读取； + - 投影到运行时 Codex home 中 Codex 需要的配置和认证目标； + - 不与后端 API Token 共用 Secret 对象、`targetKey`、目标文件或环境变量。 + +两类文件权限都应限制为容器用户可读，凭据值不得进入镜像、Git、ConfigMap、日志、规格或 Web。 + +日志、规格、状态和验收输出只允许披露： + +- `sourceRef`； +- `targetKey`； +- presence； +- 非敏感 fingerprint。 + +平台不得保存、回显或从运行中 Pod、环境变量、进程参数、日志和工件反解凭据值。 + +终端和后端 Pod 必须满足以下负面边界： + +- `automountServiceAccountToken: false` 或等价禁止挂载； +- 不包含 kubeconfig、云控制面凭据或 Kubernetes 客户端配置； +- 不允许访问 Kubernetes API、宿主机文件、容器运行时套接字和特权设备； +- 不使用 `hostPath`、`hostNetwork`、`hostPID`、`hostIPC` 或特权容器； +- 只挂载声明的会话 PVC、只读配置和两类相互隔离的只读 Secret 文件。 + +### 6.4 DELIVERY-L1-REQ-004 YAML-first 受控部署 + +| 编号 | 短名 | 主责模块 | 关联模块 | +| --- | --- | --- | --- | +| DELIVERY-L1-REQ-004 | 受控部署 | PJ2026-020704 受控部署 | PJ2026-020703 隔离凭据、PJ2026-020705 自动交付 | + +owning YAML 应定义： + +- 目标集群、命名空间和镜像摘要； +- 配置引用、Secret `sourceRef` 和 PVC； +- 资源限制、健康检查、服务和公网暴露。 + +代码或手工命令中不得保留第二套部署事实。 + +Kubernetes 工作负载应使用非 root 用户、只读根文件系统或等价最小写面、明确 CPU/内存/临时存储限制，并把可写内容限制在声明的会话 PVC 和必要临时目录。 + +公网入口应通过 YAML 声明的 IPv4 和端口暴露 Web、同源 API 与终端代理。验收使用该公网 IP 原入口，不以 Pod IP、端口转发、SSH、临时 NodePort 或集群内部地址替代。 + +### 6.5 DELIVERY-L1-REQ-005 GitOps 自动交付链 + +| 编号 | 短名 | 主责模块 | 关联模块 | +| --- | --- | --- | --- | +| DELIVERY-L1-REQ-005 | 自动交付 | PJ2026-020705 自动交付 | PJ2026-020704 受控部署、PJ2026-020706 运行验收 | + +平台应建立以下自动交付链： + +- GitHub 提交进入受控 mirror； +- Pipeline as Code 触发 commit-pinned Tekton 流水线； +- Tekton 验证源码、CLI、配置和镜像； +- 验证通过后发布不可变镜像摘要； +- 更新 GitOps 目标； +- Argo 协调 Kubernetes。 + +交付职责边界如下： + +- mirror 只提供受控源码同步，不成为可手工修改的第二真相； +- Tekton 负责构建、检查和产物发布，不直接修改生产 Deployment； +- Argo 只消费通过审核和验证的 GitOps 目标 revision。 + +任何链路失败都应保留 commit、流水线、镜像摘要、GitOps revision、Argo 协调和运行验收之间的追溯关系。自动重试不能跳过失败门禁或回退到手工复制。 + +### 6.6 DELIVERY-L1-REQ-006 公网原入口运行验收 + +| 编号 | 短名 | 主责模块 | 关联模块 | +| --- | --- | --- | --- | +| DELIVERY-L1-REQ-006 | 运行验收 | PJ2026-020706 运行验收 | PJ2026-020701 内置终端、PJ2026-020702 Agent 工具、PJ2026-020703 隔离凭据、PJ2026-020704 受控部署、PJ2026-020705 自动交付 | + +每次自动交付完成后，平台应从 YAML 选中的公网 IPv4 原入口验证健康、Web、同源 API、内置终端、Codex 启动、正式 CLI、工件读取和版本可见性。 + +运行验收应完成以下会话恢复检查： + +- 主动触发一次有界终端会话并记录 sessionId； +- 执行无外部写入的 CLI 只读命令； +- 重启或滚动更新后端； +- 验证同一 session 依靠 PVC 中的 Codex sessions 与 supervisor 元数据自动恢复。 + +安全负面验收至少检查： + +- 容器没有 ServiceAccount token 和 kubeconfig； +- 不能访问 Kubernetes API； +- 没有宿主机目录、容器运行时套接字、主机命名空间或特权能力； +- `~/.env/TOKEN` 只报告 `sourceRef`、`targetKey`、presence 和 fingerprint； +- 运行时 Codex home 的 pika 配置只报告 `sourceRef`、`targetKey`、presence 和 fingerprint； +- 两类凭据值都不出现在日志、规格、API、终端回显或工件； +- 运行 commit、镜像摘要、GitOps revision 和公网验收结果可以关联。 + +## 7. 过程控制 + +### 7.1 当前实现状态与目标差距 + +- 当前 `/root/selfmedia` 以主机 Bun 进程运行 CLI 和 Web；这只能作为功能样片和临时公网验收基线。 +- 当前不存在本规格定义的 Kubernetes 应用、Web Codex 终端、会话 PVC、两类隔离凭据投影和 GitOps 自动交付链。 +- 本规格作为目标约束已经生效，但实现状态仍为“未实现”。 +- 所有第 6 章内容都是目标需求，不得用于宣称已经上线。 +- 后续新增或修改的相关源码文件应标记： + - `SPEC: PJ2026-0207 平台交付 draft-2026-07-13-p0`。 +- 纯 YAML、生成文件、锁文件和二进制镜像可以不加文件头： + - owning YAML、渲染器、容器构建入口和交付流水线必须能追溯到本规格。 + +### 7.2 实施顺序 + +- 先交付不含 Kubernetes 权限的后端容器、源码、CLI 和只读健康入口。 +- 再交付会话 PVC、Web 内置终端和 Codex 新建/恢复。 +- 随后接入两类 YAML-first 凭据 `sourceRef`： + - 后端 API `~/.env/TOKEN`； + - Codex `~/.codex/*.pika` 源到运行时 Codex home 的配置与认证文件。 +- 两类凭据隔离后，再接入权限负面检查和受限 Skill/API。 +- 完成 YAML-first Kubernetes、公网 IPv4 原入口和运行验收后，才接入自动交付链。 +- 最后用 GitHub、mirror、PaC、Tekton、GitOps 和 Argo 全链替换手工发布路径。 + +### 7.3 安全门禁 + +- 发现 ServiceAccount token、kubeconfig、Kubernetes API 可达、hostPath、容器套接字或特权能力时，运行验收必须失败。 +- 任一后端或 Codex 凭据值出现在日志、规格、终端、API、工件或镜像层时，必须停止交付并轮换受影响凭据。 +- 终端不能直接调用部署控制面；需要部署动作时只能通过受控自动交付入口产生可审查变更。 +- 受限 Skill/API 策略变化先更新 owning YAML；扩大集群或主机权限必须先更新本规格并重新评审边界。 + +### 7.4 原入口验收 + +- 使用公网 IPv4 打开 Web 并创建 Codex 终端。 +- 验证容器内源码 commit、CLI 版本和已部署镜像摘要一致。 +- 通过终端执行配置计划、步骤列表或等价只读命令。 +- 重启后端 Pod，验证同一 sessionId 和终端上下文依靠 PVC 中的 Codex sessions 与 supervisor 元数据恢复。 +- 检查 `~/.env/TOKEN` 的 `sourceRef`、`targetKey`、presence、文件权限和 fingerprint。 +- 检查运行时 Codex home 的 pika 配置 `sourceRef`、`targetKey`、presence、文件权限和 fingerprint。 +- 不读取、保存或输出任一凭据值。 +- 运行权限负面检查和 GitHub 到 Argo 的交付追溯检查。 + +### 7.5 回写边界 + +- 内容步骤、作业、日志或工件契约问题回写 [流水线平台](PJ2026-0205-pipeline-platform.md)。 +- 发布包、人工批准和平台发布材料问题回写 [发布运营](PJ2026-0206-publishing-operations.md)。 +- Web 终端、容器、凭据隔离、Kubernetes、自动链和部署后验收主归属本 L1。 +- 稳定安全边界、部署数据流、自动链或原入口验收变化时，先更新本规格和 [L0 总规格](PJ2026-02-smart-media-factory.md)。 From a83fd9d0af0bd9dbbd1cb24cf0bc6e84674cfbe6 Mon Sep 17 00:00:00 2001 From: Codex Date: Mon, 13 Jul 2026 07:12:15 +0200 Subject: [PATCH 02/14] =?UTF-8?q?docs:=20=E6=94=B6=E5=8F=A3=20Artificer=20?= =?UTF-8?q?=E9=80=92=E5=BD=92=E6=B4=BE=E5=8D=95=E9=AA=8C=E6=94=B6?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/MDTODO/artificer-runtime-capabilities.md | 6 ++--- .../R5.2_Task_Report.md | 22 ++++++++++++++++++ .../R5.3_Task_Report.md | 19 +++++++++++++++ .../R5_Task_Report.md | 7 ++++++ .../R5.1_Task_Report.md | 23 +++++++++++++++++++ .../pr-merge-driven-automatic-delivery.md | 4 ++-- 6 files changed, 76 insertions(+), 5 deletions(-) create mode 100644 docs/MDTODO/details/artificer-runtime-capabilities/R5.2_Task_Report.md create mode 100644 docs/MDTODO/details/artificer-runtime-capabilities/R5.3_Task_Report.md create mode 100644 docs/MDTODO/details/artificer-runtime-capabilities/R5_Task_Report.md create mode 100644 docs/MDTODO/details/pr-merge-driven-automatic-delivery/R5.1_Task_Report.md diff --git a/docs/MDTODO/artificer-runtime-capabilities.md b/docs/MDTODO/artificer-runtime-capabilities.md index c92398e6..55f79ce9 100644 --- a/docs/MDTODO/artificer-runtime-capabilities.md +++ b/docs/MDTODO/artificer-runtime-capabilities.md @@ -59,7 +59,7 @@ 主代理 review/合并后只观察自动 CI/CD；用新 Artificer canary 验证 active steer 后 backend retry 无需重发仍继续最新指令，并确认 workspace/resource bundle/tool credential 保持正确，完成任务后将详细报告写入[任务报告](./details/artificer-runtime-capabilities/R4.3_Task_Report.md)。 -## R5 [in_progress] +## R5 [completed] 解决 [AgentRun #346](https://github.com/pikasTech/agentrun/issues/346)：由 owning Aipod YAML 提示词明确当前 Artificer 是执行者并禁止递归创建或派发 Artificer；由原生子代理做最小修复，不改变模型、provider、SecretRef、workspace、credential 或 resume 语义，完成任务后将详细报告写入[任务报告](./details/artificer-runtime-capabilities/R5_Task_Report.md)。 @@ -67,10 +67,10 @@ 由原生子代理按 [AgentRun #346](https://github.com/pikasTech/agentrun/issues/346) 修改 config/aipods/artificer.yaml 的单一 prompt authority 并补 render 定向测试，禁止 Artificer 递归修自己，完成任务后将详细报告写入[任务报告](./details/artificer-runtime-capabilities/R5.1_Task_Report.md)。 -### R5.2 [in_progress] +### R5.2 [completed] 主代理 review/合并 [AgentRun #346](https://github.com/pikasTech/agentrun/issues/346) 的最小 PR，只观察自动 CI/CD，不新增代码默认值或第二 prompt authority，完成任务后将详细报告写入[任务报告](./details/artificer-runtime-capabilities/R5.2_Task_Report.md)。 -### R5.3 +### R5.3 [completed] 自动上线后执行最短 Artificer canary，确认直接处理业务任务且没有创建第二 task，并记录 task/run/session 证据，完成任务后将详细报告写入[任务报告](./details/artificer-runtime-capabilities/R5.3_Task_Report.md)。 diff --git a/docs/MDTODO/details/artificer-runtime-capabilities/R5.2_Task_Report.md b/docs/MDTODO/details/artificer-runtime-capabilities/R5.2_Task_Report.md new file mode 100644 index 00000000..10a4e1bb --- /dev/null +++ b/docs/MDTODO/details/artificer-runtime-capabilities/R5.2_Task_Report.md @@ -0,0 +1,22 @@ +# R5.2 任务报告 + +## 结论 + +主代理已独立 review 并合并 [AgentRun PR #347](https://github.com/pikasTech/agentrun/pull/347)。merge commit 为 `4155132aa7e8de1936b2bf447ea889a607ee59d7`。 + +## Review 证据 + +- `git diff --check` 通过。 +- 定向运行 `81-artificer-nonrecursive-prompt`、`76-aipod-spec`、`80-artificer-model-selection`，全部通过。 +- 规则唯一位于 owning YAML；通用 renderer 仅拼接 YAML 默认提示和用户 prompt。 +- 未改变模型、reasoning、provider、SecretRef、workspace、resource bundle、Git identity 或 session resume。 + +## 自动交付证据 + +未执行人工 mirror、PipelineRun、sync、refresh、trigger 或 rollout。PaC watcher 临时恢复后，已有自动事件自行继续： + +- PipelineRun：`agentrun-nc01-v02-ci-4155132aa7e8de1936b2bf447ea889a607ee5glss5`； +- PipelineRun：`Succeeded`，47 秒，3 个 TaskRun 全部成功； +- GitOps：`fcb77f4cf8e1520003b81928eaef9177d439cfe6`； +- Argo：`Synced / Healthy`，revision `fcb77f4cf8e1`； +- runtime manager source：`4155132aa7e8`，`1/1 ready`。 diff --git a/docs/MDTODO/details/artificer-runtime-capabilities/R5.3_Task_Report.md b/docs/MDTODO/details/artificer-runtime-capabilities/R5.3_Task_Report.md new file mode 100644 index 00000000..48096b52 --- /dev/null +++ b/docs/MDTODO/details/artificer-runtime-capabilities/R5.3_Task_Report.md @@ -0,0 +1,19 @@ +# R5.3 任务报告 + +## 结论 + +自动上线后的最短 Artificer canary 通过。当前 Artificer 直接完成任务，没有调用工具，也没有创建或派发第二个 AgentRun/Artificer task。 + +## 证据 + +- task：`qt_147d7694b83b4a62a99a6bcefa19eb11`； +- run：`run_795aaea568dc44898654ce25d35099bd`； +- command：`cmd_0ab1266c1c5c48048266efdb74edd324`； +- session：`sess_artificer_fc1728414554120dad66f19a`； +- runner：`rjob_954e449ac40f4b26bd6bd74a9cecc3f1`； +- terminal：`completed`、`OK=true`、authoritative； +- 唯一结果：`ARTIFICER_NONRECURSIVE_OK`。 + +事件 1 显示 YAML 默认提示已在用户业务提示之前进入实际初始 prompt。事件 21-32 只有 thread/turn 状态、单条 assistant message 和 terminal 状态，没有 tool call、AgentRun create/apply/dispatch 或 Code Queue 操作。canary 前后活动 task 视图未出现第二个新增任务。 + +本次 canary 为只读任务，未修改文件或外部状态。 diff --git a/docs/MDTODO/details/artificer-runtime-capabilities/R5_Task_Report.md b/docs/MDTODO/details/artificer-runtime-capabilities/R5_Task_Report.md new file mode 100644 index 00000000..576b840a --- /dev/null +++ b/docs/MDTODO/details/artificer-runtime-capabilities/R5_Task_Report.md @@ -0,0 +1,7 @@ +# R5 任务报告 + +[AgentRun #346](https://github.com/pikasTech/agentrun/issues/346) 已完成。禁止 Artificer 递归派单的职责提示由 `config/aipods/artificer.yaml#spec.payloadDefaults.prompt` 单一控制；通用 renderer 只做默认提示与用户提示合成，没有第二规则权威。 + +实现已由 [AgentRun PR #347](https://github.com/pikasTech/agentrun/pull/347) 合并，并通过自动 PaC、GitOps、Argo 和 runtime 发布。上线后真实只读 canary `qt_147d7694b83b4a62a99a6bcefa19eb11` 直接返回 `ARTIFICER_NONRECURSIVE_OK`，完整事件中没有任何 tool call 或第二 task 派单。 + +默认模型和运行事实保持 `gpt-5.6-sol`、reasoning `medium`、backend profile `gpt-pika`；workspace、resource bundle、credential、Git identity 和 session resume 均未改变。 diff --git a/docs/MDTODO/details/pr-merge-driven-automatic-delivery/R5.1_Task_Report.md b/docs/MDTODO/details/pr-merge-driven-automatic-delivery/R5.1_Task_Report.md new file mode 100644 index 00000000..e3615c62 --- /dev/null +++ b/docs/MDTODO/details/pr-merge-driven-automatic-delivery/R5.1_Task_Report.md @@ -0,0 +1,23 @@ +# R5.1 任务报告 + +## 结论 + +[UniDesk #1900](https://github.com/pikasTech/unidesk/issues/1900) 的直接断点不是 PR、Pipeline 定义或 AgentRun，而是 NC01 的 `pipelines-as-code-watcher` 未启动。GitHub merge、Gitea mirror/webhook 和 PaC 创建 PipelineRun 均正常；唯一负责把 `queued/PipelineRunPending` 推进为 `started` 的 watcher 不可用，因此自动链停在 0 TaskRun。 + +## 运行面证据 + +- watcher Deployment：`replicas=1`、`ready=0`、`unavailable=1`。 +- 故障 Pod：`pipelines-as-code-watcher-7f6c87497c-h2dzz`，UID `7fe65fbe-a18e-4513-89f2-5fac37ae0b71`。 +- projected ServiceAccount token 被 node authorizer 拒绝：`node requested token bound to a pod scheduled on a different node` / `no relationship found between node`。 +- 同一控制面恢复窗口创建的 `tempo`、`todo-note` Pod 出现相同错误；稍后新建 Pod 已能正常挂载 token，排除 watcher ServiceAccount RBAC 和 AgentRun Pipeline 定义。 +- AgentRun merge commit `4155132aa7e8de1936b2bf447ea889a607ee59d7` 的 PipelineRun 已由 PaC 创建并标记 `queued`，但长期 `startTime=null`、0 TaskRun。 + +## 有界临时恢复 + +在 issue 和 MDTODO 建立后，只删除上述精确 Pending watcher Pod，由既有 Deployment/ReplicaSet 自动重建。未修改、删除或重跑任何 PipelineRun，也未执行 mirror、sync、refresh、trigger 或 rollout。 + +新 Pod `pipelines-as-code-watcher-7f6c87497c-7lhfh` 随后 Ready，并接管 leader lease；既有 #347 PipelineRun 自动从 `queued` 进入 `Running`，产生 TaskRun。该证据证明恢复仍由原 PaC 自动权威完成。 + +## 长期归属 + +长期修复归 [#1900](https://github.com/pikasTech/unidesk/issues/1900) 的 NC01 k3s node-authorizer / Pod 生命周期恢复。不得把本次精确 Pod 重建固化为额外 delivery actor、补跑入口或自动围栏。 diff --git a/docs/MDTODO/pr-merge-driven-automatic-delivery.md b/docs/MDTODO/pr-merge-driven-automatic-delivery.md index b7681dcb..72933f97 100644 --- a/docs/MDTODO/pr-merge-driven-automatic-delivery.md +++ b/docs/MDTODO/pr-merge-driven-automatic-delivery.md @@ -68,11 +68,11 @@ 解决 [UniDesk #1900](https://github.com/pikasTech/unidesk/issues/1900)：修复 NC01 PaC watcher 因 ServiceAccount token 的 node/pod 关系失败而 Pending，恢复 GitHub PR merge 驱动的唯一自动交付链，禁止人工补触发、第二权威或旁路，完成任务后将详细报告写入[任务报告](./details/pr-merge-driven-automatic-delivery/R5_Task_Report.md)。 -### R5.1 [in_progress] +### R5.1 [completed] 只读定位 [UniDesk #1900](https://github.com/pikasTech/unidesk/issues/1900) 的唯一根因，核对 watcher Pod、ServiceAccount token、node/pod 关系、owning YAML 与最近控制面变更，不改运行面，完成任务后将详细报告写入[任务报告](./details/pr-merge-driven-automatic-delivery/R5.1_Task_Report.md)。 -### R5.2 +### R5.2 [in_progress] 按 [UniDesk #1900](https://github.com/pikasTech/unidesk/issues/1900) 实施最小持久化修复和定向验证，不新增第二交付权威、合同、租约、安全机制或围栏，完成任务后将详细报告写入[任务报告](./details/pr-merge-driven-automatic-delivery/R5.2_Task_Report.md)。 From 48aa08bf73c75583f2e85a01d6daac4a92c4eb1e Mon Sep 17 00:00:00 2001 From: Codex Date: Mon, 13 Jul 2026 07:16:51 +0200 Subject: [PATCH 03/14] =?UTF-8?q?docs:=20=E6=94=B9=E4=B8=BA=20Monitor=20?= =?UTF-8?q?=E7=A9=BA=E5=BA=93=E5=88=87=E6=8D=A2?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../R2.4.1_Task_Report.md | 24 +++++++++++++++++++ .../web-sentinel-runtime-reliability.md | 12 +++++----- 2 files changed, 30 insertions(+), 6 deletions(-) create mode 100644 docs/MDTODO/details/web-sentinel-runtime-reliability/R2.4.1_Task_Report.md diff --git a/docs/MDTODO/details/web-sentinel-runtime-reliability/R2.4.1_Task_Report.md b/docs/MDTODO/details/web-sentinel-runtime-reliability/R2.4.1_Task_Report.md new file mode 100644 index 00000000..2262fa42 --- /dev/null +++ b/docs/MDTODO/details/web-sentinel-runtime-reliability/R2.4.1_Task_Report.md @@ -0,0 +1,24 @@ +# R2.4.1 任务报告 + +## 结论 + +[UniDesk #1888](https://github.com/pikasTech/unidesk/issues/1888) 的 migration Job failure envelope 可见性已修复并通过真实运行验收。实现由 [PR #1897](https://github.com/pikasTech/unidesk/pull/1897) 合并，merge commit `2f4522db`；后续自动发布 source `2fdadce835b8` 已包含该修复。 + +## 根因与修复 + +- `sentinelPayloadFromLogs` 在返回 nested `data` 时丢失外层 `command`。 +- JSON 扫描器把 pretty envelope 内嵌 `data` 误当作最后完整对象。 +- 修复后保留顶层 envelope，并继承外层 command；默认 failure 投影返回失败 container、phase、exitCode、语义化 code/message 与日志下钻，保持脱敏。 +- 37 个定向测试通过。 + +## 真实 Job 验收 + +Job `hwlab-web-probe-monitor-migration-2fdadce835b8-mrirnjg7` 的 import 失败被直接投影为： + +- `container=import`； +- `phase=Failed`； +- `exitCode=1`； +- `code=payload_hash_conflict`； +- `message=ingest identity already has a different terminal envelope`。 + +不再返回 result-missing，也没有依赖临时 dump。用户随后明确放弃 SQLite 历史迁移，因此该冲突不再继续修复或作为 Release B 门禁。 diff --git a/docs/MDTODO/web-sentinel-runtime-reliability.md b/docs/MDTODO/web-sentinel-runtime-reliability.md index 9e7f23c8..7a6e8f1b 100644 --- a/docs/MDTODO/web-sentinel-runtime-reliability.md +++ b/docs/MDTODO/web-sentinel-runtime-reliability.md @@ -42,26 +42,26 @@ PR 合并后只等待自动发布链，使用受控 Web sentinel 原入口产生 ### R2.4 [in_progress] -按 [#1868](https://github.com/pikasTech/unidesk/issues/1868) 已冻结规格推进实施：先并行完成 [#1875](https://github.com/pikasTech/unidesk/issues/1875) Host PG 前置和 [#1874](https://github.com/pikasTech/unidesk/issues/1874) 中心 service（实际实现由其子任务 [#1876](https://github.com/pikasTech/unidesk/issues/1876) 跟踪）；前置合并后完成 [#1872](https://github.com/pikasTech/unidesk/issues/1872) runner ingest/迁移工具；最后完成 [#1873](https://github.com/pikasTech/unidesk/issues/1873) 数据迁移、中心 API/UI/CLI 原子切换、自动 CI/CD 与原入口验收，并退役旧 SQLite/runner dashboard，完成任务后将详细报告写入[任务报告](./details/web-sentinel-runtime-reliability/R2.4_Task_Report.md)。 +按 [#1868](https://github.com/pikasTech/unidesk/issues/1868) 已冻结规格推进实施：Host PG、中心 service、runner ingest 与新 Monitor 工作台基础均已完成；按 [#1873 最新决策](https://github.com/pikasTech/unidesk/issues/1873#issuecomment-4954659225) 不迁移 SQLite 历史数据，最后只完成 Monitor 专属 PG 空库验证、Release B 中心 API/UI/CLI 原子切换、自动 CI/CD 与原入口验收，并退役旧 SQLite/runner dashboard，完成任务后将详细报告写入[任务报告](./details/web-sentinel-runtime-reliability/R2.4_Task_Report.md)。 -#### R2.4.1 [in_progress] +#### R2.4.1 [completed] 修复 [#1873](https://github.com/pikasTech/unidesk/issues/1873) 与 [#1888](https://github.com/pikasTech/unidesk/issues/1888) 的 one-shot migration Job 失败可见性：正确解包外层 command 与 nested data.error，默认有界返回失败 container、phase、exit、语义化 code/message 和日志下钻，不输出 Secret、不依赖临时 dump，也不改变迁移执行语义，完成任务后将详细报告写入[任务报告](./details/web-sentinel-runtime-reliability/R2.4.1_Task_Report.md)。 ### R2.5 [in_progress] -按 [#1873 运行面证据](https://github.com/pikasTech/unidesk/issues/1873#issuecomment-4953413607) 处理迁移前已不可恢复的 2 runs/5 artifacts：以 owning YAML 精确列出 runId、stateDir、artifactCount 与 report SHA，导入结构化事实并写入 `legacy-artifact-bytes-missing-before-migration` timeline/finding，不制造 locator或声称已验证 bytes；任何未列明缺口继续 fail closed，完成 PG verify、Release B 原子切换和旧 SQLite/dashboard 退役后关闭，完成任务后将详细报告写入[任务报告](./details/web-sentinel-runtime-reliability/R2.5_Task_Report.md)。 +按 [#1873 用户最新决策](https://github.com/pikasTech/unidesk/issues/1873#issuecomment-4954659225) 放弃全部 SQLite 历史数据迁移，以空 Monitor Host PG 从新数据开始；先清空并验证 Monitor 专属数据库/schema，再完成 Release B 原子切换，SQLite、legacy gap 与 migration 不进入运行路径，不双写、不回读、不补洞，完成任务后将详细报告写入[任务报告](./details/web-sentinel-runtime-reliability/R2.5_Task_Report.md)。 #### R2.5.1 [in_progress] -按 [#1873 证据](https://github.com/pikasTech/unidesk/issues/1873#issuecomment-4953413607) 在 owning YAML 精确声明仅有的 2 runs/5 artifacts 历史缺口；export/import/verify 导入结构化事实和 missing-before-migration finding，不制造 locator，任何未声明缺口继续 fail closed，完成任务后将详细报告写入[任务报告](./details/web-sentinel-runtime-reliability/R2.5.1_Task_Report.md)。 +按 [#1873 最新决策](https://github.com/pikasTech/unidesk/issues/1873#issuecomment-4954659225) 冻结空库切换边界：取消 legacy gap、SQLite export/import/verify 和 payload conflict 作为门禁，确认只清空 Monitor 自有 Host PG database/schema，不影响共享 Host PG 其他服务，完成任务后将详细报告写入[任务报告](./details/web-sentinel-runtime-reliability/R2.5.1_Task_Report.md)。 #### R2.5.2 -Release A 自动上线后通过受控 one-shot Job 完成 freeze/snapshot/import/PG verify，核对完整 run/finding/metadata/payload/latest identity 与精确历史缺口，并把真实运行证据写回 [#1873](https://github.com/pikasTech/unidesk/issues/1873)，完成任务后将详细报告写入[任务报告](./details/web-sentinel-runtime-reliability/R2.5.2_Task_Report.md)。 +通过 YAML-first 受控入口按 [#1873 最新决策](https://github.com/pikasTech/unidesk/issues/1873#issuecomment-4954659225) 清空或重建 Monitor 专属 Host PG database/schema，验收 schema ready、业务数据 0 行且不读取 SQLite，再允许 Release B，完成任务后将详细报告写入[任务报告](./details/web-sentinel-runtime-reliability/R2.5.2_Task_Report.md)。 #### R2.5.3 -迁移验证通过后实施 [#1873](https://github.com/pikasTech/unidesk/issues/1873) Release B 原子切换，并与 R2.6 / [#1887](https://github.com/pikasTech/unidesk/issues/1887) 的 Monitor 前端重写同步收口：中心 service 成为唯一 writer/query authority，Web/CLI/scheduler 只读中心 API，删除旧 SQLite/query/dashboard/fallback 与旧页面；只观察自动 CI/CD并完成 CLI、Monitor Web 与原入口验收，完成任务后将详细报告写入[任务报告](./details/web-sentinel-runtime-reliability/R2.5.3_Task_Report.md)。 +空 Host PG 验证通过后实施 [#1873](https://github.com/pikasTech/unidesk/issues/1873) Release B，并与 R2.6 / [#1887](https://github.com/pikasTech/unidesk/issues/1887) 的 Monitor 前端重写同步收口：中心 service 成为唯一 writer/query authority，新数据从切换后开始；删除或屏蔽旧 SQLite/query/dashboard/fallback 与旧页面，只观察自动 CI/CD 并完成 CLI、Monitor Web 原入口验收，完成任务后将详细报告写入[任务报告](./details/web-sentinel-runtime-reliability/R2.5.3_Task_Report.md)。 ### R2.6 [in_progress] From 8c3d4fb2894de0932cd2c32a053915cac5b9ea69 Mon Sep 17 00:00:00 2001 From: Codex Date: Mon, 13 Jul 2026 07:29:48 +0200 Subject: [PATCH 04/14] =?UTF-8?q?docs:=20=E5=AE=8C=E6=88=90=20HWLAB=20Web?= =?UTF-8?q?=20=E5=8E=9F=E5=85=A5=E5=8F=A3=E9=AA=8C=E6=94=B6?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../R1.12_Task_Report.md | 20 +++++++++++++++++++ docs/MDTODO/hwlab-web-product-experience.md | 6 +++++- 2 files changed, 25 insertions(+), 1 deletion(-) create mode 100644 docs/MDTODO/details/hwlab-web-product-experience/R1.12_Task_Report.md diff --git a/docs/MDTODO/details/hwlab-web-product-experience/R1.12_Task_Report.md b/docs/MDTODO/details/hwlab-web-product-experience/R1.12_Task_Report.md new file mode 100644 index 00000000..3097c388 --- /dev/null +++ b/docs/MDTODO/details/hwlab-web-product-experience/R1.12_Task_Report.md @@ -0,0 +1,20 @@ +# R1.12 任务报告 + +## NC01/v03 原入口交付 + +- HWLAB PR [#2520](https://github.com/pikasTech/HWLAB/pull/2520) 与 follow-up [#2521](https://github.com/pikasTech/HWLAB/pull/2521) 已合并到 `v0.3`，最终源码提交为 `5970ae77ed8bf74882b05cfa96577f2e67c0d7ba`。 +- PaC 自动交付使用 PipelineRun `hwlab-nc01-v03-ci-poll-vgrwx`，生成 GitOps 提交 `a4afa56c7b0e`；NC01/v03 Argo 状态为 `Synced/Healthy`。 +- 固定正式入口为 `https://hwlab.pikapython.com`，未用裸 IP 或本地开发页替代验收。 + +## 页面与状态验证 + +- HWPOD 浏览器验证覆盖 1920×1080、960×600、390×844：设备与 Node 卡片/列表、路径实体深链 Drawer、筛选/游标切换、reduced-motion、节点接入 artifact 元数据及六阶段 readiness 均通过，无 document 级横向溢出。 +- 非 HWPOD 固定 Playwright 烟测覆盖 19 个路由 × 960/390，5/5 通过；专项 4/4 覆盖 OpenCode retry、三栏 separator 键盘调整、Projects 公共 Workbench Launch、R205 深链与 101→201→205 增量加载。 +- HWPOD 后端 17/17、共享前端 5/5；页面主 PR `bun run check` 114/114、后端合同 8/8、Vitest 11/11；MDTODO follow-up 后端 13/13、Vitest 2/2、Playwright 3/3、TypeScript 与构建通过。 +- typed read model 覆盖一 node 多设备、离线 node、在线无 spec、invalid spec、capability mismatch、busy/in-flight、更新元数据、readiness 与脱敏诊断；partial/stale/restricted/error 不会被 ready 状态吞没。 + +## Web observe 证据与处置 + +- observer `webobs-mriq0nci-aaf44f` 在正式 MDTODO 深链采样 20 次；Project Management 相关 API 均返回 200，任务树、正文和公共 Workbench Launch 可见。 +- 该次所选任务本身没有 report link，因此没有把“报告预览”误记为线上成功；projection-only 报告与报告交互由专项回归覆盖。 +- analyzer 对任务树贴底事实栏的误报已由 UniDesk PR [#1899](https://github.com/pikasTech/unidesk/pull/1899) 修复；observer 随后已受控 force-stop，目标 PID 与独立 Chrome session 均复核为空。 diff --git a/docs/MDTODO/hwlab-web-product-experience.md b/docs/MDTODO/hwlab-web-product-experience.md index ecf919cf..a416b0bc 100644 --- a/docs/MDTODO/hwlab-web-product-experience.md +++ b/docs/MDTODO/hwlab-web-product-experience.md @@ -59,6 +59,10 @@ 依赖 R1.4-R1.10，完成迁移收口：删除所有已被新设计系统替代的旧页面 shell、私有 Panel/Table/Dialog/Viewer、字段名猜测、名称去重、关键事实前端拼装、重复 CSS 和阻碍最新目标的兼容入口；把 Workbench 与普通页面样式按职责拆分，所有源码文件满足体积治理。保持稳定业务 ID、API、权限、Secret、深链和 Workbench 用户主路径无回退，完成任务后将详细报告写入[任务报告](./details/hwlab-web-product-experience/R1.11_Task_Report.md)。 -### R1.12 +### R1.12 [completed] 依赖 R1.4-R1.11，在 NC01/v03 YAML-selected Cloud Web 原入口完成全量验收：覆盖 1920x1080、960x600、约 390px、键盘/焦点/ARIA、reduced-motion、深链刷新/前进后退、卡片/列表语义一致、loading/refreshing/partial/empty/error/ready、大表/长树/长日志性能和权限差异。用 web-probe observe/analyze 留存证据；HWPOD 覆盖一 node 多设备、离线 node、在线无 spec、invalid spec、mismatch、busy、更新和诊断错误，完成任务后将详细报告写入[任务报告](./details/hwlab-web-product-experience/R1.12_Task_Report.md)。 + +### R1.13 [in_progress] + +补强 web-probe 与 Web 哨兵资源治理：人工 observe start 只以 /proc/meminfo 的 MemAvailable 判定真实物理内存，<=4GiB 时先阻断并要求受控 GC 后复核；哨兵 <4GiB 时不创建 observer/Chrome，跳过本轮等待下一 cadence。将阈值、TTL、样本/截图/性能/历史/响应体上限收敛到 owning YAML，保证成功、失败、timeout、signal 全终态关闭 browser/context，并在 NC01/v03 验证 12GiB swap 不计入资格，完成任务后将详细报告写入[任务报告](./details/hwlab-web-product-experience/R1.13_Task_Report.md)。 From 6bac8676c29d20acd410acadb797959a210e9d15 Mon Sep 17 00:00:00 2001 From: AgentRun Codex Date: Mon, 13 Jul 2026 05:39:44 +0000 Subject: [PATCH 05/14] =?UTF-8?q?feat:=20=E5=A2=9E=E5=8A=A0=20Monitor=20Ho?= =?UTF-8?q?st=20PG=20=E7=A9=BA=E5=BA=93=E5=85=A5=E5=8F=A3?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- config/platform-db/postgres-nc01.yaml | 7 + scripts/src/platform-db-monitor-reset.test.ts | 60 +++++++ .../platform-db-nc01-monitor-config.test.ts | 1 + scripts/src/platform-db.ts | 160 ++++++++++++++++++ 4 files changed, 228 insertions(+) create mode 100644 scripts/src/platform-db-monitor-reset.test.ts diff --git a/config/platform-db/postgres-nc01.yaml b/config/platform-db/postgres-nc01.yaml index 0294ea2d..29ba644d 100644 --- a/config/platform-db/postgres-nc01.yaml +++ b/config/platform-db/postgres-nc01.yaml @@ -213,6 +213,13 @@ objects: locale: C.UTF-8 extensions: [] +managedOperations: + monitorReset: + database: web_probe_monitor + owner: web_probe_monitor + schema: monitor + secretRef: platform-db/web-probe-monitor-nc01-db.env + exports: connectionStrings: - name: agentrun-nc01-v02-database-url diff --git a/scripts/src/platform-db-monitor-reset.test.ts b/scripts/src/platform-db-monitor-reset.test.ts new file mode 100644 index 00000000..cb5e72eb --- /dev/null +++ b/scripts/src/platform-db-monitor-reset.test.ts @@ -0,0 +1,60 @@ +import { describe, expect, test } from "bun:test"; +import { rootPath } from "./config"; +import { readYamlRecord } from "./platform-infra-ops-library"; +import { monitorResetScript, monitorStatusScript, runPlatformDbCommand } from "./platform-db"; + +const configPath = "config/platform-db/postgres-nc01.yaml"; +const parsed = readYamlRecord>(rootPath(configPath), "postgres-host-cluster"); +const pg = { objects: parsed.objects }; +const target = parsed.managedOperations.monitorReset; + +describe("platform-db Monitor 专属空库入口", () => { + test("plan 精确显示 YAML 目标且默认零写入", async () => { + const result = await runPlatformDbCommand({} as never, ["postgres", "monitor", "plan", "--config", configPath]); + + expect(result).toMatchObject({ + ok: true, + target: "web-probe-monitor", + config: configPath, + database: "web_probe_monitor", + owner: "web_probe_monitor", + schema: "monitor", + secretRef: "platform-db/web-probe-monitor-nc01-db.env", + mutation: false, + valuesPrinted: false, + }); + expect(JSON.stringify(result)).not.toContain("PASSWORD"); + expect(JSON.stringify(result)).not.toContain("DATABASE_URL"); + }); + + test("reset 未确认时只返回计划", async () => { + const result = await runPlatformDbCommand({} as never, ["postgres", "monitor", "reset", "--config", configPath]); + + expect(result).toMatchObject({ ok: true, mutation: false, database: "web_probe_monitor" }); + expect(result).not.toHaveProperty("confirmed"); + }); + + test("确认脚本只重建 Monitor database 并应用 central schema", () => { + const script = monitorResetScript(pg as never, target); + + expect(script).toContain("database='web_probe_monitor'"); + expect(script).toContain("owner='web_probe_monitor'"); + expect(script).toContain('drop database if exists "web_probe_monitor";'); + expect(script).toContain("create schema if not exists monitor;"); + expect(script).toContain('set role "web_probe_monitor";'); + expect(script).not.toContain("drop database agentrun_v02"); + expect(script).not.toContain("drop database decision_center"); + }); + + test("status/reset 验证 schema ready、业务 0 行与其他数据库 OID 不变", () => { + const status = monitorStatusScript(pg as never, target); + const reset = monitorResetScript(pg as never, target); + + expect(status).toContain('\"schemaReady\":%s'); + expect(status).toContain('\"businessRows\":%s'); + expect(status).toContain('\"siblingDatabaseOids\":\"%s\"'); + expect(reset).toContain('[ "$before" != "$after" ]'); + expect(reset).toContain('[ "$rows" != "0" ]'); + expect(reset).toContain('\"siblingDatabasesUnchanged\":true'); + }); +}); diff --git a/scripts/src/platform-db-nc01-monitor-config.test.ts b/scripts/src/platform-db-nc01-monitor-config.test.ts index 4da08779..8df91c41 100644 --- a/scripts/src/platform-db-nc01-monitor-config.test.ts +++ b/scripts/src/platform-db-nc01-monitor-config.test.ts @@ -35,6 +35,7 @@ describe("NC01 Host PG Monitor declaration", () => { consumers: [{ scope: "hwlab-web-probe-monitor", secret: "hwlab-web-probe-monitor-db", key: "DATABASE_URL" }], }); expect(monitorExport.render.format).not.toContain("web_probe_monitor:"); + expect(config.managedOperations.monitorReset).toEqual({ database: "web_probe_monitor", owner: "web_probe_monitor", schema: "monitor", secretRef: "platform-db/web-probe-monitor-nc01-db.env" }); }); test("既有应用 role 保持原有独立 sourceRef", () => { diff --git a/scripts/src/platform-db.ts b/scripts/src/platform-db.ts index c261b541..51734a14 100644 --- a/scripts/src/platform-db.ts +++ b/scripts/src/platform-db.ts @@ -5,6 +5,7 @@ import { dirname, isAbsolute, join } from "node:path"; import type { UniDeskConfig } from "./config"; import { repoRoot, rootPath } from "./config"; import { startJob } from "./jobs"; +import { MONITOR_CENTRAL_MIGRATIONS, MONITOR_CENTRAL_SCHEMA_VERSION } from "./monitor-central-persistence"; import { runSshCommandCapture, type SshCaptureResult } from "./ssh"; import { compactText, fingerprintEnvValues as fingerprintValues, parseEnvFile, parseJsonOutput } from "./platform-infra-ops-library"; @@ -21,6 +22,13 @@ interface PlatformDbOptions { raw: boolean; } +interface MonitorResetConfig { + database: string; + owner: string; + schema: string; + secretRef: string; +} + interface PostgresHostConfig { configPath: string; version: number; @@ -139,6 +147,7 @@ interface PostgresHostConfig { encryption: { enabled: boolean }; }; }; + managedOperations?: { monitorReset: MonitorResetConfig }; } interface PgHbaRule { @@ -300,6 +309,9 @@ export function platformDbHelp(): unknown { `bun scripts/cli.ts platform-db postgres apply --config ${defaultConfigPath} --dry-run`, `bun scripts/cli.ts platform-db postgres apply --config ${defaultConfigPath} --confirm`, `bun scripts/cli.ts platform-db postgres apply --config ${defaultConfigPath} --confirm --wait`, + "bun scripts/cli.ts platform-db postgres monitor plan --config config/platform-db/postgres-nc01.yaml", + "bun scripts/cli.ts platform-db postgres monitor status --config config/platform-db/postgres-nc01.yaml", + "bun scripts/cli.ts platform-db postgres monitor reset --config config/platform-db/postgres-nc01.yaml --confirm", ], description: "Manage YAML-declared host-native PostgreSQL for UniDesk platform state. The PK01 declaration uses PostgreSQL 16 for Sub2API and does not deploy k3s.", safety: { @@ -307,6 +319,7 @@ export function platformDbHelp(): unknown { defaultMutation: false, exportSecrets: "export-secrets only materializes YAML-declared local Secret source/export files; it does not touch the remote PostgreSQL service.", apply: "apply --confirm returns an async UniDesk job; the job uses short trans calls and a remote PK01 job instead of keeping one long SSH session open.", + monitorReset: "monitor reset is restricted by owning YAML to the dedicated web_probe_monitor database and requires --confirm; it cannot accept a database name or arbitrary SQL.", redaction: "Passwords and full DATABASE_URL values are never printed; output is limited to key names, presence, fingerprints and state.", noK3s: "PK01 PostgreSQL is host-systemd, not Docker or k3s.", }, @@ -316,6 +329,11 @@ export function platformDbHelp(): unknown { export async function runPlatformDbCommand(config: UniDeskConfig, args: string[]): Promise> { const [target, action = "plan"] = args; if (target !== "postgres") return unsupported(args); + if (action === "monitor") { + const monitorAction = args[2] ?? "plan"; + const options = parseOptions(args.slice(3)); + return await monitorCommand(config, monitorAction, options); + } const options = parseOptions(args.slice(2)); if (action === "plan") return await plan(config, options); if (action === "status") return await status(config, options); @@ -324,6 +342,46 @@ export async function runPlatformDbCommand(config: UniDeskConfig, args: string[] return unsupported(args); } +async function monitorCommand(config: UniDeskConfig, action: string, options: PlatformDbOptions): Promise> { + if (!(["plan", "status", "reset"] as const).includes(action as "plan" | "status" | "reset")) return unsupported(["postgres", "monitor", action]); + const pg = readPostgresHostConfig(options.configPath); + const target = requireMonitorResetConfig(pg); + const base = { + action: `platform-db-postgres-monitor-${action}`, + target: "web-probe-monitor", + config: pg.configPath, + database: target.database, + owner: target.owner, + schema: target.schema, + schemaVersion: MONITOR_CENTRAL_SCHEMA_VERSION, + secretRef: target.secretRef, + valuesPrinted: false, + }; + if (action === "plan" || (action === "reset" && !options.confirm)) { + return { + ok: true, + ...base, + mutation: false, + plan: monitorResetPlan(target), + next: { + status: `bun scripts/cli.ts platform-db postgres monitor status --config ${pg.configPath}`, + confirm: `bun scripts/cli.ts platform-db postgres monitor reset --config ${pg.configPath} --confirm`, + }, + }; + } + const script = action === "status" ? monitorStatusScript(pg, target) : monitorResetScript(pg, target); + const capture = await runSshCommandCapture(config, pg.node.route, ["sh"], script); + const parsed = parseJsonOutput(capture.stdout); + return { + ok: capture.exitCode === 0 && parsed?.ok === true, + ...base, + mutation: action === "reset", + ...(action === "reset" ? { confirmed: true } : {}), + state: parsed, + remote: compactCapture(capture, { full: options.full || capture.exitCode !== 0 }), + }; +} + function unsupported(args: string[]): Record { return { ok: false, @@ -589,6 +647,7 @@ function readPostgresHostConfig(pathArg: string): PostgresHostConfig { const objects = objectField(parsed, "objects", configPath); const exportsConfig = objectField(parsed, "exports", configPath); const backup = objectField(parsed, "backup", configPath); + const managedOperations = parsed.managedOperations === undefined ? null : objectField(parsed, "managedOperations", configPath); const logicalDump = objectField(backup, "logicalDump", `${configPath}.backup`); const destination = objectField(logicalDump, "destination", `${configPath}.backup.logicalDump`); const encryption = objectField(logicalDump, "encryption", `${configPath}.backup.logicalDump`); @@ -618,6 +677,13 @@ function readPostgresHostConfig(pathArg: string): PostgresHostConfig { databaseNames.add(database.name); } const connectionStrings = arrayOfRecords(exportsConfig.connectionStrings, `${configPath}.exports.connectionStrings`).map((item, index) => connectionStringExport(item, `${configPath}.exports.connectionStrings[${index}]`)); + const monitorReset = managedOperations === null ? null : monitorResetConfig(objectField(managedOperations, "monitorReset", `${configPath}.managedOperations`), `${configPath}.managedOperations.monitorReset`); + if (monitorReset !== null) { + const database = databases.find((item) => item.name === monitorReset.database); + if (monitorReset.database !== "web_probe_monitor" || monitorReset.owner !== "web_probe_monitor" || monitorReset.schema !== "monitor" || database?.owner !== monitorReset.owner) { + throw new Error(`${configPath}.managedOperations.monitorReset is restricted to the dedicated web_probe_monitor database/schema`); + } + } return { configPath: relativeConfigPath(configPath), version, @@ -729,6 +795,16 @@ function readPostgresHostConfig(pathArg: string): PostgresHostConfig { }, }, }, + ...(monitorReset === null ? {} : { managedOperations: { monitorReset } }), + }; +} + +function monitorResetConfig(item: Record, path: string): MonitorResetConfig { + return { + database: pgIdentifier(stringField(item, "database", path), `${path}.database`), + owner: pgIdentifier(stringField(item, "owner", path), `${path}.owner`), + schema: pgIdentifier(stringField(item, "schema", path), `${path}.schema`), + secretRef: stringField(item, "secretRef", path), }; } @@ -988,6 +1064,90 @@ function configSummary(pg: PostgresHostConfig): Record { }; } +function requireMonitorResetConfig(pg: PostgresHostConfig): MonitorResetConfig { + const target = pg.managedOperations?.monitorReset; + if (target === undefined) throw new Error(`${pg.configPath}.managedOperations.monitorReset is required`); + return target; +} + +function monitorResetPlan(target: MonitorResetConfig): string[] { + return [ + `terminate connections only to ${target.database}`, + `drop and recreate only ${target.database} owned by ${target.owner}`, + `apply central Monitor schema ${MONITOR_CENTRAL_SCHEMA_VERSION} as ${target.owner}`, + "verify schema ready and Monitor business row count is 0", + "verify sibling YAML-declared database OIDs are unchanged", + ]; +} + +export function monitorStatusScript(pg: PostgresHostConfig, target: MonitorResetConfig): string { + const siblings = pg.objects.databases.filter((database) => database.name !== target.database).map((database) => database.name).join(","); + const siblingArray = pg.objects.databases.filter((database) => database.name !== target.database).map((database) => sqlLiteral(database.name)).join(", "); + return `${monitorScriptPrelude()} +database=${shellQuote(target.database)} +schema_version=${shellQuote(MONITOR_CENTRAL_SCHEMA_VERSION)} +siblings=${shellQuote(siblings)} +database_oid=$(sudo -n runuser -u postgres -- psql -At -d postgres -c "select oid from pg_database where datname = ${sqlLiteral(target.database)};") +sibling_oids=$(sudo -n runuser -u postgres -- psql -At -d postgres -c "select coalesce(string_agg(datname || ':' || oid, ',' order by datname), '') from pg_database where datname in (${siblingArray});") +schema_ready=false +schema_version_observed="" +business_rows="" +if [ -n "$database_oid" ]; then + migration_table=$(sudo -n runuser -u postgres -- psql -At -d "$database" -c "select coalesce(to_regclass('monitor.schema_migrations')::text, '');") + if [ -n "$migration_table" ]; then + schema_version_observed=$(sudo -n runuser -u postgres -- psql -At -d "$database" -c "select coalesce((select version from monitor.schema_migrations order by applied_at desc limit 1), '');") + fi + if [ "$schema_version_observed" = "$schema_version" ]; then + schema_ready=true + business_rows=$(sudo -n runuser -u postgres -- psql -At -d "$database" -c "select (select count(*) from monitor.runners) + (select count(*) from monitor.runs) + (select count(*) from monitor.run_payloads) + (select count(*) from monitor.findings) + (select count(*) from monitor.artifact_locators) + (select count(*) from monitor.timeline_events) + (select count(*) from monitor.runtime_metadata) + (select count(*) from monitor.import_manifests);") + fi +fi +empty=false +ready=false +if [ "$schema_ready" = true ] && [ "$business_rows" = "0" ]; then empty=true; ready=true; fi +printf '{"ok":true,"databaseExists":%s,"schemaReady":%s,"empty":%s,"ready":%s,"databaseOid":"%s","schemaVersion":"%s","businessRows":%s,"siblingDatabaseOids":"%s"}\n' "$([ -n "$database_oid" ] && printf true || printf false)" "$schema_ready" "$empty" "$ready" "$database_oid" "$schema_version_observed" "$([ -n "$business_rows" ] && printf '%s' "$business_rows" || printf null)" "$sibling_oids" +`; +} + +export function monitorResetScript(pg: PostgresHostConfig, target: MonitorResetConfig): string { + const siblings = pg.objects.databases.filter((database) => database.name !== target.database).map((database) => database.name).join(","); + const siblingArray = pg.objects.databases.filter((database) => database.name !== target.database).map((database) => sqlLiteral(database.name)).join(", "); + const migrations = MONITOR_CENTRAL_MIGRATIONS.map((statement) => `${statement};`).join("\n"); + return `${monitorScriptPrelude()} +database=${shellQuote(target.database)} +owner=${shellQuote(target.owner)} +siblings=${shellQuote(siblings)} +before=$(sudo -n runuser -u postgres -- psql -At -d postgres -c "select coalesce(string_agg(datname || ':' || oid, ',' order by datname), '') from pg_database where datname in (${siblingArray});") +sudo -n runuser -u postgres -- psql -v ON_ERROR_STOP=1 -d postgres <<'SQL' +select pg_terminate_backend(pid) from pg_stat_activity where datname = ${sqlLiteral(target.database)} and pid <> pg_backend_pid(); +drop database if exists "${target.database}"; +create database "${target.database}" owner "${target.owner}" encoding 'UTF8' locale 'C.UTF-8' template template0; +SQL +sudo -n runuser -u postgres -- psql -v ON_ERROR_STOP=1 -d "$database" <<'SQL' +set role "${target.owner}"; +${migrations} +insert into monitor.schema_migrations (version, applied_at) values (${sqlLiteral(MONITOR_CENTRAL_SCHEMA_VERSION)}, now()) on conflict (version) do nothing; +SQL +after=$(sudo -n runuser -u postgres -- psql -At -d postgres -c "select coalesce(string_agg(datname || ':' || oid, ',' order by datname), '') from pg_database where datname in (${siblingArray});") +rows=$(sudo -n runuser -u postgres -- psql -At -d "$database" -c "select (select count(*) from monitor.runners) + (select count(*) from monitor.runs) + (select count(*) from monitor.run_payloads) + (select count(*) from monitor.findings) + (select count(*) from monitor.artifact_locators) + (select count(*) from monitor.timeline_events) + (select count(*) from monitor.runtime_metadata) + (select count(*) from monitor.import_manifests);") +version=$(sudo -n runuser -u postgres -- psql -At -d "$database" -c "select version from monitor.schema_migrations order by applied_at desc limit 1;") +if [ "$before" != "$after" ] || [ "$rows" != "0" ] || [ "$version" != ${shellQuote(MONITOR_CENTRAL_SCHEMA_VERSION)} ]; then exit 1; fi +printf '{"ok":true,"database":"%s","schema":"monitor","schemaVersion":"%s","businessRows":0,"siblingDatabasesUnchanged":true}\n' "$database" "$version" +`; +} + +function monitorScriptPrelude(): string { + return `set -eu +if ! command -v sudo >/dev/null 2>&1 || ! sudo -n true >/dev/null 2>&1; then + printf '{"ok":false,"error":"sudo-noninteractive-unavailable"}\n' + exit 1 +fi`; +} + +function sqlLiteral(value: string): string { + return `'${value.replaceAll("'", "''")}'`; +} + function inspectSecrets(pg: PostgresHostConfig, materialize: boolean): SecretInspection { const root = secretRoot(pg); const materials = new Map(); From eed475abdc0821d340664474efaff338a1f29002 Mon Sep 17 00:00:00 2001 From: Codex Date: Mon, 13 Jul 2026 07:52:59 +0200 Subject: [PATCH 06/14] =?UTF-8?q?feat:=20=E6=8E=A5=E5=85=A5=20Sub2Rank=20?= =?UTF-8?q?=E8=87=AA=E5=8A=A8=E4=BA=A4=E4=BB=98=E9=93=BE?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- config/platform-infra/gitea.yaml | 22 ++ config/platform-infra/pipelines-as-code.yaml | 67 +++++ config/platform-infra/sub2rank.yaml | 53 +++- ...atform-infra-gitea-gitops-delivery.test.ts | 7 + ...infra-pipelines-as-code-source-artifact.ts | 24 +- .../src/platform-infra-pipelines-as-code.ts | 3 +- scripts/src/platform-infra-sub2rank-config.ts | 185 ++++++++++++- .../src/platform-infra-sub2rank-pipeline.ts | 256 ++++++++++++++++++ scripts/src/platform-infra-sub2rank.ts | 164 +++++------ 9 files changed, 671 insertions(+), 110 deletions(-) create mode 100644 scripts/src/platform-infra-sub2rank-pipeline.ts diff --git a/config/platform-infra/gitea.yaml b/config/platform-infra/gitea.yaml index 3ecbef31..df2919b0 100644 --- a/config/platform-infra/gitea.yaml +++ b/config/platform-infra/gitea.yaml @@ -288,6 +288,28 @@ sourceAuthority: readUrl: http://git-mirror-http.devops-infra.svc.cluster.local:8080/pikasTech/unidesk.git configRef: config/cicd-branch-followers.yaml#controller.source.gitMirrorReadUrl disposition: replaced-by-gitea + - key: sub2rank-nc01 + targetId: NC01 + upstream: + repository: pikasTech/sub2rank + cloneUrl: https://github.com/pikasTech/sub2rank.git + branch: master + gitea: + owner: mirrors + name: pikasTech-sub2rank + mirrorMode: controlled-push + publicRead: true + readUrl: http://gitea-http.devops-infra.svc.cluster.local:3000/mirrors/pikasTech-sub2rank.git + gitops: + branch: master + flushDisposition: not-a-gitops-branch + snapshot: + naming: application-source-snapshot + prefix: refs/unidesk/snapshots/gitea-actions/sub2rank-master-nc01 + legacyGitMirror: + readUrl: http://gitea-http.devops-infra.svc.cluster.local:3000/mirrors/pikasTech-sub2rank.git + configRef: config/platform-infra/sub2rank.yaml#application.remote + disposition: replaced-by-gitea - key: hwlab-jd01-v03 targetId: JD01 upstream: diff --git a/config/platform-infra/pipelines-as-code.yaml b/config/platform-infra/pipelines-as-code.yaml index 4b53e015..81a90780 100644 --- a/config/platform-infra/pipelines-as-code.yaml +++ b/config/platform-infra/pipelines-as-code.yaml @@ -121,6 +121,38 @@ repositories: unidesk_host_runtime_service_port: "4288" unidesk_host_health_path: /health unidesk_host_health_url: http://todo-note.unidesk.svc.cluster.local:4288/health + - id: sub2rank-nc01 + name: sub2rank-nc01 + namespace: devops-infra + providerType: gitea + url: https://gitea.pikapython.com/mirrors/pikasTech-sub2rank + cloneUrl: http://gitea-http.devops-infra.svc.cluster.local:3000/mirrors/pikasTech-sub2rank.git + owner: mirrors + repo: pikasTech-sub2rank + secretName: pac-gitea-sub2rank-nc01 + tokenKey: token + webhookSecretKey: webhook.secret + concurrencyLimit: 1 + params: + git_read_url: http://gitea-http.devops-infra.svc.cluster.local:3000/mirrors/pikasTech-sub2rank.git + source_branch: master + source_snapshot_prefix: refs/unidesk/snapshots/gitea-actions/sub2rank-master-nc01 + node: NC01 + image_repository: 127.0.0.1:5000/sub2rank/sub2rank + registry_probe_base: http://127.0.0.1:5000 + gitops_branch: unidesk-host-gitops + gitops_manifest_path: deploy/gitops/platform-infra/sub2rank-nc01/resources.yaml + pipeline_name: platform-infra-sub2rank-nc01-pac + pipeline_run_prefix: sub2rank-nc01 + service_account: default + pipeline_timeout: 600s + workspace_pvc_size: 4Gi + runtime_namespace: platform-infra + runtime_deployment: sub2rank + runtime_service: sub2rank + runtime_service_port: "8080" + health_path: /health + health_url: http://sub2rank.platform-infra.svc.cluster.local:8080/health - extends: templates.repositories.hwlabV03 variables: NODE: JD01 @@ -192,6 +224,41 @@ consumers: argoNamespace: argocd argoApplication: platform-infra-gitea-nc01 closeoutGitOpsMirrorFlush: false + - id: platform-infra-sub2rank-nc01 + repositoryRef: sub2rank-nc01 + node: NC01 + lane: platform-infra + namespace: devops-infra + pipeline: platform-infra-sub2rank-nc01-pac + pipelineRunPrefix: sub2rank-nc01 + argoNamespace: argocd + argoApplication: sub2rank-nc01 + closeoutGitOpsMirrorFlush: true + closeoutGitOpsMirrorLane: v03 + argoBootstrap: + project: default + repoUrl: http://git-mirror-http.devops-infra.svc.cluster.local:8080/pikasTech/unidesk.git + targetRevision: unidesk-host-gitops + path: deploy/gitops/platform-infra/sub2rank-nc01 + destinationNamespace: platform-infra + automated: true + deliveryProvenance: + required: true + markerValue: admission-pac-v2:platform-infra-sub2rank-nc01 + executionServiceAccountName: default + gitOps: + repoUrl: http://git-mirror-http.devops-infra.svc.cluster.local:8080/pikasTech/unidesk.git + targetRevision: unidesk-host-gitops + sourceArtifact: + mode: embedded-pipeline-spec + renderer: sub2rank-platform-service + configRef: config/platform-infra/sub2rank.yaml#delivery + pipelineRunPath: .tekton/sub2rank-nc01-pac.yaml + maxKeepRuns: 8 + taskRunTemplate: + hostNetwork: true + dnsPolicy: ClusterFirstWithHostNet + fsGroup: 1000 - extends: templates.consumers.hwlabV03 variables: NODE: JD01 diff --git a/config/platform-infra/sub2rank.yaml b/config/platform-infra/sub2rank.yaml index 4f6e27f5..7594352a 100644 --- a/config/platform-infra/sub2rank.yaml +++ b/config/platform-infra/sub2rank.yaml @@ -10,15 +10,66 @@ defaults: targetId: NC01 application: + repository: pikasTech/sub2rank + remote: https://github.com/pikasTech/sub2rank.git + branch: master sourceRoot: /root/sub2rank configRef: /root/sub2rank/config/sub2rank.yaml + sourceConfigPath: config/sub2rank.yaml + dockerfile: Dockerfile runtimeTarget: k8s image: repository: 127.0.0.1:5000/sub2rank/sub2rank - tag: 14f51f5e1242d0e048f51bbea5937e3022361e9a pullPolicy: IfNotPresent +delivery: + enabled: true + pipeline: + namespace: devops-infra + name: platform-infra-sub2rank-nc01-pac + runPrefix: sub2rank-nc01 + serviceAccountName: default + timeout: 600s + workspaceSize: 4Gi + toolsImage: 127.0.0.1:5000/hwlab/hwlab-ci-node-tools:node22-alpine-bun-v1 + buildkitImage: 127.0.0.1:5000/hwlab/buildkit:rootless + sourceSnapshotPrefix: refs/unidesk/snapshots/gitea-actions/sub2rank-master-nc01 + build: + networkMode: host + proxy: + http: http://127.0.0.1:10808 + https: http://127.0.0.1:10808 + all: http://127.0.0.1:10808 + noProxy: + - localhost + - 127.0.0.1 + - "::1" + - 127.0.0.1:5000 + - localhost:5000 + - .svc + - .svc.cluster.local + - .cluster.local + - hyueapi.com + - .hyueapi.com + gitops: + readUrl: http://git-mirror-http.devops-infra.svc.cluster.local:8080/pikasTech/unidesk.git + writeUrl: http://git-mirror-write.devops-infra.svc.cluster.local:8080/pikasTech/unidesk.git + branch: unidesk-host-gitops + manifestPath: deploy/gitops/platform-infra/sub2rank-nc01/resources.yaml + releaseStatePath: deploy/gitops-state/platform-infra/sub2rank-nc01.json + maxPushAttempts: 3 + author: + name: UniDesk Sub2Rank CI + email: sub2rank-ci@unidesk.local + application: + name: sub2rank-nc01 + namespace: argocd + project: default + path: deploy/gitops/platform-infra/sub2rank-nc01 + destinationNamespace: platform-infra + automated: true + targets: - id: NC01 route: NC01:k3s diff --git a/scripts/src/platform-infra-gitea-gitops-delivery.test.ts b/scripts/src/platform-infra-gitea-gitops-delivery.test.ts index 0d58a6bc..3c8fbdff 100644 --- a/scripts/src/platform-infra-gitea-gitops-delivery.test.ts +++ b/scripts/src/platform-infra-gitea-gitops-delivery.test.ts @@ -78,12 +78,19 @@ test("owning YAML renders one child Application and the complete durable bridge "Deployment", "Role", "RoleBinding", + "Role", + "RoleBinding", "ValidatingAdmissionPolicy", "ValidatingAdmissionPolicyBinding", "ServiceAccount", "ConfigMap", "Deployment", ]); + const pacReadOnlyNamespaces = documents + .filter((item) => item.kind === "Role" && item.metadata?.name === "unidesk-pac-consumer-runner") + .map((item) => item.metadata.namespace) + .sort(); + expect(pacReadOnlyNamespaces).toEqual(["agentrun-ci", "devops-infra"]); const candidate = documents.find((item) => item.kind === "ConfigMap" && item.metadata?.name === "gitea-github-sync-candidate"); const activeConfig = documents.find((item) => item.kind === "ConfigMap" && item.metadata?.name === "gitea-github-sync-config"); const bridge = documents.find((item) => item.kind === "Deployment" && item.metadata?.name === "gitea-github-sync"); diff --git a/scripts/src/platform-infra-pipelines-as-code-source-artifact.ts b/scripts/src/platform-infra-pipelines-as-code-source-artifact.ts index 344a1ea9..7ffdca93 100644 --- a/scripts/src/platform-infra-pipelines-as-code-source-artifact.ts +++ b/scripts/src/platform-infra-pipelines-as-code-source-artifact.ts @@ -17,9 +17,10 @@ import type { RenderedCliResult } from "./output"; import { renderedCliResult } from "./agentrun/render"; import { stableJsonSha256, stableJsonValue } from "./stable-json"; import { pipelineProvenanceAnnotations, pipelineProvenanceFromManifest, withPipelineProvenanceAnnotations } from "./pipeline-provenance"; +import { renderSub2RankDesiredPipeline, sub2RankOwningSourceRemote } from "./platform-infra-sub2rank-pipeline"; export type PacSourceArtifactMode = "embedded-pipeline-spec" | "remote-pipeline-annotation"; -export type PacSourceArtifactRenderer = "agentrun-control-plane" | "hwlab-runtime-lane"; +export type PacSourceArtifactRenderer = "agentrun-control-plane" | "hwlab-runtime-lane" | "sub2rank-platform-service"; export type PacSourceArtifactAction = "plan" | "check" | "write" | "status" | "verify-runtime"; export type PacSourceArtifactRuntimeAlignment = "not-requested" | "aligned" | "drifted" | "missing" | "unavailable"; @@ -441,7 +442,9 @@ function renderDesiredArtifact(binding: PacSourceArtifactBinding, sourceWorktree const sourceArtifact = binding.consumer.sourceArtifact; const rendered = sourceArtifact.renderer === "agentrun-control-plane" ? renderAgentRunDesiredPipeline(binding) - : renderHwlabDesiredPipeline(binding, sourceWorktree); + : sourceArtifact.renderer === "hwlab-runtime-lane" + ? renderHwlabDesiredPipeline(binding, sourceWorktree) + : renderSub2RankDesiredPipeline(binding); const pipeline = sourceArtifact.renderer === "hwlab-runtime-lane" ? rendered.pipeline : withPipelineProvenanceAnnotations(rendered.pipeline, rendered.provenance); @@ -581,7 +584,7 @@ function embeddedPipelineRun(binding: PacSourceArtifactBinding, desiredSpec: Rec name: `${binding.consumer.pipelineRunPrefix}-{{ revision }}`, namespace: binding.consumer.namespace, annotations: pipelineRunAnnotations(binding, provenance), - labels: pipelineRunLabels(binding, "agentrun"), + labels: pipelineRunLabels(binding, provenance.renderer === "sub2rank-platform-service" ? "sub2rank" : "agentrun"), }, spec: { pipelineSpec: desiredSpec, @@ -647,7 +650,7 @@ function pipelineRunAnnotations(binding: PacSourceArtifactBinding, provenance: P }; } -function pipelineRunLabels(binding: PacSourceArtifactBinding, partOf: "agentrun" | "hwlab"): Record { +function pipelineRunLabels(binding: PacSourceArtifactBinding, partOf: "agentrun" | "hwlab" | "sub2rank"): Record { return partOf === "agentrun" ? { "app.kubernetes.io/part-of": "agentrun", @@ -656,13 +659,19 @@ function pipelineRunLabels(binding: PacSourceArtifactBinding, partOf: "agentrun" "agentrun.pikastech.local/source-commit": "{{ revision }}", "agentrun.pikastech.local/trigger": "pipelines-as-code", } - : { + : partOf === "hwlab" ? { "app.kubernetes.io/name": `${binding.consumer.id}-pac`, "app.kubernetes.io/part-of": "hwlab", "unidesk.ai/source-commit": "{{ revision }}", "hwlab.pikastech.local/gitops-target": binding.consumer.lane, "hwlab.pikastech.local/source-commit": "{{ revision }}", "hwlab.pikastech.local/trigger": "pipelines-as-code", + } : { + "app.kubernetes.io/name": "sub2rank", + "app.kubernetes.io/part-of": "platform-infra", + "unidesk.ai/node": binding.consumer.node, + "unidesk.ai/source-commit": "{{ revision }}", + "unidesk.ai/trigger": "pipelines-as-code", }; } @@ -808,6 +817,9 @@ function owningSourceRemote(binding: PacSourceArtifactBinding): string { if (binding.consumer.sourceArtifact.renderer === "agentrun-control-plane") { return resolveAgentRunLaneTarget({ node: binding.consumer.node, lane: binding.consumer.lane }).spec.source.worktreeRemote; } + if (binding.consumer.sourceArtifact.renderer === "sub2rank-platform-service") { + return sub2RankOwningSourceRemote(); + } if (!isHwlabRuntimeLane(binding.consumer.lane)) throw new PacSourceArtifactError("owning-source-lane-unresolved"); return hwlabRuntimeLaneSpecForNode(binding.consumer.lane, binding.consumer.node).gitUrl; } @@ -923,7 +935,7 @@ function assertPipelineIdentity(pipeline: Record, binding: PacS function provenanceFromManifest(manifest: Record): PacSourceArtifactProvenance | null { const provenance = pipelineProvenanceFromManifest(manifest); if (provenance === null) return null; - if (provenance.renderer !== "agentrun-control-plane" && provenance.renderer !== "hwlab-runtime-lane") return null; + if (provenance.renderer !== "agentrun-control-plane" && provenance.renderer !== "hwlab-runtime-lane" && provenance.renderer !== "sub2rank-platform-service") return null; if (provenance.mode !== "embedded-pipeline-spec" && provenance.mode !== "remote-pipeline-annotation") return null; return provenance as PacSourceArtifactProvenance; } diff --git a/scripts/src/platform-infra-pipelines-as-code.ts b/scripts/src/platform-infra-pipelines-as-code.ts index eb5b7d99..21eff00c 100644 --- a/scripts/src/platform-infra-pipelines-as-code.ts +++ b/scripts/src/platform-infra-pipelines-as-code.ts @@ -537,7 +537,7 @@ function parseConsumerDeliveryProvenance(value: Record, path: s function parseSourceArtifact(value: Record, path: string): PacSourceArtifactSpec { const mode = y.enumField(value, "mode", path, ["embedded-pipeline-spec", "remote-pipeline-annotation"] as const); - const renderer = y.enumField(value, "renderer", path, ["agentrun-control-plane", "hwlab-runtime-lane"] as const); + const renderer = y.enumField(value, "renderer", path, ["agentrun-control-plane", "hwlab-runtime-lane", "sub2rank-platform-service"] as const); const pipelineRunPath = sourceArtifactPath(y.stringField(value, "pipelineRunPath", path), `${path}.pipelineRunPath`); const pipelinePath = value.pipelinePath === undefined ? null : sourceArtifactPath(y.stringField(value, "pipelinePath", path), `${path}.pipelinePath`); const taskRunTemplate = y.objectField(value, "taskRunTemplate", path); @@ -545,6 +545,7 @@ function parseSourceArtifact(value: Record, path: string): PacS if (mode === "remote-pipeline-annotation" && pipelinePath === null) throw new Error(`${path}.pipelinePath is required for remote-pipeline-annotation`); if (renderer === "agentrun-control-plane" && mode !== "embedded-pipeline-spec") throw new Error(`${path}.renderer agentrun-control-plane requires embedded-pipeline-spec`); if (renderer === "hwlab-runtime-lane" && mode !== "remote-pipeline-annotation") throw new Error(`${path}.renderer hwlab-runtime-lane requires remote-pipeline-annotation`); + if (renderer === "sub2rank-platform-service" && mode !== "embedded-pipeline-spec") throw new Error(`${path}.renderer sub2rank-platform-service requires embedded-pipeline-spec`); return { mode, renderer, diff --git a/scripts/src/platform-infra-sub2rank-config.ts b/scripts/src/platform-infra-sub2rank-config.ts index e977a4e7..aa1f6d7f 100644 --- a/scripts/src/platform-infra-sub2rank-config.ts +++ b/scripts/src/platform-infra-sub2rank-config.ts @@ -1,6 +1,6 @@ import { createHash } from "node:crypto"; -import { readFileSync } from "node:fs"; -import { relative } from "node:path"; +import { existsSync, readFileSync } from "node:fs"; +import { dirname, relative, resolve } from "node:path"; import { rootPath } from "./config"; import type { PublicServiceExposure, PublicServiceTarget } from "./platform-infra-public-service"; import { assertNoDuplicateYamlMappingKeys, createYamlFieldReader, readYamlRecord, resolveRepoPath } from "./platform-infra-ops-library"; @@ -16,20 +16,61 @@ export interface Sub2RankConfig { metadata: { id: string; owner: string; relatedIssues: number[] }; defaults: { targetId: string }; application: { + repository: string; + remote: string; + branch: string; sourceRoot: string; configRef: string; + sourceConfigPath: string; + dockerfile: string; runtimeTarget: string; configText: string; configSha256: string; sourceCommit: string; sourceClean: boolean; sourceRemotePresent: boolean; + sourceRemoteMatches: boolean; configMatchesSourceCommit: boolean; deploymentBlockPresent: boolean; automaticCreditEnabled: boolean; server: { listenPort: number; databasePath: string; envKeys: string[] }; }; - image: { repository: string; tag: string; pullPolicy: "Always" | "IfNotPresent" | "Never" }; + image: { repository: string; pullPolicy: "Always" | "IfNotPresent" | "Never" }; + delivery: { + enabled: boolean; + pipeline: { + namespace: string; + name: string; + runPrefix: string; + serviceAccountName: string; + timeout: string; + workspaceSize: string; + toolsImage: string; + buildkitImage: string; + sourceSnapshotPrefix: string; + }; + build: { + networkMode: "host"; + proxy: { http: string; https: string; all: string; noProxy: string[] }; + }; + gitops: { + readUrl: string; + writeUrl: string; + branch: string; + manifestPath: string; + releaseStatePath: string; + maxPushAttempts: number; + author: { name: string; email: string }; + application: { + name: string; + namespace: string; + project: string; + path: string; + destinationNamespace: string; + automated: true; + }; + }; + }; targets: Sub2RankTarget[]; runtime: { sharedNetworkPolicyRef: { name: string; managedBySub2Rank: false }; @@ -82,15 +123,17 @@ export function readSub2RankConfig(): Sub2RankConfig { const defaultsRecord = y.objectField(root, "defaults", ""); const applicationRecord = y.objectField(root, "application", ""); const imageRecord = y.objectField(root, "image", ""); + const deliveryRecord = y.objectField(root, "delivery", ""); const runtimeRecord = y.objectField(root, "runtime", ""); const defaults = { targetId: simpleId(y.stringField(defaultsRecord, "targetId", "defaults"), "defaults.targetId") }; const application = parseApplication(applicationRecord); const image = parseImage(imageRecord); + const delivery = parseDelivery(deliveryRecord); const runtimeBase = parseRuntime(runtimeRecord); const secret = resolveSecretDeclaration(runtimeBase.secret.configRef, runtimeBase.secret.declarationId); const targets = parseTargets(root.targets, secret); if (!targets.some((target) => target.id === defaults.targetId)) throw new Error(`${sub2RankConfigLabel}.targets must include defaults.targetId ${defaults.targetId}`); - validateResolvedConfig(application, image, runtimeBase, secret, targets); + validateResolvedConfig(application, image, delivery, runtimeBase, secret, targets); return { version, kind: "platform-infra-sub2rank", @@ -102,6 +145,7 @@ export function readSub2RankConfig(): Sub2RankConfig { defaults, application, image, + delivery, targets, runtime: { ...runtimeBase, secret: { ...runtimeBase.secret, ...secret } }, }; @@ -116,10 +160,16 @@ export function resolveSub2RankTarget(config: Sub2RankConfig, targetId: string | } function parseApplication(record: Record): Sub2RankConfig["application"] { + const repository = repositoryValue(y.stringField(record, "repository", "application"), "application.repository"); + const remote = y.stringField(record, "remote", "application"); + const branch = simpleId(y.stringField(record, "branch", "application"), "application.branch"); const sourceRoot = y.absolutePathField(record, "sourceRoot", "application"); const configRef = y.absolutePathField(record, "configRef", "application"); + const sourceConfigPath = safeRelativePath(y.stringField(record, "sourceConfigPath", "application"), "application.sourceConfigPath"); + const dockerfile = safeRelativePath(y.stringField(record, "dockerfile", "application"), "application.dockerfile"); const runtimeTarget = simpleId(y.stringField(record, "runtimeTarget", "application"), "application.runtimeTarget"); if (!configRef.startsWith(`${sourceRoot.replace(/\/+$/u, "")}/`)) throw new Error(`${sub2RankConfigLabel}.application.configRef must be inside application.sourceRoot`); + if (!existsSync(resolve(sourceRoot, dockerfile))) throw new Error(`${sub2RankConfigLabel}.application.dockerfile does not exist under application.sourceRoot`); const configText = readFileSync(configRef, "utf8"); assertNoDuplicateYamlMappingKeys(configText, configRef); const app = y.asRecord(Bun.YAML.parse(configText), configRef); @@ -140,18 +190,25 @@ function parseApplication(record: Record): Sub2RankConfig["appl const sourceCommit = gitText(sourceRoot, ["rev-parse", "HEAD"], "resolve Sub2Rank source commit"); if (!/^[a-f0-9]{40}$/u.test(sourceCommit)) throw new Error(`${sourceRoot} HEAD must resolve to a full Git commit`); const statusShort = gitText(sourceRoot, ["status", "--porcelain"], "read Sub2Rank source status", true); - const remote = gitText(sourceRoot, ["remote"], "read Sub2Rank source remotes", true); + const observedRemote = gitText(sourceRoot, ["remote", "get-url", "origin"], "read Sub2Rank origin", true); const configRelativePath = relative(sourceRoot, configRef).replaceAll("\\", "/"); + if (configRelativePath !== sourceConfigPath) throw new Error(`${sub2RankConfigLabel}.application.sourceConfigPath must resolve to application.configRef`); const committedConfig = gitText(sourceRoot, ["show", `${sourceCommit}:${configRelativePath}`], "read committed Sub2Rank application config", true, false); return { + repository, + remote, + branch, sourceRoot, configRef, + sourceConfigPath, + dockerfile, runtimeTarget, configText, configSha256: createHash("sha256").update(configText).digest("hex"), sourceCommit, sourceClean: statusShort.length === 0, - sourceRemotePresent: remote.split(/\r?\n/u).some((value) => value.trim().length > 0), + sourceRemotePresent: observedRemote.length > 0, + sourceRemoteMatches: sameRepositoryIdentity(remote, observedRemote), configMatchesSourceCommit: committedConfig === configText.trimEnd(), deploymentBlockPresent: app.deployment !== undefined, automaticCreditEnabled, @@ -165,11 +222,66 @@ function parseApplication(record: Record): Sub2RankConfig["appl function parseImage(record: Record): Sub2RankConfig["image"] { const repository = y.stringField(record, "repository", "image"); - const tag = y.stringField(record, "tag", "image"); const pullPolicy = y.enumField(record, "pullPolicy", "image", ["Always", "IfNotPresent", "Never"] as const); if (!/^[A-Za-z0-9._:/-]+$/u.test(repository) || repository.endsWith("/")) throw new Error(`${sub2RankConfigLabel}.image.repository has an unsupported format`); - if (!/^[A-Za-z0-9._-]+$/u.test(tag)) throw new Error(`${sub2RankConfigLabel}.image.tag has an unsupported format`); - return { repository, tag, pullPolicy }; + return { repository, pullPolicy }; +} + +function parseDelivery(record: Record): Sub2RankConfig["delivery"] { + const pipeline = y.objectField(record, "pipeline", "delivery"); + const build = y.objectField(record, "build", "delivery"); + const proxy = y.objectField(build, "proxy", "delivery.build"); + const gitops = y.objectField(record, "gitops", "delivery"); + const author = y.objectField(gitops, "author", "delivery.gitops"); + const application = y.objectField(gitops, "application", "delivery.gitops"); + const timeout = y.stringField(pipeline, "timeout", "delivery.pipeline"); + if (!/^[1-9][0-9]*s$/u.test(timeout)) throw new Error(`${sub2RankConfigLabel}.delivery.pipeline.timeout must be positive seconds`); + const sourceSnapshotPrefix = y.stringField(pipeline, "sourceSnapshotPrefix", "delivery.pipeline"); + if (!/^refs\/[A-Za-z0-9._/-]+$/u.test(sourceSnapshotPrefix) || sourceSnapshotPrefix.includes("..")) throw new Error(`${sub2RankConfigLabel}.delivery.pipeline.sourceSnapshotPrefix must be a safe refs/ prefix`); + const email = y.stringField(author, "email", "delivery.gitops.author"); + if (!/^[^@\s]+@[^@\s]+$/u.test(email)) throw new Error(`${sub2RankConfigLabel}.delivery.gitops.author.email must be an email address`); + const automated = y.booleanField(application, "automated", "delivery.gitops.application"); + if (!automated) throw new Error(`${sub2RankConfigLabel}.delivery.gitops.application.automated must remain true`); + return { + enabled: y.booleanField(record, "enabled", "delivery"), + pipeline: { + namespace: y.kubernetesNameField(pipeline, "namespace", "delivery.pipeline"), + name: y.kubernetesNameField(pipeline, "name", "delivery.pipeline"), + runPrefix: y.kubernetesNameField(pipeline, "runPrefix", "delivery.pipeline"), + serviceAccountName: y.kubernetesNameField(pipeline, "serviceAccountName", "delivery.pipeline"), + timeout, + workspaceSize: storageSize(y.stringField(pipeline, "workspaceSize", "delivery.pipeline")), + toolsImage: imageValue(y.stringField(pipeline, "toolsImage", "delivery.pipeline"), "delivery.pipeline.toolsImage"), + buildkitImage: imageValue(y.stringField(pipeline, "buildkitImage", "delivery.pipeline"), "delivery.pipeline.buildkitImage"), + sourceSnapshotPrefix, + }, + build: { + networkMode: y.enumField(build, "networkMode", "delivery.build", ["host"] as const), + proxy: { + http: httpUrlValue(y.stringField(proxy, "http", "delivery.build.proxy"), "delivery.build.proxy.http"), + https: httpUrlValue(y.stringField(proxy, "https", "delivery.build.proxy"), "delivery.build.proxy.https"), + all: httpUrlValue(y.stringField(proxy, "all", "delivery.build.proxy"), "delivery.build.proxy.all"), + noProxy: y.stringArrayField(proxy, "noProxy", "delivery.build.proxy"), + }, + }, + gitops: { + readUrl: httpUrlValue(y.stringField(gitops, "readUrl", "delivery.gitops"), "delivery.gitops.readUrl"), + writeUrl: httpUrlValue(y.stringField(gitops, "writeUrl", "delivery.gitops"), "delivery.gitops.writeUrl"), + branch: simpleId(y.stringField(gitops, "branch", "delivery.gitops"), "delivery.gitops.branch"), + manifestPath: safeRelativePath(y.stringField(gitops, "manifestPath", "delivery.gitops"), "delivery.gitops.manifestPath"), + releaseStatePath: safeRelativePath(y.stringField(gitops, "releaseStatePath", "delivery.gitops"), "delivery.gitops.releaseStatePath"), + maxPushAttempts: positiveInteger(gitops, "maxPushAttempts", "delivery.gitops"), + author: { name: y.stringField(author, "name", "delivery.gitops.author"), email }, + application: { + name: y.kubernetesNameField(application, "name", "delivery.gitops.application"), + namespace: y.kubernetesNameField(application, "namespace", "delivery.gitops.application"), + project: y.kubernetesNameField(application, "project", "delivery.gitops.application"), + path: safeRelativePath(y.stringField(application, "path", "delivery.gitops.application"), "delivery.gitops.application.path"), + destinationNamespace: y.kubernetesNameField(application, "destinationNamespace", "delivery.gitops.application"), + automated: true, + }, + }, + }; } function parseRuntime(record: Record): Omit & { @@ -328,13 +440,19 @@ function resolveSecretDeclaration(configRef: string, declarationId: string): Res function validateResolvedConfig( application: Sub2RankConfig["application"], image: Sub2RankConfig["image"], + delivery: Sub2RankConfig["delivery"], runtime: ReturnType, secret: ResolvedSecretDeclaration, targets: Sub2RankTarget[], ): void { - if (!/^[a-f0-9]{40}$/u.test(image.tag) || image.tag !== application.sourceCommit) throw new Error(`${sub2RankConfigLabel}.image.tag must equal the full application source commit ${application.sourceCommit}`); - if (!application.sourceClean) throw new Error(`${sub2RankConfigLabel}.application.sourceRoot must be clean before rendering deployment config`); if (!application.configMatchesSourceCommit) throw new Error(`${sub2RankConfigLabel}.application.configRef must match the file stored at application source commit ${application.sourceCommit}`); + if (!application.sourceRemoteMatches) throw new Error(`${sub2RankConfigLabel}.application.remote must identify the application.sourceRoot origin repository`); + if (!sameRepositoryIdentity(application.remote, `https://github.com/${application.repository}.git`)) throw new Error(`${sub2RankConfigLabel}.application.repository must match application.remote`); + if (!delivery.enabled) throw new Error(`${sub2RankConfigLabel}.delivery.enabled must remain true while Sub2Rank is deployed`); + if (delivery.gitops.maxPushAttempts > 5) throw new Error(`${sub2RankConfigLabel}.delivery.gitops.maxPushAttempts must be <= 5`); + if (dirname(delivery.gitops.manifestPath).replaceAll("\\", "/") !== delivery.gitops.application.path) throw new Error(`${sub2RankConfigLabel}.delivery.gitops.manifestPath must be directly under delivery.gitops.application.path`); + if (!delivery.build.proxy.noProxy.includes("hyueapi.com") || !delivery.build.proxy.noProxy.includes(".hyueapi.com")) throw new Error(`${sub2RankConfigLabel}.delivery.build.proxy.noProxy must retain hyueapi.com and .hyueapi.com`); + if (!image.repository.startsWith("127.0.0.1:5000/")) throw new Error(`${sub2RankConfigLabel}.image.repository must use the NC01 local registry`); const required = new Set(runtime.secret.requiredTargetKeys); const provided = new Set(secret.mappings.map((item) => item.targetKey)); const missing = [...required].filter((key) => !provided.has(key)); @@ -345,6 +463,7 @@ function validateResolvedConfig( for (const target of targets) { if (target.route !== secret.route || target.namespace !== secret.namespace) throw new Error(`${sub2RankConfigLabel}.targets[${target.id}] route/namespace must match runtime Secret target ${secret.targetId}`); if (target.publicExposure.frpc.localPort !== runtime.service.port) throw new Error(`${sub2RankConfigLabel}.targets[${target.id}].publicExposure.frpc.localPort must match runtime.service.port`); + if (target.namespace !== delivery.gitops.application.destinationNamespace) throw new Error(`${sub2RankConfigLabel}.delivery.gitops.application.destinationNamespace must match the runtime target namespace`); } } @@ -403,6 +522,50 @@ function sourceRefValue(value: unknown, path: string): string { return text; } +function safeRelativePath(value: string, path: string): string { + if (value.startsWith("/") || value.split(/[\\/]/u).includes("..") || !/^[A-Za-z0-9._/-]+$/u.test(value)) throw new Error(`${sub2RankConfigLabel}.${path} must be a safe relative path`); + return value; +} + +function repositoryValue(value: string, path: string): string { + if (!/^[A-Za-z0-9_.-]+\/[A-Za-z0-9_.-]+$/u.test(value)) throw new Error(`${sub2RankConfigLabel}.${path} must be owner/repository`); + return value; +} + +function httpUrlValue(value: string, path: string): string { + let parsed: URL; + try { + parsed = new URL(value); + } catch { + throw new Error(`${sub2RankConfigLabel}.${path} must be an HTTP URL`); + } + if (parsed.protocol !== "http:" && parsed.protocol !== "https:") throw new Error(`${sub2RankConfigLabel}.${path} must be an HTTP URL`); + if (parsed.username.length > 0 || parsed.password.length > 0 || parsed.hash.length > 0) throw new Error(`${sub2RankConfigLabel}.${path} must not contain credentials or a fragment`); + return value; +} + +function sameRepositoryIdentity(left: string, right: string): boolean { + const identity = (value: string): string | null => { + const scp = /^[^/@\s]+@([^:\s]+):(.+)$/u.exec(value.trim()); + let host = scp?.[1] ?? ""; + let path = scp?.[2] ?? ""; + if (host.length === 0 || path.length === 0) { + try { + const parsed = new URL(value.trim()); + host = parsed.hostname; + path = parsed.pathname; + } catch { + return null; + } + } + const parts = path.replace(/^\/+|\/+$/gu, "").split("/"); + if (parts.length !== 2) return null; + return `${host.toLowerCase()}/${parts[0]?.toLowerCase()}/${parts[1]?.replace(/\.git$/iu, "").toLowerCase()}`; + }; + const leftIdentity = identity(left); + return leftIdentity !== null && leftIdentity === identity(right); +} + function kubernetesNameValue(value: unknown, path: string): string { const text = stringValue(value, path); if (!/^[a-z0-9](?:[-a-z0-9.]*[a-z0-9])?$/u.test(text)) throw new Error(`${path} must be a Kubernetes name`); diff --git a/scripts/src/platform-infra-sub2rank-pipeline.ts b/scripts/src/platform-infra-sub2rank-pipeline.ts new file mode 100644 index 00000000..c65ef6f2 --- /dev/null +++ b/scripts/src/platform-infra-sub2rank-pipeline.ts @@ -0,0 +1,256 @@ +import { stableJsonSha256 } from "./stable-json"; +import { + readSub2RankConfig, + resolveSub2RankTarget, + sub2RankConfigLabel, + type Sub2RankConfig, + type Sub2RankTarget, +} from "./platform-infra-sub2rank-config"; +import { + renderSub2RankManifest, + sub2RankConfigBase64Placeholder, + sub2RankConfigSha256Placeholder, + sub2RankImagePlaceholder, + sub2RankSourceCommitPlaceholder, +} from "./platform-infra-sub2rank"; +import type { PacSourceArtifactBinding, PacSourceArtifactMode, PacSourceArtifactRenderer } from "./platform-infra-pipelines-as-code-source-artifact"; + +export interface Sub2RankPipelineProvenance { + readonly configRef: string; + readonly effectiveConfigSha256: string; + readonly renderer: PacSourceArtifactRenderer; + readonly mode: PacSourceArtifactMode; +} + +export function sub2RankOwningSourceRemote(): string { + return readSub2RankConfig().application.remote; +} + +export function renderSub2RankDesiredPipeline(binding: PacSourceArtifactBinding): { + pipeline: Record; + provenance: Sub2RankPipelineProvenance; +} { + const config = readSub2RankConfig(); + const target = resolveSub2RankTarget(config, binding.consumer.node); + const expectedConfigRef = `${sub2RankConfigLabel}#delivery`; + if (binding.consumer.sourceArtifact.configRef !== expectedConfigRef) throw new Error(`Sub2Rank sourceArtifact.configRef must equal ${expectedConfigRef}`); + assertBinding(config, target, binding); + const manifestTemplate = renderSub2RankManifest(config, target, { + imageRef: sub2RankImagePlaceholder, + appConfigBase64: sub2RankConfigBase64Placeholder, + appConfigSha256: sub2RankConfigSha256Placeholder, + sourceCommit: sub2RankSourceCommitPlaceholder, + }); + return { + pipeline: renderPipeline(config, target, binding, manifestTemplate), + provenance: { + configRef: expectedConfigRef, + effectiveConfigSha256: stableJsonSha256(pipelineOwner(config, target)), + renderer: "sub2rank-platform-service", + mode: "embedded-pipeline-spec", + }, + }; +} + +function renderPipeline( + config: Sub2RankConfig, + target: Sub2RankTarget, + binding: PacSourceArtifactBinding, + manifestTemplate: string, +): Record { + const delivery = config.delivery; + const params = [ + { name: "git-read-url", type: "string", default: binding.repository.cloneUrl }, + { name: "source-branch", type: "string", default: config.application.branch }, + { name: "revision", type: "string" }, + { name: "source-stage-ref", type: "string" }, + { name: "config-path", type: "string", default: config.application.sourceConfigPath }, + { name: "dockerfile", type: "string", default: config.application.dockerfile }, + { name: "image-repository", type: "string", default: config.image.repository }, + { name: "tools-image", type: "string", default: delivery.pipeline.toolsImage }, + { name: "buildkit-image", type: "string", default: delivery.pipeline.buildkitImage }, + { name: "build-network", type: "string", default: delivery.build.networkMode }, + { name: "build-http-proxy", type: "string", default: delivery.build.proxy.http }, + { name: "build-https-proxy", type: "string", default: delivery.build.proxy.https }, + { name: "build-all-proxy", type: "string", default: delivery.build.proxy.all }, + { name: "build-no-proxy", type: "string", default: delivery.build.proxy.noProxy.join(",") }, + { name: "gitops-read-url", type: "string", default: delivery.gitops.readUrl }, + { name: "gitops-write-url", type: "string", default: delivery.gitops.writeUrl }, + { name: "gitops-branch", type: "string", default: delivery.gitops.branch }, + { name: "gitops-manifest-path", type: "string", default: delivery.gitops.manifestPath }, + { name: "gitops-release-state-path", type: "string", default: delivery.gitops.releaseStatePath }, + { name: "gitops-max-push-attempts", type: "string", default: String(delivery.gitops.maxPushAttempts) }, + { name: "gitops-author-name", type: "string", default: delivery.gitops.author.name }, + { name: "gitops-author-email", type: "string", default: delivery.gitops.author.email }, + { name: "manifest-template-b64", type: "string", default: Buffer.from(manifestTemplate, "utf8").toString("base64") }, + ]; + const task = (name: string, steps: readonly Record[], runAfter: readonly string[] = []): Record => ({ + name, + ...(runAfter.length === 0 ? {} : { runAfter }), + workspaces: [{ name: "source", workspace: "source" }], + taskSpec: { params: params.map(({ name }) => ({ name })), workspaces: [{ name: "source" }], steps }, + params: params.map(({ name: paramName }) => ({ name: paramName, value: `$(params.${paramName})` })), + }); + return { + apiVersion: "tekton.dev/v1", + kind: "Pipeline", + metadata: { + name: delivery.pipeline.name, + namespace: delivery.pipeline.namespace, + labels: { + "app.kubernetes.io/name": "sub2rank", + "app.kubernetes.io/part-of": "platform-infra", + "app.kubernetes.io/managed-by": "unidesk", + "unidesk.ai/node": target.id, + }, + }, + spec: { + params, + workspaces: [{ name: "source" }], + tasks: [ + task("source-validate", [{ + name: "checkout-and-cli-validate", + image: "$(params.tools-image)", + imagePullPolicy: "IfNotPresent", + env: proxyEnv(), + script: sourceValidateScript(), + }]), + task("image-build", [{ + name: "build-and-push", + image: "$(params.buildkit-image)", + imagePullPolicy: "IfNotPresent", + env: [ + ...proxyEnv(), + { name: "BUILDKITD_FLAGS", value: "--oci-worker-no-process-sandbox --oci-worker-net=host --allow-insecure-entitlement network.host" }, + { name: "SOURCE_ROOT", value: "$(workspaces.source.path)/repo" }, + { name: "SOURCE_COMMIT", value: "$(params.revision)" }, + { name: "IMAGE_REPOSITORY", value: "$(params.image-repository)" }, + { name: "DOCKERFILE", value: "$(params.dockerfile)" }, + { name: "BUILD_NETWORK", value: "$(params.build-network)" }, + { name: "BUILD_METADATA_FILE", value: "$(workspaces.source.path)/build-metadata.json" }, + { name: "BUILD_RESULT_FILE", value: "$(workspaces.source.path)/build-result.json" }, + ], + securityContext: { privileged: true, runAsUser: 1000, runAsGroup: 1000 }, + script: buildScript(), + }], ["source-validate"]), + task("gitops-publish", [{ + name: "publish-digest-manifest", + image: "$(params.tools-image)", + imagePullPolicy: "IfNotPresent", + env: proxyEnv(), + script: gitopsPublishScript(), + }], ["image-build"]), + ], + }, + }; +} + +function sourceValidateScript(): string { + return [ + "#!/bin/sh", + "set -eu", + "root=\"$(workspaces.source.path)\"", + "rm -rf \"$root/repo\"", + "git clone --filter=blob:none --no-checkout \"$(params.git-read-url)\" \"$root/repo\"", + "cd \"$root/repo\"", + "git fetch --depth=1 --filter=blob:none origin \"+$(params.source-stage-ref):refs/remotes/origin/sub2rank-source-snapshot\"", + "git checkout --detach \"$(params.revision)\"", + "test \"$(git rev-parse HEAD)\" = \"$(params.revision)\"", + "bun install --frozen-lockfile", + "bun scripts/sub2rank-cli.ts --config \"$(params.config-path)\" config validate", + "chmod -R a+rX,g+rwX \"$root/repo\"", + "printf '{\"ok\":true,\"phase\":\"source-validate\",\"sourceCommit\":\"%s\",\"configPath\":\"%s\",\"valuesPrinted\":false}\\n' \"$(params.revision)\" \"$(params.config-path)\"", + ].join("\n"); +} + +function buildScript(): string { + return [ + "#!/bin/sh", + "set -eu", + "exec \"$(workspaces.source.path)/repo/scripts/ci/build-image.sh\"", + ].join("\n"); +} + +function gitopsPublishScript(): string { + return [ + "#!/bin/sh", + "set -eu", + "root=\"$(workspaces.source.path)\"", + "exec bun \"$root/repo/scripts/ci/publish-gitops.mjs\" \\", + " --source-root \"$root/repo\" \\", + " --source-commit \"$(params.revision)\" \\", + " --config-path \"$(params.config-path)\" \\", + " --metadata \"$root/build-metadata.json\" \\", + " --manifest-template-b64 \"$(params.manifest-template-b64)\" \\", + " --image-repository \"$(params.image-repository)\" \\", + " --gitops-read-url \"$(params.gitops-read-url)\" \\", + " --gitops-write-url \"$(params.gitops-write-url)\" \\", + " --gitops-branch \"$(params.gitops-branch)\" \\", + " --manifest-path \"$(params.gitops-manifest-path)\" \\", + " --release-state-path \"$(params.gitops-release-state-path)\" \\", + " --max-push-attempts \"$(params.gitops-max-push-attempts)\" \\", + " --author-name \"$(params.gitops-author-name)\" \\", + " --author-email \"$(params.gitops-author-email)\" \\", + " --worktree \"$root/gitops\"", + ].join("\n"); +} + +function proxyEnv(): Record[] { + return [ + { name: "HTTP_PROXY", value: "$(params.build-http-proxy)" }, + { name: "http_proxy", value: "$(params.build-http-proxy)" }, + { name: "HTTPS_PROXY", value: "$(params.build-https-proxy)" }, + { name: "https_proxy", value: "$(params.build-https-proxy)" }, + { name: "ALL_PROXY", value: "$(params.build-all-proxy)" }, + { name: "all_proxy", value: "$(params.build-all-proxy)" }, + { name: "NO_PROXY", value: "$(params.build-no-proxy)" }, + { name: "no_proxy", value: "$(params.build-no-proxy)" }, + ]; +} + +function assertBinding(config: Sub2RankConfig, target: Sub2RankTarget, binding: PacSourceArtifactBinding): void { + const delivery = config.delivery; + const expected: Readonly> = { + node: target.id, + lane: "platform-infra", + namespace: delivery.pipeline.namespace, + pipeline: delivery.pipeline.name, + pipelineRunPrefix: delivery.pipeline.runPrefix, + service_account: delivery.pipeline.serviceAccountName, + source_branch: config.application.branch, + source_snapshot_prefix: delivery.pipeline.sourceSnapshotPrefix, + image_repository: config.image.repository, + gitops_branch: delivery.gitops.branch, + gitops_manifest_path: delivery.gitops.manifestPath, + pipeline_name: delivery.pipeline.name, + pipeline_run_prefix: delivery.pipeline.runPrefix, + }; + const observed: Readonly> = { + node: binding.consumer.node, + lane: binding.consumer.lane, + namespace: binding.consumer.namespace, + pipeline: binding.consumer.pipeline, + pipelineRunPrefix: binding.consumer.pipelineRunPrefix, + ...binding.repository.params, + }; + for (const [key, value] of Object.entries(expected)) { + if (observed[key] !== value) throw new Error(`Sub2Rank PaC binding ${key}=${String(observed[key])} does not match owning YAML ${value}`); + } +} + +function pipelineOwner(config: Sub2RankConfig, target: Sub2RankTarget): Record { + return { + application: { + repository: config.application.repository, + remote: config.application.remote, + branch: config.application.branch, + sourceConfigPath: config.application.sourceConfigPath, + dockerfile: config.application.dockerfile, + runtimeTarget: config.application.runtimeTarget, + }, + image: config.image, + delivery: config.delivery, + target, + runtime: config.runtime, + }; +} diff --git a/scripts/src/platform-infra-sub2rank.ts b/scripts/src/platform-infra-sub2rank.ts index f4ebfa74..691e20d6 100644 --- a/scripts/src/platform-infra-sub2rank.ts +++ b/scripts/src/platform-infra-sub2rank.ts @@ -1,12 +1,9 @@ import { createHash } from "node:crypto"; import type { UniDeskConfig } from "./config"; -import { startJob } from "./jobs"; import { - applyManifestWithExistingSecretScript, applyPk01CaddyBlock, capture, compactCapture, - dryRunManifestScript, parseJsonOutput, publicDnsProbe, publicHttpProbe, @@ -20,20 +17,25 @@ import { readSub2RankConfig, resolveSub2RankTarget, sub2RankConfigLabel, type Su const serviceId = "sub2rank"; +export const sub2RankImagePlaceholder = "__SUB2RANK_IMAGE_REF__"; +export const sub2RankConfigBase64Placeholder = "__SUB2RANK_CONFIG_BASE64__"; +export const sub2RankConfigSha256Placeholder = "__SUB2RANK_CONFIG_SHA256__"; +export const sub2RankSourceCommitPlaceholder = "__SUB2RANK_SOURCE_COMMIT__"; + export function sub2RankHelp(): Record { return { - command: "platform-infra sub2rank plan|apply|status|validate", + command: "platform-infra sub2rank plan|public-exposure|status|validate", output: "json", usage: [ "bun scripts/cli.ts platform-infra sub2rank plan [--target NC01]", - "bun scripts/cli.ts platform-infra sub2rank apply [--target NC01] --dry-run", - "bun scripts/cli.ts platform-infra sub2rank apply [--target NC01] --confirm", + "bun scripts/cli.ts platform-infra sub2rank public-exposure [--target NC01] --dry-run", + "bun scripts/cli.ts platform-infra sub2rank public-exposure [--target NC01] --confirm", "bun scripts/cli.ts platform-infra sub2rank status [--target NC01] [--full|--raw]", "bun scripts/cli.ts platform-infra sub2rank validate [--target NC01] [--full|--raw]", ], configTruth: sub2RankConfigLabel, applicationConfigRef: "/root/sub2rank/config/sub2rank.yaml", - artifactPolicy: "Consumes the prebuilt image declared by owning YAML; apply never runs Docker or builds from a host worktree.", + artifactPolicy: "Application PR merge is the sole delivery trigger; PaC/Tekton builds the image and publishes a digest-pinned GitOps manifest.", secretPolicy: "Resolves the existing Secret declaration from config/secrets-distribution.yaml and never reads or prints runtime Secret values.", }; } @@ -41,7 +43,7 @@ export function sub2RankHelp(): Record { export async function runSub2RankCommand(config: UniDeskConfig, args: string[]): Promise> { const [action = "plan"] = args; if (action === "plan") return plan(parseOpsCommonOptions(args.slice(1))); - if (action === "apply") return await apply(config, parseOpsApplyOptions(args.slice(1))); + if (action === "public-exposure") return await publicExposure(config, parseOpsApplyOptions(args.slice(1))); if (action === "status") return await status(config, parseOpsCommonOptions(args.slice(1))); if (action === "validate") return await validate(config, parseOpsCommonOptions(args.slice(1))); return { ok: false, error: "unsupported-platform-infra-sub2rank-command", args, help: sub2RankHelp() }; @@ -50,7 +52,7 @@ export async function runSub2RankCommand(config: UniDeskConfig, args: string[]): function plan(options: OpsCommonOptions): Record { const sub2rank = readSub2RankConfig(); const target = resolveSub2RankTarget(sub2rank, options.targetId); - const yaml = renderManifest(sub2rank, target); + const yaml = renderSub2RankManifest(sub2rank, target); const policy = policyChecks(sub2rank, target, yaml); return { ok: policy.every((item) => item.ok), @@ -59,104 +61,58 @@ function plan(options: OpsCommonOptions): Record { config: configSummary(sub2rank, target), policy, artifact: { - image: `${sub2rank.image.repository}:${sub2rank.image.tag}`, - buildDisposition: "not-performed-by-apply", + imageRepository: sub2rank.image.repository, + buildDisposition: "pac-tekton-on-application-pr-merge", sourceRoot: sub2rank.application.sourceRoot, sourceCommit: sub2rank.application.sourceCommit, sourceClean: sub2rank.application.sourceClean, sourceRemotePresent: sub2rank.application.sourceRemotePresent, - publication: sub2rank.application.sourceRemotePresent - ? { supported: false, blocker: "remote-source-not-registered-with-controlled-ci", required: "Register the repository as a YAML-owned PaC/Tekton source authority before publishing the image." } - : { supported: false, blocker: "remote-source-authority-missing", required: "Create or select an independent Git remote source authority, then register it with the YAML-owned PaC/Tekton producer." }, + sourceRemoteMatches: sub2rank.application.sourceRemoteMatches, + publication: { + supported: sub2rank.delivery.enabled, + authority: "GitHub PR merge -> Gitea snapshot -> PaC -> Tekton -> GitOps -> Argo", + branch: sub2rank.application.branch, + pipeline: sub2rank.delivery.pipeline.name, + }, }, topology: `client -> PK01 Caddy -> PK01 frps:${target.publicExposure.frpc.remotePort} -> ${target.id} frpc -> ${sub2rank.runtime.service.name}.${target.namespace}.svc.cluster.local:${sub2rank.runtime.service.port}`, next: { secrets: "bun scripts/cli.ts secrets sync --config config/secrets-distribution.yaml --scope sub2rank --confirm", - dryRun: `bun scripts/cli.ts platform-infra sub2rank apply --target ${target.id} --dry-run`, - apply: `bun scripts/cli.ts platform-infra sub2rank apply --target ${target.id} --confirm`, + publicExposureDryRun: `bun scripts/cli.ts platform-infra sub2rank public-exposure --target ${target.id} --dry-run`, + publicExposureApply: `bun scripts/cli.ts platform-infra sub2rank public-exposure --target ${target.id} --confirm`, status: `bun scripts/cli.ts platform-infra sub2rank status --target ${target.id}`, validate: `bun scripts/cli.ts platform-infra sub2rank validate --target ${target.id}`, }, }; } -async function apply(config: UniDeskConfig, options: OpsApplyOptions): Promise> { +async function publicExposure(config: UniDeskConfig, options: OpsApplyOptions): Promise> { const sub2rank = readSub2RankConfig(); const target = resolveSub2RankTarget(sub2rank, options.targetId); - const yaml = renderManifest(sub2rank, target); - const policy = policyChecks(sub2rank, target, yaml); - if (!policy.every((item) => item.ok)) { - return { ok: false, action: "platform-infra-sub2rank-apply", mode: "policy-blocked", mutation: false, target: targetSummary(target), policy }; - } - if (options.confirm && !options.wait) { - const job = startJob( - `platform_infra_sub2rank_apply_${target.id.toLowerCase()}`, - ["bun", "scripts/cli.ts", "platform-infra", "sub2rank", "apply", "--target", target.id, "--confirm", "--wait"], - `Apply ${target.id} Sub2Rank manifests and reconcile the YAML-owned PK01 Caddy site through the controlled UniDesk CLI`, - ); + if (options.dryRun) { return { ok: true, - action: "platform-infra-sub2rank-apply", - mode: "async-job", - mutation: true, - target: targetSummary(target), - job, - statusCommand: `bun scripts/cli.ts job status ${job.id} --tail-bytes 12000`, - next: { - status: `bun scripts/cli.ts job status ${job.id} --tail-bytes 12000`, - runtime: `bun scripts/cli.ts platform-infra sub2rank status --target ${target.id}`, - validate: `bun scripts/cli.ts platform-infra sub2rank validate --target ${target.id}`, - }, - }; - } - if (options.dryRun) { - const remote = await capture(config, target.route, ["sh"], dryRunManifestScript({ - yaml, - target, - fieldManager: sub2rank.runtime.rollout.fieldManager, - manifestName: serviceId, - })); - const parsed = parseJsonOutput(remote.stdout); - return { - ok: remote.exitCode === 0 && parsed?.ok === true, - action: "platform-infra-sub2rank-apply", + action: "platform-infra-sub2rank-public-exposure", mode: "dry-run", mutation: false, target: targetSummary(target), - policy, - remote: parsed ?? compactCapture(remote, { full: true }), + exposure: { + publicBaseUrl: target.publicExposure.publicBaseUrl, + upstream: `127.0.0.1:${target.publicExposure.frpc.remotePort}`, + managedBlock: serviceId, + }, + next: `bun scripts/cli.ts platform-infra sub2rank public-exposure --target ${target.id} --confirm`, }; } - const resources = runtimeResources(sub2rank, target); - const remote = await capture(config, target.route, ["sh"], applyManifestWithExistingSecretScript({ - yaml, - target, - fieldManager: sub2rank.runtime.rollout.fieldManager, - manifestName: serviceId, - secretName: resources.secretName, - requiredSecretKeys: resources.requiredSecretKeys, - deploymentNames: [resources.appDeploymentName, target.publicExposure.frpc.deploymentName], - waitTimeoutSeconds: sub2rank.runtime.rollout.waitTimeoutSeconds, - })); - const parsed = parseJsonOutput(remote.stdout); - const remoteOk = remote.exitCode === 0 && parsed?.ok === true; - const caddy = remoteOk - ? await applyPk01CaddyBlock(config, serviceId, target.publicExposure) - : { ok: false, mutation: false, disposition: "skipped-runtime-apply-failed" }; + const caddy = await applyPk01CaddyBlock(config, serviceId, target.publicExposure); return { - ok: remoteOk && caddy.ok === true, - action: "platform-infra-sub2rank-apply", + ok: caddy.ok === true, + action: "platform-infra-sub2rank-public-exposure", mode: "confirmed", mutation: true, target: targetSummary(target), - policy, - secret: secretSummary(sub2rank), - remote: parsed ?? compactCapture(remote, { full: true }), pk01Caddy: caddy, - next: { - status: `bun scripts/cli.ts platform-infra sub2rank status --target ${target.id}`, - validate: `bun scripts/cli.ts platform-infra sub2rank validate --target ${target.id}`, - }, + next: `Merge the application PR only after the PaC repository and source snapshot chain are ready.`, }; } @@ -201,10 +157,20 @@ async function validate(config: UniDeskConfig, options: OpsCommonOptions): Promi }; } -function renderManifest(config: Sub2RankConfig, target: Sub2RankTarget): string { +export interface Sub2RankManifestRenderOptions { + imageRef?: string; + appConfigBase64?: string; + appConfigSha256?: string; + sourceCommit?: string; +} + +export function renderSub2RankManifest(config: Sub2RankConfig, target: Sub2RankTarget, options: Sub2RankManifestRenderOptions = {}): string { const runtime = config.runtime; - const image = `${config.image.repository}:${config.image.tag}`; - const configHash = createHash("sha256").update(JSON.stringify({ deployment: config, target, appConfigSha256: config.application.configSha256 })).digest("hex").slice(0, 16); + const image = options.imageRef ?? `${config.image.repository}:pac-preview`; + const appConfigBase64 = options.appConfigBase64 ?? Buffer.from(config.application.configText, "utf8").toString("base64"); + const appConfigSha256 = options.appConfigSha256 ?? config.application.configSha256; + const sourceCommit = options.sourceCommit ?? config.application.sourceCommit; + const configHash = createHash("sha256").update(JSON.stringify({ deployment: deploymentHashInput(config), target })).digest("hex").slice(0, 16); const appEnv = runtime.secret.appEnvTargetKeys.map((key) => ` - name: ${key}\n valueFrom:\n secretKeyRef:\n name: ${runtime.secret.secretName}\n key: ${key}`).join("\n"); const command = runtime.command.map((value) => ` - ${yamlQuoted(value)}`).join("\n"); return `apiVersion: v1 @@ -216,9 +182,8 @@ metadata: app.kubernetes.io/name: ${runtime.service.name} app.kubernetes.io/part-of: platform-infra app.kubernetes.io/managed-by: unidesk -data: - ${runtime.configMap.key}: | -${indent(config.application.configText, 4)} +binaryData: + ${runtime.configMap.key}: ${yamlQuoted(appConfigBase64)} --- apiVersion: v1 kind: PersistentVolumeClaim @@ -276,15 +241,16 @@ spec: app.kubernetes.io/name: ${runtime.service.name} app.kubernetes.io/part-of: platform-infra annotations: - unidesk.ai/sub2rank-config-sha256: "${config.application.configSha256}" + unidesk.ai/sub2rank-config-sha256: "${appConfigSha256}" unidesk.ai/sub2rank-deployment-hash: "${configHash}" + unidesk.ai/source-commit: "${sourceCommit}" unidesk.ai/public-base-url: "${target.publicExposure.publicBaseUrl}" spec: securityContext: fsGroup: 1000 containers: - name: ${runtime.service.name} - image: ${image} + image: ${yamlQuoted(image)} imagePullPolicy: ${config.image.pullPolicy} command: ${command} @@ -357,7 +323,13 @@ function configSummary(config: Sub2RankConfig, target: Sub2RankTarget): Record { sourceCommit: config.application.sourceCommit, sourceClean: config.application.sourceClean, sourceRemotePresent: config.application.sourceRemotePresent, + sourceRemoteMatches: config.application.sourceRemoteMatches, configMatchesSourceCommit: config.application.configMatchesSourceCommit, }, secret: { path: config.runtime.secret.configRef, declarationId: config.runtime.secret.declarationId }, @@ -421,9 +394,18 @@ function runtimeResources(config: Sub2RankConfig, target: Sub2RankTarget): Publi }; } -function indent(value: string, spaces: number): string { - const prefix = " ".repeat(spaces); - return value.trimEnd().split("\n").map((line) => `${prefix}${line}`).join("\n"); +function deploymentHashInput(config: Sub2RankConfig): Record { + return { + application: { + repository: config.application.repository, + branch: config.application.branch, + sourceConfigPath: config.application.sourceConfigPath, + dockerfile: config.application.dockerfile, + runtimeTarget: config.application.runtimeTarget, + }, + image: config.image, + runtime: config.runtime, + }; } function yamlQuoted(value: string): string { From 80ca187cb3f2bcb0519ec27e13183f74df29011d Mon Sep 17 00:00:00 2001 From: Codex Date: Mon, 13 Jul 2026 07:56:37 +0200 Subject: [PATCH 07/14] =?UTF-8?q?feat:=20=E6=8E=A5=E5=85=A5=20selfmedia=20?= =?UTF-8?q?=E5=8F=97=E6=8E=A7=E4=BA=A4=E4=BB=98?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- config/platform-infra/gitea.yaml | 20 + config/platform-infra/pipelines-as-code.yaml | 76 ++++ config/secrets-distribution.yaml | 43 +++ config/selfmedia.yaml | 262 +++++++++++++ scripts/cli.ts | 9 + scripts/src/platform-infra-gitea-config.ts | 11 +- ...atform-infra-gitea-gitops-delivery.test.ts | 2 + scripts/src/platform-infra-gitea-remote.sh | 11 +- scripts/src/platform-infra-gitea.ts | 5 +- .../src/platform-infra-pac-provenance.test.ts | 30 +- ...platform-infra-pipelines-as-code-remote.sh | 193 +++++++++- ...infra-pipelines-as-code-source-artifact.ts | 46 ++- .../src/platform-infra-pipelines-as-code.ts | 98 ++++- scripts/src/secrets.ts | 202 ++++++++-- scripts/src/selfmedia-config.ts | 299 +++++++++++++++ scripts/src/selfmedia-delivery-renderer.ts | 356 ++++++++++++++++++ scripts/src/selfmedia.ts | 253 +++++++++++++ 17 files changed, 1845 insertions(+), 71 deletions(-) create mode 100644 config/selfmedia.yaml create mode 100644 scripts/src/selfmedia-config.ts create mode 100644 scripts/src/selfmedia-delivery-renderer.ts create mode 100644 scripts/src/selfmedia.ts diff --git a/config/platform-infra/gitea.yaml b/config/platform-infra/gitea.yaml index 3ecbef31..8b714a14 100644 --- a/config/platform-infra/gitea.yaml +++ b/config/platform-infra/gitea.yaml @@ -332,6 +332,26 @@ sourceAuthority: readUrl: http://git-mirror-http.devops-infra.svc.cluster.local:8080/pikasTech/HWLAB.git configRef: config/hwlab-node-lanes.yaml#lanes.v03.targets.NC01.git.readUrl disposition: replaced-by-gitea + - key: selfmedia-nc01 + targetId: NC01 + upstream: + repository: pikainc/selfmedia + cloneUrl: https://github.com/pikainc/selfmedia.git + branch: master + visibility: private + gitea: + owner: mirrors + name: pikainc-selfmedia + mirrorMode: controlled-push + publicRead: false + readUrl: http://gitea-http.devops-infra.svc.cluster.local:3000/mirrors/pikainc-selfmedia.git + gitops: + branch: nc01-selfmedia-gitops + flushDisposition: gitea-writeback + snapshot: + naming: gitea-actions-immutable-source + prefix: refs/unidesk/snapshots/gitea-actions/selfmedia-master-nc01 + legacyGitMirror: null targets: - id: JD01 diff --git a/config/platform-infra/pipelines-as-code.yaml b/config/platform-infra/pipelines-as-code.yaml index 4b53e015..f733b945 100644 --- a/config/platform-infra/pipelines-as-code.yaml +++ b/config/platform-infra/pipelines-as-code.yaml @@ -130,6 +130,41 @@ repositories: variables: NODE: NC01 LANE: v03 + - id: selfmedia-nc01 + name: selfmedia-nc01 + namespace: selfmedia-ci + providerType: gitea + url: https://gitea.pikapython.com/mirrors/pikainc-selfmedia + cloneUrl: http://gitea-http.devops-infra.svc.cluster.local:3000/mirrors/pikainc-selfmedia.git + owner: mirrors + repo: pikainc-selfmedia + secretName: pac-gitea-selfmedia-nc01 + tokenKey: token + webhookSecretKey: webhook.secret + concurrencyLimit: 1 + params: + git_read_url: http://gitea-http.devops-infra.svc.cluster.local:3000/mirrors/pikainc-selfmedia.git + source_branch: master + source_snapshot_prefix: refs/unidesk/snapshots/gitea-actions/selfmedia-master-nc01 + node: NC01 + pipeline_name: selfmedia-nc01-pac + pipeline_run_prefix: selfmedia-nc01 + service_account: selfmedia-nc01-tekton-runner + pipeline_timeout: 2h0m0s + image_repository: 127.0.0.1:5000/selfmedia/newsroom + registry_probe_base: http://127.0.0.1:5000 + gitops_read_url: http://gitea-http.devops-infra.svc.cluster.local:3000/mirrors/pikainc-selfmedia.git + gitops_write_url: http://gitea-http.devops-infra.svc.cluster.local:3000/mirrors/pikainc-selfmedia.git + gitops_branch: nc01-selfmedia-gitops + gitops_username: unidesk-admin + gitops_secret_name: pac-gitea-selfmedia-nc01 + gitops_manifest_path: deploy/gitops/nc01/resources.yaml + runtime_namespace: selfmedia + runtime_deployment: selfmedia + runtime_service: selfmedia + runtime_service_port: "4317" + health_path: /healthz + health_url: http://selfmedia.selfmedia.svc.cluster.local:4317/healthz consumers: - extends: templates.consumers.agentrunV02 variables: @@ -211,6 +246,47 @@ consumers: hostNetwork: true dnsPolicy: ClusterFirstWithHostNet fsGroup: 1000 + - id: selfmedia-nc01 + repositoryRef: selfmedia-nc01 + node: NC01 + lane: selfmedia + namespace: selfmedia-ci + pipeline: selfmedia-nc01-pac + pipelineRunPrefix: selfmedia-nc01 + argoNamespace: argocd + argoApplication: selfmedia-nc01 + closeoutGitOpsMirrorFlush: false + argoBootstrap: + project: default + repoUrl: http://gitea-http.devops-infra.svc.cluster.local:3000/mirrors/pikainc-selfmedia.git + targetRevision: nc01-selfmedia-gitops + path: deploy/gitops/nc01 + destinationNamespace: selfmedia + automated: true + repositoryCredential: + secretName: argocd-repo-selfmedia-nc01 + username: unidesk-admin + deliveryProvenance: + required: true + markerValue: admission-pac-v2:selfmedia-nc01 + executionServiceAccountName: selfmedia-nc01-tekton-runner + gitOps: + repoUrl: http://gitea-http.devops-infra.svc.cluster.local:3000/mirrors/pikainc-selfmedia.git + targetRevision: nc01-selfmedia-gitops + runnerServiceAccount: + name: selfmedia-nc01-tekton-runner + automountServiceAccountToken: false + roleBindingName: selfmedia-nc01-tekton-runner + sourceArtifact: + mode: embedded-pipeline-spec + renderer: selfmedia-runtime + configRef: config/selfmedia.yaml#delivery.targets.NC01 + pipelineRunPath: .tekton/selfmedia-nc01-pac.yaml + maxKeepRuns: 8 + taskRunTemplate: + hostNetwork: true + dnsPolicy: ClusterFirstWithHostNet + fsGroup: 1000 templates: repositories: agentrunV02: diff --git a/config/secrets-distribution.yaml b/config/secrets-distribution.yaml index 5c5c8143..763323c0 100644 --- a/config/secrets-distribution.yaml +++ b/config/secrets-distribution.yaml @@ -119,6 +119,25 @@ sources: - DATABASE_URL createIfMissing: enabled: false + externalFiles: + - sourceRef: ~/.env/TOKEN + type: raw-file + required: true + createIfMissing: + enabled: true + randomBase64Url: + bytes: 32 + prefix: smt_ + - sourceRef: ~/.codex/config.toml.pika + type: raw-file + required: true + createIfMissing: + enabled: false + - sourceRef: ~/.codex/auth.json.pika + type: raw-file + required: true + createIfMissing: + enabled: false targets: - id: platform-infra-g14 @@ -141,8 +160,32 @@ targets: namespace: hwlab-v03 scope: hwlab enabled: true + - id: selfmedia-nc01 + route: NC01:k3s + namespace: selfmedia + scope: selfmedia + enabled: true kubernetesSecrets: + - name: selfmedia-runtime + targetId: selfmedia-nc01 + secretName: selfmedia-runtime + type: Opaque + data: + - sourceRef: ~/.env/TOKEN + sourceKey: contents + targetKey: TOKEN + - name: selfmedia-codex + targetId: selfmedia-nc01 + secretName: selfmedia-codex + type: Opaque + data: + - sourceRef: ~/.codex/config.toml.pika + sourceKey: contents + targetKey: config.toml + - sourceRef: ~/.codex/auth.json.pika + sourceKey: contents + targetKey: auth.json - name: sub2rank-runtime targetId: sub2rank-nc01 secretName: sub2rank-secrets diff --git a/config/selfmedia.yaml b/config/selfmedia.yaml new file mode 100644 index 00000000..82dacec9 --- /dev/null +++ b/config/selfmedia.yaml @@ -0,0 +1,262 @@ +version: 1 +kind: selfmedia-platform-delivery +metadata: + id: selfmedia-factory + owner: unidesk + repository: pikainc/selfmedia + description: 自媒体工厂在 NC01 的构建、GitOps、运行时与一次性切换真相。 +defaults: + targetId: NC01 +delivery: + targets: + NC01: + node: NC01 + lane: selfmedia + route: NC01:k3s + ci: + namespace: selfmedia-ci + pipeline: selfmedia-nc01-pac + pipelineRunPrefix: selfmedia-nc01 + serviceAccountName: selfmedia-nc01-tekton-runner + serviceAccountAutomount: false + roleBindingName: selfmedia-nc01-tekton-runner + workspaceSize: 16Gi + pipelineTimeout: 2h0m0s + toolImage: 127.0.0.1:5000/hwlab/hwlab-ci-node-tools:node22-alpine-bun-v1 + buildkitImage: 127.0.0.1:5000/hwlab/buildkit:rootless + source: + repository: pikainc/selfmedia + worktreeRemote: https://github.com/pikainc/selfmedia.git + branch: master + readUrl: http://gitea-http.devops-infra.svc.cluster.local:3000/mirrors/pikainc-selfmedia.git + snapshotPrefix: refs/unidesk/snapshots/gitea-actions/selfmedia-master-nc01 + build: + dockerfile: deploy/Dockerfile + imageRepository: 127.0.0.1:5000/selfmedia/newsroom + networkMode: host + proxy: + http: http://127.0.0.1:10808 + https: http://127.0.0.1:10808 + all: http://127.0.0.1:10808 + noProxy: + - localhost + - 127.0.0.1 + - "::1" + - 127.0.0.1:5000 + - localhost:5000 + - .svc + - .svc.cluster.local + - .cluster.local + - hyueapi.com + - .hyueapi.com + gitops: + readUrl: http://gitea-http.devops-infra.svc.cluster.local:3000/mirrors/pikainc-selfmedia.git + writeUrl: http://gitea-http.devops-infra.svc.cluster.local:3000/mirrors/pikainc-selfmedia.git + branch: nc01-selfmedia-gitops + manifestPath: deploy/gitops/nc01/resources.yaml + releaseStatePath: deploy/gitops-state/nc01/release.json + credentialSecretName: pac-gitea-selfmedia-nc01 + credentialTokenKey: token + credentialUsername: unidesk-admin + author: + name: SelfMedia NC01 CI + email: selfmedia-nc01-ci@unidesk.local + deployment: + version: selfmedia.pikapython.com/v1 + kind: SelfMediaDeployment + metadata: + name: selfmedia-nc01 + owner: unidesk + target: + id: NC01 + route: NC01:k3s + namespace: selfmedia + publicHost: 152.53.229.148 + runtime: + replicas: 1 + serviceAccountName: selfmedia + imagePullPolicy: IfNotPresent + command: + - /usr/bin/tini + - -- + - bun + - scripts/selfmedia-cli.ts + - server + - foreground + service: + name: selfmedia + type: ClusterIP + port: 4317 + targetPort: 4317 + health: + path: /healthz + startupFailureThreshold: 30 + startupPeriodSeconds: 5 + readinessPeriodSeconds: 10 + livenessPeriodSeconds: 20 + resources: + requests: + cpu: 250m + memory: 768Mi + limits: + cpu: "4" + memory: 6Gi + securityContext: + runAsUser: 10001 + runAsGroup: 10001 + fsGroup: 10001 + runAsNonRoot: true + readOnlyRootFilesystem: true + allowPrivilegeEscalation: false + seccompProfile: RuntimeDefault + dropCapabilities: + - ALL + writablePaths: + tmp: + mountPath: /tmp + sizeLimit: 2Gi + codexHome: + mountPath: /home/codex/.codex + sizeLimit: 2Gi + cache: + mountPath: /home/codex/.cache + sizeLimit: 2Gi + toolCache: + mountPath: /app/.tools/cache + sizeLimit: 1Gi + publicExposure: + enabled: true + mode: hostPort-http-edge + address: http://152.53.229.148:4317 + hostPort: 4317 + edge: + name: selfmedia-public-edge + image: caddy:2.10.2-alpine + containerPort: 8080 + upstream: selfmedia.selfmedia.svc.cluster.local:4317 + resources: + requests: + cpu: 20m + memory: 32Mi + limits: + cpu: 200m + memory: 128Mi + storage: + media: + claimName: selfmedia-data + storageClassName: local-path + accessModes: + - ReadWriteOnce + size: 40Gi + mounts: + data: + mountPath: /app/data + subPath: data + state: + mountPath: /app/.state + subPath: state + logs: + mountPath: /app/logs + subPath: logs + codexSessions: + claimName: selfmedia-codex-sessions + storageClassName: local-path + accessModes: + - ReadWriteOnce + size: 5Gi + mounts: + sessions: + mountPath: /home/codex/.codex/sessions + subPath: sessions + state: + mountPath: /home/codex/.codex/state + subPath: state + secrets: + runtime: + sourceRef: ~/.env/TOKEN + target: + secretName: selfmedia-runtime + targetKey: TOKEN + mountPath: /home/codex/.env/TOKEN + codex: + sources: + - sourceRef: ~/.codex/config.toml.pika + targetKey: config.toml + - sourceRef: ~/.codex/auth.json.pika + targetKey: auth.json + target: + secretName: selfmedia-codex + files: + - targetKey: config.toml + mountPath: /home/codex/.codex/config.toml.pika + - targetKey: auth.json + mountPath: /home/codex/.codex/auth.json.pika + networkPolicy: + enabled: true + dnsNamespace: kube-system + dnsPort: 53 + blockedCidrs: + - 10.0.0.0/8 + - 172.16.0.0/12 + - 192.168.0.0/16 + - 169.254.169.254/32 + - 152.53.229.148/32 + codex: + package: "@openai/codex" + version: 0.144.1 + executable: codex + home: /home/codex + codexHome: /home/codex/.codex + sessionsPath: /home/codex/.codex/sessions + supervisorStatePath: /home/codex/.codex/state/selfmedia-supervisor.json + resumeArgs: + - resume + - --last + sourcePath: /app + skillsPath: /home/codex/.codex/skills +cutover: + targets: + NC01: + mode: host-process-to-kubernetes + source: + workspace: /root/selfmedia + publicAddress: http://152.53.229.148:4317 + healthUrl: http://127.0.0.1:4317/healthz + stopCommand: + - bun + - scripts/selfmedia-cli.ts + - server + - stop + startCommand: + - bun + - scripts/selfmedia-cli.ts + - server + - start + dataPaths: + - data + - .state + - logs + excludes: + - .state/server.json + destination: + route: NC01:k3s + namespace: selfmedia + application: selfmedia-nc01 + publicAddress: http://152.53.229.148:4317 + healthUrl: http://152.53.229.148:4317/healthz + dataClaim: selfmedia-data + codexClaim: selfmedia-codex-sessions + prepare: + requiresFinalConfirmation: true + phases: + - 校验源服务健康、Secret presence/fingerprint 与目标 PVC + - 在线种子复制并记录文件数和字节数 + - 停止旧服务、执行最终增量并确认宿主 4317 已释放 + - 允许 Argo 首次同步并验证公网健康 + rollback: + phases: + - 暂停 selfmedia-nc01 Argo 自动同步 + - 缩容 selfmedia 与 selfmedia-public-edge + - 确认宿主 4317 已释放 + - 启动旧服务并验证原公网地址 + dataPolicy: 回滚默认不反向覆盖旧数据，需单独的数据恢复决策。 diff --git a/scripts/cli.ts b/scripts/cli.ts index 7a120349..32c9102c 100644 --- a/scripts/cli.ts +++ b/scripts/cli.ts @@ -444,6 +444,15 @@ async function main(): Promise { return; } + if (top === "selfmedia") { + const { runSelfMediaCommand } = await import("./src/selfmedia"); + const result = await runSelfMediaCommand(args.slice(1)); + const ok = (result as { ok?: unknown }).ok !== false; + emitJson(commandName, result, ok); + if (!ok) process.exitCode = 1; + return; + } + if (top === "health") { const result = runHealthCommand(args.slice(1)); const ok = (result as { ok?: unknown }).ok !== false; diff --git a/scripts/src/platform-infra-gitea-config.ts b/scripts/src/platform-infra-gitea-config.ts index c3e6436a..dcdaebb3 100644 --- a/scripts/src/platform-infra-gitea-config.ts +++ b/scripts/src/platform-infra-gitea-config.ts @@ -277,6 +277,7 @@ export interface GiteaMirrorRepository { repository: string; cloneUrl: string; branch: string; + visibility: "public" | "private"; }; gitea: { owner: string; @@ -296,7 +297,7 @@ export interface GiteaMirrorRepository { readUrl: string; configRef: string; disposition: "replaced-by-gitea" | "retained-for-gitops-flush" | "migration-readonly"; - }; + } | null; } export function readGiteaConfig(): GiteaConfig { @@ -653,7 +654,9 @@ function parseMirrorRepository(record: Record, index: number): const gitea = y.objectField(record, "gitea", path); const gitops = y.objectField(record, "gitops", path); const snapshot = y.objectField(record, "snapshot", path); - const legacyGitMirror = y.objectField(record, "legacyGitMirror", path); + const legacyGitMirror = record.legacyGitMirror === null || record.legacyGitMirror === undefined + ? null + : y.objectField(record, "legacyGitMirror", path); return { key: y.stringField(record, "key", path), targetId: y.stringField(record, "targetId", path), @@ -661,6 +664,7 @@ function parseMirrorRepository(record: Record, index: number): repository: repositoryField(upstream, "repository", `${path}.upstream`), cloneUrl: gitCloneUrlField(upstream, "cloneUrl", `${path}.upstream`), branch: y.stringField(upstream, "branch", `${path}.upstream`), + visibility: upstream.visibility === undefined ? "public" : y.enumField(upstream, "visibility", `${path}.upstream`, ["public", "private"] as const), }, gitea: { owner: y.stringField(gitea, "owner", `${path}.gitea`), @@ -676,7 +680,7 @@ function parseMirrorRepository(record: Record, index: number): snapshot: { prefix: refPrefixField(snapshot, "prefix", `${path}.snapshot`), }, - legacyGitMirror: { + legacyGitMirror: legacyGitMirror === null ? null : { readUrl: urlField(legacyGitMirror, "readUrl", `${path}.legacyGitMirror`), configRef: y.stringField(legacyGitMirror, "configRef", `${path}.legacyGitMirror`), disposition: y.enumField(legacyGitMirror, "disposition", `${path}.legacyGitMirror`, ["replaced-by-gitea", "retained-for-gitops-flush", "migration-readonly"] as const), @@ -816,6 +820,7 @@ function validateConfig(gitea: GiteaConfig): void { const readUrl = new URL(repo.gitea.readUrl); if (readUrl.hostname !== `${gitea.app.serviceName}.${resolveTarget(gitea, repo.targetId).namespace}.svc.cluster.local`) throw new Error(`${configLabel}.sourceAuthority.repositories.${repo.key}.gitea.readUrl must use the internal k3s Gitea Service DNS`); if (readUrl.hostname === new URL(gitea.app.publicExposure.publicBaseUrl).hostname) throw new Error(`${configLabel}.sourceAuthority.repositories.${repo.key}.gitea.readUrl must not use the public Web UI hostname`); + if (repo.upstream.visibility === "private" && repo.gitea.publicRead) throw new Error(`${configLabel}.sourceAuthority.repositories.${repo.key}.gitea.publicRead must be false for a private upstream`); } } diff --git a/scripts/src/platform-infra-gitea-gitops-delivery.test.ts b/scripts/src/platform-infra-gitea-gitops-delivery.test.ts index 0d58a6bc..6f60e1b6 100644 --- a/scripts/src/platform-infra-gitea-gitops-delivery.test.ts +++ b/scripts/src/platform-infra-gitea-gitops-delivery.test.ts @@ -78,6 +78,8 @@ test("owning YAML renders one child Application and the complete durable bridge "Deployment", "Role", "RoleBinding", + "Role", + "RoleBinding", "ValidatingAdmissionPolicy", "ValidatingAdmissionPolicyBinding", "ServiceAccount", diff --git a/scripts/src/platform-infra-gitea-remote.sh b/scripts/src/platform-infra-gitea-remote.sh index b5ba9dec..075a81f6 100644 --- a/scripts/src/platform-infra-gitea-remote.sh +++ b/scripts/src/platform-infra-gitea-remote.sh @@ -515,7 +515,8 @@ for i, repo in enumerate(repos): key = repo["key"] source_branch = repo["upstream"]["branch"] gitops_branch = repo["gitops"]["branch"] - branches = [source_branch] + ([] if gitops_branch == source_branch or gitops_branch == "not-a-gitops-branch" else [gitops_branch]) + sync_gitops_from_upstream = repo["gitops"].get("flushDisposition") != "gitea-writeback" + branches = [source_branch] + ([] if not sync_gitops_from_upstream or gitops_branch == source_branch or gitops_branch == "not-a-gitops-branch" else [gitops_branch]) work = f"$tmp/repo-{i}.git" gitea_write_url = base_url + urllib.parse.urlparse(repo["gitea"]["readUrl"]).path script += [ @@ -525,13 +526,14 @@ for i, repo in enumerate(repos): f"if [ \"$fetch_rc\" -eq 0 ]; then sha=$(git -C {shlex.quote(work)} rev-parse refs/remotes/github/{shlex.quote(source_branch)} 2>$tmp/{key}.revparse.err); revparse_rc=$?; printf '%s\\n' \"$sha\" >$tmp/{key}.revparse.out; else sha=''; revparse_rc=1; : >$tmp/{key}.revparse.out; printf '%s\\n' 'fetch failed; revparse skipped' >$tmp/{key}.revparse.err; fi; printf '%s' \"$revparse_rc\" >$tmp/{key}.revparse.rc", f"if [ \"$revparse_rc\" -eq 0 ]; then snapshot_ref={shlex.quote(repo['snapshot']['prefix'])}/$sha; git -C {shlex.quote(work)} -c http.extraHeader=\"$GITEA_AUTH_HEADER\" push --force {shlex.quote(gitea_write_url)} $sha:refs/heads/{shlex.quote(source_branch)} $sha:$snapshot_ref >$tmp/{key}.push.out 2>$tmp/{key}.push.err; push_rc=$?; else snapshot_ref=''; push_rc=1; : >$tmp/{key}.push.out; printf '%s\\n' 'revparse failed; push skipped' >$tmp/{key}.push.err; fi; printf '%s' \"$push_rc\" >$tmp/{key}.push.rc", ] - if gitops_branch != source_branch and gitops_branch != "not-a-gitops-branch": + if sync_gitops_from_upstream and gitops_branch != source_branch and gitops_branch != "not-a-gitops-branch": script.append(f"if [ \"$fetch_rc\" -eq 0 ]; then gitops_sha=$(git -C {shlex.quote(work)} rev-parse refs/remotes/github/{shlex.quote(gitops_branch)} 2>/dev/null || true); else gitops_sha=''; fi") script.append(f"if [ -n \"$gitops_sha\" ] && [ \"$push_rc\" -eq 0 ]; then git -C {shlex.quote(work)} -c http.extraHeader=\"$GITEA_AUTH_HEADER\" push --force {shlex.quote(gitea_write_url)} $gitops_sha:refs/heads/{shlex.quote(gitops_branch)} >>$tmp/{key}.push.out 2>>$tmp/{key}.push.err; gitops_push_rc=$?; else gitops_push_rc=0; fi; printf '%s' \"$gitops_push_rc\" >$tmp/{key}.gitops-push.rc") else: script.append(f"printf '%s' '0' >$tmp/{key}.gitops-push.rc") script.append(f"if [ \"$push_rc\" -eq 0 ]; then printf '%s %s\\n' \"$sha\" \"$snapshot_ref\" >$tmp/{key}.bundle; fi") - meta.append({"key": key, "sourceBranch": source_branch, "gitopsBranch": gitops_branch, "readUrl": repo["gitea"]["readUrl"], "writeUrlHost": urllib.parse.urlparse(gitea_write_url).netloc, "snapshotPrefix": repo["snapshot"]["prefix"], "legacyReadUrl": repo["legacyGitMirror"]["readUrl"]}) + legacy = repo.get("legacyGitMirror") or {} + meta.append({"key": key, "sourceBranch": source_branch, "gitopsBranch": gitops_branch, "gitopsUpstreamSync": sync_gitops_from_upstream, "readUrl": repo["gitea"]["readUrl"], "writeUrlHost": urllib.parse.urlparse(gitea_write_url).netloc, "snapshotPrefix": repo["snapshot"]["prefix"], "legacyReadUrl": legacy.get("readUrl"), "legacyDisposition": legacy.get("disposition", "native-gitea-only")}) open(sys.argv[2], "w", encoding="utf-8").write("\n".join(script) + "\n") json.dump(meta, open(sys.argv[3], "w", encoding="utf-8"), ensure_ascii=False, indent=2) PY @@ -621,7 +623,8 @@ for repo in repos: gitops = repo["gitops"]["branch"] gitea_probe_url = base_url + urllib.parse.urlparse(repo["gitea"]["readUrl"]).path script.append(f"git -c http.extraHeader=\"$GITEA_AUTH_HEADER\" ls-remote {shlex.quote(gitea_probe_url)} refs/heads/{shlex.quote(branch)} '{shlex.quote(repo['snapshot']['prefix'])}/*' refs/heads/{shlex.quote(gitops)} >$tmp/{key}.ls.out 2>$tmp/{key}.ls.err") - meta.append({"key": key, "branch": branch, "gitopsBranch": gitops, "readUrl": repo["gitea"]["readUrl"], "probeUrlHost": urllib.parse.urlparse(gitea_probe_url).netloc, "snapshotPrefix": repo["snapshot"]["prefix"], "legacyReadUrl": repo["legacyGitMirror"]["readUrl"], "legacyDisposition": repo["legacyGitMirror"]["disposition"]}) + legacy = repo.get("legacyGitMirror") or {} + meta.append({"key": key, "branch": branch, "gitopsBranch": gitops, "readUrl": repo["gitea"]["readUrl"], "probeUrlHost": urllib.parse.urlparse(gitea_probe_url).netloc, "snapshotPrefix": repo["snapshot"]["prefix"], "legacyReadUrl": legacy.get("readUrl"), "legacyDisposition": legacy.get("disposition", "native-gitea-only")}) open(sys.argv[2], "w", encoding="utf-8").write("\n".join(script) + "\n") json.dump(meta, open(sys.argv[3], "w", encoding="utf-8"), ensure_ascii=False) PY diff --git a/scripts/src/platform-infra-gitea.ts b/scripts/src/platform-infra-gitea.ts index ab5468ca..20b7c759 100644 --- a/scripts/src/platform-infra-gitea.ts +++ b/scripts/src/platform-infra-gitea.ts @@ -1483,13 +1483,14 @@ function repositorySummary(gitea: GiteaConfig, repo: GiteaMirrorRepository): Rec key: repo.key, upstreamRepository: repo.upstream.repository, upstreamBranch: repo.upstream.branch, + upstreamVisibility: repo.upstream.visibility, giteaOwner: repo.gitea.owner, giteaRepo: repo.gitea.name, mirrorMode: repo.gitea.mirrorMode, readUrl: repo.gitea.readUrl, snapshotPrefix: repo.snapshot.prefix, - legacyReadUrl: repo.legacyGitMirror.readUrl, - legacyDisposition: repo.legacyGitMirror.disposition, + legacyReadUrl: repo.legacyGitMirror?.readUrl ?? null, + legacyDisposition: repo.legacyGitMirror?.disposition ?? "native-gitea-only", sourceAuthority: "gitea", rootUrlMatches: repo.gitea.readUrl.startsWith(gitea.app.server.rootUrl), }; diff --git a/scripts/src/platform-infra-pac-provenance.test.ts b/scripts/src/platform-infra-pac-provenance.test.ts index e85d7ab4..7159f25c 100644 --- a/scripts/src/platform-infra-pac-provenance.test.ts +++ b/scripts/src/platform-infra-pac-provenance.test.ts @@ -59,7 +59,7 @@ test("versioned admission desired state protects controller proof, candidate ide toState: "started", }]); expect(contract.child.enabled).toBe(false); - expect(contract.consumers.map((item) => item.id)).toEqual(["agentrun-nc01-v02"]); + expect(contract.consumers.map((item) => item.id)).toEqual(["agentrun-nc01-v02", "selfmedia-nc01"]); const items = documents(renderPacAdmissionDesiredFragment("NC01")); expect(items.map((item) => item.kind)).toEqual(["ValidatingAdmissionPolicy", "ValidatingAdmissionPolicyBinding"]); const policy = items[0]; @@ -79,6 +79,7 @@ test("versioned admission desired state protects controller proof, candidate ide expect(expressions).toContain("object.metadata.name.startsWith"); expect(expressions).toContain('object.metadata.labels["tekton.dev/pipeline"]'); expect(expressions).toContain("agentrun-nc01-v02-tekton-runner"); + expect(expressions).toContain("selfmedia-nc01-tekton-runner"); expect(messages).toContain("execution ServiceAccount"); expect(expressions).toContain(contract.child.parentUidAnnotation); expect(expressions).toContain("oldObject"); @@ -117,16 +118,20 @@ test("versioned admission desired state protects controller proof, candidate ide expect(queueGuard.expression).not.toContain('== "Cancelled"'); }); -test("required consumer default RBAC has one automatic desired owner and no Tekton mutation verbs", () => { +test("required consumer default RBAC has one automatic desired owner per namespace and no Tekton mutation verbs", () => { const rbac = documents(renderPacConsumerRbacDesiredFragment("NC01")); - expect(rbac.map((item) => item.kind)).toEqual(["Role", "RoleBinding"]); - const role = rbac[0]; - expect(role.metadata.annotations["argocd.argoproj.io/sync-wave"]).toBe("-1"); - expect(role.metadata.annotations["unidesk.ai/effective-config-sha256"]).toBe(pacConsumerRbacDesiredIdentity("NC01").configSha256); - const verbs = role.rules.flatMap((rule: any) => rule.verbs); - for (const verb of ["create", "patch", "update", "delete", "deletecollection"]) expect(verbs).not.toContain(verb); + expect(rbac.map((item) => item.kind)).toEqual(["Role", "RoleBinding", "Role", "RoleBinding"]); + expect(rbac.filter((item) => item.kind === "Role").map((item) => item.metadata.namespace)).toEqual(["agentrun-ci", "selfmedia-ci"]); + for (const role of rbac.filter((item) => item.kind === "Role")) { + expect(role.metadata.annotations["argocd.argoproj.io/sync-wave"]).toBe("-1"); + expect(role.metadata.annotations["unidesk.ai/effective-config-sha256"]).toBe(pacConsumerRbacDesiredIdentity("NC01").configSha256); + const verbs = role.rules.flatMap((rule: any) => rule.verbs); + for (const verb of ["create", "patch", "update", "delete", "deletecollection"]) expect(verbs).not.toContain(verb); + } const desiredKinds = documents(renderPlatformInfraGiteaDesiredFragments("NC01")).map((item) => item.kind); expect(desiredKinds).toEqual([ + "Role", + "RoleBinding", "Role", "RoleBinding", "ValidatingAdmissionPolicy", @@ -348,6 +353,15 @@ test("history evaluates admission and default RBAC in each consumer namespace", expect(remote).toContain("'delivery plan source commit does not match the selected PipelineRun'"); expect(remote).not.toContain("'g14-ci-plan structured evidence is missing'"); expect(remote).not.toContain("for (const consumer of consumers) consumer.admissionState = admissionState"); + expect(remote).toContain("consumer_bootstrap_state()"); + expect(remote).toContain('get serviceaccount "$UNIDESK_PAC_RUNNER_SERVICE_ACCOUNT_NAME"'); + expect(remote).toContain('get rolebinding "$UNIDESK_PAC_RUNNER_ROLE_BINDING_NAME"'); + expect(remote).toContain('get secret "$UNIDESK_PAC_ARGO_REPOSITORY_SECRET_NAME"'); + expect(remote).toContain("sa.automountServiceAccountToken === false"); + expect(remote).toContain("const expectedKeys = ['password', 'url', 'username']"); + expect(remote).toContain("passwordDecoded: false"); + expect(remote).not.toContain("Buffer.from(argoSecret.data.password"); + expect(remote).not.toContain("--from-literal=password="); }); test("id-specific PaC debug keeps fixture failures but does not dump every passing check", () => { diff --git a/scripts/src/platform-infra-pipelines-as-code-remote.sh b/scripts/src/platform-infra-pipelines-as-code-remote.sh index 30d88de2..d9818316 100644 --- a/scripts/src/platform-infra-pipelines-as-code-remote.sh +++ b/scripts/src/platform-infra-pipelines-as-code-remote.sh @@ -202,6 +202,50 @@ roleRef: EOF } +pac_repository_secret_manifest() { + node -e ' +const fs = require("node:fs"); +const input = fs.readFileSync(0); +const split = input.indexOf(0); +if (split < 0) process.exit(2); +const token = input.subarray(0, split); +const webhook = input.subarray(split + 1); +const data = {}; +data[process.env.UNIDESK_PAC_TOKEN_KEY] = token.toString("base64"); +data[process.env.UNIDESK_PAC_WEBHOOK_SECRET_KEY] = webhook.toString("base64"); +process.stdout.write(JSON.stringify({ + apiVersion: "v1", + kind: "Secret", + metadata: { name: process.env.UNIDESK_PAC_SECRET_NAME, namespace: process.env.UNIDESK_PAC_TARGET_NAMESPACE }, + type: "Opaque", + data, +})); +' +} + +argo_repository_secret_manifest() { + node -e ' +const fs = require("node:fs"); +const token = fs.readFileSync(0); +const encoded = (value) => Buffer.from(value, "utf8").toString("base64"); +process.stdout.write(JSON.stringify({ + apiVersion: "v1", + kind: "Secret", + metadata: { + name: process.env.UNIDESK_PAC_ARGO_REPOSITORY_SECRET_NAME, + namespace: process.env.UNIDESK_PAC_ARGO_NAMESPACE, + labels: { "argocd.argoproj.io/secret-type": "repository" }, + }, + type: "Opaque", + data: { + url: encoded(process.env.UNIDESK_PAC_ARGO_REPOSITORY_URL || ""), + username: encoded(process.env.UNIDESK_PAC_ARGO_REPOSITORY_USERNAME || ""), + password: token.toString("base64"), + }, +})); +' +} + apply_release() { tmp=$(mktemp) printf '%s' "$UNIDESK_PAC_RELEASE_MANIFEST_B64" | base64 -d > "$tmp" @@ -238,15 +282,25 @@ apply_action() { else consumer_rbac_manifest | kubectl apply --server-side --force-conflicts --field-manager="$UNIDESK_PAC_FIELD_MANAGER" -f - >/dev/null fi + if [ -n "${UNIDESK_PAC_RUNNER_SERVICE_ACCOUNT_MANIFEST_B64:-}" ]; then + runner_tmp=$(mktemp) + printf '%s' "$UNIDESK_PAC_RUNNER_SERVICE_ACCOUNT_MANIFEST_B64" | base64 -d >"$runner_tmp" + kubectl apply --server-side --force-conflicts --field-manager="$UNIDESK_PAC_FIELD_MANAGER-runner-bootstrap" -f "$runner_tmp" >/dev/null + rm -f "$runner_tmp" + fi token=$(ensure_token) test -n "$token" - kubectl -n "$UNIDESK_PAC_TARGET_NAMESPACE" create secret generic "$UNIDESK_PAC_SECRET_NAME" \ - --from-literal="$UNIDESK_PAC_TOKEN_KEY=$token" \ - --from-literal="$UNIDESK_PAC_WEBHOOK_SECRET_KEY=$UNIDESK_PAC_WEBHOOK_SECRET" \ - --dry-run=client -o yaml | kubectl apply -f - >/dev/null + { printf '%s' "$token"; printf '\0'; printf '%s' "$UNIDESK_PAC_WEBHOOK_SECRET"; } \ + | pac_repository_secret_manifest \ + | kubectl apply --server-side --force-conflicts --field-manager="$UNIDESK_PAC_FIELD_MANAGER-repository-secret" -f - >/dev/null repository_manifest | kubectl apply --server-side --force-conflicts --field-manager="$UNIDESK_PAC_FIELD_MANAGER" -f - >/dev/null argo_bootstrap=false if [ -n "${UNIDESK_PAC_ARGO_BOOTSTRAP_MANIFEST_B64:-}" ]; then + if [ -n "${UNIDESK_PAC_ARGO_REPOSITORY_SECRET_NAME:-}" ]; then + printf '%s' "$token" \ + | argo_repository_secret_manifest \ + | kubectl apply --server-side --force-conflicts --field-manager="$UNIDESK_PAC_FIELD_MANAGER-argocd-repository" -f - >/dev/null + fi argo_tmp=$(mktemp) printf '%s' "$UNIDESK_PAC_ARGO_BOOTSTRAP_MANIFEST_B64" | base64 -d >"$argo_tmp" kubectl apply --server-side --force-conflicts --field-manager="$UNIDESK_PAC_FIELD_MANAGER" -f "$argo_tmp" >/dev/null @@ -1287,11 +1341,121 @@ process.stdout.write(JSON.stringify(rows)); NODE } +consumer_bootstrap_state() { + if [ -z "${UNIDESK_PAC_RUNNER_SERVICE_ACCOUNT_NAME:-}" ] && [ -z "${UNIDESK_PAC_ARGO_REPOSITORY_SECRET_NAME:-}" ]; then + printf '{"configured":false,"ready":null,"reasons":["consumer-bootstrap-not-declared"],"valuesPrinted":false}' + return + fi + sa_file=$(mktemp) + role_binding_file=$(mktemp) + argo_secret_file=$(mktemp) + if [ -n "${UNIDESK_PAC_RUNNER_SERVICE_ACCOUNT_NAME:-}" ]; then + kubectl -n "$UNIDESK_PAC_TARGET_NAMESPACE" get serviceaccount "$UNIDESK_PAC_RUNNER_SERVICE_ACCOUNT_NAME" -o json >"$sa_file" 2>/dev/null || printf '{}' >"$sa_file" + kubectl -n "$UNIDESK_PAC_TARGET_NAMESPACE" get rolebinding "$UNIDESK_PAC_RUNNER_ROLE_BINDING_NAME" -o json >"$role_binding_file" 2>/dev/null || printf '{}' >"$role_binding_file" + else + printf '{}' >"$sa_file" + printf '{}' >"$role_binding_file" + fi + if [ -n "${UNIDESK_PAC_ARGO_REPOSITORY_SECRET_NAME:-}" ]; then + kubectl -n "$UNIDESK_PAC_ARGO_NAMESPACE" get secret "$UNIDESK_PAC_ARGO_REPOSITORY_SECRET_NAME" -o json >"$argo_secret_file" 2>/dev/null || printf '{}' >"$argo_secret_file" + else + printf '{}' >"$argo_secret_file" + fi + node - "$sa_file" "$role_binding_file" "$argo_secret_file" <<'NODE' +const crypto = require('node:crypto'); +const fs = require('node:fs'); +function read(path) { + try { return JSON.parse(fs.readFileSync(path, 'utf8') || '{}'); } catch { return {}; } +} +function fingerprint(value) { + return `sha256:${crypto.createHash('sha256').update(value).digest('hex')}`; +} +const sa = read(process.argv[2]); +const roleBinding = read(process.argv[3]); +const argoSecret = read(process.argv[4]); +const namespace = process.env.UNIDESK_PAC_TARGET_NAMESPACE || ''; +const runnerName = process.env.UNIDESK_PAC_RUNNER_SERVICE_ACCOUNT_NAME || ''; +const roleBindingName = process.env.UNIDESK_PAC_RUNNER_ROLE_BINDING_NAME || ''; +const argoNamespace = process.env.UNIDESK_PAC_ARGO_NAMESPACE || ''; +const argoSecretName = process.env.UNIDESK_PAC_ARGO_REPOSITORY_SECRET_NAME || ''; +const argoUrl = process.env.UNIDESK_PAC_ARGO_REPOSITORY_URL || ''; +const reasons = []; +const runnerConfigured = runnerName.length > 0; +const saExists = !runnerConfigured || sa.metadata?.name === runnerName; +const automountDisabled = !runnerConfigured || sa.automountServiceAccountToken === false; +const subjects = Array.isArray(roleBinding.subjects) ? roleBinding.subjects : []; +const roleBindingExact = !runnerConfigured || ( + roleBinding.metadata?.name === roleBindingName + && roleBinding.metadata?.namespace === namespace + && subjects.length === 1 + && subjects[0]?.kind === 'ServiceAccount' + && subjects[0]?.name === runnerName + && subjects[0]?.namespace === namespace + && roleBinding.roleRef?.apiGroup === 'rbac.authorization.k8s.io' + && roleBinding.roleRef?.kind === 'Role' + && roleBinding.roleRef?.name === 'unidesk-pac-consumer-runner' +); +if (!saExists) reasons.push('runner-service-account-missing'); +if (!automountDisabled) reasons.push('runner-service-account-automount-not-false'); +if (!roleBindingExact) reasons.push('runner-rolebinding-drifted-or-missing'); +const argoConfigured = argoSecretName.length > 0; +const argoExists = !argoConfigured || (argoSecret.metadata?.name === argoSecretName && argoSecret.metadata?.namespace === argoNamespace); +const argoLabelReady = !argoConfigured || argoSecret.metadata?.labels?.['argocd.argoproj.io/secret-type'] === 'repository'; +const keys = Object.keys(argoSecret.data || {}).sort(); +const expectedKeys = ['password', 'url', 'username']; +const keysReady = !argoConfigured || JSON.stringify(keys) === JSON.stringify(expectedKeys); +let observedUrlFingerprint = null; +if (argoConfigured && typeof argoSecret.data?.url === 'string') { + try { observedUrlFingerprint = fingerprint(Buffer.from(argoSecret.data.url, 'base64').toString('utf8')); } catch {} +} +const expectedUrlFingerprint = argoConfigured ? fingerprint(argoUrl) : null; +const urlReady = !argoConfigured || observedUrlFingerprint === expectedUrlFingerprint; +const passwordPresent = !argoConfigured || (typeof argoSecret.data?.password === 'string' && argoSecret.data.password.length > 0); +if (!argoExists) reasons.push('argocd-repository-secret-missing'); +if (!argoLabelReady) reasons.push('argocd-repository-secret-label-drifted'); +if (!keysReady) reasons.push('argocd-repository-secret-keys-drifted'); +if (!urlReady) reasons.push('argocd-repository-secret-url-drifted'); +if (!passwordPresent) reasons.push('argocd-repository-secret-password-missing'); +process.stdout.write(JSON.stringify({ + configured: runnerConfigured || argoConfigured, + ready: reasons.length === 0, + runner: { + configured: runnerConfigured, + serviceAccount: runnerName || null, + exists: saExists, + automountServiceAccountToken: sa.metadata?.name === runnerName ? sa.automountServiceAccountToken ?? null : null, + automountDisabled, + roleBinding: roleBindingName || null, + roleBindingExact, + }, + argoRepository: { + configured: argoConfigured, + secretName: argoSecretName || null, + exists: argoExists, + labelReady: argoLabelReady, + expectedKeys, + observedKeys: keys, + keysReady, + expectedUrlFingerprint, + observedUrlFingerprint, + urlReady, + passwordPresent, + passwordDecoded: false, + }, + reasons, + valuesPrinted: false, +})); +NODE + rm -f "$sa_file" "$role_binding_file" "$argo_secret_file" +} + status_action() { crd=$(kubectl get crd repositories.pipelinesascode.tekton.dev -o name 2>/dev/null || true) controller_ready=$(kubectl -n "$UNIDESK_PAC_RELEASE_NAMESPACE" get deploy "$UNIDESK_PAC_CONTROLLER_SERVICE_NAME" -o jsonpath='{.status.readyReplicas}/{.spec.replicas}' 2>/dev/null || echo "0/0") repository_condition=$(condition_status "$UNIDESK_PAC_TARGET_NAMESPACE" repository "$UNIDESK_PAC_REPOSITORY_NAME") prepare_admission_provenance_state + consumer_bootstrap=$(consumer_bootstrap_state) + bootstrap_ready=$(UNIDESK_PAC_BOOTSTRAP_STATE="$consumer_bootstrap" node -e 'const s=JSON.parse(process.env.UNIDESK_PAC_BOOTSTRAP_STATE||"{}"); process.stdout.write(s.configured===true&&s.ready!==true?"false":"true")') pipelines=$(pipeline_rows) latest=$(printf '%s' "$pipelines" | node -e 'const fs=require("fs"); const a=JSON.parse(fs.readFileSync(0,"utf8")||"[]"); process.stdout.write(a[0]?.name||"")') export UNIDESK_PAC_TARGET_PIPELINERUN="$latest" @@ -1316,9 +1480,10 @@ NODE runtime=$(json_normalize "$(runtime_summary)") diagnostics=$(cicd_diagnostics "$pipelines" "$artifact" "$argo" "$runtime") admission_ready=$(UNIDESK_PAC_STATE="$UNIDESK_PAC_ADMISSION_STATE_JSON" node -e 'const s=JSON.parse(process.env.UNIDESK_PAC_STATE||"{}"); process.stdout.write(s.ready===true?"true":"false")') - diagnostics=$(UNIDESK_PAC_DIAGNOSTICS="$diagnostics" UNIDESK_PAC_STATE="$UNIDESK_PAC_ADMISSION_STATE_JSON" node <<'NODE' + diagnostics=$(UNIDESK_PAC_DIAGNOSTICS="$diagnostics" UNIDESK_PAC_STATE="$UNIDESK_PAC_ADMISSION_STATE_JSON" UNIDESK_PAC_BOOTSTRAP_STATE="$consumer_bootstrap" node <<'NODE' const diagnostics = JSON.parse(process.env.UNIDESK_PAC_DIAGNOSTICS || '{}'); const admissionProvenance = JSON.parse(process.env.UNIDESK_PAC_STATE || '{}'); +const consumerBootstrap = JSON.parse(process.env.UNIDESK_PAC_BOOTSTRAP_STATE || '{}'); if (process.env.UNIDESK_PAC_DELIVERY_PROVENANCE_REQUIRED === '1' && admissionProvenance.ready !== true) { process.stdout.write(JSON.stringify({ ...diagnostics, @@ -1329,16 +1494,28 @@ if (process.env.UNIDESK_PAC_DELIVERY_PROVENANCE_REQUIRED === '1' && admissionPro admissionProvenance, valuesPrinted: false, })); +} else if (consumerBootstrap.configured === true && consumerBootstrap.ready !== true) { + process.stdout.write(JSON.stringify({ + ...diagnostics, + ok: false, + code: 'pac-consumer-bootstrap-not-ready', + phase: 'consumer-bootstrap-not-ready', + hint: 'declared runner ServiceAccount/RoleBinding or Argo repository credential is missing or drifted', + admissionProvenance, + consumerBootstrap, + valuesPrinted: false, + })); } else { - process.stdout.write(JSON.stringify({ ...diagnostics, admissionProvenance, valuesPrinted: false })); + process.stdout.write(JSON.stringify({ ...diagnostics, admissionProvenance, consumerBootstrap, valuesPrinted: false })); } NODE ) - printf '{"ok":%s,"crdPresent":%s,"controllerReady":"%s","admissionProvenance":%s,"repository":{"name":"%s","repo":"%s/%s","url":"%s","condition":"%s"},"repositoryCondition":"%s","webhooks":%s,"pipelineRuns":%s,"taskRuns":%s,"artifact":%s,"argo":%s,"runtime":%s,"diagnostics":%s,"valuesPrinted":false}\n' \ - "$( [ -n "$crd" ] && [ "$controller_ready" != "0/0" ] && { [ "${UNIDESK_PAC_DELIVERY_PROVENANCE_REQUIRED:-0}" != "1" ] || [ "$admission_ready" = "true" ]; } && echo true || echo false )" \ + printf '{"ok":%s,"crdPresent":%s,"controllerReady":"%s","admissionProvenance":%s,"consumerBootstrap":%s,"repository":{"name":"%s","repo":"%s/%s","url":"%s","condition":"%s"},"repositoryCondition":"%s","webhooks":%s,"pipelineRuns":%s,"taskRuns":%s,"artifact":%s,"argo":%s,"runtime":%s,"diagnostics":%s,"valuesPrinted":false}\n' \ + "$( [ -n "$crd" ] && [ "$controller_ready" != "0/0" ] && [ "$bootstrap_ready" = "true" ] && { [ "${UNIDESK_PAC_DELIVERY_PROVENANCE_REQUIRED:-0}" != "1" ] || [ "$admission_ready" = "true" ]; } && echo true || echo false )" \ "$( [ -n "$crd" ] && echo true || echo false )" \ "$(json_string "$controller_ready")" \ "$UNIDESK_PAC_ADMISSION_STATE_JSON" \ + "$consumer_bootstrap" \ "$(json_string "$UNIDESK_PAC_REPOSITORY_NAME")" \ "$(json_string "$UNIDESK_PAC_GITEA_OWNER")" \ "$(json_string "$UNIDESK_PAC_GITEA_REPO")" \ diff --git a/scripts/src/platform-infra-pipelines-as-code-source-artifact.ts b/scripts/src/platform-infra-pipelines-as-code-source-artifact.ts index 344a1ea9..a72e8764 100644 --- a/scripts/src/platform-infra-pipelines-as-code-source-artifact.ts +++ b/scripts/src/platform-infra-pipelines-as-code-source-artifact.ts @@ -17,9 +17,10 @@ import type { RenderedCliResult } from "./output"; import { renderedCliResult } from "./agentrun/render"; import { stableJsonSha256, stableJsonValue } from "./stable-json"; import { pipelineProvenanceAnnotations, pipelineProvenanceFromManifest, withPipelineProvenanceAnnotations } from "./pipeline-provenance"; +import { renderSelfMediaDesiredPipeline, selfMediaSourceWorktreeRemote } from "./selfmedia-delivery-renderer"; export type PacSourceArtifactMode = "embedded-pipeline-spec" | "remote-pipeline-annotation"; -export type PacSourceArtifactRenderer = "agentrun-control-plane" | "hwlab-runtime-lane"; +export type PacSourceArtifactRenderer = "agentrun-control-plane" | "hwlab-runtime-lane" | "selfmedia-runtime"; export type PacSourceArtifactAction = "plan" | "check" | "write" | "status" | "verify-runtime"; export type PacSourceArtifactRuntimeAlignment = "not-requested" | "aligned" | "drifted" | "missing" | "unavailable"; @@ -441,7 +442,9 @@ function renderDesiredArtifact(binding: PacSourceArtifactBinding, sourceWorktree const sourceArtifact = binding.consumer.sourceArtifact; const rendered = sourceArtifact.renderer === "agentrun-control-plane" ? renderAgentRunDesiredPipeline(binding) - : renderHwlabDesiredPipeline(binding, sourceWorktree); + : sourceArtifact.renderer === "hwlab-runtime-lane" + ? renderHwlabDesiredPipeline(binding, sourceWorktree) + : renderSelfMediaPipeline(binding, sourceWorktree); const pipeline = sourceArtifact.renderer === "hwlab-runtime-lane" ? rendered.pipeline : withPipelineProvenanceAnnotations(rendered.pipeline, rendered.provenance); @@ -512,6 +515,19 @@ function renderHwlabDesiredPipeline(binding: PacSourceArtifactBinding, sourceWor }; } +function renderSelfMediaPipeline(binding: PacSourceArtifactBinding, sourceWorktree: string): { pipeline: Record; provenance: PacSourceArtifactProvenance } { + const rendered = renderSelfMediaDesiredPipeline(binding, sourceWorktree); + return { + pipeline: rendered.pipeline, + provenance: { + configRef: rendered.configRef, + effectiveConfigSha256: rendered.effectiveConfigSha256, + renderer: "selfmedia-runtime", + mode: "embedded-pipeline-spec", + }, + }; +} + function renderHwlabPipelineFromOwningYaml(spec: HwlabRuntimeLaneSpec, sourceWorktree: string): Record { const temporaryRoot = mkdtempSync(resolve(tmpdir(), "unidesk-pac-source-artifact-")); const temporarySource = resolve(temporaryRoot, "source"); @@ -581,9 +597,10 @@ function embeddedPipelineRun(binding: PacSourceArtifactBinding, desiredSpec: Rec name: `${binding.consumer.pipelineRunPrefix}-{{ revision }}`, namespace: binding.consumer.namespace, annotations: pipelineRunAnnotations(binding, provenance), - labels: pipelineRunLabels(binding, "agentrun"), + labels: pipelineRunLabels(binding, provenance.renderer === "selfmedia-runtime" ? "selfmedia" : "agentrun"), }, spec: { + ...(binding.repository.params.pipeline_timeout === undefined ? {} : { timeouts: { pipeline: binding.repository.params.pipeline_timeout } }), pipelineSpec: desiredSpec, taskRunTemplate: taskRunTemplate(binding), params: pipelineRunParams(binding, desiredSpec), @@ -647,16 +664,26 @@ function pipelineRunAnnotations(binding: PacSourceArtifactBinding, provenance: P }; } -function pipelineRunLabels(binding: PacSourceArtifactBinding, partOf: "agentrun" | "hwlab"): Record { - return partOf === "agentrun" - ? { +function pipelineRunLabels(binding: PacSourceArtifactBinding, partOf: "agentrun" | "hwlab" | "selfmedia"): Record { + if (partOf === "agentrun") { + return { "app.kubernetes.io/part-of": "agentrun", "agentrun.pikastech.local/lane": "v0.2", "agentrun.pikastech.local/node": binding.consumer.node, "agentrun.pikastech.local/source-commit": "{{ revision }}", "agentrun.pikastech.local/trigger": "pipelines-as-code", - } - : { + }; + } + if (partOf === "selfmedia") { + return { + "app.kubernetes.io/name": `${binding.consumer.id}-pac`, + "app.kubernetes.io/part-of": "selfmedia-factory", + "unidesk.ai/source-commit": "{{ revision }}", + "selfmedia.pikapython.com/source-commit": "{{ revision }}", + "selfmedia.pikapython.com/trigger": "pipelines-as-code", + }; + } + return { "app.kubernetes.io/name": `${binding.consumer.id}-pac`, "app.kubernetes.io/part-of": "hwlab", "unidesk.ai/source-commit": "{{ revision }}", @@ -808,6 +835,7 @@ function owningSourceRemote(binding: PacSourceArtifactBinding): string { if (binding.consumer.sourceArtifact.renderer === "agentrun-control-plane") { return resolveAgentRunLaneTarget({ node: binding.consumer.node, lane: binding.consumer.lane }).spec.source.worktreeRemote; } + if (binding.consumer.sourceArtifact.renderer === "selfmedia-runtime") return selfMediaSourceWorktreeRemote(binding.consumer.node); if (!isHwlabRuntimeLane(binding.consumer.lane)) throw new PacSourceArtifactError("owning-source-lane-unresolved"); return hwlabRuntimeLaneSpecForNode(binding.consumer.lane, binding.consumer.node).gitUrl; } @@ -923,7 +951,7 @@ function assertPipelineIdentity(pipeline: Record, binding: PacS function provenanceFromManifest(manifest: Record): PacSourceArtifactProvenance | null { const provenance = pipelineProvenanceFromManifest(manifest); if (provenance === null) return null; - if (provenance.renderer !== "agentrun-control-plane" && provenance.renderer !== "hwlab-runtime-lane") return null; + if (provenance.renderer !== "agentrun-control-plane" && provenance.renderer !== "hwlab-runtime-lane" && provenance.renderer !== "selfmedia-runtime") return null; if (provenance.mode !== "embedded-pipeline-spec" && provenance.mode !== "remote-pipeline-annotation") return null; return provenance as PacSourceArtifactProvenance; } diff --git a/scripts/src/platform-infra-pipelines-as-code.ts b/scripts/src/platform-infra-pipelines-as-code.ts index eb5b7d99..f7429ce8 100644 --- a/scripts/src/platform-infra-pipelines-as-code.ts +++ b/scripts/src/platform-infra-pipelines-as-code.ts @@ -154,6 +154,10 @@ interface PacConsumer { path: string; destinationNamespace: string; automated: boolean; + repositoryCredential: { + secretName: string; + username: string; + } | null; } | null; deliveryProvenance: { required: true; @@ -165,6 +169,11 @@ interface PacConsumer { closeoutGitOpsMirrorFlush: boolean; closeoutGitOpsMirrorLane: "v02" | "v03" | null; sourceArtifact: PacSourceArtifactSpec | null; + runnerServiceAccount: { + name: string; + automountServiceAccountToken: false; + roleBindingName: string; + } | null; } interface CommonOptions { @@ -495,6 +504,7 @@ function parseConsumer(consumer: Record, path: string, defaultR const id = typeof consumer.id === "string" && consumer.id.length > 0 ? consumer.id : `${y.stringField(consumer, "node", path)}-${y.stringField(consumer, "lane", path)}`; const argoBootstrap = consumer.argoBootstrap === undefined ? null : y.objectField(consumer, "argoBootstrap", path); const deliveryProvenance = consumer.deliveryProvenance === undefined ? null : parseConsumerDeliveryProvenance(y.objectField(consumer, "deliveryProvenance", path), `${path}.deliveryProvenance`); + const runnerServiceAccount = consumer.runnerServiceAccount === undefined ? null : y.objectField(consumer, "runnerServiceAccount", path); return { id, node: y.stringField(consumer, "node", path), @@ -511,15 +521,32 @@ function parseConsumer(consumer: Record, path: string, defaultR path: y.stringField(argoBootstrap, "path", `${path}.argoBootstrap`), destinationNamespace: y.kubernetesNameField(argoBootstrap, "destinationNamespace", `${path}.argoBootstrap`), automated: y.booleanField(argoBootstrap, "automated", `${path}.argoBootstrap`), + repositoryCredential: argoBootstrap.repositoryCredential === undefined ? null : (() => { + const credential = y.objectField(argoBootstrap, "repositoryCredential", `${path}.argoBootstrap`); + return { + secretName: y.kubernetesNameField(credential, "secretName", `${path}.argoBootstrap.repositoryCredential`), + username: y.stringField(credential, "username", `${path}.argoBootstrap.repositoryCredential`), + }; + })(), }, deliveryProvenance, repositoryRef: typeof consumer.repositoryRef === "string" && consumer.repositoryRef.length > 0 ? consumer.repositoryRef : defaultRepositoryRef, closeoutGitOpsMirrorFlush: y.booleanField(consumer, "closeoutGitOpsMirrorFlush", path), closeoutGitOpsMirrorLane: consumer.closeoutGitOpsMirrorLane === undefined ? null : y.enumField(consumer, "closeoutGitOpsMirrorLane", path, ["v02", "v03"] as const), sourceArtifact: consumer.sourceArtifact === undefined ? null : parseSourceArtifact(y.objectField(consumer, "sourceArtifact", path), `${path}.sourceArtifact`), + runnerServiceAccount: runnerServiceAccount === null ? null : { + name: y.kubernetesNameField(runnerServiceAccount, "name", `${path}.runnerServiceAccount`), + automountServiceAccountToken: parseFalse(runnerServiceAccount, "automountServiceAccountToken", `${path}.runnerServiceAccount`), + roleBindingName: y.kubernetesNameField(runnerServiceAccount, "roleBindingName", `${path}.runnerServiceAccount`), + }, }; } +function parseFalse(record: Record, key: string, path: string): false { + if (y.booleanField(record, key, path) !== false) throw new Error(`${path}.${key} must be false`); + return false; +} + function parseConsumerDeliveryProvenance(value: Record, path: string): PacConsumer["deliveryProvenance"] { const required = y.booleanField(value, "required", path); if (!required) return null; @@ -537,7 +564,7 @@ function parseConsumerDeliveryProvenance(value: Record, path: s function parseSourceArtifact(value: Record, path: string): PacSourceArtifactSpec { const mode = y.enumField(value, "mode", path, ["embedded-pipeline-spec", "remote-pipeline-annotation"] as const); - const renderer = y.enumField(value, "renderer", path, ["agentrun-control-plane", "hwlab-runtime-lane"] as const); + const renderer = y.enumField(value, "renderer", path, ["agentrun-control-plane", "hwlab-runtime-lane", "selfmedia-runtime"] as const); const pipelineRunPath = sourceArtifactPath(y.stringField(value, "pipelineRunPath", path), `${path}.pipelineRunPath`); const pipelinePath = value.pipelinePath === undefined ? null : sourceArtifactPath(y.stringField(value, "pipelinePath", path), `${path}.pipelinePath`); const taskRunTemplate = y.objectField(value, "taskRunTemplate", path); @@ -545,6 +572,7 @@ function parseSourceArtifact(value: Record, path: string): PacS if (mode === "remote-pipeline-annotation" && pipelinePath === null) throw new Error(`${path}.pipelinePath is required for remote-pipeline-annotation`); if (renderer === "agentrun-control-plane" && mode !== "embedded-pipeline-spec") throw new Error(`${path}.renderer agentrun-control-plane requires embedded-pipeline-spec`); if (renderer === "hwlab-runtime-lane" && mode !== "remote-pipeline-annotation") throw new Error(`${path}.renderer hwlab-runtime-lane requires remote-pipeline-annotation`); + if (renderer === "selfmedia-runtime" && mode !== "embedded-pipeline-spec") throw new Error(`${path}.renderer selfmedia-runtime requires embedded-pipeline-spec`); return { mode, renderer, @@ -772,6 +800,19 @@ function validateConfig(config: PacConfig): void { || consumer.argoBootstrap.targetRevision !== consumer.deliveryProvenance.gitOps.targetRevision)) { throw new Error(`${configLabel}.consumers.${consumer.id}.deliveryProvenance.gitOps must match argoBootstrap source identity`); } + if (consumer.runnerServiceAccount !== null && consumer.runnerServiceAccount.name !== consumer.deliveryProvenance.executionServiceAccountName) { + throw new Error(`${configLabel}.consumers.${consumer.id}.runnerServiceAccount.name must match deliveryProvenance.executionServiceAccountName`); + } + } + if (consumer.sourceArtifact?.renderer === "selfmedia-runtime") { + if (consumer.runnerServiceAccount === null) throw new Error(`${configLabel}.consumers.${consumer.id}.runnerServiceAccount is required for selfmedia-runtime`); + if (consumer.argoBootstrap?.repositoryCredential === null || consumer.argoBootstrap?.repositoryCredential === undefined) { + throw new Error(`${configLabel}.consumers.${consumer.id}.argoBootstrap.repositoryCredential is required for the private selfmedia repository`); + } + if (consumer.closeoutGitOpsMirrorFlush) throw new Error(`${configLabel}.consumers.${consumer.id}.closeoutGitOpsMirrorFlush must be false for Gitea writeback GitOps`); + if (repository.params.gitops_secret_name !== repository.secretName) throw new Error(`${configLabel}.repositories.${repository.id}.params.gitops_secret_name must match secretName`); + if (repository.params.gitops_username !== consumer.argoBootstrap.repositoryCredential.username) throw new Error(`${configLabel}.consumers.${consumer.id}.Argo and Tekton Gitea usernames must match`); + if (repository.params.gitops_read_url !== consumer.argoBootstrap.repoUrl) throw new Error(`${configLabel}.consumers.${consumer.id}.Argo repoUrl must match params.gitops_read_url`); } } if (!config.gitea.webhook.events.includes("push")) throw new Error(`${configLabel}.gitea.webhook.events must include push`); @@ -1227,6 +1268,9 @@ function remoteScript(action: "apply" | "status" | "history" | "debug-step", pac UNIDESK_PAC_CONSUMER_CONFIG_B64: Buffer.from(JSON.stringify(consumerConfig), "utf8").toString("base64"), UNIDESK_PAC_DELIVERY_PROVENANCE_REQUIRED: consumer.deliveryProvenance?.required === true ? "1" : "0", UNIDESK_PAC_CONSUMER_RBAC_MANIFEST_B64: Buffer.from(renderPacConsumerRbacDesiredFragment(target.id), "utf8").toString("base64"), + UNIDESK_PAC_RUNNER_SERVICE_ACCOUNT_MANIFEST_B64: Buffer.from(runnerServiceAccountManifest(consumer), "utf8").toString("base64"), + UNIDESK_PAC_RUNNER_SERVICE_ACCOUNT_NAME: consumer.runnerServiceAccount?.name ?? "", + UNIDESK_PAC_RUNNER_ROLE_BINDING_NAME: consumer.runnerServiceAccount?.roleBindingName ?? "", UNIDESK_PAC_CONSUMER_RBAC_CONFIG_SHA256: rbacIdentity.configSha256, UNIDESK_PAC_ADMISSION_POLICY_NAME: admissionIdentity.policyName, UNIDESK_PAC_ADMISSION_BINDING_NAME: admissionIdentity.bindingName, @@ -1242,6 +1286,9 @@ function remoteScript(action: "apply" | "status" | "history" | "debug-step", pac UNIDESK_PAC_ARGO_NAMESPACE: consumer.argoNamespace, UNIDESK_PAC_ARGO_APPLICATION: consumer.argoApplication, UNIDESK_PAC_ARGO_BOOTSTRAP_MANIFEST_B64: Buffer.from(argoBootstrapManifest(consumer), "utf8").toString("base64"), + UNIDESK_PAC_ARGO_REPOSITORY_SECRET_NAME: consumer.argoBootstrap?.repositoryCredential?.secretName ?? "", + UNIDESK_PAC_ARGO_REPOSITORY_USERNAME: consumer.argoBootstrap?.repositoryCredential?.username ?? "", + UNIDESK_PAC_ARGO_REPOSITORY_URL: consumer.argoBootstrap?.repoUrl ?? "", UNIDESK_PAC_PART_OF: pacPartOf(consumer.id), UNIDESK_PAC_SPEC: kubernetesLabelValue(pac.metadata.relatedIssues.includes(1555) ? "GH-1552-GH-1555" : pac.metadata.spec), }; @@ -1288,9 +1335,36 @@ function pacPartOf(consumerId: string): string { if (consumerId === "unidesk-host") return "unidesk-host"; if (consumerId.startsWith("sentinel")) return "hwlab-web-probe-sentinel"; if (consumerId.startsWith("hwlab-")) return "hwlab"; + if (consumerId.startsWith("selfmedia-")) return "selfmedia-factory"; return "agentrun"; } +function runnerServiceAccountManifest(consumer: PacConsumer): string { + const runner = consumer.runnerServiceAccount; + if (runner === null) return ""; + const labels = { + "app.kubernetes.io/managed-by": "unidesk", + "app.kubernetes.io/part-of": pacPartOf(consumer.id), + "unidesk.ai/pac-runner": consumer.id, + }; + const objects = [ + { + apiVersion: "v1", + kind: "ServiceAccount", + metadata: { name: runner.name, namespace: consumer.namespace, labels }, + automountServiceAccountToken: runner.automountServiceAccountToken, + }, + { + apiVersion: "rbac.authorization.k8s.io/v1", + kind: "RoleBinding", + metadata: { name: runner.roleBindingName, namespace: consumer.namespace, labels }, + subjects: [{ kind: "ServiceAccount", name: runner.name, namespace: consumer.namespace }], + roleRef: { apiGroup: "rbac.authorization.k8s.io", kind: "Role", name: "unidesk-pac-consumer-runner" }, + }, + ]; + return `${objects.map((item) => Bun.YAML.stringify(item).trim()).join("\n---\n")}\n`; +} + function argoBootstrapManifest(consumer: PacConsumer): string { const bootstrap = consumer.argoBootstrap; if (bootstrap === null) return ""; @@ -1411,6 +1485,7 @@ function statusSummary(payload: Record): Record): Record): Record 0, + }, valuesPrinted: false, }; } @@ -1666,6 +1747,8 @@ function compactPlanJson(result: Record): Record): Record): RenderedCliResult { const argo = record(summary.argo); const diagnostics = record(summary.diagnostics); const admissionProvenance = record(summary.admissionProvenance); + const consumerBootstrap = record(summary.consumerBootstrap); + const bootstrapRunner = record(consumerBootstrap.runner); + const bootstrapArgo = record(consumerBootstrap.argoRepository); const sourceObservation = record(summary.sourceObservation); const deliveryPlan = record(sourceObservation.plan); const repository = record(summary.repository); @@ -1970,6 +2057,13 @@ function renderStatus(result: Record): RenderedCliResult { ` admission-provenance: ready=${boolText(admissionProvenance.ready)} policy=${stringValue(admissionProvenance.policyName)} binding=${stringValue(admissionProvenance.bindingName)} active-after=${stringValue(admissionProvenance.activeAfter)} default-can-mutate=${boolText(admissionProvenance.defaultCanMutate)}`, ` admission-reasons: ${Array.isArray(admissionProvenance.reasons) ? admissionProvenance.reasons.join(",") || "-" : "-"}`, ]), + ...(consumerBootstrap.configured !== true + ? [" consumer-bootstrap: not-declared"] + : [ + ` consumer-bootstrap: ready=${boolText(consumerBootstrap.ready)} runner=${stringValue(bootstrapRunner.serviceAccount)} exists=${boolText(bootstrapRunner.exists)} automount-disabled=${boolText(bootstrapRunner.automountDisabled)} rolebinding-exact=${boolText(bootstrapRunner.roleBindingExact)}`, + ` argocd-repository-secret: name=${stringValue(bootstrapArgo.secretName)} exists=${boolText(bootstrapArgo.exists)} label=${boolText(bootstrapArgo.labelReady)} keys=${boolText(bootstrapArgo.keysReady)} url=${boolText(bootstrapArgo.urlReady)} password-present=${boolText(bootstrapArgo.passwordPresent)} password-decoded=${boolText(bootstrapArgo.passwordDecoded)}`, + ` consumer-bootstrap-reasons: ${Array.isArray(consumerBootstrap.reasons) ? consumerBootstrap.reasons.join(",") || "-" : "-"}`, + ]), "", "GITEA HOOKS", ...(webhooks.length === 0 ? ["-"] : table(["HOOK", "ACTIVE", "EVENTS", "URL"], webhooks.map((item) => [stringValue(item.id), boolText(item.active), Array.isArray(item.events) ? item.events.join(",") : stringValue(item.events), short(stringValue(item.url), 56)]))), diff --git a/scripts/src/secrets.ts b/scripts/src/secrets.ts index 6c689084..3ae1a934 100644 --- a/scripts/src/secrets.ts +++ b/scripts/src/secrets.ts @@ -1,5 +1,6 @@ -import { randomBytes } from "node:crypto"; +import { createHash, randomBytes } from "node:crypto"; import { chmodSync, existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs"; +import { homedir } from "node:os"; import { dirname, isAbsolute, join } from "node:path"; import type { UniDeskConfig } from "./config"; import { rootPath } from "./config"; @@ -39,23 +40,36 @@ interface SecretDistributionConfig { version: number; kind: "unidesk-secret-distribution"; metadata: { id: string; owner: string; relatedIssues: number[] }; - sources: { root: string; files: SourceFileConfig[] }; + sources: { root: string; files: EnvSourceFileConfig[]; externalFiles: RawSourceFileConfig[] }; targets: DistributionTarget[]; kubernetesSecrets: KubernetesSecretConfig[]; } -interface SourceFileConfig { +interface SourceCreateIfMissingConfig { + enabled: boolean; + values: Record; + randomHex: Record; + randomBase64Url: Record; +} + +interface EnvSourceFileConfig { sourceRef: string; type: "env"; requiredKeys: string[]; - createIfMissing: { - enabled: boolean; - values: Record; - randomHex: Record; - randomBase64Url: Record; - }; + required: true; + createIfMissing: SourceCreateIfMissingConfig; } +interface RawSourceFileConfig { + sourceRef: string; + type: "raw-file"; + requiredKeys: ["contents"]; + required: boolean; + createIfMissing: SourceCreateIfMissingConfig; +} + +type SourceFileConfig = EnvSourceFileConfig | RawSourceFileConfig; + interface DistributionTarget { id: string; route: string; @@ -80,8 +94,10 @@ interface SecretDataMapping { interface SourceMaterial { sourceRef: string; + type: SourceFileConfig["type"]; sourcePath: string; exists: boolean; + required: boolean; requiredKeys: string[]; presentKeys: string[]; missingKeys: string[]; @@ -190,7 +206,7 @@ function parseOptions(args: string[]): SecretsOptions { function plan(options: SecretsOptions): Record { const distribution = readSecretDistributionConfig(options.configPath); assertSelectedTargets(distribution, options); - const sources = inspectSources(distribution, false); + const sources = inspectSources(distribution, false, selectedSourceRefs(distribution, options)); const desired = desiredSecrets(distribution, options, sources); return { ok: sources.ok && desired.every((secret) => secret.missingKeys.length === 0), @@ -200,7 +216,7 @@ function plan(options: SecretsOptions): Record { localSources: sourceSummary(sources), desiredSecrets: desired.map(desiredSecretSummary), policy: { - sourceAuthority: "local YAML-declared sourceRef files under sources.root", + sourceAuthority: "local YAML-declared managed and external sourceRef files", runtimeReverseEngineering: false, valuesPrinted: false, }, @@ -233,7 +249,7 @@ async function sync(config: UniDeskConfig, options: SecretsOptions): Promise secret.missingKeys); @@ -257,7 +273,7 @@ async function sync(config: UniDeskConfig, options: SecretsOptions): Promise> { const distribution = readSecretDistributionConfig(options.configPath); assertSelectedTargets(distribution, options); - const sources = inspectSources(distribution, false); + const sources = inspectSources(distribution, false, selectedSourceRefs(distribution, options)); const desired = desiredSecrets(distribution, options, sources); const perTarget = await Promise.all(groupDesiredSecretsByTarget(desired).map(async (group) => await statusTargetSecrets(config, group.target, group.secrets, options))); return { @@ -293,6 +309,9 @@ function readSecretDistributionConfig(pathArg: string): SecretDistributionConfig sources: { root: stringField(sources, "root", `${label}.sources`), files: arrayOfRecords(sources.files, `${label}.sources.files`).map((item, index) => parseSourceFile(item, `${label}.sources.files[${index}]`)), + externalFiles: sources.externalFiles === undefined + ? [] + : arrayOfRecords(sources.externalFiles, `${label}.sources.externalFiles`).map((item, index) => parseRawSourceFile(item, `${label}.sources.externalFiles[${index}]`)), }, targets: arrayOfRecords(root.targets, `${label}.targets`).map((item, index) => parseTarget(item, `${label}.targets[${index}]`)), kubernetesSecrets: arrayOfRecords(root.kubernetesSecrets, `${label}.kubernetesSecrets`).map((item, index) => parseKubernetesSecret(item, `${label}.kubernetesSecrets[${index}]`)), @@ -301,7 +320,7 @@ function readSecretDistributionConfig(pathArg: string): SecretDistributionConfig return config; } -function parseSourceFile(record: Record, path: string): SourceFileConfig { +function parseSourceFile(record: Record, path: string): EnvSourceFileConfig { const type = stringField(record, "type", path); if (type !== "env") throw new Error(`${path}.type must be env`); const createRaw = record.createIfMissing === undefined ? {} : objectField(record, "createIfMissing", path); @@ -310,6 +329,7 @@ function parseSourceFile(record: Record, path: string): SourceF sourceRef: sourceRefField(record, "sourceRef", path), type, requiredKeys: stringArrayField(record, "requiredKeys", path).map((key, index) => envKeyValue(key, `${path}.requiredKeys[${index}]`)), + required: true, createIfMissing: { enabled: createRaw.enabled === undefined ? false : booleanField(createRaw, "enabled", `${path}.createIfMissing`), values: createRaw.values === undefined ? {} : stringMapField(createRaw, "values", `${path}.createIfMissing`), @@ -319,6 +339,35 @@ function parseSourceFile(record: Record, path: string): SourceF }; } +function parseRawSourceFile(record: Record, path: string): RawSourceFileConfig { + const type = stringField(record, "type", path); + if (type !== "raw-file") throw new Error(`${path}.type must be raw-file`); + const createRaw = objectField(record, "createIfMissing", path); + const enabled = booleanField(createRaw, "enabled", `${path}.createIfMissing`); + const randomHex = createRaw.randomHex === undefined + ? {} + : { contents: randomByteCount(createRaw.randomHex, `${path}.createIfMissing.randomHex`) }; + const randomBase64Url = createRaw.randomBase64Url === undefined + ? {} + : { contents: randomBase64UrlSpec(createRaw.randomBase64Url, `${path}.createIfMissing.randomBase64Url`) }; + if (createRaw.values !== undefined) throw new Error(`${path}.createIfMissing.values is not allowed for raw-file sources`); + const generatorCount = Object.keys(randomHex).length + Object.keys(randomBase64Url).length; + if (enabled && generatorCount !== 1) throw new Error(`${path}.createIfMissing must declare exactly one of randomHex or randomBase64Url when enabled`); + if (!enabled && generatorCount !== 0) throw new Error(`${path}.createIfMissing generators require enabled: true`); + return { + sourceRef: externalSourceRefField(record, "sourceRef", path), + type, + requiredKeys: ["contents"], + required: booleanField(record, "required", path), + createIfMissing: { + enabled, + values: {}, + randomHex, + randomBase64Url, + }, + }; +} + function parseTarget(record: Record, path: string): DistributionTarget { return { id: simpleId(stringField(record, "id", path), `${path}.id`), @@ -338,17 +387,22 @@ function parseKubernetesSecret(record: Record, path: string): K secretName: kubernetesNameField(record, "secretName", path), type, data: arrayOfRecords(record.data, `${path}.data`).map((item, index) => ({ - sourceRef: sourceRefField(item, "sourceRef", `${path}.data[${index}]`), - sourceKey: envKeyField(item, "sourceKey", `${path}.data[${index}]`), + sourceRef: declaredSourceRefField(item, "sourceRef", `${path}.data[${index}]`), + sourceKey: sourceDataKeyField(item, "sourceKey", `${path}.data[${index}]`), targetKey: kubernetesSecretKeyField(item, "targetKey", `${path}.data[${index}]`), })), }; } function validateDistributionConfig(config: SecretDistributionConfig): void { - if (config.sources.files.length === 0) throw new Error(`${config.configPath}.sources.files must not be empty`); + const sourceFiles = allSourceFiles(config); + if (sourceFiles.length === 0) throw new Error(`${config.configPath}.sources must declare files or externalFiles`); if (config.targets.length === 0) throw new Error(`${config.configPath}.targets must not be empty`); - const sources = new Map(config.sources.files.map((item) => [item.sourceRef, item])); + const sources = new Map(); + for (const source of sourceFiles) { + if (sources.has(source.sourceRef)) throw new Error(`${config.configPath}.sources declares duplicate sourceRef ${source.sourceRef}`); + sources.set(source.sourceRef, source); + } const targets = new Set(config.targets.map((item) => item.id)); const targetSecrets = new Set(); for (const secret of config.kubernetesSecrets) { @@ -360,20 +414,25 @@ function validateDistributionConfig(config: SecretDistributionConfig): void { for (const item of secret.data) { const source = sources.get(item.sourceRef); if (source === undefined) throw new Error(`${config.configPath}.kubernetesSecrets.${secret.name} references undeclared sourceRef ${item.sourceRef}`); - if (!source.requiredKeys.includes(item.sourceKey)) throw new Error(`${config.configPath}.kubernetesSecrets.${secret.name} maps ${item.sourceRef}.${item.sourceKey}, but that key is not listed in sources.files.requiredKeys`); + if (!source.requiredKeys.includes(item.sourceKey)) throw new Error(`${config.configPath}.kubernetesSecrets.${secret.name} maps undeclared source key ${item.sourceRef}.${item.sourceKey}`); if (targetKeys.has(item.targetKey)) throw new Error(`${config.configPath}.kubernetesSecrets.${secret.name} maps duplicate target key ${item.targetKey}`); targetKeys.add(item.targetKey); } } } -function inspectSources(config: SecretDistributionConfig, materialize: boolean): SourceInspection { +function allSourceFiles(config: SecretDistributionConfig): SourceFileConfig[] { + return [...config.sources.files, ...config.sources.externalFiles]; +} + +function inspectSources(config: SecretDistributionConfig, materialize: boolean, selectedRefs: Set): SourceInspection { const root = secretRoot(config); const materials = new Map(); - const entries = config.sources.files.map((source) => { - const sourcePath = join(root, source.sourceRef); + const entries = allSourceFiles(config).filter((source) => selectedRefs.has(source.sourceRef)).map((source) => { + const sourcePath = source.type === "env" ? join(root, source.sourceRef) : externalSourcePath(source.sourceRef); const exists = existsSync(sourcePath); - const existing = exists ? parseEnvFile(readFileSync(sourcePath, "utf8")) : {}; + const existingText = exists ? readFileSync(sourcePath, "utf8") : ""; + const existing = exists ? source.type === "env" ? parseEnvFile(existingText) : { contents: existingText } : {}; const next = { ...existing }; const generatedKeys: string[] = []; const unmaterializedGeneratedKeys: string[] = []; @@ -402,11 +461,30 @@ function inspectSources(config: SecretDistributionConfig, materialize: boolean): const missingBefore = source.requiredKeys.filter((key) => existing[key] === undefined || existing[key].length === 0); const missingKeys = source.requiredKeys.filter((key) => next[key] === undefined || next[key].length === 0); const action = missingKeys.length > 0 ? "blocked" : !exists ? "create" : missingBefore.length > 0 ? "update" : "none"; - if (materialize && action !== "blocked" && (action === "create" || action === "update")) writeEnvFile(sourcePath, next); + if (materialize && action !== "blocked" && (action === "create" || action === "update")) { + if (source.type === "env") writeEnvFile(sourcePath, next); + else writeRawFile(sourcePath, next.contents); + } + const materialized = materialize && action !== "blocked" && (action === "create" || action === "update"); + const presence = materialized || (exists && existingText.length > 0); + const bytes = materialized + ? source.type === "env" + ? Buffer.byteLength(readFileSync(sourcePath, "utf8"), "utf8") + : Buffer.byteLength(next.contents, "utf8") + : exists + ? Buffer.byteLength(existingText, "utf8") + : null; + const fingerprint = missingKeys.length === 0 && unmaterializedGeneratedKeys.length === 0 + ? source.type === "raw-file" + ? fingerprintRawFileContent(next.contents) + : fingerprintValues(next, source.requiredKeys) + : null; const material: SourceMaterial = { sourceRef: source.sourceRef, + type: source.type, sourcePath, exists, + required: source.required, requiredKeys: source.requiredKeys, presentKeys: source.requiredKeys.filter((key) => next[key] !== undefined && next[key].length > 0), missingKeys, @@ -414,11 +492,22 @@ function inspectSources(config: SecretDistributionConfig, materialize: boolean): generatedKeys, unmaterializedGeneratedKeys, values: next, - fingerprint: missingKeys.length === 0 && unmaterializedGeneratedKeys.length === 0 ? fingerprintValues(next, source.requiredKeys) : null, + fingerprint, }; materials.set(source.sourceRef, material); + if (source.type === "raw-file") { + return { + sourceRef: source.sourceRef, + type: source.type, + presence, + bytes, + fingerprint, + valuesPrinted: false, + }; + } return { sourceRef: source.sourceRef, + type: source.type, sourcePath: redactRepoPath(sourcePath), exists, requiredKeys: source.requiredKeys, @@ -432,7 +521,7 @@ function inspectSources(config: SecretDistributionConfig, materialize: boolean): }; }); return { - ok: entries.every((entry) => (entry.missingKeys as string[]).length === 0), + ok: Array.from(materials.values()).every((material) => !material.required || material.missingKeys.length === 0), root: redactRepoPath(root), entries, materials, @@ -489,6 +578,13 @@ function selectedTargets(config: SecretDistributionConfig, options: SecretsOptio .filter((target) => options.targetId === null || target.id === options.targetId); } +function selectedSourceRefs(config: SecretDistributionConfig, options: SecretsOptions): Set { + const targetIds = new Set(selectedTargets(config, options).map((target) => target.id)); + return new Set(config.kubernetesSecrets + .filter((secret) => targetIds.has(secret.targetId)) + .flatMap((secret) => secret.data.map((item) => item.sourceRef))); +} + async function applyTargetSecrets(config: UniDeskConfig, target: DistributionTarget, secrets: DesiredSecret[], options: SecretsOptions): Promise> { if (secrets.length === 0) return { ok: true, target: targetSummary(target), mode: "skipped-no-secrets" }; const yaml = renderSecretManifest(target, secrets); @@ -561,21 +657,21 @@ else apply_rc=1 fi python3 - "$ns_rc" "$apply_rc" "$tmp/ns.out" "$tmp/ns.err" "$tmp/apply.out" "$tmp/apply.err" <<'PY' -import base64, json, sys +import base64, json, os, sys ns_rc, apply_rc = int(sys.argv[1]), int(sys.argv[2]) -def text(path, limit=5000): +def byte_count(path): try: - return open(path, encoding="utf-8", errors="replace").read()[-limit:] + return os.path.getsize(path) except FileNotFoundError: - return "" + return 0 payload = { "ok": ns_rc == 0 and apply_rc == 0, "namespace": "${target.namespace}", "secrets": json.loads(base64.b64decode("${summaryB64}").decode("utf-8")), "valuesPrinted": False, "steps": { - "namespace": {"exitCode": ns_rc, "stdout": text(sys.argv[3]), "stderr": text(sys.argv[4])}, - "apply": {"exitCode": apply_rc, "stdout": text(sys.argv[5]), "stderr": text(sys.argv[6])}, + "namespace": {"exitCode": ns_rc, "stdoutBytes": byte_count(sys.argv[3]), "stderrBytes": byte_count(sys.argv[4]), "outputOmitted": True}, + "apply": {"exitCode": apply_rc, "stdoutBytes": byte_count(sys.argv[5]), "stderrBytes": byte_count(sys.argv[6]), "outputOmitted": True}, }, } print(json.dumps(payload, ensure_ascii=False, indent=2)) @@ -645,13 +741,17 @@ function groupDesiredSecretsByTarget(secrets: DesiredSecret[]): Array<{ target: } function configSummary(config: SecretDistributionConfig, options: SecretsOptions): Record { + const sourceRefs = selectedSourceRefs(config, options); return { path: config.configPath, metadata: config.metadata, root: redactRepoPath(secretRoot(config)), scope: options.scope, targetId: options.targetId, - sources: config.sources.files.map((item) => ({ sourceRef: item.sourceRef, requiredKeys: item.requiredKeys, createIfMissing: item.createIfMissing.enabled })), + sources: [ + ...config.sources.files.filter((item) => sourceRefs.has(item.sourceRef)).map((item) => ({ sourceRef: item.sourceRef, type: item.type, requiredKeys: item.requiredKeys, createIfMissing: item.createIfMissing.enabled })), + ...config.sources.externalFiles.filter((item) => sourceRefs.has(item.sourceRef)).map((item) => ({ sourceRef: item.sourceRef, type: item.type, required: item.required, createIfMissing: item.createIfMissing.enabled })), + ], targets: config.targets.filter((target) => (options.scope === null || target.scope === options.scope) && (options.targetId === null || target.id === options.targetId)).map(targetSummary), valuesPrinted: false, }; @@ -748,6 +848,16 @@ function writeEnvFile(path: string, values: Record): void { chmodSync(path, 0o600); } +function writeRawFile(path: string, value: string): void { + mkdirSync(dirname(path), { recursive: true, mode: 0o700 }); + writeFileSync(path, value, { encoding: "utf8", mode: 0o600 }); + chmodSync(path, 0o600); +} + +function fingerprintRawFileContent(value: string): string { + return `sha256:${createHash("sha256").update(value, "utf8").digest("hex")}`; +} + function quoteEnv(value: string): string { if (/^[A-Za-z0-9_./:@%?&=+-]+$/u.test(value)) return value; return `'${value.replaceAll("'", "'\"'\"'")}'`; @@ -837,8 +947,30 @@ function sourceRefField(obj: Record, key: string, path: string) return value; } -function envKeyField(obj: Record, key: string, path: string): string { - return envKeyValue(stringField(obj, key, path), `${path}.${key}`); +function externalSourceRefField(obj: Record, key: string, path: string): string { + const value = stringField(obj, key, path); + if (value.includes("\0") || value.split("/").includes("..")) throw new Error(`${path}.${key} must not contain NUL or parent traversal`); + if (value.startsWith("~/") && value.length > 2) return value; + if (isAbsolute(value)) return value; + throw new Error(`${path}.${key} must be an absolute path or a ~/ home path`); +} + +function declaredSourceRefField(obj: Record, key: string, path: string): string { + const value = stringField(obj, key, path); + if (value.startsWith("~/") || isAbsolute(value)) return externalSourceRefField(obj, key, path); + return sourceRefField(obj, key, path); +} + +function externalSourcePath(sourceRef: string): string { + if (sourceRef.startsWith("~/")) return join(homedir(), sourceRef.slice(2)); + if (isAbsolute(sourceRef)) return sourceRef; + throw new Error(`external sourceRef ${sourceRef} must be an absolute path or a ~/ home path`); +} + +function sourceDataKeyField(obj: Record, key: string, path: string): string { + const value = stringField(obj, key, path); + if (!/^[A-Za-z_][A-Za-z0-9._-]*$/u.test(value)) throw new Error(`${path}.${key} must be a source data key`); + return value; } function envKeyValue(value: string, path: string): string { diff --git a/scripts/src/selfmedia-config.ts b/scripts/src/selfmedia-config.ts new file mode 100644 index 00000000..4f4dde63 --- /dev/null +++ b/scripts/src/selfmedia-config.ts @@ -0,0 +1,299 @@ +import { readFileSync } from "node:fs"; + +import { rootPath } from "./config"; + +export const selfMediaConfigPath = rootPath("config", "selfmedia.yaml"); +export const selfMediaConfigRef = "config/selfmedia.yaml"; + +export interface SelfMediaDeliveryTarget { + readonly id: string; + readonly node: string; + readonly lane: string; + readonly route: string; + readonly ci: { + readonly namespace: string; + readonly pipeline: string; + readonly pipelineRunPrefix: string; + readonly serviceAccountName: string; + readonly serviceAccountAutomount: false; + readonly roleBindingName: string; + readonly workspaceSize: string; + readonly pipelineTimeout: string; + readonly toolImage: string; + readonly buildkitImage: string; + }; + readonly source: { + readonly repository: string; + readonly worktreeRemote: string; + readonly branch: string; + readonly readUrl: string; + readonly snapshotPrefix: string; + }; + readonly build: Record & { + readonly dockerfile: string; + readonly imageRepository: string; + readonly networkMode: string; + readonly proxy: Record; + }; + readonly gitops: Record & { + readonly readUrl: string; + readonly writeUrl: string; + readonly branch: string; + readonly manifestPath: string; + readonly releaseStatePath: string; + readonly credentialSecretName: string; + readonly credentialTokenKey: string; + readonly credentialUsername: string; + }; + readonly deployment: Record; +} + +export interface SelfMediaCutoverTarget { + readonly id: string; + readonly mode: "host-process-to-kubernetes"; + readonly source: { + readonly workspace: string; + readonly publicAddress: string; + readonly healthUrl: string; + readonly stopCommand: readonly string[]; + readonly startCommand: readonly string[]; + readonly dataPaths: readonly string[]; + readonly excludes: readonly string[]; + }; + readonly destination: { + readonly route: string; + readonly namespace: string; + readonly application: string; + readonly publicAddress: string; + readonly healthUrl: string; + readonly dataClaim: string; + readonly codexClaim: string; + }; + readonly prepare: { + readonly requiresFinalConfirmation: true; + readonly phases: readonly string[]; + }; + readonly rollback: { + readonly phases: readonly string[]; + readonly dataPolicy: string; + }; +} + +interface SelfMediaConfig { + readonly defaultTargetId: string; + readonly deliveryTargets: Readonly>; + readonly cutoverTargets: Readonly>; +} + +export function readSelfMediaConfig(): SelfMediaConfig { + const root = record(Bun.YAML.parse(readFileSync(selfMediaConfigPath, "utf8")), selfMediaConfigRef); + if (integer(root.version, "version") !== 1) throw new Error(`${selfMediaConfigRef}.version must be 1`); + if (text(root.kind, "kind") !== "selfmedia-platform-delivery") throw new Error(`${selfMediaConfigRef}.kind must be selfmedia-platform-delivery`); + const defaults = record(root.defaults, "defaults"); + const delivery = record(root.delivery, "delivery"); + const cutover = record(root.cutover, "cutover"); + const deliveryRecords = record(delivery.targets, "delivery.targets"); + const cutoverRecords = record(cutover.targets, "cutover.targets"); + const deliveryTargets = Object.fromEntries(Object.entries(deliveryRecords).map(([id, value]) => [id, parseDeliveryTarget(id, record(value, `delivery.targets.${id}`))])); + const cutoverTargets = Object.fromEntries(Object.entries(cutoverRecords).map(([id, value]) => [id, parseCutoverTarget(id, record(value, `cutover.targets.${id}`))])); + const defaultTargetId = text(defaults.targetId, "defaults.targetId"); + if (deliveryTargets[defaultTargetId] === undefined || cutoverTargets[defaultTargetId] === undefined) { + throw new Error(`${selfMediaConfigRef}.defaults.targetId must resolve in delivery.targets and cutover.targets`); + } + return { defaultTargetId, deliveryTargets, cutoverTargets }; +} + +export function resolveSelfMediaDeliveryTarget(targetId?: string | null): SelfMediaDeliveryTarget { + const config = readSelfMediaConfig(); + const id = targetId ?? config.defaultTargetId; + const target = config.deliveryTargets[id]; + if (target === undefined) throw new Error(`unknown selfmedia delivery target ${id}; known targets: ${Object.keys(config.deliveryTargets).join(", ")}`); + return target; +} + +export function resolveSelfMediaCutoverTarget(targetId?: string | null): SelfMediaCutoverTarget { + const config = readSelfMediaConfig(); + const id = targetId ?? config.defaultTargetId; + const target = config.cutoverTargets[id]; + if (target === undefined) throw new Error(`unknown selfmedia cutover target ${id}; known targets: ${Object.keys(config.cutoverTargets).join(", ")}`); + return target; +} + +export function selfMediaDeliveryConfigRef(targetId: string): string { + return `${selfMediaConfigRef}#delivery.targets.${targetId}`; +} + +export function selfMediaEffectiveDeployment(target: SelfMediaDeliveryTarget): Record { + return { + ...structuredClone(target.deployment), + delivery: { + enabled: true, + source: { + branch: target.source.branch, + snapshotPrefix: target.source.snapshotPrefix, + readUrl: target.source.readUrl, + }, + build: structuredClone(target.build), + gitops: structuredClone(target.gitops), + }, + }; +} + +function parseDeliveryTarget(id: string, value: Record): SelfMediaDeliveryTarget { + const path = `delivery.targets.${id}`; + const ci = record(value.ci, `${path}.ci`); + const source = record(value.source, `${path}.source`); + const build = record(value.build, `${path}.build`); + const proxy = record(build.proxy, `${path}.build.proxy`); + const gitops = record(value.gitops, `${path}.gitops`); + const deployment = record(value.deployment, `${path}.deployment`); + const target = record(deployment.target, `${path}.deployment.target`); + const publicExposure = record(deployment.publicExposure, `${path}.deployment.publicExposure`); + const noProxy = stringArray(proxy.noProxy, `${path}.build.proxy.noProxy`); + if (!noProxy.includes("hyueapi.com") || !noProxy.includes(".hyueapi.com")) throw new Error(`${path}.build.proxy.noProxy must preserve hyueapi.com and .hyueapi.com`); + if (text(value.node, `${path}.node`) !== id) throw new Error(`${path}.node must match target id ${id}`); + if (text(target.id, `${path}.deployment.target.id`) !== id) throw new Error(`${path}.deployment.target.id must match ${id}`); + if (text(source.repository, `${path}.source.repository`) !== "pikainc/selfmedia") throw new Error(`${path}.source.repository must be pikainc/selfmedia`); + if (!text(source.snapshotPrefix, `${path}.source.snapshotPrefix`).startsWith("refs/unidesk/snapshots/")) throw new Error(`${path}.source.snapshotPrefix must be immutable`); + if (boolean(ci.serviceAccountAutomount, `${path}.ci.serviceAccountAutomount`) !== false) throw new Error(`${path}.ci.serviceAccountAutomount must be false`); + if (boolean(publicExposure.enabled, `${path}.deployment.publicExposure.enabled`) !== true) throw new Error(`${path}.deployment.publicExposure.enabled must be true`); + if (integer(publicExposure.hostPort, `${path}.deployment.publicExposure.hostPort`) !== 4317) throw new Error(`${path}.deployment.publicExposure.hostPort must be 4317`); + const gitopsWriteUrl = text(gitops.writeUrl, `${path}.gitops.writeUrl`); + if (!gitopsWriteUrl.startsWith("http://gitea-http.")) throw new Error(`${path}.gitops.writeUrl must use internal Gitea authority`); + return { + id, + node: id, + lane: text(value.lane, `${path}.lane`), + route: text(value.route, `${path}.route`), + ci: { + namespace: kubernetesName(ci.namespace, `${path}.ci.namespace`), + pipeline: kubernetesName(ci.pipeline, `${path}.ci.pipeline`), + pipelineRunPrefix: kubernetesName(ci.pipelineRunPrefix, `${path}.ci.pipelineRunPrefix`), + serviceAccountName: kubernetesName(ci.serviceAccountName, `${path}.ci.serviceAccountName`), + serviceAccountAutomount: false, + roleBindingName: kubernetesName(ci.roleBindingName, `${path}.ci.roleBindingName`), + workspaceSize: text(ci.workspaceSize, `${path}.ci.workspaceSize`), + pipelineTimeout: text(ci.pipelineTimeout, `${path}.ci.pipelineTimeout`), + toolImage: text(ci.toolImage, `${path}.ci.toolImage`), + buildkitImage: text(ci.buildkitImage, `${path}.ci.buildkitImage`), + }, + source: { + repository: "pikainc/selfmedia", + worktreeRemote: url(source.worktreeRemote, `${path}.source.worktreeRemote`), + branch: text(source.branch, `${path}.source.branch`), + readUrl: url(source.readUrl, `${path}.source.readUrl`), + snapshotPrefix: text(source.snapshotPrefix, `${path}.source.snapshotPrefix`), + }, + build: { + ...structuredClone(build), + dockerfile: relativePath(build.dockerfile, `${path}.build.dockerfile`), + imageRepository: text(build.imageRepository, `${path}.build.imageRepository`), + networkMode: text(build.networkMode, `${path}.build.networkMode`), + proxy: structuredClone(proxy), + }, + gitops: { + ...structuredClone(gitops), + readUrl: url(gitops.readUrl, `${path}.gitops.readUrl`), + writeUrl: url(gitops.writeUrl, `${path}.gitops.writeUrl`), + branch: text(gitops.branch, `${path}.gitops.branch`), + manifestPath: relativePath(gitops.manifestPath, `${path}.gitops.manifestPath`), + releaseStatePath: relativePath(gitops.releaseStatePath, `${path}.gitops.releaseStatePath`), + credentialSecretName: kubernetesName(gitops.credentialSecretName, `${path}.gitops.credentialSecretName`), + credentialTokenKey: text(gitops.credentialTokenKey, `${path}.gitops.credentialTokenKey`), + credentialUsername: text(gitops.credentialUsername, `${path}.gitops.credentialUsername`), + }, + deployment: structuredClone(deployment), + }; +} + +function parseCutoverTarget(id: string, value: Record): SelfMediaCutoverTarget { + const path = `cutover.targets.${id}`; + const source = record(value.source, `${path}.source`); + const destination = record(value.destination, `${path}.destination`); + const prepare = record(value.prepare, `${path}.prepare`); + const rollback = record(value.rollback, `${path}.rollback`); + const mode = text(value.mode, `${path}.mode`); + if (mode !== "host-process-to-kubernetes") throw new Error(`${path}.mode must be host-process-to-kubernetes`); + if (boolean(prepare.requiresFinalConfirmation, `${path}.prepare.requiresFinalConfirmation`) !== true) throw new Error(`${path}.prepare.requiresFinalConfirmation must be true`); + return { + id, + mode, + source: { + workspace: absolutePath(source.workspace, `${path}.source.workspace`), + publicAddress: url(source.publicAddress, `${path}.source.publicAddress`), + healthUrl: url(source.healthUrl, `${path}.source.healthUrl`), + stopCommand: stringArray(source.stopCommand, `${path}.source.stopCommand`), + startCommand: stringArray(source.startCommand, `${path}.source.startCommand`), + dataPaths: stringArray(source.dataPaths, `${path}.source.dataPaths`).map((item, index) => relativePath(item, `${path}.source.dataPaths[${index}]`)), + excludes: stringArray(source.excludes, `${path}.source.excludes`).map((item, index) => relativePath(item, `${path}.source.excludes[${index}]`)), + }, + destination: { + route: text(destination.route, `${path}.destination.route`), + namespace: kubernetesName(destination.namespace, `${path}.destination.namespace`), + application: kubernetesName(destination.application, `${path}.destination.application`), + publicAddress: url(destination.publicAddress, `${path}.destination.publicAddress`), + healthUrl: url(destination.healthUrl, `${path}.destination.healthUrl`), + dataClaim: kubernetesName(destination.dataClaim, `${path}.destination.dataClaim`), + codexClaim: kubernetesName(destination.codexClaim, `${path}.destination.codexClaim`), + }, + prepare: { + requiresFinalConfirmation: true, + phases: stringArray(prepare.phases, `${path}.prepare.phases`), + }, + rollback: { + phases: stringArray(rollback.phases, `${path}.rollback.phases`), + dataPolicy: text(rollback.dataPolicy, `${path}.rollback.dataPolicy`), + }, + }; +} + +function record(value: unknown, path: string): Record { + if (value === null || typeof value !== "object" || Array.isArray(value)) throw new Error(`${selfMediaConfigRef}.${path} must be an object`); + return value as Record; +} + +function text(value: unknown, path: string): string { + if (typeof value !== "string" || value.length === 0 || value.includes("\n")) throw new Error(`${selfMediaConfigRef}.${path} must be a non-empty single-line string`); + return value; +} + +function integer(value: unknown, path: string): number { + if (!Number.isInteger(value)) throw new Error(`${selfMediaConfigRef}.${path} must be an integer`); + return value as number; +} + +function boolean(value: unknown, path: string): boolean { + if (typeof value !== "boolean") throw new Error(`${selfMediaConfigRef}.${path} must be a boolean`); + return value; +} + +function stringArray(value: unknown, path: string): string[] { + if (!Array.isArray(value) || value.some((item) => typeof item !== "string" || item.length === 0 || item.includes("\n"))) throw new Error(`${selfMediaConfigRef}.${path} must be an array of non-empty single-line strings`); + return [...value] as string[]; +} + +function kubernetesName(value: unknown, path: string): string { + const result = text(value, path); + if (!/^[a-z0-9](?:[-a-z0-9]*[a-z0-9])?$/u.test(result) || result.length > 63) throw new Error(`${selfMediaConfigRef}.${path} must be a Kubernetes name`); + return result; +} + +function absolutePath(value: unknown, path: string): string { + const result = text(value, path); + if (!result.startsWith("/") || result.split("/").includes("..")) throw new Error(`${selfMediaConfigRef}.${path} must be an absolute path without ..`); + return result; +} + +function relativePath(value: unknown, path: string): string { + const result = text(value, path); + if (result.startsWith("/") || result.split("/").includes("..")) throw new Error(`${selfMediaConfigRef}.${path} must be a relative path without ..`); + return result; +} + +function url(value: unknown, path: string): string { + const result = text(value, path); + const parsed = new URL(result); + if (parsed.protocol !== "http:" && parsed.protocol !== "https:") throw new Error(`${selfMediaConfigRef}.${path} must use http or https`); + if (parsed.username.length > 0 || parsed.password.length > 0) throw new Error(`${selfMediaConfigRef}.${path} must not contain credentials`); + return result; +} diff --git a/scripts/src/selfmedia-delivery-renderer.ts b/scripts/src/selfmedia-delivery-renderer.ts new file mode 100644 index 00000000..acffbfe9 --- /dev/null +++ b/scripts/src/selfmedia-delivery-renderer.ts @@ -0,0 +1,356 @@ +import { existsSync, mkdtempSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { resolve } from "node:path"; +import { spawnSync } from "node:child_process"; + +import { + resolveSelfMediaDeliveryTarget, + selfMediaDeliveryConfigRef, + selfMediaEffectiveDeployment, + type SelfMediaDeliveryTarget, +} from "./selfmedia-config"; +import { stableJsonSha256 } from "./stable-json"; + +export interface SelfMediaRendererBinding { + readonly consumer: { + readonly id: string; + readonly node: string; + readonly lane: string; + readonly namespace: string; + readonly pipeline: string; + readonly pipelineRunPrefix: string; + readonly sourceArtifact: { + readonly configRef: string; + readonly mode: string; + readonly renderer: string; + }; + }; + readonly repository: { + readonly id: string; + readonly params: Readonly>; + }; +} + +export interface SelfMediaRenderedPipeline { + readonly pipeline: Record; + readonly configRef: string; + readonly effectiveConfigSha256: string; + readonly sourceWorktreeRemote: string; +} + +export function renderSelfMediaDesiredPipeline(binding: SelfMediaRendererBinding, sourceWorktree: string): SelfMediaRenderedPipeline { + const target = resolveSelfMediaDeliveryTarget(binding.consumer.node); + const configRef = selfMediaDeliveryConfigRef(target.id); + if (binding.consumer.sourceArtifact.renderer !== "selfmedia-runtime" || binding.consumer.sourceArtifact.mode !== "embedded-pipeline-spec") { + throw new Error("selfmedia-runtime requires embedded-pipeline-spec"); + } + if (binding.consumer.sourceArtifact.configRef !== configRef) { + throw new Error(`sourceArtifact.configRef must equal resolved owning selector ${configRef}; observed ${binding.consumer.sourceArtifact.configRef}`); + } + assertBinding(target, binding); + assertServiceRepositoryContract(sourceWorktree); + const effectiveDeployment = selfMediaEffectiveDeployment(target); + validateRuntimeRender(sourceWorktree, effectiveDeployment); + const effectiveDeploymentB64 = Buffer.from(`${Bun.YAML.stringify(effectiveDeployment).trim()}\n`, "utf8").toString("base64"); + return { + pipeline: pipeline(target, effectiveDeploymentB64), + configRef, + effectiveConfigSha256: stableJsonSha256(target), + sourceWorktreeRemote: target.source.worktreeRemote, + }; +} + +export function selfMediaSourceWorktreeRemote(targetId: string): string { + return resolveSelfMediaDeliveryTarget(targetId).source.worktreeRemote; +} + +function pipeline(target: SelfMediaDeliveryTarget, effectiveDeploymentB64: string): Record { + const params = [ + "revision", + "source-snapshot-prefix", + "git-read-url", + "gitops-read-url", + "gitops-write-url", + "gitops-username", + "gitops-secret-name", + ]; + const taskParams = params.map((name) => ({ name, type: "string" })); + const taskParamBindings = params.map((name) => ({ name, value: `$(params.${name})` })); + return { + apiVersion: "tekton.dev/v1", + kind: "Pipeline", + metadata: { + name: target.ci.pipeline, + namespace: target.ci.namespace, + labels: { + "app.kubernetes.io/name": target.ci.pipeline, + "app.kubernetes.io/part-of": "selfmedia-factory", + "app.kubernetes.io/managed-by": "unidesk", + }, + }, + spec: { + params: taskParams, + workspaces: [], + tasks: [{ + name: "build-and-publish", + params: taskParamBindings, + taskSpec: { + params: taskParams, + volumes: [ + { name: "workspace", emptyDir: { sizeLimit: target.ci.workspaceSize } }, + { name: "buildkit-state", emptyDir: { sizeLimit: "12Gi" } }, + { name: "tmp", emptyDir: { sizeLimit: "4Gi" } }, + { + name: "gitops-token", + secret: { + secretName: "$(params.gitops-secret-name)", + defaultMode: 288, + items: [{ key: target.gitops.credentialTokenKey, path: "token" }], + }, + }, + ], + steps: [ + sourceStep(target, effectiveDeploymentB64), + prepareBuildkitStep(target), + imageBuildStep(target), + gitOpsPublishStep(target), + ], + }, + }], + }, + }; +} + +function sourceStep(target: SelfMediaDeliveryTarget, effectiveDeploymentB64: string): Record { + const proxy = target.build.proxy; + return { + name: "immutable-source", + image: target.ci.toolImage, + imagePullPolicy: "IfNotPresent", + workingDir: "/workspace", + env: [ + { name: "HTTP_PROXY", value: requiredString(proxy.http, "build.proxy.http") }, + { name: "HTTPS_PROXY", value: requiredString(proxy.https, "build.proxy.https") }, + { name: "ALL_PROXY", value: requiredString(proxy.all, "build.proxy.all") }, + { name: "NO_PROXY", value: requiredStringArray(proxy.noProxy, "build.proxy.noProxy").join(",") }, + ], + script: `#!/bin/sh +set -eu +source_commit="$(params.revision)" +snapshot_prefix="$(params.source-snapshot-prefix)" +rm -rf /workspace/source /workspace/release +askpass=/tmp/selfmedia-source-git-askpass +cat >"$askpass" <<'ASKPASS' +#!/bin/sh +case "$1" in + *Username*) printf '%s' "$SELFMEDIA_GIT_USERNAME" ;; + *Password*) cat /var/run/selfmedia-gitops/token ;; + *) exit 1 ;; +esac +ASKPASS +chmod 700 "$askpass" +export SELFMEDIA_GIT_USERNAME="$(params.gitops-username)" +export GIT_ASKPASS="$askpass" +export GIT_TERMINAL_PROMPT=0 +git clone --filter=blob:none --no-checkout "$(params.git-read-url)" /workspace/source +cd /workspace/source +git fetch --depth=1 --filter=blob:none origin "+$snapshot_prefix/$source_commit:refs/remotes/origin/selfmedia-source-snapshot" +git checkout --detach "$source_commit" +test "$(git rev-parse HEAD)" = "$source_commit" +rm -f "$askpass" +printf '%s' '${effectiveDeploymentB64}' | base64 -d > /workspace/effective-deployment.yaml +bun install --frozen-lockfile +bun deploy/scripts/prepare-build.ts \\ + --config /workspace/effective-deployment.yaml \\ + --source-root /workspace/source \\ + --source-commit "$source_commit" \\ + --source-snapshot-prefix "$snapshot_prefix" \\ + --output-dir /workspace/release +`, + securityContext: nonRootSecurity(), + volumeMounts: [ + { name: "workspace", mountPath: "/workspace" }, + { name: "tmp", mountPath: "/tmp" }, + { name: "gitops-token", mountPath: "/var/run/selfmedia-gitops", readOnly: true }, + ], + }; +} + +function prepareBuildkitStep(target: SelfMediaDeliveryTarget): Record { + return { + name: "prepare-buildkit-state", + image: target.ci.toolImage, + imagePullPolicy: "IfNotPresent", + script: "#!/bin/sh\nset -eu\nmkdir -p /home/user/.local/share/buildkit\nchown -R 1000:1000 /home/user/.local/share/buildkit\n", + securityContext: { runAsUser: 0, runAsGroup: 0 }, + volumeMounts: [{ name: "buildkit-state", mountPath: "/home/user/.local/share/buildkit" }], + }; +} + +function imageBuildStep(target: SelfMediaDeliveryTarget): Record { + return { + name: "image-build", + image: target.ci.buildkitImage, + imagePullPolicy: "IfNotPresent", + workingDir: "/workspace", + env: [ + { name: "SOURCE_COMMIT", value: "$(params.revision)" }, + { name: "BUILDKITD_FLAGS", value: "--oci-worker-no-process-sandbox --oci-worker-net=host --allow-insecure-entitlement network.host" }, + ], + script: "#!/bin/sh\nset -eu\nexec /bin/sh /workspace/source/deploy/scripts/build-image.sh\n", + securityContext: { privileged: true, runAsUser: 1000, runAsGroup: 1000 }, + volumeMounts: [ + { name: "workspace", mountPath: "/workspace" }, + { name: "buildkit-state", mountPath: "/home/user/.local/share/buildkit" }, + { name: "tmp", mountPath: "/tmp" }, + ], + }; +} + +function gitOpsPublishStep(target: SelfMediaDeliveryTarget): Record { + return { + name: "gitops-publish", + image: target.ci.toolImage, + imagePullPolicy: "IfNotPresent", + workingDir: "/workspace", + env: [{ name: "SOURCE_COMMIT", value: "$(params.revision)" }], + script: `#!/bin/sh +set -eu +askpass=/tmp/selfmedia-git-askpass +cat >"$askpass" <<'ASKPASS' +#!/bin/sh +case "$1" in + *Username*) printf '%s' "$SELFMEDIA_GIT_USERNAME" ;; + *Password*) cat /var/run/selfmedia-gitops/token ;; + *) exit 1 ;; +esac +ASKPASS +chmod 700 "$askpass" +export SELFMEDIA_GIT_USERNAME="$(params.gitops-username)" +export GIT_ASKPASS="$askpass" +export GIT_TERMINAL_PROMPT=0 +bun /workspace/source/deploy/scripts/publish-gitops.ts \\ + --config /workspace/effective-deployment.yaml \\ + --source-root /workspace/source \\ + --metadata /workspace/build-metadata.json \\ + --source-commit "$SOURCE_COMMIT" \\ + --worktree /workspace/selfmedia-gitops +rm -f "$askpass" +`, + securityContext: nonRootSecurity(), + volumeMounts: [ + { name: "workspace", mountPath: "/workspace" }, + { name: "tmp", mountPath: "/tmp" }, + { name: "gitops-token", mountPath: "/var/run/selfmedia-gitops", readOnly: true }, + ], + }; +} + +function assertBinding(target: SelfMediaDeliveryTarget, binding: SelfMediaRendererBinding): void { + const expected: Record = { + node: target.node, + lane: target.lane, + namespace: target.ci.namespace, + pipeline: target.ci.pipeline, + pipelineRunPrefix: target.ci.pipelineRunPrefix, + }; + const observed: Record = { + node: binding.consumer.node, + lane: binding.consumer.lane, + namespace: binding.consumer.namespace, + pipeline: binding.consumer.pipeline, + pipelineRunPrefix: binding.consumer.pipelineRunPrefix, + }; + for (const [key, value] of Object.entries(expected)) { + if (observed[key] !== value) throw new Error(`PaC ${binding.consumer.id} ${key}=${observed[key]} does not match selfmedia owner ${value}`); + } + const requiredParams: Readonly> = { + node: target.node, + source_branch: target.source.branch, + source_snapshot_prefix: target.source.snapshotPrefix, + git_read_url: target.source.readUrl, + gitops_read_url: target.gitops.readUrl, + gitops_write_url: target.gitops.writeUrl, + gitops_branch: target.gitops.branch, + gitops_username: target.gitops.credentialUsername, + gitops_secret_name: target.gitops.credentialSecretName, + pipeline_name: target.ci.pipeline, + pipeline_run_prefix: target.ci.pipelineRunPrefix, + service_account: target.ci.serviceAccountName, + image_repository: target.build.imageRepository, + pipeline_timeout: target.ci.pipelineTimeout, + }; + for (const [key, value] of Object.entries(requiredParams)) { + if (binding.repository.params[key] !== value) throw new Error(`PaC ${binding.consumer.id} repository params.${key} must match selfmedia owner`); + } +} + +function assertServiceRepositoryContract(sourceWorktree: string): void { + const required = [ + "deploy/Dockerfile", + "deploy/scripts/prepare-build.ts", + "deploy/scripts/build-image.sh", + "deploy/scripts/publish-gitops.ts", + "deploy/scripts/render-runtime.ts", + ]; + for (const file of required) { + if (!existsSync(resolve(sourceWorktree, file))) throw new Error(`selfmedia source repository is missing renderer contract file ${file}`); + } +} + +function validateRuntimeRender(sourceWorktree: string, effectiveDeployment: Record): void { + const temporary = mkdtempSync(resolve(tmpdir(), "unidesk-selfmedia-render-")); + const configPath = resolve(temporary, "effective-deployment.yaml"); + try { + writeFileSync(configPath, `${Bun.YAML.stringify(effectiveDeployment).trim()}\n`, "utf8"); + const result = spawnSync("bun", [ + "deploy/scripts/render-runtime.ts", + "--config", configPath, + "--image", `127.0.0.1:5000/selfmedia/newsroom@sha256:${"a".repeat(64)}`, + "--source-commit", "b".repeat(40), + ], { + cwd: sourceWorktree, + encoding: "utf8", + maxBuffer: 16 * 1024 * 1024, + timeout: 60_000, + }); + if (result.error !== undefined) throw result.error; + if (result.status !== 0) { + const evidence = `${result.stderr || result.stdout || "render-runtime failed"}`.trim().slice(-2000); + throw new Error(`selfmedia effective deployment failed service renderer validation: ${evidence}`); + } + const output = result.stdout; + const required = ["kind: Deployment", "kind: Service", "kind: PersistentVolumeClaim", "kind: NetworkPolicy", "hostPort: 4317", "automountServiceAccountToken: false"]; + for (const marker of required) { + if (!output.includes(marker)) throw new Error(`selfmedia service renderer output is missing ${marker}`); + } + const forbidden = ["kind: Secret", "kind: Role\n", "kind: RoleBinding", "kind: ClusterRole", "kind: ClusterRoleBinding"]; + for (const marker of forbidden) { + if (output.includes(marker)) throw new Error(`selfmedia service renderer output contains forbidden ${marker.trim()}`); + } + if ((output.match(/kind: PersistentVolumeClaim/gu) ?? []).length !== 2) throw new Error("selfmedia service renderer must produce exactly two PVCs"); + if ((output.match(/kind: NetworkPolicy/gu) ?? []).length < 3) throw new Error("selfmedia service renderer must produce at least three NetworkPolicies"); + } finally { + rmSync(temporary, { recursive: true, force: true }); + } +} + +function nonRootSecurity(): Record { + return { + runAsUser: 1000, + runAsGroup: 1000, + runAsNonRoot: true, + allowPrivilegeEscalation: false, + capabilities: { drop: ["ALL"] }, + }; +} + +function requiredString(value: unknown, path: string): string { + if (typeof value !== "string" || value.length === 0) throw new Error(`${path} must be a non-empty string`); + return value; +} + +function requiredStringArray(value: unknown, path: string): string[] { + if (!Array.isArray(value) || value.some((item) => typeof item !== "string" || item.length === 0)) throw new Error(`${path} must be an array of non-empty strings`); + return value as string[]; +} diff --git a/scripts/src/selfmedia.ts b/scripts/src/selfmedia.ts new file mode 100644 index 00000000..148a91fb --- /dev/null +++ b/scripts/src/selfmedia.ts @@ -0,0 +1,253 @@ +import { createHash } from "node:crypto"; +import { createReadStream, existsSync, lstatSync, readdirSync } from "node:fs"; +import { relative, resolve, sep } from "node:path"; + +import { resolveSelfMediaCutoverTarget, resolveSelfMediaDeliveryTarget, selfMediaConfigRef, type SelfMediaCutoverTarget } from "./selfmedia-config"; + +type CutoverAction = "plan" | "prepare" | "status" | "rollback"; + +interface CutoverOptions { + readonly targetId: string | null; + readonly dryRun: boolean; + readonly confirm: boolean; +} + +interface SourceDataSummary { + readonly available: boolean; + readonly files: number; + readonly bytes: number; + readonly digest: string | null; + readonly digestBasis: "relative-path,size,sha256-content"; + readonly missingPaths: readonly string[]; + readonly excludedPaths: readonly string[]; + readonly valuesPrinted: false; +} + +export async function runSelfMediaCommand(args: string[]): Promise> { + if (args.length === 0 || args.includes("--help") || args.includes("-h") || args[0] === "help") return help(); + if (args[0] !== "cutover") return { ok: false, error: "unsupported-selfmedia-command", args, help: help() }; + const action = args[1]; + if (action !== "plan" && action !== "prepare" && action !== "status" && action !== "rollback") { + return { ok: false, error: "unsupported-selfmedia-cutover-command", args, help: help() }; + } + const options = parseOptions(args.slice(2)); + const target = resolveSelfMediaCutoverTarget(options.targetId); + const delivery = resolveSelfMediaDeliveryTarget(target.id); + if (action === "plan") return cutoverPlan(target, delivery.route); + if (action === "status") return await cutoverStatus(target, delivery.route); + if (action === "prepare") return await cutoverPrepare(target, delivery.route, options); + return cutoverRollback(target, delivery.route, options); +} + +function help(): Record { + return { + ok: true, + command: "selfmedia cutover plan|prepare|status|rollback", + configTruth: selfMediaConfigRef, + usage: [ + "bun scripts/cli.ts selfmedia cutover plan --target NC01", + "bun scripts/cli.ts selfmedia cutover prepare --target NC01 --dry-run", + "bun scripts/cli.ts selfmedia cutover status --target NC01", + "bun scripts/cli.ts selfmedia cutover rollback --target NC01 --dry-run", + ], + boundary: "当前入口只做配置校验、源数据摘要和切换计划；不停止服务、不复制数据、不访问 k8s、不切换端口。", + valuesPrinted: false, + }; +} + +function parseOptions(args: string[]): CutoverOptions { + let targetId: string | null = null; + let dryRun = false; + let confirm = false; + for (let index = 0; index < args.length; index += 1) { + const token = args[index]; + if (token === "--target") { + const value = args[index + 1]; + if (value === undefined || value.startsWith("--")) throw new Error("--target requires a value"); + targetId = value; + index += 1; + continue; + } + if (token === "--dry-run") { + dryRun = true; + continue; + } + if (token === "--confirm") { + confirm = true; + continue; + } + if (token === "--json") continue; + throw new Error(`unsupported selfmedia cutover option ${token}`); + } + if (dryRun && confirm) throw new Error("--dry-run and --confirm are mutually exclusive"); + return { targetId, dryRun, confirm }; +} + +function cutoverPlan(target: SelfMediaCutoverTarget, deliveryRoute: string): Record { + return { + ok: true, + action: "selfmedia-cutover-plan", + target: target.id, + mode: target.mode, + mutation: false, + configTruth: `${selfMediaConfigRef}#cutover.targets.${target.id}`, + routeAligned: target.destination.route === deliveryRoute, + source: { + workspace: target.source.workspace, + healthUrl: target.source.healthUrl, + dataPaths: target.source.dataPaths, + excludes: target.source.excludes, + }, + destination: target.destination, + phases: target.prepare.phases, + rollback: { + phases: target.rollback.phases, + dataPolicy: target.rollback.dataPolicy, + }, + safety: { + requiresFinalConfirmation: target.prepare.requiresFinalConfirmation, + oldLifecycleStateExcluded: target.source.excludes.includes(".state/server.json"), + runtimeExecutorAvailable: false, + }, + next: `bun scripts/cli.ts selfmedia cutover prepare --target ${target.id} --dry-run`, + valuesPrinted: false, + }; +} + +async function cutoverStatus(target: SelfMediaCutoverTarget, deliveryRoute: string): Promise> { + const sourceData = await summarizeSourceData(target); + return { + ok: sourceData.available, + action: "selfmedia-cutover-status", + target: target.id, + mutation: false, + configReady: target.destination.route === deliveryRoute && target.source.excludes.includes(".state/server.json"), + sourceWorkspacePresent: existsSync(target.source.workspace), + sourceData, + runtimeObservation: { + status: "not-requested", + reason: "read-only CLI does not contact the host service, k8s, Argo, or public endpoint", + }, + next: sourceData.available + ? `bun scripts/cli.ts selfmedia cutover prepare --target ${target.id} --dry-run` + : "Restore or select the YAML-declared source workspace before preparing cutover.", + valuesPrinted: false, + }; +} + +async function cutoverPrepare(target: SelfMediaCutoverTarget, deliveryRoute: string, options: CutoverOptions): Promise> { + const sourceData = await summarizeSourceData(target); + const confirmedButUnsupported = options.confirm; + return { + ok: sourceData.available && !confirmedButUnsupported, + action: "selfmedia-cutover-prepare", + target: target.id, + mode: options.dryRun || !options.confirm ? "dry-run" : "blocked-confirmed", + mutation: false, + configReady: target.destination.route === deliveryRoute && target.source.excludes.includes(".state/server.json"), + sourceData, + proposedPhases: target.prepare.phases, + portHandoff: { + port: 4317, + oldServiceStopCommand: target.source.stopCommand, + destinationHealthUrl: target.destination.healthUrl, + executed: false, + }, + blocked: confirmedButUnsupported ? { + code: "selfmedia-cutover-runtime-executor-not-implemented", + reason: "本次交付只建立 YAML 所有权和只读/干运行入口，禁止从此命令执行真实切换。", + } : null, + next: confirmedButUnsupported + ? "由后续受控执行器实现 PVC seed、最终增量、端口释放、Argo 同步和原入口验收后再开放 --confirm。" + : `bun scripts/cli.ts selfmedia cutover status --target ${target.id}`, + valuesPrinted: false, + }; +} + +function cutoverRollback(target: SelfMediaCutoverTarget, deliveryRoute: string, options: CutoverOptions): Record { + const confirmedButUnsupported = options.confirm; + return { + ok: !confirmedButUnsupported, + action: "selfmedia-cutover-rollback", + target: target.id, + mode: options.dryRun || !options.confirm ? "dry-run" : "blocked-confirmed", + mutation: false, + configReady: target.destination.route === deliveryRoute, + proposedPhases: target.rollback.phases, + dataPolicy: target.rollback.dataPolicy, + blocked: confirmedButUnsupported ? { + code: "selfmedia-cutover-runtime-executor-not-implemented", + reason: "回滚命令当前只输出计划，不操作 Argo、Deployment、宿主进程或数据。", + } : null, + valuesPrinted: false, + }; +} + +async function summarizeSourceData(target: SelfMediaCutoverTarget): Promise { + const root = resolve(target.source.workspace); + const missingPaths: string[] = []; + const files: { absolute: string; relative: string; size: number }[] = []; + for (const declared of target.source.dataPaths) { + const absolute = resolve(root, declared); + if (!inside(root, absolute) || !existsSync(absolute)) { + missingPaths.push(declared); + continue; + } + collectFiles(root, absolute, target.source.excludes, files); + } + files.sort((left, right) => left.relative.localeCompare(right.relative)); + const digest = createHash("sha256"); + let bytes = 0; + for (const file of files) { + const contentDigest = await hashFile(file.absolute); + bytes += file.size; + digest.update(file.relative); + digest.update("\0"); + digest.update(String(file.size)); + digest.update("\0"); + digest.update(contentDigest); + digest.update("\n"); + } + return { + available: missingPaths.length === 0, + files: files.length, + bytes, + digest: missingPaths.length === 0 ? `sha256:${digest.digest("hex")}` : null, + digestBasis: "relative-path,size,sha256-content", + missingPaths, + excludedPaths: target.source.excludes, + valuesPrinted: false, + }; +} + +function collectFiles(root: string, current: string, excludes: readonly string[], output: { absolute: string; relative: string; size: number }[]): void { + const currentRelative = relative(root, current).split(sep).join("/"); + if (isExcluded(currentRelative, excludes)) return; + const stat = lstatSync(current); + if (stat.isSymbolicLink()) return; + if (stat.isFile()) { + output.push({ absolute: current, relative: currentRelative, size: stat.size }); + return; + } + if (!stat.isDirectory()) return; + for (const entry of readdirSync(current).sort()) collectFiles(root, resolve(current, entry), excludes, output); +} + +function isExcluded(path: string, excludes: readonly string[]): boolean { + return excludes.some((excluded) => path === excluded || path.startsWith(`${excluded}/`)); +} + +function inside(root: string, candidate: string): boolean { + return candidate === root || candidate.startsWith(`${root}${sep}`); +} + +async function hashFile(path: string): Promise { + const digest = createHash("sha256"); + await new Promise((resolvePromise, reject) => { + const stream = createReadStream(path); + stream.on("data", (chunk) => digest.update(chunk)); + stream.on("error", reject); + stream.on("end", resolvePromise); + }); + return digest.digest("hex"); +} From 18ac5d33ba52e8a894fba0eb836b2ec9ebe797fd Mon Sep 17 00:00:00 2001 From: Codex Date: Mon, 13 Jul 2026 07:57:47 +0200 Subject: [PATCH 08/14] =?UTF-8?q?fix:=20=E5=AE=8C=E5=96=84=20Sub2Rank=20Pa?= =?UTF-8?q?C=20=E9=A6=96=E5=8F=91=E6=A0=A1=E9=AA=8C?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- scripts/src/cicd-delivery-authority.test.ts | 6 +++--- .../src/platform-infra-pac-provenance.test.ts | 20 +++++++++++-------- ...infra-pipelines-as-code-source-artifact.ts | 1 + 3 files changed, 16 insertions(+), 11 deletions(-) diff --git a/scripts/src/cicd-delivery-authority.test.ts b/scripts/src/cicd-delivery-authority.test.ts index 62ca14f3..262f91ae 100644 --- a/scripts/src/cicd-delivery-authority.test.ts +++ b/scripts/src/cicd-delivery-authority.test.ts @@ -51,9 +51,9 @@ const pacEvaluator = createRequire(import.meta.url)("../native/cicd/pac-status-e describe("YAML 组合的 CI/CD delivery authority", () => { test("组合 PaC 与 Gitea YAML，覆盖所有实际 consumer 且 id 唯一", () => { const catalog = readCicdDeliveryAuthorityCatalog(); - expect(catalog.consumers).toHaveLength(8); - expect(new Set(catalog.consumers.map((item) => item.consumerId)).size).toBe(8); - for (const consumerId of ["agentrun-nc01-v02", "hwlab-nc01-v03", "sentinel-nc01-v03", "unidesk-host", "platform-infra-gitea-nc01"]) { + expect(catalog.consumers).toHaveLength(9); + expect(new Set(catalog.consumers.map((item) => item.consumerId)).size).toBe(9); + for (const consumerId of ["agentrun-nc01-v02", "hwlab-nc01-v03", "sentinel-nc01-v03", "unidesk-host", "platform-infra-gitea-nc01", "platform-infra-sub2rank-nc01"]) { const authority = resolveCicdDeliveryAuthority({ consumerId }); expect(authority.kind).toBe("pac-pr-merge"); expect(authority.mutationHintsAllowed).toBe(false); diff --git a/scripts/src/platform-infra-pac-provenance.test.ts b/scripts/src/platform-infra-pac-provenance.test.ts index e85d7ab4..872d49f0 100644 --- a/scripts/src/platform-infra-pac-provenance.test.ts +++ b/scripts/src/platform-infra-pac-provenance.test.ts @@ -59,7 +59,7 @@ test("versioned admission desired state protects controller proof, candidate ide toState: "started", }]); expect(contract.child.enabled).toBe(false); - expect(contract.consumers.map((item) => item.id)).toEqual(["agentrun-nc01-v02"]); + expect(contract.consumers.map((item) => item.id)).toEqual(["agentrun-nc01-v02", "platform-infra-sub2rank-nc01"]); const items = documents(renderPacAdmissionDesiredFragment("NC01")); expect(items.map((item) => item.kind)).toEqual(["ValidatingAdmissionPolicy", "ValidatingAdmissionPolicyBinding"]); const policy = items[0]; @@ -117,16 +117,20 @@ test("versioned admission desired state protects controller proof, candidate ide expect(queueGuard.expression).not.toContain('== "Cancelled"'); }); -test("required consumer default RBAC has one automatic desired owner and no Tekton mutation verbs", () => { +test("required consumers have one automatic desired owner per namespace and no Tekton mutation verbs", () => { const rbac = documents(renderPacConsumerRbacDesiredFragment("NC01")); - expect(rbac.map((item) => item.kind)).toEqual(["Role", "RoleBinding"]); - const role = rbac[0]; - expect(role.metadata.annotations["argocd.argoproj.io/sync-wave"]).toBe("-1"); - expect(role.metadata.annotations["unidesk.ai/effective-config-sha256"]).toBe(pacConsumerRbacDesiredIdentity("NC01").configSha256); - const verbs = role.rules.flatMap((rule: any) => rule.verbs); - for (const verb of ["create", "patch", "update", "delete", "deletecollection"]) expect(verbs).not.toContain(verb); + expect(rbac.map((item) => item.kind)).toEqual(["Role", "RoleBinding", "Role", "RoleBinding"]); + expect(rbac.map((item) => item.metadata.namespace)).toEqual(["agentrun-ci", "agentrun-ci", "devops-infra", "devops-infra"]); + for (const role of rbac.filter((item) => item.kind === "Role")) { + expect(role.metadata.annotations["argocd.argoproj.io/sync-wave"]).toBe("-1"); + expect(role.metadata.annotations["unidesk.ai/effective-config-sha256"]).toBe(pacConsumerRbacDesiredIdentity("NC01").configSha256); + const verbs = role.rules.flatMap((rule: any) => rule.verbs); + for (const verb of ["create", "patch", "update", "delete", "deletecollection"]) expect(verbs).not.toContain(verb); + } const desiredKinds = documents(renderPlatformInfraGiteaDesiredFragments("NC01")).map((item) => item.kind); expect(desiredKinds).toEqual([ + "Role", + "RoleBinding", "Role", "RoleBinding", "ValidatingAdmissionPolicy", diff --git a/scripts/src/platform-infra-pipelines-as-code-source-artifact.ts b/scripts/src/platform-infra-pipelines-as-code-source-artifact.ts index 7ffdca93..f5550b76 100644 --- a/scripts/src/platform-infra-pipelines-as-code-source-artifact.ts +++ b/scripts/src/platform-infra-pipelines-as-code-source-artifact.ts @@ -587,6 +587,7 @@ function embeddedPipelineRun(binding: PacSourceArtifactBinding, desiredSpec: Rec labels: pipelineRunLabels(binding, provenance.renderer === "sub2rank-platform-service" ? "sub2rank" : "agentrun"), }, spec: { + timeouts: { pipeline: binding.repository.params.pipeline_timeout }, pipelineSpec: desiredSpec, taskRunTemplate: taskRunTemplate(binding), params: pipelineRunParams(binding, desiredSpec), From f1e9375973fdd6b0f79ea6d0e2df54706603e2c1 Mon Sep 17 00:00:00 2001 From: AgentRun Codex Date: Mon, 13 Jul 2026 06:00:09 +0000 Subject: [PATCH 09/14] =?UTF-8?q?fix:=20=E4=BF=AE=E6=AD=A3=20Monitor=20sta?= =?UTF-8?q?tus=20=E5=B0=B1=E7=BB=AA=E9=80=80=E5=87=BA=E8=AF=AD=E4=B9=89?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- scripts/src/platform-db-monitor-reset.test.ts | 12 +++++++++++- scripts/src/platform-db.ts | 12 ++++++++++-- 2 files changed, 21 insertions(+), 3 deletions(-) diff --git a/scripts/src/platform-db-monitor-reset.test.ts b/scripts/src/platform-db-monitor-reset.test.ts index cb5e72eb..c0e5504d 100644 --- a/scripts/src/platform-db-monitor-reset.test.ts +++ b/scripts/src/platform-db-monitor-reset.test.ts @@ -1,7 +1,7 @@ import { describe, expect, test } from "bun:test"; import { rootPath } from "./config"; import { readYamlRecord } from "./platform-infra-ops-library"; -import { monitorResetScript, monitorStatusScript, runPlatformDbCommand } from "./platform-db"; +import { monitorResetScript, monitorStatusScript, monitorStatusSucceeded, runPlatformDbCommand } from "./platform-db"; const configPath = "config/platform-db/postgres-nc01.yaml"; const parsed = readYamlRecord>(rootPath(configPath), "postgres-host-cluster"); @@ -57,4 +57,14 @@ describe("platform-db Monitor 专属空库入口", () => { expect(reset).toContain('[ "$rows" != "0" ]'); expect(reset).toContain('\"siblingDatabasesUnchanged\":true'); }); + + test("status ready=true 且远端退出 0 时成功", () => { + expect(monitorStatusSucceeded(0, { ok: true, ready: true, schemaReady: true, empty: true, businessRows: 0 })).toBe(true); + }); + + test("status ready=false 时失败，即使远端误返回退出 0", () => { + expect(monitorStatusSucceeded(0, { ok: false, ready: false, schemaReady: true, empty: false, businessRows: 1 })).toBe(false); + expect(monitorStatusSucceeded(0, { ok: true, ready: false, schemaReady: false, empty: false, businessRows: null })).toBe(false); + expect(monitorStatusSucceeded(1, { ok: false, ready: false })).toBe(false); + }); }); diff --git a/scripts/src/platform-db.ts b/scripts/src/platform-db.ts index 51734a14..96141acd 100644 --- a/scripts/src/platform-db.ts +++ b/scripts/src/platform-db.ts @@ -372,8 +372,11 @@ async function monitorCommand(config: UniDeskConfig, action: string, options: Pl const script = action === "status" ? monitorStatusScript(pg, target) : monitorResetScript(pg, target); const capture = await runSshCommandCapture(config, pg.node.route, ["sh"], script); const parsed = parseJsonOutput(capture.stdout); + const commandOk = action === "status" + ? monitorStatusSucceeded(capture.exitCode, parsed) + : capture.exitCode === 0 && parsed?.ok === true; return { - ok: capture.exitCode === 0 && parsed?.ok === true, + ok: commandOk, ...base, mutation: action === "reset", ...(action === "reset" ? { confirmed: true } : {}), @@ -382,6 +385,10 @@ async function monitorCommand(config: UniDeskConfig, action: string, options: Pl }; } +export function monitorStatusSucceeded(exitCode: number, parsed: Record | null): boolean { + return exitCode === 0 && parsed?.ok === true && parsed.ready === true; +} + function unsupported(args: string[]): Record { return { ok: false, @@ -1105,7 +1112,8 @@ fi empty=false ready=false if [ "$schema_ready" = true ] && [ "$business_rows" = "0" ]; then empty=true; ready=true; fi -printf '{"ok":true,"databaseExists":%s,"schemaReady":%s,"empty":%s,"ready":%s,"databaseOid":"%s","schemaVersion":"%s","businessRows":%s,"siblingDatabaseOids":"%s"}\n' "$([ -n "$database_oid" ] && printf true || printf false)" "$schema_ready" "$empty" "$ready" "$database_oid" "$schema_version_observed" "$([ -n "$business_rows" ] && printf '%s' "$business_rows" || printf null)" "$sibling_oids" +printf '{"ok":%s,"databaseExists":%s,"schemaReady":%s,"empty":%s,"ready":%s,"databaseOid":"%s","schemaVersion":"%s","businessRows":%s,"siblingDatabaseOids":"%s"}\n' "$ready" "$([ -n "$database_oid" ] && printf true || printf false)" "$schema_ready" "$empty" "$ready" "$database_oid" "$schema_version_observed" "$([ -n "$business_rows" ] && printf '%s' "$business_rows" || printf null)" "$sibling_oids" +if [ "$ready" != true ]; then exit 1; fi `; } From 90f46dec507066829d474d55101cc8d4a2e37a81 Mon Sep 17 00:00:00 2001 From: Codex Date: Mon, 13 Jul 2026 08:03:39 +0200 Subject: [PATCH 10/14] =?UTF-8?q?docs:=20=E6=94=B6=E5=8F=A3=20Monitor=20?= =?UTF-8?q?=E7=A9=BA=E5=BA=93=E5=88=87=E6=8D=A2?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../R2.5.1_Task_Report.md | 11 +++++++++++ .../R2.5.2_Task_Report.md | 18 ++++++++++++++++++ .../MDTODO/web-sentinel-runtime-reliability.md | 4 ++-- 3 files changed, 31 insertions(+), 2 deletions(-) create mode 100644 docs/MDTODO/details/web-sentinel-runtime-reliability/R2.5.1_Task_Report.md create mode 100644 docs/MDTODO/details/web-sentinel-runtime-reliability/R2.5.2_Task_Report.md diff --git a/docs/MDTODO/details/web-sentinel-runtime-reliability/R2.5.1_Task_Report.md b/docs/MDTODO/details/web-sentinel-runtime-reliability/R2.5.1_Task_Report.md new file mode 100644 index 00000000..d8bf9860 --- /dev/null +++ b/docs/MDTODO/details/web-sentinel-runtime-reliability/R2.5.1_Task_Report.md @@ -0,0 +1,11 @@ +# R2.5.1 任务报告 + +## 结论 + +- 已按 [#1873 最新决策](https://github.com/pikasTech/unidesk/issues/1873#issuecomment-4954659225) 冻结为空库切换：旧 SQLite 历史允许全部丢弃，不执行 export/import/verify，不把 legacy gap 或 payload conflict 作为门禁。 +- #1873 正文与 MDTODO 已同步为“从新数据开始”，后续 Artificer 不再收到迁移旧数据的冲突要求。 +- 切换只允许处理 Monitor 专属 `web_probe_monitor` database/schema；AgentRun、Decision Center 等 Host PG sibling database 不在 mutation 范围。 + +## 边界 + +不双写、不回读 SQLite、不补洞，也不新增迁移兼容层、合同、租约、安全机制或围栏。R2.5.2 仅负责受控空库与 schema 就绪证据。 diff --git a/docs/MDTODO/details/web-sentinel-runtime-reliability/R2.5.2_Task_Report.md b/docs/MDTODO/details/web-sentinel-runtime-reliability/R2.5.2_Task_Report.md new file mode 100644 index 00000000..20c17069 --- /dev/null +++ b/docs/MDTODO/details/web-sentinel-runtime-reliability/R2.5.2_Task_Report.md @@ -0,0 +1,18 @@ +# R2.5.2 任务报告 + +## 结论 + +- [PR #1903](https://github.com/pikasTech/unidesk/pull/1903) 已合并，merge commit 为 `0213fe0196515e99ceba2b2f84adfad90acbe7c2`，提供 YAML-first 的 Monitor 专属 Host PG `plan/status/reset --confirm` 受控入口。 +- owning YAML 精确限制 database=`web_probe_monitor`、owner=`web_probe_monitor`、schema=`monitor`；CLI 不接受任意 database 或 SQL 参数。 +- 空库操作已将 Monitor 业务行从 938 清为 0；未导入或读取 SQLite 历史。 + +## 验证 + +- 持久化 CLI 实测：`databaseExists=true`、`schemaReady=true`、`empty=true`、`ready=true`、`businessRows=0`，schema version=`2026-07-12-p17-3`。 +- sibling OID 摘要保持为 `agentrun_v02:59744,decision_center:119248`，证明未重建这两个数据库。 +- 独立定向测试 `8 pass / 0 fail`；`check --syntax-only` 为 `11/11`；`git diff --check` 通过。 +- 审查发现并修复 `status` 假绿：`ready=false` 现在同时让远端 shell、JSON `ok` 和上层命令退出失败。 + +## 后续 + +只允许 R2.5.3 / R2.6.3 从该空库实施 Release B，开始接收新数据；禁止再次清库或恢复 SQLite 历史。 diff --git a/docs/MDTODO/web-sentinel-runtime-reliability.md b/docs/MDTODO/web-sentinel-runtime-reliability.md index 7a6e8f1b..85bb0fad 100644 --- a/docs/MDTODO/web-sentinel-runtime-reliability.md +++ b/docs/MDTODO/web-sentinel-runtime-reliability.md @@ -51,11 +51,11 @@ PR 合并后只等待自动发布链，使用受控 Web sentinel 原入口产生 按 [#1873 用户最新决策](https://github.com/pikasTech/unidesk/issues/1873#issuecomment-4954659225) 放弃全部 SQLite 历史数据迁移，以空 Monitor Host PG 从新数据开始；先清空并验证 Monitor 专属数据库/schema，再完成 Release B 原子切换，SQLite、legacy gap 与 migration 不进入运行路径，不双写、不回读、不补洞，完成任务后将详细报告写入[任务报告](./details/web-sentinel-runtime-reliability/R2.5_Task_Report.md)。 -#### R2.5.1 [in_progress] +#### R2.5.1 [completed] 按 [#1873 最新决策](https://github.com/pikasTech/unidesk/issues/1873#issuecomment-4954659225) 冻结空库切换边界：取消 legacy gap、SQLite export/import/verify 和 payload conflict 作为门禁，确认只清空 Monitor 自有 Host PG database/schema，不影响共享 Host PG 其他服务，完成任务后将详细报告写入[任务报告](./details/web-sentinel-runtime-reliability/R2.5.1_Task_Report.md)。 -#### R2.5.2 +#### R2.5.2 [completed] 通过 YAML-first 受控入口按 [#1873 最新决策](https://github.com/pikasTech/unidesk/issues/1873#issuecomment-4954659225) 清空或重建 Monitor 专属 Host PG database/schema，验收 schema ready、业务数据 0 行且不读取 SQLite，再允许 Release B，完成任务后将详细报告写入[任务报告](./details/web-sentinel-runtime-reliability/R2.5.2_Task_Report.md)。 From 2b26d78560e2f34e90fe5ced6d9f2c376f7297e6 Mon Sep 17 00:00:00 2001 From: Codex Date: Mon, 13 Jul 2026 08:03:54 +0200 Subject: [PATCH 11/14] =?UTF-8?q?docs:=20=E5=90=AF=E5=8A=A8=20Monitor=20Re?= =?UTF-8?q?lease=20B?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/MDTODO/web-sentinel-runtime-reliability.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/MDTODO/web-sentinel-runtime-reliability.md b/docs/MDTODO/web-sentinel-runtime-reliability.md index 85bb0fad..0a1a2632 100644 --- a/docs/MDTODO/web-sentinel-runtime-reliability.md +++ b/docs/MDTODO/web-sentinel-runtime-reliability.md @@ -59,7 +59,7 @@ PR 合并后只等待自动发布链，使用受控 Web sentinel 原入口产生 通过 YAML-first 受控入口按 [#1873 最新决策](https://github.com/pikasTech/unidesk/issues/1873#issuecomment-4954659225) 清空或重建 Monitor 专属 Host PG database/schema，验收 schema ready、业务数据 0 行且不读取 SQLite，再允许 Release B，完成任务后将详细报告写入[任务报告](./details/web-sentinel-runtime-reliability/R2.5.2_Task_Report.md)。 -#### R2.5.3 +#### R2.5.3 [in_progress] 空 Host PG 验证通过后实施 [#1873](https://github.com/pikasTech/unidesk/issues/1873) Release B，并与 R2.6 / [#1887](https://github.com/pikasTech/unidesk/issues/1887) 的 Monitor 前端重写同步收口：中心 service 成为唯一 writer/query authority，新数据从切换后开始；删除或屏蔽旧 SQLite/query/dashboard/fallback 与旧页面，只观察自动 CI/CD 并完成 CLI、Monitor Web 原入口验收，完成任务后将详细报告写入[任务报告](./details/web-sentinel-runtime-reliability/R2.5.3_Task_Report.md)。 @@ -75,6 +75,6 @@ PR 合并后只等待自动发布链，使用受控 Web sentinel 原入口产生 由 Artificer 按 [#1887](https://github.com/pikasTech/unidesk/issues/1887) 在独立 PR 实现全新 Monitor 页面组件、状态模型和样式，只消费 central Monitor API，覆盖筛选、timeline、详情、loading/empty/error/freshness 与刷新恢复，并运行最小前端定向测试，完成任务后将详细报告写入[任务报告](./details/web-sentinel-runtime-reliability/R2.6.2_Task_Report.md)。 -#### R2.6.3 +#### R2.6.3 [in_progress] 与 R2.5.3 的 Release B 协同合并 [#1887](https://github.com/pikasTech/unidesk/issues/1887)：删除旧页面组件、旧 query/dashboard/fallback 和废弃 CSS，只观察自动 CI/CD，先 internal 后 public 原入口做 1920x1080/960x600 web-probe、深链刷新和截图验收，完成任务后将详细报告写入[任务报告](./details/web-sentinel-runtime-reliability/R2.6.3_Task_Report.md)。 From cf8630a701bb45431382c7a70acc43f66037e3cc Mon Sep 17 00:00:00 2001 From: Codex Date: Mon, 13 Jul 2026 08:09:54 +0200 Subject: [PATCH 12/14] =?UTF-8?q?docs:=20=E8=B7=9F=E8=B8=AA=20Artificer=20?= =?UTF-8?q?=E4=B8=B4=E6=97=B6=E6=AD=A3=E6=96=87=E6=91=A9=E6=93=A6?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/MDTODO/cli-output-progressive-disclosure.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/MDTODO/cli-output-progressive-disclosure.md b/docs/MDTODO/cli-output-progressive-disclosure.md index 02a344e2..eda09bcf 100644 --- a/docs/MDTODO/cli-output-progressive-disclosure.md +++ b/docs/MDTODO/cli-output-progressive-disclosure.md @@ -130,3 +130,11 @@ ### R8.4 [completed] 补齐合并后验收发现的 [UniDesk #1890](https://github.com/pikasTech/unidesk/issues/1890) 残余：gh issue 默认 Next、readCommands、help、skill 与 reference 必须把 trans gh:/owner/repo/issue/ cat 作为人工正文读取首选，结构化 JSON/full/raw 仅作为显式机器披露，完成任务后将详细报告写入[任务报告](./details/cli-output-progressive-disclosure/R8.4_Task_Report.md)。 + +## R9 + +解决 [UniDesk #1905](https://github.com/pikasTech/unidesk/issues/1905)：收敛 Artificer 一次性 GitHub 正文输入，直接使用 quoted heredoc stdin，不再被旧参考误导为先写临时正文文件；保持既有可复用 --body-file 能力，不扩大到 Monitor 主线，完成任务后将详细报告写入[任务报告](./details/cli-output-progressive-disclosure/R9_Task_Report.md)。 + +### R9.1 + +按 [UniDesk #1905](https://github.com/pikasTech/unidesk/issues/1905) 清理冲突的 reference/skill/Next，并在 Artificer YAML prompt/resource bundle 内明确支持 stdin 时禁止一次性 /tmp 正文；补最小 render/行为测试后独立交付，当前只登记不执行，完成任务后将详细报告写入[任务报告](./details/cli-output-progressive-disclosure/R9.1_Task_Report.md)。 From f99aefafbf101161f5ab4d471a33beb923654788 Mon Sep 17 00:00:00 2001 From: Codex Date: Mon, 13 Jul 2026 08:12:47 +0200 Subject: [PATCH 13/14] =?UTF-8?q?fix:=20=E9=9A=94=E7=A6=BB=20Sub2Rank=20Pa?= =?UTF-8?q?C=20=E6=89=A7=E8=A1=8C=E8=BA=AB=E4=BB=BD?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- config/platform-infra/pipelines-as-code.yaml | 5 ++- config/platform-infra/sub2rank.yaml | 2 +- scripts/native/cicd/pac-status-evaluator.cjs | 28 +++++++++++++ ...trun-managed-repository-reconciler.test.ts | 3 ++ ...atform-infra-gitea-gitops-delivery.test.ts | 1 + .../src/platform-infra-pac-consumer-rbac.ts | 39 ++++++++++++++++++- .../src/platform-infra-pac-provenance.test.ts | 37 +++++++++++++++--- scripts/src/platform-infra-pac-provenance.ts | 13 +++++++ ...platform-infra-pipelines-as-code-remote.sh | 35 +++++++++++++++++ ...infra-pipelines-as-code-source-artifact.ts | 10 +++++ .../src/platform-infra-pipelines-as-code.ts | 18 +++++++++ .../src/platform-infra-sub2rank-pipeline.ts | 21 ++++++++++ 12 files changed, 201 insertions(+), 11 deletions(-) diff --git a/config/platform-infra/pipelines-as-code.yaml b/config/platform-infra/pipelines-as-code.yaml index 81a90780..c9ef5b00 100644 --- a/config/platform-infra/pipelines-as-code.yaml +++ b/config/platform-infra/pipelines-as-code.yaml @@ -144,7 +144,7 @@ repositories: gitops_manifest_path: deploy/gitops/platform-infra/sub2rank-nc01/resources.yaml pipeline_name: platform-infra-sub2rank-nc01-pac pipeline_run_prefix: sub2rank-nc01 - service_account: default + service_account: sub2rank-nc01-tekton-runner pipeline_timeout: 600s workspace_pvc_size: 4Gi runtime_namespace: platform-infra @@ -245,7 +245,8 @@ consumers: deliveryProvenance: required: true markerValue: admission-pac-v2:platform-infra-sub2rank-nc01 - executionServiceAccountName: default + executionServiceAccountName: sub2rank-nc01-tekton-runner + manageExecutionServiceAccount: true gitOps: repoUrl: http://git-mirror-http.devops-infra.svc.cluster.local:8080/pikasTech/unidesk.git targetRevision: unidesk-host-gitops diff --git a/config/platform-infra/sub2rank.yaml b/config/platform-infra/sub2rank.yaml index 7594352a..2c6b02e4 100644 --- a/config/platform-infra/sub2rank.yaml +++ b/config/platform-infra/sub2rank.yaml @@ -29,7 +29,7 @@ delivery: namespace: devops-infra name: platform-infra-sub2rank-nc01-pac runPrefix: sub2rank-nc01 - serviceAccountName: default + serviceAccountName: sub2rank-nc01-tekton-runner timeout: 600s workspaceSize: 4Gi toolsImage: 127.0.0.1:5000/hwlab/hwlab-ci-node-tools:node22-alpine-bun-v1 diff --git a/scripts/native/cicd/pac-status-evaluator.cjs b/scripts/native/cicd/pac-status-evaluator.cjs index f14ffb46..123a4811 100644 --- a/scripts/native/cicd/pac-status-evaluator.cjs +++ b/scripts/native/cicd/pac-status-evaluator.cjs @@ -112,6 +112,8 @@ function evaluatePacAdmissionState(inputValue) { const role = record(input.role); const roleBinding = record(input.roleBinding); const expected = record(input.expected); + const executionServiceAccounts = Array.isArray(input.executionServiceAccounts) ? input.executionServiceAccounts.map(record) : []; + const expectedExecutionServiceAccounts = Array.isArray(expected.managedExecutionServiceAccounts) ? expected.managedExecutionServiceAccounts.map(record) : []; const namespace = stringOrNull(input.namespace); const policyMetadata = record(policy.metadata); const bindingMetadata = record(binding.metadata); @@ -167,6 +169,31 @@ function evaluatePacAdmissionState(inputValue) { if (namespace === null || subjects.length !== 1 || subjects[0].kind !== "ServiceAccount" || subjects[0].name !== "default" || subjects[0].namespace !== namespace) reasons.push("default-rolebinding-subject-mismatch"); if (typeof input.defaultCanMutate !== "boolean") reasons.push("default-serviceaccount-mutation-probe-unavailable"); if (input.defaultCanMutate === true) reasons.push("default-serviceaccount-can-mutate-tekton-runs"); + const executionServiceAccountState = expectedExecutionServiceAccounts.map((expectedServiceAccount) => { + const expectedNamespace = stringOrNull(expectedServiceAccount.namespace); + const expectedName = stringOrNull(expectedServiceAccount.name); + const expectedConsumerId = stringOrNull(expectedServiceAccount.consumerId); + const live = executionServiceAccounts.find((item) => { + const metadata = record(item.metadata); + return metadata.namespace === expectedNamespace && metadata.name === expectedName; + }); + const metadata = record(live?.metadata); + const annotations = record(metadata.annotations); + const labels = record(metadata.labels); + const identity = expectedConsumerId || expectedName || "unknown"; + if (live === undefined) reasons.push(`execution-serviceaccount-missing:${identity}`); + if (live !== undefined && live.automountServiceAccountToken !== false) reasons.push(`execution-serviceaccount-automount-not-false:${identity}`); + if (live !== undefined && annotations["unidesk.ai/effective-config-sha256"] !== expected.rbacConfigSha256) reasons.push(`execution-serviceaccount-config-sha256-mismatch:${identity}`); + if (live !== undefined && labels["unidesk.ai/pac-consumer"] !== expectedConsumerId) reasons.push(`execution-serviceaccount-consumer-mismatch:${identity}`); + return { + consumerId: expectedConsumerId, + namespace: expectedNamespace, + name: expectedName, + present: live !== undefined, + automountServiceAccountToken: live === undefined ? null : live.automountServiceAccountToken === false ? false : true, + configSha256: stringOrNull(annotations["unidesk.ai/effective-config-sha256"]), + }; + }); const timestamps = [policyMetadata.creationTimestamp, bindingMetadata.creationTimestamp] .filter((value) => typeof value === "string" && Number.isFinite(Date.parse(value))) .sort((left, right) => Date.parse(left) - Date.parse(right)); @@ -189,6 +216,7 @@ function evaluatePacAdmissionState(inputValue) { observedGeneration: Number.isInteger(policyStatus.observedGeneration) ? policyStatus.observedGeneration : null, rbacConfigSha256: stringOrNull(roleAnnotations["unidesk.ai/effective-config-sha256"]), defaultCanMutate: typeof input.defaultCanMutate === "boolean" ? input.defaultCanMutate : null, + executionServiceAccounts: executionServiceAccountState, activeAfter, reasons, valuesPrinted: false, diff --git a/scripts/src/agentrun-managed-repository-reconciler.test.ts b/scripts/src/agentrun-managed-repository-reconciler.test.ts index 51c406f8..f5ecf449 100644 --- a/scripts/src/agentrun-managed-repository-reconciler.test.ts +++ b/scripts/src/agentrun-managed-repository-reconciler.test.ts @@ -89,6 +89,9 @@ describe("AgentRun YAML-owned managed repository reconciler", () => { expect(documents.map((item) => item.kind)).toEqual([ "Role", "RoleBinding", + "Role", + "RoleBinding", + "ServiceAccount", "ValidatingAdmissionPolicy", "ValidatingAdmissionPolicyBinding", "ServiceAccount", diff --git a/scripts/src/platform-infra-gitea-gitops-delivery.test.ts b/scripts/src/platform-infra-gitea-gitops-delivery.test.ts index 3c8fbdff..c0bc2342 100644 --- a/scripts/src/platform-infra-gitea-gitops-delivery.test.ts +++ b/scripts/src/platform-infra-gitea-gitops-delivery.test.ts @@ -80,6 +80,7 @@ test("owning YAML renders one child Application and the complete durable bridge "RoleBinding", "Role", "RoleBinding", + "ServiceAccount", "ValidatingAdmissionPolicy", "ValidatingAdmissionPolicyBinding", "ServiceAccount", diff --git a/scripts/src/platform-infra-pac-consumer-rbac.ts b/scripts/src/platform-infra-pac-consumer-rbac.ts index 58ee9a87..12963abc 100644 --- a/scripts/src/platform-infra-pac-consumer-rbac.ts +++ b/scripts/src/platform-infra-pac-consumer-rbac.ts @@ -5,6 +5,12 @@ import { readPacAdmissionContract } from "./platform-infra-pac-provenance"; export interface PacConsumerRbacDesiredIdentity { readonly configSha256: string; readonly namespaces: readonly string[]; + readonly executionServiceAccounts: readonly { + readonly consumerId: string; + readonly namespace: string; + readonly name: string; + readonly automountServiceAccountToken: false; + }[]; } const pacConsumerReadOnlyRules = [ @@ -19,7 +25,7 @@ export function renderPacConsumerRbacDesiredFragment(targetId: string): string { const namespaces = [...new Set(consumers.map((consumer) => consumer.namespace))]; if (namespaces.length === 0) return ""; const configSha256 = pacConsumerRbacDesiredIdentity(targetId).configSha256; - const objects = namespaces.flatMap((namespace) => { + const objects: Record[] = namespaces.flatMap((namespace) => { const labels = { "app.kubernetes.io/managed-by": "unidesk", "app.kubernetes.io/part-of": "platform-infra", @@ -45,6 +51,25 @@ export function renderPacConsumerRbacDesiredFragment(targetId: string): string { }, ]; }); + objects.push(...consumers.filter((consumer) => consumer.manageExecutionServiceAccount).map((consumer) => ({ + apiVersion: "v1", + kind: "ServiceAccount", + metadata: { + name: consumer.executionServiceAccountName, + namespace: consumer.namespace, + labels: { + "app.kubernetes.io/managed-by": "unidesk", + "app.kubernetes.io/part-of": "platform-infra", + "unidesk.ai/pac-rbac-contract": "admission-provenance-required", + "unidesk.ai/pac-consumer": consumer.id, + }, + annotations: { + "argocd.argoproj.io/sync-wave": "-2", + "unidesk.ai/effective-config-sha256": configSha256, + }, + }, + automountServiceAccountToken: false, + }))); return `${objects.map((item) => Bun.YAML.stringify(item).trim()).join("\n---\n")}\n`; } @@ -60,8 +85,18 @@ export function pacConsumerRbacDesiredIdentity(targetId: string): PacConsumerRba roleRef: pacConsumerDefaultRoleRef, }, })); + const executionServiceAccounts = consumers + .filter((consumer) => consumer.manageExecutionServiceAccount) + .map((consumer) => ({ + consumerId: consumer.id, + namespace: consumer.namespace, + name: consumer.executionServiceAccountName, + automountServiceAccountToken: false as const, + })) + .sort((left, right) => `${left.namespace}/${left.name}`.localeCompare(`${right.namespace}/${right.name}`)); return { - configSha256: `sha256:${createHash("sha256").update(JSON.stringify({ targetId, desiredSpec, contract: "admission-provenance-required-v1" })).digest("hex")}`, + configSha256: `sha256:${createHash("sha256").update(JSON.stringify({ targetId, desiredSpec, executionServiceAccounts, contract: "admission-provenance-required-v2" })).digest("hex")}`, namespaces, + executionServiceAccounts, }; } diff --git a/scripts/src/platform-infra-pac-provenance.test.ts b/scripts/src/platform-infra-pac-provenance.test.ts index 872d49f0..26c3f28f 100644 --- a/scripts/src/platform-infra-pac-provenance.test.ts +++ b/scripts/src/platform-infra-pac-provenance.test.ts @@ -25,9 +25,18 @@ function documents(value: string): any[] { test("admission contract rejects malformed required declarations instead of silently downgrading", () => { const source = Bun.YAML.parse(readFileSync(resolve(root, "config/platform-infra/pipelines-as-code.yaml"), "utf8")) as any; - const consumer = source.consumers.find((item: any) => item.deliveryProvenance !== undefined); - consumer.deliveryProvenance.required = "true"; - expect(() => parsePacAdmissionContract(source)).toThrow("deliveryProvenance.required must be boolean true or false"); + const malformedRequired = structuredClone(source); + malformedRequired.consumers.find((item: any) => item.deliveryProvenance !== undefined).deliveryProvenance.required = "true"; + expect(() => parsePacAdmissionContract(malformedRequired)).toThrow("deliveryProvenance.required must be boolean true or false"); + + const sharedDefault = structuredClone(source); + sharedDefault.consumers.find((item: any) => item.deliveryProvenance !== undefined).deliveryProvenance.executionServiceAccountName = "default"; + expect(() => parsePacAdmissionContract(sharedDefault)).toThrow("must not reserve the shared default ServiceAccount"); + + const duplicatedServiceAccount = structuredClone(source); + const duplicatedRequiredConsumers = duplicatedServiceAccount.consumers.filter((item: any) => item.deliveryProvenance !== undefined); + duplicatedRequiredConsumers[1].deliveryProvenance.executionServiceAccountName = duplicatedRequiredConsumers[0].deliveryProvenance.executionServiceAccountName; + expect(() => parsePacAdmissionContract(duplicatedServiceAccount)).toThrow("executionServiceAccountName must be unique per target"); }); test("admission queue transition contract rejects absent sources, unknown Tekton states, and duplicate transitions", () => { @@ -79,6 +88,8 @@ test("versioned admission desired state protects controller proof, candidate ide expect(expressions).toContain("object.metadata.name.startsWith"); expect(expressions).toContain('object.metadata.labels["tekton.dev/pipeline"]'); expect(expressions).toContain("agentrun-nc01-v02-tekton-runner"); + expect(expressions).toContain("sub2rank-nc01-tekton-runner"); + expect(expressions).not.toContain('serviceAccountName == "default"'); expect(messages).toContain("execution ServiceAccount"); expect(expressions).toContain(contract.child.parentUidAnnotation); expect(expressions).toContain("oldObject"); @@ -113,26 +124,34 @@ test("versioned admission desired state protects controller proof, candidate ide expect(queueGuard.expression.match(/admission-pac-v2:agentrun-nc01-v02/gu)).toHaveLength(2); expect(queueGuard.expression.match(/pipelines-as-code-watcher/gu)).toHaveLength(1); expect(queueGuard.expression.match(/agentrun-nc01-v02-tekton-runner/gu)).toHaveLength(2); + expect(queueGuard.expression.match(/sub2rank-nc01-tekton-runner/gu)).toHaveLength(2); expect(queueGuard.expression).not.toContain("admission-pac-v1:"); expect(queueGuard.expression).not.toContain('== "Cancelled"'); }); test("required consumers have one automatic desired owner per namespace and no Tekton mutation verbs", () => { const rbac = documents(renderPacConsumerRbacDesiredFragment("NC01")); - expect(rbac.map((item) => item.kind)).toEqual(["Role", "RoleBinding", "Role", "RoleBinding"]); - expect(rbac.map((item) => item.metadata.namespace)).toEqual(["agentrun-ci", "agentrun-ci", "devops-infra", "devops-infra"]); + expect(rbac.map((item) => item.kind)).toEqual(["Role", "RoleBinding", "Role", "RoleBinding", "ServiceAccount"]); + expect(rbac.map((item) => item.metadata.namespace)).toEqual(["agentrun-ci", "agentrun-ci", "devops-infra", "devops-infra", "devops-infra"]); for (const role of rbac.filter((item) => item.kind === "Role")) { expect(role.metadata.annotations["argocd.argoproj.io/sync-wave"]).toBe("-1"); expect(role.metadata.annotations["unidesk.ai/effective-config-sha256"]).toBe(pacConsumerRbacDesiredIdentity("NC01").configSha256); const verbs = role.rules.flatMap((rule: any) => rule.verbs); for (const verb of ["create", "patch", "update", "delete", "deletecollection"]) expect(verbs).not.toContain(verb); } + const executionServiceAccount = rbac.find((item) => item.kind === "ServiceAccount"); + expect(executionServiceAccount).toMatchObject({ + metadata: { name: "sub2rank-nc01-tekton-runner", namespace: "devops-infra" }, + automountServiceAccountToken: false, + }); + expect(executionServiceAccount.metadata.annotations["argocd.argoproj.io/sync-wave"]).toBe("-2"); const desiredKinds = documents(renderPlatformInfraGiteaDesiredFragments("NC01")).map((item) => item.kind); expect(desiredKinds).toEqual([ "Role", "RoleBinding", "Role", "RoleBinding", + "ServiceAccount", "ValidatingAdmissionPolicy", "ValidatingAdmissionPolicyBinding", "ServiceAccount", @@ -145,6 +164,7 @@ test("live admission evaluator requires exact desired hashes, observed generatio const contract = readPacAdmissionContract(); const admission = documents(renderPacAdmissionDesiredFragment("NC01")); const rbac = documents(renderPacConsumerRbacDesiredFragment("NC01")); + const executionServiceAccount = rbac.find((item) => item.kind === "ServiceAccount"); const policy = { ...admission[0], metadata: { ...admission[0].metadata, uid: "policy-uid", generation: 2, creationTimestamp: "2026-01-01T00:00:00Z" }, status: { observedGeneration: 2, typeChecking: { expressionWarnings: [] } } }; const binding = { ...admission[1], metadata: { ...admission[1].metadata, uid: "binding-uid", creationTimestamp: "2026-01-01T00:00:01Z" } }; const expected = { @@ -155,9 +175,14 @@ test("live admission evaluator requires exact desired hashes, observed generatio controllerIdentity: contract.controllerUsername, configSha256: policy.metadata.annotations["unidesk.ai/effective-config-sha256"], rbacConfigSha256: rbac[0].metadata.annotations["unidesk.ai/effective-config-sha256"], + managedExecutionServiceAccounts: pacConsumerRbacDesiredIdentity("NC01").executionServiceAccounts, }; - const input = { policy, binding, role: rbac[0], roleBinding: rbac[1], namespace: "agentrun-ci", defaultCanMutate: false, expected }; + const input = { policy, binding, role: rbac[0], roleBinding: rbac[1], namespace: "agentrun-ci", defaultCanMutate: false, executionServiceAccounts: [executionServiceAccount], expected }; expect(evaluator.evaluatePacAdmissionState(input)).toMatchObject({ ready: true, activeAfter: "2026-01-01T00:00:01Z", defaultCanMutate: false }); + expect(evaluator.evaluatePacAdmissionState({ ...input, executionServiceAccounts: [] })).toMatchObject({ + ready: false, + reasons: expect.arrayContaining(["execution-serviceaccount-missing:platform-infra-sub2rank-nc01"]), + }); expect(evaluator.evaluatePacAdmissionState({ ...input, policy: { diff --git a/scripts/src/platform-infra-pac-provenance.ts b/scripts/src/platform-infra-pac-provenance.ts index 434bbb02..951cc337 100644 --- a/scripts/src/platform-infra-pac-provenance.ts +++ b/scripts/src/platform-infra-pac-provenance.ts @@ -14,6 +14,7 @@ export interface PacAdmissionConsumerContract { readonly pipelineRunPrefix: string; readonly markerValue: string; readonly executionServiceAccountName: string; + readonly manageExecutionServiceAccount: boolean; readonly gitOps: { readonly repoUrl: string; readonly targetRevision: string }; } @@ -98,6 +99,7 @@ export function parsePacAdmissionContract(source: Record): PacA pipelineRunPrefix: required(consumer.pipelineRunPrefix, `consumers[${index}].pipelineRunPrefix`), markerValue: required(value.markerValue, `consumers[${index}].deliveryProvenance.markerValue`), executionServiceAccountName: required(value.executionServiceAccountName, `consumers[${index}].deliveryProvenance.executionServiceAccountName`), + manageExecutionServiceAccount: optionalBoolean(value.manageExecutionServiceAccount, `consumers[${index}].deliveryProvenance.manageExecutionServiceAccount`, false), gitOps: { repoUrl: required(gitOps.repoUrl, `consumers[${index}].deliveryProvenance.gitOps.repoUrl`), targetRevision: required(gitOps.targetRevision, `consumers[${index}].deliveryProvenance.gitOps.targetRevision`), @@ -106,6 +108,11 @@ export function parsePacAdmissionContract(source: Record): PacA }); const markerValues = new Set(consumers.map((consumer) => consumer.markerValue)); if (markerValues.size !== consumers.length) throw new Error("deliveryProvenance markerValue must be unique per consumer"); + for (const consumer of consumers) { + if (consumer.executionServiceAccountName === "default") throw new Error(`deliveryProvenance executionServiceAccountName for ${consumer.id} must not reserve the shared default ServiceAccount`); + } + const serviceAccountKeys = consumers.map((consumer) => `${consumer.node.toLowerCase()}\u0000${consumer.executionServiceAccountName}`); + if (new Set(serviceAccountKeys).size !== serviceAccountKeys.length) throw new Error("deliveryProvenance executionServiceAccountName must be unique per target"); const version = required(provenance.version, "deliveryProvenance.version"); const versionMatch = version.match(/-v([1-9][0-9]*)$/u); if (versionMatch === null) throw new Error("deliveryProvenance.version must end with a positive -vN resource epoch"); @@ -393,3 +400,9 @@ function literal(value: unknown, path: string, expec if (value !== expected) throw new Error(`${path} must be ${expected}`); return expected; } + +function optionalBoolean(value: unknown, path: string, fallback: boolean): boolean { + if (value === undefined) return fallback; + if (typeof value !== "boolean") throw new Error(`${path} must be boolean`); + return value; +} diff --git a/scripts/src/platform-infra-pipelines-as-code-remote.sh b/scripts/src/platform-infra-pipelines-as-code-remote.sh index 30d88de2..61b9a669 100644 --- a/scripts/src/platform-infra-pipelines-as-code-remote.sh +++ b/scripts/src/platform-infra-pipelines-as-code-remote.sh @@ -291,10 +291,28 @@ admission_provenance_state() { export UNIDESK_PAC_DEFAULT_CAN_MUTATE="$default_can_mutate" node - "$policy_file" "$binding_file" "$role_file" "$role_binding_file" <<'NODE' const fs = require('node:fs'); +const cp = require('node:child_process'); const { evaluatePacAdmissionState } = require(process.env.UNIDESK_PAC_EVALUATOR_PATH); function read(path) { try { return JSON.parse(fs.readFileSync(path, 'utf8') || '{}'); } catch { return {}; } } +let managedExecutionServiceAccounts = []; +try { + const parsed = JSON.parse(process.env.UNIDESK_PAC_MANAGED_EXECUTION_SERVICE_ACCOUNTS_JSON || '[]'); + if (Array.isArray(parsed)) managedExecutionServiceAccounts = parsed; +} catch {} +const waitTimeoutSeconds = Number.parseInt(process.env.UNIDESK_PAC_WAIT_TIMEOUT_SECONDS || '', 10); +const kubectlTimeoutMs = Number.isInteger(waitTimeoutSeconds) && waitTimeoutSeconds > 0 ? waitTimeoutSeconds * 1000 : null; +const executionServiceAccounts = managedExecutionServiceAccounts.map((item) => { + if (kubectlTimeoutMs === null || !item || typeof item.namespace !== 'string' || typeof item.name !== 'string') return {}; + const result = cp.spawnSync('kubectl', ['-n', item.namespace, 'get', 'serviceaccount', item.name, '-o', 'json'], { + encoding: 'utf8', + timeout: kubectlTimeoutMs, + maxBuffer: 1024 * 1024, + }); + if (result.error || result.status !== 0) return {}; + try { return JSON.parse(result.stdout || '{}'); } catch { return {}; } +}); process.stdout.write(JSON.stringify(evaluatePacAdmissionState({ policy: read(process.argv[2]), binding: read(process.argv[3]), @@ -302,6 +320,7 @@ process.stdout.write(JSON.stringify(evaluatePacAdmissionState({ roleBinding: read(process.argv[5]), namespace: process.env.UNIDESK_PAC_TARGET_NAMESPACE || null, defaultCanMutate: process.env.UNIDESK_PAC_DEFAULT_CAN_MUTATE === 'unknown' ? null : process.env.UNIDESK_PAC_DEFAULT_CAN_MUTATE === 'true', + executionServiceAccounts, expected: { targetId: process.env.UNIDESK_PAC_TARGET_ID || '', policyName: process.env.UNIDESK_PAC_ADMISSION_POLICY_NAME || '', @@ -310,6 +329,7 @@ process.stdout.write(JSON.stringify(evaluatePacAdmissionState({ controllerIdentity: process.env.UNIDESK_PAC_ADMISSION_CONTROLLER_IDENTITY || '', configSha256: process.env.UNIDESK_PAC_ADMISSION_CONFIG_SHA256 || '', rbacConfigSha256: process.env.UNIDESK_PAC_CONSUMER_RBAC_CONFIG_SHA256 || '', + managedExecutionServiceAccounts, }, }))); NODE @@ -470,6 +490,7 @@ function kubectlJson(args, fallback, context) { let admissionPolicy = null; let admissionBinding = null; const admissionRbac = new Map(); +const admissionExecutionServiceAccounts = new Map(); function defaultCanMutate(namespace) { const waitTimeoutSeconds = Number.parseInt(process.env.UNIDESK_PAC_WAIT_TIMEOUT_SECONDS || '', 10); @@ -507,6 +528,18 @@ function admissionStateForConsumer(consumer) { }); } const rbac = admissionRbac.get(consumer.namespace); + const managedExecutionServiceAccounts = Array.isArray(expected.managedExecutionServiceAccounts) ? expected.managedExecutionServiceAccounts : []; + const executionServiceAccounts = managedExecutionServiceAccounts.map((item) => { + const key = `${item.namespace || ''}/${item.name || ''}`; + if (!admissionExecutionServiceAccounts.has(key)) { + admissionExecutionServiceAccounts.set(key, kubectlJson( + ['-n', item.namespace, 'get', 'serviceaccount', item.name, '-o', 'json'], + {}, + `${consumer.id}:execution-serviceaccount:${key}`, + )); + } + return admissionExecutionServiceAccounts.get(key); + }); return evaluatePacAdmissionState({ policy: admissionPolicy, binding: admissionBinding, @@ -514,6 +547,7 @@ function admissionStateForConsumer(consumer) { roleBinding: rbac.roleBinding, namespace: consumer.namespace, defaultCanMutate: rbac.defaultCanMutate, + executionServiceAccounts, expected: { targetId: expected.targetId, policyName: expected.policyName, @@ -522,6 +556,7 @@ function admissionStateForConsumer(consumer) { controllerIdentity: expected.controllerIdentity, configSha256: expected.configSha256, rbacConfigSha256: expected.rbacConfigSha256, + managedExecutionServiceAccounts, }, }); } diff --git a/scripts/src/platform-infra-pipelines-as-code-source-artifact.ts b/scripts/src/platform-infra-pipelines-as-code-source-artifact.ts index f5550b76..bc1da2af 100644 --- a/scripts/src/platform-infra-pipelines-as-code-source-artifact.ts +++ b/scripts/src/platform-infra-pipelines-as-code-source-artifact.ts @@ -49,6 +49,16 @@ export interface PacSourceArtifactBinding { readonly namespace: string; readonly pipeline: string; readonly pipelineRunPrefix: string; + readonly argoNamespace: string; + readonly argoApplication: string; + readonly argoBootstrap: { + readonly project: string; + readonly repoUrl: string; + readonly targetRevision: string; + readonly path: string; + readonly destinationNamespace: string; + readonly automated: boolean; + } | null; readonly deliveryProvenance?: { readonly markerAnnotation: string; readonly markerValue: string; diff --git a/scripts/src/platform-infra-pipelines-as-code.ts b/scripts/src/platform-infra-pipelines-as-code.ts index 21eff00c..c8c595d4 100644 --- a/scripts/src/platform-infra-pipelines-as-code.ts +++ b/scripts/src/platform-infra-pipelines-as-code.ts @@ -159,6 +159,7 @@ interface PacConsumer { required: true; markerValue: string; executionServiceAccountName: string; + manageExecutionServiceAccount: boolean; gitOps: { repoUrl: string; targetRevision: string }; } | null; repositoryRef: string; @@ -271,6 +272,9 @@ export async function runPlatformInfraPipelinesAsCodeCommand(config: UniDeskConf namespace: consumer.namespace, pipeline: consumer.pipeline, pipelineRunPrefix: consumer.pipelineRunPrefix, + argoNamespace: consumer.argoNamespace, + argoApplication: consumer.argoApplication, + argoBootstrap: consumer.argoBootstrap, deliveryProvenance: consumer.deliveryProvenance === null ? null : { markerAnnotation: pac.deliveryProvenance.markerAnnotation, markerValue: consumer.deliveryProvenance.markerValue, @@ -528,6 +532,9 @@ function parseConsumerDeliveryProvenance(value: Record, path: s required: true, markerValue: y.stringField(value, "markerValue", path), executionServiceAccountName: y.kubernetesNameField(value, "executionServiceAccountName", path), + manageExecutionServiceAccount: value.manageExecutionServiceAccount === undefined + ? false + : y.booleanField(value, "manageExecutionServiceAccount", path), gitOps: { repoUrl: urlField(gitOps, "repoUrl", `${path}.gitOps`), targetRevision: y.stringField(gitOps, "targetRevision", `${path}.gitOps`), @@ -756,6 +763,7 @@ function validateConfig(config: PacConfig): void { } const consumerIds = new Set(); const markerValues = new Set(); + const reservedServiceAccounts = new Set(); for (const consumer of config.consumers) { if (consumerIds.has(consumer.id)) throw new Error(`${configLabel}.consumers id must be unique: ${consumer.id}`); consumerIds.add(consumer.id); @@ -765,6 +773,14 @@ function validateConfig(config: PacConfig): void { if (consumer.sourceArtifact === null) throw new Error(`${configLabel}.consumers.${consumer.id}.deliveryProvenance requires sourceArtifact ownership`); if (markerValues.has(consumer.deliveryProvenance.markerValue)) throw new Error(`${configLabel}.consumers deliveryProvenance.markerValue must be unique: ${consumer.deliveryProvenance.markerValue}`); markerValues.add(consumer.deliveryProvenance.markerValue); + if (consumer.deliveryProvenance.executionServiceAccountName === "default") { + throw new Error(`${configLabel}.consumers.${consumer.id}.deliveryProvenance.executionServiceAccountName must not reserve the shared default ServiceAccount`); + } + const serviceAccountKey = `${consumer.node.toLowerCase()}\u0000${consumer.deliveryProvenance.executionServiceAccountName}`; + if (reservedServiceAccounts.has(serviceAccountKey)) { + throw new Error(`${configLabel}.consumers deliveryProvenance.executionServiceAccountName must be unique per target: ${consumer.deliveryProvenance.executionServiceAccountName}`); + } + reservedServiceAccounts.add(serviceAccountKey); if (repository.params.service_account !== consumer.deliveryProvenance.executionServiceAccountName) { throw new Error(`${configLabel}.consumers.${consumer.id}.deliveryProvenance.executionServiceAccountName must match repository params.service_account`); } @@ -1229,6 +1245,7 @@ function remoteScript(action: "apply" | "status" | "history" | "debug-step", pac UNIDESK_PAC_DELIVERY_PROVENANCE_REQUIRED: consumer.deliveryProvenance?.required === true ? "1" : "0", UNIDESK_PAC_CONSUMER_RBAC_MANIFEST_B64: Buffer.from(renderPacConsumerRbacDesiredFragment(target.id), "utf8").toString("base64"), UNIDESK_PAC_CONSUMER_RBAC_CONFIG_SHA256: rbacIdentity.configSha256, + UNIDESK_PAC_MANAGED_EXECUTION_SERVICE_ACCOUNTS_JSON: JSON.stringify(rbacIdentity.executionServiceAccounts), UNIDESK_PAC_ADMISSION_POLICY_NAME: admissionIdentity.policyName, UNIDESK_PAC_ADMISSION_BINDING_NAME: admissionIdentity.bindingName, UNIDESK_PAC_ADMISSION_VERSION: admissionIdentity.version, @@ -1272,6 +1289,7 @@ function historyConsumerConfig(pac: PacConfig, consumer: PacConsumer): Record> = { node: binding.consumer.node, @@ -236,6 +238,25 @@ function assertBinding(config: Sub2RankConfig, target: Sub2RankTarget, binding: for (const [key, value] of Object.entries(expected)) { if (observed[key] !== value) throw new Error(`Sub2Rank PaC binding ${key}=${String(observed[key])} does not match owning YAML ${value}`); } + if (binding.consumer.deliveryProvenance?.executionServiceAccountName !== delivery.pipeline.serviceAccountName) { + throw new Error(`Sub2Rank PaC deliveryProvenance execution ServiceAccount does not match owning YAML ${delivery.pipeline.serviceAccountName}`); + } + const application = delivery.gitops.application; + if (binding.consumer.argoNamespace !== application.namespace) throw new Error(`Sub2Rank PaC argoNamespace does not match owning YAML ${application.namespace}`); + if (binding.consumer.argoApplication !== application.name) throw new Error(`Sub2Rank PaC argoApplication does not match owning YAML ${application.name}`); + const bootstrap = binding.consumer.argoBootstrap; + if (bootstrap === null) throw new Error("Sub2Rank PaC argoBootstrap must be declared"); + const expectedBootstrap = { + project: application.project, + repoUrl: delivery.gitops.readUrl, + targetRevision: delivery.gitops.branch, + path: application.path, + destinationNamespace: application.destinationNamespace, + automated: application.automated, + } as const; + for (const [key, value] of Object.entries(expectedBootstrap)) { + if (bootstrap[key as keyof typeof expectedBootstrap] !== value) throw new Error(`Sub2Rank PaC argoBootstrap.${key} does not match owning YAML ${String(value)}`); + } } function pipelineOwner(config: Sub2RankConfig, target: Sub2RankTarget): Record { From 33771c93def683182edc4e7befc9e5eae9012f58 Mon Sep 17 00:00:00 2001 From: Codex Date: Mon, 13 Jul 2026 08:41:36 +0200 Subject: [PATCH 14/14] =?UTF-8?q?docs:=20=E5=85=B3=E8=81=94=20WebProbe=20?= =?UTF-8?q?=E8=B5=84=E6=BA=90=E6=B2=BB=E7=90=86=E5=AD=90=E4=BB=BB=E5=8A=A1?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/MDTODO/hwlab-web-product-experience.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/MDTODO/hwlab-web-product-experience.md b/docs/MDTODO/hwlab-web-product-experience.md index a416b0bc..03e739a4 100644 --- a/docs/MDTODO/hwlab-web-product-experience.md +++ b/docs/MDTODO/hwlab-web-product-experience.md @@ -65,4 +65,4 @@ ### R1.13 [in_progress] -补强 web-probe 与 Web 哨兵资源治理：人工 observe start 只以 /proc/meminfo 的 MemAvailable 判定真实物理内存，<=4GiB 时先阻断并要求受控 GC 后复核；哨兵 <4GiB 时不创建 observer/Chrome，跳过本轮等待下一 cadence。将阈值、TTL、样本/截图/性能/历史/响应体上限收敛到 owning YAML，保证成功、失败、timeout、signal 全终态关闭 browser/context，并在 NC01/v03 验证 12GiB swap 不计入资格，完成任务后将详细报告写入[任务报告](./details/hwlab-web-product-experience/R1.13_Task_Report.md)。 +补强 WebProbe 全浏览器入口与 Web 哨兵资源治理（子 issue [pikasTech/unidesk#1907](https://github.com/pikasTech/unidesk/issues/1907)）：所有会创建浏览器的人工入口只以 /proc/meminfo 的 MemAvailable 判定真实物理内存，未满足 owning YAML 策略时先阻断并要求受控 GC 后复核；哨兵未满足策略时不创建 observer/Chrome，跳过本轮等待下一 cadence。将阈值、锁、TTL、样本、截图、性能、历史、响应体和终态关闭上限收敛到 owning YAML，保证成功、失败、timeout、signal 全路径关闭 browser/context，并在 NC01/v03 验证 12GiB swap 不计入资格，完成任务后将详细报告写入[任务报告](./details/hwlab-web-product-experience/R1.13_Task_Report.md)。