From 8ab0762ed604b132718f3ba348cea356593f529d Mon Sep 17 00:00:00 2001 From: Codex Date: Sat, 16 May 2026 17:49:21 +0000 Subject: [PATCH] fix: preload native k3s pause image --- TEST.md | 2 +- docs/reference/deploy.md | 2 +- scripts/src/deploy.ts | 37 ++++++++++++++++++++++++++++--------- 3 files changed, 30 insertions(+), 11 deletions(-) diff --git a/TEST.md b/TEST.md index 62c89c77..2d51190d 100644 --- a/TEST.md +++ b/TEST.md @@ -103,7 +103,7 @@ ## T23 D601 Code Queue User Service -阅读 `AGENTS.md`（本项目 `AGENTS.md` 同时承担 `SKILL.md` 对 `scripts/cli.ts` 的解释职责），然后用 cli 手动测试以下内容：运行 `bun scripts/cli.ts microservice list`，确认 `code-queue` 显示为 `providerId=D601`、`public=false`、`frontendOnly=true`、仓库 URL `https://github.com/pikasTech/unidesk`、k3s/k8s `k3s://unidesk/code-queue:4222` 逻辑服务映射、`deployment.mode=k3sctl-managed`、`runtime.orchestrator=k3sctl` 且无业务直连容器摘要；使用 `bun scripts/cli.ts codex deploy <已push的commitId>` 重建/启动 D601 Code Queue，确认命令立即返回异步 job id，`bun scripts/cli.ts job status --tail-bytes 30000` 能看到 fetch/export、rsync、Docker build、k3s image import、kubectl apply、部署 commit 戳记、rollout、legacy direct cleanup 和 health commit 验证进度，并确认 job 最终校验真实 Code Queue `/health` 返回的 `deploy.commit` 精确匹配本次 remote commit，不能由旧服务或旧 Pod 充数；同时确认主 server 根目录 `docker-compose.yml` 中不再存在 `code-queue` service，并通过 `bun scripts/cli.ts ssh D601 argv bash -lc 'systemctl is-active k3s && KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl get nodes -o wide && ! docker ps --format "{{.Names}} {{.Image}}" | grep -E "[[:space:]]rancher/k3s:" && ! docker ps --format "{{.Names}}" | grep -Fx code-queue-backend'` 或等价检查证明 D601 k3s 是 WSL 原生 systemd 服务、没有 active `rancher/k3s` 控制面容器且旧 direct Docker `code-queue-backend` 没有并行运行。运行 `bun scripts/cli.ts microservice health code-queue`、`bun scripts/cli.ts microservice proxy code-queue /api/dev-ready --raw`、`bun scripts/cli.ts microservice proxy code-queue '/api/tasks/overview?limit=5&transcriptLimit=1&compact=1&afterSeq=0&preferId='` 和 `bun scripts/cli.ts codex task <已有taskId>`，确认链路通过 backend-core、k3sctl-adapter、Kubernetes API service proxy 和 D601 active Code Queue Service，且 task id 查询返回初始 prompt、最后 assistant message、工具调用摘要、attempt/judge/error 和耗时，`queue.storage.primary=postgres`、`queue.storage.postgresReady=true`、`queue.devReady.missingTools=[]`、`queue.devReady.docker.versionOk=true`、`queue.devReady.docker.composeOk=true`；`queue.devReady.ssh.ready` 只在需要跨 Provider SSH/Windows-native 任务时作为强制项。在 D601 active Code Queue Pod 内验证主 PostgreSQL 端口映射可执行 `select 1`，主 OA Event Flow 端口映射 `/health` 可访问，本机 ClaudeQQ `http://host.docker.internal:3290/health` 可访问；这些映射不得成为任意公网入口。 +阅读 `AGENTS.md`（本项目 `AGENTS.md` 同时承担 `SKILL.md` 对 `scripts/cli.ts` 的解释职责），然后用 cli 手动测试以下内容：运行 `bun scripts/cli.ts microservice list`，确认 `code-queue` 显示为 `providerId=D601`、`public=false`、`frontendOnly=true`、仓库 URL `https://github.com/pikasTech/unidesk`、k3s/k8s `k3s://unidesk/code-queue:4222` 逻辑服务映射、`deployment.mode=k3sctl-managed`、`runtime.orchestrator=k3sctl` 且无业务直连容器摘要；使用 `bun scripts/cli.ts codex deploy <已push的commitId>` 重建/启动 D601 Code Queue，确认命令立即返回异步 job id，`bun scripts/cli.ts job status --tail-bytes 30000` 能看到 fetch/export、rsync、Docker build、native k3s provider egress proxy、有效 `rancher/mirrored-pause:3.6` sandbox 镜像导入、k3s image import、kubectl apply、部署 commit 戳记、rollout、legacy direct cleanup 和 health commit 验证进度，并确认 job 最终校验真实 Code Queue `/health` 返回的 `deploy.commit` 精确匹配本次 remote commit，不能由旧服务或旧 Pod 充数；同时确认主 server 根目录 `docker-compose.yml` 中不再存在 `code-queue` service，并通过 `bun scripts/cli.ts ssh D601 argv bash -lc 'systemctl is-active k3s && KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl get nodes -o wide && sudo ctr --address /run/k3s/containerd/containerd.sock -n k8s.io images ls | grep -F docker.io/rancher/mirrored-pause:3.6 && ! docker ps --format "{{.Names}} {{.Image}}" | grep -E "[[:space:]]rancher/k3s:" && ! docker ps --format "{{.Names}}" | grep -Fx code-queue-backend'` 或等价检查证明 D601 k3s 是 WSL 原生 systemd 服务、native containerd 已有正确 pause sandbox 镜像、没有 active `rancher/k3s` 控制面容器且旧 direct Docker `code-queue-backend` 没有并行运行。运行 `bun scripts/cli.ts microservice health code-queue`、`bun scripts/cli.ts microservice proxy code-queue /api/dev-ready --raw`、`bun scripts/cli.ts microservice proxy code-queue '/api/tasks/overview?limit=5&transcriptLimit=1&compact=1&afterSeq=0&preferId='` 和 `bun scripts/cli.ts codex task <已有taskId>`，确认链路通过 backend-core、k3sctl-adapter、Kubernetes API service proxy 和 D601 active Code Queue Service，且 task id 查询返回初始 prompt、最后 assistant message、工具调用摘要、attempt/judge/error 和耗时，`queue.storage.primary=postgres`、`queue.storage.postgresReady=true`、`queue.devReady.missingTools=[]`、`queue.devReady.docker.versionOk=true`、`queue.devReady.docker.composeOk=true`；`queue.devReady.ssh.ready` 只在需要跨 Provider SSH/Windows-native 任务时作为强制项。在 D601 active Code Queue Pod 内验证主 PostgreSQL 端口映射可执行 `select 1`，主 OA Event Flow 端口映射 `/health` 可访问，本机 ClaudeQQ `http://host.docker.internal:3290/health` 可访问；这些映射不得成为任意公网入口。 随后登录公网 frontend `http://74.48.78.17:18081/`，进入 `用户服务 / Code Queue`，确认页面显示默认模型 `gpt-5.5`、默认执行 Provider `D601`、默认工作目录 `/workspace`、模型下拉菜单包含 `gpt-5.4-mini`/`gpt-5.4`/`gpt-5.5`、入队份数、队列指标、任务 ID、复制任务 ID、引用按钮、任务耗时、引用任务 ID、清空输入、创建成功提示、任务提交表单、Trace 输出、attempt 表、MiniMax/fallback judge 状态、追加 prompt、打断和重试控件；通过页面提交一个小任务，确认任务进入 queued/running/succeeded 或可解释的 failed 状态，并且输出区能看到运行中的 Codex 消息。批量验收时设置 `入队份数=5` 或用 `---` 分隔 5 段 prompt，一次性入队 5 条任务，确认 5 条任务按顺序运行并全部进入 succeeded 或可解释的非成功终态，不能只运行第一条后停止；其中任一任务被 judge 判定 `fail` 时只能把当前任务标为 failed，后续 queued 任务仍必须继续推进。测试异常中断时可以提交长任务后点击 `打断`，确认任务变为 canceled 或被 judge 标记为非成功终态；自动重试只应在服务端/传输异常、任务正常结束但 execution record 显示未完成、或 judge 判定 retry 时发生；retry 必须复用已有 Codex thread 并 append 继续执行 prompt，只有当前任务 complete 后才推进队列中的下一个任务。MiniMax judge 必须能处理 Markdown fence/夹杂文本等 JSON 去噪；若去噪后仍失败，必须把解析错误和上一轮去噪前原始回答反馈给 MiniMax 修复后重试，日志中应出现 `judge_json_parse_retry`，且 repair 成功时仍以 `source=minimax` 返回。Codex provider key 只能通过 `OPENAI_API_KEY`、`CRS_OAI_KEY` 这类运行时环境透传，MiniMax API key 只能通过 D601 env-file 运行时环境传入，禁止写入 `config.json`、Dockerfile、源码或测试文档。 diff --git a/docs/reference/deploy.md b/docs/reference/deploy.md index b5a5f911..a1d24695 100644 --- a/docs/reference/deploy.md +++ b/docs/reference/deploy.md @@ -69,7 +69,7 @@ The reconciler selects the executor from `config.json`: - `deployment.mode=unidesk-direct` on `main-server`: build the image on the main server, then use the fixed UniDesk Compose project and `up -d --no-build --no-deps --force-recreate `. - `deployment.mode=unidesk-direct` on a provider: dispatch `host.ssh` to that provider, build on the provider, then use the service's provider-local compose file and project. The executor resolves the actual Compose project, image name, build context, Dockerfile and target from the running container labels and `docker compose config`; it must not guess an image tag that the service will not actually run. - Control bridges that UniDesk needs in order to inspect or repair an orchestrator must stay in this direct class. In particular, `k3sctl-adapter` is a UniDesk-managed bridge to native k3s and must remain outside k3s; Docker packaging on Docker Desktop/WSL must create an explicit host-local bridge, currently an adapter-container SSH local tunnel, to reach `/etc/rancher/k3s/k3s.yaml` and WSL `127.0.0.1:6443`. -- `deployment.mode=k3sctl-managed`: dispatch to the active control target, build on that target, verify or install native k3s on the host OS/WSL distro, import the image into native k3s/containerd, apply the existing Kubernetes manifest, stamp the Deployment and wait for rollout. The executor must use the native kubeconfig and containerd socket, for example `/etc/rancher/k3s/k3s.yaml` and `/run/k3s/containerd/containerd.sock`; running k3s itself in Docker is forbidden for both control-plane and worker nodes. A `rancher/k3s` image or legacy container may only be used as a temporary artifact source during migration, and any active containerized k3s control plane must be stopped before verification succeeds. Code Queue's k3s migration executor must also stop/remove the legacy direct Docker `code-queue-backend` after k3s rollout, so there is never a second scheduler running beside the native k3s scheduler. +- `deployment.mode=k3sctl-managed`: dispatch to the active control target, build on that target, verify or install native k3s on the host OS/WSL distro, import the image into native k3s/containerd, apply the existing Kubernetes manifest, stamp the Deployment and wait for rollout. The executor must use the native kubeconfig and containerd socket, for example `/etc/rancher/k3s/k3s.yaml` and `/run/k3s/containerd/containerd.sock`; running k3s itself in Docker is forbidden for both control-plane and worker nodes. A `rancher/k3s` image or legacy container may only be used as a temporary artifact source during migration, and any active containerized k3s control plane must be stopped before verification succeeds. The executor must preload a valid `rancher/mirrored-pause:3.6` sandbox image into native k3s containerd through the provider-gateway one-shot egress path, verify its entrypoint is `/pause`, and reject fake or sleep-based replacement images. Code Queue's k3s migration executor must also stop/remove the legacy direct Docker `code-queue-backend` after k3s rollout, so there is never a second scheduler running beside the native k3s scheduler. Existing service-specific commands such as Code Queue deploy should converge onto this reconciler path instead of keeping a parallel implementation. diff --git a/scripts/src/deploy.ts b/scripts/src/deploy.ts index 5ef8382e..f5387056 100644 --- a/scripts/src/deploy.ts +++ b/scripts/src/deploy.ts @@ -784,6 +784,19 @@ function ensureNativeK3sScript(): string { ...rootAccessPrelude(), `native_k3s_image=${shellQuote(nativeK3sImage)}`, `native_ctr_address=${shellQuote(nativeK3sCtrAddress)}`, + `build_proxy=${shellQuote(providerGatewayWsEgressProxyUrl)}`, + "configure_provider_deploy_proxy() {", + " export DOCKER_CONFIG=/tmp/unidesk-docker-config-clean", + " mkdir -p \"$DOCKER_CONFIG\"", + " [ -f \"$DOCKER_CONFIG/config.json\" ] || printf '{}' > \"$DOCKER_CONFIG/config.json\"", + " export HTTP_PROXY=\"$build_proxy\" HTTPS_PROXY=\"$build_proxy\" ALL_PROXY=\"$build_proxy\"", + " export NO_PROXY=\"localhost,127.0.0.1,::1,host.docker.internal\"", + " if ! curl -fsSI --max-time 20 -x \"$build_proxy\" https://github.com >/dev/null; then", + " echo native_k3s_provider_egress_proxy_unavailable=$build_proxy >&2", + " exit 1", + " fi", + " echo native_k3s_provider_egress_proxy=provider-gateway-ws-egress:$build_proxy", + "}", "install_native_k3s_binaries() {", " missing=0", " for binary in k3s ctr containerd containerd-shim-runc-v2 crictl runc cni flannel bridge host-local loopback portmap bandwidth firewall; do", @@ -821,18 +834,24 @@ function ensureNativeK3sScript(): string { " fi", "}", "install_pause_image() {", - " if root_exec ctr --address \"$native_ctr_address\" -n k8s.io images ls | grep -q 'docker.io/rancher/mirrored-pause:3.6'; then return; fi", - " docker image inspect rancher/mirrored-pause:3.6 >/dev/null 2>&1 || {", - " docker image inspect \"$native_k3s_image\" >/dev/null 2>&1 || docker pull \"$native_k3s_image\"", - " tmp_dir=$(mktemp -d)", - " printf '%s\\n' \"FROM $native_k3s_image\" 'ENTRYPOINT [\"/bin/sleep\", \"365d\"]' > \"$tmp_dir/Dockerfile\"", - " docker build -q -t rancher/mirrored-pause:3.6 \"$tmp_dir\" >/dev/null", - " rm -rf \"$tmp_dir\"", - " }", - " docker save rancher/mirrored-pause:3.6 -o /tmp/unidesk-k3s-pause.tar", + " pause_image=rancher/mirrored-pause:3.6", + " pause_entrypoint=$(docker image inspect \"$pause_image\" --format '{{json .Config.Entrypoint}}' 2>/dev/null || true)", + " if ! printf '%s' \"$pause_entrypoint\" | grep -q '\"/pause\"'; then", + " docker image rm \"$pause_image\" >/dev/null 2>&1 || true", + " docker pull --platform linux/amd64 \"$pause_image\"", + " pause_entrypoint=$(docker image inspect \"$pause_image\" --format '{{json .Config.Entrypoint}}' 2>/dev/null || true)", + " fi", + " if ! printf '%s' \"$pause_entrypoint\" | grep -q '\"/pause\"'; then", + " echo native_k3s_pause_image_invalid_entrypoint=$pause_entrypoint >&2", + " exit 1", + " fi", + " docker save \"$pause_image\" -o /tmp/unidesk-k3s-pause.tar", + " root_exec ctr --address \"$native_ctr_address\" -n k8s.io images rm docker.io/rancher/mirrored-pause:3.6 >/dev/null 2>&1 || true", " root_exec ctr --address \"$native_ctr_address\" -n k8s.io images import /tmp/unidesk-k3s-pause.tar >/dev/null", + " root_exec ctr --address \"$native_ctr_address\" -n k8s.io images ls | grep -F 'docker.io/rancher/mirrored-pause:3.6' >/dev/null", " echo native_k3s_imported_pause_image=rancher/mirrored-pause:3.6", "}", + "configure_provider_deploy_proxy", "install_native_k3s_binaries", "unmount_invalid_wsl_kubelet_mounts", "if ! systemctl cat k3s.service >/dev/null 2>&1; then",