fix(code-queue): add issue 3 gateway diagnostics

This commit is contained in:
Codex
2026-05-19 03:32:16 +00:00
parent c00c695197
commit 9d46ca2531
18 changed files with 1494 additions and 39 deletions
+5 -3
View File
@@ -20,8 +20,8 @@ bun scripts/cli.ts codex deploy <commitId>
2. 在 D601 的 deploy cache 中通过本机 provider-gateway WS egress proxy 执行 `git fetch` remote,并用 `git archive <commitId>` 导出 tracked files 到一次性 export 目录;不得让 D601 直连 GitHub,也不得临时创建 SSH SOCKS、公网 master proxy 或 backend-core/provider-ingress fallback。
3.`rsync --delete` 同步导出的 repo 到 `/home/ubuntu/cq-deploy`,保留 `.state/``logs/``.git/``node_modules/``dist/`
4. 在 D601 用目标 Docker daemon 的本地 BuildKit builder 构建 `unidesk-code-queue:d601`,复用 D601 上已有基础镜像、inline cache 和 Code Queue build-baseprovider-gateway WS egress 是唯一允许的构建代理通道,只作为本次 build 的环境变量与 build-arg 注入,并配合本次 build 的 `--network host` 让 RUN 阶段访问 D601 宿主 loopback proxy,不能污染 D601 宿主 Docker/HTTP proxy 配置,不能新建 SSH SOCKS、公网 master proxy 或直连 fallback。
5. `docker save` 镜像并导入原生 k3s containerd`docker save unidesk-code-queue:d601 | sudo ctr --address /run/k3s/containerd/containerd.sock -n k8s.io images import -`
6. `kubectl apply -f src/components/microservices/k3sctl-adapter/k3s/code-queue.k8s.yaml`,其中包含 Code Queue`d601-tcp-egress-gateway`
5. `docker save` 镜像并导入原生 k3s containerd`docker save unidesk-code-queue:d601 | sudo ctr --address /run/k3s/containerd/containerd.sock -n k8s.io images import -`导入后必须用同一个 containerd socket 验证 `unidesk-code-queue:d601` tag 存在;D601 Docker daemon 的本地 tag 不是 k3s containerd 的 source of truth。
6. `kubectl apply -f src/components/microservices/k3sctl-adapter/k3s/code-queue.k8s.yaml`,其中包含 Code Queue`d601-provider-egress-proxy``d601-tcp-egress-gateway`。apply 后必须验证 `code-queue``code-queue-read``code-queue-write``d601-provider-egress-proxy``d601-tcp-egress-gateway` 这些 Deployment 的 container image 都是 `unidesk-code-queue:d601`,不能让 kubelet 回退到 Docker Hub 或其他外部 registry
7. 将解析后的 40 位 remote commit 写入 `deployment/code-queue``CODE_QUEUE_DEPLOY_COMMIT` / `CODE_QUEUE_DEPLOY_REQUESTED_COMMIT`,并记录到 Deployment annotation。
8. `kubectl -n unidesk rollout restart deployment/d601-tcp-egress-gateway deployment/code-queue` 并等待 rollout 完成。
9. 通过 backend-core 的真实微服务代理读取 Code Queue `/health`,强制校验 `deploy.commit` 等于本次解析出的 remote commit;如果健康的是旧服务或旧 Pod,job 必须失败。
@@ -37,7 +37,9 @@ bun scripts/cli.ts microservice health code-queue
bun scripts/cli.ts microservice proxy code-queue '/api/tasks/overview?limit=5&transcriptLimit=1&compact=1&afterSeq=0&preferId='
```
D601 原生 k3s 的人工诊断必须显式使用 host kubeconfig`KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n unidesk ...`。D601 上的默认 `kubectl` context 可能指向 Docker Desktop 或其他本地集群,不能作为 UniDesk Code Queue 部署是否 ready 的证据。部署后直接查 k3s 时,至少确认 `deployment/code-queue``code-queue-read``code-queue-write``d601-provider-egress-proxy``d601-tcp-egress-gateway` readyPod 环境中的 `UNIDESK_DEPLOY_REQUESTED_COMMIT`/`CODE_QUEUE_DEPLOY_REQUESTED_COMMIT` 等于期望 commit,并且 scheduler `/health` 暴露 PostgreSQL ready、`storage.lastError=null`、egress proxy connected 和 MiniMax `NO_PROXY` 例外。
D601 原生 k3s 的人工诊断必须显式使用 host kubeconfig`KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n unidesk ...`。D601 上的默认 `kubectl` context 可能指向 Docker Desktop 或其他本地集群,不能作为 UniDesk Code Queue 部署是否 ready 的证据。部署后直接查 k3s 时,至少确认 `deployment/code-queue``code-queue-read``code-queue-write``d601-provider-egress-proxy``d601-tcp-egress-gateway` ready两个 egress Service 的 Endpoints 非空,Pod 环境中的 `UNIDESK_DEPLOY_REQUESTED_COMMIT`/`CODE_QUEUE_DEPLOY_REQUESTED_COMMIT` 等于期望 commit,并且 scheduler `/health` 暴露 PostgreSQL ready、`storage.lastError=null`、egress proxy connected、stale active/retry_wait reconcile 状态和 MiniMax `NO_PROXY` 例外。
`bun scripts/cli.ts microservice diagnostics code-queue` 必须作为 Code Queue 的长期诊断入口。它必须显式报告 `d601-provider-egress-proxy``d601-tcp-egress-gateway` 的 Deployment available、Endpoint non-empty、scheduler 到 PostgreSQL route、以及 stale active/retry_wait reconcile;任何一项失败都应让诊断结果 degraded/failing,而不是只显示 scheduler HTTP healthy。
## Boundaries
+4
View File
@@ -163,6 +163,10 @@ The reconciler selects the executor from `config.json`:
- Control bridges that UniDesk needs in order to inspect or repair an orchestrator must stay in this direct class. In particular, `k3sctl-adapter` is a UniDesk-managed bridge to native k3s and must remain outside k3s; Docker packaging on Docker Desktop/WSL must create an explicit host-local bridge, currently an adapter-container SSH local tunnel, to reach `/etc/rancher/k3s/k3s.yaml` and WSL `127.0.0.1:6443`.
- `deployment.mode=k3sctl-managed`: the target behavior is to build on the active control target, verify native k3s on the host OS/WSL distro, import the image into native k3s/containerd, apply the existing Kubernetes manifest, stamp the Deployment and wait for rollout. On D601, persistent dev apply is currently allowed only for `backend-core` and `frontend` in `unidesk-dev`; normal production services still cannot use a maintenance-channel direct rollout. The executor must use the native kubeconfig and containerd socket, for example `/etc/rancher/k3s/k3s.yaml` and `/run/k3s/containerd/containerd.sock`; running k3s itself in Docker is forbidden for both control-plane and worker nodes. A `rancher/k3s` image or legacy container may only be used as a temporary artifact source during migration, and any active containerized k3s control plane must be stopped before verification succeeds. The executor must preload a valid `rancher/mirrored-pause:3.6` sandbox image into native k3s containerd through the provider-gateway one-shot egress path, verify its entrypoint is `/pause`, and reject fake or sleep-based replacement images. Code Queue's k3s migration executor must also stop/remove the legacy direct Docker `code-queue-backend` after k3s rollout, so there is never a second scheduler running beside the native k3s scheduler.
D601 Docker local images are not the source of truth for k3s runtime availability. For Code Queue, the deploy gate must verify `unidesk-code-queue:d601` exists in native k3s containerd after import with `ctr --address /run/k3s/containerd/containerd.sock -n k8s.io images ls`, and it must fail before rollout if the tag is missing. The same gate must verify every production Code Queue Deployment that uses the image (`code-queue`, `code-queue-read`, `code-queue-write`, `d601-provider-egress-proxy`, `d601-tcp-egress-gateway`) still references exactly `unidesk-code-queue:d601`; otherwise kubelet may attempt an external registry pull and leave base gateways in `ImagePullBackOff`.
Code Queue health and diagnostics must cover its k3s dependencies, not only scheduler HTTP health. `bun scripts/cli.ts microservice diagnostics code-queue` and the `/health` aggregation must mark the service degraded/failing when `d601-provider-egress-proxy` or `d601-tcp-egress-gateway` Deployment availability or Endpoint readiness is missing, when the scheduler reports `storage.lastError` or PostgreSQL route failure through `d601-tcp-egress-gateway.unidesk.svc.cluster.local:15432`, or when stale active/retry_wait reconcile reports recoverable active tasks without a local run.
Existing service-specific commands such as Code Queue deploy are disabled as direct D601 deploy paths. Their build/import/rollout semantics should converge later into one controlled target-side deployment path instead of keeping parallel implementations.
Decision Center is a standard `k3sctl-managed` service in this model, but D601 maintenance-channel direct apply must not deploy it. Future controlled CD for Decision Center should build `src/components/microservices/decision-center/Dockerfile` on D601, import `unidesk-decision-center:d601` into native k3s containerd, apply `src/components/microservices/k3sctl-adapter/k3s/decision-center.k8s.yaml`, stamp the Deployment, and verify health through `/api/microservices/decision-center/health`. It must not add a main-server Compose service, NodePort, hostPort, or provider-gateway direct HTTP backend for Decision Center.