From b605c78875e997b00c09c5e26bcdd2838691cacb Mon Sep 17 00:00:00 2001 From: Codex Date: Mon, 18 May 2026 00:53:11 +0000 Subject: [PATCH] feat: enable dev core deploy apply --- AGENTS.md | 2 +- docs/plan/d601-k3s-dev-environment.md | 3 +- docs/reference/cli.md | 2 +- docs/reference/deploy.md | 10 +- scripts/cli.ts | 10 +- scripts/src/deploy.ts | 277 ++++++++++++++++-- .../k3s/dev/unidesk-dev-core.k8s.yaml | 46 ++- 7 files changed, 312 insertions(+), 38 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index f6c04e32..176566a5 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -36,7 +36,7 @@ UniDesk 是一个以主 server 为统一入口的分布式工作平台；本文 - `bun scripts/cli.ts ssh [ssh-like args...]`：通过 provider-gateway 的 Host SSH / WSL SSH 维护桥打开近似原生 ssh 的交互会话或远端命令，并在远端 PATH 注入 `apply_patch`、`glob` 与 `skill-discover`；`apply-patch`、`py`、`skills`、结构化 `find`、`glob` 和 `argv` 子命令用于避免远端补丁、Python stdin、skill 发现与常用只读命令的嵌套转义问题，使用规则见 `docs/reference/cli.md` 和 `docs/reference/provider-gateway.md`。 - `bun scripts/cli.ts microservice list/status/health/diagnostics/tunnel-self-test/proxy`：管理和验证挂载在主 server、计算节点 Docker 或 k3s 控制面上的用户服务，`proxy` 支持受控 JSON body，OA Event Flow/Todo Note/Baidu Netdisk/Code Queue Manager on main-server、k3s Control/Code Queue 执行面/MDTODO/Decision Center/FindJob/Pipeline/MET Nonlinear on D601 的规则见 `docs/reference/microservices.md`。 - `bun scripts/cli.ts decision upload/list/show/health`：通过 backend-core 用户服务代理上传会议记录/决议 Markdown、列出记录和查看详情；Decision Center 运行在 D601 k3s，规则见 `docs/reference/microservices.md`。 -- `bun scripts/cli.ts deploy check/plan/apply [--file deploy.json] [--service ]`：按根目录 `deploy.json` 的服务 repo 和 commit 期望状态校验或更新用户服务，目标侧自行 fetch、构建、部署和 live commit 验证；规则见 `docs/reference/deploy.md`。 +- `bun scripts/cli.ts deploy check/plan/apply [--file deploy.json|--env dev|prod] [--service ]`：按根目录或固定环境 ref 的服务 repo 和 commit 期望状态校验或更新用户服务；`--env dev` 当前只开放 backend-core/frontend 开发环境部署，目标侧自行 fetch、构建、部署和 live commit 验证；规则见 `docs/reference/deploy.md`。 - `bun scripts/cli.ts dev-env validate [--manifest path] [--kubectl-dry-run]` / `dev-env prewarm-images`：离线校验 D601 `unidesk-dev` 生产隔离护栏，或把开发底座基础镜像预热到 D601 原生 k3s containerd，规则见 `docs/reference/deploy.md` 与 `docs/reference/microservices.md`。 - `bun scripts/cli.ts ci install/status/run/logs`：在 D601 原生 k3s 上安装和运行 Tekton CI，只做每 commit 检查和 Code Queue 只读性能门禁，不部署 CD；规则见 `docs/reference/ci.md`。 - `bun scripts/cli.ts codex deploy `：Code Queue 兼容部署入口，会生成临时 desired manifest 并调用 `deploy apply --service code-queue` 的同一条 target-side build 与 live commit 验证路径；规则见 `docs/reference/codex-deploy.md`。 diff --git a/docs/plan/d601-k3s-dev-environment.md b/docs/plan/d601-k3s-dev-environment.md index de35fdd1..11bdf81f 100644 --- a/docs/plan/d601-k3s-dev-environment.md +++ b/docs/plan/d601-k3s-dev-environment.md @@ -190,6 +190,7 @@ - dev frontend `/health` 返回 ok，并且只代理到 dev backend-core。 - dev backend/frontend 重部署期间，生产 `bun scripts/cli.ts server status` 仍健康。 - 重建 dev backend/frontend 不触碰主 server Docker Compose 容器。 +- 当前实施切片：`deploy apply --env dev --service backend-core|frontend` 先支持 dev core 两个服务；`deploy/dev` 只保存 `deploy.json`，不保存 k8s manifest 或源码。 ## 阶段 4：code-queue-mgr-dev @@ -342,4 +343,4 @@ 5. Phase 7：把生产部署迁移到 `deploy/prod`。 6. Phase 8：强化操作员和 LLM 安全检查。 -第一个里程碑完成条件：`deploy apply --env dev` 可以根据 `origin/deploy/dev:deploy.json` 声明的 commit id，把 backend-core、frontend、code-queue-mgr 以及 Code Queue read/write/scheduler 部署进 `unidesk-dev`；反复 dev redeploy 不改变生产主 server status，也不改变生产 Code Queue state。 +第一个完整里程碑完成条件：`deploy apply --env dev` 可以根据 `origin/deploy/dev:deploy.json` 声明的 commit id，把 backend-core、frontend、code-queue-mgr 以及 Code Queue read/write/scheduler 部署进 `unidesk-dev`；反复 dev redeploy 不改变生产主 server status，也不改变生产 Code Queue state。阶段 3 的阶段性里程碑先以 `--service backend-core` 和 `--service frontend` 分别部署成功为准。 diff --git a/docs/reference/cli.md b/docs/reference/cli.md index 6529a6fd..dcc97d85 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -21,7 +21,7 @@ UniDesk 的统一 CLI 入口是根目录 `scripts/cli.ts`，运行方式固定 - `ssh skills [--scope all|wsl|windows] [--limit N]` 发现目标节点上的 WSL/Linux skill 根目录；当 provider 是 WSL 时同一次调用还会扫描 Windows 用户目录下的 `.agents/skills` 与 `.codex/skills`。 - `microservice list/status/health/diagnostics/tunnel-self-test/proxy` 通过 backend-core 内网 API 管理挂载在计算节点 Docker 或 k3s 控制面中的用户服务（底层命令名仍为 microservice）；`health`、`diagnostics`、`tunnel-self-test` 和 `proxy` 会走真实 backend-core -> provider-gateway 或 k3sctl-adapter -> 节点服务链路，`proxy` 支持受控 JSON 请求体并对超大响应 body 默认输出有界预览，规则见 `docs/reference/microservices.md`。 - `decision upload/list/show/health` 通过 backend-core 用户服务代理访问 D601 k3s Decision Center，用于上传会议记录/决议 Markdown、列出权威记录、查看详情和健康检查；它不得直连 D601 Service、NodePort 或 provider-gateway 业务 HTTP。 -- `deploy check/plan/apply` 默认从根目录 `deploy.json` 读取服务 repo 与 commit 期望状态，join `config.json` 和现有 manifest 后使用 target-side build 单一路径校验或更新直管服务与 k3s 代管服务；`deploy plan --env dev|prod` 在 Phase 0 只从固定 Git ref 读取 manifest 并输出 dry-run 环境计划，不使用本地 dirty worktree；规则见 `docs/reference/deploy.md`。 +- `deploy check/plan/apply` 默认从根目录 `deploy.json` 读取服务 repo 与 commit 期望状态，join `config.json` 和现有 manifest 后使用 target-side build 单一路径校验或更新直管服务与 k3s 代管服务；`deploy plan --env dev|prod` 只从固定 Git ref 读取 manifest 并输出 dry-run 环境计划，不使用本地 dirty worktree；`deploy apply --env dev --service backend-core|frontend` 可按 `origin/deploy/dev:deploy.json` 部署第一版 dev core，`--env prod` apply 仍禁用；规则见 `docs/reference/deploy.md`。 - `dev-env validate [--manifest path] [--kubectl-dry-run]` 离线校验 D601 `unidesk-dev` namespace、dev PostgreSQL 底座和 dev backend/frontend manifest。默认检查 `src/components/microservices/k3sctl-adapter/k3s/dev/unidesk-dev-foundation.k8s.yaml`；也可显式校验 `src/components/microservices/k3sctl-adapter/k3s/dev/unidesk-dev-core.k8s.yaml`。所有 namespaced 对象必须只落到 `unidesk-dev`，foundation manifest 必须包含 `postgres-dev` StatefulSet/Service、dev secret/config、迁移 Job 和 DB URL guard，core manifest 必须包含 `backend-core-dev`/`frontend-dev` Deployment/Service。加 `--kubectl-dry-run` 时额外执行 `kubectl apply --dry-run=client --validate=false -f `，仍不 apply 资源。 - `dev-env prewarm-images [--image image] [--provider-id D601] [--no-pull] [--proxy-url URL] [--pull-timeout-ms N] [--dry-run]` 创建异步 job，通过 UniDesk SSH 维护桥在 D601 上把开发底座依赖镜像从 Docker 缓存导入原生 k3s containerd。默认镜像是 `postgres:16-alpine` 和 `rancher/mirrored-library-busybox:1.36.1`，用于避免 `postgres-dev` 与 local-path helper pod 卡在外部 registry 拉取。该命令固定验证 `/etc/rancher/k3s/k3s.yaml` 指向的 native k3s 上下文，并输出 `dev_env_containerd_image_ready=...` 作为成功判据；它不 apply manifest、不修改生产 `unidesk` namespace。 - `codex deploy ` 是 Code Queue 兼容部署入口，会生成临时 desired manifest 并调用 `deploy apply --service code-queue` 的同一条 target-side build、k3s import、rollout 和 live commit 验证路径；详细规则见 `docs/reference/codex-deploy.md`。 diff --git a/docs/reference/deploy.md b/docs/reference/deploy.md index 6951e3da..20cd05a6 100644 --- a/docs/reference/deploy.md +++ b/docs/reference/deploy.md @@ -29,7 +29,9 @@ Environment mode never reads the local working tree manifest. The mapping is fix - `dev -> origin/deploy/dev` - `prod -> origin/deploy/prod` -The current Phase 0 implementation enables only dry-run `check` and `plan` for `--env`. It fetches the fixed ref, reads `deploy.json` from that ref, validates the declared environment, and reports the manifest commit/blob, service commit IDs, target namespace, database fingerprint and Provider identity. `deploy apply --env ...` is intentionally rejected until the dev infrastructure executors exist. +`deploy check --env ...` and `deploy plan --env ...` fetch the fixed ref, read `deploy.json` from that ref, validate the declared environment, and report the manifest commit/blob, service commit IDs, target namespace, database fingerprint and Provider identity without mutating runtime resources. `deploy apply --env dev` is enabled only for the first dev-core slice, currently `backend-core` and `frontend`. If no `--service` is given and the dev manifest still includes unsupported later-stage services such as Code Queue, the command fails before changing runtime resources. `deploy apply --env prod` remains disabled until the production environment executor and authorization policy are explicitly added. + +The `deploy/dev` and `deploy/prod` branches are environment desired-state branches, not source branches. They should contain only `deploy.json`; Kubernetes manifests, Dockerfiles and executor code continue to live on `master` and are selected through the commit IDs declared in the environment manifest. `config.json.microservices[].repository.commitId` is retained for catalog compatibility, but `deploy.json` is the deployment version authority for the reconciler. @@ -60,7 +62,7 @@ Phase 3 introduces the dev backend/frontend manifest at `src/components/microser `backend-core-dev` must use `unidesk-dev-runtime-config` and `unidesk-dev-runtime-secrets`, connect to `postgres-dev.../unidesk_dev`, expose HTTP on 8080 and provider ingress on 8081, and write logs under `/var/log/unidesk-dev`. `frontend-dev` must set `CORE_INTERNAL_URL=http://backend-core-dev.unidesk-dev.svc.cluster.local:8080` and must not proxy to production backend-core. -The manifest uses placeholder image tags and deploy commit values until `deploy apply --env dev` supports target-side dev builds. A controller or operator must replace those placeholders from `origin/deploy/dev:deploy.json` before real rollout. Client dry-run and static validation are the required checks before any controlled apply: +The manifest keeps placeholder image tags and deploy commit values in source control. `deploy apply --env dev --service backend-core|frontend` fetches `origin/deploy/dev:deploy.json`, materializes the requested source commit on D601, copies the dev core control manifest, narrows it to the selected Service/Deployment pair, replaces placeholders with the requested commit and dev image tag, builds on D601, imports the image into native k3s containerd, applies only the `unidesk-dev` objects and stamps the Deployment. Client dry-run and static validation are the required checks before any controlled apply: - `bun scripts/cli.ts dev-env validate --manifest src/components/microservices/k3sctl-adapter/k3s/dev/unidesk-dev-core.k8s.yaml` - `KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl apply --dry-run=client --validate=false -f src/components/microservices/k3sctl-adapter/k3s/dev/unidesk-dev-core.k8s.yaml` @@ -73,9 +75,9 @@ backend-core and frontend keep their production health payload shape by default. `bun scripts/cli.ts deploy plan [--file deploy.json] [--service ]` prints the same live state plus the intended action: `noop`, `deploy` or `unsupported`. -`bun scripts/cli.ts deploy plan --env dev [--service ]` reads `origin/deploy/dev:deploy.json` and prints a dry-run environment plan without checking or mutating live runtime resources. `deploy check --env dev` uses the same Phase 0 dry-run path. `--env prod` is available for parity but is also dry-run only in Phase 0; it reads `origin/deploy/prod:deploy.json` and must not use a dirty local `deploy.json`. +`bun scripts/cli.ts deploy plan --env dev [--service ]` reads `origin/deploy/dev:deploy.json` and prints a dry-run environment plan without checking or mutating live runtime resources. `deploy check --env dev` uses the same dry-run environment plan. `--env prod` is available for parity as a dry-run planning path; it reads `origin/deploy/prod:deploy.json` and must not use a dirty local `deploy.json`. -`bun scripts/cli.ts deploy apply [--file deploy.json] [--service ] [--dry-run] [--force]` starts an asynchronous job. Use `bun scripts/cli.ts job status --tail-bytes 30000` to observe progress. `--dry-run` resolves the same plan but does not build or replace runtime objects. `--force` rebuilds even when the live commit matches. +`bun scripts/cli.ts deploy apply [--file deploy.json | --env dev] [--service ] [--dry-run] [--force]` starts an asynchronous job. Use `bun scripts/cli.ts job status --tail-bytes 30000` to observe progress. `--dry-run` resolves the same plan but does not build or replace runtime objects. `--force` rebuilds even when the live commit matches. Environment apply is currently limited to `--env dev --service backend-core` and `--env dev --service frontend`; `--env prod` apply is rejected. All deploy commands output JSON. Long operations must use `.state/jobs/` and bounded log tails; no deploy path may succeed with missing progress output. diff --git a/scripts/cli.ts b/scripts/cli.ts index 33147f19..dcdea905 100644 --- a/scripts/cli.ts +++ b/scripts/cli.ts @@ -54,7 +54,7 @@ function help(): unknown { { command: "decision upload [--title text] [--type meeting|decision] [--level G0|G1|G2|G3|P0|P1|P2|P3|none] [--status active|blocked|parked|done] [--linked-goal-id id] [--evidence url]", description: "Upload a meeting note or decision record through backend-core -> decision-center user-service proxy." }, { command: "decision list [--type ...] [--status ...] [--level ...] [--linked-goal-id id] [--limit N]", description: "List Decision Center records through the user-service proxy." }, { command: "decision show ", description: "Show one Decision Center record." }, - { command: "deploy check|plan|apply [--file deploy.json|--env dev|prod] [--service id] [--dry-run] [--force]", description: "Reconcile services from a repo+commit manifest; --env uses fixed environment refs for dry-run planning in Phase 0." }, + { command: "deploy check|plan|apply [--file deploy.json|--env dev|prod] [--service id] [--dry-run] [--force]", description: "Reconcile services from a repo+commit manifest; --env reads fixed environment refs and can apply supported dev services." }, { command: "dev-env validate|prewarm-images", description: "Validate D601 unidesk-dev guardrails or prewarm dev foundation images into native k3s containerd through a bounded async job." }, { command: "schedule list|get|runs|run|delete", description: "Manage backend-core scheduled tasks and run history; schedule run supports --wait-ms N." }, { command: "schedule upsert-pgdata-backup [--time HH:MM] [--remote-base /SERVER_DATA/UNIDESK_PG_DATA]", description: "Create or update the daily PGDATA physical backup task that uploads monthly rotated archives to Baidu Netdisk." }, @@ -149,14 +149,6 @@ async function main(): Promise { return; } - if (top === "deploy" && (args.includes("--env") || args.includes("--environment"))) { - const result = await runDeployCommand(null, args.slice(1)); - const ok = (result as { ok?: unknown }).ok !== false; - emitJson(commandName, result, ok); - if (!ok) process.exitCode = 1; - return; - } - if (top === "dev-env") { const result = runDevEnvCommand(args.slice(1)); const ok = (result as { ok?: unknown }).ok !== false; diff --git a/scripts/src/deploy.ts b/scripts/src/deploy.ts index 0ccdad14..5acd0f56 100644 --- a/scripts/src/deploy.ts +++ b/scripts/src/deploy.ts @@ -131,6 +131,7 @@ const nativeK3sInstallVersion = "v1.34.1+k3s1"; const nativeK3sImage = "rancher/k3s:v1.34.1-k3s1"; const nativeK3sCtrAddress = "/run/k3s/containerd/containerd.sock"; const unideskRepoUrl = "https://github.com/pikasTech/unidesk"; +const devApplySupportedServiceIds = new Set(["backend-core", "frontend"]); const deployEnvironmentTargets: Record = { dev: { environment: "dev", @@ -184,7 +185,7 @@ function deployHelp(action: string | undefined = undefined): Record", default: defaultDeployFile, description: "Desired-state manifest path relative to the repo root. JSON and ESM JS manifests are supported, for example deploy.json or develop.js." }, - { name: "--env ", description: "Read deploy.json from the fixed environment ref: dev=origin/deploy/dev, prod=origin/deploy/prod. Phase 0 supports check/plan only." }, + { name: "--env ", description: "Read deploy.json from the fixed environment ref: dev=origin/deploy/dev, prod=origin/deploy/prod. Apply is currently enabled for supported dev services only." }, { name: "--service ", description: "Limit reconcile to one service from the manifest." }, { name: "--dry-run", description: "Prepare and validate without mutating the target service." }, { name: "--force", description: "Redeploy even when the live commit appears up to date." }, @@ -567,7 +568,96 @@ function frontendCoreDeployService(config: UniDeskConfig): UniDeskMicroserviceCo }; } -function coreDeployService(config: UniDeskConfig, id: string): UniDeskMicroserviceConfig | undefined { +function devCoreDeployService(id: string): UniDeskMicroserviceConfig | undefined { + const specs: Record = { + "backend-core": { + name: "UniDesk Dev Backend Core", + description: "Isolated dev backend-core deployed into D601 native k3s namespace unidesk-dev.", + dockerfile: "src/components/backend-core/Dockerfile", + composeService: "backend-core-dev", + containerName: "k3s:backend-core-dev", + nodeBaseUrl: "k3s://backend-core-dev", + nodePort: 8080, + healthPath: "/health", + route: "/dev/backend-core", + allowedMethods: ["GET", "HEAD", "POST", "PUT", "PATCH", "DELETE"], + allowedPathPrefixes: ["/", "/api/", "/logs"], + }, + frontend: { + name: "UniDesk Dev Frontend", + description: "Isolated dev frontend deployed into D601 native k3s namespace unidesk-dev.", + dockerfile: "src/components/frontend/Dockerfile", + composeService: "frontend-dev", + containerName: "k3s:frontend-dev", + nodeBaseUrl: "k3s://frontend-dev", + nodePort: 8080, + healthPath: "/health", + route: "/dev/frontend", + allowedMethods: ["GET", "HEAD"], + allowedPathPrefixes: ["/"], + }, + }; + const spec = specs[id]; + if (spec === undefined) return undefined; + return { + id, + name: spec.name, + providerId: "D601", + description: spec.description, + repository: { + url: unideskRepoUrl, + commitId: "deploy-dev", + dockerfile: spec.dockerfile, + composeFile: "src/components/microservices/k3sctl-adapter/k3s/dev/unidesk-dev-core.k8s.yaml", + composeService: spec.composeService, + containerName: spec.containerName, + }, + backend: { + nodeBaseUrl: spec.nodeBaseUrl, + nodeBindHost: `k3s://unidesk-dev/${spec.composeService}`, + nodePort: spec.nodePort, + proxyMode: "dev-k3s-direct", + frontendOnly: true, + public: false, + allowedMethods: spec.allowedMethods, + allowedPathPrefixes: spec.allowedPathPrefixes, + healthPath: spec.healthPath, + timeoutMs: 30_000, + }, + deployment: { + mode: "k3sctl-managed", + adapterServiceId: "k3sctl-adapter", + k3sServiceId: spec.composeService, + namespace: "unidesk-dev", + expectedNodeIds: ["D601"], + activeNodeId: "D601", + }, + development: { + providerId: "D601", + sshPassthrough: true, + worktreePath: `/home/ubuntu/unidesk-dev-core-deploy/${id}`, + }, + frontend: { + route: spec.route, + integrated: false, + }, + }; +} + +function coreDeployService(config: UniDeskConfig, id: string, environment: DeployEnvironment | null): UniDeskMicroserviceConfig | undefined { + if (environment === "dev") return devCoreDeployService(id); if (id === "frontend") return frontendCoreDeployService(config); return undefined; } @@ -576,6 +666,12 @@ function isCoreDeployService(service: UniDeskMicroserviceConfig): boolean { return service.id === "frontend" && service.providerId === "main-server" && service.backend.proxyMode === "core-direct"; } +function isDevK3sCoreService(service: UniDeskMicroserviceConfig): boolean { + return service.deployment.mode === "k3sctl-managed" + && service.deployment.namespace === deployEnvironmentTargets.dev.namespace + && devApplySupportedServiceIds.has(service.id); +} + function isDirectComposeDeployMode(service: UniDeskMicroserviceConfig): boolean { return service.deployment.mode === "unidesk-direct" || service.deployment.mode === "internal-sidecar"; } @@ -588,7 +684,14 @@ function selectServices(config: UniDeskConfig, manifest: DeployManifest, service const selected = serviceId === null ? manifest.services : manifest.services.filter((service) => service.id === serviceId); if (serviceId !== null && selected.length === 0) throw new Error(`deploy manifest does not contain service: ${serviceId}`); return selected.map((desired) => { - const service = configById.get(desired.id) ?? coreDeployService(config, desired.id); + if (manifest.environment === "dev") { + const service = devCoreDeployService(desired.id); + if (service === undefined) { + throw new Error(`deploy --env dev service ${desired.id} is not enabled in this executor yet; currently supported: ${[...devApplySupportedServiceIds].join(", ")}`); + } + return { desired, config: service }; + } + const service = configById.get(desired.id) ?? coreDeployService(config, desired.id, manifest.environment); if (service === undefined) throw new Error(`deploy manifest service ${desired.id} is not present in config.json microservices or supported core deploy services`); return { desired, config: service }; }); @@ -643,6 +746,7 @@ function targetExportDir(service: UniDeskMicroserviceConfig, runId: string): str } function targetWorkDir(service: UniDeskMicroserviceConfig): string { + if (isDevK3sCoreService(service)) return service.development.worktreePath; if (service.deployment.mode === "k3sctl-managed") return k3sDeployDir; if (targetIsMain(service) && isUnideskRepo(service.repository.url)) { return rootPath(".state", "deploy", "work", safeId(service.id)); @@ -676,6 +780,7 @@ function sourceBuildContext(service: UniDeskMicroserviceConfig): string { } function buildImageTag(service: UniDeskMicroserviceConfig): string { + if (isDevK3sCoreService(service)) return `unidesk-${service.id}:dev`; if (service.deployment.mode === "k3sctl-managed") return `unidesk-${service.id}:d601`; if (targetIsMain(service)) { if (["project-manager", "baidu-netdisk", "oa-event-flow"].includes(service.repository.composeService)) return service.repository.composeService; @@ -706,6 +811,7 @@ function directDockerfileOverride(service: UniDeskMicroserviceConfig): string { function k8sManifestPath(service: UniDeskMicroserviceConfig): string { const composeFile = service.repository.composeFile; + if (composeFile.endsWith(".k8s.yaml")) return composeFile; if (!composeFile.endsWith(".k3s.json")) throw new Error(`${service.id} k3s service composeFile must point to *.k3s.json`); return composeFile.replace(/\.k3s\.json$/u, ".k8s.yaml"); } @@ -916,8 +1022,24 @@ function claudeqqDeployAssetOverlayCommands(): string[] { } function syncK8sControlManifestsScript(service: UniDeskMicroserviceConfig): string { + if (isDevK3sCoreService(service)) { + const manifest = k8sManifestPath(service); + if (!existsSync(rootPath(manifest))) throw new Error(`${service.id} dev k3s control manifest missing: ${manifest}`); + const encoded = Buffer.from(readFileSync(rootPath(manifest), "utf8"), "utf8").toString("base64"); + return [ + "set -euo pipefail", + `target_root=${shellQuote(targetWorkDir(service))}`, + `relative_path=${shellQuote(manifest)}`, + "target_file=\"$target_root/$relative_path\"", + "mkdir -p \"$(dirname \"$target_file\")\"", + `printf %s ${shellQuote(encoded)} | base64 -d > "$target_file"`, + "printf 'synced_k3s_control_manifest=%s\\n' \"$target_file\"", + ].join("\n"); + } if (service.deployment.mode !== "k3sctl-managed" || isUnideskRepo(service.repository.url)) return ""; - const manifests = [service.repository.composeFile, k8sManifestPath(service)]; + const manifests = service.repository.composeFile.endsWith(".k8s.yaml") + ? [service.repository.composeFile] + : [service.repository.composeFile, k8sManifestPath(service)]; const commands = [ "set -euo pipefail", `target_root=${shellQuote(targetWorkDir(service))}`, @@ -970,7 +1092,14 @@ function buildImageScript(service: UniDeskMicroserviceConfig, desired: DeployMan sourceProxyPrelude(service), `image=${shellQuote(image)}`, `dockerfile=${shellQuote(dockerfile)}`, - "docker buildx version >/dev/null", + "if ! docker buildx version >/dev/null 2>&1; then", + " if ! docker buildx inspect default >/dev/null 2>&1; then echo target_build_builder=missing >&2; exit 1; fi", + "fi", + ...(isDevK3sCoreService(service) ? [ + "if ! docker image inspect oven/bun:1-alpine >/dev/null 2>&1; then", + " docker pull oven/bun:1-alpine", + "fi", + ] : []), "builder_args=()", "if docker buildx inspect --builder default >/dev/null 2>&1; then builder_args=(--builder default); echo target_build_builder=default; else echo target_build_builder=implicit; fi", "docker buildx inspect \"${builder_args[@]}\" --bootstrap || true", @@ -981,6 +1110,54 @@ function buildImageScript(service: UniDeskMicroserviceConfig, desired: DeployMan ].filter((line) => line.length > 0).join("\n"); } +function patchDevK3sCoreManifestScript(service: UniDeskMicroserviceConfig, desired: DeployManifestService, resolvedCommit: string): string { + const manifest = `${targetWorkDir(service)}/${k8sManifestPath(service)}`; + const image = buildImageTag(service); + const serviceId = service.id; + const deploymentName = service.repository.composeService; + const manifestServiceId = deploymentName; + return [ + "set -euo pipefail", + `manifest=${shellQuote(manifest)}`, + `service_id=${shellQuote(serviceId)}`, + `deployment_name=${shellQuote(deploymentName)}`, + `image=${shellQuote(image)}`, + `repo=${shellQuote(desired.repo)}`, + `commit=${shellQuote(resolvedCommit)}`, + `requested_commit=${shellQuote(desired.commitId)}`, + `manifest_service_id=${shellQuote(manifestServiceId)}`, + `python3 - "$manifest" "$service_id" "$deployment_name" "$image" "$repo" "$commit" "$requested_commit" "$manifest_service_id" <<'PY'`, + "import re", + "import sys", + "path, service_id, deployment_name, image, repo, commit, requested_commit, manifest_service_id = sys.argv[1:]", + "text = open(path, encoding='utf-8').read()", + "segments = re.split(r'(?m)^---\\s*$', text)", + "kept = []", + "for segment in segments:", + " if not segment.strip():", + " continue", + " if f'\\n name: {deployment_name}\\n' in ('\\n' + segment + '\\n'):", + " kept.append(segment)", + "if not kept:", + " raise SystemExit(f'deployment/service {deployment_name} not found in {path}')", + "patched = []", + "for segment in kept:", + " segment = segment.replace('unidesk.ai/image-source: deploy-dev-commit', 'unidesk.ai/image-source: deploy-env-commit')", + " segment = re.sub(r'image: unidesk-[^\\n]+:dev-placeholder', f'image: {image}', segment)", + " segment = re.sub(r'value: replace-with-deploy-dev-commit', f'value: {commit}', segment)", + " segment = segment.replace(f'value: {manifest_service_id}', f'value: {service_id}')", + " patched.append(segment.strip() + '\\n')", + "out = '---\\n'.join(patched)", + "open(path, 'w', encoding='utf-8').write(out)", + "print(f'dev_k3s_manifest_patched={path}')", + "print(f'dev_k3s_manifest_service={service_id}')", + "print(f'dev_k3s_manifest_deployment={deployment_name}')", + "print(f'dev_k3s_manifest_image={image}')", + "print(f'dev_k3s_manifest_commit={commit}')", + "PY", + ].join("\n"); +} + function directComposeResolveScript(service: UniDeskMicroserviceConfig): string { const projectHint = targetIsMain(service) ? "unidesk" : ""; return [ @@ -1322,6 +1499,10 @@ function cleanupLegacyDirectCodeQueueScript(service: UniDeskMicroserviceConfig): ].join("\n"); } +function k8sNamespaceForService(service: UniDeskMicroserviceConfig): string { + return service.deployment.namespace ?? k8sNamespace; +} + function k8sDeploymentsForService(service: UniDeskMicroserviceConfig): string[] { if (service.id === "code-queue") return ["d601-provider-egress-proxy", "d601-tcp-egress-gateway", "code-queue", "code-queue-read", "code-queue-write"]; return [service.repository.composeService]; @@ -1329,11 +1510,12 @@ function k8sDeploymentsForService(service: UniDeskMicroserviceConfig): string[] function applyK8sScript(service: UniDeskMicroserviceConfig): string { const manifest = `${targetWorkDir(service)}/${k8sManifestPath(service)}`; + const namespace = k8sNamespaceForService(service); const cleanup = service.id === "code-queue" ? [ - `KUBECONFIG=${shellQuote(k8sKubeconfig)} kubectl -n ${shellQuote(k8sNamespace)} delete endpointslice d601-provider-egress-proxy --ignore-not-found`, - `KUBECONFIG=${shellQuote(k8sKubeconfig)} kubectl -n ${shellQuote(k8sNamespace)} delete deployment code-queue-d518 --ignore-not-found`, - `KUBECONFIG=${shellQuote(k8sKubeconfig)} kubectl -n ${shellQuote(k8sNamespace)} delete service code-queue-d518 --ignore-not-found`, + `KUBECONFIG=${shellQuote(k8sKubeconfig)} kubectl -n ${shellQuote(namespace)} delete endpointslice d601-provider-egress-proxy --ignore-not-found`, + `KUBECONFIG=${shellQuote(k8sKubeconfig)} kubectl -n ${shellQuote(namespace)} delete deployment code-queue-d518 --ignore-not-found`, + `KUBECONFIG=${shellQuote(k8sKubeconfig)} kubectl -n ${shellQuote(namespace)} delete service code-queue-d518 --ignore-not-found`, ].join("\n") : ""; return [ @@ -1344,6 +1526,7 @@ function applyK8sScript(service: UniDeskMicroserviceConfig): string { function stampK8sScript(service: UniDeskMicroserviceConfig, desired: DeployManifestService, resolvedCommit: string): string { const deployments = k8sDeploymentsForService(service).map((name) => `deployment/${name}`); + const namespace = k8sNamespaceForService(service); const envPairs = [ `UNIDESK_DEPLOY_SERVICE_ID=${service.id}`, `UNIDESK_DEPLOY_REPO=${desired.repo}`, @@ -1362,9 +1545,9 @@ function stampK8sScript(service: UniDeskMicroserviceConfig, desired: DeployManif ]; return [ "set -euo pipefail", - `KUBECONFIG=${shellQuote(k8sKubeconfig)} kubectl -n ${shellQuote(k8sNamespace)} set env ${deployments.map(shellQuote).join(" ")} ${envPairs.map(shellQuote).join(" ")}`, - `KUBECONFIG=${shellQuote(k8sKubeconfig)} kubectl -n ${shellQuote(k8sNamespace)} annotate ${deployments.map(shellQuote).join(" ")} ${annotatePairs.map(shellQuote).join(" ")} --overwrite`, - `actual=$(KUBECONFIG=${shellQuote(k8sKubeconfig)} kubectl -n ${shellQuote(k8sNamespace)} get deployment ${shellQuote(service.repository.composeService)} -o jsonpath='{.metadata.annotations.unidesk\\.ai/deploy-commit}')`, + `KUBECONFIG=${shellQuote(k8sKubeconfig)} kubectl -n ${shellQuote(namespace)} set env ${deployments.map(shellQuote).join(" ")} ${envPairs.map(shellQuote).join(" ")}`, + `KUBECONFIG=${shellQuote(k8sKubeconfig)} kubectl -n ${shellQuote(namespace)} annotate ${deployments.map(shellQuote).join(" ")} ${annotatePairs.map(shellQuote).join(" ")} --overwrite`, + `actual=$(KUBECONFIG=${shellQuote(k8sKubeconfig)} kubectl -n ${shellQuote(namespace)} get deployment ${shellQuote(service.repository.composeService)} -o jsonpath='{.metadata.annotations.unidesk\\.ai/deploy-commit}')`, `test "$actual" = ${shellQuote(resolvedCommit)}`, "printf 'k8s_deploy_commit=%s\\n' \"$actual\"", ].join("\n"); @@ -1372,18 +1555,20 @@ function stampK8sScript(service: UniDeskMicroserviceConfig, desired: DeployManif function rolloutK8sScript(service: UniDeskMicroserviceConfig): string { const deployments = k8sDeploymentsForService(service); + const namespace = k8sNamespaceForService(service); return [ "set -euo pipefail", - `KUBECONFIG=${shellQuote(k8sKubeconfig)} kubectl -n ${shellQuote(k8sNamespace)} rollout restart ${deployments.map((name) => shellQuote(`deployment/${name}`)).join(" ")}`, - ...deployments.map((name) => `KUBECONFIG=${shellQuote(k8sKubeconfig)} kubectl -n ${shellQuote(k8sNamespace)} rollout status ${shellQuote(`deployment/${name}`)} --timeout=180s`), - `KUBECONFIG=${shellQuote(k8sKubeconfig)} kubectl -n ${shellQuote(k8sNamespace)} get deploy ${deployments.map(shellQuote).join(" ")} -o wide`, + `KUBECONFIG=${shellQuote(k8sKubeconfig)} kubectl -n ${shellQuote(namespace)} rollout restart ${deployments.map((name) => shellQuote(`deployment/${name}`)).join(" ")}`, + ...deployments.map((name) => `KUBECONFIG=${shellQuote(k8sKubeconfig)} kubectl -n ${shellQuote(namespace)} rollout status ${shellQuote(`deployment/${name}`)} --timeout=180s`), + `KUBECONFIG=${shellQuote(k8sKubeconfig)} kubectl -n ${shellQuote(namespace)} get deploy ${deployments.map(shellQuote).join(" ")} -o wide`, ].join("\n"); } function k8sCommitProbeScript(service: UniDeskMicroserviceConfig): string { + const namespace = k8sNamespaceForService(service); return [ "set -euo pipefail", - `commit=$(KUBECONFIG=${shellQuote(k8sKubeconfig)} kubectl -n ${shellQuote(k8sNamespace)} get deployment ${shellQuote(service.repository.composeService)} -o jsonpath='{.metadata.annotations.unidesk\\.ai/deploy-commit}' 2>/dev/null || true)`, + `commit=$(KUBECONFIG=${shellQuote(k8sKubeconfig)} kubectl -n ${shellQuote(namespace)} get deployment ${shellQuote(service.repository.composeService)} -o jsonpath='{.metadata.annotations.unidesk\\.ai/deploy-commit}' 2>/dev/null || true)`, "printf '%s\\n' \"$commit\"", ].join("\n"); } @@ -1444,10 +1629,41 @@ async function directHttpJson(url: string, timeoutMs: number): Promise } } +function devK3sServiceHealthScript(service: UniDeskMicroserviceConfig): string { + const namespace = k8sNamespaceForService(service); + const deployment = service.repository.composeService; + const path = service.backend.healthPath; + return [ + "set -euo pipefail", + `namespace=${shellQuote(namespace)}`, + `deployment=${shellQuote(deployment)}`, + `path=${shellQuote(path)}`, + `pod=$(KUBECONFIG=${shellQuote(k8sKubeconfig)} kubectl -n "$namespace" get pod -l app.kubernetes.io/name=${shellQuote(service.id)},unidesk.ai/environment=dev -o jsonpath='{.items[0].metadata.name}' 2>/dev/null || true)`, + "if [ -z \"$pod\" ]; then echo '{\"ok\":false,\"error\":\"dev deployment pod not found\"}'; exit 0; fi", + `KUBECONFIG=${shellQuote(k8sKubeconfig)} kubectl -n "$namespace" exec "$pod" -- wget -q -T 5 -O - "http://127.0.0.1:8080$path"`, + ].join("\n"); +} + async function serviceHealth(config: UniDeskConfig, service: UniDeskMicroserviceConfig): Promise { if (isCoreDeployService(service)) { return await directHttpJson(`${service.backend.nodeBaseUrl}${service.backend.healthPath}`, service.backend.timeoutMs); } + if (isDevK3sCoreService(service)) { + const result = await runTargetCommand(config, service, devK3sServiceHealthScript(service), "/home/ubuntu", 30_000, 20_000); + let body: unknown = null; + try { + body = result.stdout.trim().length > 0 ? JSON.parse(result.stdout.trim()) : null; + } catch { + body = { text: result.stdout.trim() }; + } + return { + ok: result.ok, + status: result.ok ? 200 : 502, + body, + raw: result.raw, + error: result.ok ? undefined : result.stderr || result.stdout || "dev k3s service health failed", + }; + } return coreInternalFetch(`/api/microservices/${encodeURIComponent(service.id)}/health`); } @@ -1699,6 +1915,7 @@ async function step( } async function readDockerImageCommit(config: UniDeskConfig, service: UniDeskMicroserviceConfig): Promise { + if (isDevK3sCoreService(service)) return null; const result = await runTargetCommand(config, service, dockerCommitProbeScript(service), targetIsMain(service) ? repoRoot : "/home/ubuntu", 30_000, 20_000); const commit = parseFullCommit(result.stdout); return commit.length > 0 ? commit : null; @@ -1824,7 +2041,7 @@ async function applyOneService(config: UniDeskConfig, service: UniDeskMicroservi const runId = `${Date.now().toString(36)}-${Math.random().toString(16).slice(2, 8)}`; const exportDir = targetExportDir(service, runId); - if (service.id === "code-queue" && !targetIsMain(service)) { + if (!targetIsMain(service) && isUnideskRepo(desired.repo)) { const identity = await ensureGithubSshIdentityStep(config, service); if (!pushStep(steps, identity)) return { ok: false, serviceId: service.id, startedAt, finishedAt: nowIso(), before, steps }; } @@ -1846,6 +2063,11 @@ async function applyOneService(config: UniDeskConfig, service: UniDeskMicroservi if (!pushStep(steps, controlManifestSync)) return { ok: false, serviceId: service.id, startedAt, finishedAt: nowIso(), resolvedCommit, before, steps }; } + if (isDevK3sCoreService(service)) { + const patchManifest = await step(config, service, "patch-dev-k3s-manifest", patchDevK3sCoreManifestScript(service, desired, resolvedCommit), targetWorkDir(service), 60_000, false); + if (!pushStep(steps, patchManifest)) return { ok: false, serviceId: service.id, startedAt, finishedAt: nowIso(), resolvedCommit, before, steps }; + } + const buildScript = isDirectComposeDeployMode(service) ? buildDirectImageScript(service, desired, resolvedCommit) : buildImageScript(service, desired, resolvedCommit); @@ -1965,6 +2187,12 @@ function environmentDryRunPlan( }; } +function unsupportedDevApplyServices(manifest: DeployManifest, serviceId: string | null): string[] { + if (manifest.environment !== "dev") return []; + const services = serviceId === null ? manifest.services : manifest.services.filter((service) => service.id === serviceId); + return services.map((service) => service.id).filter((id) => !devApplySupportedServiceIds.has(id)); +} + async function runApplyNow(config: UniDeskConfig, manifest: DeployManifest, options: DeployOptions): Promise> { const selected = selectServices(config, manifest, options.serviceId); const startedAt = nowIso(); @@ -1989,7 +2217,8 @@ async function runApplyNow(config: UniDeskConfig, manifest: DeployManifest, opti function applyJob(config: UniDeskConfig, args: string[], options: DeployOptions): Record { const runArgs = args.includes("--run-now") ? args : [...args, "--run-now"]; const command = [process.execPath, rootPath("scripts", "cli.ts"), "deploy", ...runArgs]; - const job = startJob("deploy_apply", command, `Reconcile services from ${options.file}${options.serviceId === null ? "" : ` service=${options.serviceId}`}`); + const source = options.environment === null ? options.file : `${deployEnvironmentTargets[options.environment].gitRef}:deploy.json`; + const job = startJob("deploy_apply", command, `Reconcile services from ${source}${options.serviceId === null ? "" : ` service=${options.serviceId}`}`); return { ok: true, mode: "async-job", @@ -2008,9 +2237,17 @@ export async function runDeployCommand(config: UniDeskConfig | null, args: strin const action = actionRaw as DeployAction; const options = parseOptions(args.slice(1)); if (options.environment !== null) { - if (action === "apply") throw new Error("deploy apply --env is not enabled in Phase 0; use deploy plan --env dev|prod for dry-run only"); const { manifest, source } = readEnvironmentDeployManifest(options.environment); - return environmentDryRunPlan(manifest, source, options, action); + if (action === "check" || action === "plan") return environmentDryRunPlan(manifest, source, options, action); + if (options.environment !== "dev") throw new Error("deploy apply --env prod is not enabled yet"); + const unsupported = unsupportedDevApplyServices(manifest, options.serviceId); + if (unsupported.length > 0) { + throw new Error(`deploy apply --env dev currently supports only ${[...devApplySupportedServiceIds].join(", ")}; unsupported selected services: ${unsupported.join(", ")}`); + } + if (config === null) throw new Error("deploy apply --env dev requires config.json"); + const resolved = resolveManifestCommits(manifest, options.serviceId); + if (!options.runNow) return applyJob(config, args, options); + return await runApplyNow(config, resolved, options); } if (config === null) throw new Error("deploy local manifest mode requires config.json"); const manifest = resolveManifestCommits(await readDeployManifest(options.file), options.serviceId); diff --git a/src/components/microservices/k3sctl-adapter/k3s/dev/unidesk-dev-core.k8s.yaml b/src/components/microservices/k3sctl-adapter/k3s/dev/unidesk-dev-core.k8s.yaml index e2d96412..35432365 100644 --- a/src/components/microservices/k3sctl-adapter/k3s/dev/unidesk-dev-core.k8s.yaml +++ b/src/components/microservices/k3sctl-adapter/k3s/dev/unidesk-dev-core.k8s.yaml @@ -34,6 +34,7 @@ metadata: annotations: unidesk.ai/deploy-ref: origin/deploy/dev unidesk.ai/image-source: deploy-dev-commit + unidesk.ai/deploy-service-id: backend-core spec: replicas: 1 strategy: @@ -77,10 +78,30 @@ spec: value: "8080" - name: PROVIDER_PORT value: "8081" + - name: DATABASE_URL + valueFrom: + secretKeyRef: + name: unidesk-dev-runtime-secrets + key: DATABASE_URL + - name: PROVIDER_TOKEN + valueFrom: + secretKeyRef: + name: unidesk-dev-runtime-secrets + key: PROVIDER_TOKEN + - name: DATABASE_VOLUME_NAME + valueFrom: + configMapKeyRef: + name: unidesk-dev-runtime-config + key: DATABASE_VOLUME_NAME + - name: DATABASE_VOLUME_SIZE + valueFrom: + configMapKeyRef: + name: unidesk-dev-runtime-config + key: DATABASE_VOLUME_SIZE - name: UNIDESK_DATABASE_NAME value: unidesk_dev - name: UNIDESK_DEPLOY_SERVICE_ID - value: backend-core-dev + value: backend-core - name: UNIDESK_DEPLOY_REPO value: https://github.com/pikasTech/unidesk - name: UNIDESK_DEPLOY_COMMIT @@ -156,6 +177,7 @@ metadata: annotations: unidesk.ai/deploy-ref: origin/deploy/dev unidesk.ai/image-source: deploy-dev-commit + unidesk.ai/deploy-service-id: frontend spec: replicas: 1 strategy: @@ -201,10 +223,30 @@ spec: value: http://frontend-dev.unidesk-dev.svc.cluster.local:8080 - name: PROVIDER_INGRESS_PUBLIC_URL value: ws://backend-core-dev.unidesk-dev.svc.cluster.local:8081/ws/provider + - name: AUTH_USERNAME + valueFrom: + secretKeyRef: + name: unidesk-dev-runtime-secrets + key: AUTH_USERNAME + - name: AUTH_PASSWORD + valueFrom: + secretKeyRef: + name: unidesk-dev-runtime-secrets + key: AUTH_PASSWORD + - name: SESSION_SECRET + valueFrom: + secretKeyRef: + name: unidesk-dev-runtime-secrets + key: SESSION_SECRET + - name: SESSION_TTL_SECONDS + valueFrom: + configMapKeyRef: + name: unidesk-dev-runtime-config + key: SESSION_TTL_SECONDS - name: UNIDESK_DATABASE_NAME value: unidesk_dev - name: UNIDESK_DEPLOY_SERVICE_ID - value: frontend-dev + value: frontend - name: UNIDESK_DEPLOY_REPO value: https://github.com/pikasTech/unidesk - name: UNIDESK_DEPLOY_COMMIT