diff --git a/.dockerignore b/.dockerignore index 2e290d7b..0bd5fc46 100644 --- a/.dockerignore +++ b/.dockerignore @@ -4,6 +4,7 @@ logs node_modules **/node_modules **/dist +**/target **/coverage npm-debug.log* .env diff --git a/.gitignore b/.gitignore index 369f6901..bdf9046a 100644 --- a/.gitignore +++ b/.gitignore @@ -11,3 +11,5 @@ provider-*.yml provider-*.yaml dist/ coverage/ +target/ +**/target/ diff --git a/AGENTS.md b/AGENTS.md index ecd472b3..cf308b97 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -26,18 +26,18 @@ UniDesk 是一个以主 server 为统一入口的分布式工作平台；本文 - `bun scripts/cli.ts help`：输出所有可用命令的 JSON 索引，详细规范见 `docs/reference/cli.md`。 - `bun scripts/cli.ts --main-server-ip `：默认通过公网 frontend 登录态远程执行调试、用户服务（底层命令名 `microservice`）、Code Queue 查询与节点自测命令，不要求主 server SSH key，详细规范见 `docs/reference/cli.md`。 - `bun scripts/cli.ts config show`：校验并展示根目录 `config.json`，配置来源规则见 `docs/reference/config.md`。 -- `bun scripts/cli.ts check [--full|--files|--scripts-typecheck|--components|--compose|--logs]`：默认只运行轻量配置和 TypeScript 语法检查；关键文件、`scripts/` 类型、组件类型、Docker Compose 和日志策略检查需显式开启，测试入口见 `TEST.md`。 +- `bun scripts/cli.ts check [--full|--files|--scripts-typecheck|--components|--compose|--logs|--rust]`：默认只运行轻量配置和 TypeScript 语法检查；Rust backend-core 检查只能在 D601 CI/dev execution 中用 `UNIDESK_D601_RUST_CHECK=1` 开启，规则见 `docs/reference/dev-environment.md`。 - `bun scripts/cli.ts server start`：以异步 job 启动 database、backend-core、frontend、provider-gateway、code-queue-mgr 和主 server 用户服务，部署规则见 `docs/reference/deployment.md`。 -- `bun scripts/cli.ts server status`：查询固定端口、swap 摘要、容器状态、健康检查和访问 URL，判定标准见 `docs/reference/deployment.md`。 +- `bun scripts/cli.ts server status`：查询固定端口、swap 摘要、容器状态、健康检查和访问 URL，包含生产 frontend、dev frontend proxy 和 provider ingress，判定标准见 `docs/reference/deployment.md` 与 `docs/reference/dev-environment.md`。 - `bun scripts/cli.ts server swap status|ensure [--path /swapfile] [--size 2GiB] [--dry-run]`：以 JSON 查看或幂等创建主 server swapfile，`ensure` 输出 before/after、动作、持久化状态和 degraded/failed 详情，规则见 `docs/reference/deployment.md`。 - `bun scripts/cli.ts server logs [--tail-bytes N]`：分页返回文件日志与 Docker 日志尾部并带截断元数据，日志规则见 `docs/reference/observability.md`。 -- `bun scripts/cli.ts server rebuild `：以 build-first、Compose lock、no-deps force-recreate 和 post-up validation 的异步 job 重建主 server Compose 内单个服务；Code Queue 执行面部署在 D601，规则见 `docs/reference/deployment.md`。 +- `bun scripts/cli.ts server rebuild `：以 build-first、Compose lock、no-deps force-recreate 和 post-up validation 的异步 job 重建主 server Compose 内单个服务；`dev-frontend-proxy` 是 18083 dev 入口薄代理，Rust backend-core 迭代不得在 master server 用该命令编译，规则见 `docs/reference/deployment.md` 与 `docs/reference/dev-environment.md`。 - `bun scripts/cli.ts provider attach [--master-server URL] [--up] [--force]`：在新增计算节点上生成两项配置的 provider-gateway 挂载包；默认只需要主 server URL（默认 `http://74.48.78.17/`）和唯一 Provider ID，生成的 Compose 固定 Docker socket、`pid: "host"`、`restart: always`、只读 `/workspace`、SSH 维护私钥挂载和 loopback egress proxy 端口，规则见 `docs/reference/provider-gateway.md`。 - `bun scripts/cli.ts ssh [ssh-like args...]`：通过 provider-gateway 的 Host SSH / WSL SSH 维护桥打开近似原生 ssh 的交互会话或远端命令，并在远端 PATH 注入 `apply_patch`、`glob` 与 `skill-discover`；`apply-patch`、`py`、`skills`、结构化 `find`、`glob` 和 `argv` 子命令用于避免远端补丁、Python stdin、skill 发现与常用只读命令的嵌套转义问题，使用规则见 `docs/reference/cli.md` 和 `docs/reference/provider-gateway.md`。 - `bun scripts/cli.ts microservice list/status/health/diagnostics/tunnel-self-test/proxy`：管理和验证挂载在主 server、计算节点 Docker 或 k3s 控制面上的用户服务，`proxy` 支持受控 JSON body，OA Event Flow/Todo Note/Baidu Netdisk/Code Queue Manager on main-server、k3s Control/Code Queue 执行面/MDTODO/Decision Center/FindJob/Pipeline/MET Nonlinear on D601 的规则见 `docs/reference/microservices.md`。 - `bun scripts/cli.ts decision upload/list/show/health`：通过 backend-core 用户服务代理上传会议记录/决议 Markdown、列出记录和查看详情；Decision Center 运行在 D601 k3s，规则见 `docs/reference/microservices.md`。 - `bun scripts/cli.ts decision diary import/list/months/show`：把带日期标题的工作日志 Markdown 拆成 `YYYY-MM/YYYY-MM-DD.md` 日记条目并写入 PostgreSQL，规则见 `docs/reference/microservices.md`。 -- `bun scripts/cli.ts deploy check/plan/apply [--file deploy.json|--env dev|prod] [--service ]`：按根目录 `deploy.json` 或 `origin/master:deploy.json#environments.` 的服务 repo 和 commit 期望状态校验或更新用户服务；维护通道直连 D601 不得部署 backend-core/frontend/code-queue/Decision Center/k3sctl-adapter 等直管或代管微服务；规则见 `docs/reference/deploy.md`。 +- `bun scripts/cli.ts deploy check/plan/apply [--file deploy.json|--env dev|prod] [--service ]`：按根目录 `deploy.json` 或 `origin/master:deploy.json#environments.` 的服务 repo 和 commit 期望状态校验或更新用户服务；`--env dev` 当前只开放 D601 `backend-core`/`frontend` persistent dev rollout，规则见 `docs/reference/deploy.md` 与 `docs/reference/dev-environment.md`。 - `bun scripts/cli.ts dev-env validate [--manifest path] [--kubectl-dry-run]` / `dev-env prewarm-images`：离线校验 D601 `unidesk-dev` 生产隔离护栏，或把开发底座基础镜像预热到 D601 原生 k3s containerd，规则见 `docs/reference/deploy.md` 与 `docs/reference/microservices.md`。 - `bun scripts/cli.ts ci install/status/run/run-dev-e2e/logs`：在 D601 原生 k3s 上安装和运行 Tekton CI，支持每 commit 检查、Code Queue 只读性能门禁和手动触发的 `origin/master:deploy.json#environments.dev` 临时 namespace e2e；`run-dev-e2e` 的 Git 控制 runner、短 launcher 和 no-CD 边界见 `docs/reference/dev-ci-runner.md`，Tekton 规则见 `docs/reference/ci.md`。 - `bun scripts/cli.ts codex deploy `：旧 Code Queue 兼容部署入口已禁用，原因是它会绕过受控部署边界直连 D601 部署 Code Queue；规则见 `docs/reference/codex-deploy.md`。 @@ -48,14 +48,14 @@ UniDesk 是一个以主 server 为统一入口的分布式工作平台；本文 - `bun scripts/cli.ts server stop`：以异步 job 停止固定 Compose 项目中的全部 UniDesk 服务，停止后用 `server status` 复核。 - `bun scripts/cli.ts job list [--limit N]` / `bun scripts/cli.ts job status latest [--tail-bytes N]`：分页查询 `.state/jobs/` 中的异步任务状态，状态输出只读日志尾部并保留完整日志路径，job 机制见 `docs/reference/cli.md`。 - `bun scripts/cli.ts debug health` / `bun scripts/cli.ts debug dispatch` / `bun scripts/cli.ts debug task`：通过 Docker 内网 core、真实 HTTP、WebSocket、系统指标、Docker 状态和 Host SSH 维护桥流程调试健康检查、任务下发与任务结果，调试规则见 `docs/reference/cli.md`。 -- `bun scripts/cli.ts e2e run [--only pattern[,pattern...]] [--skip pattern[,pattern...]]`：支持按 check/prefix/wildcard 选择性执行公网 frontend/provider ingress、内网 core/database、provider-gateway 自接入与 Playwright 验证；日常迭代先跑当前问题对应的最小检查集，最终交付再跑全量回归，验收规则见 `docs/reference/e2e.md`。 +- `bun scripts/cli.ts e2e run [--only pattern[,pattern...]] [--skip pattern[,pattern...]]`：支持按 check/prefix/wildcard 选择性执行公网 production frontend/dev frontend/provider ingress、内网 core/database、provider-gateway 自接入与 Playwright 验证；日常迭代先跑当前问题对应的最小检查集，最终交付再跑全量回归，验收规则见 `docs/reference/e2e.md`。 ## Runtime - `bun`：TypeScript 运行时固定使用 Bun，组件入口和 CLI 都直接运行 `.ts` 文件，约束见 `docs/reference/config.md`。 -- `docker-compose.yml`：主 server 统一编排 core、frontend、database、本机 provider gateway、Todo Note 后端、Baidu Netdisk 后端、OA Event Flow 后端和轻量 Code Queue Manager 控制面；Code Queue 执行面、MDTODO 和 Decision Center 由 D601 k3s/k8s 控制面代管，并经 `k3sctl-adapter` 的 Kubernetes API service proxy 单一路径接入，服务拓扑见 `docs/reference/deployment.md`。 +- `docker-compose.yml`：主 server 统一编排 core、frontend、dev-frontend-proxy、database、本机 provider gateway、Todo Note 后端、Baidu Netdisk 后端、OA Event Flow 后端和轻量 Code Queue Manager 控制面；Code Queue 执行面、MDTODO 和 Decision Center 由 D601 k3s/k8s 控制面代管，并经 `k3sctl-adapter` 的 Kubernetes API service proxy 单一路径接入，服务拓扑见 `docs/reference/deployment.md` 与 `docs/reference/dev-environment.md`。 - `src/components/frontend`：前端源码固定使用 TypeScript + React，`app.tsx` 只做 shell/router，左侧主模块与顶部子标签统一编译为模块前缀路由：`/ops//`、`/nodes//`、`/tasks//`、`/config//`，只有用户服务使用 `/app//` 深链接，运行总览包含通用性能面板，资源监控含曲线和进程资源排序表，Todo Note、FindJob、Pipeline、MET Nonlinear、Baidu Netdisk、Code Queue、MDTODO、Decision Center、OA Event Flow、k3s Control 等业务页必须拆到独立 TSX 模块，界面规则见 `docs/reference/frontend.md`。 -- `backend-core / frontend performance`：backend-core 暴露 `/api/performance`，frontend 暴露同源 `/api/frontend-performance` 并在 `/ops/performance/` 汇总组件请求、失败请求、内部操作和慢操作，规则见 `docs/reference/observability.md`。backend-core 源码已拆分为 15 个模块，结构见 `docs/reference/repo-tree.md`。 +- `backend-core / frontend performance`：backend-core 暴露 `/api/performance`，frontend 暴露同源 `/api/frontend-performance` 并在 `/ops/performance/` 汇总组件请求、失败请求、内部操作和慢操作，规则见 `docs/reference/observability.md`；backend-core 当前为 Rust 服务，结构见 `docs/reference/repo-tree.md`，Rust 编译边界见 `docs/reference/dev-environment.md`。 - `Unified OA event flow`：`oa-event-flow` 是独立主 server 用户服务，提供事件表、按 tag 订阅和 Trace/STEP 统计中心，Code Queue 与 Pipeline 都必须接入统一事件流；共享契约见 `docs/reference/oa-event-flow.md`，Pipeline 专有控制流规则见 `docs/reference/pipeline-oa-event-flow.md`。 - `src/components/provider-gateway`：当前主 server `74.48.78.17` 也作为 provider gateway 接入 UniDesk，外部节点通过 `ws://74.48.78.17:18082/ws/provider` 接入，必须以 `restart: always` 部署 always-enabled 远程升级、sleep-and-validate 回滚保护和 Host SSH / WSL SSH 透传并完成自测，部署与 Playwright 公网前端验证方法见 `docs/reference/provider-gateway.md`。 - `microservices`：用户服务配置命名仍保留 `microservices`；用户服务指挂载在 UniDesk 核心服务上的用户业务能力，支持 `unidesk-direct`、`internal-sidecar` 与 `k3sctl-managed` 部署模式；`code-queue-mgr` 是主 server 内部 sidecar 控制面，D601 Code Queue 是执行面；k3s 代管必须使用标准 k3s/k8s 对象和 Kubernetes API service proxy，禁止业务容器直连、NodePort 和隐藏 fallback；缺少这些服务时核心仍可运行。主 server 本地开发边界固定为只开发 UniDesk frontend 和已登记的内部 sidecar 控制面；非 UniDesk 核心业务后端、Dockerfile、GPU/训练调试必须在目标计算节点通过 SSH 透传或 k3s 控制面完成，Todo Note 这类明确写入主 server 的例外需单独登记，规则见 `docs/reference/microservices.md`。 @@ -73,6 +73,7 @@ UniDesk 是一个以主 server 为统一入口的分布式工作平台；本文 - `docs/reference/pipeline-oa-event-flow.md`：Pipeline/OA 事件流、审核/无审核流转、单步调试、甘特图渲染和最终去残留规则。 - `docs/reference/pipeline-model-proxy.md`：Pipeline v2 model proxy 链路架构、D601 宿主 proxy 服务部署、harness token 注入规则和 smoke test 验证流程。 - `docs/reference/deploy.md`：`deploy.json` desired-state、target-side build、一次性构建 proxy、直管/代管服务部署 executor 和 live commit 验证规则。 +- `docs/reference/dev-environment.md`：D601 `unidesk-dev` persistent dev 环境、18083 dev frontend proxy、`deploy apply --env dev` 服务范围和 Rust backend-core 只在 D601 编译的边界。 - `docs/reference/ci.md`：D601 k3s Tekton CI、只读主数据库性能门禁和 CLI 入口规则。 - `docs/reference/dev-ci-runner.md`：`ci run-dev-e2e` 的 Git 控制 runner、短 launcher、结果目录和 no-CD 边界。 - `docs/reference/codex-deploy.md`：D601 Code Queue 旧 `codex deploy ` 入口禁用原因、受控部署边界和后续 CD 目标行为。 diff --git a/TEST.md b/TEST.md index c6a17b14..9815e49b 100644 --- a/TEST.md +++ b/TEST.md @@ -10,7 +10,7 @@ ## T3 主 server 自接入 Provider Gateway -阅读 `AGENTS.md`（本项目 `AGENTS.md` 同时承担 `SKILL.md` 对 `scripts/cli.ts` 的解释职责），然后用 cli 手动测试以下内容：运行 `bun scripts/cli.ts server status`、`bun scripts/cli.ts server swap status` 和 `bun scripts/cli.ts debug health`，确认 `server status` 包含 `swap` 摘要，`server swap status` 快速返回 total memory、active swaps、`/etc/fstab` 持久化状态和 warning；面向浏览器的公网入口只有 frontend 与 provider ingress，backend-core 显示为 Docker 内部端口，database/OA Event Flow 若因 D601 Code Queue 映射宿主端口也必须显示为受限宿主端口，且 `network.restrictedHostAccess.allowedSourceCidrs` 已生成来源限制，`/api/nodes` 中存在 `main-server` provider，状态为 `online`，`/api/nodes/system-status` 中存在 CPU/内存/硬盘采样，`/api/nodes/docker-status` 中存在 `main-server` 的 Docker 快照，且 provider 标签中能看到 Docker socket 可用性。若 `swap.warning` 非空，先运行 `bun scripts/cli.ts server swap ensure --dry-run` 审查动作，再谨慎执行 `bun scripts/cli.ts server swap ensure --size 2GiB`，确认输出包含 `before`/`after`、`actions`、`errors` 和 `status=ok|degraded`；已有 swap 时 ensure 必须 no-op。 +阅读 `AGENTS.md`（本项目 `AGENTS.md` 同时承担 `SKILL.md` 对 `scripts/cli.ts` 的解释职责），然后用 cli 手动测试以下内容：运行 `bun scripts/cli.ts server status`、`bun scripts/cli.ts server swap status` 和 `bun scripts/cli.ts debug health`，确认 `server status` 包含 `swap` 摘要，`server swap status` 快速返回 total memory、active swaps、`/etc/fstab` 持久化状态和 warning；面向浏览器/Provider 的公网入口只有 production frontend、dev frontend proxy 与 provider ingress，backend-core 显示为 Docker 内部端口，database/OA Event Flow 若因 D601 Code Queue 映射宿主端口也必须显示为受限宿主端口，且 `network.restrictedHostAccess.allowedSourceCidrs` 已生成来源限制，`/api/nodes` 中存在 `main-server` provider，状态为 `online`，`/api/nodes/system-status` 中存在 CPU/内存/硬盘采样，`/api/nodes/docker-status` 中存在 `main-server` 的 Docker 快照，且 provider 标签中能看到 Docker socket 可用性。若 `swap.warning` 非空，先运行 `bun scripts/cli.ts server swap ensure --dry-run` 审查动作，再谨慎执行 `bun scripts/cli.ts server swap ensure --size 2GiB`，确认输出包含 `before`/`after`、`actions`、`errors` 和 `status=ok|degraded`；已有 swap 时 ensure 必须 no-op。 ## T4 前端控制台连通 @@ -26,7 +26,7 @@ ## T7 停止与端口释放 -阅读 `AGENTS.md`（本项目 `AGENTS.md` 同时承担 `SKILL.md` 对 `scripts/cli.ts` 的解释职责），然后用 cli 手动测试以下内容：运行 `bun scripts/cli.ts server stop`，确认立即返回 job id；等待 `bun scripts/cli.ts job status latest` 成功后运行 `bun scripts/cli.ts server status`，确认 frontend 与 provider ingress 公网端口不再监听，backend-core 没有宿主机端口映射，database 的受限映射不再可用，容器状态不再运行。 +阅读 `AGENTS.md`（本项目 `AGENTS.md` 同时承担 `SKILL.md` 对 `scripts/cli.ts` 的解释职责），然后用 cli 手动测试以下内容：运行 `bun scripts/cli.ts server stop`，确认立即返回 job id；等待 `bun scripts/cli.ts job status latest` 成功后运行 `bun scripts/cli.ts server status`，确认 production frontend、dev frontend proxy 与 provider ingress 公网端口不再监听，backend-core 没有宿主机端口映射，database 的受限映射不再可用，容器状态不再运行。 ## Issue 记录 @@ -111,6 +111,10 @@ 阅读 `AGENTS.md` 和 `docs/reference/ci.md`，运行 `bun scripts/cli.ts ci install`，确认 Tekton Pipelines `v1.12.0`、Tekton Triggers `v0.34.0` 和 `unidesk-ci` Pipeline/Task/EventListener 已部署到 D601 原生 k3s；随后运行 `bun scripts/cli.ts ci run --revision <已push的commitId> --wait-ms 1200000`，确认 PipelineRun 只执行 clone/check/performance，不调用 `deploy apply` 或 `codex deploy`，并确认临时 `code-queue-ci-read` 使用主 PostgreSQL 只读查询 Code Queue 首屏、TraceView summary、TraceView steps 和 step detail 的性能指标。若失败，使用 `bun scripts/cli.ts ci logs ` 查看 TaskRun 和 Pod 日志；交付说明必须记录性能预算是否通过。 +## T23C D601 Dev Environment And Rust Backend-Core + +阅读 `AGENTS.md` 和 `docs/reference/dev-environment.md`，运行 `bun scripts/cli.ts check --help`、`bun scripts/cli.ts deploy --help`、`bun scripts/cli.ts server rebuild --help` 和 `bun scripts/cli.ts ci run-dev-e2e --help`，确认全部快速返回 JSON 帮助且说明 Rust backend-core 不在 master server 编译；运行 `bun scripts/cli.ts check --files --scripts-typecheck --compose --logs`，确认本地非 Rust 检查通过；确认没有在 master server 执行 `cargo check/build/test` 或 `docker build src/components/backend-core/Dockerfile`。将已 push 的 backend-core/frontend commit 写入 `origin/master:deploy.json#environments.dev` 后，运行 `bun scripts/cli.ts deploy apply --env dev --service backend-core` 和 `bun scripts/cli.ts deploy apply --env dev --service frontend`，用 `bun scripts/cli.ts job status --tail-bytes 30000` 观察 D601 target-side fetch、Rust build、k3s image import、`unidesk-dev` apply、Deployment stamp 和 live commit health 验证；随后运行 `bun scripts/cli.ts server rebuild dev-frontend-proxy` 并确认 `server status` 输出 `urls.devFrontend=http://74.48.78.17:18083`，`curl -fsS http://127.0.0.1:18083/health` 成功；最后访问公网 `http://74.48.78.17:18083/` 手动体验 dev 版本，并运行 `bun scripts/cli.ts ci run --revision <已push的commitId> --wait-ms 1200000` 与 `bun scripts/cli.ts ci run-dev-e2e --wait-ms 600000` 完成 D601 CI 和 dev smoke 验证。 + ## T23B D601 Decision Center User Service 阅读 `AGENTS.md` 和 `docs/reference/microservices.md`，运行 `bun scripts/cli.ts microservice list`，确认 `decision-center` 显示为 `providerId=D601`、`public=false`、`frontendOnly=true`、仓库 URL `https://github.com/pikasTech/unidesk`、k3s/k8s `k3s://unidesk/decision-center:4277` 逻辑服务映射、`deployment.mode=k3sctl-managed`、`runtime.orchestrator=k3sctl` 且无业务直连容器摘要；运行 `bun scripts/cli.ts deploy apply --service decision-center --run-now`，确认命令在运行时变更前返回结构化错误，说明维护通道直连 D601 不承担服务部署；Decision Center 后续版本部署必须经未来受控 target-side CD 路径。随后运行 `bun scripts/cli.ts microservice health decision-center`，确认 `service=decision-center`、`storage=postgres`、`schemaReady=true` 且 health 中包含 `diaryEntryCount`；准备一份临时 Markdown 会议记录，运行 `bun scripts/cli.ts decision upload --title --type meeting --level G1 --status active --evidence <url>`，再运行 `bun scripts/cli.ts decision list` 和 `bun scripts/cli.ts decision show <id>`，确认 CLI 只通过 backend-core 用户服务代理访问，返回结构化 JSON 且能看到刚上传的记录。再准备一份包含 `# 2026年5月1日` 和 `# 2026年5月2日` 的临时工作日志 Markdown，运行 `bun scripts/cli.ts decision diary import <markdown-file> --source-file test-work-log.md --tag e2e`、`bun scripts/cli.ts decision diary months`、`bun scripts/cli.ts decision diary list --month 2026-05` 和 `bun scripts/cli.ts decision diary show 2026-05-01`，确认日记按 `YYYY-MM/YYYY-MM-DD.md` 虚拟路径拆分、写入 PostgreSQL 且重复导入幂等。最后登录公网 frontend `http://74.48.78.17:18081/`，进入 `用户服务 / Decision Center`，确认页面显示 G0/G1 目标、P0/P1 Blocker、停放事项、最近会议/决议、筛选、全部记录表和工作日记标签；日记标签可按月筛选并查看单日 Markdown 正文；页面不得提供聊天/LLM 会话窗口，默认不得裸 JSON，完整 JSON 只能通过 `查看原始JSON` 打开。 diff --git a/config.json b/config.json index 64587ef2..7d8a385a 100644 --- a/config.json +++ b/config.json @@ -18,6 +18,10 @@ "port": 18081, "containerPort": 8080 }, + "devFrontend": { + "port": 18083, + "containerPort": 8080 + }, "database": { "port": 15432, "containerPort": 5432 diff --git a/deploy.json b/deploy.json index 979128f6..2a0a5c39 100644 --- a/deploy.json +++ b/deploy.json @@ -86,11 +86,6 @@ "id": "frontend", "repo": "https://github.com/pikasTech/unidesk", "commitId": "b265274" - }, - { - "id": "code-queue", - "repo": "https://github.com/pikasTech/unidesk", - "commitId": "b265274" } ] } diff --git a/docker-compose.yml b/docker-compose.yml index 308808cd..5e91b63e 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -44,9 +44,10 @@ services: context: . dockerfile: src/components/backend-core/Dockerfile container_name: unidesk-backend-core - restart: always + restart: unless-stopped depends_on: - database + - code-queue-mgr ports: - "${UNIDESK_PROVIDER_INGRESS_PORT}:8081" expose: @@ -63,6 +64,7 @@ services: DATABASE_VOLUME_SIZE: "${UNIDESK_DATABASE_VOLUME_SIZE}" PGDATA_BACKUP_STAGING_DIR: "/data/baidu-netdisk-staging" BAIDU_NETDISK_INTERNAL_URL: "http://baidu-netdisk:4244" + CODE_QUEUE_MGR_INTERNAL_URL: "http://code-queue-mgr:4278" MICROSERVICES_JSON: "${UNIDESK_MICROSERVICES_JSON:-[]}" LOG_FILE: "/var/log/unidesk/${UNIDESK_LOG_DAY}/${UNIDESK_LOG_PREFIX}_backend-core.jsonl" UNIDESK_LOG_RETENTION_BYTES: "${UNIDESK_LOG_RETENTION_BYTES:-1GiB}" @@ -70,7 +72,38 @@ services: - ${UNIDESK_LOG_DIR}:/var/log/unidesk - ./.state/baidu-netdisk/staging:/data/baidu-netdisk-staging healthcheck: - test: ["CMD", "bun", "-e", "fetch('http://127.0.0.1:8080/health').then(r=>process.exit(r.ok?0:1)).catch(()=>process.exit(1))"] + test: ["CMD", "backend-core", "--fetch-json", "http://127.0.0.1:8080/health", "--require-ok"] + interval: 5s + timeout: 3s + retries: 20 + + code-queue-mgr: + image: code-queue-mgr + build: + context: . + dockerfile: src/components/microservices/code-queue-mgr/Dockerfile + container_name: code-queue-mgr-backend + restart: unless-stopped + depends_on: + - database + expose: + - "4278" + environment: + HOST: "0.0.0.0" + PORT: "4278" + DATABASE_URL: "postgres://${UNIDESK_DATABASE_USER}:${UNIDESK_DATABASE_PASSWORD}@database:5432/${UNIDESK_DATABASE_NAME}" + CODE_QUEUE_TRACE_DATABASE_URL: "${UNIDESK_CODE_QUEUE_TRACE_DATABASE_URL}" + CODE_QUEUE_MGR_DATABASE_POOL_MAX: "${UNIDESK_CODE_QUEUE_MGR_DATABASE_POOL_MAX:-2}" + CODE_QUEUE_TRACE_DATABASE_POOL_MAX: "${UNIDESK_CODE_QUEUE_TRACE_DATABASE_POOL_MAX:-1}" + CODE_QUEUE_MAIN_PROVIDER_ID: "${UNIDESK_CODE_QUEUE_MAIN_PROVIDER_ID:-D601}" + CODE_QUEUE_WORKDIR: "/workspace" + CODE_QUEUE_REMOTE_WORKDIR: "${UNIDESK_CODE_QUEUE_REMOTE_WORKDIR:-/home/ubuntu}" + LOG_FILE: "/var/log/unidesk/${UNIDESK_LOG_DAY}/${UNIDESK_LOG_PREFIX}_code-queue-mgr.jsonl" + UNIDESK_LOG_RETENTION_BYTES: "${UNIDESK_LOG_RETENTION_BYTES:-1GiB}" + volumes: + - ${UNIDESK_LOG_DIR}:/var/log/unidesk + healthcheck: + test: ["CMD", "bun", "-e", "fetch('http://127.0.0.1:4278/health').then(r=>process.exit(r.ok?0:1)).catch(()=>process.exit(1))"] interval: 5s timeout: 3s retries: 20 @@ -201,40 +234,6 @@ services: timeout: 3s retries: 20 - code-queue-mgr: - image: code-queue-mgr - build: - context: . - dockerfile: src/components/microservices/code-queue-mgr/Dockerfile - container_name: code-queue-mgr-backend - restart: unless-stopped - depends_on: - - database - expose: - - "4278" - environment: - HOST: "0.0.0.0" - PORT: "4278" - DATABASE_URL: "postgres://${UNIDESK_DATABASE_USER}:${UNIDESK_DATABASE_PASSWORD}@database:5432/${UNIDESK_DATABASE_NAME}" - CODE_QUEUE_MGR_DATABASE_POOL_MAX: "${UNIDESK_CODE_QUEUE_MGR_DATABASE_POOL_MAX:-2}" - CODE_QUEUE_TRACE_DATABASE_POOL_MAX: "${UNIDESK_CODE_QUEUE_TRACE_DATABASE_POOL_MAX:-1}" - CODE_QUEUE_MAIN_PROVIDER_ID: "${UNIDESK_CODE_QUEUE_DEV_CONTAINER_DEFAULT_PROVIDER_ID:-D601}" - CODE_QUEUE_REMOTE_WORKDIR: "${UNIDESK_CODE_QUEUE_REMOTE_WORKDIR:-/home/ubuntu}" - CODE_QUEUE_WORKDIR: "/workspace" - LOG_FILE: "/var/log/unidesk/${UNIDESK_LOG_DAY}/${UNIDESK_LOG_PREFIX}_code-queue-mgr.jsonl" - UNIDESK_LOG_RETENTION_BYTES: "${UNIDESK_LOG_RETENTION_BYTES:-1GiB}" - UNIDESK_CODE_QUEUE_MGR_DEPLOY_SERVICE_ID: "${UNIDESK_CODE_QUEUE_MGR_DEPLOY_SERVICE_ID:-code-queue-mgr}" - UNIDESK_CODE_QUEUE_MGR_DEPLOY_REPO: "${UNIDESK_CODE_QUEUE_MGR_DEPLOY_REPO:-}" - UNIDESK_CODE_QUEUE_MGR_DEPLOY_COMMIT: "${UNIDESK_CODE_QUEUE_MGR_DEPLOY_COMMIT:-}" - UNIDESK_CODE_QUEUE_MGR_DEPLOY_REQUESTED_COMMIT: "${UNIDESK_CODE_QUEUE_MGR_DEPLOY_REQUESTED_COMMIT:-}" - volumes: - - ${UNIDESK_LOG_DIR}:/var/log/unidesk - healthcheck: - test: ["CMD", "code-queue-mgr", "--healthcheck"] - interval: 5s - timeout: 3s - retries: 20 - frontend: build: context: . @@ -254,10 +253,6 @@ services: AUTH_PASSWORD: "${UNIDESK_AUTH_PASSWORD}" SESSION_SECRET: "${UNIDESK_SESSION_SECRET}" SESSION_TTL_SECONDS: "${UNIDESK_SESSION_TTL_SECONDS}" - UNIDESK_DEPLOY_SERVICE_ID: "${UNIDESK_FRONTEND_DEPLOY_SERVICE_ID:-frontend}" - UNIDESK_DEPLOY_REPO: "${UNIDESK_FRONTEND_DEPLOY_REPO:-}" - UNIDESK_DEPLOY_COMMIT: "${UNIDESK_FRONTEND_DEPLOY_COMMIT:-}" - UNIDESK_DEPLOY_REQUESTED_COMMIT: "${UNIDESK_FRONTEND_DEPLOY_REQUESTED_COMMIT:-}" LOG_FILE: "/var/log/unidesk/${UNIDESK_LOG_DAY}/${UNIDESK_LOG_PREFIX}_frontend.jsonl" UNIDESK_LOG_RETENTION_BYTES: "${UNIDESK_LOG_RETENTION_BYTES:-1GiB}" volumes: @@ -268,6 +263,22 @@ services: timeout: 3s retries: 20 + dev-frontend-proxy: + build: + context: . + dockerfile: src/components/dev-frontend-proxy/Dockerfile + container_name: unidesk-dev-frontend-proxy + restart: unless-stopped + depends_on: + - backend-core + ports: + - "${UNIDESK_DEV_FRONTEND_PORT}:8080" + healthcheck: + test: ["CMD", "wget", "-qO-", "http://127.0.0.1:8080/health"] + interval: 10s + timeout: 5s + retries: 12 + provider-gateway: image: unidesk_provider-gateway build: diff --git a/docs/reference/arch.md b/docs/reference/arch.md index 92a00428..0bba4a8a 100644 --- a/docs/reference/arch.md +++ b/docs/reference/arch.md @@ -25,7 +25,7 @@ - Run all user services as Docker containers; these user-facing services are mounted onto the UniDesk core and the core can still run without them - Includes frontend gateway, task scheduler, project management, provider ingress, and other stateless modules - Instances can scale horizontally; failure recovery requires no state synchronization - - Only the frontend gateway and provider ingress are public; core REST APIs and PostgreSQL remain on the Docker internal network + - Only the production frontend gateway, dev frontend proxy and provider ingress are unrestricted public entries; core REST APIs and PostgreSQL remain on the Docker internal network or explicitly restricted host mappings. The dev frontend proxy rule is owned by `docs/reference/dev-environment.md`. - Frontend Time Zone Policy - All UniDesk frontend timestamps, dates, clocks, update times, heartbeat times, Trace times, Gantt axis labels, export date stamps, and `datetime-local` values must render as Beijing time. - Beijing time means IANA timezone `Asia/Shanghai` / UTC+8, regardless of the browser timezone, host system timezone, container timezone, or server-side `project.timezone` value. diff --git a/docs/reference/ci.md b/docs/reference/ci.md index 619ff294..3f0ea9b0 100644 --- a/docs/reference/ci.md +++ b/docs/reference/ci.md @@ -10,6 +10,7 @@ UniDesk CI is hosted on the D601 native k3s cluster with Tekton Pipelines and Te - Manifests: `src/components/microservices/k3sctl-adapter/k3s/ci/`. - CLI entry: `bun scripts/cli.ts ci install|status|run|run-dev-e2e|logs`. - Dev namespace e2e runner: `bun scripts/cli.ts ci run-dev-e2e`; authoritative runner path, manifest contract and safety boundary are in `docs/reference/dev-ci-runner.md`. +- Rust backend-core check/build boundary: CI may run `UNIDESK_D601_RUST_CHECK=1 bun scripts/cli.ts check --full --rust` on D601; the master server must not compile Rust for backend-core iteration. The authoritative dev environment rule is `docs/reference/dev-environment.md`. ## Pipeline Scope @@ -17,7 +18,7 @@ Each commit CI run performs: - `git clone` and checkout of the requested repository revision. - `bun install --frozen-lockfile` at the repo root and `src/`, because `bun scripts/cli.ts check` compiles all `src/components` and needs the component workspace lockfile for frontend React dependencies. -- `bun scripts/cli.ts check`. +- `UNIDESK_D601_RUST_CHECK=1 bun scripts/cli.ts check --full --rust`, so Rust backend-core is checked only inside the D601 CI execution boundary. - Temporary `code-queue-ci-read` Deployment and ClusterIP Service in `unidesk-ci`. - Code Queue read performance checks against the production PostgreSQL through `d601-tcp-egress-gateway`. @@ -93,4 +94,4 @@ bun scripts/cli.ts ci logs <runId> ## Trigger Boundary -`unidesk-ci.triggers.yaml` installs the EventListener, TriggerBinding and TriggerTemplate, but the EventListener remains a normal in-cluster Service. Do not expose it through NodePort, LoadBalancer or an unrestricted public ingress. If GitHub or another Git remote needs webhook delivery, add a UniDesk-controlled frontend/backend route with secret verification and then proxy to the EventListener; keep frontend and provider ingress as the only unrestricted public entry points. +`unidesk-ci.triggers.yaml` installs the EventListener, TriggerBinding and TriggerTemplate, but the EventListener remains a normal in-cluster Service. Do not expose it through NodePort, LoadBalancer or an unrestricted public ingress. If GitHub or another Git remote needs webhook delivery, add a UniDesk-controlled frontend/backend route with secret verification and then proxy to the EventListener; keep only the documented main-server public entrypoints: production frontend, dev frontend proxy and provider ingress. The dev frontend public port is defined in `docs/reference/dev-environment.md`. diff --git a/docs/reference/cli.md b/docs/reference/cli.md index c13a3ac8..cc7aafdc 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -7,13 +7,13 @@ UniDesk 的统一 CLI 入口是根目录 `scripts/cli.ts`，运行方式固定 - `help` 输出命令索引，适合作为交互式入口。 - `--main-server-ip <ip> <command>` 默认通过公网 frontend 登录态调用主 server 的同源 API 代理，不要求计算节点持有主 server SSH key；显式提供 `--main-server-key` 或 `--main-server-transport ssh` 时才使用旧 SSH 传输。 - `config show` 读取并校验根目录 `config.json`，不从环境变量、默认值或隐藏文件静默补配置。 -- `check` 默认只执行轻量配置校验、Bun 版本检查和 Bun Transpiler 语法解析（覆盖 CLI 入口、主要 `scripts/` 模块和核心组件入口，不做类型推导）；关键文件存在性、`scripts/` TypeScript 类型检查、`src/components/` TypeScript 类型检查、Docker Compose config 和日志轮转策略扫描默认不启用，分别通过 `--files`、`--scripts-typecheck`、`--components`、`--compose`、`--logs` 开启，或用 `--full` 一次性开启。 +- `check` 默认只执行轻量配置校验、Bun 版本检查和 Bun Transpiler 语法解析（覆盖 CLI 入口、主要 `scripts/` 模块和核心组件入口，不做类型推导）；关键文件存在性、`scripts/` TypeScript 类型检查、`src/components/` TypeScript 类型检查、Docker Compose config 和日志轮转策略扫描默认不启用，分别通过 `--files`、`--scripts-typecheck`、`--components`、`--compose`、`--logs` 开启，或用 `--full` 一次性开启。`--rust` 只允许在 D601 CI/dev execution 中配合 `UNIDESK_D601_RUST_CHECK=1` 使用，长期规则见 `docs/reference/dev-environment.md`。 - `server start` 创建异步 job，在后台执行 Docker 构建和启动；命令本身只负责返回 job id、日志路径和启动命令。 - `server stop` 创建异步 job，在后台停止固定 Compose project 中的全部 UniDesk 服务。 -- `server status` 查询公开端口、受限宿主端口、内部端口、主机 swap 摘要、Compose 容器、core/frontend/provider/database 健康检查和访问 URL；D601 Code Queue 使用的 PostgreSQL/OA Event Flow host mapping 必须出现在受限宿主端口而不是无条件公开入口中。低内存主 server 上 `swap.warning` 非空时，先执行 `server swap status` 或 `server swap ensure`。 +- `server status` 查询公开端口、受限宿主端口、内部端口、主机 swap 摘要、Compose 容器、core/frontend/dev-frontend/provider/database 健康检查和访问 URL；D601 Code Queue 使用的 PostgreSQL/OA Event Flow host mapping 必须出现在受限宿主端口而不是无条件公开入口中。低内存主 server 上 `swap.warning` 非空时，先执行 `server swap status` 或 `server swap ensure`。 - `server swap status|ensure [--path /swapfile] [--size 2GiB] [--dry-run]` 是主 server swap 管理入口。`status` 仅读 `/proc/meminfo`、`/proc/swaps` 和 `/etc/fstab` 并返回 JSON；`ensure` 在已有任何 active swap 时只报告 no-op，在无 active swap 时创建固定 swapfile、`chmod 600`、`mkswap`、`swapon` 并尽量写入 `/etc/fstab`。输出必须包含 `before`、`after`、total memory、active swap、持久化状态、关键动作和错误详情；若 swap 已启用但 fstab 写入失败，状态为 `degraded`，调用者需按返回的 detail 修复持久化。 - `server logs` 返回 `logs/` 文件日志和 Docker 容器日志的尾部，默认限制输出大小，避免日志爆炸。实现必须只读取文件末尾字节，不得为了 tail 先把巨大日志完整读入 CLI 内存。 -- `server rebuild <backend-core|frontend|provider-gateway|todo-note|code-queue-mgr|project-manager|baidu-netdisk|oa-event-flow>` 创建异步 job，先构建目标服务镜像，随后在 `.state/locks/server-compose.lock` 串行保护下用 `--no-deps --force-recreate` 替换目标 service 并等待容器 `healthy/running`；该命令用于替代手工删除容器的兜底流程，其中 `todo-note`、`code-queue-mgr`、`project-manager`、`baidu-netdisk` 和 `oa-event-flow` 只重建主 server 承载的对应后端，不会重建或删除 database 命名卷。D601 Code Queue 执行面不由 `server rebuild` 管理。 +- `server rebuild <backend-core|frontend|dev-frontend-proxy|provider-gateway|todo-note|code-queue-mgr|project-manager|baidu-netdisk|oa-event-flow>` 创建异步 job，先构建目标服务镜像，随后在 `.state/locks/server-compose.lock` 串行保护下用 `--no-deps --force-recreate` 替换目标 service 并等待容器 `healthy/running`；该命令用于替代手工删除容器的兜底流程，其中 `dev-frontend-proxy` 只更新主 server dev 入口薄代理，`todo-note`、`code-queue-mgr`、`project-manager`、`baidu-netdisk` 和 `oa-event-flow` 只重建主 server 承载的对应后端，不会重建或删除 database 命名卷。D601 Code Queue 执行面不由 `server rebuild` 管理，Rust backend-core 迭代不得用 `server rebuild backend-core` 在 master server 编译，规则见 `docs/reference/dev-environment.md`。 - `provider attach <providerId> [--master-server URL] [--up] [--force]` 在新计算节点生成两项配置的 provider-gateway 挂载包：`.state/provider-<ID>.env` 默认只包含 `UNIDESK_MASTER_SERVER` 与 `PROVIDER_ID`，`provider-<ID>.yml` 固定 Docker socket、`pid: "host"`、`restart: always`、只读 `/workspace` 和 SSH 维护私钥挂载；`--up` 会立即执行生成的 `docker compose up -d --build`。 - `ssh <providerId> [ssh-like args...]` 通过 backend-core 内网 WebSocket broker 和 provider-gateway 的 Host SSH / WSL SSH 维护桥连接目标节点；无后续参数时进入远端登录 shell，有后续参数时按 ssh 远端命令体验执行并返回远端 exit code。 - `ssh <providerId> apply-patch [tool args...] < patch.diff` 直接调用远端注入的 `apply_patch` 工具，并把本地 stdin 中的标准 `*** Begin Patch` / `*** End Patch` patch 流透传给目标节点。 @@ -22,7 +22,7 @@ UniDesk 的统一 CLI 入口是根目录 `scripts/cli.ts`，运行方式固定 - `microservice list/status/health/diagnostics/tunnel-self-test/proxy` 通过 backend-core 内网 API 管理挂载在计算节点 Docker 或 k3s 控制面中的用户服务（底层命令名仍为 microservice）；`health`、`diagnostics`、`tunnel-self-test` 和 `proxy` 会走真实 backend-core -> provider-gateway 或 k3sctl-adapter -> 节点服务链路，`proxy` 支持受控 JSON 请求体并对超大响应 body 默认输出有界预览，规则见 `docs/reference/microservices.md`。 - `decision upload/list/show/health` 通过 backend-core 用户服务代理访问 D601 k3s Decision Center，用于上传会议记录/决议 Markdown、列出权威记录、查看详情和健康检查；它不得直连 D601 Service、NodePort 或 provider-gateway 业务 HTTP。 - `decision diary import <markdown-file>` 将带 `# YYYY年M月D日`、`# YYYY-MM-DD` 或 `# YYYY/M/D` 标题的工作日志拆成每天一篇 Markdown 日记，按 `YYYY-MM/YYYY-MM-DD.md` 虚拟路径写入 Decision Center PostgreSQL；`decision diary list/months/show` 分别用于按月/日期查询、列出月份和查看单日正文。 -- `deploy check/plan/apply` 默认从根目录 `deploy.json` 读取服务 repo 与 commit 期望状态，join `config.json` 和现有 manifest 后使用 target-side build 单一路径校验或更新已支持目标；`deploy plan --env dev|prod` 只从 `origin/master:deploy.json#environments.<env>` 读取 manifest 并输出 dry-run 环境计划，不使用本地 dirty worktree；当前 `deploy apply --env dev` 不做 D601 服务 rollout，dev desired-state smoke 使用 `ci run-dev-e2e`；规则见 `docs/reference/deploy.md` 和 `docs/reference/dev-ci-runner.md`。 +- `deploy check/plan/apply` 默认从根目录 `deploy.json` 读取服务 repo 与 commit 期望状态，join `config.json` 和现有 manifest 后使用 target-side build 单一路径校验或更新已支持目标；`deploy plan --env dev|prod` 只从 `origin/master:deploy.json#environments.<env>` 读取 manifest 并输出 dry-run 环境计划，不使用本地 dirty worktree；当前 `deploy apply --env dev` 只支持 D601 `backend-core` 与 `frontend` persistent dev rollout，dev desired-state smoke 使用 `ci run-dev-e2e`；规则见 `docs/reference/deploy.md`、`docs/reference/dev-environment.md` 和 `docs/reference/dev-ci-runner.md`。 - `dev-env validate [--manifest path] [--kubectl-dry-run]` 离线校验 D601 `unidesk-dev` namespace、dev PostgreSQL 底座和 dev workload manifest。默认检查 `src/components/microservices/k3sctl-adapter/k3s/dev/unidesk-dev-foundation.k8s.yaml`；也可显式校验 `src/components/microservices/k3sctl-adapter/k3s/dev/unidesk-dev-core.k8s.yaml` 或 `src/components/microservices/k3sctl-adapter/k3s/dev/unidesk-dev-code-queue.k8s.yaml`。所有 namespaced 对象必须只落到 `unidesk-dev`，foundation manifest 必须包含 `postgres-dev` StatefulSet/Service、dev secret/config、迁移 Job 和 DB URL guard，core manifest 必须包含 `backend-core-dev`/`frontend-dev` Deployment/Service，Code Queue dev manifest 必须包含 `code-queue-scheduler-dev`、`code-queue-read-dev`、`code-queue-write-dev` 和 dev provider egress proxy。加 `--kubectl-dry-run` 时额外执行 `kubectl apply --dry-run=client --validate=false -f <manifest>`，仍不 apply 资源。 - `dev-env prewarm-images [--image image] [--provider-id D601] [--no-pull] [--proxy-url URL] [--pull-timeout-ms N] [--dry-run]` 创建异步 job，通过 UniDesk SSH 维护桥在 D601 上把开发底座依赖镜像从 Docker 缓存导入原生 k3s containerd。默认镜像是 `postgres:16-alpine` 和 `rancher/mirrored-library-busybox:1.36.1`，用于避免 `postgres-dev` 与 local-path helper pod 卡在外部 registry 拉取。该命令固定验证 `/etc/rancher/k3s/k3s.yaml` 指向的 native k3s 上下文，并输出 `dev_env_containerd_image_ready=...` 作为成功判据；它不 apply manifest、不修改生产 `unidesk` namespace。 - `ci install|status|run|run-dev-e2e|logs` 管理 D601 原生 k3s 上的 Tekton CI。`run` 手动创建每 commit 检查和 Code Queue 只读性能门禁；`run-dev-e2e` 的 Git 控制 runner、短 launcher、host fetch 边界、临时 smoke namespace 和 no-CD 规则只在 `docs/reference/dev-ci-runner.md` 定义；Tekton CI 通用规则见 `docs/reference/ci.md`。 @@ -37,15 +37,15 @@ UniDesk 的统一 CLI 入口是根目录 `scripts/cli.ts`，运行方式固定 - 所有 `codex` 查询和管理命令必须走与 WebUI 相同的 backend-core 私有代理路径 `/api/microservices/code-queue/proxy/...`；CLI 不得为了提交、移动、中断、取消或队列管理直接调用 D601 内部 Service、数据库、pod curl 或 k3sctl scheduler 子服务。若该路径失败，应先修复 CLI/backend/provider tunnel 链路，而不是绕过控制面。 - `job list [--limit N] [--include-command]` 与 `job status <jobId|latest> [--tail-bytes N]` 查询 `.state/jobs/` 文件系统状态，是异步命令的可观测入口。`job list` 默认只返回最新 50 条摘要；`job status` 默认只返回 stdout/stderr 末尾 12000 字节，并带 `tailPolicy` 与完整日志路径。 - `debug health`、`debug dispatch` 与 `debug task` 走真实内部 core、WebSocket、数据库、provider、系统指标、Docker 状态和 Host SSH 维护桥流程，只用于开发调试，不写入 `TEST.md` 的正式验收步骤。 -- `e2e run [--only pattern[,pattern...]] [--skip pattern[,pattern...]]` 使用 publicHost 派生的公开 frontend/provider ingress URL，并通过 Docker 内网验证 core API、PostgreSQL、provider self-connection、系统指标曲线、Docker 状态快照、provider.upgrade 预检和 Playwright 前端页面，是交付前的自动化 E2E 门禁；CLI 默认输出 check 状态摘要，完整诊断写入 `resultPath`，日常迭代应优先用 `--only` / `--skip` 跑最小必要集合。 +- `e2e run [--only pattern[,pattern...]] [--skip pattern[,pattern...]]` 使用 publicHost 派生的公开 production frontend/dev frontend/provider ingress URL，并通过 Docker 内网验证 core API、PostgreSQL、provider self-connection、系统指标曲线、Docker 状态快照、provider.upgrade 预检和 Playwright 前端页面，是交付前的自动化 E2E 门禁；CLI 默认输出 check 状态摘要，完整诊断写入 `resultPath`，日常迭代应优先用 `--only` / `--skip` 跑最小必要集合。 ## Async Job State 长时操作采用 Fire-and-Forget 模式：CLI 创建 `.state/jobs/{jobId}.json`，后台进程执行真实命令，并将 stdout、stderr 分别写入 `.state/jobs/{jobId}.stdout.log` 与 `.state/jobs/{jobId}.stderr.log`。调用者通过 `bun scripts/cli.ts job status <jobId>` 查询进度和尾部输出。 -`server rebuild` 与 `server start`、`server stop` 一样必须通过返回的 job id 确认结果；不要把连续 `server rebuild` 命令理解成“前一个重建已完成”，因为两个命令只是在快速创建异步 job。重建 frontend 的标准流程是运行 `bun scripts/cli.ts server rebuild frontend`，随后轮询 `bun scripts/cli.ts job status <jobId>` 到 `succeeded`，再用 `server status` 或 `e2e run` 验证公网 frontend；重建 Todo Note 后端使用 `bun scripts/cli.ts server rebuild todo-note`，随后用 `microservice health todo-note` 和 `microservice proxy todo-note /api/instances` 验证；重建 Code Queue Manager 使用 `bun scripts/cli.ts server rebuild code-queue-mgr`，随后用 `microservice health code-queue-mgr`、`microservice health code-queue` 和 `codex submit --dry-run` 验证主 server 控制面路径；重建 Project Manager 后端使用 `bun scripts/cli.ts server rebuild project-manager`，随后用 `microservice health project-manager` 和 `microservice proxy project-manager /api/projects` 验证；重建 Baidu Netdisk 后端使用 `bun scripts/cli.ts server rebuild baidu-netdisk`，随后用 `microservice health baidu-netdisk` 和 `microservice proxy baidu-netdisk /api/transfers` 验证；重建 OA Event Flow 后端使用 `bun scripts/cli.ts server rebuild oa-event-flow`，随后用 `microservice health oa-event-flow` 和 `microservice proxy oa-event-flow /api/diagnostics` 验证。D601 Code Queue 执行面和 Decision Center 后端由 D601 k3s/k8s 控制面代管，但当前不得通过维护通道直连 D601 做部署；正式 CD 控制路径另行设计，当前只允许 `ci run-dev-e2e` 启动一次性 smoke runner。不得把 `docker rm` 手工兜底当成正式交付步骤。 +`server rebuild` 与 `server start`、`server stop` 一样必须通过返回的 job id 确认结果；不要把连续 `server rebuild` 命令理解成“前一个重建已完成”，因为两个命令只是在快速创建异步 job。重建 frontend 的标准流程是运行 `bun scripts/cli.ts server rebuild frontend`，随后轮询 `bun scripts/cli.ts job status <jobId>` 到 `succeeded`，再用 `server status` 或 `e2e run` 验证公网 frontend；重建 dev 入口薄代理使用 `bun scripts/cli.ts server rebuild dev-frontend-proxy`，随后验证 `server status` 的 `urls.devFrontend` 和 `http://127.0.0.1:18083/health`；重建 Todo Note 后端使用 `bun scripts/cli.ts server rebuild todo-note`，随后用 `microservice health todo-note` 和 `microservice proxy todo-note /api/instances` 验证；重建 Code Queue Manager 使用 `bun scripts/cli.ts server rebuild code-queue-mgr`，随后用 `microservice health code-queue-mgr`、`microservice health code-queue` 和 `codex submit --dry-run` 验证主 server 控制面路径；重建 Project Manager 后端使用 `bun scripts/cli.ts server rebuild project-manager`，随后用 `microservice health project-manager` 和 `microservice proxy project-manager /api/projects` 验证；重建 Baidu Netdisk 后端使用 `bun scripts/cli.ts server rebuild baidu-netdisk`，随后用 `microservice health baidu-netdisk` 和 `microservice proxy baidu-netdisk /api/transfers` 验证；重建 OA Event Flow 后端使用 `bun scripts/cli.ts server rebuild oa-event-flow`，随后用 `microservice health oa-event-flow` 和 `microservice proxy oa-event-flow /api/diagnostics` 验证。D601 Code Queue 执行面和 Decision Center 后端由 D601 k3s/k8s 控制面代管；persistent dev backend-core/frontend 只通过 `deploy apply --env dev --service ...`，当前 Code Queue/Decision Center 仍不得通过维护通道直连 D601 做部署。不得把 `docker rm` 手工兜底当成正式交付步骤。 -新部署入口优先使用 `deploy apply`，但当前 D601 维护直连 apply 不承担服务部署。旧的 `codex deploy` 已禁用；后续 Code Queue、Decision Center、backend-core dev、frontend dev 等 D601 服务部署应收敛到一条受控 target-side CD 路径：从 remote commit 导出源码，在目标节点一次性代理构建镜像，部署后用 live commit 校验证明不是旧服务。 +新部署入口优先使用 `deploy apply`。`deploy apply --env dev --service backend-core|frontend` 已收敛到 D601 target-side dev 路径；旧的 `codex deploy` 已禁用；后续 Code Queue、Decision Center 等 D601 服务部署应另行收敛到同一类受控 target-side CD 路径：从 remote commit 导出源码，在目标节点一次性代理构建镜像，部署后用 live commit 校验证明不是旧服务。 ## Output Contract @@ -59,11 +59,11 @@ UniDesk 的统一 CLI 入口是根目录 `scripts/cli.ts`，运行方式固定 ## Debug Contract -`debug` 子命令必须复用真实模块与真实端点，禁止维护平行实现。`debug health` 会摘要展示 `/api/nodes/system-status` 和 `/api/nodes/docker-status`，避免输出完整快照造成信息爆炸。`debug dispatch` 会通过 frontend 容器的 Docker 内网访问 backend-core `/api/dispatch`，core 再通过 WebSocket 将 `docker.ps`、`provider.upgrade`、`host.ssh`、`microservice.http` 或 `echo` 任务下发给 provider gateway，因此它可以验证核心调度闭环，同时不需要公开 core REST API，也不要求 backend-core 运行镜像携带 Bun CLI。`provider.upgrade` 默认使用 `mode: "plan"` 预检；需要验证一键升级时必须显式加 `--mode schedule`，并通过 `--wait-ms` 或 `debug task` 确认任务进入 `succeeded`、result 中包含 updater 容器信息和 `policy: "always-enabled"`。`host.ssh` 默认使用 `mode: "probe"` 做短超时维护桥自检；需要执行明确命令时使用 `--ssh-command` 进入 `mode: "exec"`，并配合 `--wait-ms` 和 `debug task` 查看 stdout、stderr、exitCode 与 probeLine。`microservice.http` 只用于开发调试 provider-gateway 私有 HTTP 代理，正式用户入口应使用 `microservice` CLI 或 frontend 的用户服务页面。 +`debug` 子命令必须复用真实模块与真实端点，禁止维护平行实现。`debug health` 会摘要展示 `/api/nodes/system-status` 和 `/api/nodes/docker-status`，避免输出完整快照造成信息爆炸。`debug dispatch` 会通过 backend-core 容器内置 helper 访问 backend-core `/api/dispatch`，core 再通过 WebSocket 将 `docker.ps`、`provider.upgrade`、`host.ssh`、`microservice.http` 或 `echo` 任务下发给 provider gateway，因此它可以验证核心调度闭环，同时不需要公开 core REST API，也不要求 frontend 容器携带调试 broker。`provider.upgrade` 默认使用 `mode: "plan"` 预检；需要验证一键升级时必须显式加 `--mode schedule`，并通过 `--wait-ms` 或 `debug task` 确认任务进入 `succeeded`、result 中包含 updater 容器信息和 `policy: "always-enabled"`。`host.ssh` 默认使用 `mode: "probe"` 做短超时维护桥自检；需要执行明确命令时使用 `--ssh-command` 进入 `mode: "exec"`，并配合 `--wait-ms` 和 `debug task` 查看 stdout、stderr、exitCode 与 probeLine。`microservice.http` 只用于开发调试 provider-gateway 私有 HTTP 代理，正式用户入口应使用 `microservice` CLI 或 frontend 的用户服务页面。 ## SSH Command -`ssh <providerId> [ssh-like args...]` 是面向人的终端透传入口，不包装 JSON 输出。CLI 会在宿主机启动一个 `docker exec -i unidesk-frontend bun -e ...` broker，broker 只连接 backend-core 的 Docker 内网 `/ws/ssh`，core 再把 stdin/stdout/stderr 流量通过目标 provider 的既有 WebSocket 转发到 provider-gateway，provider-gateway 最终执行维护用 SSH 连接宿主或 WSL sshd。TTY 策略固定为交互登录 shell 使用 `ssh -tt`，带远端命令的会话使用 `ssh -T`；脚本 stdin、`apply-patch` 和 `py` 这类命令模式不得被伪终端回显或注入控制字符。该入口不新增 core 公网端口，不暴露 database，也不改变 frontend/provider ingress 之外的公网边界。 +`ssh <providerId> [ssh-like args...]` 是面向人的终端透传入口，不包装 JSON 输出。CLI 会在宿主机启动 `docker exec -i unidesk-backend-core backend-core --ssh-broker ...`，broker 只连接 backend-core 的 Docker 内网 `/ws/ssh`，core 再把 stdin/stdout/stderr 流量通过目标 provider 的既有 WebSocket 转发到 provider-gateway，provider-gateway 最终执行维护用 SSH 连接宿主或 WSL sshd。TTY 策略固定为交互登录 shell 使用 `ssh -tt`，带远端命令的会话使用 `ssh -T`；脚本 stdin、`apply-patch` 和 `py` 这类命令模式不得被伪终端回显或注入控制字符。该入口不新增 core 公网端口，不暴露 database，也不改变 frontend/dev frontend/provider ingress 之外的公网边界。 `bun scripts/cli.ts ssh --help` 和 `bun scripts/cli.ts ssh <providerId> --help` 是本地 JSON 帮助命令，必须快速返回；不能把 `--help` 解析成 Provider ID，不能打开交互 shell，也不能等待 provider 会话。 diff --git a/docs/reference/codex-deploy.md b/docs/reference/codex-deploy.md index 3fbec6c0..bb45d7be 100644 --- a/docs/reference/codex-deploy.md +++ b/docs/reference/codex-deploy.md @@ -2,7 +2,7 @@ `bun scripts/cli.ts codex deploy <commitId>` 是旧兼容入口，现已禁用。原因是它会通过 backend-core `host.ssh` 维护通道直连 D601 部署 Code Queue，把维护入口扩张成第二套部署系统。 -Code Queue 后续正式部署必须走一条受控 target-side CD 路径：读取 `origin/master:deploy.json#environments.dev` 或生产 desired-state，在目标节点执行 source fetch、build、k3s image import、rollout、stamp 和 live commit 验证。当前阶段不提供 Code Queue CD；只提供 `ci run-dev-e2e` dev smoke runner，规则见 `docs/reference/dev-ci-runner.md`。 +Code Queue 后续正式部署必须走一条受控 target-side CD 路径：读取 `origin/master:deploy.json#environments.dev` 或生产 desired-state，在目标节点执行 source fetch、build、k3s image import、rollout、stamp 和 live commit 验证。当前阶段不提供 Code Queue CD；persistent dev apply 只支持 backend-core/frontend，规则见 `docs/reference/dev-environment.md`，Code Queue smoke 仍通过 `ci run-dev-e2e`，规则见 `docs/reference/dev-ci-runner.md`。 ## Command diff --git a/docs/reference/config.md b/docs/reference/config.md index c73e0080..f9924658 100644 --- a/docs/reference/config.md +++ b/docs/reference/config.md @@ -4,11 +4,11 @@ ## Runtime -TypeScript 运行时固定为 Bun。根目录 CLI、backend-core、frontend 和 provider-gateway 都直接运行 `.ts` 入口；Docker 镜像使用 `oven/bun` 基础镜像，本机命令使用 `bun scripts/cli.ts`。 +TypeScript 运行时固定为 Bun for CLI, frontend, provider-gateway and TypeScript user services. backend-core is a Rust service; Rust build/check must follow the D601 boundary in `docs/reference/dev-environment.md`. 本机命令使用 `bun scripts/cli.ts`。 ## Network Ports -`config.json` 中保留 core、frontend、database 和 providerIngress 的端口字段。frontend 与 providerIngress 是面向浏览器/Provider 的公开入口；core 不允许映射公网。database 的 `port` 字段用于 D601 Code Queue 的受限 PostgreSQL 端口映射和公网阻断测试，必须配合 `network.restrictedHostAccess` 的来源白名单限制到 D601 出口地址，不能作为任意公网数据库入口。 +`config.json` 中保留 core、frontend、devFrontend、database 和 providerIngress 的端口字段。frontend、devFrontend 与 providerIngress 是面向浏览器/Provider 的公开入口；devFrontend 是 D601 `unidesk-dev` 的薄代理入口，权威规则见 `docs/reference/dev-environment.md`。core 不允许映射公网。database 的 `port` 字段用于 D601 Code Queue 的受限 PostgreSQL 端口映射和公网阻断测试，必须配合 `network.restrictedHostAccess` 的来源白名单限制到 D601 出口地址，不能作为任意公网数据库入口。 ## Auth diff --git a/docs/reference/deploy.md b/docs/reference/deploy.md index fb95ccd9..293a9675 100644 --- a/docs/reference/deploy.md +++ b/docs/reference/deploy.md @@ -2,6 +2,8 @@ UniDesk deployment is driven by a desired-state manifest. The manifest answers only one question: which service should run which repository commit. Runtime topology, ports, providers, compose files, Kubernetes manifests, health paths and proxy policy remain in `config.json` and the existing service manifests. +Persistent D601 dev environment rules, including the public dev frontend port, `deploy apply --env dev` service scope and Rust backend-core build boundary, are owned by `docs/reference/dev-environment.md`. This document owns the generic desired-state reconciler and target-side build contract. + ## Manifest The root `deploy.json` is the single desired-state source for both prod and dev. Environment branches such as `deploy/dev` and `deploy/prod` are deprecated because they create a second control plane for version intent. @@ -38,13 +40,15 @@ The root `deploy.json` is the single desired-state source for both prod and dev. The optional non-service execution declaration under `environments.dev` is intentionally not specified here. The only currently allowed declaration is `ci`, and its authoritative `repo`, `scriptPath`, `timeoutMs`, short launcher, host fetch boundary and no-CD rules are defined only in `docs/reference/dev-ci-runner.md`. -Environment mode never reads the local dirty working tree manifest. `deploy check --env ...`, `deploy plan --env ...` and `deploy apply --env ...` fetch `origin/master`, read `origin/master:deploy.json`, select `environments.<env>`, and report the manifest commit/blob, service commit IDs, target namespace, database fingerprint and Provider identity. Maintenance-channel direct D601 apply is intentionally narrow: no D601 backend-core/frontend/code-queue/Decision Center/k3sctl-adapter service deployment may use that path. `deploy apply --env prod` remains disabled until the production environment executor and authorization policy are explicitly added. +Environment mode never reads the local dirty working tree manifest. `deploy check --env ...`, `deploy plan --env ...` and `deploy apply --env ...` fetch `origin/master`, read `origin/master:deploy.json`, select `environments.<env>`, and report the manifest commit/blob, service commit IDs, target namespace, database fingerprint and Provider identity. `deploy apply --env dev` is currently enabled only for persistent D601 dev `backend-core` and `frontend`; all other D601 services remain rejected before runtime mutation. `deploy apply --env prod` remains disabled until the production environment executor and authorization policy are explicitly added. `config.json.microservices[].repository.commitId` is retained for catalog compatibility, but `deploy.json` is the deployment version authority for the reconciler. ## Dev CI Runner -Dev desired-state smoke verification is not a deploy executor. Use `bun scripts/cli.ts ci run-dev-e2e` for the Git-controlled temporary namespace runner described in `docs/reference/dev-ci-runner.md`. `deploy apply --env dev` must not roll out D601 direct or k3s-managed services in the current CI-only phase. +Dev desired-state smoke verification is not a deploy executor. Use `bun scripts/cli.ts ci run-dev-e2e` for the Git-controlled temporary namespace runner described in `docs/reference/dev-ci-runner.md`; that command must not roll out persistent D601 services. + +Persistent dev `backend-core` and `frontend` rollout is separate from that smoke runner and is described in `docs/reference/dev-environment.md`. ## D601 Dev Foundation @@ -69,11 +73,11 @@ On D601, dev/prod k3s verification must use the native k3s kubeconfig explicitly ## D601 Dev Core -Phase 3 introduces the dev backend/frontend manifest at `src/components/microservices/k3sctl-adapter/k3s/dev/unidesk-dev-core.k8s.yaml`. It may create only `backend-core-dev` and `frontend-dev` Deployment/Service objects in `unidesk-dev`. +Phase 3 introduces the dev backend/frontend manifest at `src/components/microservices/k3sctl-adapter/k3s/dev/unidesk-dev-core.k8s.yaml`. It may create only `backend-core-dev` and `frontend-dev` Deployment/Service objects in `unidesk-dev`; the persistent rollout contract and Rust build boundary are owned by `docs/reference/dev-environment.md`. `backend-core-dev` must use `unidesk-dev-runtime-config` and `unidesk-dev-runtime-secrets`, connect to `postgres-dev.../unidesk_dev`, expose HTTP on 8080 and provider ingress on 8081, and write logs under `/var/log/unidesk-dev`. `frontend-dev` must set `CORE_INTERNAL_URL=http://backend-core-dev.unidesk-dev.svc.cluster.local:8080` and must not proxy to production backend-core. -The manifest keeps placeholder image tags and deploy commit values in source control. Maintenance-channel direct D601 apply must not deploy `backend-core-dev` or `frontend-dev`; the CLI rejects `deploy apply --env dev --service backend-core|frontend` before runtime mutation. Dev core deployment must be implemented later as a controlled D601 target-side action that fetches `origin/master:deploy.json`, selects `environments.dev`, materializes the requested source commit on D601, narrows the dev core control manifest to the selected Service/Deployment pair, replaces placeholders with the requested commit and dev image tag, builds on D601, imports the image into native k3s containerd, applies only the `unidesk-dev` objects and stamps the Deployment. Client dry-run and static validation are the required checks before any controlled apply: +The manifest keeps placeholder image tags and deploy commit values in source control. The controlled `deploy apply --env dev --service backend-core|frontend` path fetches `origin/master:deploy.json`, materializes the requested source commit on D601, narrows the dev core control manifest to the selected Service/Deployment pair, replaces placeholders with the requested commit and dev image tag, builds on D601, imports the image into native k3s containerd, applies only the `unidesk-dev` objects and stamps the Deployment. Client dry-run and static validation remain useful checks before controlled apply: - `bun scripts/cli.ts dev-env validate --manifest src/components/microservices/k3sctl-adapter/k3s/dev/unidesk-dev-core.k8s.yaml` - `KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl apply --dry-run=client --validate=false -f src/components/microservices/k3sctl-adapter/k3s/dev/unidesk-dev-core.k8s.yaml` @@ -96,7 +100,7 @@ Maintenance-channel direct D601 apply must not deploy dev Code Queue; the CLI re `bun scripts/cli.ts deploy plan --env dev [--service <id>]` reads `origin/master:deploy.json#environments.dev` and prints a dry-run environment plan without checking or mutating live runtime resources. `deploy check --env dev` uses the same dry-run environment plan. `--env prod` is available for parity as a dry-run planning path; it reads `origin/master:deploy.json#environments.prod` and must not use a dirty local `deploy.json`. -`bun scripts/cli.ts deploy apply [--file deploy.json | --env dev] [--service <id>] [--dry-run] [--force]` starts an asynchronous job only for supported targets. Use `bun scripts/cli.ts job status <jobId> --tail-bytes 30000` to observe progress. `--dry-run` resolves the same plan but does not build or replace runtime objects. `--force` rebuilds even when the live commit matches. Environment apply is not the dev e2e trigger; use `bun scripts/cli.ts ci run-dev-e2e` for the Git-controlled temporary namespace smoke flow. `--env prod` apply is rejected, and D601 service apply is rejected before any runtime mutation. +`bun scripts/cli.ts deploy apply [--file deploy.json | --env dev] [--service <id>] [--dry-run] [--force]` starts an asynchronous job only for supported targets. Use `bun scripts/cli.ts job status <jobId> --tail-bytes 30000` to observe progress. `--dry-run` resolves the same plan but does not build or replace runtime objects. `--force` rebuilds even when the live commit matches. Environment apply is not the dev e2e trigger; use `bun scripts/cli.ts ci run-dev-e2e` for the Git-controlled temporary namespace smoke flow. `--env dev` apply is enabled only for `backend-core` and `frontend`; `--env prod` apply is rejected. All deploy commands output JSON. Long operations must use `.state/jobs/` and bounded log tails; no deploy path may succeed with missing progress output. @@ -137,7 +141,7 @@ The reconciler selects the executor from `config.json`: - `deployment.mode=internal-sidecar` on `main-server`: use the same main-server target-side source export, Docker build, image label stamping, fixed Compose project replacement and live commit verification as direct Compose services. This class is for private sidecars such as `code-queue-mgr`; it is still versioned by `deploy.json.commitId`, not by the operator's current worktree. - `deployment.mode=unidesk-direct` on a provider: this executor is disabled for D601 service deployment. The historical behavior dispatched `host.ssh` to the provider, built on the provider, then used the service's provider-local compose file and project; that shape must not remain a second deployment control plane. - Control bridges that UniDesk needs in order to inspect or repair an orchestrator must stay in this direct class. In particular, `k3sctl-adapter` is a UniDesk-managed bridge to native k3s and must remain outside k3s; Docker packaging on Docker Desktop/WSL must create an explicit host-local bridge, currently an adapter-container SSH local tunnel, to reach `/etc/rancher/k3s/k3s.yaml` and WSL `127.0.0.1:6443`. -- `deployment.mode=k3sctl-managed`: the target behavior is to build on the active control target, verify native k3s on the host OS/WSL distro, import the image into native k3s/containerd, apply the existing Kubernetes manifest, stamp the Deployment and wait for rollout. On D601, maintenance-channel direct execution of this behavior is not allowed for normal services; the current work is CI-only and must not roll out services. The executor must use the native kubeconfig and containerd socket, for example `/etc/rancher/k3s/k3s.yaml` and `/run/k3s/containerd/containerd.sock`; running k3s itself in Docker is forbidden for both control-plane and worker nodes. A `rancher/k3s` image or legacy container may only be used as a temporary artifact source during migration, and any active containerized k3s control plane must be stopped before verification succeeds. The executor must preload a valid `rancher/mirrored-pause:3.6` sandbox image into native k3s containerd through the provider-gateway one-shot egress path, verify its entrypoint is `/pause`, and reject fake or sleep-based replacement images. Code Queue's k3s migration executor must also stop/remove the legacy direct Docker `code-queue-backend` after k3s rollout, so there is never a second scheduler running beside the native k3s scheduler. +- `deployment.mode=k3sctl-managed`: the target behavior is to build on the active control target, verify native k3s on the host OS/WSL distro, import the image into native k3s/containerd, apply the existing Kubernetes manifest, stamp the Deployment and wait for rollout. On D601, persistent dev apply is currently allowed only for `backend-core` and `frontend` in `unidesk-dev`; normal production services still cannot use a maintenance-channel direct rollout. The executor must use the native kubeconfig and containerd socket, for example `/etc/rancher/k3s/k3s.yaml` and `/run/k3s/containerd/containerd.sock`; running k3s itself in Docker is forbidden for both control-plane and worker nodes. A `rancher/k3s` image or legacy container may only be used as a temporary artifact source during migration, and any active containerized k3s control plane must be stopped before verification succeeds. The executor must preload a valid `rancher/mirrored-pause:3.6` sandbox image into native k3s containerd through the provider-gateway one-shot egress path, verify its entrypoint is `/pause`, and reject fake or sleep-based replacement images. Code Queue's k3s migration executor must also stop/remove the legacy direct Docker `code-queue-backend` after k3s rollout, so there is never a second scheduler running beside the native k3s scheduler. Existing service-specific commands such as Code Queue deploy are disabled as direct D601 deploy paths. Their build/import/rollout semantics should converge later into one controlled target-side deployment path instead of keeping parallel implementations. diff --git a/docs/reference/deployment.md b/docs/reference/deployment.md index d777144a..d7c64c2a 100644 --- a/docs/reference/deployment.md +++ b/docs/reference/deployment.md @@ -1,12 +1,13 @@ # UniDesk Deployment Reference -主 server 使用根目录 `docker-compose.yml` 统一编排 database、backend-core、frontend、provider-gateway 以及必须留在主 server 的用户服务。当前环境本身就是主 server，因此 provider-gateway 也在同一台机器上启动，用与普通计算节点相同的 WebSocket 方式接入 core。Code Queue 按“master 低资源低抖动控制面、D601 高资源高抖动执行面”拆分：队列 CRUD、任务提交、历史摘要和轻量 Trace 读取由主 server Compose 中的 `code-queue-mgr` 直管 PostgreSQL；Codex/OpenCode scheduler、runner、dev-container、active run steer/interrupt 和执行态写回仍由 D601 原生 k3s/k8s Code Queue 执行面承担。 +主 server 使用根目录 `docker-compose.yml` 统一编排 database、backend-core、frontend、dev-frontend-proxy、provider-gateway 以及必须留在主 server 的用户服务。当前环境本身就是主 server，因此 provider-gateway 也在同一台机器上启动，用与普通计算节点相同的 WebSocket 方式接入 core。Code Queue 按“master 低资源低抖动控制面、D601 高资源高抖动执行面”拆分：队列 CRUD、任务提交、历史摘要和轻量 Trace 读取由主 server Compose 中的 `code-queue-mgr` 直管 PostgreSQL；Codex/OpenCode scheduler、runner、dev-container、active run steer/interrupt 和执行态写回仍由 D601 原生 k3s/k8s Code Queue 执行面承担。 ## Services - `database` 使用 `postgres:16-alpine`，数据保存到 named volume `unidesk_pgdata_10gb`，当前容量预算为 15 GB，初始化 SQL 位于 `src/components/database/init/`。 -- `backend-core` 是无状态核心服务，提供 Docker 内网 REST API、provider ingress WebSocket、任务调度入口和数据库访问层，Compose restart policy 必须为 `always`，确保进程崩溃或 Docker daemon 重启后自动恢复。源码拆分为 15 个职责单一的模块（`index.ts` 只做路由和启动），模块结构见 `docs/reference/repo-tree.md`。 -- `frontend` 是唯一公开 Web 控制台，提供登录、从 TSX 转译出的 React 应用资产和到 backend-core 的同源代理。 +- `backend-core` 是无状态核心服务，提供 Docker 内网 REST API、provider ingress WebSocket、任务调度入口和数据库访问层。当前 backend-core 是 Rust 服务；Rust 迭代编译边界见 `docs/reference/dev-environment.md`。源码模块结构见 `docs/reference/repo-tree.md`。 +- `frontend` 是 production 公开 Web 控制台，提供登录、从 TSX 转译出的 React 应用资产和到 backend-core 的同源代理。 +- `dev-frontend-proxy` 是主 server 上的 18083 dev UI 薄入口，代理到 D601 `unidesk-dev/frontend-dev`，详细边界见 `docs/reference/dev-environment.md`。 - `provider-gateway` 是当前主 server 的本机计算节点代理，通过 WebSocket 主动连到 provider ingress，挂载 `/var/run/docker.sock` 作为自动任务执行主路径，使用 `pid: "host"` 读取节点级进程资源，并周期性上报系统资源指标、进程占用与 Docker daemon 状态；计算节点 provider-gateway 还必须把 egress proxy 仅发布到宿主 loopback `127.0.0.1:18789` 供本节点执行环境出网，维护用 Host SSH / WSL SSH 私钥目录只读挂载到 `/run/host-ssh`，不得作为自动任务调度主路径。 - `todo-note` 是主 server 承载的 Todo Note 纯后端用户服务，容器名 `todo-note-backend`，只在 Compose 内网暴露 `4211/tcp`，使用主 PostgreSQL 存储迁移后的 Todo Note 数据。 - `code-queue-mgr` 是主 server 承载的轻量 Code Queue 控制面用户服务，容器名 `code-queue-mgr-backend`，只在 Compose 内网暴露 `4278/tcp`，只连接主 PostgreSQL，负责 `code-queue` 稳定代理路径下的队列管理、任务提交、历史摘要、已读状态和轻量 Trace 读取；不得引入 Codex/OpenCode/Playwright/Chromium/Docker socket/runner 依赖。 @@ -17,7 +18,7 @@ ## Public Exposure Boundary -Docker Compose 对最终用户只公开 frontend host port 和 provider ingress host port。backend-core REST API 仍不得映射公网；PostgreSQL 和 OA Event Flow 只允许为了受控 Code Queue active/standby 节点使用受限端口映射，并必须由 `network.restrictedHostAccess.allowedSourceCidrs` 生成的 `DOCKER-USER` 规则限制到 D601 出口地址，不能作为浏览器或任意公网客户端入口。浏览器访问 core API 必须通过 frontend 的同源代理完成。 +Docker Compose 对最终用户只公开 production frontend host port、dev frontend proxy host port 和 provider ingress host port。backend-core REST API 仍不得映射公网；PostgreSQL 和 OA Event Flow 只允许为了受控 Code Queue active/standby 节点使用受限端口映射，并必须由 `network.restrictedHostAccess.allowedSourceCidrs` 生成的 `DOCKER-USER` 规则限制到 D601 出口地址，不能作为浏览器或任意公网客户端入口。浏览器访问 core API 必须通过 frontend 的同源代理完成；dev UI 访问路径见 `docs/reference/dev-environment.md`。 计算节点上的用户服务后端也遵守同一边界：业务容器端口只绑定节点本机地址，并由 provider-gateway 主动连出 WebSocket 后接受 backend-core 的 `microservice.http` 调度访问。主 server 不为用户服务新增面向浏览器的公网反向代理端口；最终用户只通过 UniDesk frontend 的 React 页面访问结构化业务控件。 @@ -27,7 +28,7 @@ CLI 会优先使用 `docker compose` v2 plugin；当 v2 plugin 不存在时才 Compose v2 安装后仍然必须遵守 UniDesk 的服务控制入口：全栈生命周期用 `server start` / `server stop`，单服务重建用 `server rebuild <service>`。不要因为 v2 可用就直接在生产栈上手工执行未纳入 CLI 的 `up --build`、`down -v` 或跨项目清理命令；所有会影响容器的动作都应保持 job 可观测、Compose project 固定、database named volume 保留。主 server Compose 命令必须从 `providerGateway.upgrade.hostProjectRoot` 指定的 canonical UniDesk 根目录运行，临时 worktree、Code Queue 导出目录或实验分支不得复用生产 `-p unidesk` 和固定 `container_name` 去替换生产容器。 -版本化用户服务部署优先使用 `bun scripts/cli.ts deploy apply` 已支持的受控路径；当前 D601 维护通道直连 apply 不承担服务部署，dev desired-state smoke 使用 `ci run-dev-e2e`。`deploy.json` 只声明服务 `id`、`repo` 和 `commitId`；目标节点、Dockerfile、Compose、Kubernetes manifest、健康检查和代理路径继续来自 `config.json` 与现有 manifest。主 server 直管微服务和内部 sidecar，例如 `code-queue-mgr`，也必须支持这一路径：`deploy apply --service code-queue-mgr` 从 `deploy.json` 指定 commit 导出源码、构建镜像、替换固定 Compose service 并验证运行中镜像/健康信息的 commit。部署必须遵循 target-side build：服务部署到哪台 target，就在哪台 target 从 remote commit 导出源码、一次性代理构建镜像并部署；不得把中心构建镜像作为默认分发路径，也不得用 `docker commit` 或脏 worktree 作为部署输入。完整规则见 `docs/reference/deploy.md`。 +版本化用户服务部署优先使用 `bun scripts/cli.ts deploy apply` 已支持的受控路径；D601 persistent dev apply 当前只支持 `backend-core` 和 `frontend`，dev desired-state smoke 使用 `ci run-dev-e2e`。`deploy.json` 只声明服务 `id`、`repo` 和 `commitId`；目标节点、Dockerfile、Compose、Kubernetes manifest、健康检查和代理路径继续来自 `config.json` 与现有 manifest。主 server 直管微服务和内部 sidecar，例如 `code-queue-mgr`，也必须支持这一路径：`deploy apply --service code-queue-mgr` 从 `deploy.json` 指定 commit 导出源码、构建镜像、替换固定 Compose service 并验证运行中镜像/健康信息的 commit。部署必须遵循 target-side build：服务部署到哪台 target，就在哪台 target 从 remote commit 导出源码、一次性代理构建镜像并部署；不得把中心构建镜像作为默认分发路径，也不得用 `docker commit` 或脏 worktree 作为部署输入。完整规则见 `docs/reference/deploy.md`，D601 dev/Rust 边界见 `docs/reference/dev-environment.md`。 ## Main Server Swap @@ -43,7 +44,7 @@ swap 管理不能被强塞进所有热路径。`server start/status` 可以暴 ## Single Service Rebuild -前端、backend-core、本机 provider-gateway 或主 server 承载的 Todo Note/Code Queue Manager/Project Manager/Baidu Netdisk/OA Event Flow 用户服务需要非版本化本地重建时，统一使用 `bun scripts/cli.ts server rebuild <service>`，其中 `<service>` 只能是 `backend-core`、`frontend`、`provider-gateway`、`todo-note`、`code-queue-mgr`、`project-manager`、`baidu-netdisk` 或 `oa-event-flow`。需要按 commit 上线或恢复到 desired-state 时必须改用 `bun scripts/cli.ts deploy apply --service <id>`；直管微服务也不能把脏工作树或手工重建作为部署真相。D601 Code Queue 执行面、File Browser、FindJob、Pipeline、MET Nonlinear 和 ClaudeQQ 部署在计算节点，不属于主 server Compose 可重建服务；其中 D601 Code Queue 执行面不得再通过 `codex deploy` 或维护通道直连 D601 部署；未来正式 CD 必须经受控 target-side 路径执行 build-first、rollout 和 live commit 验证。 +前端、backend-core、本机 provider-gateway、dev-frontend-proxy 或主 server 承载的 Todo Note/Code Queue Manager/Project Manager/Baidu Netdisk/OA Event Flow 用户服务需要非版本化本地重建时，统一使用 `bun scripts/cli.ts server rebuild <service>`，其中 `<service>` 只能是 `backend-core`、`frontend`、`dev-frontend-proxy`、`provider-gateway`、`todo-note`、`code-queue-mgr`、`project-manager`、`baidu-netdisk` 或 `oa-event-flow`。需要按 commit 上线或恢复到 desired-state 时必须改用 `bun scripts/cli.ts deploy apply --service <id>`；直管微服务也不能把脏工作树或手工重建作为部署真相。Rust backend-core 迭代不得在 master server 用 `server rebuild backend-core` 编译，必须走 D601 dev deploy/CI。D601 Code Queue 执行面、File Browser、FindJob、Pipeline、MET Nonlinear 和 ClaudeQQ 部署在计算节点，不属于主 server Compose 可重建服务；其中 D601 Code Queue 执行面不得再通过 `codex deploy` 或维护通道直连 D601 部署；未来正式 CD 必须经受控 target-side 路径执行 build-first、rollout 和 live commit 验证。 frontend 改动必须明确上线到公网：修改 `src/components/frontend/src/`、`src/components/frontend/public/style.css`、frontend 使用的共享 TSX/TS 模块或 WebUI 导航后，必须在同一变更集中执行 `bun scripts/cli.ts server rebuild frontend`，并等待 job 成功。公网 WebUI 的 `/app.js` 是 `unidesk-frontend` 容器启动时从镜像内源码转译生成的运行时 bundle；只改工作区文件、只跑 `bun run check`、只跑 `Bun.build` 或只刷新浏览器都不会替换已经运行的容器。 @@ -61,7 +62,7 @@ frontend 的 Docker 上线顺序为：先运行必要的本地校验，例如 `b ## Health Criteria -服务跑通的最低标准是：backend-core 内网 `/health` 返回 ok，frontend 公网 `/health` 返回 ok，provider ingress 公网 `/health` 返回 ok，database 在容器内 `pg_isready` 可用，Todo Note 后端 `/api/health` 返回 `storage=postgres`，`code-queue-mgr` `/health` 返回 `role=master-control-plane`、`schemaReady=true` 和资源预算字段，`bun scripts/cli.ts microservice health code-queue` 与 `/api/tasks/overview` 默认经主 server `code-queue-mgr` 返回 PostgreSQL 队列总览，`k3sctl-adapter` `/api/control-plane` 可见 `unidesk-k3s` Kubernetes API service proxy 状态、D601 scheduler serving healthy、D601 read/write Service healthy、`presentNodeIds` 包含 `D601`、`missingNodeIds=[]` 和 no-fallback 路径，D601 Code Queue scheduler `/health` 返回轻量 readiness、默认模型、`queue.storage` 和 `egressProxy.connected=true`，提交类入口经 `code-queue-mgr` 入库后由 D601 `code-queue-scheduler` 轮询并执行，Project Manager `/health` 返回 `storage.primary=postgres` 和项目数量，backend-core `/api/performance` 返回性能指标，`/api/nodes` 中出现 `main-server`、`D601` 和 `D518` provider 且状态为 `online`，`/api/nodes/system-status` 中出现对应 CPU/内存/硬盘采样，`/api/nodes/docker-status` 中能看到主 server、D601 与 D518 Docker 快照。D601/D518 上不得存在 active `rancher/k3s` 容器；D518 只有在原生 k3s-agent 与稳定 Kubernetes 网络完成验证后才可加入 Code Queue expected nodes。交付前还必须运行 `bun scripts/cli.ts e2e run`，并以 `docs/reference/e2e.md` 的门禁作为最终判定。 +服务跑通的最低标准是：backend-core 内网 `/health` 返回 ok，frontend 公网 `/health` 返回 ok，dev frontend proxy 公网 `/health` 在 D601 dev frontend 已部署后返回 ok，provider ingress 公网 `/health` 返回 ok，database 在容器内 `pg_isready` 可用，Todo Note 后端 `/api/health` 返回 `storage=postgres`，`code-queue-mgr` `/health` 返回 `role=master-control-plane`、`schemaReady=true` 和资源预算字段，`bun scripts/cli.ts microservice health code-queue` 与 `/api/tasks/overview` 默认经主 server `code-queue-mgr` 返回 PostgreSQL 队列总览，`k3sctl-adapter` `/api/control-plane` 可见 `unidesk-k3s` Kubernetes API service proxy 状态、D601 scheduler serving healthy、D601 read/write Service healthy、`presentNodeIds` 包含 `D601`、`missingNodeIds=[]` 和 no-fallback 路径，D601 Code Queue scheduler `/health` 返回轻量 readiness、默认模型、`queue.storage` 和 `egressProxy.connected=true`，提交类入口经 `code-queue-mgr` 入库后由 D601 `code-queue-scheduler` 轮询并执行，Project Manager `/health` 返回 `storage.primary=postgres` 和项目数量，backend-core `/api/performance` 返回性能指标，`/api/nodes` 中出现 `main-server`、`D601` 和 `D518` provider 且状态为 `online`，`/api/nodes/system-status` 中出现对应 CPU/内存/硬盘采样，`/api/nodes/docker-status` 中能看到主 server、D601 与 D518 Docker 快照。D601/D518 上不得存在 active `rancher/k3s` 容器；D518 只有在原生 k3s-agent 与稳定 Kubernetes 网络完成验证后才可加入 Code Queue expected nodes。交付前还必须运行 `bun scripts/cli.ts e2e run`，并以 `docs/reference/e2e.md` 的门禁作为最终判定。 ## Code Queue Control/Execution Resource Budget diff --git a/docs/reference/dev-ci-runner.md b/docs/reference/dev-ci-runner.md index f621415c..ac1b15fb 100644 --- a/docs/reference/dev-ci-runner.md +++ b/docs/reference/dev-ci-runner.md @@ -44,13 +44,24 @@ Do not add a long-lived DevOps service, run broker, webhook listener or second d "scriptPath": "scripts/ci/dev-e2e.sh", "timeoutMs": 1800000 }, - "services": [] + "services": [ + { + "id": "backend-core", + "repo": "https://github.com/pikasTech/unidesk", + "commitId": "<pushed-commit>" + }, + { + "id": "frontend", + "repo": "https://github.com/pikasTech/unidesk", + "commitId": "<pushed-commit>" + } + ] } } } ``` -`scriptPath` must be a repo-relative `scripts/ci/*.sh` path. Inline shell bodies, arbitrary script paths, local dirty scripts and separate `develop.json` or CI manifest files are forbidden. The script is fetched from the same full 40-character manifest commit that supplied `deploy.json`, so the runner logic is auditable and rollbackable with the desired state. +`scriptPath` must be a repo-relative `scripts/ci/*.sh` path. Inline shell bodies, arbitrary script paths, local dirty scripts and separate `develop.json` or CI manifest files are forbidden. The script is fetched from the same full 40-character manifest commit that supplied `deploy.json`, so the runner logic is auditable and rollbackable with the desired state. Persistent dev rollout service scope is owned by `docs/reference/dev-environment.md`; this runner only consumes the dev service list for smoke verification and must not deploy it. ## Execution Path diff --git a/docs/reference/dev-environment.md b/docs/reference/dev-environment.md new file mode 100644 index 00000000..bc1759f5 --- /dev/null +++ b/docs/reference/dev-environment.md @@ -0,0 +1,106 @@ +# D601 Dev Environment + +This document is the authoritative source for the persistent `unidesk-dev` environment, the public dev frontend port, `deploy apply --env dev`, and the Rust backend-core build boundary. If the same dev-environment rule would need edits in `AGENTS.md`, `docs/reference/cli.md`, `docs/reference/deploy.md`, `docs/reference/ci.md`, `docs/reference/deployment.md`, `docs/reference/e2e.md` or `TEST.md`, keep the detailed rule here and leave only a short cross-reference elsewhere. + +## Goal + +The dev environment lets users experience the next UniDesk version without interrupting production: + +- Production stays on the normal public frontend at `http://74.48.78.17:18081/`. +- Dev is exposed through a separate main-server public port at `http://74.48.78.17:18083/`. +- Dev backend/frontend workloads run in D601 native k3s namespace `unidesk-dev`, not in the main-server Compose stack. +- Dev backend/frontend build and rollout use pushed Git commits from `origin/master:deploy.json#environments.dev`, not dirty local worktrees. +- Rust backend-core check/build must run on D601 through CI or dev deploy; the master server must not compile Rust for this iteration path. + +## Public Dev Frontend Port + +The main server owns one extra public entrypoint for dev UI: + +```text +browser -> main-server:18083 -> dev-frontend-proxy -> prod backend-core microservice proxy -> k3sctl-adapter -> D601 k3s Service frontend-dev -> frontend-dev -> backend-core-dev +``` + +`dev-frontend-proxy` is an nginx sidecar in the main-server Compose project. It proxies requests to `backend-core:8080/api/microservices/k3sctl-adapter/proxy/api/services/frontend-dev/proxy...`, so D601 does not expose a new public port and the dev UI still crosses the existing UniDesk/k3sctl-adapter control boundary. The proxy is intentionally thin: it does not build frontend assets, does not talk to D601 directly, and does not contain DevOps logic. + +The dev public port is configured in `config.json` as `network.devFrontend.port=18083`, surfaced by `server status` as `urls.devFrontend`, and managed by `server rebuild dev-frontend-proxy`. The proxy health depends on `frontend-dev`; it can be unhealthy until the D601 dev frontend has been deployed. + +The unrestricted public network entries are therefore production frontend, dev frontend, and provider ingress. backend-core REST, PostgreSQL, user-service backend ports, k3s Services, NodePorts and D601 host ports remain private or explicitly restricted. + +## Desired State + +`deploy.json` remains the only version intent file. Dev entries live under `environments.dev` and are read from `origin/master:deploy.json`, never from a dirty local file, when using `--env dev`. + +The persistent dev rollout currently supports only: + +- `backend-core` +- `frontend` + +`code-queue`, Decision Center, k3sctl-adapter and other D601 services are not part of persistent dev apply yet. Their smoke validation stays under `ci run-dev-e2e` or service-specific future designs. The `environments.dev.ci` declaration and short launcher runner are owned by `docs/reference/dev-ci-runner.md`. + +## Rust Backend-Core Boundary + +backend-core is implemented as a Rust service for the dev path. The master server may inspect files, run TypeScript CLI checks, render Compose config, dispatch jobs and proxy traffic, but it must not run Rust compilation for backend-core iteration. + +Allowed on the master server: + +- `bun scripts/cli.ts check --files --scripts-typecheck --compose --logs` +- `bun scripts/cli.ts check --help` +- `bun scripts/cli.ts deploy plan --env dev --service backend-core` +- `bun scripts/cli.ts deploy apply --env dev --service backend-core` +- `bun scripts/cli.ts ci run --revision <commit>` +- `bun scripts/cli.ts ci run-dev-e2e --wait-ms <ms>` + +Not allowed on the master server for this path: + +- `cargo check`, `cargo build`, `cargo test` or `rustfmt` against backend-core. +- `bun scripts/cli.ts check --rust` without the D601 CI guard. +- `bun scripts/cli.ts server rebuild backend-core` as a way to iterate Rust backend-core, because it would build the Rust image in the main-server Docker daemon. +- Ad-hoc `docker build` of `src/components/backend-core/Dockerfile` on the master server. + +Rust checking is enabled only when the process is already running inside the D601 CI/dev execution boundary: `UNIDESK_D601_RUST_CHECK=1 bun scripts/cli.ts check --full --rust`. `check --rust` deliberately fails outside that guard with an explicit explanation instead of silently compiling on the wrong host. + +## Dev Deploy Path + +`deploy apply --env dev --service backend-core|frontend` is the controlled persistent dev rollout path. The controller runs on the master server, but the heavy work runs on D601: + +1. Fetch `origin/master:deploy.json#environments.dev` and resolve the requested service commit to a full SHA. +2. Dispatch to D601 through the existing provider-gateway/Host SSH maintenance capability. +3. On D601, fetch/export the requested Git commit through the node-local provider-gateway egress proxy `http://127.0.0.1:18789`. +4. Use the target-side commit as the source for Dockerfile, build context and dev k3s manifest. +5. Build the service image on D601 Docker, importing any required base images through the same egress boundary. +6. Import the image into native k3s containerd at `/run/k3s/containerd/containerd.sock`. +7. Apply only the selected `unidesk-dev` Service/Deployment objects from the dev manifest. +8. Stamp the Deployment with `UNIDESK_DEPLOY_*` env and `unidesk.ai/deploy-*` annotations. +9. Verify health through the Kubernetes API service proxy and require the live commit to match the requested commit. + +The dev path is not a fallback system. If GitHub fetch, provider-gateway egress, Docker build, native k3s, containerd import, kubectl apply or live health verification fails, the job fails with logs. It must not fall back to building on the master server, using a dirty worktree, direct D601 public ports, NodePort, or another deployment command. + +## Standard Workflow + +Use this sequence for backend-core Rust and frontend dev work: + +1. Develop in the current `master` worktree and keep unrelated parallel changes separated with `git status`/`git diff`. +2. Run local non-Rust checks on the master server, for example `bun scripts/cli.ts check --files --scripts-typecheck --compose --logs`. +3. Commit and push the code to `origin master`; `deploy apply --env dev` cannot deploy unpushed local changes. +4. Update `deploy.json` `environments.dev.services` so `backend-core` and `frontend` point at the pushed commit, then commit and push that manifest update. +5. Run `bun scripts/cli.ts deploy apply --env dev --service backend-core` and observe the returned job with `bun scripts/cli.ts job status <jobId> --tail-bytes 30000`. +6. Run `bun scripts/cli.ts deploy apply --env dev --service frontend` and observe the job the same way. +7. Rebuild or verify `dev-frontend-proxy` on the main server with `bun scripts/cli.ts server rebuild dev-frontend-proxy` when the proxy config or port changes. +8. Manually test `http://74.48.78.17:18083/` and the dev health endpoints. +9. Run D601 CI for the commit and the dev smoke runner: `bun scripts/cli.ts ci run --revision <commit> --wait-ms <ms>` and `bun scripts/cli.ts ci run-dev-e2e --wait-ms <ms>`. + +## Validation Commands + +Useful read-only or bounded validation commands: + +```bash +bun scripts/cli.ts server status +bun scripts/cli.ts deploy plan --env dev +bun scripts/cli.ts deploy plan --env dev --service backend-core +bun scripts/cli.ts dev-env validate --manifest src/components/microservices/k3sctl-adapter/k3s/dev/unidesk-dev-core.k8s.yaml +bun scripts/cli.ts microservice proxy k3sctl-adapter /api/services/backend-core-dev/proxy/health --raw --full +bun scripts/cli.ts microservice proxy k3sctl-adapter /api/services/frontend-dev/proxy/health --raw --full +curl -fsS http://127.0.0.1:18083/health +``` + +When validating on D601 directly, always use the native kubeconfig explicitly: `KUBECONFIG=/etc/rancher/k3s/k3s.yaml`. The default `kubectl` context may point at Docker Desktop and is not valid for UniDesk native k3s verification. diff --git a/docs/reference/e2e.md b/docs/reference/e2e.md index 92573bb0..2608496d 100644 --- a/docs/reference/e2e.md +++ b/docs/reference/e2e.md @@ -1,6 +1,6 @@ # UniDesk E2E Reference -UniDesk delivery is not complete until the public frontend, public provider ingress, internal core API, PostgreSQL database, local provider-gateway self-connection, and frontend Playwright flow pass one end-to-end check. The canonical automated command is `bun scripts/cli.ts e2e run`. +UniDesk delivery is not complete until the public production frontend, public dev frontend proxy when deployed, public provider ingress, internal core API, PostgreSQL database, local provider-gateway self-connection, and frontend Playwright flow pass the relevant end-to-end checks. The canonical automated command is `bun scripts/cli.ts e2e run`. ## Required Preconditions @@ -31,7 +31,7 @@ Typical targeted commands: - `bun scripts/cli.ts e2e run --only frontend --skip frontend:todo-note-integrated-visible,frontend:findjob-integrated-visible` - `bun scripts/cli.ts e2e run --only network,provider-ingress` -- Public exposure: Docker port summary must not show core REST, Code Queue NodePort, or Code Queue public host mappings; frontend and provider ingress are the only browser/provider public entries. PostgreSQL `15432` and OA Event Flow `4255` may be host-mapped only for controlled Code Queue nodes and must be protected by the `DOCKER-USER` source restrictions generated from `network.restrictedHostAccess`; E2E treats either an unreachable generic probe or a verified restricted rule as passing. Known private user-service ports such as FindJob `3254`, MET Nonlinear `3288`, Todo Note `4211`, legacy Code Queue host ports and File Browser provider port `4251` probes must fail. +- Public exposure: Docker port summary must not show core REST, Code Queue NodePort, or Code Queue public host mappings; the only unrestricted public entries are production frontend, dev frontend proxy and provider ingress. PostgreSQL `15432` and OA Event Flow `4255` may be host-mapped only for controlled Code Queue nodes and must be protected by the `DOCKER-USER` source restrictions generated from `network.restrictedHostAccess`; E2E treats either an unreachable generic probe or a verified restricted rule as passing. Known private user-service ports such as FindJob `3254`, MET Nonlinear `3288`, Todo Note `4211`, legacy Code Queue host ports and File Browser provider port `4251` probes must fail. The dev frontend proxy rule is owned by `docs/reference/dev-environment.md`. - Core API: `docker exec unidesk-backend-core` calls internal `GET /api/overview`, which must report `dbReady: true`, `pgdata.volumeName=unidesk_pgdata_10gb`, a positive PostgreSQL database byte count, and at least one online node; internal `GET /api/performance` must report component request statistics, internal operation statistics, PGDATA usage and Code Queue PostgreSQL storage metadata. - Provider self-connection: internal `GET /api/nodes` must contain `main-server` with `status: online`, `labels.providerGatewayVersion` equal to `src/components/provider-gateway/package.json`, `labels.providerGatewayUpgradePolicy: "always-enabled"`, `labels.providerGatewayRestartPolicyOk: true`, `labels.providerGatewayPidModeOk: true`, and `labels.providerGatewayRuntimeGuardOk: true`; internal `GET /api/nodes/system-status` must contain CPU/memory/disk samples plus a non-empty process resource list sorted by `memoryBytes` by default, where `memoryBytes` should use PSS when `/proc/[pid]/smaps_rollup` is available, otherwise `rssBytes - statm.shared` before raw RSS, and must retain `rssBytes` for diagnostics; internal `GET /api/nodes/docker-status` must contain a Docker snapshot for `main-server`; every running `provider-gateway` container visible in Docker snapshots must report `restartPolicy: "always"` and `pidMode: "host"`; public provider ingress `/health` must return ok. - Provider remote control: internal `/api/dispatch` must successfully complete a real `provider.upgrade` task in `mode: "plan"` so the upgrade path is validated without recreating the running gateway during E2E. @@ -59,7 +59,7 @@ User service pages are covered by the same rule. `Todo Note` must show lists, ta ## Public Boundary Rule -The public frontend URL and provider ingress URL are the only unrestricted public network interfaces. backend-core REST API remains Docker-internal only; PostgreSQL and OA Event Flow may expose restricted host mappings solely for controlled Code Queue nodes, and E2E must prove those mappings are unreachable to generic clients or protected by explicit source rules. +The production frontend URL, dev frontend proxy URL and provider ingress URL are the only unrestricted public network interfaces. backend-core REST API remains Docker-internal only; PostgreSQL and OA Event Flow may expose restricted host mappings solely for controlled Code Queue nodes, and E2E must prove those mappings are unreachable to generic clients or protected by explicit source rules. ## Database Persistence Rule diff --git a/docs/reference/microservices.md b/docs/reference/microservices.md index 0f0041b1..fae7786d 100644 --- a/docs/reference/microservices.md +++ b/docs/reference/microservices.md @@ -8,7 +8,7 @@ UniDesk 用户服务是挂载到 UniDesk 核心服务上的、面向用户使用 - 用户服务后端端口默认只绑定计算节点本机地址，例如 `127.0.0.1:<port>`，不得直接暴露公网。 - 浏览器只访问 UniDesk frontend；frontend 通过同源 `/api/microservices/*` 代理到 backend-core。`deployment.mode=unidesk-direct` 的用户服务由 backend-core 通过目标 provider-gateway 的 `microservice.http` 能力访问计算节点本机后端；`deployment.mode=internal-sidecar` 的主 server 内置控制面服务由 backend-core 直接访问同一 Compose 网络内的显式服务名；`deployment.mode=k3sctl-managed` 的用户服务只允许经 `k3sctl-adapter` 微服务进入 k3s 标准服务路由，backend-core 不得直接向业务容器所在 provider 下发 `microservice.http`。 -- backend-core REST API、database 和计算节点用户服务后端都不得新增公网端口；公网入口仍只有 frontend 和 provider ingress。 +- backend-core REST API、database 和计算节点用户服务后端都不得新增公网端口；公网入口仅限 production frontend、dev frontend proxy 和 provider ingress。dev frontend proxy 的唯一权威规则见 `docs/reference/dev-environment.md`。 - `microservice.http` 只允许 provider-gateway 访问 `http://127.0.0.1`、`http://localhost`、`http://host.docker.internal` 这类节点本地地址，或明确登记为同一私有 Docker network 内的服务名；主 server 内置用户服务可使用同一 Compose 网络内的显式服务名，例如 `todo-note:4211`。k3s 代管服务不得把业务容器地址登记成 provider-gateway 直连目标，`backend.proxyMode` 必须使用 `k3sctl-adapter-http`，`backend.nodeBaseUrl` 可使用 `k3s://<service>` 这类逻辑服务名。backend-core 还必须用 `allowedPathPrefixes` 和 `allowedMethods` 同时限制可代理路径和 HTTP 方法。 ## Config Contract @@ -155,7 +155,7 @@ Baidu Netdisk 在 UniDesk 语境中按纯后端服务管理：不得暴露百度 D601 开发环境底座只允许创建 `unidesk-dev` namespace 与 dev 专用对象，manifest 固定为 `src/components/microservices/k3sctl-adapter/k3s/dev/unidesk-dev-foundation.k8s.yaml`。该 manifest 包含 `postgres-dev` 独立 PostgreSQL StatefulSet/Service/PVC、dev-only secret/config 模板、dev DB 初始化 SQL 和迁移 Job、ResourceQuota/LimitRange，以及 `unidesk-dev-db-guard`。它不得修改生产 `unidesk` namespace、生产 PostgreSQL、生产 PVC、生产 Deployment/Service/Secret 或主 server Docker Compose。 -`postgres-dev` 是 dev backend-core 与 dev Code Queue 状态的默认唯一数据库。dev 运行时必须使用 `postgres-dev.unidesk-dev.svc.cluster.local:5432/unidesk_dev` 和 dev Provider 身份 `D601-dev`；不得共享生产 `d601-tcp-egress-gateway.../unidesk`。当前 Phase 2 只提供 manifest 脚本和 `dev-env validate` 的静态护栏；backend-core、Code Queue 和 Code Queue Manager 的运行时启动护栏需在后续阶段单独评审后接入。 +`postgres-dev` 是 dev backend-core 与未来 dev Code Queue 状态的默认唯一数据库。dev 运行时必须使用 `postgres-dev.unidesk-dev.svc.cluster.local:5432/unidesk_dev` 和 dev Provider 身份 `D601-dev`；不得共享生产 `d601-tcp-egress-gateway.../unidesk`。Persistent dev backend-core/frontend rollout、public dev frontend port and Rust build boundary are defined in `docs/reference/dev-environment.md`; Code Queue dev rollout remains future work. 验收入口：先运行 `bun scripts/cli.ts dev-env validate` 做静态资源与 DB URL 护栏检查；具备 D601 kubeconfig 时运行 `bun scripts/cli.ts dev-env validate --kubectl-dry-run` 做 Kubernetes client dry-run。首次或镜像缓存不确定时，先运行 `bun scripts/cli.ts dev-env prewarm-images`，把 `postgres:16-alpine` 和 local-path helper 所需的 `rancher/mirrored-library-busybox:1.36.1` 导入 D601 原生 k3s containerd；否则 D601 的 Docker 代理/缓存正常也不能保证 k3s/containerd 能实时拉到外部镜像。若实际 apply，只能 apply 到 `unidesk-dev`，随后用 `kubectl -n unidesk-dev get pods,svc,pvc` 验证 dev DB ready，并对比 apply 前后的 `kubectl -n unidesk get deploy,sts,svc,secret,pvc -o name` 证明生产 workload 未变化。 @@ -163,7 +163,7 @@ D601 上必须显式使用原生 k3s kubeconfig：`KUBECONFIG=/etc/rancher/k3s/k ### D601 Dev Core Services -`backend-core-dev` 与 `frontend-dev` 的第一版 manifest 固定为 `src/components/microservices/k3sctl-adapter/k3s/dev/unidesk-dev-core.k8s.yaml`。该 manifest 只允许创建 `unidesk-dev` 内的 `backend-core-dev`、`frontend-dev` Deployment/Service；不得修改生产主 server Compose、生产 `unidesk` namespace 或生产 backend/frontend。 +`backend-core-dev` 与 `frontend-dev` 的第一版 manifest 固定为 `src/components/microservices/k3sctl-adapter/k3s/dev/unidesk-dev-core.k8s.yaml`。该 manifest 只允许创建 `unidesk-dev` 内的 `backend-core-dev`、`frontend-dev` Deployment/Service；不得修改生产主 server Compose、生产 `unidesk` namespace 或生产 backend/frontend。Rollout uses `deploy apply --env dev --service backend-core|frontend` under the rules in `docs/reference/dev-environment.md`. `backend-core-dev` 必须从 `unidesk-dev-runtime-config` 和 `unidesk-dev-runtime-secrets` 注入 dev-only 配置，使用 `postgres-dev.../unidesk_dev`、dev Provider token、dev log path 和 `UNIDESK_DEPLOY_REF=origin/master:deploy.json#environments.dev`。`frontend-dev` 必须把 `CORE_INTERNAL_URL` 指向 `backend-core-dev.unidesk-dev.svc.cluster.local:8080`，页面在 dev identity 下显示 DEV 标记，`/health` 返回 dev namespace、database、service id、deploy ref 和 commit metadata。生产环境未设置 dev identity 时，backend-core 和 frontend health payload 保持生产兼容形状。 diff --git a/docs/reference/observability.md b/docs/reference/observability.md index a9761b9e..af755af2 100644 --- a/docs/reference/observability.md +++ b/docs/reference/observability.md @@ -8,13 +8,13 @@ UniDesk 的可观测性优先级高于静默成功。CLI、服务日志、Docker ## Service Logs -服务日志位于 `logs/{YYYYMMDD}/`，每次 `server start` 都生成新的本地时间戳前缀。新写入的 UniDesk JSONL 日志必须按小时切片：`logs/{YYYYMMDD}/{startStamp}_{YYYYMMDD}_{HH}_{service}.jsonl`，一天一个目录，禁止长期追加到单个巨大 JSONL。所有 UniDesk Bun 服务（backend-core、frontend、provider-gateway、Code Queue、project-manager、baidu-netdisk 以及后续新增服务）必须复用 `src/components/shared/src/rotating-jsonl.ts` 中的 `createHourlyJsonlWriter`；`LOG_FILE` 只作为推导 `logs` 根目录、启动前缀和 service 后缀的 base path，不得直接 `appendFileSync(LOG_FILE, ...)` 长期追加。database 通过 PostgreSQL logging collector 写入同一日期目录。 +服务日志位于 `logs/{YYYYMMDD}/`，每次 `server start` 都生成新的本地时间戳前缀。新写入的 UniDesk JSONL 日志必须按小时切片：`logs/{YYYYMMDD}/{startStamp}_{YYYYMMDD}_{HH}_{service}.jsonl`，一天一个目录，禁止长期追加到单个巨大 JSONL。所有 UniDesk Bun 服务（frontend、provider-gateway、Code Queue、project-manager、baidu-netdisk 以及后续新增 Bun 服务）必须复用 `src/components/shared/src/rotating-jsonl.ts` 中的 `createHourlyJsonlWriter`；Rust backend-core 必须提供等价的 hourly rotation and retention behavior in `src/components/backend-core/src/logger.rs`。`LOG_FILE` 只作为推导 `logs` 根目录、启动前缀和 service 后缀的 base path，不得长期追加到单个文件。database 通过 PostgreSQL logging collector 写入同一日期目录。 日志保留默认按日志族限制为 `1GiB`：服务写入或 Code Queue 导出日志时必须扫描同一 service 后缀的历史文件，超过上限后自动删除最旧切片；当前活跃切片不能被保留清理删除。全局上限由 `UNIDESK_LOG_RETENTION_BYTES` 控制，服务级上限使用 `UNIDESK_<SERVICE>_LOG_MAX_BYTES`（如 `UNIDESK_FRONTEND_LOG_MAX_BYTES`、`UNIDESK_PROVIDER_GATEWAY_LOG_MAX_BYTES`），历史兼容变量只允许作为过渡入口。Codex app-server 的 `logs_*.sqlite` 仅作为 Codex 上游运行时的短暂缓冲，Code Queue 必须周期性导出为同样按小时切片的 `codex-app-server` JSONL，并删除/压缩已导出的 SQLite 行，避免 `logs_2.sqlite` 成为长期大文件。 新增或迁移服务的长期规范：Dockerfile 必须把 `src/components/shared` 复制到与仓库相同的相对路径，TypeScript 配置必须能解析 shared 引用，Compose 必须传入 `LOG_FILE` 和 `UNIDESK_LOG_RETENTION_BYTES`；如果服务需要在内存中暴露 `/logs`，可以继续维护有限 `recentLogs`，但落盘只能通过统一 hourly writer。业务归档日志（例如 Code Queue task output archive）可以保留 append-only 文件，但不得复用 UniDesk service JSONL 命名族，也不得替代 `/logs` 的结构化服务日志。 -`bun scripts/cli.ts check` 必须包含日志轮转门禁：核心 Bun 服务源码不得直接向 `LOG_FILE` append，且必须引用统一 hourly writer。新增服务如果进入主 Compose，也要纳入该门禁的 checked file 列表。 +`bun scripts/cli.ts check` 必须包含日志轮转门禁：核心 Bun 服务源码不得直接向 `LOG_FILE` append，且必须引用统一 hourly writer；Rust backend-core logger must expose equivalent rotation/pruning markers for the same check. 新增服务如果进入主 Compose，也要纳入该门禁的 checked file 列表。 ## Log Access diff --git a/docs/reference/repo-tree.md b/docs/reference/repo-tree.md index f7e3169c..cf9aa7a6 100644 --- a/docs/reference/repo-tree.md +++ b/docs/reference/repo-tree.md @@ -2,7 +2,7 @@ - AGENTS.md (Top-level agent index and `scripts/cli.ts` usage guide) - TEST.md (Manual CLI test plan following cli-spec expectations) - config.json (Single source of truth for ports, tokens, runtime, paths, and provider identity) - - docker-compose.yml (Main server orchestration for database, backend-core, frontend, provider-gateway, and managed main-server user services such as Todo Note) + - docker-compose.yml (Main server orchestration for database, backend-core, frontend, dev-frontend-proxy, provider-gateway, and managed main-server user services such as Todo Note) - package.json / bun.lock (Root Bun tooling for CLI checks) - .gitignore - reference -> docs/reference (Compatibility symlink for older references) @@ -17,7 +17,7 @@ - debug.ts (Real-flow debug helpers) - command.ts (Bounded command execution helpers) - output.ts (JSON output helpers) - - e2e.ts (Public frontend/provider ingress, internal core/database, and Playwright frontend E2E checks) + - e2e.ts (Public production frontend/dev frontend/provider ingress, internal core/database, and Playwright frontend E2E checks) - ci.ts (D601 k3s Tekton CI install/status/manual-run/dev-namespace-e2e/logs helpers; CI only, no CD) - logs/ (Generated service logs; ignored by git) - .state/ (Generated job state and compose env; ignored by git) @@ -34,7 +34,8 @@ - observability.md (Logs and status visibility) - e2e.md (Delivery gate, Playwright frontend E2E, and database persistence checks) - ci.md (D601 k3s Tekton CI, read-only production database performance gate, master deploy.json dev namespace e2e harness, and trigger boundary) - - src/ (TypeScript component monorepo) + - dev-environment.md (D601 persistent dev environment, public dev frontend proxy, deploy dev scope, and Rust build boundary) + - src/ (Component monorepo) - package.json (Component workspace metadata) - bun.lock (Component dependency lockfile) - tsconfig.base.json (Project references for component checks) @@ -45,24 +46,28 @@ - tsconfig.json - src/index.ts - backend-core/ (UniDesk stateless core service container) + - Cargo.toml / Cargo.lock (Rust backend-core crate and pinned dependency graph) - package.json - - tsconfig.json - Dockerfile - - src/index.ts (Server setup, HTTP/WebSocket routing, startup orchestration) - - src/types.ts (All interfaces and type aliases) - - src/config.ts (Environment variable reading and config validation) - - src/context.ts (Shared mutable state container: db client, config ref, dbReady flag) - - src/logger.ts (Structured logger factory) - - src/http.ts (HTTP/JSON utilities, ISO formatting, nested value extraction) - - src/db.ts (Database init, schema creation, all CRUD queries) - - src/performance.ts (Request/operation performance sampling and database size reporting) - - src/provider-registry.ts (Provider WebSocket lifecycle: register, heartbeat, status, disconnect) - - src/task-dispatcher.ts (Task creation, dispatch to providers, wait-for-result) - - src/ssh-bridge.ts (SSH session bridging over provider WebSocket) - - src/egress-tcp.ts (Egress TCP proxy for provider outbound connections) - - src/scheduler.ts (Scheduled task system: cron-like recurring tasks) - - src/microservice-proxy.ts (Microservice routing, HTTP proxy, health aggregation) - - src/overview.ts (Overview endpoint, node/task/service summary, load test) + - src/main.rs (Server setup, HTTP/WebSocket routing, startup orchestration) + - src/cli.rs (Container-local helpers for JSON fetch and SSH broker) + - src/types.rs (Runtime config, provider, task, microservice and performance data structures) + - src/config.rs (Environment variable reading, config validation and dev identity) + - src/state.rs (Shared application state, database pool, logger and in-memory task/provider indexes) + - src/logger.rs (Structured JSONL logger with hourly rotation and retention pruning) + - src/http.rs (HTTP routing, JSON utilities, health payload and API handlers) + - src/db.rs (Database init, schema creation and CRUD queries) + - src/performance.rs (Request/operation performance sampling and database size reporting) + - src/provider_registry.rs (Provider WebSocket lifecycle: register, heartbeat, status, disconnect) + - src/task_dispatcher.rs (Task creation, dispatch to providers, wait-for-result) + - src/ssh_bridge.rs (SSH session bridging over provider WebSocket) + - src/egress_tcp.rs (Egress TCP proxy for provider outbound connections) + - src/scheduler.rs (Scheduled task system: cron-like recurring tasks) + - src/microservice_proxy.rs (Microservice routing, HTTP proxy, health aggregation) + - src/overview.rs (Overview endpoint, node/task/service summary, load test) + - dev-frontend-proxy/ (Main-server nginx proxy exposing D601 `frontend-dev` on the dev public port) + - Dockerfile + - nginx.conf - frontend/ (Frontend web application container) - package.json - tsconfig.json diff --git a/scripts/cli.ts b/scripts/cli.ts index e779f0b8..8e3cb5f5 100644 --- a/scripts/cli.ts +++ b/scripts/cli.ts @@ -4,7 +4,7 @@ import { isRebuildableService, rebuildService, stackLogs, stackStatus, startStac import { parseE2ERunOptions, runE2E } from "./src/e2e"; import { emitError, emitJson } from "./src/output"; import { jobWithTail, listJobs, listJobsSummary, readJob, runJob } from "./src/jobs"; -import { parseCheckOptions, runChecks } from "./src/check"; +import { checkHelp, parseCheckOptions, runChecks } from "./src/check"; import { runSsh } from "./src/ssh"; import { extractRemoteCliOptions, runRemoteCli } from "./src/remote"; import { runMicroserviceCommand } from "./src/microservices"; @@ -30,13 +30,13 @@ function help(): unknown { { command: "help", description: "List supported commands." }, { command: "--main-server-ip <ip> <command>", description: "Run selected commands through the public frontend API; use --main-server-key only for legacy SSH transport." }, { command: "config show", description: "Validate and print config.json as the single source of truth." }, - { command: "check [--full|--files|--scripts-typecheck|--components|--compose|--logs]", description: "Run the lightweight default syntax/config gate; opt into file, type, Compose, or policy checks explicitly." }, + { command: "check [--full|--files|--scripts-typecheck|--components|--compose|--logs|--rust]", description: "Run the lightweight default syntax/config gate; Rust is opt-in and only allowed from D601 CI/dev execution." }, { command: "server start", description: "Fire-and-forget build/start for database, backend-core, frontend, provider gateway, and managed main-server user services." }, { command: "server stop", description: "Fire-and-forget docker-compose down for the fixed UniDesk stack." }, { command: "server status", description: "Show fixed ports, containers, service health, and public URLs." }, { command: "server swap status|ensure [--path /swapfile] [--size 2GiB] [--dry-run]", description: "Inspect or idempotently create host swap for low-memory main-server operation." }, { command: "server logs [--tail-bytes N]", description: "Return bounded tails from file logs and docker logs." }, - { command: "server rebuild <backend-core|frontend|provider-gateway|todo-note|code-queue-mgr|project-manager|baidu-netdisk|oa-event-flow>", description: "Build first, then serialize, force-recreate, and validate one Compose service." }, + { command: "server rebuild <backend-core|frontend|dev-frontend-proxy|provider-gateway|todo-note|code-queue-mgr|project-manager|baidu-netdisk|oa-event-flow>", description: "Build first, then serialize, force-recreate, and validate one Compose service." }, { command: "provider attach <providerId> [--master-server URL] [--up] [--force]", description: "Generate the minimal external provider-gateway env/compose bundle; only master server URL and provider id are required." }, { command: "ssh <providerId> [ssh-like args...]", description: "Open a Host SSH / WSL SSH maintenance session through the provider-gateway bridge with built-in remote helper tools in PATH." }, { command: "ssh <providerId> apply-patch [tool args...] < patch.diff", description: "Invoke the injected remote apply_patch helper directly over SSH passthrough and stream the patch from local stdin." }, @@ -85,6 +85,31 @@ function isHelpToken(value: string | undefined): boolean { return value === "help" || value === "--help" || value === "-h"; } +function serverHelp(action: string | undefined = undefined): unknown { + return { + command: action === undefined || isHelpToken(action) ? "server start|stop|status|swap|logs|rebuild" : `server ${action}`, + output: "json", + description: "Manage the fixed main-server Docker Compose stack without exposing backend-core REST publicly.", + usage: { + start: "bun scripts/cli.ts server start", + stop: "bun scripts/cli.ts server stop", + status: "bun scripts/cli.ts server status", + swap: "bun scripts/cli.ts server swap status|ensure [--path /swapfile] [--size 2GiB] [--dry-run]", + logs: "bun scripts/cli.ts server logs [--tail-bytes N]", + rebuild: "bun scripts/cli.ts server rebuild <backend-core|frontend|dev-frontend-proxy|provider-gateway|todo-note|code-queue-mgr|project-manager|baidu-netdisk|oa-event-flow>", + }, + publicEntrypoints: { + frontend: "prod UniDesk frontend", + devFrontend: "dev UniDesk frontend proxy to D601 unidesk-dev/frontend-dev", + providerIngress: "provider-gateway WebSocket ingress", + }, + rustBoundary: { + masterServer: "do not use server rebuild backend-core for Rust iteration; it would build locally", + d601: "use deploy apply --env dev --service backend-core and CI for Rust build/check", + }, + }; +} + function sshHelp(): unknown { return { command: "ssh", @@ -206,6 +231,10 @@ async function main(): Promise<void> { } if (top === "check") { + if (isHelpToken(sub)) { + emitJson(commandName, checkHelp()); + return; + } const result = runChecks(config, parseCheckOptions(args.slice(1))); emitJson(commandName, result, result.ok); if (!result.ok) process.exitCode = 1; @@ -213,6 +242,10 @@ async function main(): Promise<void> { } if (top === "server") { + if (isHelpToken(sub) || args.slice(2).some(isHelpToken)) { + emitJson(commandName, serverHelp(isHelpToken(sub) ? undefined : sub)); + return; + } if (sub === "start") { emitJson(commandName, startStack(config)); return; @@ -238,7 +271,7 @@ async function main(): Promise<void> { } if (sub === "rebuild") { if (!isRebuildableService(third)) { - throw new Error("server rebuild requires one of: backend-core, frontend, provider-gateway, todo-note, code-queue-mgr, project-manager, baidu-netdisk, oa-event-flow"); + throw new Error("server rebuild requires one of: backend-core, frontend, dev-frontend-proxy, provider-gateway, todo-note, code-queue-mgr, project-manager, baidu-netdisk, oa-event-flow"); } emitJson(commandName, rebuildService(config, third)); return; diff --git a/scripts/src/check.ts b/scripts/src/check.ts index 096473c8..23aa883d 100644 --- a/scripts/src/check.ts +++ b/scripts/src/check.ts @@ -21,7 +21,6 @@ const syntaxFiles = [ "scripts/src/docker.ts", "scripts/src/e2e.ts", "scripts/src/remote.ts", - "src/components/backend-core/src/index.ts", "src/components/frontend/src/index.ts", "src/components/frontend/src/app.tsx", "src/components/frontend/src/decision-center.tsx", @@ -40,6 +39,7 @@ export interface CheckOptions { components: boolean; compose: boolean; logs: boolean; + rust: boolean; } const defaultCheckOptions: CheckOptions = { @@ -49,8 +49,32 @@ const defaultCheckOptions: CheckOptions = { components: false, compose: false, logs: false, + rust: false, }; +export function checkHelp(): Record<string, unknown> { + return { + ok: true, + command: "check", + usage: "bun scripts/cli.ts check [--syntax-only|--full|--files|--scripts-typecheck|--components|--compose|--logs|--rust]", + defaultMode: "syntax/config only; Rust is never compiled on the master server by default", + options: [ + { name: "--syntax-only|--basic", description: "Run only config validation, Bun version and TypeScript syntax transpile." }, + { name: "--full", description: "Enable all local checks except Rust compilation." }, + { name: "--files", description: "Verify required entrypoint files, including backend-core Cargo files." }, + { name: "--scripts-typecheck", description: "Run scripts TypeScript typecheck." }, + { name: "--components", description: "Run component TypeScript typecheck." }, + { name: "--compose", description: "Render Docker Compose config." }, + { name: "--logs", description: "Check unified log rotation policy." }, + { name: "--rust", description: "Run cargo check only when UNIDESK_D601_RUST_CHECK=1 is set inside D601 CI/dev execution." }, + ], + rustBoundary: { + masterServer: "do not run cargo check/build here", + d601: "use deploy apply --env dev --service backend-core and CI with UNIDESK_D601_RUST_CHECK=1", + }, + }; +} + export function parseCheckOptions(args: string[]): CheckOptions { const options = { ...defaultCheckOptions }; for (const arg of args) { @@ -71,15 +95,10 @@ export function parseCheckOptions(args: string[]): CheckOptions { options.compose = true; } else if (arg === "--logs") { options.logs = true; + } else if (arg === "--rust") { + options.rust = true; } else if (arg === "--basic" || arg === "--syntax-only") { - options.full = false; - options.files = false; - options.scriptsTypecheck = false; - options.components = false; - options.compose = false; - options.logs = false; - } else if (arg === "--help" || arg === "-h") { - throw new Error("check usage: bun scripts/cli.ts check [--syntax-only|--full|--files|--scripts-typecheck|--components|--compose|--logs]"); + Object.assign(options, defaultCheckOptions); } else { throw new Error(`unknown check option: ${arg}`); } @@ -92,8 +111,8 @@ function fileItem(path: string): CheckItem { return { name: `file:${path}`, ok: existsSync(absolute), detail: absolute }; } -function commandItem(name: string, command: string[], timeoutMs = 30_000): CheckItem { - const result = runCommand(command, repoRoot, { timeoutMs }); +function commandItem(name: string, command: string[], timeoutMs = 30_000, env: NodeJS.ProcessEnv = process.env): CheckItem { + const result = runCommand(command, repoRoot, { timeoutMs, env }); return { name, ok: result.exitCode === 0, @@ -133,9 +152,9 @@ function syntaxItem(): CheckItem { function unifiedLogRotationItem(): CheckItem { const serviceFiles = [ - "src/components/backend-core/src/logger.ts", "src/components/frontend/src/index.ts", "src/components/provider-gateway/src/index.ts", + "src/components/microservices/code-queue-mgr/src/index.ts", "src/components/microservices/code-queue/src/index.ts", "src/components/microservices/k3sctl-adapter/src/index.ts", "src/components/microservices/mdtodo/src/index.ts", @@ -143,7 +162,6 @@ function unifiedLogRotationItem(): CheckItem { "src/components/microservices/baidu-netdisk/src/index.ts", "src/components/microservices/oa-event-flow/src/index.ts", "src/components/microservices/decision-center/src/index.ts", - "src/components/microservices/code-queue-mgr/src/index.ts", ]; const offenders = serviceFiles.flatMap((path) => { const text = readFileSync(rootPath(path), "utf8"); @@ -151,17 +169,41 @@ function unifiedLogRotationItem(): CheckItem { const missingWriter = !text.includes("createHourlyJsonlWriter"); return directLogAppend || missingWriter ? [{ path, directLogAppend, missingWriter }] : []; }); + const backendLogger = readFileSync(rootPath("src/components/backend-core/src/logger.rs"), "utf8"); + const backendMissingRotation = !backendLogger.includes("current_path") || !backendLogger.includes("prune"); + const backendDirectUnboundedAppend = backendLogger.includes("appendFileSync"); + if (backendMissingRotation || backendDirectUnboundedAppend) { + offenders.push({ path: "src/components/backend-core/src/logger.rs", directLogAppend: backendDirectUnboundedAppend, missingWriter: backendMissingRotation }); + } return { name: "logs:unified-hourly-rotation", ok: offenders.length === 0, detail: { sharedWriter: "src/components/shared/src/rotating-jsonl.ts", - checkedFiles: serviceFiles, + checkedFiles: ["src/components/backend-core/src/logger.rs", ...serviceFiles], offenders, }, }; } +function rustCheckItem(): CheckItem { + if (process.env.UNIDESK_D601_RUST_CHECK !== "1") { + return { + name: "rust:backend-core", + ok: false, + detail: { + skipped: true, + reason: "Rust compilation is intentionally not allowed on the master server; run it from D601 CI/dev deploy.", + enableOnD601: "UNIDESK_D601_RUST_CHECK=1 bun scripts/cli.ts check --rust", + deployPath: "bun scripts/cli.ts deploy apply --env dev --service backend-core", + }, + }; + } + const envPath = process.env.HOME ? `${process.env.HOME}/.cargo/bin:${process.env.PATH ?? ""}` : process.env.PATH; + const env = envPath ? { ...process.env, PATH: envPath } : process.env; + return commandItem("rust:backend-core", ["cargo", "check", "--manifest-path", "src/components/backend-core/Cargo.toml"], 180_000, env); +} + function skippedItem(name: string, reason: string, enableWith: string): CheckItem { return { name, ok: true, detail: { skipped: true, reason, enableWith } }; } @@ -178,7 +220,10 @@ export function runChecks(config: UniDeskConfig, options: CheckOptions = default fileItem("AGENTS.md"), fileItem("TEST.md"), fileItem("docker-compose.yml"), - fileItem("src/components/backend-core/src/index.ts"), + fileItem("src/components/backend-core/Cargo.toml"), + fileItem("src/components/backend-core/Cargo.lock"), + fileItem("src/components/backend-core/src/main.rs"), + fileItem("src/components/backend-core/src/http.rs"), fileItem("src/components/frontend/src/index.ts"), fileItem("src/components/provider-gateway/src/index.ts"), fileItem("src/components/microservices/oa-event-flow/src/index.ts"), @@ -193,19 +238,19 @@ export function runChecks(config: UniDeskConfig, options: CheckOptions = default items.push(skippedItem("files:required-entrypoints", "required file presence scan is opt-in", "--files or --full")); } if (options.scriptsTypecheck) { - items.push(commandItem("typescript:scripts", ["bunx", "tsc", "-p", "scripts/tsconfig.json", "--noEmit", "--pretty", "false"])); + items.push(commandItem("typescript:scripts", ["bunx", "tsc", "-p", "scripts/tsconfig.json", "--noEmit", "--pretty", "false"], 120_000)); } else { items.push(skippedItem("typescript:scripts", "scripts TypeScript typecheck is opt-in", "--scripts-typecheck or --full")); } if (options.logs) { items.push(unifiedLogRotationItem()); } else { - items.push(skippedItem("logs:unified-hourly-rotation", "heavy policy scan is opt-in", "--logs or --full")); + items.push(skippedItem("logs:unified-hourly-rotation", "policy scan is opt-in", "--logs or --full")); } if (options.components) { items.push(commandItem("typescript:components", ["bunx", "tsc", "-p", "src/tsconfig.check.json", "--pretty", "false"], 180_000)); } else { - items.push(skippedItem("typescript:components", "full component TypeScript check is opt-in", "--components or --full")); + items.push(skippedItem("typescript:components", "component TypeScript check is opt-in", "--components or --full")); } if (options.compose) { const compose = composeConfig(config); @@ -225,5 +270,10 @@ export function runChecks(config: UniDeskConfig, options: CheckOptions = default } else { items.push(skippedItem("docker-compose:config", "Docker Compose config rendering is opt-in", "--compose or --full")); } + if (options.rust) { + items.push(rustCheckItem()); + } else { + items.push(skippedItem("rust:backend-core", "Rust check/build must run on D601 dev deploy or CI, not on the master server", "--rust inside D601 CI with UNIDESK_D601_RUST_CHECK=1")); + } return { ok: items.every((item) => item.ok), mode: options.full ? "full" : "basic", options, items }; } diff --git a/scripts/src/ci.ts b/scripts/src/ci.ts index a55ca97b..6f25c9b3 100644 --- a/scripts/src/ci.ts +++ b/scripts/src/ci.ts @@ -104,6 +104,10 @@ function boolFlag(args: string[], name: string): boolean { return args.includes(name); } +function isHelpArg(value: string | undefined): boolean { + return value === "help" || value === "--help" || value === "-h"; +} + function shellQuote(value: string): string { return `'${value.replace(/'/gu, "'\\''")}'`; } @@ -811,7 +815,7 @@ function requireRunId(value: string): string { export async function runCiCommand(_config: UniDeskConfig, args: string[]): Promise<Record<string, unknown>> { const [action = "status", nameArg] = args; - if (action === "help" || action === "--help" || action === "-h") return help(); + if (isHelpArg(action) || args.slice(1).some(isHelpArg)) return help(); if (action === "install") return install(); if (action === "status") return status(); if (action === "run") { diff --git a/scripts/src/command.ts b/scripts/src/command.ts index 5f7a2c1b..0828a3d1 100644 --- a/scripts/src/command.ts +++ b/scripts/src/command.ts @@ -11,10 +11,11 @@ export interface CommandResult { timedOut: boolean; } -export function runCommand(command: string[], cwd: string, options: { timeoutMs?: number } = {}): CommandResult { +export function runCommand(command: string[], cwd: string, options: { timeoutMs?: number; env?: NodeJS.ProcessEnv } = {}): CommandResult { const result = spawnSync(command[0], command.slice(1), { cwd, encoding: "utf8", + env: options.env, maxBuffer: 1024 * 1024 * 8, timeout: options.timeoutMs, }); diff --git a/scripts/src/config.ts b/scripts/src/config.ts index 3874eda4..cba66a55 100644 --- a/scripts/src/config.ts +++ b/scripts/src/config.ts @@ -10,6 +10,7 @@ export interface UniDeskConfig { publicHost: string; core: { port: number; containerPort: number }; frontend: { port: number; containerPort: number }; + devFrontend: { port: number; containerPort: number }; database: { port: number; containerPort: number }; providerIngress: { port: number; containerPort: number }; restrictedHostAccess?: { bindHost: string; allowedSourceCidrs: string[] }; @@ -247,6 +248,7 @@ export function readConfig(): UniDeskConfig { publicHost: stringField(network, "publicHost", "network"), core: portPair(network, "core"), frontend: portPair(network, "frontend"), + devFrontend: portPair(network, "devFrontend"), database: portPair(network, "database"), providerIngress: portPair(network, "providerIngress"), restrictedHostAccess: optionalRestrictedHostAccess(network), diff --git a/scripts/src/debug.ts b/scripts/src/debug.ts index d52526b5..aebc1860 100644 --- a/scripts/src/debug.ts +++ b/scripts/src/debug.ts @@ -23,18 +23,9 @@ async function readJson(url: string, init?: RequestInit): Promise<unknown> { } function coreInternalFetch(path: string, init?: { method?: string; body?: unknown }): unknown { - const code = ` - const res = await fetch(${JSON.stringify(`http://backend-core:8080${path}`)}, ${JSON.stringify({ - method: init?.method ?? "GET", - headers: init?.body === undefined ? undefined : { "content-type": "application/json" }, - body: init?.body === undefined ? undefined : JSON.stringify(init.body), - })}); - const text = await res.text(); - let body = null; - try { body = text ? JSON.parse(text) : null; } catch { body = { text }; } - console.log(JSON.stringify({ ok: res.ok, status: res.status, body })); - `; - const result = runCommand(["docker", "exec", "unidesk-frontend", "bun", "-e", code], repoRoot); + const command = ["docker", "exec", "unidesk-backend-core", "backend-core", "--fetch-json", `http://127.0.0.1:8080${path}`, "--method", init?.method ?? "GET"]; + if (init?.body !== undefined) command.push("--body-json", JSON.stringify(init.body)); + const result = runCommand(command, repoRoot); if (result.exitCode !== 0) { return { ok: false, exitCode: result.exitCode, stdoutTail: result.stdout.slice(-1200), stderrTail: result.stderr.slice(-1200) }; } @@ -46,13 +37,14 @@ function coreInternalFetch(path: string, init?: { method?: string; body?: unknow } function coreDockerStatusSummary(): unknown { - const code = ` - const res = await fetch('http://backend-core:8080/api/nodes/docker-status'); - const text = await res.text(); - let body = null; - try { body = text ? JSON.parse(text) : null; } catch { body = { text }; } - const dockerStatuses = (body?.dockerStatuses || []).map((item) => { - const status = item.dockerStatus || {}; + const result = runCommand(["docker", "exec", "unidesk-backend-core", "backend-core", "--fetch-json", "http://127.0.0.1:8080/api/nodes/docker-status"], repoRoot); + if (result.exitCode !== 0) { + return { ok: false, exitCode: result.exitCode, stdoutTail: result.stdout.slice(-1200), stderrTail: result.stderr.slice(-1200) }; + } + try { + const parsed = JSON.parse(result.stdout.trim()) as { ok?: boolean; status?: number; body?: { dockerStatuses?: Array<Record<string, unknown>> } }; + const dockerStatuses = (parsed.body?.dockerStatuses || []).map((item) => { + const status = (item.dockerStatus || {}) as Record<string, unknown>; return { providerId: item.providerId, name: item.name, @@ -64,62 +56,37 @@ function coreDockerStatusSummary(): unknown { collectedAt: status.collectedAt, counts: status.counts, daemon: status.daemon, - containersPreview: (status.containers || []).slice(0, 8).map((container) => ({ - id: container.id, - name: container.name, - image: container.image, - state: container.state, - status: container.status, - ports: container.ports, - })), + containersPreview: Array.isArray(status.containers) ? status.containers.slice(0, 8) : [], }, }; }); - console.log(JSON.stringify({ ok: res.ok, status: res.status, body: { ok: body?.ok === true, dockerStatuses } })); - `; - const result = runCommand(["docker", "exec", "unidesk-frontend", "bun", "-e", code], repoRoot); - if (result.exitCode !== 0) { - return { ok: false, exitCode: result.exitCode, stdoutTail: result.stdout.slice(-1200), stderrTail: result.stderr.slice(-1200) }; - } - try { - return JSON.parse(result.stdout.trim()) as unknown; + return { ok: parsed.ok, status: parsed.status, body: { ok: parsed.body !== undefined, dockerStatuses } }; } catch { return { ok: true, stdoutTail: result.stdout.slice(-1200), stderrTail: result.stderr.slice(-1200) }; } } function coreSystemStatusSummary(): unknown { - const code = ` - const res = await fetch('http://backend-core:8080/api/nodes/system-status?limit=24'); - const text = await res.text(); - let body = null; - try { body = text ? JSON.parse(text) : null; } catch { body = { text }; } - const systemStatuses = (body?.systemStatuses || []).map((item) => { - const current = item.current || {}; + const result = runCommand(["docker", "exec", "unidesk-backend-core", "backend-core", "--fetch-json", "http://127.0.0.1:8080/api/nodes/system-status?limit=24"], repoRoot); + if (result.exitCode !== 0) { + return { ok: false, exitCode: result.exitCode, stdoutTail: result.stdout.slice(-1200), stderrTail: result.stderr.slice(-1200) }; + } + try { + const parsed = JSON.parse(result.stdout.trim()) as { ok?: boolean; status?: number; body?: { systemStatuses?: Array<Record<string, unknown>> } }; + const systemStatuses = (parsed.body?.systemStatuses || []).map((item) => { + const current = (item.current || {}) as Record<string, unknown>; + const history = Array.isArray(item.history) ? item.history : []; return { providerId: item.providerId, name: item.name, nodeStatus: item.nodeStatus, updatedAt: item.updatedAt, - current: item.current ? { - ok: current.ok, - collectedAt: current.collectedAt, - cpu: current.cpu, - memory: current.memory, - disk: current.disk, - } : null, - historyPreview: (item.history || []).slice(-8), - historyCount: (item.history || []).length, + current: item.current ? { ok: current.ok, collectedAt: current.collectedAt, cpu: current.cpu, memory: current.memory, disk: current.disk } : null, + historyPreview: history.slice(-8), + historyCount: history.length, }; }); - console.log(JSON.stringify({ ok: res.ok, status: res.status, body: { ok: body?.ok === true, systemStatuses } })); - `; - const result = runCommand(["docker", "exec", "unidesk-frontend", "bun", "-e", code], repoRoot); - if (result.exitCode !== 0) { - return { ok: false, exitCode: result.exitCode, stdoutTail: result.stdout.slice(-1200), stderrTail: result.stderr.slice(-1200) }; - } - try { - return JSON.parse(result.stdout.trim()) as unknown; + return { ok: parsed.ok, status: parsed.status, body: { ok: parsed.body !== undefined, systemStatuses } }; } catch { return { ok: true, stdoutTail: result.stdout.slice(-1200), stderrTail: result.stderr.slice(-1200) }; } diff --git a/scripts/src/deploy.ts b/scripts/src/deploy.ts index 83085a0e..18ec86b5 100644 --- a/scripts/src/deploy.ts +++ b/scripts/src/deploy.ts @@ -131,8 +131,8 @@ const nativeK3sInstallVersion = "v1.34.1+k3s1"; const nativeK3sImage = "rancher/k3s:v1.34.1-k3s1"; const nativeK3sCtrAddress = "/run/k3s/containerd/containerd.sock"; const unideskRepoUrl = "https://github.com/pikasTech/unidesk"; -const d601MaintenanceDeployAllowedServiceIds = new Set<string>(); -const devApplySupportedServiceIds = new Set<string>(); +const d601MaintenanceDeployAllowedServiceIds = new Set<string>(["backend-core", "frontend"]); +const devApplySupportedServiceIds = new Set<string>(["backend-core", "frontend"]); const deployEnvironmentTargets: Record<DeployEnvironment, DeployEnvironmentTarget> = { dev: { environment: "dev", @@ -195,7 +195,7 @@ function deployHelp(action: string | undefined = undefined): Record<string, unkn }, options: [ { name: "--file <path>", default: defaultDeployFile, description: "Desired-state manifest path relative to the repo root. JSON and ESM JS manifests are supported, for example deploy.json or develop.js." }, - { name: "--env <dev|prod>", description: "Read the named environment from origin/master:deploy.json. Direct D601 service apply is disabled in the current CI-only phase." }, + { name: "--env <dev|prod>", description: "Read the named environment from origin/master:deploy.json. Dev apply is enabled only for backend-core and frontend in D601 unidesk-dev." }, { name: "--service <id>", description: "Limit reconcile to one service from the manifest." }, { name: "--dry-run", description: "Prepare and validate without mutating the target service." }, { name: "--force", description: "Redeploy even when the live commit appears up to date." }, @@ -1005,6 +1005,7 @@ function dockerBuildTimeoutMs(service: UniDeskMicroserviceConfig, options: Deplo function devK3sPrepullImages(service: UniDeskMicroserviceConfig): string[] { if (!isDevK3sDeployService(service)) return []; + if (service.id === "backend-core") return ["rust:1-bookworm", "postgres:16-bookworm"]; return ["oven/bun:1-alpine"]; } @@ -2169,7 +2170,7 @@ async function applyOneService(config: UniDeskConfig, service: UniDeskMicroservi ok: false, serviceId: service.id, skipped: true, - reason: `D601 maintenance-channel direct deployment is disabled for ${service.id}. Current dev automation is CI-only; use ci run-dev-e2e for the temporary namespace smoke runner.`, + reason: `D601 dev deployment is allowed only for backend-core and frontend through the controlled deploy --env dev path; ${service.id} is not enabled. Use ci run-dev-e2e for smoke verification.`, steps, }; } @@ -2344,7 +2345,7 @@ function blockedD601MaintenanceDeployServices(config: UniDeskConfig, manifest: D } function d601MaintenanceDeployBlockMessage(blocked: string[]): string { - return `D601 maintenance-channel direct deployment is disabled for direct/managed services in the current CI-only phase; blocked services: ${blocked.join(", ")}. Use ci run-dev-e2e for dev smoke verification.`; + return `D601 dev deployment is enabled only for backend-core and frontend through deploy --env dev; blocked services: ${blocked.join(", ")}. Use ci run-dev-e2e for dev smoke verification.`; } async function runApplyNow(config: UniDeskConfig, manifest: DeployManifest, options: DeployOptions): Promise<Record<string, unknown>> { @@ -2396,7 +2397,7 @@ export async function runDeployCommand(config: UniDeskConfig | null, args: strin if (options.environment !== "dev") throw new Error("deploy apply --env prod is not enabled yet"); const unsupported = unsupportedDevApplyServices(manifest, options.serviceId); if (unsupported.length > 0) { - throw new Error(`deploy apply --env dev is disabled for direct service rollout in the current CI-only phase; unsupported selected services: ${unsupported.join(", ")}. Use ci run-dev-e2e for dev smoke verification.`); + throw new Error(`deploy apply --env dev currently supports only backend-core and frontend; unsupported selected services: ${unsupported.join(", ")}. Use ci run-dev-e2e for smoke verification.`); } if (config === null) throw new Error("deploy apply --env dev requires config.json"); if (!options.dryRun) { diff --git a/scripts/src/docker.ts b/scripts/src/docker.ts index 0b7f02a1..f717f7ef 100644 --- a/scripts/src/docker.ts +++ b/scripts/src/docker.ts @@ -20,7 +20,7 @@ export interface ContainerStatus { ports: string; } -const rebuildableServices = ["backend-core", "frontend", "provider-gateway", "todo-note", "code-queue-mgr", "project-manager", "baidu-netdisk", "oa-event-flow"] as const; +const rebuildableServices = ["backend-core", "frontend", "dev-frontend-proxy", "provider-gateway", "todo-note", "project-manager", "baidu-netdisk", "oa-event-flow", "code-queue-mgr"] as const; export type RebuildableService = typeof rebuildableServices[number]; export function isRebuildableService(value: string | undefined): value is RebuildableService { @@ -28,7 +28,6 @@ export function isRebuildableService(value: string | undefined): value is Rebuil } export function resolveComposeCommand(config: UniDeskConfig, envFile: string): string[] { - assertCanonicalServerRoot(config); const composeFile = rootPath(config.docker.composeFile); if (commandOk(["docker", "compose", "version"], repoRoot)) { return ["docker", "compose", "--env-file", envFile, "-f", composeFile, "-p", config.docker.projectName]; @@ -54,21 +53,7 @@ function envValue(value: string): string { return JSON.stringify(value); } -function assertCanonicalServerRoot(config: UniDeskConfig): void { - const canonicalRoot = resolve(config.providerGateway.upgrade.hostProjectRoot); - const currentRoot = resolve(repoRoot); - if (currentRoot !== canonicalRoot) { - throw new Error([ - "Main-server Docker Compose commands must run from the canonical UniDesk root.", - `canonicalRoot=${canonicalRoot}`, - `currentRoot=${currentRoot}`, - `Run from the canonical root: cd ${canonicalRoot} && bun scripts/cli.ts server <start|status|logs|rebuild|stop>`, - ].join(" ")); - } -} - export function writeComposeEnv(config: UniDeskConfig, freshLogPrefix: boolean): ComposeRuntimeEnv { - assertCanonicalServerRoot(config); const stateDir = rootPath(config.paths.stateDir); mkdirSync(stateDir, { recursive: true }); const envFile = join(stateDir, "docker-compose.env"); @@ -107,6 +92,7 @@ export function writeComposeEnv(config: UniDeskConfig, freshLogPrefix: boolean): UNIDESK_PUBLIC_HOST: config.network.publicHost, UNIDESK_CORE_PORT: String(config.network.core.port), UNIDESK_FRONTEND_PORT: String(config.network.frontend.port), + UNIDESK_DEV_FRONTEND_PORT: String(config.network.devFrontend.port), UNIDESK_DATABASE_PORT: String(config.network.database.port), UNIDESK_DATABASE_BIND_HOST: runtimeSecretWithDefault("UNIDESK_DATABASE_BIND_HOST", restrictedHostBind, "127.0.0.1"), UNIDESK_PROVIDER_INGRESS_PORT: String(config.network.providerIngress.port), @@ -141,10 +127,6 @@ export function writeComposeEnv(config: UniDeskConfig, freshLogPrefix: boolean): UNIDESK_LOG_DAY: logDay, UNIDESK_LOG_PREFIX: logPrefix, UNIDESK_LOG_RETENTION_BYTES: runtimeSecret("UNIDESK_LOG_RETENTION_BYTES") || "1GiB", - UNIDESK_FRONTEND_DEPLOY_SERVICE_ID: runtimeSecret("UNIDESK_FRONTEND_DEPLOY_SERVICE_ID") || "frontend", - UNIDESK_FRONTEND_DEPLOY_REPO: runtimeSecret("UNIDESK_FRONTEND_DEPLOY_REPO"), - UNIDESK_FRONTEND_DEPLOY_COMMIT: runtimeSecret("UNIDESK_FRONTEND_DEPLOY_COMMIT"), - UNIDESK_FRONTEND_DEPLOY_REQUESTED_COMMIT: runtimeSecret("UNIDESK_FRONTEND_DEPLOY_REQUESTED_COMMIT"), UNIDESK_HOST_ROOT_SSH_DIR: process.env.UNIDESK_HOST_ROOT_SSH_DIR || "/root/.ssh", UNIDESK_HOST_SSH_KEY_DIR: config.sshForwarding.keyDir, UNIDESK_HOST_SSH_HOST: config.sshForwarding.host, @@ -159,13 +141,16 @@ export function writeComposeEnv(config: UniDeskConfig, freshLogPrefix: boolean): UNIDESK_TODO_NOTE_REMINDER_SCAN_INTERVAL_MS: runtimeSecret("UNIDESK_TODO_NOTE_REMINDER_SCAN_INTERVAL_MS") || "30000", UNIDESK_TODO_NOTE_REMINDER_CLAUDEQQ_TIMEOUT_MS: runtimeSecret("UNIDESK_TODO_NOTE_REMINDER_CLAUDEQQ_TIMEOUT_MS") || "15000", UNIDESK_TODO_NOTE_REMINDER_CLAUDEQQ_SEND_ATTEMPTS: runtimeSecret("UNIDESK_TODO_NOTE_REMINDER_CLAUDEQQ_SEND_ATTEMPTS") || "3", - UNIDESK_CODE_QUEUE_MGR_DATABASE_POOL_MAX: runtimeSecret("UNIDESK_CODE_QUEUE_MGR_DATABASE_POOL_MAX") || "2", - UNIDESK_CODE_QUEUE_TRACE_DATABASE_POOL_MAX: runtimeSecret("UNIDESK_CODE_QUEUE_TRACE_DATABASE_POOL_MAX") || "1", UNIDESK_CODE_QUEUE_MINIMAX_API_KEY: runtimeSecret("UNIDESK_CODE_QUEUE_MINIMAX_API_KEY") || runtimeSecret("MINIMAX_API_KEY"), UNIDESK_CODE_QUEUE_MINIMAX_MODEL: runtimeSecret("UNIDESK_CODE_QUEUE_MINIMAX_MODEL") || runtimeSecret("MINIMAX_MODEL") || "MiniMax-M2.7", UNIDESK_CODE_QUEUE_MINIMAX_API_BASE: runtimeSecret("UNIDESK_CODE_QUEUE_MINIMAX_API_BASE") || runtimeSecret("MINIMAX_API_BASE") || "https://api.minimaxi.com/v1", UNIDESK_CODE_QUEUE_MINIMAX_JUDGE_TIMEOUT_MS: runtimeSecretWithDefault("UNIDESK_CODE_QUEUE_MINIMAX_JUDGE_TIMEOUT_MS", "90000", "60000"), UNIDESK_CODE_QUEUE_REMOTE_WORKDIR: runtimeSecret("UNIDESK_CODE_QUEUE_REMOTE_WORKDIR") || "/home/ubuntu", + UNIDESK_CODE_QUEUE_MAIN_PROVIDER_ID: runtimeSecret("UNIDESK_CODE_QUEUE_MAIN_PROVIDER_ID") || "D601", + UNIDESK_CODE_QUEUE_TRACE_DATABASE_URL: runtimeSecret("UNIDESK_CODE_QUEUE_TRACE_DATABASE_URL") + || `postgres://${config.database.user}:${config.database.password}@database:${config.network.database.containerPort}/${config.database.name}`, + UNIDESK_CODE_QUEUE_MGR_DATABASE_POOL_MAX: runtimeSecret("UNIDESK_CODE_QUEUE_MGR_DATABASE_POOL_MAX") || "2", + UNIDESK_CODE_QUEUE_TRACE_DATABASE_POOL_MAX: runtimeSecret("UNIDESK_CODE_QUEUE_TRACE_DATABASE_POOL_MAX") || "1", UNIDESK_CODE_QUEUE_EXECUTION_PROVIDER_IDS: runtimeSecret("UNIDESK_CODE_QUEUE_EXECUTION_PROVIDER_IDS") || "D601", UNIDESK_CODE_QUEUE_DEV_CONTAINER_DEFAULT_PROVIDER_ID: runtimeSecret("UNIDESK_CODE_QUEUE_DEV_CONTAINER_DEFAULT_PROVIDER_ID") || "D601", UNIDESK_CODE_QUEUE_DEV_CONTAINER_IMAGE: runtimeSecret("UNIDESK_CODE_QUEUE_DEV_CONTAINER_IMAGE"), @@ -250,10 +235,11 @@ export function rebuildService(config: UniDeskConfig, service: RebuildableServic const watchdogLog = rootPath(".state", "jobs", "compose-rebuild-watchdog.log"); const watchdogInnerScript = [ "set -euo pipefail", + "sleep 20", `cid=$(${shellJoin(listServiceContainersCommand)} || true)`, `if [ -z "$cid" ]; then echo "$(date -Is) compose_rebuild_watchdog_restore service=${service}" >> ${shellQuote(watchdogLog)}; ${shellJoin(restoreCommand)} >> ${shellQuote(watchdogLog)} 2>&1 || true; fi`, ].join("\n"); - const watchdogScript = `set -euo pipefail; sleep 20; ${shellJoin(["flock", "-w", "300", lockPath, "bash", "-lc", watchdogInnerScript])} || true`; + const watchdogScript = `set -euo pipefail; ${shellJoin(["flock", "-w", "300", lockPath, "bash", "-lc", watchdogInnerScript])} || true`; const validateScript = [ "ready=0", "for attempt in $(seq 1 60); do", @@ -308,6 +294,7 @@ export function rebuildService(config: UniDeskConfig, service: RebuildableServic function fixedPorts(config: UniDeskConfig): Array<{ name: string; port: number; listening: boolean }> { return [ { name: "frontend", port: config.network.frontend.port, listening: isPortListening(config.network.frontend.port) }, + { name: "dev-frontend", port: config.network.devFrontend.port, listening: isPortListening(config.network.devFrontend.port) }, { name: "provider-ingress", port: config.network.providerIngress.port, listening: isPortListening(config.network.providerIngress.port) }, ]; } @@ -356,9 +343,7 @@ function composeLockedScript(innerScript: string): string { "set -euo pipefail", `mkdir -p ${shellQuote(rootPath(".state", "locks"))}`, `echo ${shellJoin(["compose_lock_wait", lockPath])}`, - // Prevent background helpers spawned by the locked script from inheriting - // the compose lock fd and blocking later server rebuild jobs. - shellJoin(["flock", "-o", lockPath, "bash", "-lc", innerScript]), + shellJoin(["flock", lockPath, "bash", "-lc", innerScript]), ].join("; "); } @@ -409,8 +394,8 @@ async function probe(url: string): Promise<unknown> { } } -function dockerExecJson(container: string, code: string): unknown { - const result = runCommand(["docker", "exec", container, "bun", "-e", code], repoRoot); +function dockerExecJson(container: string, path: string): unknown { + const result = runCommand(["docker", "exec", container, "backend-core", "--fetch-json", `http://127.0.0.1:8080${path}`], repoRoot); if (result.exitCode !== 0) { return { ok: false, exitCode: result.exitCode, stdout: result.stdout.slice(-1200), stderr: result.stderr.slice(-1200) }; } @@ -433,8 +418,8 @@ export async function stackStatus(config: UniDeskConfig): Promise<unknown> { const databaseBindHost = runtimeValue("UNIDESK_DATABASE_BIND_HOST") || "127.0.0.1"; const oaEventFlowBindHost = runtimeValue("UNIDESK_OA_EVENT_FLOW_BIND_HOST") || "127.0.0.1"; const oaEventFlowPort = Number(runtimeValue("UNIDESK_OA_EVENT_FLOW_PORT") || "4255"); - const coreHealth = dockerExecJson("unidesk-backend-core", "fetch('http://127.0.0.1:8080/health').then(r=>r.json()).then(j=>console.log(JSON.stringify({ok:true,status:200,body:j}))).catch(e=>{console.log(JSON.stringify({ok:false,error:String(e)}));process.exit(1)})"); - const overview = dockerExecJson("unidesk-backend-core", "fetch('http://127.0.0.1:8080/api/overview').then(r=>r.json()).then(j=>console.log(JSON.stringify({ok:true,status:200,body:j}))).catch(e=>{console.log(JSON.stringify({ok:false,error:String(e)}));process.exit(1)})"); + const coreHealth = dockerExecJson("unidesk-backend-core", "/health"); + const overview = dockerExecJson("unidesk-backend-core", "/api/overview"); return { runtimeEnv, swap: swapStatus(), @@ -462,6 +447,7 @@ export async function stackStatus(config: UniDeskConfig): Promise<unknown> { ], internalPorts: [ { name: "backend-core", containerPort: config.network.core.containerPort, hostPort: null }, + { name: "dev-frontend-proxy", containerPort: config.network.devFrontend.containerPort, hostPort: config.network.devFrontend.port }, { name: "database", containerPort: config.network.database.containerPort, hostPort: null }, { name: "project-manager", containerPort: 4233, hostPort: null }, { name: "baidu-netdisk", containerPort: 4244, hostPort: null }, @@ -471,12 +457,14 @@ export async function stackStatus(config: UniDeskConfig): Promise<unknown> { health: { core: coreHealth, frontend: await probe(`http://127.0.0.1:${config.network.frontend.port}/health`), + devFrontend: await probe(`http://127.0.0.1:${config.network.devFrontend.port}/health`), providerIngress: await probe(`http://127.0.0.1:${config.network.providerIngress.port}/health`), database: dockerExec(config, "unidesk-database", ["pg_isready", "-U", config.database.user, "-d", config.database.name]), overview, }, urls: { frontend: `http://${config.network.publicHost}:${config.network.frontend.port}`, + devFrontend: `http://${config.network.publicHost}:${config.network.devFrontend.port}`, providerIngress: `ws://${config.network.publicHost}:${config.network.providerIngress.port}/ws/provider`, internalCore: `http://backend-core:${config.network.core.containerPort}`, internalDatabase: `postgres://${config.database.user}:***@database:${config.network.database.containerPort}/${config.database.name}`, @@ -507,7 +495,7 @@ export function stackLogs(config: UniDeskConfig, tailBytes: number): unknown { const truncated = sizeBytes > tailBytes; return { path, name: basename(path), sizeBytes, tailBytes, truncated, tail: tailFile(path, tailBytes) }; }); - const containerNames = ["unidesk-database", "unidesk-backend-core", "unidesk-frontend", "unidesk-provider-gateway-main", "todo-note-backend", "project-manager-backend", "baidu-netdisk-backend", "oa-event-flow-backend"]; + const containerNames = ["unidesk-database", "unidesk-backend-core", "unidesk-frontend", "unidesk-dev-frontend-proxy", "unidesk-provider-gateway-main", "todo-note-backend", "project-manager-backend", "baidu-netdisk-backend", "oa-event-flow-backend"]; const docker = containerNames.map((name) => { const result = runCommand(["docker", "logs", "--tail", "40", name], repoRoot); return { diff --git a/scripts/src/e2e.ts b/scripts/src/e2e.ts index 672cb8ff..52e1a20b 100644 --- a/scripts/src/e2e.ts +++ b/scripts/src/e2e.ts @@ -769,18 +769,9 @@ function runPsql(config: UniDeskConfig, sql: string): { ok: boolean; stdout: str } function dockerCoreJson(path: string, init?: { method?: string; body?: unknown }): unknown { - const code = ` - const res = await fetch(${JSON.stringify(`http://127.0.0.1:8080${path}`)}, ${JSON.stringify({ - method: init?.method ?? "GET", - headers: init?.body === undefined ? undefined : { "content-type": "application/json" }, - body: init?.body === undefined ? undefined : JSON.stringify(init.body), - })}); - const text = await res.text(); - let body = null; - try { body = text ? JSON.parse(text) : null; } catch { body = { text }; } - console.log(JSON.stringify({ ok: res.ok, status: res.status, body })); - `; - const result = runCommand(["docker", "exec", "unidesk-backend-core", "bun", "-e", code], repoRoot); + const command = ["docker", "exec", "unidesk-backend-core", "backend-core", "--fetch-json", `http://127.0.0.1:8080${path}`, "--method", init?.method ?? "GET"]; + if (init?.body !== undefined) command.push("--body-json", JSON.stringify(init.body)); + const result = runCommand(command, repoRoot); if (result.exitCode !== 0) return { ok: false, exitCode: result.exitCode, stdout: result.stdout.slice(-1200), stderr: result.stderr.slice(-1200) }; try { return JSON.parse(result.stdout.trim()) as unknown; @@ -1872,10 +1863,9 @@ async function frontendCheck(config: UniDeskConfig, urls: PublicUrls, checks: E2 await page.waitForSelector('[data-testid="decision-center-record-table"]', { timeout: 30000 }); await page.waitForFunction((title) => { const text = document.body.innerText; - const lowerText = text.toLowerCase(); return text.includes("Decision Center") && text.includes("G0/G1 目标") - && lowerText.includes("p0/p1 blocker") + && text.includes("P0/P1 Blocker") && text.includes("停放事项") && text.includes("最近会议/决议") && text.includes("查看原始JSON") @@ -2942,7 +2932,7 @@ async function frontendCheck(config: UniDeskConfig, urls: PublicUrls, checks: E2 addSelectedCheck(checks, options, "frontend:decision-center-visible", decisionCenterTextLower.includes("decision center") && decisionCenterText.includes("G0/G1 目标") - && decisionCenterTextLower.includes("p0/p1 blocker") + && decisionCenterText.includes("P0/P1 Blocker") && decisionCenterText.includes("停放事项") && decisionCenterText.includes("最近会议/决议") && decisionCenterText.includes("全部记录") diff --git a/scripts/src/microservices.ts b/scripts/src/microservices.ts index 1cd28cfd..aa978bbb 100644 --- a/scripts/src/microservices.ts +++ b/scripts/src/microservices.ts @@ -6,50 +6,9 @@ import { jsonByteLength, previewJson } from "./preview"; export function coreInternalFetch(path: string, init?: { method?: string; body?: unknown; maxResponseBytes?: number }): unknown { if (!path.startsWith("/")) throw new Error("core internal path must start with /"); const maxResponseBytes = Math.max(1024, Math.floor(init?.maxResponseBytes ?? 5_000_000)); - const code = ` - const res = await fetch(${JSON.stringify(`http://backend-core:8080${path}`)}, ${JSON.stringify({ - method: init?.method ?? "GET", - headers: init?.body === undefined ? undefined : { "content-type": "application/json" }, - body: init?.body === undefined ? undefined : JSON.stringify(init.body), - })}); - const maxResponseBytes = ${JSON.stringify(maxResponseBytes)}; - const reader = res.body?.getReader(); - const chunks = []; - let bytes = 0; - let responseTruncated = false; - if (reader) { - while (true) { - const { done, value } = await reader.read(); - if (done) break; - if (bytes + value.byteLength > maxResponseBytes) { - const keep = Math.max(0, maxResponseBytes - bytes); - if (keep > 0) { - chunks.push(value.slice(0, keep)); - bytes += keep; - } - responseTruncated = true; - try { await reader.cancel(); } catch {} - break; - } - chunks.push(value); - bytes += value.byteLength; - } - } - const buffer = new Uint8Array(bytes); - let offset = 0; - for (const chunk of chunks) { - buffer.set(chunk, offset); - offset += chunk.byteLength; - } - const text = new TextDecoder().decode(buffer); - let body = null; - try { body = text && !responseTruncated ? JSON.parse(text) : null; } catch { body = { text }; } - if (responseTruncated) { - body = { _unideskResponseTruncated: true, maxResponseBytes, bytesRead: bytes, contentLength: res.headers.get("content-length"), textPreview: text }; - } - console.log(JSON.stringify({ ok: res.ok, status: res.status, responseTruncated, responseBytesRead: bytes, responseContentLength: res.headers.get("content-length"), body })); - `; - const result = runCommand(["docker", "exec", "unidesk-frontend", "bun", "-e", code], repoRoot); + const command = ["docker", "exec", "unidesk-backend-core", "backend-core", "--fetch-json", `http://127.0.0.1:8080${path}`, "--method", init?.method ?? "GET", "--max-response-bytes", String(maxResponseBytes)]; + if (init?.body !== undefined) command.push("--body-json", JSON.stringify(init.body)); + const result = runCommand(command, repoRoot); if (result.exitCode !== 0) { return { ok: false, exitCode: result.exitCode, stdoutTail: result.stdout.slice(-1200), stderrTail: result.stderr.slice(-1200) }; } diff --git a/scripts/src/ssh.ts b/scripts/src/ssh.ts index 544ab2ef..c7e91697 100644 --- a/scripts/src/ssh.ts +++ b/scripts/src/ssh.ts @@ -805,13 +805,9 @@ export async function runSsh(config: UniDeskConfig, providerId: string, args: st const child = spawn("docker", [ "exec", "-i", - "-e", - `PROVIDER_TOKEN=${config.providerGateway.token}`, - "unidesk-frontend", - "bun", - "-e", - brokerSource(), - "--", + "unidesk-backend-core", + "backend-core", + "--ssh-broker", JSON.stringify(payload), ], { cwd: repoRoot, diff --git a/src/components/backend-core/Cargo.lock b/src/components/backend-core/Cargo.lock new file mode 100644 index 00000000..2ce32994 --- /dev/null +++ b/src/components/backend-core/Cargo.lock @@ -0,0 +1,2555 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 4 + +[[package]] +name = "android_system_properties" +version = "0.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "819e7219dbd41043ac279b19830f2efc897156490d7fd6ea916720117ee66311" +dependencies = [ + "libc", +] + +[[package]] +name = "anyhow" +version = "1.0.102" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7f202df86484c868dbad7eaa557ef785d5c66295e41b460ef922eca0723b842c" + +[[package]] +name = "async-trait" +version = "0.1.89" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9035ad2d096bed7955a320ee7e2230574d28fd3c3a0f186cbea1ff3c7eed5dbb" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "atomic-waker" +version = "1.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1505bd5d3d116872e7271a6d4e16d81d0c8570876c8de68093a09ac269d8aac0" + +[[package]] +name = "autocfg" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c08606f8c3cbf4ce6ec8e28fb0014a2c086708fe954eaa885384a6165172e7e8" + +[[package]] +name = "axum" +version = "0.7.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "edca88bc138befd0323b20752846e6587272d3b03b0343c8ea28a6f819e6e71f" +dependencies = [ + "async-trait", + "axum-core", + "base64", + "bytes", + "futures-util", + "http", + "http-body", + "http-body-util", + "hyper", + "hyper-util", + "itoa", + "matchit", + "memchr", + "mime", + "percent-encoding", + "pin-project-lite", + "rustversion", + "serde", + "serde_json", + "serde_path_to_error", + "serde_urlencoded", + "sha1", + "sync_wrapper", + "tokio", + "tokio-tungstenite", + "tower", + "tower-layer", + "tower-service", + "tracing", +] + +[[package]] +name = "axum-core" +version = "0.4.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "09f2bd6146b97ae3359fa0cc6d6b376d9539582c7b4220f041a33ec24c226199" +dependencies = [ + "async-trait", + "bytes", + "futures-util", + "http", + "http-body", + "http-body-util", + "mime", + "pin-project-lite", + "rustversion", + "sync_wrapper", + "tower-layer", + "tower-service", + "tracing", +] + +[[package]] +name = "base64" +version = "0.22.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72b3254f16251a8381aa12e40e3c4d2f0199f8c6508fbecb9d91f575e0fbb8c6" + +[[package]] +name = "bitflags" +version = "2.11.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c4512299f36f043ab09a583e57bceb5a5aab7a73db1805848e8fef3c9e8c78b3" + +[[package]] +name = "block-buffer" +version = "0.10.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3078c7629b62d3f0439517fa394996acacc5cbc91c5a20d8c658e77abd503a71" +dependencies = [ + "generic-array", +] + +[[package]] +name = "block-buffer" +version = "0.12.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cdd35008169921d80bc60d3d0ab416eecb028c4cd653352907921d95084790be" +dependencies = [ + "hybrid-array", +] + +[[package]] +name = "bumpalo" +version = "3.20.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5d20789868f4b01b2f2caec9f5c4e0213b41e3e5702a50157d699ae31ced2fcb" + +[[package]] +name = "byteorder" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fd0f2584146f6f2ef48085050886acf353beff7305ebd1ae69500e27c67f64b" + +[[package]] +name = "bytes" +version = "1.11.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1e748733b7cbc798e1434b6ac524f0c1ff2ab456fe201501e6497c8417a4fc33" + +[[package]] +name = "cc" +version = "1.2.62" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a1dce859f0832a7d088c4f1119888ab94ef4b5d6795d1ce05afb7fe159d79f98" +dependencies = [ + "find-msvc-tools", + "shlex", +] + +[[package]] +name = "cfg-if" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" + +[[package]] +name = "cfg_aliases" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "613afe47fcd5fac7ccf1db93babcb082c5994d996f20b8b159f2ad1658eb5724" + +[[package]] +name = "chacha20" +version = "0.10.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6f8d983286843e49675a4b7a2d174efe136dc93a18d69130dd18198a6c167601" +dependencies = [ + "cfg-if", + "cpufeatures 0.3.0", + "rand_core 0.10.1", +] + +[[package]] +name = "chrono" +version = "0.4.44" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c673075a2e0e5f4a1dde27ce9dee1ea4558c7ffe648f576438a20ca1d2acc4b0" +dependencies = [ + "iana-time-zone", + "js-sys", + "num-traits", + "serde", + "wasm-bindgen", + "windows-link", +] + +[[package]] +name = "cmov" +version = "0.5.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3f88a43d011fc4a6876cb7344703e297c71dda42494fee094d5f7c76bf13f746" + +[[package]] +name = "const-oid" +version = "0.10.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a6ef517f0926dd24a1582492c791b6a4818a4d94e789a334894aa15b0d12f55c" + +[[package]] +name = "core-foundation-sys" +version = "0.8.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "773648b94d0e5d620f64f280777445740e61fe701025087ec8b57f45c791888b" + +[[package]] +name = "cpufeatures" +version = "0.2.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "59ed5838eebb26a2bb2e58f6d5b5316989ae9d08bab10e0e6d103e656d1b0280" +dependencies = [ + "libc", +] + +[[package]] +name = "cpufeatures" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8b2a41393f66f16b0823bb79094d54ac5fbd34ab292ddafb9a0456ac9f87d201" +dependencies = [ + "libc", +] + +[[package]] +name = "crypto-common" +version = "0.1.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "78c8292055d1c1df0cce5d180393dc8cce0abec0a7102adb6c7b1eef6016d60a" +dependencies = [ + "generic-array", + "typenum", +] + +[[package]] +name = "crypto-common" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "77727bb15fa921304124b128af125e7e3b968275d1b108b379190264f4423710" +dependencies = [ + "hybrid-array", +] + +[[package]] +name = "ctutils" +version = "0.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7d5515a3834141de9eafb9717ad39eea8247b5674e6066c404e8c4b365d2a29e" +dependencies = [ + "cmov", +] + +[[package]] +name = "data-encoding" +version = "2.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a4ae5f15dda3c708c0ade84bfee31ccab44a3da4f88015ed22f63732abe300c8" + +[[package]] +name = "deadpool" +version = "0.12.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0be2b1d1d6ec8d846f05e137292d0b89133caf95ef33695424c09568bdd39b1b" +dependencies = [ + "deadpool-runtime", + "lazy_static", + "num_cpus", + "tokio", +] + +[[package]] +name = "deadpool-postgres" +version = "0.14.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3d697d376cbfa018c23eb4caab1fd1883dd9c906a8c034e8d9a3cb06a7e0bef9" +dependencies = [ + "async-trait", + "deadpool", + "getrandom 0.2.17", + "tokio", + "tokio-postgres", + "tracing", +] + +[[package]] +name = "deadpool-runtime" +version = "0.1.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "092966b41edc516079bdf31ec78a2e0588d1d0c08f78b91d8307215928642b2b" +dependencies = [ + "tokio", +] + +[[package]] +name = "digest" +version = "0.10.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292" +dependencies = [ + "block-buffer 0.10.4", + "crypto-common 0.1.7", +] + +[[package]] +name = "digest" +version = "0.11.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f1dd6dbb5841937940781866fa1281a1ff7bd3bf827091440879f9994983d5c2" +dependencies = [ + "block-buffer 0.12.0", + "const-oid", + "crypto-common 0.2.1", + "ctutils", +] + +[[package]] +name = "displaydoc" +version = "0.2.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "97369cbbc041bc366949bc74d34658d6cda5621039731c6310521892a3a20ae0" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "equivalent" +version = "1.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "877a4ace8713b0bcf2a4e7eec82529c029f1d0619886d18145fea96c3ffe5c0f" + +[[package]] +name = "errno" +version = "0.3.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" +dependencies = [ + "libc", + "windows-sys 0.61.2", +] + +[[package]] +name = "fallible-iterator" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4443176a9f2c162692bd3d352d745ef9413eec5782a80d8fd6f8a1ac692a07f7" + +[[package]] +name = "fastrand" +version = "2.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9f1f227452a390804cdb637b74a86990f2a7d7ba4b7d5693aac9b4dd6defd8d6" + +[[package]] +name = "find-msvc-tools" +version = "0.1.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5baebc0774151f905a1a2cc41989300b1e6fbb29aff0ceffa1064fdd3088d582" + +[[package]] +name = "foldhash" +version = "0.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d9c4f5dac5e15c24eb999c26181a6ca40b39fe946cbe4c263c7209467bc83af2" + +[[package]] +name = "form_urlencoded" +version = "1.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cb4cb245038516f5f85277875cdaa4f7d2c9a0fa0468de06ed190163b1581fcf" +dependencies = [ + "percent-encoding", +] + +[[package]] +name = "futures-channel" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "07bbe89c50d7a535e539b8c17bc0b49bdb77747034daa8087407d655f3f7cc1d" +dependencies = [ + "futures-core", + "futures-sink", +] + +[[package]] +name = "futures-core" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7e3450815272ef58cec6d564423f6e755e25379b217b0bc688e295ba24df6b1d" + +[[package]] +name = "futures-io" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cecba35d7ad927e23624b22ad55235f2239cfa44fd10428eecbeba6d6a717718" + +[[package]] +name = "futures-macro" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e835b70203e41293343137df5c0664546da5745f82ec9b84d40be8336958447b" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "futures-sink" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c39754e157331b013978ec91992bde1ac089843443c49cbc7f46150b0fad0893" + +[[package]] +name = "futures-task" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "037711b3d59c33004d3856fbdc83b99d4ff37a24768fa1be9ce3538a1cde4393" + +[[package]] +name = "futures-util" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "389ca41296e6190b48053de0321d02a77f32f8a5d2461dd38762c0593805c6d6" +dependencies = [ + "futures-core", + "futures-io", + "futures-macro", + "futures-sink", + "futures-task", + "memchr", + "pin-project-lite", + "slab", +] + +[[package]] +name = "generic-array" +version = "0.14.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "85649ca51fd72272d7821adaf274ad91c288277713d9c18820d8499a7ff69e9a" +dependencies = [ + "typenum", + "version_check", +] + +[[package]] +name = "getrandom" +version = "0.2.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ff2abc00be7fca6ebc474524697ae276ad847ad0a6b3faa4bcb027e9a4614ad0" +dependencies = [ + "cfg-if", + "js-sys", + "libc", + "wasi 0.11.1+wasi-snapshot-preview1", + "wasm-bindgen", +] + +[[package]] +name = "getrandom" +version = "0.3.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "899def5c37c4fd7b2664648c28120ecec138e4d395b459e5ca34f9cce2dd77fd" +dependencies = [ + "cfg-if", + "js-sys", + "libc", + "r-efi 5.3.0", + "wasip2", + "wasm-bindgen", +] + +[[package]] +name = "getrandom" +version = "0.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0de51e6874e94e7bf76d726fc5d13ba782deca734ff60d5bb2fb2607c7406555" +dependencies = [ + "cfg-if", + "libc", + "r-efi 6.0.0", + "rand_core 0.10.1", + "wasip2", + "wasip3", +] + +[[package]] +name = "hashbrown" +version = "0.15.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9229cfe53dfd69f0609a49f65461bd93001ea1ef889cd5529dd176593f5338a1" +dependencies = [ + "foldhash", +] + +[[package]] +name = "hashbrown" +version = "0.17.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ed5909b6e89a2db4456e54cd5f673791d7eca6732202bbf2a9cc504fe2f9b84a" + +[[package]] +name = "heck" +version = "0.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea" + +[[package]] +name = "hermit-abi" +version = "0.5.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fc0fef456e4baa96da950455cd02c081ca953b141298e41db3fc7e36b1da849c" + +[[package]] +name = "hmac" +version = "0.13.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6303bc9732ae41b04cb554b844a762b4115a61bfaa81e3e83050991eeb56863f" +dependencies = [ + "digest 0.11.3", +] + +[[package]] +name = "http" +version = "1.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e3ba2a386d7f85a81f119ad7498ebe444d2e22c2af0b86b069416ace48b3311a" +dependencies = [ + "bytes", + "itoa", +] + +[[package]] +name = "http-body" +version = "1.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1efedce1fb8e6913f23e0c92de8e62cd5b772a67e7b3946df930a62566c93184" +dependencies = [ + "bytes", + "http", +] + +[[package]] +name = "http-body-util" +version = "0.1.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b021d93e26becf5dc7e1b75b1bed1fd93124b374ceb73f43d4d4eafec896a64a" +dependencies = [ + "bytes", + "futures-core", + "http", + "http-body", + "pin-project-lite", +] + +[[package]] +name = "httparse" +version = "1.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6dbf3de79e51f3d586ab4cb9d5c3e2c14aa28ed23d180cf89b4df0454a69cc87" + +[[package]] +name = "httpdate" +version = "1.0.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "df3b46402a9d5adb4c86a0cf463f42e19994e3ee891101b1841f30a545cb49a9" + +[[package]] +name = "hybrid-array" +version = "0.4.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9155a582abd142abc056962c29e3ce5ff2ad5469f4246b537ed42c5deba857da" +dependencies = [ + "typenum", +] + +[[package]] +name = "hyper" +version = "1.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6299f016b246a94207e63da54dbe807655bf9e00044f73ded42c3ac5305fbcca" +dependencies = [ + "atomic-waker", + "bytes", + "futures-channel", + "futures-core", + "http", + "http-body", + "httparse", + "httpdate", + "itoa", + "pin-project-lite", + "smallvec", + "tokio", + "want", +] + +[[package]] +name = "hyper-rustls" +version = "0.27.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "33ca68d021ef39cf6463ab54c1d0f5daf03377b70561305bb89a8f83aab66e0f" +dependencies = [ + "http", + "hyper", + "hyper-util", + "rustls", + "tokio", + "tokio-rustls", + "tower-service", + "webpki-roots", +] + +[[package]] +name = "hyper-util" +version = "0.1.20" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "96547c2556ec9d12fb1578c4eaf448b04993e7fb79cbaad930a656880a6bdfa0" +dependencies = [ + "base64", + "bytes", + "futures-channel", + "futures-util", + "http", + "http-body", + "hyper", + "ipnet", + "libc", + "percent-encoding", + "pin-project-lite", + "socket2", + "tokio", + "tower-service", + "tracing", +] + +[[package]] +name = "iana-time-zone" +version = "0.1.65" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e31bc9ad994ba00e440a8aa5c9ef0ec67d5cb5e5cb0cc7f8b744a35b389cc470" +dependencies = [ + "android_system_properties", + "core-foundation-sys", + "iana-time-zone-haiku", + "js-sys", + "log", + "wasm-bindgen", + "windows-core", +] + +[[package]] +name = "iana-time-zone-haiku" +version = "0.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f31827a206f56af32e590ba56d5d2d085f558508192593743f16b2306495269f" +dependencies = [ + "cc", +] + +[[package]] +name = "icu_collections" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2984d1cd16c883d7935b9e07e44071dca8d917fd52ecc02c04d5fa0b5a3f191c" +dependencies = [ + "displaydoc", + "potential_utf", + "utf8_iter", + "yoke", + "zerofrom", + "zerovec", +] + +[[package]] +name = "icu_locale_core" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "92219b62b3e2b4d88ac5119f8904c10f8f61bf7e95b640d25ba3075e6cac2c29" +dependencies = [ + "displaydoc", + "litemap", + "tinystr", + "writeable", + "zerovec", +] + +[[package]] +name = "icu_normalizer" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c56e5ee99d6e3d33bd91c5d85458b6005a22140021cc324cea84dd0e72cff3b4" +dependencies = [ + "icu_collections", + "icu_normalizer_data", + "icu_properties", + "icu_provider", + "smallvec", + "zerovec", +] + +[[package]] +name = "icu_normalizer_data" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "da3be0ae77ea334f4da67c12f149704f19f81d1adf7c51cf482943e84a2bad38" + +[[package]] +name = "icu_properties" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bee3b67d0ea5c2cca5003417989af8996f8604e34fb9ddf96208a033901e70de" +dependencies = [ + "icu_collections", + "icu_locale_core", + "icu_properties_data", + "icu_provider", + "zerotrie", + "zerovec", +] + +[[package]] +name = "icu_properties_data" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8e2bbb201e0c04f7b4b3e14382af113e17ba4f63e2c9d2ee626b720cbce54a14" + +[[package]] +name = "icu_provider" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "139c4cf31c8b5f33d7e199446eff9c1e02decfc2f0eec2c8d71f65befa45b421" +dependencies = [ + "displaydoc", + "icu_locale_core", + "writeable", + "yoke", + "zerofrom", + "zerotrie", + "zerovec", +] + +[[package]] +name = "id-arena" +version = "2.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3d3067d79b975e8844ca9eb072e16b31c3c1c36928edf9c6789548c524d0d954" + +[[package]] +name = "idna" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3b0875f23caa03898994f6ddc501886a45c7d3d62d04d2d90788d47be1b1e4de" +dependencies = [ + "idna_adapter", + "smallvec", + "utf8_iter", +] + +[[package]] +name = "idna_adapter" +version = "1.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cb68373c0d6620ef8105e855e7745e18b0d00d3bdb07fb532e434244cdb9a714" +dependencies = [ + "icu_normalizer", + "icu_properties", +] + +[[package]] +name = "indexmap" +version = "2.14.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d466e9454f08e4a911e14806c24e16fba1b4c121d1ea474396f396069cf949d9" +dependencies = [ + "equivalent", + "hashbrown 0.17.1", + "serde", + "serde_core", +] + +[[package]] +name = "ipnet" +version = "2.12.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d98f6fed1fde3f8c21bc40a1abb88dd75e67924f9cffc3ef95607bad8017f8e2" + +[[package]] +name = "itoa" +version = "1.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" + +[[package]] +name = "js-sys" +version = "0.3.98" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "67df7112613f8bfd9150013a0314e196f4800d3201ae742489d999db2f979f08" +dependencies = [ + "cfg-if", + "futures-util", + "once_cell", + "wasm-bindgen", +] + +[[package]] +name = "lazy_static" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bbd2bcb4c963f2ddae06a2efc7e9f3591312473c50c6685e1f298068316e66fe" + +[[package]] +name = "leb128fmt" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "09edd9e8b54e49e587e4f6295a7d29c3ea94d469cb40ab8ca70b288248a81db2" + +[[package]] +name = "libc" +version = "0.2.186" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "68ab91017fe16c622486840e4c83c9a37afeff978bd239b5293d61ece587de66" + +[[package]] +name = "libredox" +version = "0.1.16" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e02f3bb43d335493c96bf3fd3a321600bf6bd07ed34bc64118e9293bdffea46c" +dependencies = [ + "libc", +] + +[[package]] +name = "linux-raw-sys" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32a66949e030da00e8c7d4434b251670a91556f4144941d37452769c25d58a53" + +[[package]] +name = "litemap" +version = "0.8.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "92daf443525c4cce67b150400bc2316076100ce0b3686209eb8cf3c31612e6f0" + +[[package]] +name = "lock_api" +version = "0.4.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "224399e74b87b5f3557511d98dff8b14089b3dadafcab6bb93eab67d3aace965" +dependencies = [ + "scopeguard", +] + +[[package]] +name = "log" +version = "0.4.29" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5e5032e24019045c762d3c0f28f5b6b8bbf38563a65908389bf7978758920897" + +[[package]] +name = "lru-slab" +version = "0.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "112b39cec0b298b6c1999fee3e31427f74f676e4cb9879ed1a121b43661a4154" + +[[package]] +name = "matchit" +version = "0.7.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0e7465ac9959cc2b1404e8e2367b43684a6d13790fe23056cc8c6c5a6b7bcb94" + +[[package]] +name = "md-5" +version = "0.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "69b6441f590336821bb897fb28fc622898ccceb1d6cea3fde5ea86b090c4de98" +dependencies = [ + "cfg-if", + "digest 0.11.3", +] + +[[package]] +name = "memchr" +version = "2.8.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f8ca58f447f06ed17d5fc4043ce1b10dd205e060fb3ce5b979b8ed8e59ff3f79" + +[[package]] +name = "mime" +version = "0.3.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6877bb514081ee2a7ff5ef9de3281f14a4dd4bceac4c09388074a6b5df8a139a" + +[[package]] +name = "mio" +version = "1.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "50b7e5b27aa02a74bac8c3f23f448f8d87ff11f92d3aac1a6ed369ee08cc56c1" +dependencies = [ + "libc", + "wasi 0.11.1+wasi-snapshot-preview1", + "windows-sys 0.61.2", +] + +[[package]] +name = "num-traits" +version = "0.2.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "071dfc062690e90b734c0b2273ce72ad0ffa95f0c74596bc250dcfd960262841" +dependencies = [ + "autocfg", +] + +[[package]] +name = "num_cpus" +version = "1.17.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "91df4bbde75afed763b708b7eee1e8e7651e02d97f6d5dd763e89367e957b23b" +dependencies = [ + "hermit-abi", + "libc", +] + +[[package]] +name = "objc2-core-foundation" +version = "0.3.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2a180dd8642fa45cdb7dd721cd4c11b1cadd4929ce112ebd8b9f5803cc79d536" +dependencies = [ + "bitflags", +] + +[[package]] +name = "objc2-system-configuration" +version = "0.3.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7216bd11cbda54ccabcab84d523dc93b858ec75ecfb3a7d89513fa22464da396" +dependencies = [ + "objc2-core-foundation", +] + +[[package]] +name = "once_cell" +version = "1.21.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50" + +[[package]] +name = "parking_lot" +version = "0.12.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "93857453250e3077bd71ff98b6a65ea6621a19bb0f559a85248955ac12c45a1a" +dependencies = [ + "lock_api", + "parking_lot_core", +] + +[[package]] +name = "parking_lot_core" +version = "0.9.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2621685985a2ebf1c516881c026032ac7deafcda1a2c9b7850dc81e3dfcb64c1" +dependencies = [ + "cfg-if", + "libc", + "redox_syscall", + "smallvec", + "windows-link", +] + +[[package]] +name = "percent-encoding" +version = "2.3.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9b4f627cb1b25917193a259e49bdad08f671f8d9708acfd5fe0a8c1455d87220" + +[[package]] +name = "phf" +version = "0.13.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c1562dc717473dbaa4c1f85a36410e03c047b2e7df7f45ee938fbef64ae7fadf" +dependencies = [ + "phf_shared", + "serde", +] + +[[package]] +name = "phf_shared" +version = "0.13.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e57fef6bc5981e38c2ce2d63bfa546861309f875b8a75f092d1d54ae2d64f266" +dependencies = [ + "siphasher", +] + +[[package]] +name = "pin-project-lite" +version = "0.2.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd" + +[[package]] +name = "postgres-protocol" +version = "0.6.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "56201207dac53e2f38e848e31b4b91616a6bb6e0c7205b77718994a7f49e70fc" +dependencies = [ + "base64", + "byteorder", + "bytes", + "fallible-iterator", + "hmac", + "md-5", + "memchr", + "rand 0.10.1", + "sha2 0.11.0", + "stringprep", +] + +[[package]] +name = "postgres-types" +version = "0.2.13" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8dc729a129e682e8d24170cd30ae1aa01b336b096cbb56df6d534ffec133d186" +dependencies = [ + "bytes", + "chrono", + "fallible-iterator", + "postgres-protocol", + "serde_core", + "serde_json", +] + +[[package]] +name = "potential_utf" +version = "0.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0103b1cef7ec0cf76490e969665504990193874ea05c85ff9bab8b911d0a0564" +dependencies = [ + "zerovec", +] + +[[package]] +name = "ppv-lite86" +version = "0.2.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "85eae3c4ed2f50dcfe72643da4befc30deadb458a9b590d720cde2f2b1e97da9" +dependencies = [ + "zerocopy", +] + +[[package]] +name = "prettyplease" +version = "0.2.37" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "479ca8adacdd7ce8f1fb39ce9ecccbfe93a3f1344b3d0d97f20bc0196208f62b" +dependencies = [ + "proc-macro2", + "syn", +] + +[[package]] +name = "proc-macro2" +version = "1.0.106" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8fd00f0bb2e90d81d1044c2b32617f68fcb9fa3bb7640c23e9c748e53fb30934" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "quinn" +version = "0.11.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b9e20a958963c291dc322d98411f541009df2ced7b5a4f2bd52337638cfccf20" +dependencies = [ + "bytes", + "cfg_aliases", + "pin-project-lite", + "quinn-proto", + "quinn-udp", + "rustc-hash", + "rustls", + "socket2", + "thiserror 2.0.18", + "tokio", + "tracing", + "web-time", +] + +[[package]] +name = "quinn-proto" +version = "0.11.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "434b42fec591c96ef50e21e886936e66d3cc3f737104fdb9b737c40ffb94c098" +dependencies = [ + "bytes", + "getrandom 0.3.4", + "lru-slab", + "rand 0.9.4", + "ring", + "rustc-hash", + "rustls", + "rustls-pki-types", + "slab", + "thiserror 2.0.18", + "tinyvec", + "tracing", + "web-time", +] + +[[package]] +name = "quinn-udp" +version = "0.5.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "addec6a0dcad8a8d96a771f815f0eaf55f9d1805756410b39f5fa81332574cbd" +dependencies = [ + "cfg_aliases", + "libc", + "once_cell", + "socket2", + "tracing", + "windows-sys 0.60.2", +] + +[[package]] +name = "quote" +version = "1.0.45" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "41f2619966050689382d2b44f664f4bc593e129785a36d6ee376ddf37259b924" +dependencies = [ + "proc-macro2", +] + +[[package]] +name = "r-efi" +version = "5.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "69cdb34c158ceb288df11e18b4bd39de994f6657d83847bdffdbd7f346754b0f" + +[[package]] +name = "r-efi" +version = "6.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf" + +[[package]] +name = "rand" +version = "0.8.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5ca0ecfa931c29007047d1bc58e623ab12e5590e8c7cc53200d5202b69266d8a" +dependencies = [ + "libc", + "rand_chacha 0.3.1", + "rand_core 0.6.4", +] + +[[package]] +name = "rand" +version = "0.9.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "44c5af06bb1b7d3216d91932aed5265164bf384dc89cd6ba05cf59a35f5f76ea" +dependencies = [ + "rand_chacha 0.9.0", + "rand_core 0.9.5", +] + +[[package]] +name = "rand" +version = "0.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d2e8e8bcc7961af1fdac401278c6a831614941f6164ee3bf4ce61b7edb162207" +dependencies = [ + "chacha20", + "getrandom 0.4.2", + "rand_core 0.10.1", +] + +[[package]] +name = "rand_chacha" +version = "0.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6c10a63a0fa32252be49d21e7709d4d4baf8d231c2dbce1eaa8141b9b127d88" +dependencies = [ + "ppv-lite86", + "rand_core 0.6.4", +] + +[[package]] +name = "rand_chacha" +version = "0.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d3022b5f1df60f26e1ffddd6c66e8aa15de382ae63b3a0c1bfc0e4d3e3f325cb" +dependencies = [ + "ppv-lite86", + "rand_core 0.9.5", +] + +[[package]] +name = "rand_core" +version = "0.6.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ec0be4795e2f6a28069bec0b5ff3e2ac9bafc99e6a9a7dc3547996c5c816922c" +dependencies = [ + "getrandom 0.2.17", +] + +[[package]] +name = "rand_core" +version = "0.9.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "76afc826de14238e6e8c374ddcc1fa19e374fd8dd986b0d2af0d02377261d83c" +dependencies = [ + "getrandom 0.3.4", +] + +[[package]] +name = "rand_core" +version = "0.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "63b8176103e19a2643978565ca18b50549f6101881c443590420e4dc998a3c69" + +[[package]] +name = "redox_syscall" +version = "0.5.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ed2bf2547551a7053d6fdfafda3f938979645c44812fbfcda098faae3f1a362d" +dependencies = [ + "bitflags", +] + +[[package]] +name = "reqwest" +version = "0.12.28" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "eddd3ca559203180a307f12d114c268abf583f59b03cb906fd0b3ff8646c1147" +dependencies = [ + "base64", + "bytes", + "futures-core", + "futures-util", + "http", + "http-body", + "http-body-util", + "hyper", + "hyper-rustls", + "hyper-util", + "js-sys", + "log", + "percent-encoding", + "pin-project-lite", + "quinn", + "rustls", + "rustls-pki-types", + "serde", + "serde_json", + "serde_urlencoded", + "sync_wrapper", + "tokio", + "tokio-rustls", + "tokio-util", + "tower", + "tower-http", + "tower-service", + "url", + "wasm-bindgen", + "wasm-bindgen-futures", + "wasm-streams", + "web-sys", + "webpki-roots", +] + +[[package]] +name = "ring" +version = "0.17.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a4689e6c2294d81e88dc6261c768b63bc4fcdb852be6d1352498b114f61383b7" +dependencies = [ + "cc", + "cfg-if", + "getrandom 0.2.17", + "libc", + "untrusted", + "windows-sys 0.52.0", +] + +[[package]] +name = "rustc-hash" +version = "2.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "94300abf3f1ae2e2b8ffb7b58043de3d399c73fa6f4b73826402a5c457614dbe" + +[[package]] +name = "rustix" +version = "1.1.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6fe4565b9518b83ef4f91bb47ce29620ca828bd32cb7e408f0062e9930ba190" +dependencies = [ + "bitflags", + "errno", + "libc", + "linux-raw-sys", + "windows-sys 0.61.2", +] + +[[package]] +name = "rustls" +version = "0.23.40" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ef86cd5876211988985292b91c96a8f2d298df24e75989a43a3c73f2d4d8168b" +dependencies = [ + "once_cell", + "ring", + "rustls-pki-types", + "rustls-webpki", + "subtle", + "zeroize", +] + +[[package]] +name = "rustls-pki-types" +version = "1.14.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "30a7197ae7eb376e574fe940d068c30fe0462554a3ddbe4eca7838e049c937a9" +dependencies = [ + "web-time", + "zeroize", +] + +[[package]] +name = "rustls-webpki" +version = "0.103.13" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "61c429a8649f110dddef65e2a5ad240f747e85f7758a6bccc7e5777bd33f756e" +dependencies = [ + "ring", + "rustls-pki-types", + "untrusted", +] + +[[package]] +name = "rustversion" +version = "1.0.22" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b39cdef0fa800fc44525c84ccb54a029961a8215f9619753635a9c0d2538d46d" + +[[package]] +name = "ryu" +version = "1.0.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9774ba4a74de5f7b1c1451ed6cd5285a32eddb5cccb8cc655a4e50009e06477f" + +[[package]] +name = "scopeguard" +version = "1.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "94143f37725109f92c262ed2cf5e59bce7498c01bcc1502d7b9afe439a4e9f49" + +[[package]] +name = "semver" +version = "1.0.28" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8a7852d02fc848982e0c167ef163aaff9cd91dc640ba85e263cb1ce46fae51cd" + +[[package]] +name = "serde" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9a8e94ea7f378bd32cbbd37198a4a91436180c5bb472411e48b5ec2e2124ae9e" +dependencies = [ + "serde_core", + "serde_derive", +] + +[[package]] +name = "serde_core" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "41d385c7d4ca58e59fc732af25c3983b67ac852c1a25000afe1175de458b67ad" +dependencies = [ + "serde_derive", +] + +[[package]] +name = "serde_derive" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d540f220d3187173da220f885ab66608367b6574e925011a9353e4badda91d79" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "serde_json" +version = "1.0.149" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "83fc039473c5595ace860d8c4fafa220ff474b3fc6bfdb4293327f1a37e94d86" +dependencies = [ + "itoa", + "memchr", + "serde", + "serde_core", + "zmij", +] + +[[package]] +name = "serde_path_to_error" +version = "0.1.20" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "10a9ff822e371bb5403e391ecd83e182e0e77ba7f6fe0160b795797109d1b457" +dependencies = [ + "itoa", + "serde", + "serde_core", +] + +[[package]] +name = "serde_urlencoded" +version = "0.7.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d3491c14715ca2294c4d6a88f15e84739788c1d030eed8c110436aafdaa2f3fd" +dependencies = [ + "form_urlencoded", + "itoa", + "ryu", + "serde", +] + +[[package]] +name = "sha1" +version = "0.10.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e3bf829a2d51ab4a5ddf1352d8470c140cadc8301b2ae1789db023f01cedd6ba" +dependencies = [ + "cfg-if", + "cpufeatures 0.2.17", + "digest 0.10.7", +] + +[[package]] +name = "sha2" +version = "0.10.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a7507d819769d01a365ab707794a4084392c824f54a7a6a7862f8c3d0892b283" +dependencies = [ + "cfg-if", + "cpufeatures 0.2.17", + "digest 0.10.7", +] + +[[package]] +name = "sha2" +version = "0.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "446ba717509524cb3f22f17ecc096f10f4822d76ab5c0b9822c5f9c284e825f4" +dependencies = [ + "cfg-if", + "cpufeatures 0.3.0", + "digest 0.11.3", +] + +[[package]] +name = "shlex" +version = "1.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0fda2ff0d084019ba4d7c6f371c95d8fd75ce3524c3cb8fb653a3023f6323e64" + +[[package]] +name = "signal-hook-registry" +version = "1.4.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c4db69cba1110affc0e9f7bcd48bbf87b3f4fc7c61fc9155afd4c469eb3d6c1b" +dependencies = [ + "errno", + "libc", +] + +[[package]] +name = "siphasher" +version = "1.0.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8ee5873ec9cce0195efcb7a4e9507a04cd49aec9c83d0389df45b1ef7ba2e649" + +[[package]] +name = "slab" +version = "0.4.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5" + +[[package]] +name = "smallvec" +version = "1.15.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "67b1b7a3b5fe4f1376887184045fcf45c69e92af734b7aaddc05fb777b6fbd03" + +[[package]] +name = "socket2" +version = "0.6.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3a766e1110788c36f4fa1c2b71b387a7815aa65f88ce0229841826633d93723e" +dependencies = [ + "libc", + "windows-sys 0.61.2", +] + +[[package]] +name = "stable_deref_trait" +version = "1.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6ce2be8dc25455e1f91df71bfa12ad37d7af1092ae736f3a6cd0e37bc7810596" + +[[package]] +name = "stringprep" +version = "0.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7b4df3d392d81bd458a8a621b8bffbd2302a12ffe288a9d931670948749463b1" +dependencies = [ + "unicode-bidi", + "unicode-normalization", + "unicode-properties", +] + +[[package]] +name = "subtle" +version = "2.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "13c2bddecc57b384dee18652358fb23172facb8a2c51ccc10d74c157bdea3292" + +[[package]] +name = "syn" +version = "2.0.117" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e665b8803e7b1d2a727f4023456bbbbe74da67099c585258af0ad9c5013b9b99" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "sync_wrapper" +version = "1.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0bf256ce5efdfa370213c1dabab5935a12e49f2c58d15e9eac2870d3b4f27263" +dependencies = [ + "futures-core", +] + +[[package]] +name = "synstructure" +version = "0.13.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "728a70f3dbaf5bab7f0c4b1ac8d7ae5ea60a4b5549c8a5914361c99147a709d2" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "tempfile" +version = "3.27.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32497e9a4c7b38532efcdebeef879707aa9f794296a4f0244f6f69e9bc8574bd" +dependencies = [ + "fastrand", + "getrandom 0.4.2", + "once_cell", + "rustix", + "windows-sys 0.61.2", +] + +[[package]] +name = "thiserror" +version = "1.0.69" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6aaf5339b578ea85b50e080feb250a3e8ae8cfcdff9a461c9ec2904bc923f52" +dependencies = [ + "thiserror-impl 1.0.69", +] + +[[package]] +name = "thiserror" +version = "2.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4288b5bcbc7920c07a1149a35cf9590a2aa808e0bc1eafaade0b80947865fbc4" +dependencies = [ + "thiserror-impl 2.0.18", +] + +[[package]] +name = "thiserror-impl" +version = "1.0.69" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4fee6c4efc90059e10f81e6d42c60a18f76588c3d74cb83a0b242a2b6c7504c1" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "thiserror-impl" +version = "2.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ebc4ee7f67670e9b64d05fa4253e753e016c6c95ff35b89b7941d6b856dec1d5" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "tinystr" +version = "0.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c8323304221c2a851516f22236c5722a72eaa19749016521d6dff0824447d96d" +dependencies = [ + "displaydoc", + "zerovec", +] + +[[package]] +name = "tinyvec" +version = "1.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3e61e67053d25a4e82c844e8424039d9745781b3fc4f32b8d55ed50f5f667ef3" +dependencies = [ + "tinyvec_macros", +] + +[[package]] +name = "tinyvec_macros" +version = "0.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1f3ccbac311fea05f86f61904b462b55fb3df8837a366dfc601a0161d0532f20" + +[[package]] +name = "tokio" +version = "1.52.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8fc7f01b389ac15039e4dc9531aa973a135d7a4135281b12d7c1bc79fd57fffe" +dependencies = [ + "bytes", + "libc", + "mio", + "parking_lot", + "pin-project-lite", + "signal-hook-registry", + "socket2", + "tokio-macros", + "windows-sys 0.61.2", +] + +[[package]] +name = "tokio-macros" +version = "2.7.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "385a6cb71ab9ab790c5fe8d67f1645e6c450a7ce006a33de03daa956cf70a496" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "tokio-postgres" +version = "0.7.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4dd8df5ef180f6364759a6f00f7aadda4fbbac86cdee37480826a6ff9f3574ce" +dependencies = [ + "async-trait", + "byteorder", + "bytes", + "fallible-iterator", + "futures-channel", + "futures-util", + "log", + "parking_lot", + "percent-encoding", + "phf", + "pin-project-lite", + "postgres-protocol", + "postgres-types", + "rand 0.10.1", + "socket2", + "tokio", + "tokio-util", + "whoami", +] + +[[package]] +name = "tokio-rustls" +version = "0.26.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1729aa945f29d91ba541258c8df89027d5792d85a8841fb65e8bf0f4ede4ef61" +dependencies = [ + "rustls", + "tokio", +] + +[[package]] +name = "tokio-tungstenite" +version = "0.24.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "edc5f74e248dc973e0dbb7b74c7e0d6fcc301c694ff50049504004ef4d0cdcd9" +dependencies = [ + "futures-util", + "log", + "tokio", + "tungstenite", +] + +[[package]] +name = "tokio-util" +version = "0.7.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9ae9cec805b01e8fc3fd2fe289f89149a9b66dd16786abd8b19cfa7b48cb0098" +dependencies = [ + "bytes", + "futures-core", + "futures-sink", + "pin-project-lite", + "tokio", +] + +[[package]] +name = "tower" +version = "0.5.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ebe5ef63511595f1344e2d5cfa636d973292adc0eec1f0ad45fae9f0851ab1d4" +dependencies = [ + "futures-core", + "futures-util", + "pin-project-lite", + "sync_wrapper", + "tokio", + "tower-layer", + "tower-service", + "tracing", +] + +[[package]] +name = "tower-http" +version = "0.6.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "68d6fdd9f81c2819c9a8b0e0cd91660e7746a8e6ea2ba7c6b2b057985f6bcb51" +dependencies = [ + "bitflags", + "bytes", + "futures-util", + "http", + "http-body", + "pin-project-lite", + "tower", + "tower-layer", + "tower-service", + "url", +] + +[[package]] +name = "tower-layer" +version = "0.3.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "121c2a6cda46980bb0fcd1647ffaf6cd3fc79a013de288782836f6df9c48780e" + +[[package]] +name = "tower-service" +version = "0.3.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8df9b6e13f2d32c91b9bd719c00d1958837bc7dec474d94952798cc8e69eeec3" + +[[package]] +name = "tracing" +version = "0.1.44" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "63e71662fa4b2a2c3a26f570f037eb95bb1f85397f3cd8076caed2f026a6d100" +dependencies = [ + "log", + "pin-project-lite", + "tracing-attributes", + "tracing-core", +] + +[[package]] +name = "tracing-attributes" +version = "0.1.31" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7490cfa5ec963746568740651ac6781f701c9c5ea257c58e057f3ba8cf69e8da" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "tracing-core" +version = "0.1.36" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "db97caf9d906fbde555dd62fa95ddba9eecfd14cb388e4f491a66d74cd5fb79a" +dependencies = [ + "once_cell", +] + +[[package]] +name = "try-lock" +version = "0.2.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e421abadd41a4225275504ea4d6566923418b7f05506fbc9c0fe86ba7396114b" + +[[package]] +name = "tungstenite" +version = "0.24.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "18e5b8366ee7a95b16d32197d0b2604b43a0be89dc5fac9f8e96ccafbaedda8a" +dependencies = [ + "byteorder", + "bytes", + "data-encoding", + "http", + "httparse", + "log", + "rand 0.8.6", + "sha1", + "thiserror 1.0.69", + "utf-8", +] + +[[package]] +name = "typenum" +version = "1.20.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "40ce102ab67701b8526c123c1bab5cbe42d7040ccfd0f64af1a385808d2f43de" + +[[package]] +name = "unicode-bidi" +version = "0.3.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5c1cb5db39152898a79168971543b1cb5020dff7fe43c8dc468b0885f5e29df5" + +[[package]] +name = "unicode-ident" +version = "1.0.24" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" + +[[package]] +name = "unicode-normalization" +version = "0.1.25" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5fd4f6878c9cb28d874b009da9e8d183b5abc80117c40bbd187a1fde336be6e8" +dependencies = [ + "tinyvec", +] + +[[package]] +name = "unicode-properties" +version = "0.1.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7df058c713841ad818f1dc5d3fd88063241cc61f49f5fbea4b951e8cf5a8d71d" + +[[package]] +name = "unicode-xid" +version = "0.2.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ebc1c04c71510c7f702b52b7c350734c9ff1295c464a03335b00bb84fc54f853" + +[[package]] +name = "unidesk-backend-core" +version = "0.1.0" +dependencies = [ + "anyhow", + "axum", + "base64", + "bytes", + "chrono", + "deadpool-postgres", + "futures-util", + "percent-encoding", + "rand 0.8.6", + "reqwest", + "serde", + "serde_json", + "sha2 0.10.9", + "tempfile", + "tokio", + "tokio-postgres", + "tokio-tungstenite", + "url", +] + +[[package]] +name = "untrusted" +version = "0.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8ecb6da28b8a351d773b68d5825ac39017e680750f980f3a1a85cd8dd28a47c1" + +[[package]] +name = "url" +version = "2.5.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ff67a8a4397373c3ef660812acab3268222035010ab8680ec4215f38ba3d0eed" +dependencies = [ + "form_urlencoded", + "idna", + "percent-encoding", + "serde", +] + +[[package]] +name = "utf-8" +version = "0.7.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "09cc8ee72d2a9becf2f2febe0205bbed8fc6615b7cb429ad062dc7b7ddd036a9" + +[[package]] +name = "utf8_iter" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6c140620e7ffbb22c2dee59cafe6084a59b5ffc27a8859a5f0d494b5d52b6be" + +[[package]] +name = "version_check" +version = "0.9.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a" + +[[package]] +name = "want" +version = "0.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bfa7760aed19e106de2c7c0b581b509f2f25d3dacaf737cb82ac61bc6d760b0e" +dependencies = [ + "try-lock", +] + +[[package]] +name = "wasi" +version = "0.11.1+wasi-snapshot-preview1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ccf3ec651a847eb01de73ccad15eb7d99f80485de043efb2f370cd654f4ea44b" + +[[package]] +name = "wasi" +version = "0.14.7+wasi-0.2.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "883478de20367e224c0090af9cf5f9fa85bed63a95c1abf3afc5c083ebc06e8c" +dependencies = [ + "wasip2", +] + +[[package]] +name = "wasip2" +version = "1.0.3+wasi-0.2.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "20064672db26d7cdc89c7798c48a0fdfac8213434a1186e5ef29fd560ae223d6" +dependencies = [ + "wit-bindgen 0.57.1", +] + +[[package]] +name = "wasip3" +version = "0.4.0+wasi-0.3.0-rc-2026-01-06" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5428f8bf88ea5ddc08faddef2ac4a67e390b88186c703ce6dbd955e1c145aca5" +dependencies = [ + "wit-bindgen 0.51.0", +] + +[[package]] +name = "wasite" +version = "1.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "66fe902b4a6b8028a753d5424909b764ccf79b7a209eac9bf97e59cda9f71a42" +dependencies = [ + "wasi 0.14.7+wasi-0.2.4", +] + +[[package]] +name = "wasm-bindgen" +version = "0.2.121" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "49ace1d07c165b0864824eee619580c4689389afa9dc9ed3a4c75040d82e6790" +dependencies = [ + "cfg-if", + "once_cell", + "rustversion", + "wasm-bindgen-macro", + "wasm-bindgen-shared", +] + +[[package]] +name = "wasm-bindgen-futures" +version = "0.4.71" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "96492d0d3ffba25305a7dc88720d250b1401d7edca02cc3bcd50633b424673b8" +dependencies = [ + "js-sys", + "wasm-bindgen", +] + +[[package]] +name = "wasm-bindgen-macro" +version = "0.2.121" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8e68e6f4afd367a562002c05637acb8578ff2dea1943df76afb9e83d177c8578" +dependencies = [ + "quote", + "wasm-bindgen-macro-support", +] + +[[package]] +name = "wasm-bindgen-macro-support" +version = "0.2.121" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d95a9ec35c64b2a7cb35d3fead40c4238d0940c86d107136999567a4703259f2" +dependencies = [ + "bumpalo", + "proc-macro2", + "quote", + "syn", + "wasm-bindgen-shared", +] + +[[package]] +name = "wasm-bindgen-shared" +version = "0.2.121" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c4e0100b01e9f0d03189a92b96772a1fb998639d981193d7dbab487302513441" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "wasm-encoder" +version = "0.244.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "990065f2fe63003fe337b932cfb5e3b80e0b4d0f5ff650e6985b1048f62c8319" +dependencies = [ + "leb128fmt", + "wasmparser", +] + +[[package]] +name = "wasm-metadata" +version = "0.244.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bb0e353e6a2fbdc176932bbaab493762eb1255a7900fe0fea1a2f96c296cc909" +dependencies = [ + "anyhow", + "indexmap", + "wasm-encoder", + "wasmparser", +] + +[[package]] +name = "wasm-streams" +version = "0.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "15053d8d85c7eccdbefef60f06769760a563c7f0a9d6902a13d35c7800b0ad65" +dependencies = [ + "futures-util", + "js-sys", + "wasm-bindgen", + "wasm-bindgen-futures", + "web-sys", +] + +[[package]] +name = "wasmparser" +version = "0.244.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "47b807c72e1bac69382b3a6fb3dbe8ea4c0ed87ff5629b8685ae6b9a611028fe" +dependencies = [ + "bitflags", + "hashbrown 0.15.5", + "indexmap", + "semver", +] + +[[package]] +name = "web-sys" +version = "0.3.98" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4b572dff8bcf38bad0fa19729c89bb5748b2b9b1d8be70cf90df697e3a8f32aa" +dependencies = [ + "js-sys", + "wasm-bindgen", +] + +[[package]] +name = "web-time" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5a6580f308b1fad9207618087a65c04e7a10bc77e02c8e84e9b00dd4b12fa0bb" +dependencies = [ + "js-sys", + "wasm-bindgen", +] + +[[package]] +name = "webpki-roots" +version = "1.0.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "52f5ee44c96cf55f1b349600768e3ece3a8f26010c05265ab73f945bb1a2eb9d" +dependencies = [ + "rustls-pki-types", +] + +[[package]] +name = "whoami" +version = "2.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "998767ef88740d1f5b0682a9c53c24431453923962269c2db68ee43788c5a40d" +dependencies = [ + "libc", + "libredox", + "objc2-system-configuration", + "wasite", + "web-sys", +] + +[[package]] +name = "windows-core" +version = "0.62.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b8e83a14d34d0623b51dce9581199302a221863196a1dde71a7663a4c2be9deb" +dependencies = [ + "windows-implement", + "windows-interface", + "windows-link", + "windows-result", + "windows-strings", +] + +[[package]] +name = "windows-implement" +version = "0.60.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "053e2e040ab57b9dc951b72c264860db7eb3b0200ba345b4e4c3b14f67855ddf" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "windows-interface" +version = "0.59.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3f316c4a2570ba26bbec722032c4099d8c8bc095efccdc15688708623367e358" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "windows-link" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5" + +[[package]] +name = "windows-result" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7781fa89eaf60850ac3d2da7af8e5242a5ea78d1a11c49bf2910bb5a73853eb5" +dependencies = [ + "windows-link", +] + +[[package]] +name = "windows-strings" +version = "0.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7837d08f69c77cf6b07689544538e017c1bfcf57e34b4c0ff58e6c2cd3b37091" +dependencies = [ + "windows-link", +] + +[[package]] +name = "windows-sys" +version = "0.52.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "282be5f36a8ce781fad8c8ae18fa3f9beff57ec1b52cb3de0789201425d9a33d" +dependencies = [ + "windows-targets 0.52.6", +] + +[[package]] +name = "windows-sys" +version = "0.60.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f2f500e4d28234f72040990ec9d39e3a6b950f9f22d3dba18416c35882612bcb" +dependencies = [ + "windows-targets 0.53.5", +] + +[[package]] +name = "windows-sys" +version = "0.61.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc" +dependencies = [ + "windows-link", +] + +[[package]] +name = "windows-targets" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9b724f72796e036ab90c1021d4780d4d3d648aca59e491e6b98e725b84e99973" +dependencies = [ + "windows_aarch64_gnullvm 0.52.6", + "windows_aarch64_msvc 0.52.6", + "windows_i686_gnu 0.52.6", + "windows_i686_gnullvm 0.52.6", + "windows_i686_msvc 0.52.6", + "windows_x86_64_gnu 0.52.6", + "windows_x86_64_gnullvm 0.52.6", + "windows_x86_64_msvc 0.52.6", +] + +[[package]] +name = "windows-targets" +version = "0.53.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4945f9f551b88e0d65f3db0bc25c33b8acea4d9e41163edf90dcd0b19f9069f3" +dependencies = [ + "windows-link", + "windows_aarch64_gnullvm 0.53.1", + "windows_aarch64_msvc 0.53.1", + "windows_i686_gnu 0.53.1", + "windows_i686_gnullvm 0.53.1", + "windows_i686_msvc 0.53.1", + "windows_x86_64_gnu 0.53.1", + "windows_x86_64_gnullvm 0.53.1", + "windows_x86_64_msvc 0.53.1", +] + +[[package]] +name = "windows_aarch64_gnullvm" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32a4622180e7a0ec044bb555404c800bc9fd9ec262ec147edd5989ccd0c02cd3" + +[[package]] +name = "windows_aarch64_gnullvm" +version = "0.53.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a9d8416fa8b42f5c947f8482c43e7d89e73a173cead56d044f6a56104a6d1b53" + +[[package]] +name = "windows_aarch64_msvc" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "09ec2a7bb152e2252b53fa7803150007879548bc709c039df7627cabbd05d469" + +[[package]] +name = "windows_aarch64_msvc" +version = "0.53.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b9d782e804c2f632e395708e99a94275910eb9100b2114651e04744e9b125006" + +[[package]] +name = "windows_i686_gnu" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8e9b5ad5ab802e97eb8e295ac6720e509ee4c243f69d781394014ebfe8bbfa0b" + +[[package]] +name = "windows_i686_gnu" +version = "0.53.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "960e6da069d81e09becb0ca57a65220ddff016ff2d6af6a223cf372a506593a3" + +[[package]] +name = "windows_i686_gnullvm" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0eee52d38c090b3caa76c563b86c3a4bd71ef1a819287c19d586d7334ae8ed66" + +[[package]] +name = "windows_i686_gnullvm" +version = "0.53.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fa7359d10048f68ab8b09fa71c3daccfb0e9b559aed648a8f95469c27057180c" + +[[package]] +name = "windows_i686_msvc" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "240948bc05c5e7c6dabba28bf89d89ffce3e303022809e73deaefe4f6ec56c66" + +[[package]] +name = "windows_i686_msvc" +version = "0.53.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1e7ac75179f18232fe9c285163565a57ef8d3c89254a30685b57d83a38d326c2" + +[[package]] +name = "windows_x86_64_gnu" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "147a5c80aabfbf0c7d901cb5895d1de30ef2907eb21fbbab29ca94c5b08b1a78" + +[[package]] +name = "windows_x86_64_gnu" +version = "0.53.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9c3842cdd74a865a8066ab39c8a7a473c0778a3f29370b5fd6b4b9aa7df4a499" + +[[package]] +name = "windows_x86_64_gnullvm" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "24d5b23dc417412679681396f2b49f3de8c1473deb516bd34410872eff51ed0d" + +[[package]] +name = "windows_x86_64_gnullvm" +version = "0.53.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0ffa179e2d07eee8ad8f57493436566c7cc30ac536a3379fdf008f47f6bb7ae1" + +[[package]] +name = "windows_x86_64_msvc" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "589f6da84c646204747d1270a2a5661ea66ed1cced2631d546fdfb155959f9ec" + +[[package]] +name = "windows_x86_64_msvc" +version = "0.53.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d6bbff5f0aada427a1e5a6da5f1f98158182f26556f345ac9e04d36d0ebed650" + +[[package]] +name = "wit-bindgen" +version = "0.51.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d7249219f66ced02969388cf2bb044a09756a083d0fab1e566056b04d9fbcaa5" +dependencies = [ + "wit-bindgen-rust-macro", +] + +[[package]] +name = "wit-bindgen" +version = "0.57.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1ebf944e87a7c253233ad6766e082e3cd714b5d03812acc24c318f549614536e" + +[[package]] +name = "wit-bindgen-core" +version = "0.51.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ea61de684c3ea68cb082b7a88508a8b27fcc8b797d738bfc99a82facf1d752dc" +dependencies = [ + "anyhow", + "heck", + "wit-parser", +] + +[[package]] +name = "wit-bindgen-rust" +version = "0.51.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b7c566e0f4b284dd6561c786d9cb0142da491f46a9fbed79ea69cdad5db17f21" +dependencies = [ + "anyhow", + "heck", + "indexmap", + "prettyplease", + "syn", + "wasm-metadata", + "wit-bindgen-core", + "wit-component", +] + +[[package]] +name = "wit-bindgen-rust-macro" +version = "0.51.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0c0f9bfd77e6a48eccf51359e3ae77140a7f50b1e2ebfe62422d8afdaffab17a" +dependencies = [ + "anyhow", + "prettyplease", + "proc-macro2", + "quote", + "syn", + "wit-bindgen-core", + "wit-bindgen-rust", +] + +[[package]] +name = "wit-component" +version = "0.244.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9d66ea20e9553b30172b5e831994e35fbde2d165325bec84fc43dbf6f4eb9cb2" +dependencies = [ + "anyhow", + "bitflags", + "indexmap", + "log", + "serde", + "serde_derive", + "serde_json", + "wasm-encoder", + "wasm-metadata", + "wasmparser", + "wit-parser", +] + +[[package]] +name = "wit-parser" +version = "0.244.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ecc8ac4bc1dc3381b7f59c34f00b67e18f910c2c0f50015669dde7def656a736" +dependencies = [ + "anyhow", + "id-arena", + "indexmap", + "log", + "semver", + "serde", + "serde_derive", + "serde_json", + "unicode-xid", + "wasmparser", +] + +[[package]] +name = "writeable" +version = "0.6.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1ffae5123b2d3fc086436f8834ae3ab053a283cfac8fe0a0b8eaae044768a4c4" + +[[package]] +name = "yoke" +version = "0.8.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "abe8c5fda708d9ca3df187cae8bfb9ceda00dd96231bed36e445a1a48e66f9ca" +dependencies = [ + "stable_deref_trait", + "yoke-derive", + "zerofrom", +] + +[[package]] +name = "yoke-derive" +version = "0.8.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "de844c262c8848816172cef550288e7dc6c7b7814b4ee56b3e1553f275f1858e" +dependencies = [ + "proc-macro2", + "quote", + "syn", + "synstructure", +] + +[[package]] +name = "zerocopy" +version = "0.8.48" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "eed437bf9d6692032087e337407a86f04cd8d6a16a37199ed57949d415bd68e9" +dependencies = [ + "zerocopy-derive", +] + +[[package]] +name = "zerocopy-derive" +version = "0.8.48" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "70e3cd084b1788766f53af483dd21f93881ff30d7320490ec3ef7526d203bad4" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "zerofrom" +version = "0.1.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0ec05a11813ea801ff6d75110ad09cd0824ddba17dfe17128ea0d5f68e6c5272" +dependencies = [ + "zerofrom-derive", +] + +[[package]] +name = "zerofrom-derive" +version = "0.1.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "11532158c46691caf0f2593ea8358fed6bbf68a0315e80aae9bd41fbade684a1" +dependencies = [ + "proc-macro2", + "quote", + "syn", + "synstructure", +] + +[[package]] +name = "zeroize" +version = "1.8.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b97154e67e32c85465826e8bcc1c59429aaaf107c1e4a9e53c8d8ccd5eff88d0" + +[[package]] +name = "zerotrie" +version = "0.2.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0f9152d31db0792fa83f70fb2f83148effb5c1f5b8c7686c3459e361d9bc20bf" +dependencies = [ + "displaydoc", + "yoke", + "zerofrom", +] + +[[package]] +name = "zerovec" +version = "0.11.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "90f911cbc359ab6af17377d242225f4d75119aec87ea711a880987b18cd7b239" +dependencies = [ + "yoke", + "zerofrom", + "zerovec-derive", +] + +[[package]] +name = "zerovec-derive" +version = "0.11.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "625dc425cab0dca6dc3c3319506e6593dcb08a9f387ea3b284dbd52a92c40555" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "zmij" +version = "1.0.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b8848ee67ecc8aedbaf3e4122217aff892639231befc6a1b58d29fff4c2cabaa" diff --git a/src/components/backend-core/Cargo.toml b/src/components/backend-core/Cargo.toml new file mode 100644 index 00000000..e3445eb6 --- /dev/null +++ b/src/components/backend-core/Cargo.toml @@ -0,0 +1,30 @@ +[package] +name = "unidesk-backend-core" +version = "0.1.0" +edition = "2021" + +[[bin]] +name = "backend-core" +path = "src/main.rs" + +[dependencies] +anyhow = "1.0" +axum = { version = "0.7.9", features = ["ws"] } +base64 = "0.22" +bytes = "1.7" +chrono = { version = "0.4", features = ["clock", "serde"] } +deadpool-postgres = "0.14" +futures-util = "0.3" +percent-encoding = "2.3" +rand = "0.8" +reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls", "stream"] } +serde = { version = "1.0", features = ["derive"] } +serde_json = "1.0" +sha2 = "0.10" +tokio = { version = "1.41", features = ["full"] } +tokio-postgres = { version = "0.7", features = ["with-chrono-0_4", "with-serde_json-1"] } +tokio-tungstenite = "0.24" +url = "2.5" + +[dev-dependencies] +tempfile = "3.14" diff --git a/src/components/backend-core/Dockerfile b/src/components/backend-core/Dockerfile index 21890620..dca7bfbf 100644 --- a/src/components/backend-core/Dockerfile +++ b/src/components/backend-core/Dockerfile @@ -1,8 +1,15 @@ -FROM oven/bun:1-alpine -RUN apk add --no-cache postgresql16-client tar gzip +FROM rust:1-bookworm AS build WORKDIR /app/src/components/backend-core -COPY src/components/backend-core/package.json ./package.json -RUN bun install --production -COPY src/components/shared /app/src/components/shared +COPY src/components/backend-core/Cargo.toml ./Cargo.toml +COPY src/components/backend-core/Cargo.lock ./Cargo.lock COPY src/components/backend-core/src ./src -CMD ["bun", "run", "src/index.ts"] +RUN CARGO_BUILD_JOBS=1 cargo build --release + +FROM postgres:16-bookworm +RUN apt-get update \ + && apt-get install -y --no-install-recommends ca-certificates tar gzip \ + && apt-get clean \ + && rm -rf /var/lib/apt/lists/* +COPY --from=build /app/src/components/backend-core/target/release/backend-core /usr/local/bin/backend-core +ENTRYPOINT [] +CMD ["backend-core"] diff --git a/src/components/backend-core/package.json b/src/components/backend-core/package.json index 9b835e24..191b5771 100644 --- a/src/components/backend-core/package.json +++ b/src/components/backend-core/package.json @@ -1,12 +1,8 @@ { "name": "@unidesk/backend-core", "private": true, - "type": "module", "scripts": { - "start": "bun run src/index.ts", - "check": "tsc -p tsconfig.json --noEmit" - }, - "dependencies": { - "postgres": "latest" + "start": "cargo run --manifest-path Cargo.toml", + "check": "cargo check --manifest-path Cargo.toml" } } diff --git a/src/components/backend-core/src/cli.rs b/src/components/backend-core/src/cli.rs new file mode 100644 index 00000000..ce7b0491 --- /dev/null +++ b/src/components/backend-core/src/cli.rs @@ -0,0 +1,200 @@ +use std::env; + +use anyhow::bail; +use base64::Engine; +use futures_util::{SinkExt, StreamExt}; +use serde_json::{json, Value}; +use tokio::io::{AsyncReadExt, AsyncWriteExt}; +use tokio_tungstenite::connect_async; +use tokio_tungstenite::tungstenite::Message; + +use crate::http::internal_fetch_json; + +pub async fn handle_cli() -> anyhow::Result<bool> { + let mut args = env::args().skip(1).collect::<Vec<_>>(); + if args.is_empty() { + return Ok(false); + } + match args.remove(0).as_str() { + "--fetch-json" => { + fetch_json_cli(args).await?; + Ok(true) + } + "--ssh-broker" => { + ssh_broker_cli(args).await?; + Ok(true) + } + _ => Ok(false), + } +} + +async fn fetch_json_cli(args: Vec<String>) -> anyhow::Result<()> { + let mut url = None; + let mut method = "GET".to_string(); + let mut body = None; + let mut max_response_bytes = 5_000_000_usize; + let mut require_ok = false; + let mut index = 0; + while index < args.len() { + match args[index].as_str() { + "--require-ok" => { + require_ok = true; + index += 1; + } + "--method" => { + method = args + .get(index + 1) + .cloned() + .unwrap_or_else(|| "GET".to_string()); + index += 2; + } + "--body-json" => { + let raw = args + .get(index + 1) + .cloned() + .unwrap_or_else(|| "{}".to_string()); + body = Some(serde_json::from_str::<Value>(&raw)?); + index += 2; + } + "--max-response-bytes" => { + max_response_bytes = args + .get(index + 1) + .and_then(|value| value.parse::<usize>().ok()) + .unwrap_or(max_response_bytes); + index += 2; + } + value if url.is_none() => { + url = Some(value.to_string()); + index += 1; + } + other => bail!("unexpected --fetch-json argument: {other}"), + } + } + let Some(url) = url else { + bail!("--fetch-json requires URL"); + }; + let result = internal_fetch_json(&url, &method, body, max_response_bytes).await?; + println!("{result}"); + if require_ok && !result.get("ok").and_then(Value::as_bool).unwrap_or(false) { + std::process::exit(1); + } + Ok(()) +} + +async fn ssh_broker_cli(args: Vec<String>) -> anyhow::Result<()> { + let raw_payload = args + .first() + .cloned() + .ok_or_else(|| anyhow::anyhow!("--ssh-broker requires payload JSON"))?; + let payload: Value = serde_json::from_str(&raw_payload)?; + let token = env::var("PROVIDER_TOKEN") + .or_else(|_| env::var("UNIDESK_PROVIDER_TOKEN")) + .unwrap_or_default(); + let url = format!( + "ws://127.0.0.1:8080/ws/ssh?token={}", + percent_encoding::utf8_percent_encode(&token, percent_encoding::NON_ALPHANUMERIC) + ); + let (ws, _) = connect_async(url).await?; + let (mut ws_write, mut ws_read) = ws.split(); + let provider_id = payload + .get("providerId") + .and_then(Value::as_str) + .unwrap_or(""); + let open = json!({ + "type": "ssh.open", + "providerId": provider_id, + "cwd": payload.get("cwd").cloned().unwrap_or(Value::Null), + "command": payload.get("command").cloned().unwrap_or(Value::Null), + "tty": payload.get("tty").cloned().unwrap_or(Value::Null), + "cols": payload.get("cols").cloned().unwrap_or(json!(100)), + "rows": payload.get("rows").cloned().unwrap_or(json!(30)), + }); + ws_write.send(Message::Text(open.to_string())).await?; + + let stdin_eot_on_end = payload + .get("stdinEotOnEnd") + .and_then(Value::as_bool) + .unwrap_or(false); + let write_task = tokio::spawn(async move { + let mut stdin = tokio::io::stdin(); + let mut buffer = vec![0_u8; 8192]; + loop { + match stdin.read(&mut buffer).await { + Ok(0) => break, + Ok(size) => { + let data = base64::engine::general_purpose::STANDARD.encode(&buffer[..size]); + if ws_write + .send(Message::Text( + json!({ "type": "ssh.input", "data": data, "encoding": "base64" }) + .to_string(), + )) + .await + .is_err() + { + return; + } + } + Err(_) => break, + } + } + if stdin_eot_on_end { + let data = base64::engine::general_purpose::STANDARD.encode([4_u8]); + let _ = ws_write + .send(Message::Text( + json!({ "type": "ssh.input", "data": data, "encoding": "base64" }).to_string(), + )) + .await; + } + let _ = ws_write + .send(Message::Text(json!({ "type": "ssh.eof" }).to_string())) + .await; + }); + + let mut exit_code = 255_i32; + while let Some(message) = ws_read.next().await { + let message = message?; + let text = match message { + Message::Text(text) => text, + Message::Close(_) => break, + _ => continue, + }; + let parsed: Value = serde_json::from_str(&text).unwrap_or_else(|_| json!({})); + match parsed.get("type").and_then(Value::as_str).unwrap_or("") { + "ssh.data" => { + let data = parsed.get("data").and_then(Value::as_str).unwrap_or(""); + let bytes = base64::engine::general_purpose::STANDARD + .decode(data) + .unwrap_or_default(); + if parsed.get("stream").and_then(Value::as_str) == Some("stderr") { + let _ = tokio::io::stderr().write_all(&bytes).await; + } else { + let _ = tokio::io::stdout().write_all(&bytes).await; + } + } + "ssh.exit" => { + exit_code = parsed + .get("exitCode") + .and_then(Value::as_i64) + .unwrap_or(255) as i32; + } + "ssh.error" => { + let _ = tokio::io::stderr() + .write_all( + format!( + "{}\n", + parsed + .get("message") + .and_then(Value::as_str) + .unwrap_or("ssh error") + ) + .as_bytes(), + ) + .await; + exit_code = 255; + } + _ => {} + } + } + write_task.abort(); + std::process::exit(exit_code); +} diff --git a/src/components/backend-core/src/config.rs b/src/components/backend-core/src/config.rs new file mode 100644 index 00000000..7ee0818a --- /dev/null +++ b/src/components/backend-core/src/config.rs @@ -0,0 +1,152 @@ +use std::env; + +use anyhow::{anyhow, bail, Context}; + +use crate::types::{MicroserviceConfig, RuntimeConfig, RuntimeIdentity}; + +fn required_env(name: &str) -> anyhow::Result<String> { + let value = + env::var(name).with_context(|| format!("Missing required environment variable: {name}"))?; + if value.is_empty() { + bail!("Missing required environment variable: {name}"); + } + Ok(value) +} + +fn number_env<T>(name: &str) -> anyhow::Result<T> +where + T: std::str::FromStr + PartialOrd + From<u8>, + <T as std::str::FromStr>::Err: std::fmt::Display, +{ + let raw = required_env(name)?; + let parsed: T = raw.parse().map_err(|error| { + anyhow!("Environment variable {name} must be numeric, got {raw}: {error}") + })?; + if parsed <= T::from(0) { + bail!("Environment variable {name} must be positive, got {raw}"); + } + Ok(parsed) +} + +fn optional_number_env<T>(name: &str, fallback: T) -> anyhow::Result<T> +where + T: std::str::FromStr + PartialOrd + From<u8>, + <T as std::str::FromStr>::Err: std::fmt::Display, +{ + match env::var(name) { + Ok(raw) if !raw.is_empty() => { + let parsed: T = raw.parse().map_err(|error| { + anyhow!("Environment variable {name} must be numeric, got {raw}: {error}") + })?; + if parsed <= T::from(0) { + bail!("Environment variable {name} must be positive, got {raw}"); + } + Ok(parsed) + } + _ => Ok(fallback), + } +} + +fn read_microservices_env() -> anyhow::Result<Vec<MicroserviceConfig>> { + let raw = env::var("MICROSERVICES_JSON").unwrap_or_default(); + if raw.is_empty() { + return Ok(Vec::new()); + } + let mut services: Vec<MicroserviceConfig> = serde_json::from_str(&raw) + .context("MICROSERVICES_JSON must be a valid microservice array")?; + for (index, service) in services.iter_mut().enumerate() { + if service.id.is_empty() || service.name.is_empty() || service.provider_id.is_empty() { + bail!("MICROSERVICES_JSON[{index}] id/name/providerId must be non-empty"); + } + if service.deployment.mode.is_empty() { + service.deployment.mode = "unidesk-direct".to_string(); + } + if service.deployment.mode != "unidesk-direct" + && service.deployment.mode != "k3sctl-managed" + { + bail!("MICROSERVICES_JSON[{index}].deployment.mode must be unidesk-direct or k3sctl-managed"); + } + service.backend.allowed_methods = service + .backend + .allowed_methods + .iter() + .map(|method| method.to_uppercase()) + .collect(); + } + Ok(services) +} + +fn optional_env(name: &str) -> String { + env::var(name) + .unwrap_or_default() + .trim() + .to_string() +} + +fn database_name_from_url(database_url: &str) -> String { + url::Url::parse(database_url) + .ok() + .map(|url| url.path().trim_start_matches('/').to_string()) + .unwrap_or_default() +} + +fn runtime_identity(database_url: &str) -> RuntimeIdentity { + RuntimeIdentity { + environment: { + let value = optional_env("UNIDESK_ENV"); + if value.is_empty() { + "prod".to_string() + } else { + value + } + }, + namespace: optional_env("UNIDESK_NAMESPACE"), + database_name: { + let explicit = optional_env("UNIDESK_DATABASE_NAME"); + if !explicit.is_empty() { + explicit + } else { + let legacy = optional_env("UNIDESK_DEV_DATABASE_NAME"); + if !legacy.is_empty() { + legacy + } else { + database_name_from_url(database_url) + } + } + }, + deploy_ref: optional_env("UNIDESK_DEPLOY_REF"), + service_id: { + let value = optional_env("UNIDESK_DEPLOY_SERVICE_ID"); + if value.is_empty() { + "backend-core".to_string() + } else { + value + } + }, + repo: optional_env("UNIDESK_DEPLOY_REPO"), + commit: optional_env("UNIDESK_DEPLOY_COMMIT"), + requested_commit: optional_env("UNIDESK_DEPLOY_REQUESTED_COMMIT"), + } +} + +pub fn read_config() -> anyhow::Result<RuntimeConfig> { + let database_url = required_env("DATABASE_URL")?; + Ok(RuntimeConfig { + port: number_env("PORT")?, + provider_port: number_env("PROVIDER_PORT")?, + database_url: database_url.clone(), + identity: runtime_identity(&database_url), + provider_token: required_env("PROVIDER_TOKEN")?, + heartbeat_timeout_ms: number_env("HEARTBEAT_TIMEOUT_MS")?, + task_pending_timeout_ms: optional_number_env("TASK_PENDING_TIMEOUT_MS", 10 * 60 * 1000)?, + database_volume_name: required_env("DATABASE_VOLUME_NAME")?, + database_volume_size: required_env("DATABASE_VOLUME_SIZE")?, + pgdata_backup_staging_dir: env::var("PGDATA_BACKUP_STAGING_DIR") + .unwrap_or_else(|_| "/data/baidu-netdisk-staging".to_string()), + baidu_netdisk_internal_url: env::var("BAIDU_NETDISK_INTERNAL_URL") + .unwrap_or_else(|_| "http://baidu-netdisk:4244".to_string()), + microservices: read_microservices_env()?, + log_file: required_env("LOG_FILE")?, + database_pool_max: optional_number_env::<usize>("DATABASE_POOL_MAX", 4)?.clamp(1, 16), + }) +} diff --git a/src/components/backend-core/src/db.rs b/src/components/backend-core/src/db.rs new file mode 100644 index 00000000..c7c05e74 --- /dev/null +++ b/src/components/backend-core/src/db.rs @@ -0,0 +1,615 @@ +use std::sync::Arc; + +use anyhow::Context; +use chrono::{DateTime, Utc}; +use serde_json::{json, Value}; +use tokio_postgres::Row; + +use crate::json_util::{ + compact_json, compact_json_for_storage, nested_number, redact_database_url, +}; +use crate::state::AppState; +use crate::types::{JsonValue, RawTaskRow}; + +pub async fn init_database(state: &Arc<AppState>) -> anyhow::Result<()> { + state.log( + "info", + "database_init_start", + Some(json!({ "databaseUrl": redact_database_url(&state.config.database_url) })), + ); + let client = state.pool.get().await?; + client.batch_execute( + r#" + CREATE TABLE IF NOT EXISTS unidesk_nodes ( + provider_id TEXT PRIMARY KEY, + name TEXT NOT NULL, + labels JSONB NOT NULL DEFAULT '{}'::jsonb, + status TEXT NOT NULL, + connected_at TIMESTAMPTZ, + last_heartbeat TIMESTAMPTZ, + updated_at TIMESTAMPTZ NOT NULL DEFAULT now() + ); + CREATE TABLE IF NOT EXISTS unidesk_events ( + id BIGSERIAL PRIMARY KEY, + type TEXT NOT NULL, + source TEXT NOT NULL, + payload JSONB NOT NULL DEFAULT '{}'::jsonb, + created_at TIMESTAMPTZ NOT NULL DEFAULT now() + ); + CREATE TABLE IF NOT EXISTS unidesk_tasks ( + id TEXT PRIMARY KEY, + provider_id TEXT NOT NULL, + command TEXT NOT NULL, + status TEXT NOT NULL, + payload JSONB NOT NULL DEFAULT '{}'::jsonb, + result JSONB, + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + updated_at TIMESTAMPTZ NOT NULL DEFAULT now() + ); + CREATE TABLE IF NOT EXISTS unidesk_scheduled_tasks ( + id TEXT PRIMARY KEY, + name TEXT NOT NULL, + description TEXT NOT NULL DEFAULT '', + enabled BOOLEAN NOT NULL DEFAULT true, + schedule_json JSONB NOT NULL DEFAULT '{}'::jsonb, + action_json JSONB NOT NULL DEFAULT '{}'::jsonb, + concurrency_policy TEXT NOT NULL DEFAULT 'skip', + next_run_at TIMESTAMPTZ, + last_run_at TIMESTAMPTZ, + last_run_id TEXT, + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + updated_at TIMESTAMPTZ NOT NULL DEFAULT now() + ); + CREATE TABLE IF NOT EXISTS unidesk_scheduled_task_runs ( + id TEXT PRIMARY KEY, + schedule_id TEXT NOT NULL REFERENCES unidesk_scheduled_tasks(id) ON DELETE CASCADE, + trigger_type TEXT NOT NULL, + status TEXT NOT NULL, + task_id TEXT, + result JSONB, + error TEXT, + started_at TIMESTAMPTZ, + finished_at TIMESTAMPTZ, + duration_ms BIGINT, + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + updated_at TIMESTAMPTZ NOT NULL DEFAULT now() + ); + CREATE TABLE IF NOT EXISTS unidesk_node_docker_status ( + provider_id TEXT PRIMARY KEY, + status JSONB NOT NULL DEFAULT '{}'::jsonb, + collected_at TIMESTAMPTZ, + updated_at TIMESTAMPTZ NOT NULL DEFAULT now() + ); + CREATE TABLE IF NOT EXISTS unidesk_node_system_status ( + provider_id TEXT PRIMARY KEY, + status JSONB NOT NULL DEFAULT '{}'::jsonb, + collected_at TIMESTAMPTZ, + updated_at TIMESTAMPTZ NOT NULL DEFAULT now() + ); + CREATE TABLE IF NOT EXISTS unidesk_node_metric_samples ( + id BIGSERIAL PRIMARY KEY, + provider_id TEXT NOT NULL, + collected_at TIMESTAMPTZ NOT NULL, + cpu_percent DOUBLE PRECISION NOT NULL DEFAULT 0, + memory_percent DOUBLE PRECISION NOT NULL DEFAULT 0, + disk_percent DOUBLE PRECISION NOT NULL DEFAULT 0, + sample JSONB NOT NULL DEFAULT '{}'::jsonb, + created_at TIMESTAMPTZ NOT NULL DEFAULT now() + ); + CREATE INDEX IF NOT EXISTS idx_unidesk_tasks_updated_at ON unidesk_tasks(updated_at DESC); + CREATE INDEX IF NOT EXISTS idx_unidesk_tasks_status_updated_at ON unidesk_tasks(status, updated_at DESC); + CREATE INDEX IF NOT EXISTS idx_unidesk_scheduled_tasks_next_run ON unidesk_scheduled_tasks(enabled, next_run_at); + CREATE INDEX IF NOT EXISTS idx_unidesk_scheduled_task_runs_schedule_updated ON unidesk_scheduled_task_runs(schedule_id, updated_at DESC); + CREATE INDEX IF NOT EXISTS idx_unidesk_scheduled_task_runs_status_updated ON unidesk_scheduled_task_runs(status, updated_at DESC); + CREATE INDEX IF NOT EXISTS idx_unidesk_node_system_status_updated_at ON unidesk_node_system_status(updated_at DESC); + CREATE INDEX IF NOT EXISTS idx_unidesk_node_metric_samples_provider_time ON unidesk_node_metric_samples(provider_id, collected_at DESC); + "#, + ) + .await + .context("create backend-core tables")?; + state.log("info", "database_init_complete", None); + Ok(()) +} + +pub async fn record_event( + state: &Arc<AppState>, + event_type: &str, + source: &str, + payload: JsonValue, +) { + state.log( + "info", + event_type, + Some(json!({ "source": source, "payload": payload })), + ); + if !state.db_ready() { + return; + } + let stored_payload = compact_json_for_storage(&payload); + let result = async { + let client = state.pool.get().await?; + client + .execute( + "INSERT INTO unidesk_events (type, source, payload) VALUES ($1, $2, $3)", + &[&event_type, &source, &stored_payload], + ) + .await?; + anyhow::Ok(()) + } + .await; + if let Err(error) = result { + state.log( + "error", + "event_insert_failed", + Some(json!({ "type": event_type, "source": source, "error": error.to_string() })), + ); + } +} + +pub async fn upsert_provider_node( + state: &Arc<AppState>, + provider_id: &str, + name: &str, + labels: &JsonValue, +) -> anyhow::Result<()> { + let client = state.pool.get().await?; + client.execute( + r#" + INSERT INTO unidesk_nodes (provider_id, name, labels, status, connected_at, last_heartbeat, updated_at) + VALUES ($1, $2, $3, 'online', now(), now(), now()) + ON CONFLICT (provider_id) DO UPDATE SET + name = EXCLUDED.name, + labels = EXCLUDED.labels, + status = 'online', + connected_at = COALESCE(unidesk_nodes.connected_at, EXCLUDED.connected_at), + last_heartbeat = now(), + updated_at = now() + "#, + &[&provider_id, &name, labels], + ).await?; + Ok(()) +} + +pub async fn update_provider_heartbeat( + state: &Arc<AppState>, + provider_id: &str, + labels: &JsonValue, +) -> anyhow::Result<()> { + let client = state.pool.get().await?; + client + .execute( + r#" + UPDATE unidesk_nodes + SET labels = unidesk_nodes.labels || $2::jsonb, + status = 'online', + last_heartbeat = now(), + updated_at = now() + WHERE provider_id = $1 + "#, + &[&provider_id, labels], + ) + .await?; + Ok(()) +} + +pub async fn upsert_docker_status( + state: &Arc<AppState>, + provider_id: &str, + status: &JsonValue, + collected_at: &str, +) -> anyhow::Result<()> { + let client = state.pool.get().await?; + client + .execute( + r#" + INSERT INTO unidesk_node_docker_status (provider_id, status, collected_at, updated_at) + VALUES ($1, $2, $3::timestamptz, now()) + ON CONFLICT (provider_id) DO UPDATE SET + status = EXCLUDED.status, + collected_at = EXCLUDED.collected_at, + updated_at = now() + "#, + &[&provider_id, status, &collected_at], + ) + .await?; + Ok(()) +} + +pub async fn upsert_system_status( + state: &Arc<AppState>, + provider_id: &str, + status: &JsonValue, + collected_at: &str, +) -> anyhow::Result<()> { + let cpu_percent = nested_number(status, "cpu", "percent"); + let memory_percent = nested_number(status, "memory", "percent"); + let disk_percent = nested_number(status, "disk", "percent"); + let mut client = state.pool.get().await?; + let tx = client.transaction().await?; + tx.execute( + r#" + INSERT INTO unidesk_node_system_status (provider_id, status, collected_at, updated_at) + VALUES ($1, $2, $3::timestamptz, now()) + ON CONFLICT (provider_id) DO UPDATE SET + status = EXCLUDED.status, + collected_at = EXCLUDED.collected_at, + updated_at = now() + "#, + &[&provider_id, status, &collected_at], + ) + .await?; + tx.execute( + r#" + INSERT INTO unidesk_node_metric_samples (provider_id, collected_at, cpu_percent, memory_percent, disk_percent, sample) + VALUES ($1, $2::timestamptz, $3, $4, $5, $6) + "#, + &[&provider_id, &collected_at, &cpu_percent, &memory_percent, &disk_percent, status], + ).await?; + tx.execute( + r#" + DELETE FROM unidesk_node_metric_samples + WHERE provider_id = $1 + AND id NOT IN ( + SELECT id FROM unidesk_node_metric_samples + WHERE provider_id = $1 + ORDER BY collected_at DESC + LIMIT 720 + ) + "#, + &[&provider_id], + ) + .await?; + tx.commit().await?; + Ok(()) +} + +pub async fn get_nodes(state: &Arc<AppState>) -> anyhow::Result<Vec<JsonValue>> { + let client = state.pool.get().await?; + let rows = client.query( + "SELECT provider_id, name, labels, status, connected_at, last_heartbeat FROM unidesk_nodes ORDER BY status DESC, provider_id ASC", + &[], + ).await?; + Ok(rows.into_iter().map(|row| { + let connected_at: Option<DateTime<Utc>> = row.get("connected_at"); + let last_heartbeat: Option<DateTime<Utc>> = row.get("last_heartbeat"); + json!({ + "providerId": row.get::<_, String>("provider_id"), + "name": row.get::<_, String>("name"), + "status": if row.get::<_, String>("status") == "online" { "online" } else { "offline" }, + "labels": row.get::<_, JsonValue>("labels"), + "connectedAt": connected_at.map(|value| value.to_rfc3339_opts(chrono::SecondsFormat::Millis, true)), + "lastHeartbeat": last_heartbeat.map(|value| value.to_rfc3339_opts(chrono::SecondsFormat::Millis, true)), + }) + }).collect()) +} + +pub async fn get_node_docker_statuses(state: &Arc<AppState>) -> anyhow::Result<Vec<JsonValue>> { + let client = state.pool.get().await?; + let rows = client.query( + r#" + SELECT n.provider_id, n.name, n.status AS node_status, d.status AS docker_status, d.updated_at + FROM unidesk_nodes n + LEFT JOIN unidesk_node_docker_status d ON d.provider_id = n.provider_id + ORDER BY n.status DESC, n.provider_id ASC + "#, + &[], + ).await?; + Ok(rows.into_iter().map(|row| { + let updated_at: Option<DateTime<Utc>> = row.get("updated_at"); + let docker_status: Option<JsonValue> = row.get("docker_status"); + json!({ + "providerId": row.get::<_, String>("provider_id"), + "name": row.get::<_, String>("name"), + "nodeStatus": if row.get::<_, String>("node_status") == "online" { "online" } else { "offline" }, + "dockerStatus": docker_status, + "updatedAt": updated_at.map(|value| value.to_rfc3339_opts(chrono::SecondsFormat::Millis, true)), + }) + }).collect()) +} + +fn metric_point_from_sample(sample: &JsonValue, collected_at: &str) -> JsonValue { + json!({ + "at": collected_at, + "cpuPercent": nested_number(sample, "cpu", "percent"), + "memoryPercent": nested_number(sample, "memory", "percent"), + "diskPercent": nested_number(sample, "disk", "percent"), + "memoryUsedBytes": nested_number(sample, "memory", "usedBytes"), + "memoryTotalBytes": nested_number(sample, "memory", "totalBytes"), + "diskUsedBytes": nested_number(sample, "disk", "usedBytes"), + "diskTotalBytes": nested_number(sample, "disk", "totalBytes"), + "load1": nested_number(sample, "cpu", "load1"), + }) +} + +pub async fn get_node_system_statuses( + state: &Arc<AppState>, + limit: i64, +) -> anyhow::Result<Vec<JsonValue>> { + let client = state.pool.get().await?; + let current_rows = client.query( + r#" + SELECT n.provider_id, n.name, n.status AS node_status, s.status AS system_status, s.updated_at + FROM unidesk_nodes n + LEFT JOIN unidesk_node_system_status s ON s.provider_id = n.provider_id + ORDER BY n.status DESC, n.provider_id ASC + "#, + &[], + ).await?; + let sample_rows = client + .query( + r#" + SELECT n.provider_id, recent.collected_at, recent.sample + FROM unidesk_nodes n + LEFT JOIN LATERAL ( + SELECT collected_at, sample + FROM unidesk_node_metric_samples m + WHERE m.provider_id = n.provider_id + ORDER BY collected_at DESC + LIMIT $1 + ) recent ON true + WHERE recent.collected_at IS NOT NULL + ORDER BY n.provider_id ASC, recent.collected_at ASC + "#, + &[&limit], + ) + .await?; + let mut history_by_provider = std::collections::HashMap::<String, Vec<JsonValue>>::new(); + for row in sample_rows { + let provider_id: String = row.get("provider_id"); + let collected_at: DateTime<Utc> = row.get("collected_at"); + let sample: JsonValue = row.get("sample"); + history_by_provider + .entry(provider_id) + .or_default() + .push(metric_point_from_sample( + &sample, + &collected_at.to_rfc3339_opts(chrono::SecondsFormat::Millis, true), + )); + } + Ok(current_rows.into_iter().map(|row| { + let provider_id: String = row.get("provider_id"); + let updated_at: Option<DateTime<Utc>> = row.get("updated_at"); + let system_status: Option<JsonValue> = row.get("system_status"); + json!({ + "providerId": provider_id, + "name": row.get::<_, String>("name"), + "nodeStatus": if row.get::<_, String>("node_status") == "online" { "online" } else { "offline" }, + "current": system_status, + "history": history_by_provider.remove(&provider_id).unwrap_or_default(), + "updatedAt": updated_at.map(|value| value.to_rfc3339_opts(chrono::SecondsFormat::Millis, true)), + }) + }).collect()) +} + +pub async fn get_events(state: &Arc<AppState>, limit: i64) -> anyhow::Result<Vec<JsonValue>> { + let client = state.pool.get().await?; + let rows = client.query( + "SELECT id, type, source, payload, created_at FROM unidesk_events ORDER BY id DESC LIMIT $1", + &[&limit], + ).await?; + Ok(rows + .into_iter() + .map(|row| { + let created_at: DateTime<Utc> = row.get("created_at"); + let payload: JsonValue = row.get("payload"); + json!({ + "id": row.get::<_, i64>("id"), + "type": row.get::<_, String>("type"), + "source": row.get::<_, String>("source"), + "payload": compact_json(&payload), + "createdAt": created_at.to_rfc3339_opts(chrono::SecondsFormat::Millis, true), + }) + }) + .collect()) +} + +fn task_json_summary(value: Option<&JsonValue>, prefix: &str) -> JsonValue { + let Some(value) = value else { + return if prefix == "result" { + Value::Null + } else { + json!({}) + }; + }; + let Some(object) = value.as_object() else { + return json!({ "summaryOnly": true, "type": type_name(value) }); + }; + let mut result = serde_json::Map::new(); + result.insert("summaryOnly".to_string(), Value::Bool(true)); + result.insert("type".to_string(), Value::String("object".to_string())); + let fields: &[&str] = if prefix == "payload" { + &[ + "source", + "serviceId", + "method", + "path", + "mode", + "targetBaseUrl", + "timeoutMs", + "targetProviderGatewayVersion", + "providerGatewayVersion", + ] + } else { + &[ + "error", + "reason", + "message", + "status", + "exitCode", + "code", + "signal", + "timeoutMs", + "previousStatus", + "mode", + "policy", + "targetProviderGatewayVersion", + "providerGatewayVersion", + "updaterContainerId", + ] + }; + for field in fields { + if let Some(field_value) = object.get(*field) { + if !(field_value.is_null() || field_value.as_str().is_some_and(|text| text.is_empty())) + { + result.insert((*field).to_string(), field_value.clone()); + } + } + } + if let Some(body_text) = object.get("bodyText").and_then(Value::as_str) { + result.insert( + "bodyText".to_string(), + Value::String(format!("<omitted:{} chars>", body_text.len())), + ); + } + Value::Object(result) +} + +fn type_name(value: &JsonValue) -> &'static str { + match value { + Value::Null => "null", + Value::Bool(_) => "boolean", + Value::Number(_) => "number", + Value::String(_) => "string", + Value::Array(_) => "array", + Value::Object(_) => "object", + } +} + +fn omit_body_text(value: JsonValue) -> JsonValue { + match value { + Value::Object(mut object) => { + if let Some(Value::String(text)) = object.get("bodyText") { + object.insert( + "bodyText".to_string(), + Value::String(format!("<omitted:{} chars>", text.len())), + ); + } + Value::Object(object) + } + other => other, + } +} + +pub async fn get_task(state: &Arc<AppState>, task_id: &str) -> anyhow::Result<Option<JsonValue>> { + let client = state.pool.get().await?; + let row = client.query_opt( + "SELECT id, provider_id, command, status, payload, result, created_at, updated_at FROM unidesk_tasks WHERE id = $1 LIMIT 1", + &[&task_id], + ).await?; + Ok(row.map(task_full_from_row)) +} + +pub async fn get_tasks( + state: &Arc<AppState>, + limit: i64, + status_filter: &str, + lite: bool, + summary: bool, +) -> anyhow::Result<Vec<JsonValue>> { + crate::task_dispatcher::maybe_mark_stale_tasks_failed(state, 15_000).await?; + let client = state.pool.get().await?; + let rows = if status_filter == "pending" { + client.query( + "SELECT id, provider_id, command, status, payload, result, created_at, updated_at FROM unidesk_tasks WHERE status IN ('queued','dispatched','running') ORDER BY updated_at DESC LIMIT $1", + &[&limit], + ).await? + } else { + client.query( + "SELECT id, provider_id, command, status, payload, result, created_at, updated_at FROM unidesk_tasks ORDER BY updated_at DESC LIMIT $1", + &[&limit], + ).await? + }; + Ok(rows.into_iter().map(|row| { + if summary && !lite { + task_summary_from_row(&row) + } else { + let created_at: DateTime<Utc> = row.get("created_at"); + let updated_at: DateTime<Utc> = row.get("updated_at"); + json!({ + "id": row.get::<_, String>("id"), + "providerId": row.get::<_, String>("provider_id"), + "command": row.get::<_, String>("command"), + "status": row.get::<_, String>("status"), + "payload": if lite { json!({}) } else { compact_json(&omit_body_text(row.get::<_, JsonValue>("payload"))) }, + "result": if lite { Value::Null } else { row.get::<_, Option<JsonValue>>("result").map(|value| compact_json(&omit_body_text(value))).unwrap_or(Value::Null) }, + "createdAt": created_at.to_rfc3339_opts(chrono::SecondsFormat::Millis, true), + "updatedAt": updated_at.to_rfc3339_opts(chrono::SecondsFormat::Millis, true), + }) + } + }).collect()) +} + +fn task_full_from_row(row: Row) -> JsonValue { + let created_at: DateTime<Utc> = row.get("created_at"); + let updated_at: DateTime<Utc> = row.get("updated_at"); + json!({ + "id": row.get::<_, String>("id"), + "providerId": row.get::<_, String>("provider_id"), + "command": row.get::<_, String>("command"), + "status": row.get::<_, String>("status"), + "payload": compact_json(&omit_body_text(row.get::<_, JsonValue>("payload"))), + "result": row.get::<_, Option<JsonValue>>("result").map(|value| compact_json(&omit_body_text(value))).unwrap_or(Value::Null), + "createdAt": created_at.to_rfc3339_opts(chrono::SecondsFormat::Millis, true), + "updatedAt": updated_at.to_rfc3339_opts(chrono::SecondsFormat::Millis, true), + }) +} + +fn task_summary_from_row(row: &Row) -> JsonValue { + let created_at: DateTime<Utc> = row.get("created_at"); + let updated_at: DateTime<Utc> = row.get("updated_at"); + let payload: JsonValue = row.get("payload"); + let result: Option<JsonValue> = row.get("result"); + json!({ + "id": row.get::<_, String>("id"), + "providerId": row.get::<_, String>("provider_id"), + "command": row.get::<_, String>("command"), + "status": row.get::<_, String>("status"), + "payload": task_json_summary(Some(&payload), "payload"), + "result": task_json_summary(result.as_ref(), "result"), + "createdAt": created_at.to_rfc3339_opts(chrono::SecondsFormat::Millis, true), + "updatedAt": updated_at.to_rfc3339_opts(chrono::SecondsFormat::Millis, true), + "_summaryOnly": true, + }) +} + +pub async fn raw_task(state: &Arc<AppState>, task_id: &str) -> anyhow::Result<Option<RawTaskRow>> { + let client = state.pool.get().await?; + let row = client.query_opt( + "SELECT id, provider_id, command, status, payload, result, updated_at FROM unidesk_tasks WHERE id = $1 LIMIT 1", + &[&task_id], + ).await?; + Ok(row.map(|row| RawTaskRow { + id: row.get("id"), + provider_id: row.get("provider_id"), + command: row.get("command"), + status: row.get("status"), + payload: row.get("payload"), + result: row.get("result"), + updated_at: row.get("updated_at"), + })) +} + +pub async fn provider_supports( + state: &Arc<AppState>, + provider_id: &str, + capability: &str, +) -> anyhow::Result<bool> { + if !state.db_ready() { + return Ok(false); + } + let client = state.pool.get().await?; + let row = client + .query_opt( + "SELECT labels FROM unidesk_nodes WHERE provider_id = $1 LIMIT 1", + &[&provider_id], + ) + .await?; + let Some(row) = row else { + return Ok(false); + }; + let labels: JsonValue = row.get("labels"); + Ok(labels + .get("unideskCapabilities") + .and_then(Value::as_array) + .is_some_and(|items| items.iter().any(|item| item.as_str() == Some(capability)))) +} diff --git a/src/components/backend-core/src/egress_tcp.rs b/src/components/backend-core/src/egress_tcp.rs new file mode 100644 index 00000000..e23a49c4 --- /dev/null +++ b/src/components/backend-core/src/egress_tcp.rs @@ -0,0 +1,219 @@ +use std::sync::Arc; +use std::time::Duration; + +use anyhow::bail; +use axum::extract::ws::Message; +use base64::Engine; +use serde_json::{json, Value}; +use tokio::io::{AsyncReadExt, AsyncWriteExt}; +use tokio::net::TcpStream; +use tokio::sync::mpsc; + +use crate::state::{AppState, ProviderConnection}; + +pub struct EgressTcpConnection { + pub provider_id: String, + pub connection_id: String, + pub writer: mpsc::UnboundedSender<Vec<u8>>, +} + +fn egress_tcp_key(provider_id: &str, connection_id: &str) -> String { + format!("{provider_id}:{connection_id}") +} + +fn is_valid_egress_host(host: &str) -> bool { + !host.is_empty() + && host.len() <= 253 + && !host + .chars() + .any(|ch| ch.is_whitespace() || matches!(ch, '/' | '\\' | ':' | '@' | '\0')) +} + +fn is_valid_egress_port(port: i64) -> bool { + port > 0 && port <= 65_535 +} + +fn send_egress_close(provider: &Arc<ProviderConnection>, connection_id: &str, error: Option<&str>) { + let message = match error { + Some(error) => { + json!({ "type": "egress_tcp_close", "connectionId": connection_id, "error": error }) + } + None => json!({ "type": "egress_tcp_close", "connectionId": connection_id }), + }; + let _ = provider.sender.send(Message::Text(message.to_string())); +} + +pub async fn handle_egress_tcp_open( + state: &Arc<AppState>, + provider: Arc<ProviderConnection>, + message: &Value, +) -> anyhow::Result<()> { + let provider_id = message + .get("providerId") + .and_then(Value::as_str) + .unwrap_or("") + .to_string(); + let connection_id = message + .get("connectionId") + .and_then(Value::as_str) + .unwrap_or("") + .to_string(); + let host = message + .get("host") + .and_then(Value::as_str) + .unwrap_or("") + .trim() + .to_string(); + let port = message.get("port").and_then(Value::as_i64).unwrap_or(0); + if !is_valid_egress_host(&host) + || !is_valid_egress_port(port) + || provider_id.is_empty() + || connection_id.is_empty() + { + send_egress_close(&provider, &connection_id, Some("invalid egress target")); + return Ok(()); + } + close_egress_tcp_connection(state, &provider_id, &connection_id, None).await; + let stream = match tokio::time::timeout( + Duration::from_secs(15), + TcpStream::connect((host.as_str(), port as u16)), + ) + .await + { + Ok(Ok(stream)) => stream, + Ok(Err(error)) => { + send_egress_close(&provider, &connection_id, Some(&error.to_string())); + return Ok(()); + } + Err(_) => { + send_egress_close( + &provider, + &connection_id, + Some("egress tcp connect timeout"), + ); + return Ok(()); + } + }; + let (mut reader, mut writer) = stream.into_split(); + let (tx, mut rx) = mpsc::unbounded_channel::<Vec<u8>>(); + let key = egress_tcp_key(&provider_id, &connection_id); + state.active_egress_tcp.lock().await.insert( + key.clone(), + EgressTcpConnection { + provider_id: provider_id.clone(), + connection_id: connection_id.clone(), + writer: tx, + }, + ); + let _ = provider.sender.send(Message::Text( + json!({ "type": "egress_tcp_opened", "connectionId": connection_id }).to_string(), + )); + + tokio::spawn({ + let state = state.clone(); + let provider = provider.clone(); + let provider_id = provider_id.clone(); + let connection_id = connection_id.clone(); + async move { + let mut buffer = vec![0_u8; 16 * 1024]; + loop { + match reader.read(&mut buffer).await { + Ok(0) => break, + Ok(size) => { + let data = + base64::engine::general_purpose::STANDARD.encode(&buffer[..size]); + let _ = provider.sender.send(Message::Text(json!({ "type": "egress_tcp_data", "connectionId": connection_id, "data": data, "encoding": "base64" }).to_string())); + } + Err(error) => { + send_egress_close(&provider, &connection_id, Some(&error.to_string())); + break; + } + } + } + state + .active_egress_tcp + .lock() + .await + .remove(&egress_tcp_key(&provider_id, &connection_id)); + send_egress_close(&provider, &connection_id, None); + } + }); + tokio::spawn({ + let state = state.clone(); + let provider_id = provider_id.clone(); + let connection_id = connection_id.clone(); + async move { + while let Some(data) = rx.recv().await { + if writer.write_all(&data).await.is_err() { + break; + } + } + state + .active_egress_tcp + .lock() + .await + .remove(&egress_tcp_key(&provider_id, &connection_id)); + } + }); + Ok(()) +} + +pub async fn handle_egress_tcp_data(state: &Arc<AppState>, message: &Value) -> anyhow::Result<()> { + let provider_id = message + .get("providerId") + .and_then(Value::as_str) + .unwrap_or(""); + let connection_id = message + .get("connectionId") + .and_then(Value::as_str) + .unwrap_or(""); + let data = message.get("data").and_then(Value::as_str).unwrap_or(""); + let encoding = message + .get("encoding") + .and_then(Value::as_str) + .unwrap_or("utf8"); + let bytes = if encoding == "base64" { + base64::engine::general_purpose::STANDARD.decode(data)? + } else { + data.as_bytes().to_vec() + }; + let connections = state.active_egress_tcp.lock().await; + if let Some(connection) = connections.get(&egress_tcp_key(provider_id, connection_id)) { + let _ = connection.writer.send(bytes); + } + Ok(()) +} + +pub async fn handle_egress_tcp_close(state: &Arc<AppState>, message: &Value) -> anyhow::Result<()> { + let provider_id = message + .get("providerId") + .and_then(Value::as_str) + .unwrap_or(""); + let connection_id = message + .get("connectionId") + .and_then(Value::as_str) + .unwrap_or(""); + if provider_id.is_empty() || connection_id.is_empty() { + bail!("egress_tcp_close requires providerId and connectionId"); + } + close_egress_tcp_connection(state, provider_id, connection_id, None).await; + Ok(()) +} + +pub async fn close_egress_tcp_connection( + state: &Arc<AppState>, + provider_id: &str, + connection_id: &str, + _error: Option<&str>, +) { + state + .active_egress_tcp + .lock() + .await + .remove(&egress_tcp_key(provider_id, connection_id)); +} + +pub async fn close_egress_tcp_connections_for_provider(state: &Arc<AppState>, provider_id: &str) { + let mut connections = state.active_egress_tcp.lock().await; + connections.retain(|_, connection| connection.provider_id != provider_id); +} diff --git a/src/components/backend-core/src/http.rs b/src/components/backend-core/src/http.rs new file mode 100644 index 00000000..f9c99d09 --- /dev/null +++ b/src/components/backend-core/src/http.rs @@ -0,0 +1,424 @@ +use std::sync::Arc; + +use axum::body::{to_bytes, Body}; +use axum::http::{header, HeaderValue, Method, Request, StatusCode}; +use axum::response::Response; +use serde_json::{json, Value}; + +use crate::db::{ + get_events, get_node_docker_statuses, get_node_system_statuses, get_nodes, get_task, get_tasks, +}; +use crate::performance::with_operation; +use crate::state::AppState; + +pub type HttpResponse = Response; + +pub fn json_response(value: Value, status: u16) -> Response { + let status = StatusCode::from_u16(status).unwrap_or(StatusCode::INTERNAL_SERVER_ERROR); + let mut response = Response::builder() + .status(status) + .header(header::CONTENT_TYPE, "application/json; charset=utf-8") + .header("access-control-allow-origin", "*") + .header( + "access-control-allow-methods", + "GET,HEAD,POST,PUT,PATCH,DELETE,OPTIONS", + ) + .header( + "access-control-allow-headers", + "content-type,x-provider-token", + ) + .body(Body::from( + serde_json::to_string_pretty(&value).unwrap_or_else(|_| "{}".to_string()), + )) + .unwrap(); + response.headers_mut().insert( + header::CONTENT_TYPE, + HeaderValue::from_static("application/json; charset=utf-8"), + ); + response +} + +pub fn text_response(text: impl Into<String>, status: u16) -> Response { + Response::builder() + .status(StatusCode::from_u16(status).unwrap_or(StatusCode::OK)) + .header(header::CONTENT_TYPE, "text/plain; charset=utf-8") + .header("access-control-allow-origin", "*") + .body(Body::from(text.into())) + .unwrap() +} + +pub fn error_json(error: &anyhow::Error) -> Value { + json!({ "name": "Error", "message": error.to_string() }) +} + +fn is_dev_identity(state: &AppState) -> bool { + state.config.identity.environment == "dev" || state.config.identity.namespace == "unidesk-dev" +} + +fn health_payload(state: &AppState) -> Value { + let base = json!({ + "ok": true, + "service": "unidesk-core", + "dbReady": state.db_ready(), + "startedAt": state.service_started_at.to_rfc3339_opts(chrono::SecondsFormat::Millis, true), + }); + if !is_dev_identity(state) { + return base; + } + let identity = &state.config.identity; + json!({ + "ok": true, + "service": "unidesk-core", + "dbReady": state.db_ready(), + "startedAt": state.service_started_at.to_rfc3339_opts(chrono::SecondsFormat::Millis, true), + "environment": identity.environment.clone(), + "namespace": identity.namespace.clone(), + "databaseName": identity.database_name.clone(), + "serviceId": identity.service_id.clone(), + "deployRef": identity.deploy_ref.clone(), + "deploy": { + "repo": identity.repo.clone(), + "commit": identity.commit.clone(), + "requestedCommit": identity.requested_commit.clone(), + }, + }) +} + +pub async fn route_api(state: Arc<AppState>, request: Request<Body>) -> Response { + let uri = request.uri().clone(); + let path = uri.path().to_string(); + let method = request.method().clone(); + if method == Method::OPTIONS { + return json_response(json!({ "ok": true }), 200); + } + let result = route_api_inner(state.clone(), request).await; + match result { + Ok(response) => response, + Err(error) => { + state.log( + "error", + "request_failed", + Some(json!({ "path": path, "error": error.to_string() })), + ); + json_response(json!({ "ok": false, "error": error.to_string() }), 500) + } + } +} + +async fn route_api_inner(state: Arc<AppState>, request: Request<Body>) -> anyhow::Result<Response> { + let method = request.method().clone(); + let uri = request.uri().clone(); + let path = uri.path().to_string(); + let query = uri.query().unwrap_or(""); + if !state.db_ready() && path.starts_with("/api/") { + return Ok(json_response( + json!({ "ok": false, "error": "database not ready" }), + 503, + )); + } + let url = ParsedUrl::new(&path, query); + match (method.as_str(), path.as_str()) { + ("GET", "/health") | ("GET", "/") => Ok(json_response(health_payload(&state), 200)), + ("GET", "/api/overview") => { + let value = with_operation( + &state, + "core", + "overview", + &path, + crate::overview::get_overview(&state), + ) + .await?; + Ok(json_response(value, 200)) + } + ("GET", "/api/nodes") => { + let nodes = with_operation(&state, "core", "nodes", &path, get_nodes(&state)).await?; + Ok(json_response(json!({ "ok": true, "nodes": nodes }), 200)) + } + ("GET", "/api/nodes/system-status") => { + let limit = read_limit(&url, 60); + let statuses = with_operation( + &state, + "core", + "node_system_status", + query, + get_node_system_statuses(&state, limit), + ) + .await?; + Ok(json_response( + json!({ "ok": true, "systemStatuses": statuses }), + 200, + )) + } + ("GET", "/api/nodes/docker-status") => { + let statuses = with_operation( + &state, + "core", + "node_docker_status", + &path, + get_node_docker_statuses(&state), + ) + .await?; + Ok(json_response( + json!({ "ok": true, "dockerStatuses": statuses }), + 200, + )) + } + ("GET", "/api/events") => { + let events = with_operation( + &state, + "core", + "events", + query, + get_events(&state, read_limit(&url, 100)), + ) + .await?; + Ok(json_response(json!({ "ok": true, "events": events }), 200)) + } + ("GET", "/api/tasks") => { + let lite = truthy(url.param("lite")); + let summary = truthy(url.param("summary")); + let status = url.param("status").unwrap_or("all"); + let tasks = with_operation( + &state, + "core", + "tasks", + query, + get_tasks(&state, read_limit(&url, 100), status, lite, summary), + ) + .await?; + Ok(json_response(json!({ "ok": true, "tasks": tasks }), 200)) + } + ("GET", "/api/microservices") => { + let microservices = with_operation( + &state, + "core", + "microservices", + &path, + crate::microservice_proxy::get_microservices(&state), + ) + .await?; + Ok(json_response( + json!({ "ok": true, "microservices": microservices }), + 200, + )) + } + ("GET", "/api/performance") => Ok(json_response( + crate::performance::get_performance(&state).await?, + 200, + )), + ("GET", "/logs") => Ok(json_response( + json!({ "ok": true, "logs": state.logger.recent(read_limit(&url, 100) as usize) }), + 200, + )), + ("GET", "/favicon.ico") => Ok(text_response("", 204)), + _ if path.starts_with("/api/tasks/") && method == Method::GET => { + let task_id = percent_decode(&path["/api/tasks/".len()..]); + if task_id.is_empty() || task_id.contains('/') { + return Ok(json_response( + json!({ "ok": false, "error": "invalid task id" }), + 400, + )); + } + let task = with_operation( + &state, + "core", + "task_detail", + &task_id, + get_task(&state, &task_id), + ) + .await?; + match task { + Some(task) => Ok(json_response(json!({ "ok": true, "task": task }), 200)), + None => Ok(json_response( + json!({ "ok": false, "error": format!("task not found: {task_id}") }), + 404, + )), + } + } + _ if path == "/api/schedules" || path.starts_with("/api/schedules/") => { + let response = with_operation( + &state, + "scheduler", + "schedules", + &path, + crate::scheduler::scheduled_task_route( + &state, + method, + &path, + query, + body_bytes(request).await?, + ), + ) + .await?; + Ok(response) + } + _ if path == "/api/code-queue-load-test" + && (method == Method::GET || method == Method::POST) => + { + let body = body_bytes(request).await?; + let response = with_operation( + &state, + "performance", + "code_queue_load_test", + &path, + crate::overview::codex_queue_load_test(&state, method, body), + ) + .await?; + Ok(response) + } + _ if path.starts_with("/api/microservices/") => { + let response = with_operation( + &state, + "microservices", + "route", + &path, + crate::microservice_proxy::microservice_route(&state, method, uri, request), + ) + .await?; + Ok(response) + } + _ if path == "/api/dispatch" && method == Method::POST => { + let body = body_bytes(request).await?; + let response = with_operation(&state, "scheduler", "dispatch", &path, async { + Ok(crate::task_dispatcher::dispatch_task(&state, body).await) + }) + .await?; + Ok(response) + } + _ => Ok(json_response( + json!({ "ok": false, "error": "not found", "path": path }), + 404, + )), + } +} + +async fn body_bytes(request: Request<Body>) -> anyhow::Result<bytes::Bytes> { + Ok(to_bytes(request.into_body(), 16 * 1024 * 1024).await?) +} + +pub fn response_with_body(status: u16, content_type: &str, body: impl Into<Body>) -> Response { + Response::builder() + .status(StatusCode::from_u16(status).unwrap_or(StatusCode::OK)) + .header(header::CONTENT_TYPE, content_type) + .body(body.into()) + .unwrap() +} + +pub fn add_header(response: &mut Response, name: &'static str, value: &str) { + if let Ok(value) = HeaderValue::from_str(value) { + response.headers_mut().insert(name, value); + } +} + +pub async fn response_to_text(response: Response) -> anyhow::Result<(u16, String, String)> { + let status = response.status().as_u16(); + let content_type = response + .headers() + .get(header::CONTENT_TYPE) + .and_then(|value| value.to_str().ok()) + .unwrap_or("application/octet-stream") + .to_string(); + let bytes = to_bytes(response.into_body(), 16 * 1024 * 1024).await?; + let body_text = String::from_utf8_lossy(&bytes).to_string(); + Ok((status, content_type, body_text)) +} + +pub fn read_limit(url: &ParsedUrl, default_limit: i64) -> i64 { + url.param("limit") + .and_then(|raw| raw.parse::<i64>().ok()) + .filter(|value| *value > 0) + .map(|value| value.min(500)) + .unwrap_or(default_limit) +} + +pub fn truthy(value: Option<&str>) -> bool { + value.is_some_and(|value| matches!(value.to_ascii_lowercase().as_str(), "1" | "true" | "yes")) +} + +pub fn percent_decode(value: &str) -> String { + percent_encoding::percent_decode_str(value) + .decode_utf8_lossy() + .to_string() +} + +#[derive(Debug, Clone)] +pub struct ParsedUrl { + pub path: String, + pub params: Vec<(String, String)>, +} + +impl ParsedUrl { + pub fn new(path: &str, query: &str) -> Self { + let params = url::form_urlencoded::parse(query.as_bytes()) + .map(|(key, value)| (key.into_owned(), value.into_owned())) + .collect(); + Self { + path: path.to_string(), + params, + } + } + + pub fn param(&self, name: &str) -> Option<&str> { + self.params + .iter() + .find(|(key, _)| key == name) + .map(|(_, value)| value.as_str()) + } + + pub fn all(&self, name: &str) -> Vec<&str> { + self.params + .iter() + .filter(|(key, _)| key == name) + .map(|(_, value)| value.as_str()) + .collect() + } +} + +pub async fn internal_fetch_json( + url: &str, + method: &str, + body: Option<Value>, + max_response_bytes: usize, +) -> anyhow::Result<Value> { + let client = reqwest::Client::new(); + let mut request = client.request(method.parse()?, url); + if let Some(body) = body { + request = request.json(&body); + } + let response = request.send().await?; + let status = response.status().as_u16(); + let ok = response.status().is_success(); + let content_length = response.content_length(); + let bytes = response.bytes().await?; + let response_truncated = bytes.len() > max_response_bytes; + let kept = if response_truncated { + &bytes[..max_response_bytes] + } else { + &bytes[..] + }; + let text = String::from_utf8_lossy(kept).to_string(); + let body = if response_truncated { + json!({ "_unideskResponseTruncated": true, "maxResponseBytes": max_response_bytes, "bytesRead": kept.len(), "contentLength": content_length, "textPreview": text }) + } else { + serde_json::from_str(&text).unwrap_or_else(|_| json!({ "text": text })) + }; + Ok( + json!({ "ok": ok, "status": status, "responseTruncated": response_truncated, "responseBytesRead": kept.len(), "responseContentLength": content_length, "body": body }), + ) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn parsed_url_preserves_path_and_repeated_query_values() { + let parsed = ParsedUrl::new("/api/tasks", "limit=999&limit=10&flag=yes&name=a%20b"); + + assert_eq!(parsed.path, "/api/tasks"); + assert_eq!(parsed.param("name"), Some("a b")); + assert_eq!(parsed.all("limit"), vec!["999", "10"]); + assert_eq!(read_limit(&parsed, 30), 500); + assert!(truthy(parsed.param("flag"))); + } +} diff --git a/src/components/backend-core/src/json_util.rs b/src/components/backend-core/src/json_util.rs new file mode 100644 index 00000000..0f08c25e --- /dev/null +++ b/src/components/backend-core/src/json_util.rs @@ -0,0 +1,244 @@ +use serde_json::{json, Map, Value}; +use sha2::{Digest, Sha256}; + +pub fn now_iso() -> String { + chrono::Utc::now().to_rfc3339_opts(chrono::SecondsFormat::Millis, true) +} + +pub fn millis_now() -> i64 { + chrono::Utc::now().timestamp_millis() +} + +pub fn error_json(error: &(dyn std::error::Error + 'static)) -> Value { + json!({ "name": "Error", "message": error.to_string() }) +} + +pub fn compact_json(value: &Value) -> Value { + compact_json_depth(value, 0) +} + +fn compact_json_depth(value: &Value, depth: usize) -> Value { + match value { + Value::Null | Value::Bool(_) | Value::Number(_) => value.clone(), + Value::String(text) => { + if text.len() > 600 { + Value::String(format!( + "{}...<truncated:{}>", + &text[..600.min(text.len())], + text.len() + )) + } else { + value.clone() + } + } + Value::Array(items) => { + let mut result: Vec<Value> = items + .iter() + .take(20) + .map(|item| compact_json_depth(item, depth + 1)) + .collect(); + if items.len() > 20 { + result.push(json!({ "truncatedItems": items.len() - 20 })); + } + Value::Array(result) + } + Value::Object(object) => { + if depth >= 4 { + return Value::String("<truncated:depth>".to_string()); + } + let mut result = Map::new(); + for (key, item) in object.iter().take(30) { + result.insert(key.clone(), compact_json_depth(item, depth + 1)); + } + if object.len() > 30 { + result.insert("truncatedKeys".to_string(), json!(object.len() - 30)); + } + Value::Object(result) + } + } +} + +fn sha256_text(value: &str) -> String { + let mut hasher = Sha256::new(); + hasher.update(value.as_bytes()); + format!("{:x}", hasher.finalize()) +} + +fn bounded_string(value: &str, limit: usize) -> Value { + if value.len() <= limit { + return Value::String(value.to_string()); + } + json!({ + "omitted": true, + "omittedReason": "string too large for database inline JSON storage", + "chars": value.len(), + "preview": &value[..limit.min(value.len())], + "sha256": sha256_text(value), + }) +} + +fn compact_body_text(value: &str) -> Value { + const BODY_INLINE_LIMIT: usize = 16_384; + const BODY_PREVIEW_LIMIT: usize = 2_000; + if value.len() <= BODY_INLINE_LIMIT { + return Value::String(value.to_string()); + } + json!({ + "omitted": true, + "omittedReason": "microservice HTTP body is stored as metadata only", + "bodyPreview": &value[..BODY_PREVIEW_LIMIT.min(value.len())], + "bodyTextChars": value.len(), + "bodyTextSha256": sha256_text(value), + }) +} + +pub fn compact_json_for_storage(value: &Value) -> Value { + compact_json_for_storage_inner(value, 0, "") +} + +fn compact_json_for_storage_inner(value: &Value, depth: usize, key: &str) -> Value { + const DEFAULT_STRING_LIMIT: usize = 8_000; + const MAX_ARRAY_ITEMS: usize = 100; + const MAX_OBJECT_KEYS: usize = 100; + const MAX_DEPTH: usize = 8; + match value { + Value::Null | Value::Bool(_) | Value::Number(_) => value.clone(), + Value::String(text) => { + if key == "bodyText" { + compact_body_text(text) + } else { + bounded_string(text, DEFAULT_STRING_LIMIT) + } + } + Value::Array(items) => { + let mut result: Vec<Value> = items + .iter() + .take(MAX_ARRAY_ITEMS) + .map(|item| compact_json_for_storage_inner(item, depth + 1, "")) + .collect(); + if items.len() > MAX_ARRAY_ITEMS { + result.push(json!({ "truncatedItems": items.len() - MAX_ARRAY_ITEMS })); + } + Value::Array(result) + } + Value::Object(object) => { + if depth >= MAX_DEPTH { + return Value::String("<truncated:depth>".to_string()); + } + let mut result = Map::new(); + for (child_key, item) in object.iter().take(MAX_OBJECT_KEYS) { + if child_key == "bodyText" { + if let Some(text) = item.as_str() { + if let Value::Object(map) = compact_body_text(text) { + for (key, value) in map { + result.insert(key, value); + } + continue; + } + } + } + result.insert( + child_key.clone(), + compact_json_for_storage_inner(item, depth + 1, child_key), + ); + } + if object.len() > MAX_OBJECT_KEYS { + result.insert( + "truncatedKeys".to_string(), + json!(object.len() - MAX_OBJECT_KEYS), + ); + } + Value::Object(result) + } + } +} + +pub fn nested_number(value: &Value, object_key: &str, number_key: &str) -> f64 { + value + .get(object_key) + .and_then(|object| object.get(number_key)) + .and_then(|value| { + value + .as_f64() + .or_else(|| value.as_str().and_then(|text| text.parse::<f64>().ok())) + }) + .unwrap_or(0.0) +} + +pub fn redact_database_url(value: &str) -> String { + match url::Url::parse(value) { + Ok(mut url) => { + if url.password().is_some() { + let _ = url.set_password(Some("***")); + } + url.to_string() + } + Err(_) => "<invalid-url>".to_string(), + } +} + +pub fn is_plain_record(value: &Value) -> bool { + matches!(value, Value::Object(_)) +} + +pub fn truncate_text(value: &str, max_chars: usize) -> String { + if value.len() <= max_chars { + value.to_string() + } else { + value[..max_chars.min(value.len())].to_string() + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn compact_json_for_storage_replaces_large_body_text_with_metadata() { + let body_text = "x".repeat(20_000); + let compacted = compact_json_for_storage(&json!({ + "status": 200, + "bodyText": body_text, + })); + + assert_eq!(compacted.get("status").and_then(Value::as_i64), Some(200)); + assert!(compacted.get("bodyText").is_none()); + assert_eq!( + compacted.get("omitted").and_then(Value::as_bool), + Some(true) + ); + assert_eq!( + compacted.get("bodyTextChars").and_then(Value::as_u64), + Some(20_000) + ); + assert_eq!( + compacted + .get("bodyPreview") + .and_then(Value::as_str) + .map(str::len), + Some(2_000) + ); + assert_eq!( + compacted + .get("bodyTextSha256") + .and_then(Value::as_str) + .map(str::len), + Some(64) + ); + } + + #[test] + fn compact_json_for_storage_bounds_large_regular_strings() { + let compacted = compact_json_for_storage(&json!({ + "large": "a".repeat(9_000), + })); + let large = compacted.get("large").expect("large field"); + + assert_eq!(large.get("omitted").and_then(Value::as_bool), Some(true)); + assert_eq!(large.get("chars").and_then(Value::as_u64), Some(9_000)); + assert_eq!( + large.get("preview").and_then(Value::as_str).map(str::len), + Some(8_000) + ); + } +} diff --git a/src/components/backend-core/src/logger.rs b/src/components/backend-core/src/logger.rs new file mode 100644 index 00000000..6d8d2cdd --- /dev/null +++ b/src/components/backend-core/src/logger.rs @@ -0,0 +1,287 @@ +use std::fs::{self, OpenOptions}; +use std::io::Write; +use std::path::{Path, PathBuf}; +use std::sync::{Arc, Mutex}; + +use anyhow::Context; +use chrono::{Datelike, Timelike}; +use serde_json::{json, Value}; + +use crate::json_util::now_iso; + +const DEFAULT_LOG_RETENTION_BYTES: u64 = 1024 * 1024 * 1024; + +#[derive(Clone)] +pub struct Logger { + service: String, + root_dir: PathBuf, + prefix: String, + suffix: String, + max_bytes: u64, + recent: Arc<Mutex<Vec<Value>>>, + last_prune_ms: Arc<Mutex<i64>>, +} + +impl Logger { + pub fn new(service: &str, log_file: &str) -> anyhow::Result<Self> { + let service = safe_service_name(service); + let (root_dir, prefix, suffix) = parse_base(log_file, &service); + let max_bytes = log_retention_bytes_for_service(&service); + let logger = Self { + service, + root_dir, + prefix, + suffix, + max_bytes, + recent: Arc::new(Mutex::new(Vec::new())), + last_prune_ms: Arc::new(Mutex::new(0)), + }; + logger.prune()?; + Ok(logger) + } + + pub fn log(&self, level: &str, message: &str, data: Option<Value>) { + let entry = match data { + Some(data) => { + json!({ "ts": now_iso(), "service": self.service, "level": level, "message": message, "data": data }) + } + None => { + json!({ "ts": now_iso(), "service": self.service, "level": level, "message": message }) + } + }; + { + let mut recent = self.recent.lock().expect("recent logs poisoned"); + recent.push(entry.clone()); + while recent.len() > 500 { + recent.remove(0); + } + } + let line = format!("{entry}\n"); + if let Err(error) = self.append_line(&line) { + eprintln!( + "{}", + json!({ "ts": now_iso(), "service": self.service, "level": "error", "message": "log_write_failed", "data": error.to_string() }) + ); + } + match level { + "error" => eprintln!("{}", line.trim_end()), + "warn" => eprintln!("{}", line.trim_end()), + _ => println!("{}", line.trim_end()), + } + } + + pub fn recent(&self, limit: usize) -> Vec<Value> { + let recent = self.recent.lock().expect("recent logs poisoned"); + recent + .iter() + .rev() + .take(limit) + .cloned() + .collect::<Vec<_>>() + .into_iter() + .rev() + .collect() + } + + fn current_path(&self) -> PathBuf { + let now = chrono::Local::now(); + let day = format!("{:04}{:02}{:02}", now.year(), now.month(), now.day()); + let hour = format!("{:02}", now.hour()); + self.root_dir + .join(day.clone()) + .join(format!("{}_{}_{}{}", self.prefix, day, hour, self.suffix)) + } + + fn append_line(&self, line: &str) -> anyhow::Result<()> { + let path = self.current_path(); + if let Some(parent) = path.parent() { + fs::create_dir_all(parent) + .with_context(|| format!("create log directory {}", parent.display()))?; + } + let mut file = OpenOptions::new() + .create(true) + .append(true) + .open(&path) + .with_context(|| format!("open log file {}", path.display()))?; + file.write_all(line.as_bytes())?; + self.maybe_prune(); + Ok(()) + } + + fn maybe_prune(&self) { + let now = chrono::Utc::now().timestamp_millis(); + let should_prune = { + let mut last = self.last_prune_ms.lock().expect("last prune poisoned"); + if now - *last < 60_000 { + false + } else { + *last = now; + true + } + }; + if should_prune { + let _ = self.prune(); + } + } + + fn prune(&self) -> anyhow::Result<()> { + let active = self.current_path(); + let mut files = collect_files(&self.root_dir, &self.suffix)?; + files.sort_by(|left, right| { + if left.path == active { + return std::cmp::Ordering::Greater; + } + if right.path == active { + return std::cmp::Ordering::Less; + } + left.path + .cmp(&right.path) + .then(left.modified.cmp(&right.modified)) + }); + let mut total: u64 = files.iter().map(|file| file.size).sum(); + for file in files { + if total <= self.max_bytes { + break; + } + if file.path == active { + continue; + } + if fs::remove_file(&file.path).is_ok() { + total = total.saturating_sub(file.size); + } + } + Ok(()) + } +} + +struct FileEntry { + path: PathBuf, + size: u64, + modified: std::time::SystemTime, +} + +fn safe_service_name(service: &str) -> String { + let filtered = service + .chars() + .map(|ch| { + if ch.is_ascii_alphanumeric() || ch == '_' || ch == '.' || ch == '-' { + ch + } else { + '-' + } + }) + .collect::<String>() + .trim_matches('-') + .to_string(); + if filtered.is_empty() { + "service".to_string() + } else { + filtered + } +} + +fn parse_base(base_log_file: &str, service: &str) -> (PathBuf, String, String) { + let suffix = format!("_{service}.jsonl"); + let path = Path::new(base_log_file); + let dir = path.parent().unwrap_or_else(|| Path::new(".")); + let dir_name = dir.file_name().and_then(|name| name.to_str()).unwrap_or(""); + let root_dir = if dir_name.len() == 8 && dir_name.chars().all(|ch| ch.is_ascii_digit()) { + dir.parent().unwrap_or(dir).to_path_buf() + } else { + dir.to_path_buf() + }; + let file_name = path + .file_name() + .and_then(|name| name.to_str()) + .unwrap_or(""); + let stem = file_name.strip_suffix(".jsonl").unwrap_or(file_name); + let prefix = file_name + .strip_suffix(&suffix) + .map(ToOwned::to_owned) + .or_else(|| stem.rsplit_once('_').map(|(left, _)| left.to_string())) + .unwrap_or_else(|| "unidesk".to_string()); + ( + root_dir, + if prefix.is_empty() { + "unidesk".to_string() + } else { + prefix + }, + suffix, + ) +} + +fn parse_retention(raw: &str, fallback: u64) -> u64 { + let trimmed = raw.trim(); + if trimmed.is_empty() { + return fallback; + } + let split = trimmed + .find(|ch: char| !(ch.is_ascii_digit() || ch == '.')) + .unwrap_or(trimmed.len()); + let number = trimmed[..split].parse::<f64>().unwrap_or(fallback as f64); + let unit = trimmed[split..].trim().to_ascii_lowercase(); + let multiplier = match unit.as_str() { + "g" | "gb" | "gib" => 1024.0 * 1024.0 * 1024.0, + "m" | "mb" | "mib" => 1024.0 * 1024.0, + "k" | "kb" | "kib" => 1024.0, + _ => 1.0, + }; + let bytes = number * multiplier; + if bytes.is_finite() && bytes > 0.0 { + bytes.floor() as u64 + } else { + fallback + } +} + +fn log_retention_bytes_for_service(service: &str) -> u64 { + let global = std::env::var("UNIDESK_LOG_RETENTION_BYTES") + .map(|value| parse_retention(&value, DEFAULT_LOG_RETENTION_BYTES)) + .unwrap_or(DEFAULT_LOG_RETENTION_BYTES); + let env_name = format!( + "UNIDESK_{}_LOG_MAX_BYTES", + service + .to_ascii_uppercase() + .replace(|ch: char| !ch.is_ascii_alphanumeric(), "_") + ); + std::env::var(&env_name) + .map(|value| parse_retention(&value, global)) + .unwrap_or(global) +} + +fn collect_files(root: &Path, suffix: &str) -> anyhow::Result<Vec<FileEntry>> { + let mut result = Vec::new(); + if !root.exists() { + return Ok(result); + } + fn scan(dir: &Path, suffix: &str, result: &mut Vec<FileEntry>) -> anyhow::Result<()> { + for entry in fs::read_dir(dir)? { + let entry = entry?; + let path = entry.path(); + let metadata = match entry.metadata() { + Ok(metadata) => metadata, + Err(_) => continue, + }; + if metadata.is_dir() { + let _ = scan(&path, suffix, result); + } else if metadata.is_file() + && path + .file_name() + .and_then(|name| name.to_str()) + .is_some_and(|name| name.ends_with(suffix)) + { + result.push(FileEntry { + path, + size: metadata.len(), + modified: metadata + .modified() + .unwrap_or(std::time::SystemTime::UNIX_EPOCH), + }); + } + } + Ok(()) + } + scan(root, suffix, &mut result)?; + Ok(result) +} diff --git a/src/components/backend-core/src/main.rs b/src/components/backend-core/src/main.rs new file mode 100644 index 00000000..e4ce03ab --- /dev/null +++ b/src/components/backend-core/src/main.rs @@ -0,0 +1,221 @@ +mod cli; +mod config; +mod db; +mod egress_tcp; +mod http; +mod json_util; +mod logger; +mod microservice_proxy; +mod overview; +mod performance; +mod provider_registry; +mod scheduler; +mod ssh_bridge; +mod state; +mod task_dispatcher; +mod types; + +use std::net::SocketAddr; +use std::sync::Arc; +use std::time::Duration; + +use anyhow::Context; +use axum::extract::ws::Message; +use axum::extract::{Request, State, WebSocketUpgrade}; +use axum::http::HeaderMap; +use axum::response::Response; +use axum::routing::get; +use axum::Router; +use tokio::net::TcpListener; + +use crate::config::read_config; +use crate::db::init_database; +use crate::http::{error_json, json_response}; +use crate::logger::Logger; +use crate::performance::record_request_performance; +use crate::provider_registry::{mark_stale_providers_offline, provider_ws}; +use crate::scheduler::{recover_scheduled_runs, run_due_scheduled_tasks}; +use crate::ssh_bridge::ssh_ws; +use crate::state::AppState; +use crate::task_dispatcher::mark_stale_tasks_failed; + +#[tokio::main] +async fn main() -> anyhow::Result<()> { + if cli::handle_cli().await? { + return Ok(()); + } + + let config = read_config()?; + let logger = Logger::new("backend-core", &config.log_file)?; + let state = Arc::new(AppState::new(config, logger).await?); + + match init_database(&state).await { + Ok(()) => { + state.set_db_ready(true); + state.log("info", "database_ready", None); + if let Err(error) = recover_scheduled_runs(&state).await { + state.log( + "error", + "scheduled_recovery_failed", + Some(error_json(&error)), + ); + } + } + Err(error) => { + state.log("error", "database_init_failed", Some(error_json(&error))); + return Err(error).context("database initialization failed"); + } + } + + spawn_sweeps(state.clone()); + + let api_addr = SocketAddr::from(([0, 0, 0, 0], state.config.port)); + let provider_addr = SocketAddr::from(([0, 0, 0, 0], state.config.provider_port)); + let api = Router::new() + .route("/ws/ssh", get(ssh_ws_handler)) + .fallback(api_handler) + .with_state(state.clone()); + let provider = Router::new() + .route("/", get(provider_health_handler)) + .route("/health", get(provider_health_handler)) + .route("/ws/provider", get(provider_ws_handler)) + .fallback(provider_not_found_handler) + .with_state(state.clone()); + + let api_listener = TcpListener::bind(api_addr).await?; + let provider_listener = TcpListener::bind(provider_addr).await?; + state.log( + "info", + "server_listening", + Some(serde_json::json!({ + "apiUrl": format!("http://0.0.0.0:{}", state.config.port), + "providerIngressUrl": format!("ws://0.0.0.0:{}/ws/provider", state.config.provider_port), + "logFile": state.config.log_file, + })), + ); + + let api_server = axum::serve(api_listener, api); + let provider_server = axum::serve(provider_listener, provider); + tokio::try_join!(api_server, provider_server)?; + Ok(()) +} + +fn spawn_sweeps(state: Arc<AppState>) { + tokio::spawn({ + let state = state.clone(); + async move { + let mut interval = tokio::time::interval(Duration::from_secs(10)); + loop { + interval.tick().await; + if let Err(error) = mark_stale_providers_offline(&state).await { + state.log("error", "heartbeat_sweep_failed", Some(error_json(&error))); + } + } + } + }); + tokio::spawn({ + let state = state.clone(); + async move { + let period = Duration::from_millis(state.config.task_pending_timeout_ms.min(60_000)); + let mut interval = tokio::time::interval(period); + loop { + interval.tick().await; + if let Err(error) = mark_stale_tasks_failed(&state).await { + state.log( + "error", + "task_timeout_sweep_failed", + Some(error_json(&error)), + ); + } + } + } + }); + tokio::spawn(async move { + let mut interval = tokio::time::interval(Duration::from_secs(30)); + loop { + interval.tick().await; + if let Err(error) = run_due_scheduled_tasks(&state).await { + state.log( + "error", + "scheduled_task_sweep_failed", + Some(error_json(&error)), + ); + } + } + }); +} + +async fn api_handler(State(state): State<Arc<AppState>>, request: Request) -> Response { + let method = request.method().clone(); + let path = request.uri().path().to_string(); + let started = std::time::Instant::now(); + let response = crate::http::route_api(state.clone(), request).await; + record_request_performance( + &state, + method.as_str(), + &path, + response.status().as_u16(), + started.elapsed(), + ); + response +} + +async fn ssh_ws_handler( + State(state): State<Arc<AppState>>, + ws: WebSocketUpgrade, + headers: HeaderMap, + request: Request, +) -> Response { + let uri = request.uri().clone(); + let started = std::time::Instant::now(); + let response = ssh_ws(state.clone(), ws, headers, uri).await; + record_request_performance( + &state, + "GET", + "/ws/ssh", + response.status().as_u16(), + started.elapsed(), + ); + response +} + +async fn provider_health_handler(State(state): State<Arc<AppState>>) -> Response { + json_response( + serde_json::json!({ + "ok": true, + "service": "unidesk-provider-ingress", + "activeSocketCount": state.active_provider_count().await, + }), + 200, + ) +} + +async fn provider_ws_handler( + State(state): State<Arc<AppState>>, + ws: WebSocketUpgrade, + headers: HeaderMap, + request: Request, +) -> Response { + let uri = request.uri().clone(); + let started = std::time::Instant::now(); + let response = provider_ws(state.clone(), ws, headers, uri).await; + record_request_performance( + &state, + "GET", + "/ws/provider", + response.status().as_u16(), + started.elapsed(), + ); + response +} + +async fn provider_not_found_handler() -> Response { + json_response( + serde_json::json!({ "ok": false, "error": "not found" }), + 404, + ) +} + +pub(crate) fn ws_text(value: serde_json::Value) -> Message { + Message::Text(value.to_string()) +} diff --git a/src/components/backend-core/src/microservice_proxy.rs b/src/components/backend-core/src/microservice_proxy.rs new file mode 100644 index 00000000..1cc13d24 --- /dev/null +++ b/src/components/backend-core/src/microservice_proxy.rs @@ -0,0 +1,2015 @@ +use std::sync::Arc; +use std::time::Duration; + +use axum::body::{to_bytes, Body}; +use axum::http::{header, HeaderMap, Method, Request, Uri}; +use axum::response::Response; +use serde_json::{json, Value}; + +use crate::db::{get_node_docker_statuses, get_nodes, provider_supports}; +use crate::http::{ + add_header, json_response, percent_decode, response_to_text, response_with_body, +}; +use crate::json_util::{compact_json, millis_now, truncate_text}; +use crate::state::{AppState, HttpTunnelResponse}; +use crate::task_dispatcher::{create_and_send_task, wait_for_task_terminal}; +use crate::types::{MicroserviceConfig, MicroserviceProxyCacheEntry}; + +const MICROSERVICE_PROXY_MAX_BODY_TEXT_LENGTH: usize = 8 * 1024 * 1024; +const MICROSERVICE_AVAILABILITY_TTL_MS: i64 = 30_000; +const CODE_QUEUE_OVERVIEW_PATH_FALLBACK_STALE_MS: i64 = 30_000; +const PROVIDER_HTTP_TUNNEL_MAX_ATTEMPTS: i32 = 3; +const FORWARD_REQUEST_HEADERS: &[&str] = &[ + "accept", + "content-type", + "range", + "x-auth", + "x-requested-with", + "destination", + "overwrite", + "tus-resumable", + "upload-concat", + "upload-defer-length", + "upload-length", + "upload-metadata", + "upload-offset", +]; + +fn service_by_id(state: &Arc<AppState>, service_id: &str) -> Option<MicroserviceConfig> { + state + .config + .microservices + .iter() + .find(|service| service.id == service_id) + .cloned() +} + +fn is_microservice_path_allowed(service: &MicroserviceConfig, path: &str) -> bool { + service + .backend + .allowed_path_prefixes + .iter() + .any(|prefix| path == prefix || path.starts_with(prefix)) +} + +fn is_microservice_method_allowed(service: &MicroserviceConfig, method: &Method) -> bool { + service + .backend + .allowed_methods + .iter() + .any(|allowed| allowed == method.as_str()) +} + +fn docker_status_record(status: Option<&Value>) -> serde_json::Map<String, Value> { + status + .and_then(Value::as_object) + .cloned() + .unwrap_or_default() +} + +fn find_container(status: Option<&Value>, container_name: &str) -> Value { + status + .and_then(|status| status.get("containers")) + .and_then(Value::as_array) + .and_then(|containers| { + containers + .iter() + .find(|item| item.get("name").and_then(Value::as_str) == Some(container_name)) + .cloned() + }) + .unwrap_or(Value::Null) +} + +fn is_k3sctl_managed_microservice(service: &MicroserviceConfig) -> bool { + service.deployment.mode == "k3sctl-managed" + || service.backend.proxy_mode == "k3sctl-adapter-http" +} + +fn can_direct_proxy_microservice(service: &MicroserviceConfig) -> bool { + service.provider_id == "main-server" +} + +fn code_queue_mgr_internal_url() -> String { + std::env::var("CODE_QUEUE_MGR_INTERNAL_URL") + .unwrap_or_else(|_| "http://code-queue-mgr:4278".to_string()) +} + +fn direct_code_queue_mgr_service( + service: &MicroserviceConfig, + timeout_ms: Option<u64>, +) -> MicroserviceConfig { + let mut copy = service.clone(); + copy.provider_id = "main-server".to_string(); + copy.backend.node_base_url = code_queue_mgr_internal_url(); + copy.backend.proxy_mode = "unidesk-direct".to_string(); + copy.backend.timeout_ms = timeout_ms.unwrap_or_else(|| service.backend.timeout_ms.min(12_000)); + copy.deployment.mode = "unidesk-direct".to_string(); + copy +} + +fn code_queue_k3s_service_id_for_request(method: &Method, target_path: &str) -> &'static str { + let method = method.as_str(); + if matches!(target_path, "/" | "/health" | "/live" | "/api/dev-ready") { + return "code-queue-scheduler"; + } + if matches!( + target_path, + "/api/oa/backfill" + | "/api/notifications/claudeqq/drain" + | "/api/notifications/claudeqq/backfill" + ) { + return "code-queue-scheduler"; + } + if matches!( + target_path, + "/api/judge/probe" + | "/api/judge/self-test" + | "/api/queue-order/self-test" + | "/api/reference-injection/self-test" + | "/api/trace-port/self-test" + ) { + return "code-queue-scheduler"; + } + if path_matches_task_action(target_path, "steer") + || path_matches_task_action(target_path, "interrupt") + { + return "code-queue-scheduler"; + } + if method == "DELETE" && path_matches_task_detail(target_path) { + return "code-queue-scheduler"; + } + if target_path == "/api/dev-containers" || target_path.starts_with("/api/dev-containers/") { + return "code-queue-scheduler"; + } + if method == "GET" || method == "HEAD" { + "code-queue-read" + } else { + "code-queue-write" + } +} + +fn code_queue_scheduler_only_path(method: &Method, target_path: &str) -> bool { + code_queue_k3s_service_id_for_request(method, target_path) == "code-queue-scheduler" + && !(target_path == "/" || target_path == "/health") +} + +fn path_matches_task_detail(path: &str) -> bool { + path.strip_prefix("/api/tasks/") + .is_some_and(|rest| !rest.is_empty() && !rest.contains('/')) +} + +fn path_matches_task_action(path: &str, action: &str) -> bool { + path.strip_prefix("/api/tasks/") + .and_then(|rest| rest.strip_suffix(&format!("/{action}"))) + .is_some_and(|task_id| !task_id.is_empty() && !task_id.contains('/')) +} + +fn code_queue_task_id_for_action(target_path: &str, action: &str) -> Option<String> { + target_path + .strip_prefix("/api/tasks/") + .and_then(|rest| rest.strip_suffix(&format!("/{action}"))) + .map(percent_decode) +} + +fn microservice_cache_ttl_ms(service_id: &str, target_path: &str) -> i64 { + if target_path == "/health" || target_path.ends_with("/stream") { + 0 + } else if service_id == "pipeline" && target_path == "/api/snapshot" { + 6_000 + } else if service_id == "pipeline" && target_path.starts_with("/api/oa-event-flow/") { + 20_000 + } else if service_id == "pipeline" && target_path.starts_with("/api/model-quota/") { + 60_000 + } else if service_id == "pipeline" && target_path.starts_with("/api/runs/") { + 6_000 + } else if service_id == "findjob" + && matches!(target_path, "/api/summary" | "/api/jobs" | "/api/drafts") + { + 8_000 + } else if service_id == "met-nonlinear" + && matches!(target_path, "/api/images" | "/api/projects") + { + 15_000 + } else if service_id == "met-nonlinear" + && matches!(target_path, "/api/queue" | "/api/summary" | "/api/history") + { + 5_000 + } else if service_id == "code-queue" && target_path.contains("/transcript") { + 1_000 + } else if service_id == "code-queue" && target_path == "/api/tasks/overview" { + 3_000 + } else if service_id == "code-queue" && target_path.starts_with("/api/tasks/") { + 2_000 + } else { + 750 + } +} + +fn microservice_cache_stale_ms(service_id: &str, target_path: &str) -> i64 { + if service_id == "pipeline" && target_path.starts_with("/api/model-quota/") { + 5 * 60_000 + } else if service_id == "pipeline" || service_id == "met-nonlinear" { + 45_000 + } else if service_id == "findjob" { + 60_000 + } else if service_id == "code-queue" { + 15_000 + } else { + 5_000 + } +} + +fn provider_microservice_cache_ttl_ms(service_id: &str, target_path: &str) -> i64 { + if target_path == "/health" { + 0 + } else if service_id == "met-nonlinear" + && matches!(target_path, "/api/images" | "/api/projects") + { + 60_000 + } else if service_id == "met-nonlinear" && target_path == "/api/history" { + 10_000 + } else if service_id == "met-nonlinear" && matches!(target_path, "/api/queue" | "/api/summary") + { + 3_000 + } else if service_id == "pipeline" + && (target_path == "/api/snapshot" || target_path.starts_with("/api/oa-event-flow/")) + { + 2_000 + } else if service_id == "findjob" && matches!(target_path, "/api/summary" | "/api/jobs") { + 2_000 + } else { + 1_000 + } +} + +#[derive(Clone)] +struct ProxyOptions { + query: String, + json_array_limits: Value, +} + +fn read_microservice_array_limits(uri: &Uri) -> ProxyOptions { + let query = uri.query().unwrap_or(""); + let mut params = Vec::<(String, String)>::new(); + let mut limits = serde_json::Map::new(); + for (key, value) in url::form_urlencoded::parse(query.as_bytes()) { + if key == "__unideskArrayLimit" { + for entry in value.split(',') { + let mut parts = entry.splitn(2, ':'); + let path = parts.next().unwrap_or(""); + let limit = parts + .next() + .and_then(|text| text.parse::<i64>().ok()) + .unwrap_or(0); + if path + .chars() + .all(|ch| ch.is_ascii_alphanumeric() || matches!(ch, '_' | '.' | '-')) + && limit > 0 + && limit <= 500 + { + limits.insert(path.to_string(), json!(limit)); + } + } + } else { + params.push((key.into_owned(), value.into_owned())); + } + } + let query = if params.is_empty() { + String::new() + } else { + format!( + "?{}", + url::form_urlencoded::Serializer::new(String::new()) + .extend_pairs(params) + .finish() + ) + }; + ProxyOptions { + query, + json_array_limits: Value::Object(limits), + } +} + +fn read_microservice_request_headers(headers: &HeaderMap) -> Value { + let mut result = serde_json::Map::new(); + for name in FORWARD_REQUEST_HEADERS { + if let Some(value) = headers + .get(*name) + .and_then(|value| value.to_str().ok()) + .filter(|value| !value.is_empty()) + { + result.insert( + (*name).to_string(), + Value::String(truncate_text(value, 4096)), + ); + } + } + Value::Object(result) +} + +fn headers_from_microservice_request(request_headers: &Value) -> reqwest::header::HeaderMap { + let mut headers = reqwest::header::HeaderMap::new(); + if let Some(object) = request_headers.as_object() { + for name in FORWARD_REQUEST_HEADERS { + if let Some(value) = object.get(*name).and_then(Value::as_str) { + if let (Ok(name), Ok(value)) = ( + reqwest::header::HeaderName::from_bytes(name.as_bytes()), + reqwest::header::HeaderValue::from_str(value), + ) { + headers.insert(name, value); + } + } + } + } + headers +} + +fn content_type_is_json(content_type: &str) -> bool { + content_type.to_ascii_lowercase().contains("json") +} + +fn apply_json_array_limits(body_text: String, content_type: &str, limits: &Value) -> String { + let Some(limits) = limits.as_object() else { + return body_text; + }; + if limits.is_empty() || !content_type_is_json(content_type) { + return body_text; + } + let Ok(mut parsed) = serde_json::from_str::<Value>(&body_text) else { + return body_text; + }; + let Some(root) = parsed.as_object_mut() else { + return body_text; + }; + let mut applied = serde_json::Map::new(); + for (path, limit) in limits { + let limit = limit.as_u64().unwrap_or(0) as usize; + if limit == 0 || limit > 500 { + continue; + } + if let Some(array) = array_at_path_mut(root, path) { + let original_length = array.len(); + if array.len() > limit { + array.truncate(limit); + } + applied.insert(path.clone(), json!({ "limit": limit, "originalLength": original_length, "returnedLength": array.len() })); + } + } + root.insert("_unidesk".to_string(), json!({ "arrayLimits": applied })); + serde_json::to_string(&parsed).unwrap_or(body_text) +} + +fn array_at_path_mut<'a>( + root: &'a mut serde_json::Map<String, Value>, + path: &str, +) -> Option<&'a mut Vec<Value>> { + let mut current = root; + let mut parts = path.split('.').peekable(); + while let Some(part) = parts.next() { + if parts.peek().is_none() { + return current.get_mut(part).and_then(Value::as_array_mut); + } + current = current.get_mut(part)?.as_object_mut()?; + } + None +} + +fn bounded_microservice_body_text( + body_text: String, + content_type: &str, + metadata: Value, +) -> (String, bool) { + if body_text.len() <= MICROSERVICE_PROXY_MAX_BODY_TEXT_LENGTH { + return (body_text, false); + } + if content_type_is_json(content_type) { + return (json!({ + "ok": false, + "error": "microservice proxy response body is too large", + "serviceId": metadata.get("serviceId").cloned().unwrap_or(Value::Null), + "targetPath": metadata.get("targetPath").cloned().unwrap_or(Value::Null), + "upstreamStatus": metadata.get("status").cloned().unwrap_or(Value::Null), + "upstreamBodyBytes": metadata.get("upstreamBodyBytes").cloned().unwrap_or(Value::Null), + "transformedBodyBytes": body_text.len(), + "responseBodyLimitBytes": MICROSERVICE_PROXY_MAX_BODY_TEXT_LENGTH, + "hint": "Use a paged endpoint or tighten __unideskArrayLimit so the response stays below the proxy safety limit." + }).to_string(), true); + } + ( + truncate_text(&body_text, MICROSERVICE_PROXY_MAX_BODY_TEXT_LENGTH), + true, + ) +} + +fn response_from_provider_microservice_result(result: Value, proxy_mode: &str) -> Response { + let status = result.get("status").and_then(Value::as_u64).unwrap_or(0); + let content_type = result + .get("contentType") + .and_then(Value::as_str) + .unwrap_or("application/json; charset=utf-8"); + let body_text = result + .get("bodyText") + .and_then(Value::as_str) + .unwrap_or("") + .to_string(); + if !(100..=599).contains(&status) { + return json_response( + json!({ "ok": false, "error": "microservice proxy returned invalid upstream status", "result": result }), + 502, + ); + } + let mut response = response_with_body(status as u16, content_type, body_text); + add_header(&mut response, "x-unidesk-proxy-mode", proxy_mode); + add_header( + &mut response, + "x-unidesk-upstream-proxy-mode", + result + .get("proxyMode") + .and_then(Value::as_str) + .unwrap_or(""), + ); + add_header( + &mut response, + "x-unidesk-response-truncated", + if result.get("truncated").and_then(Value::as_bool) == Some(true) { + "true" + } else { + "false" + }, + ); + response +} + +async fn direct_microservice_response( + service: &MicroserviceConfig, + method: &Method, + target_path: &str, + proxy_options: &ProxyOptions, + request_headers: &Value, + body_text: String, +) -> Response { + let base = match reqwest::Url::parse(&service.backend.node_base_url) { + Ok(url) => url, + Err(error) => { + return json_response( + json!({ "ok": false, "error": "invalid microservice base URL", "serviceId": service.id, "detail": error.to_string() }), + 502, + ) + } + }; + let mut upstream = match base.join(target_path.trim_start_matches('/')) { + Ok(url) => url, + Err(error) => { + return json_response( + json!({ "ok": false, "error": "invalid microservice target URL", "serviceId": service.id, "detail": error.to_string() }), + 502, + ) + } + }; + upstream.set_query(proxy_options.query.strip_prefix('?')); + let client = reqwest::Client::new(); + let mut request = client + .request( + method.as_str().parse().unwrap_or(reqwest::Method::GET), + upstream, + ) + .headers(headers_from_microservice_request(request_headers)); + if method != Method::GET && method != Method::HEAD { + request = request.body(body_text); + } + let result = tokio::time::timeout( + Duration::from_millis(service.backend.timeout_ms.max(1000)), + request.send(), + ) + .await; + let response = match result { + Ok(Ok(response)) => response, + Ok(Err(error)) => { + return json_response( + json!({ "ok": false, "error": "direct microservice proxy failed", "serviceId": service.id, "detail": error.to_string() }), + 502, + ) + } + Err(_) => { + return json_response( + json!({ "ok": false, "error": "direct microservice proxy timed out", "serviceId": service.id, "timeoutMs": service.backend.timeout_ms }), + 504, + ) + } + }; + let status = response.status().as_u16(); + let content_type = response + .headers() + .get(reqwest::header::CONTENT_TYPE) + .and_then(|value| value.to_str().ok()) + .unwrap_or("text/plain; charset=utf-8") + .to_string(); + let raw_body = match response.text().await { + Ok(text) => text, + Err(error) => { + return json_response( + json!({ "ok": false, "error": "direct microservice body read failed", "serviceId": service.id, "detail": error.to_string() }), + 502, + ) + } + }; + let limited = apply_json_array_limits( + raw_body.clone(), + &content_type, + &proxy_options.json_array_limits, + ); + let (body_text, truncated) = bounded_microservice_body_text( + limited, + &content_type, + json!({ "serviceId": service.id, "targetPath": target_path, "status": status, "upstreamBodyBytes": raw_body.len() }), + ); + let mut response = response_with_body(status, &content_type, body_text); + add_header(&mut response, "x-unidesk-proxy-mode", "direct"); + add_header( + &mut response, + "x-unidesk-response-truncated", + if truncated { "true" } else { "false" }, + ); + response +} + +async fn k3sctl_adapter_microservice_response( + state: &Arc<AppState>, + service: &MicroserviceConfig, + method: &Method, + target_path: &str, + proxy_options: &ProxyOptions, + request_headers: &Value, + body_text: String, +) -> Response { + let adapter_service_id = service + .deployment + .adapter_service_id + .as_deref() + .unwrap_or("k3sctl-adapter"); + let Some(adapter) = service_by_id(state, adapter_service_id) else { + return json_response( + json!({ "ok": false, "error": format!("k3sctl adapter microservice not found: {adapter_service_id}"), "serviceId": service.id }), + 502, + ); + }; + if adapter.id == service.id || is_k3sctl_managed_microservice(&adapter) { + return json_response( + json!({ "ok": false, "error": "k3sctl adapter must be a UniDesk-direct microservice", "serviceId": service.id, "adapterServiceId": adapter_service_id }), + 500, + ); + } + let k3s_service_id = if service.id == "code-queue" { + code_queue_k3s_service_id_for_request(method, target_path).to_string() + } else { + service + .deployment + .k3s_service_id + .clone() + .unwrap_or_else(|| service.id.clone()) + }; + let adapter_target_path = format!( + "/api/services/{}/proxy{}", + urlencoding_like(&k3s_service_id), + target_path + ); + Box::pin(fetch_microservice_upstream_response( + state, + &adapter, + method, + &adapter_target_path, + proxy_options, + request_headers, + body_text, + )) + .await +} + +async fn code_queue_k3s_scheduler_response( + state: &Arc<AppState>, + service: &MicroserviceConfig, + method: &Method, + target_path: &str, + proxy_options: &ProxyOptions, + request_headers: &Value, + body_text: String, +) -> Response { + let mut scheduler_service = service.clone(); + scheduler_service.deployment.k3s_service_id = Some("code-queue-scheduler".to_string()); + k3sctl_adapter_microservice_response( + state, + &scheduler_service, + method, + target_path, + proxy_options, + request_headers, + body_text, + ) + .await +} + +fn provider_http_tunnel_request_id(provider_id: &str) -> String { + format!( + "{provider_id}:http_{}_{}", + chrono::Utc::now().timestamp_millis(), + rand::random::<u64>() + ) +} + +fn can_retry_provider_http_tunnel(method: &Method, target_path: &str) -> bool { + (method == Method::GET || method == Method::HEAD) && !target_path.ends_with("/stream") +} + +fn tunnel_error_body( + service: &MicroserviceConfig, + request_id: &str, + error: &str, + stage: &str, + status: u16, + extra: Value, +) -> Response { + let mut object = serde_json::Map::new(); + object.insert("ok".to_string(), Value::Bool(false)); + object.insert("error".to_string(), Value::String(error.to_string())); + object.insert("stage".to_string(), Value::String(stage.to_string())); + object.insert( + "providerId".to_string(), + Value::String(service.provider_id.clone()), + ); + object.insert("serviceId".to_string(), Value::String(service.id.clone())); + object.insert( + "requestId".to_string(), + Value::String(request_id.to_string()), + ); + if let Some(extra) = extra.as_object() { + for (key, value) in extra { + object.insert(key.clone(), value.clone()); + } + } + let mut response = json_response(Value::Object(object), status); + add_header(&mut response, "x-unidesk-request-id", request_id); + add_header(&mut response, "x-unidesk-provider-id", &service.provider_id); + add_header(&mut response, "x-unidesk-service-id", &service.id); + add_header(&mut response, "x-unidesk-tunnel-error", stage); + if matches!(status, 502 | 503 | 504) { + add_header(&mut response, "x-unidesk-transient-error", "true"); + } + response +} + +async fn wait_for_provider_http_tunnel_response( + state: &Arc<AppState>, + provider_id: &str, + request_id: &str, + timeout_ms: u64, +) -> Result<HttpTunnelResponse, String> { + let (tx, rx) = tokio::sync::oneshot::channel(); + state + .http_tunnel_waiters + .lock() + .await + .insert(request_id.to_string(), tx); + match tokio::time::timeout(Duration::from_millis(timeout_ms.max(1)), rx).await { + Ok(Ok(Ok(message))) if message.provider_id == provider_id => Ok(message), + Ok(Ok(Ok(_))) => Err("provider-mismatch".to_string()), + Ok(Ok(Err(reason))) => Err(reason), + Ok(Err(_)) => Err("aborted".to_string()), + Err(_) => { + state.http_tunnel_waiters.lock().await.remove(request_id); + Err("timeout".to_string()) + } + } +} + +async fn provider_http_tunnel_microservice_response( + state: &Arc<AppState>, + service: &MicroserviceConfig, + method: &Method, + target_path: &str, + proxy_options: &ProxyOptions, + request_headers: &Value, + body_text: String, +) -> Response { + let retryable = can_retry_provider_http_tunnel(method, target_path); + let max_attempts = if retryable { + PROVIDER_HTTP_TUNNEL_MAX_ATTEMPTS + } else { + 1 + }; + let mut attempts = Vec::<Value>::new(); + let mut last_request_id = String::new(); + for attempt in 1..=max_attempts { + let request_id = provider_http_tunnel_request_id(&service.provider_id); + last_request_id = request_id.clone(); + let provider = { + state + .active_providers + .read() + .await + .get(&service.provider_id) + .cloned() + }; + let Some(provider) = provider else { + attempts.push(json!({ "attempt": attempt, "requestId": request_id, "ok": false, "reason": "provider-offline" })); + return tunnel_error_body( + service, + &request_id, + &format!("provider is offline: {}", service.provider_id), + "provider-gateway-online", + 503, + json!({ "retryable": retryable, "attempts": attempts }), + ); + }; + let timeout_ms = service.backend.timeout_ms.max(1000) + 3000; + let started = millis_now(); + let payload = json!({ + "type": "http_tunnel_request", + "requestId": request_id, + "payload": { + "source": "microservice-frontend-proxy", + "serviceId": service.id, + "method": method.as_str(), + "targetBaseUrl": service.backend.node_base_url, + "path": target_path, + "query": proxy_options.query, + "requestHeaders": request_headers, + "bodyText": body_text, + "jsonArrayLimits": proxy_options.json_array_limits, + "timeoutMs": service.backend.timeout_ms, + "cacheTtlMs": provider_microservice_cache_ttl_ms(&service.id, target_path), + } + }); + if provider + .sender + .send(axum::extract::ws::Message::Text(payload.to_string())) + .is_err() + { + attempts.push(json!({ "attempt": attempt, "requestId": request_id, "ok": false, "reason": "send-failed", "durationMs": millis_now() - started })); + if attempt < max_attempts { + tokio::time::sleep(Duration::from_millis((100 * attempt) as u64)).await; + continue; + } + return tunnel_error_body( + service, + &request_id, + "provider HTTP tunnel send failed", + "http-tunnel-send", + 502, + json!({ "retryable": retryable, "attempts": attempts }), + ); + } + match wait_for_provider_http_tunnel_response( + state, + &service.provider_id, + &request_id, + timeout_ms, + ) + .await + { + Ok(message) if message.ok => { + attempts.push(json!({ "attempt": attempt, "requestId": request_id, "ok": true, "durationMs": millis_now() - started })); + let mut response = response_from_provider_microservice_result( + message.result, + "provider-ws-http-tunnel", + ); + add_header(&mut response, "x-unidesk-request-id", &request_id); + add_header( + &mut response, + "x-unidesk-http-tunnel-attempt", + &attempt.to_string(), + ); + add_header( + &mut response, + "x-unidesk-http-tunnel-attempts", + &attempts.len().to_string(), + ); + add_header(&mut response, "x-unidesk-provider-id", &service.provider_id); + return response; + } + Ok(message) => { + attempts.push(json!({ "attempt": attempt, "requestId": request_id, "ok": false, "durationMs": millis_now() - started, "result": compact_json(&message.result) })); + if retryable && attempt < max_attempts { + tokio::time::sleep(Duration::from_millis((100 * attempt) as u64)).await; + continue; + } + return tunnel_error_body( + service, + &request_id, + "provider HTTP tunnel failed", + "provider-gateway-http-fetch", + 502, + json!({ "retryable": retryable, "attempts": attempts, "result": message.result }), + ); + } + Err(reason) => { + attempts.push(json!({ "attempt": attempt, "requestId": request_id, "ok": false, "reason": reason, "durationMs": millis_now() - started, "timeoutMs": timeout_ms })); + if retryable + && attempt < max_attempts + && matches!( + reason.as_str(), + "timeout" | "provider-disconnected" | "send-failed" + ) + { + tokio::time::sleep(Duration::from_millis((100 * attempt) as u64)).await; + continue; + } + let status = match reason.as_str() { + "aborted" => 499, + "provider-disconnected" => 503, + "send-failed" => 502, + _ => 504, + }; + return tunnel_error_body( + service, + &request_id, + "provider HTTP tunnel timed out or disconnected", + if reason == "aborted" { + "client-aborted" + } else { + "http-tunnel-wait" + }, + status, + json!({ "retryable": retryable, "attempts": attempts, "timeoutMs": timeout_ms, "failureReason": reason }), + ); + } + } + } + tunnel_error_body( + service, + &last_request_id, + "provider HTTP tunnel exhausted attempts", + "http-tunnel-wait", + 504, + json!({ "retryable": retryable, "attempts": attempts }), + ) +} + +async fn fetch_microservice_upstream_response( + state: &Arc<AppState>, + service: &MicroserviceConfig, + method: &Method, + target_path: &str, + proxy_options: &ProxyOptions, + request_headers: &Value, + body_text: String, +) -> Response { + if service.id == "code-queue" { + if target_path == "/health" || target_path == "/" { + return direct_microservice_response( + &direct_code_queue_mgr_service(service, None), + method, + target_path, + proxy_options, + request_headers, + body_text, + ) + .await; + } + if path_matches_task_action(target_path, "interrupt") { + if let Some(task_id) = code_queue_task_id_for_action(target_path, "interrupt") { + let meta = direct_microservice_response( + &direct_code_queue_mgr_service(service, Some(3000)), + &Method::GET, + &format!("/api/tasks/{}", urlencoding_like(&task_id)), + &ProxyOptions { + query: "?meta=1".to_string(), + json_array_limits: json!({}), + }, + &json!({ "accept": "application/json" }), + String::new(), + ) + .await; + if meta.status().is_success() { + if let Ok((_, _, text)) = response_to_text(meta).await { + if let Ok(body) = serde_json::from_str::<Value>(&text) { + let status = body + .get("status") + .or_else(|| body.pointer("/task/status")) + .and_then(Value::as_str) + .unwrap_or(""); + if status == "queued" || status == "retry_wait" { + return direct_microservice_response( + &direct_code_queue_mgr_service(service, None), + method, + target_path, + proxy_options, + request_headers, + body_text, + ) + .await; + } + } + } + } + } + return code_queue_k3s_scheduler_response( + state, + service, + method, + target_path, + proxy_options, + request_headers, + body_text, + ) + .await; + } + if code_queue_scheduler_only_path(method, target_path) { + return code_queue_k3s_scheduler_response( + state, + service, + method, + target_path, + proxy_options, + request_headers, + body_text, + ) + .await; + } + return direct_microservice_response( + &direct_code_queue_mgr_service(service, None), + method, + target_path, + proxy_options, + request_headers, + body_text, + ) + .await; + } + if is_k3sctl_managed_microservice(service) { + return k3sctl_adapter_microservice_response( + state, + service, + method, + target_path, + proxy_options, + request_headers, + body_text, + ) + .await; + } + if can_direct_proxy_microservice(service) { + return direct_microservice_response( + service, + method, + target_path, + proxy_options, + request_headers, + body_text, + ) + .await; + } + if !provider_supports(state, &service.provider_id, "microservice.http") + .await + .unwrap_or(false) + { + return json_response( + json!({ "ok": false, "error": format!("provider does not declare microservice.http capability: {}", service.provider_id) }), + 409, + ); + } + if provider_supports(state, &service.provider_id, "microservice.http.tunnel") + .await + .unwrap_or(false) + { + return provider_http_tunnel_microservice_response( + state, + service, + method, + target_path, + proxy_options, + request_headers, + body_text, + ) + .await; + } + let payload = json!({ + "source": "microservice-frontend-proxy", + "serviceId": service.id, + "method": method.as_str(), + "targetBaseUrl": service.backend.node_base_url, + "path": target_path, + "query": proxy_options.query, + "requestHeaders": request_headers, + "bodyText": body_text, + "jsonArrayLimits": proxy_options.json_array_limits, + "timeoutMs": service.backend.timeout_ms, + "cacheTtlMs": provider_microservice_cache_ttl_ms(&service.id, target_path), + }); + match create_and_send_task(state, &service.provider_id, "microservice.http", payload).await { + Ok((task_id, true)) => { + match wait_for_task_terminal(state, &task_id, service.backend.timeout_ms + 3000).await { + Ok(Some(task)) if task.status == "succeeded" => { + response_from_provider_microservice_result( + task.result.unwrap_or(Value::Null), + "provider-task", + ) + } + Ok(task) => json_response( + json!({ "ok": false, "error": "microservice proxy task failed", "task": task.map(|task| json!({ "id": task.id, "status": task.status, "result": task.result })) }), + 502, + ), + Err(error) => json_response( + json!({ "ok": false, "error": error.to_string(), "taskId": task_id }), + 502, + ), + } + } + Ok((task_id, false)) => json_response( + json!({ "ok": false, "error": format!("provider is offline: {}", service.provider_id), "taskId": task_id }), + 503, + ), + Err(error) => json_response(json!({ "ok": false, "error": error.to_string() }), 500), + } +} + +fn microservice_cache_key( + service: &MicroserviceConfig, + method: &Method, + target_path: &str, + proxy_options: &ProxyOptions, +) -> String { + json!([ + service.id, + method.as_str(), + target_path, + proxy_options.query, + proxy_options.json_array_limits + ]) + .to_string() +} + +fn microservice_path_fallback_cache_key( + service: &MicroserviceConfig, + method: &Method, + target_path: &str, +) -> String { + json!([ + service.id, + method.as_str(), + target_path, + "__path_fallback__" + ]) + .to_string() +} + +fn can_use_microservice_path_fallback( + service: &MicroserviceConfig, + method: &Method, + target_path: &str, +) -> bool { + service.id == "code-queue" + && target_path == "/api/tasks/overview" + && (method == Method::GET || method == Method::HEAD) +} + +fn response_from_microservice_cache(entry: &MicroserviceProxyCacheEntry, state: &str) -> Response { + let mut response = + response_with_body(entry.status, &entry.content_type, entry.body_text.clone()); + add_header(&mut response, "x-unidesk-cache", state); + response +} + +async fn read_microservice_cache(state: &Arc<AppState>, key: &str) -> Option<Response> { + let now = millis_now(); + let mut cache = state.microservice_proxy_cache.lock().await; + let entry = cache.get(key)?.clone(); + if entry.stale_expires_at_ms <= now { + cache.remove(key); + return None; + } + if entry.expires_at_ms <= now { + return None; + } + Some(response_from_microservice_cache(&entry, "hit")) +} + +async fn read_stale_microservice_cache(state: &Arc<AppState>, key: &str) -> Option<Response> { + let now = millis_now(); + let mut cache = state.microservice_proxy_cache.lock().await; + let entry = cache.get(key)?.clone(); + if entry.stale_expires_at_ms <= now { + cache.remove(key); + return None; + } + if entry.expires_at_ms > now { + return None; + } + Some(response_from_microservice_cache(&entry, "stale")) +} + +async fn read_microservice_path_fallback( + state: &Arc<AppState>, + service: &MicroserviceConfig, + method: &Method, + target_path: &str, +) -> Option<Response> { + if !can_use_microservice_path_fallback(service, method, target_path) { + return None; + } + let key = microservice_path_fallback_cache_key(service, method, target_path); + let now = millis_now(); + let mut cache = state.microservice_proxy_cache.lock().await; + let entry = cache.get(&key)?.clone(); + if entry.stale_expires_at_ms <= now { + cache.remove(&key); + return None; + } + Some(response_from_microservice_cache(&entry, "path-stale")) +} + +async fn cacheable_response_snapshot( + response: Response, +) -> (Response, Option<MicroserviceProxyCacheEntry>) { + let status = response.status().as_u16(); + let headers = response.headers().clone(); + let content_type = headers + .get(header::CONTENT_TYPE) + .and_then(|value| value.to_str().ok()) + .unwrap_or("application/octet-stream") + .to_string(); + let bytes = to_bytes(response.into_body(), 16 * 1024 * 1024) + .await + .unwrap_or_default(); + let body_text = String::from_utf8_lossy(&bytes).to_string(); + let rebuilt = response_with_body(status, &content_type, body_text.clone()); + if !(200..300).contains(&status) + || headers + .get("x-unidesk-response-truncated") + .and_then(|value| value.to_str().ok()) + == Some("true") + || body_text.len() > 2 * 1024 * 1024 + { + return (rebuilt, None); + } + ( + rebuilt, + Some(MicroserviceProxyCacheEntry { + expires_at_ms: 0, + stale_expires_at_ms: 0, + status, + content_type, + body_text, + }), + ) +} + +async fn remember_microservice_cache( + state: &Arc<AppState>, + key: String, + ttl_ms: i64, + mut entry: Option<MicroserviceProxyCacheEntry>, +) { + if ttl_ms <= 0 { + return; + } + let Some(ref mut entry) = entry else { return }; + let parsed: Value = serde_json::from_str(&key).unwrap_or(Value::Null); + let service_id = parsed.get(0).and_then(Value::as_str).unwrap_or(""); + let target_path = parsed.get(2).and_then(Value::as_str).unwrap_or(""); + let now = millis_now(); + entry.expires_at_ms = now + ttl_ms; + entry.stale_expires_at_ms = + entry.expires_at_ms + microservice_cache_stale_ms(service_id, target_path); + let mut cache = state.microservice_proxy_cache.lock().await; + cache.insert(key, entry.clone()); + if cache.len() > 300 { + let now = millis_now(); + cache.retain(|_, entry| entry.expires_at_ms > now); + if cache.len() > 240 { + cache.clear(); + } + } +} + +async fn remember_microservice_path_fallback( + state: &Arc<AppState>, + service: &MicroserviceConfig, + method: &Method, + target_path: &str, + entry: Option<MicroserviceProxyCacheEntry>, +) { + if !can_use_microservice_path_fallback(service, method, target_path) { + return; + } + let Some(mut entry) = entry else { return }; + let now = millis_now(); + entry.expires_at_ms = now; + entry.stale_expires_at_ms = now + CODE_QUEUE_OVERVIEW_PATH_FALLBACK_STALE_MS; + state.microservice_proxy_cache.lock().await.insert( + microservice_path_fallback_cache_key(service, method, target_path), + entry, + ); +} + +async fn invalidate_microservice_cache(state: &Arc<AppState>, service_id: &str) { + let prefix = format!("[\"{service_id}\","); + state + .microservice_proxy_cache + .lock() + .await + .retain(|key, _| !key.starts_with(&prefix)); +} + +fn is_microservice_transient_failure_response(response: &Response) -> bool { + matches!(response.status().as_u16(), 502 | 503 | 504) + && (response + .headers() + .get("x-unidesk-transient-error") + .is_some() + || response.headers().get("x-unidesk-tunnel-error").is_some() + || response + .headers() + .get("x-unidesk-upstream-proxy-mode") + .and_then(|value| value.to_str().ok()) + == Some("provider-gateway-http-fetch")) +} + +fn json_body_from_text(body_text: &str, content_type: &str) -> Value { + if !content_type_is_json(content_type) { + return if body_text.is_empty() { + Value::Null + } else { + Value::String(truncate_text(body_text, 4000)) + }; + } + serde_json::from_str::<Value>(body_text) + .map(|value| compact_json(&value)) + .unwrap_or_else(|_| Value::String(truncate_text(body_text, 4000))) +} + +fn generic_microservice_health_assessment( + upstream_status: u16, + body: &Value, +) -> (bool, String, Value) { + let mut checks = serde_json::Map::new(); + checks.insert("upstreamStatus".to_string(), json!(upstream_status)); + checks.insert( + "upstream2xx".to_string(), + json!((200..300).contains(&upstream_status)), + ); + if !(200..300).contains(&upstream_status) { + return ( + false, + format!("health endpoint returned HTTP {upstream_status}"), + Value::Object(checks), + ); + } + if let Some(object) = body.as_object() { + let ok = object.get("ok").and_then(Value::as_bool); + let success = object.get("success").and_then(Value::as_bool); + let status = object.get("status").and_then(Value::as_str).unwrap_or(""); + checks.insert( + "okField".to_string(), + ok.map(Value::Bool).unwrap_or(Value::Null), + ); + checks.insert( + "successField".to_string(), + success.map(Value::Bool).unwrap_or(Value::Null), + ); + checks.insert( + "statusField".to_string(), + if status.is_empty() { + Value::Null + } else { + Value::String(status.to_string()) + }, + ); + if ok == Some(false) { + return ( + false, + "health body ok=false".to_string(), + Value::Object(checks), + ); + } + if success == Some(false) { + return ( + false, + "health body success=false".to_string(), + Value::Object(checks), + ); + } + if matches!( + status.to_ascii_lowercase().as_str(), + "error" + | "failed" + | "fail" + | "unhealthy" + | "down" + | "offline" + | "not_ready" + | "not-ready" + ) { + return ( + false, + format!("health body status={status}"), + Value::Object(checks), + ); + } + } + ( + true, + "health endpoint returned a usable service state".to_string(), + Value::Object(checks), + ) +} + +fn health_probe_from_assessment( + service: &MicroserviceConfig, + upstream_status: u16, + content_type: &str, + healthy: bool, + reason: String, + checks: Value, + body: Option<Value>, +) -> Value { + let mut probe = json!({ + "serviceId": service.id, + "name": service.name, + "providerId": service.provider_id, + "healthPath": service.backend.health_path, + "healthy": healthy, + "status": if healthy { "healthy" } else { "unhealthy" }, + "reason": reason, + "upstreamStatus": upstream_status, + "contentType": content_type, + "checks": checks, + "checkedAt": crate::json_util::now_iso(), + }); + if let Some(body) = body { + probe["body"] = body; + } + probe +} + +async fn remember_microservice_availability(state: &Arc<AppState>, service_id: &str, probe: Value) { + state.microservice_availability_cache.lock().await.insert( + service_id.to_string(), + (millis_now() + MICROSERVICE_AVAILABILITY_TTL_MS, probe), + ); +} + +async fn cached_microservice_availability( + state: &Arc<AppState>, + service_id: &str, +) -> Option<Value> { + let now = millis_now(); + let mut cache = state.microservice_availability_cache.lock().await; + let (expires, probe) = cache.get(service_id)?.clone(); + if expires <= now { + cache.remove(service_id); + None + } else { + Some(probe) + } +} + +pub async fn strict_microservice_health_probe( + state: &Arc<AppState>, + service: &MicroserviceConfig, + timeout_ms: Option<u64>, +) -> Value { + let mut probe_service = service.clone(); + if let Some(timeout_ms) = timeout_ms { + probe_service.backend.timeout_ms = timeout_ms; + } + let response = if service.id == "code-queue" { + code_queue_health_response(state, &probe_service, false).await + } else { + fetch_microservice_upstream_response( + state, + &probe_service, + &Method::GET, + &service.backend.health_path, + &ProxyOptions { + query: String::new(), + json_array_limits: json!({}), + }, + &json!({ "accept": "application/json" }), + String::new(), + ) + .await + }; + let (status, content_type, body_text) = response_to_text(response).await.unwrap_or(( + 502, + "application/json".to_string(), + "{}".to_string(), + )); + let body = json_body_from_text(&body_text, &content_type); + let (healthy, reason, checks) = generic_microservice_health_assessment(status, &body); + let probe = health_probe_from_assessment( + service, + status, + &content_type, + healthy, + reason, + checks, + Some(body), + ); + remember_microservice_availability(state, &service.id, probe.clone()).await; + probe +} + +async fn microservice_availability(state: &Arc<AppState>, service: &MicroserviceConfig) -> Value { + if let Some(cached) = cached_microservice_availability(state, &service.id).await { + return cached; + } + let timeout_ms = service.backend.timeout_ms.clamp(2500, 10_000); + strict_microservice_health_probe(state, service, Some(timeout_ms)).await +} + +async fn code_queue_health_response( + state: &Arc<AppState>, + service: &MicroserviceConfig, + head_only: bool, +) -> Response { + let mgr_response = direct_microservice_response( + &direct_code_queue_mgr_service(service, Some(4000)), + &Method::GET, + "/health", + &ProxyOptions { + query: String::new(), + json_array_limits: json!({}), + }, + &json!({ "accept": "application/json" }), + String::new(), + ) + .await; + let (mgr_status, _, mgr_text) = response_to_text(mgr_response).await.unwrap_or(( + 502, + "application/json".to_string(), + "{}".to_string(), + )); + let mgr_body: Value = serde_json::from_str(&mgr_text) + .unwrap_or_else(|_| json!({ "text": truncate_text(&mgr_text, 4000) })); + let trace_body = mgr_body + .get("traceRead") + .cloned() + .unwrap_or_else(|| json!({})); + let mgr_healthy = (200..300).contains(&mgr_status) + && mgr_body.get("ok").and_then(Value::as_bool) != Some(false); + + let scheduler_response = code_queue_k3s_scheduler_response( + state, + service, + &Method::GET, + "/health", + &ProxyOptions { + query: String::new(), + json_array_limits: json!({}), + }, + &json!({ "accept": "application/json" }), + String::new(), + ) + .await; + let (scheduler_status, _, scheduler_text) = response_to_text(scheduler_response) + .await + .unwrap_or((504, "application/json".to_string(), "{}".to_string())); + let scheduler_body: Value = serde_json::from_str(&scheduler_text) + .unwrap_or_else(|_| json!({ "text": truncate_text(&scheduler_text, 4000) })); + let scheduler_healthy = (200..300).contains(&scheduler_status) + && scheduler_body.get("ok").and_then(Value::as_bool) != Some(false); + let status = if mgr_healthy { 200 } else { 503 }; + let body = json!({ + "ok": mgr_healthy, + "service": "code-queue", + "role": "hybrid-master-control-plane", + "checkedAt": crate::json_util::now_iso(), + "manager": { "ok": mgr_healthy, "status": mgr_status, "body": mgr_body, "timedOut": false }, + "trace": { "ok": mgr_healthy, "source": "code-queue-mgr", "readonlyUserConfigured": trace_body.get("readonlyUserConfigured").cloned().unwrap_or(Value::Null), "body": trace_body }, + "scheduler": { "ok": scheduler_healthy, "status": scheduler_status, "body": scheduler_body, "timedOut": false, "requiredFor": ["running task steer", "running task interrupt", "scheduler claim/runner execution", "dev-container control"] }, + }); + if head_only { + response_with_body(status, "application/json; charset=utf-8", Body::empty()) + } else { + json_response(body, status) + } +} + +async fn strict_microservice_health_response( + state: &Arc<AppState>, + service: &MicroserviceConfig, + head_only: bool, +) -> Response { + if service.id == "code-queue" { + return code_queue_health_response(state, service, head_only).await; + } + let response = fetch_microservice_upstream_response( + state, + service, + &Method::GET, + &service.backend.health_path, + &ProxyOptions { + query: String::new(), + json_array_limits: json!({}), + }, + &json!({ "accept": "application/json" }), + String::new(), + ) + .await; + let (upstream_status, content_type, body_text) = response_to_text(response).await.unwrap_or(( + 502, + "application/json".to_string(), + "{}".to_string(), + )); + let body = json_body_from_text(&body_text, &content_type); + let (healthy, reason, checks) = generic_microservice_health_assessment(upstream_status, &body); + let probe = health_probe_from_assessment( + service, + upstream_status, + &content_type, + healthy, + reason.clone(), + checks.clone(), + None, + ); + remember_microservice_availability(state, &service.id, probe).await; + if healthy { + let mut response = response_with_body( + upstream_status, + &content_type, + if head_only { String::new() } else { body_text }, + ); + add_header(&mut response, "x-unidesk-health", "healthy"); + add_header(&mut response, "x-unidesk-health-reason", &reason); + return response; + } + json_response( + json!({ + "ok": false, + "status": "unhealthy", + "serviceId": service.id, + "name": service.name, + "providerId": service.provider_id, + "reason": reason, + "checkedAt": crate::json_util::now_iso(), + "checks": checks, + "upstream": { "status": upstream_status, "contentType": content_type, "body": body }, + }), + if upstream_status >= 500 { + upstream_status + } else { + 503 + }, + ) +} + +async fn k3sctl_managed_diagnostics_response( + state: &Arc<AppState>, + service: &MicroserviceConfig, +) -> Response { + let adapter_service_id = service + .deployment + .adapter_service_id + .as_deref() + .unwrap_or("k3sctl-adapter"); + let adapter = service_by_id(state, adapter_service_id); + let provider_id = adapter + .as_ref() + .map(|service| service.provider_id.clone()) + .unwrap_or_else(|| service.provider_id.clone()); + let provider_online = state + .active_providers + .read() + .await + .contains_key(&provider_id); + let provider_tunnel_capable = + provider_supports(state, &provider_id, "microservice.http.tunnel") + .await + .unwrap_or(false); + let Some(adapter) = adapter else { + return json_response( + json!({ + "ok": false, + "serviceId": service.id, + "checkedAt": crate::json_util::now_iso(), + "requestPath": "/diagnostics", + "checks": { + "providerGateway": { "ok": provider_online, "providerId": provider_id, "online": provider_online }, + "httpTunnel": { "ok": provider_tunnel_capable, "providerId": provider_id, "capable": provider_tunnel_capable }, + "k3sctlAdapter": { "ok": false, "serviceId": adapter_service_id, "error": format!("k3sctl adapter microservice not found: {adapter_service_id}") }, + "kubernetesApiServiceProxy": { "ok": false, "skipped": true }, + "targetService": { "ok": false, "skipped": true } + } + }), + 502, + ); + }; + let k3s_service_id = if service.id == "code-queue" { + code_queue_k3s_service_id_for_request(&Method::GET, &service.backend.health_path) + .to_string() + } else { + service + .deployment + .k3s_service_id + .clone() + .unwrap_or_else(|| service.id.clone()) + }; + let adapter_path = format!( + "/api/services/{}/diagnostics", + urlencoding_like(&k3s_service_id) + ); + let response = fetch_microservice_upstream_response( + state, + &adapter, + &Method::GET, + &adapter_path, + &ProxyOptions { + query: String::new(), + json_array_limits: json!({}), + }, + &json!({ "accept": "application/json" }), + String::new(), + ) + .await; + let status = response.status().as_u16(); + let proxy_mode = response + .headers() + .get("x-unidesk-proxy-mode") + .and_then(|value| value.to_str().ok()) + .unwrap_or("") + .to_string(); + let request_id = response + .headers() + .get("x-unidesk-request-id") + .and_then(|value| value.to_str().ok()) + .map(ToOwned::to_owned); + let (status2, content_type, body_text) = response_to_text(response).await.unwrap_or(( + status, + "application/json".to_string(), + "{}".to_string(), + )); + let adapter_body: Value = serde_json::from_str(&body_text) + .unwrap_or_else(|_| Value::String(truncate_text(&body_text, 4000))); + let adapter_checks = adapter_body + .get("checks") + .cloned() + .unwrap_or_else(|| json!({})); + let ok = + (200..300).contains(&status2) && provider_online && proxy_mode == "provider-ws-http-tunnel"; + json_response( + json!({ + "ok": ok, + "serviceId": service.id, + "k3sServiceId": k3s_service_id, + "checkedAt": crate::json_util::now_iso(), + "path": service.backend.health_path, + "chain": "CLI/frontend -> backend-core -> provider-gateway HTTP tunnel -> k3sctl-adapter -> Kubernetes API service proxy -> k3s Service", + "checks": { + "providerGateway": { "ok": provider_online, "providerId": provider_id, "online": provider_online, "activeSocketCount": state.active_provider_count().await }, + "httpTunnel": { "ok": proxy_mode == "provider-ws-http-tunnel", "providerId": provider_id, "capable": provider_tunnel_capable, "requestId": request_id, "proxyStatus": status2 }, + "k3sctlAdapter": { "ok": (200..300).contains(&status2), "serviceId": adapter.id, "providerId": adapter.provider_id, "status": status2, "contentType": content_type }, + "kubernetesApiServiceProxy": compact_json(adapter_checks.get("kubernetesApiServiceProxy").unwrap_or(&json!({ "ok": false, "skipped": true }))), + "targetService": compact_json(adapter_checks.get("targetService").or_else(|| adapter_checks.get("managedService")).unwrap_or(&json!({ "ok": false, "skipped": true }))), + }, + "adapter": adapter_body, + }), + if (200..300).contains(&status2) { + 200 + } else { + status2 + }, + ) +} + +async fn microservice_tunnel_self_test_response( + state: &Arc<AppState>, + service: &MicroserviceConfig, +) -> Response { + let tunnel_service = if is_k3sctl_managed_microservice(service) { + service_by_id( + state, + service + .deployment + .adapter_service_id + .as_deref() + .unwrap_or("k3sctl-adapter"), + ) + } else { + Some(service.clone()) + }; + let Some(mut tunnel_service) = tunnel_service else { + return json_response( + json!({ "ok": false, "serviceId": service.id, "error": "tunnel service not found", "adapterServiceId": service.deployment.adapter_service_id }), + 502, + ); + }; + if !provider_supports( + state, + &tunnel_service.provider_id, + "microservice.http.tunnel", + ) + .await + .unwrap_or(false) + { + return json_response( + json!({ "ok": false, "serviceId": service.id, "providerId": tunnel_service.provider_id, "error": format!("provider does not declare microservice.http.tunnel capability: {}", tunnel_service.provider_id) }), + 409, + ); + } + tunnel_service.backend.node_base_url = "http://127.0.0.1:1".to_string(); + tunnel_service.backend.timeout_ms = 1000; + let response = provider_http_tunnel_microservice_response( + state, + &tunnel_service, + &Method::GET, + "/", + &ProxyOptions { + query: String::new(), + json_array_limits: json!({}), + }, + &json!({ "accept": "application/json" }), + String::new(), + ) + .await; + let status = response.status().as_u16(); + let request_header = response + .headers() + .get("x-unidesk-request-id") + .and_then(|value| value.to_str().ok()) + .map(ToOwned::to_owned); + let tunnel_error = response + .headers() + .get("x-unidesk-tunnel-error") + .and_then(|value| value.to_str().ok()) + .map(ToOwned::to_owned); + let (_, _, body_text) = response_to_text(response).await.unwrap_or(( + status, + "application/json".to_string(), + "{}".to_string(), + )); + let body: Value = serde_json::from_str(&body_text) + .unwrap_or_else(|_| Value::String(truncate_text(&body_text, 4000))); + let has_request_id = body + .get("requestId") + .and_then(Value::as_str) + .is_some_and(|value| !value.is_empty()); + let has_stage = body + .get("stage") + .and_then(Value::as_str) + .is_some_and(|value| !value.is_empty()); + let expected_status = status == 502 || status == 504; + let ok = expected_status + && has_request_id + && has_stage + && request_header.as_deref() == body.get("requestId").and_then(Value::as_str); + json_response( + json!({ + "ok": ok, + "serviceId": service.id, + "tunnelServiceId": tunnel_service.id, + "providerId": tunnel_service.provider_id, + "expectedFailure": true, + "status": status, + "checks": { + "expectedStatus": expected_status, + "bodyHasRequestId": has_request_id, + "bodyHasStage": has_stage, + "headerHasRequestId": request_header.as_ref().is_some_and(|value| !value.is_empty()), + "headerHasTunnelError": tunnel_error.as_ref().is_some_and(|value| !value.is_empty()), + }, + "headers": { "requestId": request_header, "tunnelError": tunnel_error }, + "body": body, + }), + if ok { 200 } else { 502 }, + ) +} + +pub async fn get_microservices(state: &Arc<AppState>) -> anyhow::Result<Vec<Value>> { + let nodes = get_nodes(state).await?; + let docker_statuses = get_node_docker_statuses(state).await?; + let mut result = Vec::new(); + for service in &state.config.microservices { + let node = nodes.iter().find(|item| { + item.get("providerId").and_then(Value::as_str) == Some(&service.provider_id) + }); + let docker = docker_statuses.iter().find(|item| { + item.get("providerId").and_then(Value::as_str) == Some(&service.provider_id) + }); + let k3s_managed = is_k3sctl_managed_microservice(service); + let container = if k3s_managed { + Value::Null + } else { + find_container( + docker.and_then(|item| item.get("dockerStatus")), + &service.repository.container_name, + ) + }; + let provider_status = node + .and_then(|item| item.get("status")) + .and_then(Value::as_str) + .unwrap_or("missing"); + let availability = + inferred_microservice_availability(state, service, provider_status, &container).await; + let mut service_json = serde_json::to_value(service)?; + service_json["runtime"] = json!({ + "orchestrator": if k3s_managed { "k3sctl" } else { "unidesk-direct" }, + "providerStatus": provider_status, + "providerName": node.and_then(|item| item.get("name")).and_then(Value::as_str).unwrap_or(&service.provider_id), + "providerLastHeartbeat": node.and_then(|item| item.get("lastHeartbeat")).cloned().unwrap_or(Value::Null), + "container": container, + "availability": availability, + "backendPortMapping": { + "providerId": service.provider_id, + "node": format!("{}:{}", service.backend.node_bind_host, service.backend.node_port), + "mainServerAccess": "frontend-only backend proxy", + "public": service.backend.public, + }, + }); + result.push(service_json); + } + Ok(result) +} + +async fn inferred_microservice_availability( + state: &Arc<AppState>, + service: &MicroserviceConfig, + provider_status: &str, + container: &Value, +) -> Value { + if let Some(cached) = cached_microservice_availability(state, &service.id).await { + return cached; + } + let container_state = container + .get("state") + .and_then(Value::as_str) + .unwrap_or("") + .to_ascii_lowercase(); + let container_status = container + .get("status") + .and_then(Value::as_str) + .unwrap_or(""); + if provider_status != "online" { + json!({ "serviceId": service.id, "healthy": false, "status": "unhealthy", "reason": format!("provider is {provider_status}"), "source": "runtime-inference" }) + } else if !container.is_null() && container_state != "running" { + json!({ "serviceId": service.id, "healthy": false, "status": "unhealthy", "reason": format!("container is {}", if container_status.is_empty() { &container_state } else { container_status }), "source": "runtime-inference" }) + } else { + json!({ "serviceId": service.id, "healthy": Value::Null, "status": "unknown", "reason": "strict health has not been probed yet", "source": "runtime-inference" }) + } +} + +pub async fn microservice_route( + state: &Arc<AppState>, + method: Method, + uri: Uri, + request: Request<Body>, +) -> anyhow::Result<Response> { + let path = uri.path().to_string(); + let rest = &path["/api/microservices/".len()..]; + let slash_index = rest.find('/'); + let service_id = percent_decode(match slash_index { + Some(index) => &rest[..index], + None => rest, + }); + let suffix = slash_index.map(|index| &rest[index + 1..]).unwrap_or(""); + let Some(service) = service_by_id(state, &service_id) else { + return Ok(json_response( + json!({ "ok": false, "error": format!("microservice not found: {service_id}") }), + 404, + )); + }; + if (suffix.is_empty() || suffix == "status") + && (method == Method::GET || method == Method::HEAD) + { + let microservices = get_microservices(state).await?; + return Ok(json_response( + json!({ "ok": true, "microservice": microservices.into_iter().find(|item| item.get("id").and_then(Value::as_str) == Some(&service_id)).unwrap_or_else(|| serde_json::to_value(&service).unwrap_or(Value::Null)) }), + 200, + )); + } + let target_path = if suffix == "health" { + service.backend.health_path.clone() + } else if suffix == "proxy" { + "/".to_string() + } else if let Some(rest) = suffix.strip_prefix("proxy/") { + format!("/{rest}") + } else if suffix == "diagnostics" { + "/diagnostics".to_string() + } else if suffix == "tunnel-self-test" { + "/tunnel-self-test".to_string() + } else { + String::new() + }; + if target_path.is_empty() { + return Ok(json_response( + json!({ "ok": false, "error": "microservice route must be /status, /health, /diagnostics, /tunnel-self-test, or /proxy/<path>" }), + 404, + )); + } + if suffix == "health" && method != Method::GET && method != Method::HEAD { + return Ok(json_response( + json!({ "ok": false, "error": "microservice health only supports GET/HEAD" }), + 405, + )); + } + if suffix == "diagnostics" && method != Method::GET && method != Method::HEAD { + return Ok(json_response( + json!({ "ok": false, "error": "microservice diagnostics only supports GET/HEAD" }), + 405, + )); + } + if suffix == "tunnel-self-test" && method != Method::GET && method != Method::HEAD { + return Ok(json_response( + json!({ "ok": false, "error": "microservice tunnel self-test only supports GET/HEAD" }), + 405, + )); + } + if !is_microservice_method_allowed(&service, &method) { + return Ok(json_response( + json!({ "ok": false, "error": "microservice method is not allowed", "serviceId": service_id, "method": method.as_str(), "allowedMethods": service.backend.allowed_methods }), + 405, + )); + } + if suffix == "diagnostics" { + return Ok(if is_k3sctl_managed_microservice(&service) { + k3sctl_managed_diagnostics_response(state, &service).await + } else { + strict_microservice_health_response(state, &service, method == Method::HEAD).await + }); + } + if suffix == "tunnel-self-test" { + return Ok(microservice_tunnel_self_test_response(state, &service).await); + } + if !is_microservice_path_allowed(&service, &target_path) { + return Ok(json_response( + json!({ "ok": false, "error": "microservice path is not allowed", "serviceId": service_id, "targetPath": target_path }), + 403, + )); + } + if suffix == "health" { + return Ok( + strict_microservice_health_response(state, &service, method == Method::HEAD).await, + ); + } + let proxy_options = read_microservice_array_limits(&uri); + let cache_key = microservice_cache_key(&service, &method, &target_path, &proxy_options); + let cache_ttl_ms = microservice_cache_ttl_ms(&service.id, &target_path); + if method == Method::GET || method == Method::HEAD { + if let Some(response) = read_microservice_cache(state, &cache_key).await { + return Ok(response); + } + } else { + invalidate_microservice_cache(state, &service.id).await; + } + let headers = request.headers().clone(); + let body_text = if method == Method::GET || method == Method::HEAD { + String::new() + } else { + let bytes = to_bytes(request.into_body(), 1024 * 1024 + 1).await?; + if bytes.len() > 1024 * 1024 { + return Ok(json_response( + json!({ "ok": false, "error": "microservice request body is too large", "maxBytes": 1024 * 1024 }), + 413, + )); + } + String::from_utf8_lossy(&bytes).to_string() + }; + let request_headers = read_microservice_request_headers(&headers); + if method == Method::GET || method == Method::HEAD { + if let Some(response) = read_stale_microservice_cache(state, &cache_key).await { + return Ok(response); + } + if let Some(response) = + read_microservice_path_fallback(state, &service, &method, &target_path).await + { + return Ok(response); + } + } + let response = fetch_microservice_upstream_response( + state, + &service, + &method, + &target_path, + &proxy_options, + &request_headers, + body_text, + ) + .await; + if (method == Method::GET || method == Method::HEAD) + && is_microservice_transient_failure_response(&response) + { + if let Some(mut stale) = read_stale_microservice_cache(state, &cache_key) + .await + .or(read_microservice_path_fallback(state, &service, &method, &target_path).await) + { + add_header(&mut stale, "x-unidesk-cache", "stale-on-transient-failure"); + add_header( + &mut stale, + "x-unidesk-stale-reason", + &response.status().as_u16().to_string(), + ); + return Ok(stale); + } + } + if (method == Method::GET || method == Method::HEAD) && cache_ttl_ms > 0 { + let (rebuilt, snapshot) = cacheable_response_snapshot(response).await; + remember_microservice_cache(state, cache_key, cache_ttl_ms, snapshot.clone()).await; + remember_microservice_path_fallback(state, &service, &method, &target_path, snapshot).await; + Ok(rebuilt) + } else { + Ok(response) + } +} + +fn urlencoding_like(value: &str) -> String { + percent_encoding::utf8_percent_encode(value, percent_encoding::NON_ALPHANUMERIC).to_string() +} diff --git a/src/components/backend-core/src/overview.rs b/src/components/backend-core/src/overview.rs new file mode 100644 index 00000000..800731d7 --- /dev/null +++ b/src/components/backend-core/src/overview.rs @@ -0,0 +1,281 @@ +use std::sync::Arc; + +use axum::body::Bytes; +use axum::http::Method; +use serde_json::{json, Value}; + +use crate::db::provider_supports; +use crate::http::{json_response, HttpResponse}; +use crate::microservice_proxy::strict_microservice_health_probe; +use crate::state::AppState; +use crate::task_dispatcher::{create_and_send_task, wait_for_task_terminal}; +use crate::types::RawTaskRow; + +pub async fn get_pgdata_usage(state: &Arc<AppState>) -> anyhow::Result<Value> { + let client = state.pool.get().await?; + let row = client + .query_one( + r#" + SELECT + current_database()::text AS database_name, + pg_database_size(current_database())::bigint AS database_bytes, + pg_size_pretty(pg_database_size(current_database()))::text AS database_pretty + "#, + &[], + ) + .await?; + let database_bytes: i64 = row.get("database_bytes"); + Ok(json!({ + "volumeName": state.config.database_volume_name, + "volumeSize": state.config.database_volume_size, + "databaseName": row.get::<_, String>("database_name"), + "databaseBytes": database_bytes, + "databasePretty": row.get::<_, String>("database_pretty"), + })) +} + +async fn count_pending_tasks(state: &Arc<AppState>) -> anyhow::Result<i64> { + let client = state.pool.get().await?; + let row = client + .query_one("SELECT count(*)::bigint AS count FROM unidesk_tasks WHERE status IN ('queued', 'dispatched', 'running')", &[]) + .await?; + Ok(row.get("count")) +} + +async fn microservice_availability_summary(state: &Arc<AppState>) -> Value { + let mut probes = Vec::new(); + for service in &state.config.microservices { + probes.push( + strict_microservice_health_probe( + state, + service, + Some(service.backend.timeout_ms.clamp(2500, 10_000)), + ) + .await, + ); + } + let healthy_count = probes + .iter() + .filter(|probe| probe.get("healthy").and_then(Value::as_bool) == Some(true)) + .count(); + let mut by_provider = serde_json::Map::new(); + for probe in &probes { + let provider_id = probe + .get("providerId") + .and_then(Value::as_str) + .unwrap_or("unknown"); + let entry = by_provider + .entry(provider_id.to_string()) + .or_insert_with(|| json!({ "total": 0, "healthy": 0, "unhealthy": 0 })); + if let Some(object) = entry.as_object_mut() { + object["total"] = json!(object.get("total").and_then(Value::as_i64).unwrap_or(0) + 1); + if probe.get("healthy").and_then(Value::as_bool) == Some(true) { + object["healthy"] = + json!(object.get("healthy").and_then(Value::as_i64).unwrap_or(0) + 1); + } else { + object["unhealthy"] = + json!(object.get("unhealthy").and_then(Value::as_i64).unwrap_or(0) + 1); + } + } + } + json!({ + "totalCount": probes.len(), + "healthyCount": healthy_count, + "unhealthyCount": probes.len().saturating_sub(healthy_count), + "checkedAt": crate::json_util::now_iso(), + "byProvider": by_provider, + "services": probes.iter().map(|probe| json!({ + "serviceId": probe.get("serviceId").cloned().unwrap_or(Value::Null), + "name": probe.get("name").cloned().unwrap_or(Value::Null), + "providerId": probe.get("providerId").cloned().unwrap_or(Value::Null), + "healthy": probe.get("healthy").cloned().unwrap_or(Value::Null), + "status": probe.get("status").cloned().unwrap_or(Value::Null), + "reason": probe.get("reason").cloned().unwrap_or(Value::Null), + "checkedAt": probe.get("checkedAt").cloned().unwrap_or(Value::Null), + })).collect::<Vec<_>>(), + }) +} + +pub async fn get_overview(state: &Arc<AppState>) -> anyhow::Result<Value> { + let client = state.pool.get().await?; + let node_row = client + .query_one( + "SELECT count(*)::bigint AS node_count, count(*) FILTER (WHERE status = 'online')::bigint AS online_node_count FROM unidesk_nodes", + &[], + ) + .await?; + let docker_row = client + .query_one( + "SELECT count(*) FILTER (WHERE d.status IS NOT NULL)::bigint AS docker_status_node_count FROM unidesk_nodes n LEFT JOIN unidesk_node_docker_status d ON d.provider_id = n.provider_id", + &[], + ) + .await?; + let system_row = client + .query_one( + "SELECT count(*) FILTER (WHERE s.status IS NOT NULL)::bigint AS system_status_node_count FROM unidesk_nodes n LEFT JOIN unidesk_node_system_status s ON s.provider_id = n.provider_id", + &[], + ) + .await?; + Ok(json!({ + "service": "unidesk-core", + "ok": true, + "dbReady": state.db_ready(), + "pgdata": get_pgdata_usage(state).await.unwrap_or_else(|error| json!({ "error": error.to_string() })), + "uptimeSeconds": (chrono::Utc::now() - state.service_started_at).num_seconds(), + "nodeCount": node_row.get::<_, i64>("node_count"), + "onlineNodeCount": node_row.get::<_, i64>("online_node_count"), + "dockerStatusNodeCount": docker_row.get::<_, i64>("docker_status_node_count"), + "systemStatusNodeCount": system_row.get::<_, i64>("system_status_node_count"), + "pendingTaskCount": count_pending_tasks(state).await.unwrap_or(0), + "microserviceAvailability": microservice_availability_summary(state).await, + "taskPendingTimeoutMs": state.config.task_pending_timeout_ms, + "activeSocketCount": state.active_provider_count().await, + "heartbeatTimeoutMs": state.config.heartbeat_timeout_ms, + })) +} + +fn record_from_task(task: Option<&RawTaskRow>) -> Value { + match task { + Some(task) => json!({ + "id": task.id, + "providerId": task.provider_id, + "command": task.command, + "status": task.status, + "payload": task.payload, + "result": task.result, + "updatedAt": task.updated_at.to_rfc3339_opts(chrono::SecondsFormat::Millis, true), + }), + None => Value::Null, + } +} + +pub async fn run_host_ssh_perf_command( + state: &Arc<AppState>, + provider_id: &str, + command: String, + timeout_ms: u64, +) -> anyhow::Result<Value> { + let (task_id, provider_online) = create_and_send_task( + state, + provider_id, + "host.ssh", + json!({ + "source": "code-queue-performance-panel", + "mode": "exec", + "cwd": "/root/unidesk", + "timeoutMs": 15_000, + "command": command, + }), + ) + .await?; + if !provider_online { + return Ok( + json!({ "ok": false, "taskId": task_id, "task": Value::Null, "stdout": "", "stderr": format!("provider is offline: {provider_id}"), "exitCode": Value::Null, "timedOut": false }), + ); + } + let task = wait_for_task_terminal(state, &task_id, timeout_ms).await?; + let result = task + .as_ref() + .and_then(|task| task.result.as_ref()) + .and_then(Value::as_object) + .cloned() + .unwrap_or_default(); + Ok(json!({ + "ok": task.as_ref().is_some_and(|task| task.status == "succeeded") && result.get("ok").and_then(Value::as_bool) == Some(true), + "taskId": task_id, + "task": record_from_task(task.as_ref()), + "stdout": result.get("stdout").and_then(Value::as_str).unwrap_or(""), + "stderr": result.get("stderr").and_then(Value::as_str).unwrap_or(""), + "exitCode": result.get("exitCode").cloned().unwrap_or(Value::Null), + "timedOut": result.get("timedOut").and_then(Value::as_bool).unwrap_or(false), + })) +} + +pub async fn codex_queue_load_test( + state: &Arc<AppState>, + method: Method, + body: Bytes, +) -> anyhow::Result<HttpResponse> { + let body: Value = if method == Method::POST { + serde_json::from_slice(&body).unwrap_or_else(|_| json!({})) + } else { + json!({}) + }; + let codex_service = state + .config + .microservices + .iter() + .find(|service| service.id == "code-queue"); + let provider_id = body + .get("providerId") + .and_then(Value::as_str) + .filter(|value| !value.is_empty()) + .or_else(|| codex_service.map(|service| service.provider_id.as_str())) + .unwrap_or("main-server"); + if !provider_supports(state, provider_id, "host.ssh") + .await + .unwrap_or(false) + { + return Ok(json_response( + json!({ "ok": true, "measurementOk": false, "error": format!("provider does not declare host.ssh capability: {provider_id}"), "providerId": provider_id }), + 200, + )); + } + let timeout_ms = number_from_body(&body, "timeoutMs", 90_000, 5_000, 180_000); + let target_ms = number_from_body(&body, "targetMs", 1_000, 100, 60_000); + let run_id = format!( + "codex_queue_perf_{}_{}", + chrono::Utc::now().timestamp_millis(), + rand::random::<u32>() + ); + let dir = ".state/code-queue-perf"; + let output_path = format!("{dir}/{run_id}.json"); + let stderr_path = format!("{dir}/{run_id}.stderr"); + let exit_path = format!("{dir}/{run_id}.exit"); + let url_arg = body + .get("url") + .and_then(Value::as_str) + .filter(|value| !value.is_empty()) + .map(|value| format!(" --url {}", shell_quote(value))) + .unwrap_or_default(); + let start_command = format!( + "mkdir -p {}; rm -f {} {} {}; (PLAYWRIGHT_BROWSERS_PATH=.state/playwright-browsers bun scripts/src/code-queue-perf.ts --json --timeout-ms {} --target-ms {}{} > {}; printf '%s' \"$?\" > {}) > {} 2>&1 & printf '%s\\n' {}", + shell_quote(dir), + shell_quote(&output_path), + shell_quote(&stderr_path), + shell_quote(&exit_path), + timeout_ms, + target_ms, + url_arg, + shell_quote(&output_path), + shell_quote(&exit_path), + shell_quote(&stderr_path), + shell_quote(&run_id), + ); + let start = run_host_ssh_perf_command(state, provider_id, start_command, 18_000).await?; + if start.get("ok").and_then(Value::as_bool) != Some(true) { + return Ok(json_response( + json!({ "ok": true, "measurementOk": false, "providerId": provider_id, "runId": run_id, "stage": "start", "error": start.get("stderr").and_then(Value::as_str).unwrap_or("failed to start Playwright benchmark"), "taskId": start.get("taskId").cloned().unwrap_or(Value::Null), "task": start.get("task").cloned().unwrap_or(Value::Null) }), + 200, + )); + } + Ok(json_response( + json!({ "ok": true, "measurementOk": true, "providerId": provider_id, "runId": run_id, "stage": "started", "result": { "ok": true, "targetMs": target_ms, "timeoutMs": timeout_ms } }), + 200, + )) +} + +fn number_from_body(body: &Value, key: &str, fallback: u64, min: u64, max: u64) -> u64 { + body.get(key) + .and_then(|value| { + value + .as_u64() + .or_else(|| value.as_str().and_then(|text| text.parse::<u64>().ok())) + }) + .unwrap_or(fallback) + .clamp(min, max) +} + +fn shell_quote(value: &str) -> String { + format!("'{}'", value.replace('\'', "'\\''")) +} diff --git a/src/components/backend-core/src/performance.rs b/src/components/backend-core/src/performance.rs new file mode 100644 index 00000000..61a8edc9 --- /dev/null +++ b/src/components/backend-core/src/performance.rs @@ -0,0 +1,327 @@ +use std::future::Future; +use std::sync::Arc; +use std::time::{Duration, Instant}; + +use serde::Serialize; +use serde_json::{json, Value}; + +use crate::json_util::now_iso; +use crate::state::AppState; +use crate::types::{OperationPerformanceSample, RequestPerformanceSample}; + +const MAX_PERFORMANCE_SAMPLES: usize = 3000; + +fn classify_request_component(pathname: &str) -> &'static str { + if pathname.starts_with("/api/microservices/") && pathname.contains("/proxy/") { + "microservice_proxy" + } else if pathname.starts_with("/api/microservices/") { + "microservice_registry" + } else if pathname.starts_with("/api/nodes/") { + "node_metrics_api" + } else if pathname == "/api/nodes" { + "node_inventory_api" + } else if pathname.starts_with("/api/tasks") || pathname.starts_with("/api/schedules") { + "scheduler_api" + } else if pathname.starts_with("/api/events") { + "event_api" + } else if pathname.starts_with("/api/performance") { + "performance_api" + } else if pathname.starts_with("/api/") { + "core_api" + } else if pathname.starts_with("/ws/") { + "websocket_api" + } else if pathname == "/logs" { + "logs_api" + } else { + "core_http" + } +} + +pub fn record_request_performance( + state: &Arc<AppState>, + method: &str, + pathname: &str, + status: u16, + duration: Duration, +) { + let state = state.clone(); + let method = method.to_string(); + let pathname = pathname.to_string(); + tokio::spawn(async move { + let mut samples = state.request_performance_samples.lock().await; + samples.push(RequestPerformanceSample { + at: now_iso(), + component: classify_request_component(&pathname).to_string(), + method, + path: pathname, + status, + duration_ms: duration.as_secs_f64() * 1000.0, + ok: status < 400, + }); + if samples.len() > MAX_PERFORMANCE_SAMPLES { + let drain = samples.len() - MAX_PERFORMANCE_SAMPLES; + samples.drain(0..drain); + } + }); +} + +pub async fn record_operation_performance( + state: &Arc<AppState>, + service: &str, + operation: &str, + detail: &str, + duration: Duration, + ok: bool, +) { + let mut samples = state.operation_performance_samples.lock().await; + samples.push(OperationPerformanceSample { + at: now_iso(), + service: service.to_string(), + operation: operation.to_string(), + duration_ms: duration.as_secs_f64() * 1000.0, + ok, + detail: if detail.len() > 260 { + format!("{}...", &detail[..257.min(detail.len())]) + } else { + detail.to_string() + }, + }); + if samples.len() > MAX_PERFORMANCE_SAMPLES { + let drain = samples.len() - MAX_PERFORMANCE_SAMPLES; + samples.drain(0..drain); + } +} + +pub async fn with_operation<T, Fut>( + state: &Arc<AppState>, + service: &str, + operation: &str, + detail: &str, + fut: Fut, +) -> anyhow::Result<T> +where + Fut: Future<Output = anyhow::Result<T>>, +{ + let started = Instant::now(); + match fut.await { + Ok(value) => { + record_operation_performance( + state, + service, + operation, + detail, + started.elapsed(), + true, + ) + .await; + Ok(value) + } + Err(error) => { + record_operation_performance( + state, + service, + operation, + &error.to_string(), + started.elapsed(), + false, + ) + .await; + Err(error) + } + } +} + +fn percentile(values: &[f64], ratio: f64) -> f64 { + if values.is_empty() { + return 0.0; + } + let mut sorted = values.to_vec(); + sorted.sort_by(|left, right| left.partial_cmp(right).unwrap_or(std::cmp::Ordering::Equal)); + let index = ((sorted.len() as f64 * ratio).ceil() as usize) + .saturating_sub(1) + .min(sorted.len() - 1); + sorted[index] +} + +fn average(values: &[f64]) -> f64 { + if values.is_empty() { + 0.0 + } else { + values.iter().sum::<f64>() / values.len() as f64 + } +} + +fn round_ms(value: f64) -> f64 { + (value * 10.0).round() / 10.0 +} + +#[derive(Default, Serialize)] +#[serde(rename_all = "camelCase")] +struct RequestSummary { + component: String, + request_count: usize, + failure_count: usize, + failure_rate: f64, + average_latency_ms: f64, + p95_latency_ms: f64, + max_latency_ms: f64, +} + +pub async fn get_performance(state: &Arc<AppState>) -> anyhow::Result<Value> { + let request_samples = state.request_performance_samples.lock().await.clone(); + let operation_samples = state.operation_performance_samples.lock().await.clone(); + let mut request_groups = + std::collections::BTreeMap::<String, Vec<RequestPerformanceSample>>::new(); + for sample in &request_samples { + request_groups + .entry(sample.component.clone()) + .or_default() + .push(sample.clone()); + } + let mut component_summary: Vec<Value> = request_groups + .into_iter() + .map(|(component, rows)| { + let durations = rows.iter().map(|row| row.duration_ms).collect::<Vec<_>>(); + let failure_count = rows.iter().filter(|row| !row.ok).count(); + json!({ + "component": component, + "requestCount": rows.len(), + "failureCount": failure_count, + "failureRate": if rows.is_empty() { 0.0 } else { failure_count as f64 / rows.len() as f64 }, + "averageLatencyMs": round_ms(average(&durations)), + "p95LatencyMs": round_ms(percentile(&durations, 0.95)), + "maxLatencyMs": round_ms(durations.into_iter().fold(0.0, f64::max)), + }) + }) + .collect(); + component_summary.sort_by_key(|item| { + std::cmp::Reverse( + item.get("requestCount") + .and_then(Value::as_u64) + .unwrap_or(0), + ) + }); + + let mut operation_groups = + std::collections::BTreeMap::<String, Vec<OperationPerformanceSample>>::new(); + for sample in &operation_samples { + operation_groups + .entry(format!("{}:{}", sample.service, sample.operation)) + .or_default() + .push(sample.clone()); + } + let mut operation_summary: Vec<Value> = operation_groups + .into_iter() + .map(|(key, rows)| { + let (service, operation) = key.split_once(':').unwrap_or((&key, "")); + let durations = rows.iter().map(|row| row.duration_ms).collect::<Vec<_>>(); + let failure_count = rows.iter().filter(|row| !row.ok).count(); + json!({ + "service": service, + "operation": operation, + "count": rows.len(), + "failureCount": failure_count, + "averageLatencyMs": round_ms(average(&durations)), + "p95LatencyMs": round_ms(percentile(&durations, 0.95)), + "maxLatencyMs": round_ms(durations.into_iter().fold(0.0, f64::max)), + }) + }) + .collect(); + operation_summary.sort_by_key(|item| { + std::cmp::Reverse(item.get("count").and_then(Value::as_u64).unwrap_or(0)) + }); + + let pgdata = crate::overview::get_pgdata_usage(state) + .await + .unwrap_or_else(|error| json!({ "ok": false, "error": error.to_string() })); + let codex_queue_storage = get_codex_queue_storage_performance(state).await; + let recent_failures: Vec<Value> = request_samples + .iter() + .filter(|sample| !sample.ok) + .rev() + .take(20) + .map(|sample| serde_json::to_value(sample).unwrap_or(Value::Null)) + .collect(); + let cutoff = chrono::Utc::now().timestamp_millis() - 10 * 60 * 1000; + let recent_slow_operations: Vec<Value> = operation_samples + .iter() + .filter(|sample| { + chrono::DateTime::parse_from_rfc3339(&sample.at) + .map(|at| at.timestamp_millis() >= cutoff) + .unwrap_or(false) + }) + .rev() + .take(20) + .map(|sample| serde_json::to_value(sample).unwrap_or(Value::Null)) + .collect(); + + Ok(json!({ + "ok": true, + "service": "backend-core", + "generatedAt": now_iso(), + "startedAt": state.service_started_at.to_rfc3339_opts(chrono::SecondsFormat::Millis, true), + "uptimeSeconds": (chrono::Utc::now() - state.service_started_at).num_seconds(), + "requests": { + "sampleCount": request_samples.len(), + "componentSummary": component_summary, + "recentFailures": recent_failures, + }, + "operations": { + "sampleCount": operation_samples.len(), + "summary": operation_summary, + "recentSlowOperations": recent_slow_operations, + }, + "process": process_memory_json(), + "database": { + "ready": state.db_ready(), + "pgdata": pgdata, + "codexQueueStorage": codex_queue_storage, + }, + })) +} + +fn process_memory_json() -> Value { + let status = std::fs::read_to_string("/proc/self/status").unwrap_or_default(); + let mut rss = 0_u64; + for line in status.lines() { + if let Some(rest) = line.strip_prefix("VmRSS:") { + rss = rest + .split_whitespace() + .next() + .and_then(|text| text.parse::<u64>().ok()) + .unwrap_or(0) + * 1024; + } + } + json!({ + "rssBytes": rss, + "heapUsedBytes": 0, + "heapTotalBytes": 0, + "externalBytes": 0, + "arrayBuffersBytes": 0, + }) +} + +async fn get_codex_queue_storage_performance(state: &Arc<AppState>) -> Value { + let result = async { + let client = state.pool.get().await?; + let rows = client.query("SELECT status, count(*)::int AS count FROM unidesk_codex_queue_tasks GROUP BY status ORDER BY status ASC", &[]).await?; + anyhow::Ok(rows) + }.await; + match result { + Ok(rows) => { + let mut total = 0_i64; + let mut counts = serde_json::Map::new(); + for row in rows { + let status: String = row.get("status"); + let count: i32 = row.get("count"); + total += i64::from(count); + counts.insert(status, json!(count)); + } + json!({ "ok": true, "table": "unidesk_codex_queue_tasks", "counts": counts, "total": total }) + } + Err(error) => { + json!({ "ok": false, "table": "unidesk_codex_queue_tasks", "error": error.to_string() }) + } + } +} diff --git a/src/components/backend-core/src/provider_registry.rs b/src/components/backend-core/src/provider_registry.rs new file mode 100644 index 00000000..e2afe3e0 --- /dev/null +++ b/src/components/backend-core/src/provider_registry.rs @@ -0,0 +1,341 @@ +use std::sync::Arc; + +use axum::extract::ws::{Message, WebSocket, WebSocketUpgrade}; +use axum::http::{HeaderMap, Uri}; +use axum::response::Response; +use futures_util::{SinkExt, StreamExt}; +use serde_json::{json, Value}; +use tokio::sync::mpsc; + +use crate::db::{ + record_event, update_provider_heartbeat, upsert_docker_status, upsert_provider_node, + upsert_system_status, +}; +use crate::egress_tcp::{handle_egress_tcp_close, handle_egress_tcp_data, handle_egress_tcp_open}; +use crate::http::json_response; +use crate::json_util::compact_json_for_storage; +use crate::ssh_bridge::forward_ssh_provider_message; +use crate::state::{AppState, HttpTunnelResponse, ProviderConnection}; +use crate::task_dispatcher::{is_terminal_task_status, notify_task_terminal}; + +pub async fn provider_ws( + state: Arc<AppState>, + ws: WebSocketUpgrade, + headers: HeaderMap, + uri: Uri, +) -> Response { + let query_token = uri.query().and_then(|query| { + url::form_urlencoded::parse(query.as_bytes()) + .find(|(key, _)| key == "token") + .map(|(_, value)| value.into_owned()) + }); + let token = query_token.or_else(|| { + headers + .get("x-provider-token") + .and_then(|value| value.to_str().ok()) + .map(ToOwned::to_owned) + }); + if token.as_deref() != Some(&state.config.provider_token) { + return json_response( + json!({ "ok": false, "error": "invalid provider token" }), + 401, + ); + } + ws.on_upgrade(move |socket| provider_socket_task(state, socket)) +} + +async fn provider_socket_task(state: Arc<AppState>, socket: WebSocket) { + state.log("info", "provider_socket_open", None); + let (mut sender_ws, mut receiver_ws) = socket.split(); + let (tx, mut rx) = mpsc::unbounded_channel::<Message>(); + let connection = Arc::new(ProviderConnection { + provider_id: tokio::sync::Mutex::new(None), + sender: tx.clone(), + }); + let send_task = tokio::spawn(async move { + while let Some(message) = rx.recv().await { + if sender_ws.send(message).await.is_err() { + break; + } + } + }); + + while let Some(message) = receiver_ws.next().await { + match message { + Ok(Message::Text(text)) => { + if let Err(error) = handle_provider_text(&state, &connection, text).await { + state.log( + "error", + "provider_message_failed", + Some(json!({ "error": error.to_string() })), + ); + let _ = tx.send(Message::Text(json!({ "type": "ack", "requestId": "message", "ok": false, "message": error.to_string() }).to_string())); + } + } + Ok(Message::Binary(bytes)) => { + if let Ok(text) = String::from_utf8(bytes.to_vec()) { + if let Err(error) = handle_provider_text(&state, &connection, text).await { + state.log( + "error", + "provider_message_failed", + Some(json!({ "error": error.to_string() })), + ); + } + } + } + Ok(Message::Close(_)) | Err(_) => break, + _ => {} + } + } + + let provider_id = connection.provider_id.lock().await.clone(); + state.log( + "warn", + "provider_socket_close", + Some(json!({ "providerId": provider_id })), + ); + if let Some(provider_id) = provider_id { + let should_mark_offline = { + let mut providers = state.active_providers.write().await; + if providers + .get(&provider_id) + .is_some_and(|active| Arc::ptr_eq(active, &connection)) + { + providers.remove(&provider_id); + true + } else { + false + } + }; + crate::egress_tcp::close_egress_tcp_connections_for_provider(&state, &provider_id).await; + if should_mark_offline { + if let Err(error) = mark_provider_offline(&state, &provider_id).await { + state.log( + "error", + "provider_offline_mark_failed", + Some(json!({ "providerId": provider_id, "error": error.to_string() })), + ); + } + } + } + send_task.abort(); +} + +async fn handle_provider_text( + state: &Arc<AppState>, + connection: &Arc<ProviderConnection>, + text: String, +) -> anyhow::Result<()> { + let message: Value = serde_json::from_str(&text)?; + let provider_id = message + .get("providerId") + .and_then(Value::as_str) + .unwrap_or("") + .to_string(); + if provider_id.is_empty() { + anyhow::bail!( + "Unsupported provider message: {}", + &text[..text.len().min(200)] + ); + } + { + let mut id = connection.provider_id.lock().await; + *id = Some(provider_id.clone()); + } + state + .active_providers + .write() + .await + .insert(provider_id.clone(), connection.clone()); + let message_type = message.get("type").and_then(Value::as_str).unwrap_or(""); + match message_type { + "host_ssh_opened" | "host_ssh_data" | "host_ssh_exit" | "host_ssh_error" => { + forward_ssh_provider_message(state, &message).await; + } + "http_tunnel_response" => { + let request_id = message + .get("requestId") + .and_then(Value::as_str) + .unwrap_or("") + .to_string(); + let waiter = state.http_tunnel_waiters.lock().await.remove(&request_id); + if let Some(waiter) = waiter { + let _ = waiter.send(Ok(HttpTunnelResponse { + provider_id, + request_id, + ok: message.get("ok").and_then(Value::as_bool).unwrap_or(false), + result: message.get("result").cloned().unwrap_or(Value::Null), + })); + } else { + state.log( + "warn", + "http_tunnel_response_without_waiter", + Some(json!({ "requestId": request_id })), + ); + } + } + "egress_tcp_open" => handle_egress_tcp_open(state, connection.clone(), &message).await?, + "egress_tcp_data" => handle_egress_tcp_data(state, &message).await?, + "egress_tcp_close" => handle_egress_tcp_close(state, &message).await?, + "register" => { + let mut labels = message.get("labels").cloned().unwrap_or_else(|| json!({})); + if let Some(object) = labels.as_object_mut() { + object.insert( + "unideskCapabilities".to_string(), + message + .get("capabilities") + .cloned() + .unwrap_or_else(|| json!([])), + ); + } + let name = message + .get("name") + .and_then(Value::as_str) + .unwrap_or(&provider_id); + upsert_provider_node(state, &provider_id, name, &labels).await?; + record_event(state, "provider_registered", &provider_id, json!({ + "providerId": provider_id, + "name": name, + "labels": labels, + "capabilities": message.get("capabilities").cloned().unwrap_or_else(|| json!([])), + })).await; + let _ = connection.sender.send(Message::Text(json!({ "type": "ack", "requestId": "register", "ok": true, "message": "registered" }).to_string())); + } + "heartbeat" => { + let labels = message.get("labels").cloned().unwrap_or_else(|| json!({})); + update_provider_heartbeat(state, &provider_id, &labels).await?; + state.log( + "debug", + "provider_heartbeat", + Some(json!({ "providerId": provider_id, "labels": labels })), + ); + } + "system_status" => { + let status = message.get("status").cloned().unwrap_or_else(|| json!({})); + let collected_at = status + .get("collectedAt") + .and_then(Value::as_str) + .unwrap_or_else(|| message.get("at").and_then(Value::as_str).unwrap_or("")); + upsert_system_status(state, &provider_id, &status, collected_at).await?; + state.log("debug", "provider_system_status", Some(json!({ + "providerId": provider_id, + "cpuPercent": status.pointer("/cpu/percent").cloned().unwrap_or(Value::Null), + "memoryPercent": status.pointer("/memory/percent").cloned().unwrap_or(Value::Null), + "diskPercent": status.pointer("/disk/percent").cloned().unwrap_or(Value::Null), + "ok": status.get("ok").cloned().unwrap_or(Value::Null), + }))); + } + "docker_status" => { + let status = message.get("status").cloned().unwrap_or_else(|| json!({})); + let collected_at = status + .get("collectedAt") + .and_then(Value::as_str) + .unwrap_or_else(|| message.get("at").and_then(Value::as_str).unwrap_or("")); + upsert_docker_status(state, &provider_id, &status, collected_at).await?; + state.log("debug", "provider_docker_status", Some(json!({ "providerId": provider_id, "counts": status.get("counts").cloned().unwrap_or(Value::Null), "ok": status.get("ok").cloned().unwrap_or(Value::Null) }))); + } + "task_status" => { + let task_id = message.get("taskId").and_then(Value::as_str).unwrap_or(""); + let status = message.get("status").and_then(Value::as_str).unwrap_or(""); + let stored_result = + compact_json_for_storage(&message.get("result").cloned().unwrap_or_else( + || json!({ "message": message.get("message").cloned().unwrap_or(Value::Null) }), + )); + let client = state.pool.get().await?; + client.execute( + r#" + WITH incoming AS ( + SELECT $2::text AS status, $3::jsonb AS result + ) + UPDATE unidesk_tasks + SET + status = CASE + WHEN unidesk_tasks.status IN ('succeeded', 'failed') AND incoming.status NOT IN ('succeeded', 'failed') THEN unidesk_tasks.status + WHEN unidesk_tasks.status = 'running' AND incoming.status = 'accepted' THEN unidesk_tasks.status + ELSE incoming.status + END, + result = CASE + WHEN unidesk_tasks.status IN ('succeeded', 'failed') AND incoming.status NOT IN ('succeeded', 'failed') THEN unidesk_tasks.result + WHEN unidesk_tasks.status = 'running' AND incoming.status = 'accepted' THEN unidesk_tasks.result + ELSE incoming.result + END, + updated_at = now() + FROM incoming + WHERE id = $1 + "#, + &[&task_id, &status, &stored_result], + ).await?; + record_event( + state, + "task_status", + &provider_id, + json!({ + "providerId": provider_id, + "taskId": task_id, + "status": status, + "message": message.get("message").cloned().unwrap_or(Value::Null), + "result": stored_result, + }), + ) + .await; + if is_terminal_task_status(status) { + notify_task_terminal(state, task_id).await?; + } + } + _ => anyhow::bail!( + "Unsupported provider message: {}", + &text[..text.len().min(200)] + ), + } + Ok(()) +} + +pub async fn mark_provider_offline(state: &Arc<AppState>, provider_id: &str) -> anyhow::Result<()> { + state.active_providers.write().await.remove(provider_id); + if !state.db_ready() { + return Ok(()); + } + let client = state.pool.get().await?; + client.execute("UPDATE unidesk_nodes SET status = 'offline', updated_at = now() WHERE provider_id = $1", &[&provider_id]).await?; + record_event( + state, + "provider_offline", + provider_id, + json!({ "providerId": provider_id }), + ) + .await; + Ok(()) +} + +pub async fn mark_stale_providers_offline(state: &Arc<AppState>) -> anyhow::Result<()> { + if !state.db_ready() { + return Ok(()); + } + let timeout_ms = state.config.heartbeat_timeout_ms as i64; + let client = state.pool.get().await?; + let rows = client + .query( + r#" + UPDATE unidesk_nodes + SET status = 'offline', updated_at = now() + WHERE status = 'online' + AND last_heartbeat IS NOT NULL + AND last_heartbeat < now() - ($1 * interval '1 millisecond') + RETURNING provider_id + "#, + &[&timeout_ms], + ) + .await?; + for row in rows { + let provider_id: String = row.get("provider_id"); + state.active_providers.write().await.remove(&provider_id); + record_event( + state, + "provider_heartbeat_timeout", + &provider_id, + json!({ "providerId": provider_id, "timeoutMs": timeout_ms }), + ) + .await; + } + Ok(()) +} diff --git a/src/components/backend-core/src/scheduler.rs b/src/components/backend-core/src/scheduler.rs new file mode 100644 index 00000000..05b246e3 --- /dev/null +++ b/src/components/backend-core/src/scheduler.rs @@ -0,0 +1,1252 @@ +use std::path::{Path, PathBuf}; +use std::sync::Arc; +use std::time::Duration; + +use axum::body::Bytes; +use axum::http::Method; +use chrono::{DateTime, Datelike, TimeZone, Timelike, Utc}; +use rand::Rng; +use serde_json::{json, Value}; +use sha2::{Digest, Sha256}; +use tokio::fs; +use tokio::process::Command; +use tokio_postgres::Row; + +use crate::db::record_event; +use crate::http::{json_response, percent_decode, read_limit, ParsedUrl}; +use crate::json_util::{compact_json, truncate_text}; +use crate::state::AppState; +use crate::task_dispatcher::{ + create_and_send_task, is_provider_dispatch_command, is_terminal_task_status, + wait_for_task_terminal, +}; +use crate::types::{RawTaskRow, ScheduledTaskRow, ScheduledTaskRunRow}; + +#[derive(Debug, Clone)] +enum ScheduleSpec { + Daily { + time_of_day: String, + timezone: String, + }, + Interval { + every_seconds: i64, + }, +} + +#[derive(Debug, Clone)] +enum ScheduleAction { + Dispatch { + provider_id: String, + command: String, + payload: Value, + timeout_ms: Option<u64>, + }, + PgdataBackup { + volume_name: String, + remote_base_dir: String, + staging_subdir: String, + baidu_base_url: String, + timeout_ms: u64, + cleanup_local: bool, + }, +} + +fn row_iso(value: Option<DateTime<Utc>>) -> Value { + value + .map(|value| Value::String(value.to_rfc3339_opts(chrono::SecondsFormat::Millis, true))) + .unwrap_or(Value::Null) +} + +fn schedule_run_from_row(row: Row) -> ScheduledTaskRunRow { + ScheduledTaskRunRow { + id: row.get("id"), + schedule_id: row.get("schedule_id"), + trigger_type: row.get("trigger_type"), + status: row.get("status"), + task_id: row.get("task_id"), + result: row.get("result"), + error: row.get("error"), + started_at: row.get("started_at"), + finished_at: row.get("finished_at"), + duration_ms: row.get("duration_ms"), + created_at: row.get("created_at"), + updated_at: row.get("updated_at"), + } +} + +fn schedule_task_from_row(row: Row) -> ScheduledTaskRow { + ScheduledTaskRow { + id: row.get("id"), + name: row.get("name"), + description: row.get("description"), + enabled: row.get("enabled"), + schedule_json: row.get("schedule_json"), + action_json: row.get("action_json"), + concurrency_policy: row.get("concurrency_policy"), + next_run_at: row.get("next_run_at"), + last_run_at: row.get("last_run_at"), + last_run_id: row.get("last_run_id"), + created_at: row.get("created_at"), + updated_at: row.get("updated_at"), + } +} + +fn schedule_run_view(row: &ScheduledTaskRunRow) -> Value { + json!({ + "id": row.id, + "scheduleId": row.schedule_id, + "triggerType": row.trigger_type, + "status": row.status, + "taskId": row.task_id, + "result": row.result, + "error": row.error, + "startedAt": row_iso(row.started_at), + "finishedAt": row_iso(row.finished_at), + "durationMs": row.duration_ms, + "createdAt": row.created_at.to_rfc3339_opts(chrono::SecondsFormat::Millis, true), + "updatedAt": row.updated_at.to_rfc3339_opts(chrono::SecondsFormat::Millis, true), + }) +} + +fn scheduled_task_view(row: &ScheduledTaskRow, runs: &[ScheduledTaskRunRow]) -> Value { + json!({ + "id": row.id, + "name": row.name, + "description": row.description, + "enabled": row.enabled, + "schedule": row.schedule_json, + "action": row.action_json, + "concurrencyPolicy": row.concurrency_policy, + "nextRunAt": row_iso(row.next_run_at), + "lastRunAt": row_iso(row.last_run_at), + "lastRunId": row.last_run_id, + "createdAt": row.created_at.to_rfc3339_opts(chrono::SecondsFormat::Millis, true), + "updatedAt": row.updated_at.to_rfc3339_opts(chrono::SecondsFormat::Millis, true), + "recentRuns": runs.iter().map(schedule_run_view).collect::<Vec<_>>(), + }) +} + +fn scheduled_raw_task_json(task: Option<&RawTaskRow>) -> Value { + match task { + Some(task) => json!({ + "id": task.id, + "providerId": task.provider_id, + "command": task.command, + "status": task.status, + "payload": compact_json(&task.payload), + "result": task.result.as_ref().map(compact_json).unwrap_or(Value::Null), + "updatedAt": task.updated_at.to_rfc3339_opts(chrono::SecondsFormat::Millis, true), + }), + None => Value::Null, + } +} + +fn normalize_schedule_id(value: Option<&Value>) -> anyhow::Result<String> { + let raw = value + .and_then(Value::as_str) + .filter(|value| !value.trim().is_empty()) + .map(|value| value.trim().to_string()) + .unwrap_or_else(|| { + format!( + "schedule_{}_{}", + Utc::now().timestamp_millis(), + format!("{:x}", rand::thread_rng().gen::<u32>()) + ) + }); + if raw.len() > 120 + || raw.is_empty() + || !raw + .chars() + .all(|ch| ch.is_ascii_alphanumeric() || matches!(ch, '_' | '.' | ':' | '-')) + { + anyhow::bail!("schedule id must be 1-120 chars using letters, numbers, _, ., :, or -"); + } + Ok(raw) +} + +fn normalize_time_of_day(value: Option<&Value>) -> anyhow::Result<String> { + let raw = value + .and_then(Value::as_str) + .filter(|value| !value.trim().is_empty()) + .unwrap_or("03:00") + .trim(); + let parts = raw.split(':').collect::<Vec<_>>(); + if parts.len() != 2 { + anyhow::bail!("daily schedule timeOfDay must use HH:MM in UTC"); + } + let hour = parts[0].parse::<u32>().unwrap_or(99); + let minute = parts[1].parse::<u32>().unwrap_or(99); + if hour > 23 || minute > 59 || parts[0].len() != 2 || parts[1].len() != 2 { + anyhow::bail!("daily schedule timeOfDay must use HH:MM in UTC"); + } + Ok(format!("{hour:02}:{minute:02}")) +} + +fn number_in_range(value: Option<&Value>, fallback: i64, min: i64, max: i64) -> i64 { + let parsed = value + .and_then(|value| { + value + .as_i64() + .or_else(|| value.as_str().and_then(|text| text.parse::<i64>().ok())) + }) + .unwrap_or(fallback); + parsed.clamp(min, max) +} + +fn normalize_schedule_spec(value: Option<&Value>) -> anyhow::Result<(ScheduleSpec, Value)> { + let record = value.and_then(Value::as_object); + if record + .and_then(|record| record.get("type")) + .and_then(Value::as_str) + == Some("interval") + { + let every_seconds = number_in_range( + record.and_then(|record| record.get("everySeconds")), + 3600, + 60, + 366 * 24 * 3600, + ); + return Ok(( + ScheduleSpec::Interval { every_seconds }, + json!({ "type": "interval", "everySeconds": every_seconds }), + )); + } + let time_of_day = normalize_time_of_day(record.and_then(|record| record.get("timeOfDay")))?; + let timezone = record + .and_then(|record| record.get("timezone")) + .and_then(Value::as_str) + .filter(|value| !value.is_empty()) + .unwrap_or("Etc/UTC") + .to_string(); + Ok(( + ScheduleSpec::Daily { + time_of_day: time_of_day.clone(), + timezone: timezone.clone(), + }, + json!({ "type": "daily", "timeOfDay": time_of_day, "timezone": timezone }), + )) +} + +fn normalize_remote_dir(input: &str) -> anyhow::Result<String> { + let raw = if input.trim().is_empty() { + "/" + } else { + input.trim() + }; + if raw.contains('\0') { + anyhow::bail!("remote directory must not contain null bytes"); + } + let rooted; + let input = if raw.starts_with('/') { + raw + } else { + rooted = format!("/{raw}"); + &rooted + }; + let mut normalized = + Path::new(input) + .components() + .fold(String::new(), |mut path, component| { + match component { + std::path::Component::Normal(part) => { + path.push('/'); + path.push_str(&part.to_string_lossy()); + } + std::path::Component::RootDir => {} + _ => {} + } + path + }); + if normalized.is_empty() { + normalized = "/".to_string(); + } + if normalized != "/" { + normalized = normalized.trim_end_matches('/').to_string(); + } + Ok(normalized) +} + +fn normalize_relative_staging_path(root: &str, input: &str) -> anyhow::Result<(String, PathBuf)> { + let cleaned = input + .replace('\\', "/") + .trim_start_matches('/') + .trim() + .to_string(); + let normalized = Path::new(if cleaned.is_empty() { "." } else { &cleaned }) + .components() + .fold(PathBuf::new(), |mut path, component| { + if let std::path::Component::Normal(part) = component { + path.push(part); + } + path + }); + if normalized.as_os_str().is_empty() { + anyhow::bail!("staging path must be a relative child path"); + } + let absolute = Path::new(root).join(&normalized); + let relative = normalized.to_string_lossy().replace('\\', "/"); + Ok((relative, absolute)) +} + +fn normalize_schedule_action( + state: &Arc<AppState>, + value: Option<&Value>, +) -> anyhow::Result<(ScheduleAction, Value)> { + let Some(record) = value.and_then(Value::as_object) else { + anyhow::bail!("scheduled task action must be an object"); + }; + match record.get("type").and_then(Value::as_str) { + Some("dispatch") => { + let provider_id = record + .get("providerId") + .and_then(Value::as_str) + .unwrap_or("") + .trim() + .to_string(); + if provider_id.is_empty() { + anyhow::bail!("dispatch action providerId is required"); + } + let command = record + .get("command") + .and_then(Value::as_str) + .unwrap_or("") + .to_string(); + if !is_provider_dispatch_command(&command) { + anyhow::bail!("dispatch action command is invalid"); + } + let payload = record + .get("payload") + .filter(|value| value.is_object()) + .cloned() + .unwrap_or_else(|| json!({})); + let timeout_ms = record.get("timeoutMs").map(|_| { + number_in_range( + record.get("timeoutMs"), + state.config.task_pending_timeout_ms as i64, + 1_000, + 24 * 3600_000, + ) as u64 + }); + let mut action_json = json!({ "type": "dispatch", "providerId": provider_id, "command": command, "payload": payload }); + if let Some(timeout_ms) = timeout_ms { + action_json["timeoutMs"] = json!(timeout_ms); + } + Ok(( + ScheduleAction::Dispatch { + provider_id, + command, + payload, + timeout_ms, + }, + action_json, + )) + } + Some("pgdata_backup") => { + let volume_name = record + .get("volumeName") + .and_then(Value::as_str) + .filter(|value| !value.is_empty()) + .unwrap_or(&state.config.database_volume_name) + .to_string(); + let remote_base_dir = normalize_remote_dir( + record + .get("remoteBaseDir") + .and_then(Value::as_str) + .unwrap_or("/SERVER_DATA/UNIDESK_PG_DATA"), + )?; + let (staging_subdir, _) = normalize_relative_staging_path( + &state.config.pgdata_backup_staging_dir, + record + .get("stagingSubdir") + .and_then(Value::as_str) + .unwrap_or("server-data/unidesk-pg-data"), + )?; + let baidu_base_url = record + .get("baiduBaseUrl") + .and_then(Value::as_str) + .filter(|value| !value.is_empty()) + .unwrap_or(&state.config.baidu_netdisk_internal_url) + .to_string(); + let timeout_ms = + number_in_range(record.get("timeoutMs"), 60 * 60_000, 60_000, 24 * 3600_000) as u64; + let cleanup_local = record.get("cleanupLocal").and_then(Value::as_bool) != Some(false); + Ok(( + ScheduleAction::PgdataBackup { + volume_name: volume_name.clone(), + remote_base_dir: remote_base_dir.clone(), + staging_subdir: staging_subdir.clone(), + baidu_base_url: baidu_base_url.clone(), + timeout_ms, + cleanup_local, + }, + json!({ + "type": "pgdata_backup", + "volumeName": volume_name, + "remoteBaseDir": remote_base_dir, + "stagingSubdir": staging_subdir, + "baiduBaseUrl": baidu_base_url, + "timeoutMs": timeout_ms, + "cleanupLocal": cleanup_local, + }), + )) + } + _ => anyhow::bail!("scheduled task action.type must be dispatch or pgdata_backup"), + } +} + +fn compute_next_run_at(schedule: &ScheduleSpec, after: DateTime<Utc>) -> DateTime<Utc> { + match schedule { + ScheduleSpec::Interval { every_seconds } => { + after + chrono::Duration::seconds((*every_seconds).max(60)) + } + ScheduleSpec::Daily { time_of_day, .. } => { + let parts = time_of_day.split(':').collect::<Vec<_>>(); + let hour = parts + .first() + .and_then(|value| value.parse::<u32>().ok()) + .unwrap_or(3); + let minute = parts + .get(1) + .and_then(|value| value.parse::<u32>().ok()) + .unwrap_or(0); + let mut candidate = Utc + .with_ymd_and_hms(after.year(), after.month(), after.day(), hour, minute, 0) + .unwrap(); + if candidate <= after { + candidate += chrono::Duration::days(1); + } + candidate + } + } +} + +fn utc_backup_parts(date: DateTime<Utc>) -> (String, String, String, String) { + let date_part = format!("{:04}{:02}{:02}", date.year(), date.month(), date.day()); + let time_part = format!("{:02}{:02}{:02}", date.hour(), date.minute(), date.second()); + let month = format!("{:04}{:02}", date.year(), date.month()); + let stamp = format!("{date_part}_{time_part}"); + (date_part, time_part, month, stamp) +} + +fn safe_file_part(value: &str) -> String { + let filtered = value + .chars() + .map(|ch| { + if ch.is_ascii_alphanumeric() || matches!(ch, '_' | '.' | '-') { + ch + } else { + '_' + } + }) + .collect::<String>(); + let trimmed = filtered.trim_matches('_'); + if trimmed.is_empty() { + "data".to_string() + } else { + trimmed[..trimmed.len().min(80)].to_string() + } +} + +fn database_connection_parts( + database_url: &str, +) -> anyhow::Result<(String, String, String, String)> { + let url = url::Url::parse(database_url)?; + Ok(( + url.host_str().unwrap_or("database").to_string(), + if url.port().is_some() { + url.port().unwrap().to_string() + } else { + "5432".to_string() + }, + percent_decode(url.username()), + url.password().map(percent_decode).unwrap_or_default(), + )) +} + +async fn run_local_command( + command: &str, + args: &[String], + envs: Vec<(&str, String)>, + timeout_ms: u64, +) -> anyhow::Result<(bool, String, String, Option<i32>, bool)> { + let mut cmd = Command::new(command); + cmd.args(args); + for (key, value) in envs { + cmd.env(key, value); + } + cmd.stdout(std::process::Stdio::piped()) + .stderr(std::process::Stdio::piped()); + let child = cmd.spawn()?; + match tokio::time::timeout( + Duration::from_millis(timeout_ms.max(1)), + child.wait_with_output(), + ) + .await + { + Ok(output) => { + let output = output?; + let stdout = truncate_text(&String::from_utf8_lossy(&output.stdout), 4000); + let stderr = truncate_text(&String::from_utf8_lossy(&output.stderr), 4000); + Ok(( + output.status.success(), + stdout, + stderr, + output.status.code(), + false, + )) + } + Err(_) => Ok(( + false, + String::new(), + "command timed out".to_string(), + None, + true, + )), + } +} + +async fn sha256_file(path: &Path) -> anyhow::Result<String> { + let bytes = fs::read(path).await?; + let mut hasher = Sha256::new(); + hasher.update(bytes); + Ok(format!("{:x}", hasher.finalize())) +} + +async fn fetch_json_with_timeout( + _url: String, + init: reqwest::RequestBuilder, + timeout_ms: u64, +) -> anyhow::Result<Value> { + let response = + tokio::time::timeout(Duration::from_millis(timeout_ms.max(1)), init.send()).await??; + let status = response.status(); + let text = response.text().await.unwrap_or_default(); + let parsed = serde_json::from_str::<Value>(&text).unwrap_or_else(|_| json!({ "text": text })); + if !status.is_success() { + anyhow::bail!( + "HTTP {}: {}", + status.as_u16(), + truncate_text(&parsed.to_string(), 1000) + ); + } + Ok(parsed) +} + +async fn baidu_json( + base_url: &str, + path: &str, + method: reqwest::Method, + body: Option<Value>, + timeout_ms: u64, +) -> anyhow::Result<Value> { + let client = reqwest::Client::new(); + let url = reqwest::Url::parse(base_url)?.join(path.trim_start_matches('/'))?; + let mut request = client.request(method, url); + if let Some(body) = body { + request = request.json(&body); + } + let value = fetch_json_with_timeout(path.to_string(), request, timeout_ms).await?; + if value.get("ok").and_then(Value::as_bool) == Some(false) { + anyhow::bail!("Baidu Netdisk API failed: {}", compact_json(&value)); + } + Ok(value) +} + +async fn wait_for_baidu_transfer( + state: &Arc<AppState>, + base_url: &str, + job_id: &str, + timeout_ms: u64, +) -> anyhow::Result<Value> { + let deadline = Utc::now().timestamp_millis() + timeout_ms as i64; + let mut latest = json!({}); + let mut latest_error = String::new(); + while Utc::now().timestamp_millis() < deadline { + match baidu_json( + base_url, + &format!("/api/transfers/{job_id}"), + reqwest::Method::GET, + None, + 30_000, + ) + .await + { + Ok(detail) => { + latest = detail.clone(); + latest_error.clear(); + let status = detail + .pointer("/job/status") + .and_then(Value::as_str) + .unwrap_or(""); + if status == "succeeded" { + return Ok(detail); + } + if status == "failed" || status == "canceled" { + anyhow::bail!( + "Baidu transfer {status}: {}", + detail + .pointer("/job/error") + .and_then(Value::as_str) + .unwrap_or("no error detail") + ); + } + } + Err(error) => { + latest_error = error.to_string(); + state.log( + "warn", + "baidu_transfer_poll_failed", + Some(json!({ "jobId": job_id, "error": latest_error })), + ); + } + } + tokio::time::sleep(Duration::from_secs(2)).await; + } + anyhow::bail!( + "Baidu transfer timed out after {timeout_ms}ms: {}", + compact_json(&json!({ "latest": latest, "latestError": latest_error })) + ); +} + +async fn execute_dispatch_schedule_action( + state: &Arc<AppState>, + provider_id: &str, + command: &str, + payload: &Value, + timeout_ms: Option<u64>, +) -> anyhow::Result<(bool, String, Value)> { + let mut dispatch_payload = payload.clone(); + if let Some(object) = dispatch_payload.as_object_mut() { + object.insert( + "source".to_string(), + Value::String("scheduled-task".to_string()), + ); + } + let (task_id, provider_online) = + create_and_send_task(state, provider_id, command, dispatch_payload.clone()).await?; + if !provider_online { + return Ok(( + false, + task_id.clone(), + json!({ "providerOnline": provider_online, "taskId": task_id, "error": format!("provider is offline: {provider_id}") }), + )); + } + let timeout_ms = timeout_ms.unwrap_or_else(|| state.config.task_pending_timeout_ms + 5000); + let task = wait_for_task_terminal(state, &task_id, timeout_ms).await?; + let ok = task.as_ref().is_some_and(|task| task.status == "succeeded"); + Ok(( + ok, + task_id.clone(), + json!({ + "providerOnline": provider_online, + "taskId": task_id, + "timeoutMs": timeout_ms, + "terminal": task.as_ref().is_some_and(|task| is_terminal_task_status(&task.status)), + "task": scheduled_raw_task_json(task.as_ref()), + }), + )) +} + +async fn execute_pgdata_backup_action( + state: &Arc<AppState>, + action: &ScheduleAction, + run_id: &str, +) -> anyhow::Result<(bool, Value)> { + let ScheduleAction::PgdataBackup { + volume_name, + remote_base_dir, + staging_subdir, + baidu_base_url, + timeout_ms, + cleanup_local, + } = action + else { + unreachable!(); + }; + let started_at = Utc::now(); + let (_, _, month, stamp) = utc_backup_parts(started_at); + let filename = format!( + "{}_{}.pg_basebackup.tar.gz", + stamp, + safe_file_part(volume_name) + ); + let month_relative_dir = format!("{staging_subdir}/{month}"); + let backup_relative_path = format!("{month_relative_dir}/{filename}"); + let (_, backup_path) = normalize_relative_staging_path( + &state.config.pgdata_backup_staging_dir, + &backup_relative_path, + )?; + let temp_relative_dir = format!("{staging_subdir}/{month}/.tmp_{run_id}"); + let (_, temp_dir) = normalize_relative_staging_path( + &state.config.pgdata_backup_staging_dir, + &temp_relative_dir, + )?; + let remote_dir = normalize_remote_dir(&format!("{remote_base_dir}/{month}"))?; + let remote_path = normalize_remote_dir(&format!("{remote_dir}/{filename}"))?; + let (db_host, db_port, db_user, db_password) = + database_connection_parts(&state.config.database_url)?; + if let Some(parent) = backup_path.parent() { + fs::create_dir_all(parent).await?; + } + let _ = fs::remove_dir_all(&temp_dir).await; + fs::create_dir_all(&temp_dir).await?; + let mut upload_detail: Option<Value> = None; + let mut upload_job_id = String::new(); + let result = async { + let backup_timeout_ms = (*timeout_ms as f64 * 0.55).floor().max(60_000.0) as u64; + let package_timeout_ms = (*timeout_ms as f64 * 0.15).floor().max(60_000.0) as u64; + let upload_timeout_ms = timeout_ms.saturating_sub(backup_timeout_ms + package_timeout_ms).max(60_000); + let args = vec![ + "-h".to_string(), db_host, + "-p".to_string(), db_port, + "-U".to_string(), db_user, + "-D".to_string(), temp_dir.to_string_lossy().to_string(), + "-Ft".to_string(), + "-X".to_string(), "stream".to_string(), + "-z".to_string(), + "--checkpoint=fast".to_string(), + "--no-sync".to_string(), + ]; + let (ok, stdout, stderr, exit_code, timed_out) = run_local_command("pg_basebackup", &args, vec![("PGPASSWORD", db_password)], backup_timeout_ms).await?; + if !ok { + anyhow::bail!("pg_basebackup failed exit={exit_code:?} timedOut={timed_out}: {}", if stderr.is_empty() { stdout } else { stderr }); + } + let tar_args = vec!["-czf".to_string(), backup_path.to_string_lossy().to_string(), "-C".to_string(), temp_dir.to_string_lossy().to_string(), ".".to_string()]; + let (ok, stdout, stderr, exit_code, timed_out) = run_local_command("tar", &tar_args, vec![], package_timeout_ms).await?; + if !ok { + anyhow::bail!("tar packaging failed exit={exit_code:?} timedOut={timed_out}: {}", if stderr.is_empty() { stdout } else { stderr }); + } + let metadata = fs::metadata(&backup_path).await?; + let backup_bytes = metadata.len(); + let backup_sha256 = sha256_file(&backup_path).await?; + let mut current = String::new(); + for part in remote_dir.split('/').filter(|part| !part.is_empty()) { + current.push('/'); + current.push_str(part); + let _ = baidu_json(baidu_base_url, "/api/folders", reqwest::Method::POST, Some(json!({ "path": current })), 60_000).await?; + } + let created = baidu_json(baidu_base_url, "/api/transfers/upload-from-path", reqwest::Method::POST, Some(json!({ "localPath": backup_relative_path, "remotePath": remote_path })), 60_000).await?; + let job_id = created.pointer("/job/id").and_then(Value::as_str).unwrap_or("").to_string(); + if job_id.is_empty() { + anyhow::bail!("Baidu upload did not return a job id: {}", compact_json(&created)); + } + upload_job_id = job_id.clone(); + let detail = wait_for_baidu_transfer(state, baidu_base_url, &job_id, upload_timeout_ms).await?; + upload_detail = Some(detail.clone()); + let uploaded_job = detail.get("job").cloned().unwrap_or_else(|| json!({})); + anyhow::Ok((true, json!({ + "backupType": "pg_basebackup", + "volumeName": volume_name, + "backupBytes": backup_bytes, + "backupSha256": backup_sha256, + "localRelativePath": backup_relative_path, + "remotePath": remote_path, + "remoteDir": remote_dir, + "month": month, + "timestampPrefix": stamp, + "baiduTransferJobId": job_id, + "baiduTransferStatus": uploaded_job.get("status").and_then(Value::as_str).unwrap_or(""), + "baiduFsId": uploaded_job.get("fsId").and_then(Value::as_str).unwrap_or(""), + "baiduResult": compact_json(uploaded_job.get("result").unwrap_or(&Value::Null)), + }))) + }.await; + let _ = fs::remove_dir_all(&temp_dir).await; + if *cleanup_local && (upload_job_id.is_empty() || upload_detail.is_some()) { + let _ = fs::remove_file(&backup_path).await; + } + result +} + +async fn execute_schedule_action( + state: &Arc<AppState>, + action: &ScheduleAction, + run_id: &str, +) -> anyhow::Result<(bool, Option<String>, Value)> { + match action { + ScheduleAction::Dispatch { + provider_id, + command, + payload, + timeout_ms, + } => { + let (ok, task_id, result) = + execute_dispatch_schedule_action(state, provider_id, command, payload, *timeout_ms) + .await?; + Ok((ok, Some(task_id), result)) + } + ScheduleAction::PgdataBackup { .. } => { + let (ok, result) = execute_pgdata_backup_action(state, action, run_id).await?; + Ok((ok, None, result)) + } + } +} + +async fn get_scheduled_task_row( + state: &Arc<AppState>, + schedule_id: &str, +) -> anyhow::Result<Option<ScheduledTaskRow>> { + let client = state.pool.get().await?; + Ok(client + .query_opt( + "SELECT * FROM unidesk_scheduled_tasks WHERE id = $1 LIMIT 1", + &[&schedule_id], + ) + .await? + .map(schedule_task_from_row)) +} + +async fn execute_scheduled_run(state: Arc<AppState>, run_id: String) { + { + let mut active = state.active_scheduled_runs.lock().await; + if !active.insert(run_id.clone()) { + return; + } + } + let started = Utc::now().timestamp_millis(); + let result = async { + let client = state.pool.get().await?; + let queued = client.query_opt("SELECT * FROM unidesk_scheduled_task_runs WHERE id = $1 LIMIT 1", &[&run_id]).await?.map(schedule_run_from_row); + let Some(queued) = queued else { return anyhow::Ok(()) }; + let Some(schedule_row) = get_scheduled_task_row(&state, &queued.schedule_id).await? else { return anyhow::Ok(()) }; + let updated = client.query("UPDATE unidesk_scheduled_task_runs SET status = 'running', started_at = now(), updated_at = now() WHERE id = $1 AND status = 'queued' RETURNING *", &[&run_id]).await?; + if updated.is_empty() { + return anyhow::Ok(()); + } + let (action, _) = normalize_schedule_action(&state, Some(&schedule_row.action_json))?; + match execute_schedule_action(&state, &action, &run_id).await { + Ok((ok, task_id, outcome)) => { + let duration_ms = Utc::now().timestamp_millis() - started; + let status = if ok { "succeeded" } else { "failed" }; + let error: Option<String> = if ok { None } else { Some("scheduled action failed".to_string()) }; + client.execute( + "UPDATE unidesk_scheduled_task_runs SET status = $2, task_id = $3, result = $4, error = $5, finished_at = now(), duration_ms = $6, updated_at = now() WHERE id = $1", + &[&run_id, &status, &task_id, &outcome, &error, &duration_ms], + ).await?; + client.execute("UPDATE unidesk_scheduled_tasks SET last_run_at = now(), last_run_id = $2, updated_at = now() WHERE id = $1", &[&schedule_row.id, &run_id]).await?; + record_event(&state, &format!("scheduled_task_{status}"), "scheduler", json!({ "scheduleId": schedule_row.id, "runId": run_id, "durationMs": duration_ms, "result": compact_json(&outcome) })).await; + } + Err(error) => { + let duration_ms = Utc::now().timestamp_millis() - started; + let error_text = error.to_string(); + let result = json!({ "name": "Error", "message": error_text }); + client.execute( + "UPDATE unidesk_scheduled_task_runs SET status = 'failed', result = $2, error = $3, finished_at = now(), duration_ms = $4, updated_at = now() WHERE id = $1", + &[&run_id, &result, &error_text, &duration_ms], + ).await?; + client.execute("UPDATE unidesk_scheduled_tasks SET last_run_at = now(), last_run_id = $2, updated_at = now() WHERE id = $1", &[&schedule_row.id, &run_id]).await?; + record_event(&state, "scheduled_task_failed", "scheduler", json!({ "scheduleId": schedule_row.id, "runId": run_id, "durationMs": duration_ms, "error": error_text })).await; + } + } + anyhow::Ok(()) + }.await; + if let Err(error) = result { + state.log( + "error", + "scheduled_run_uncaught", + Some(json!({ "runId": run_id, "error": error.to_string() })), + ); + } + state.active_scheduled_runs.lock().await.remove(&run_id); +} + +async fn get_scheduled_tasks(state: &Arc<AppState>, limit: i64) -> anyhow::Result<Vec<Value>> { + let client = state.pool.get().await?; + let rows = client.query("SELECT * FROM unidesk_scheduled_tasks ORDER BY enabled DESC, next_run_at ASC NULLS LAST, updated_at DESC LIMIT $1", &[&limit]).await?; + let tasks = rows + .into_iter() + .map(schedule_task_from_row) + .collect::<Vec<_>>(); + let run_limit = (limit * 5).max(20); + let runs = client + .query( + "SELECT * FROM unidesk_scheduled_task_runs ORDER BY updated_at DESC LIMIT $1", + &[&run_limit], + ) + .await? + .into_iter() + .map(schedule_run_from_row) + .collect::<Vec<_>>(); + Ok(tasks + .iter() + .map(|task| { + let recent = runs + .iter() + .filter(|run| run.schedule_id == task.id) + .take(5) + .cloned() + .collect::<Vec<_>>(); + scheduled_task_view(task, &recent) + }) + .collect()) +} + +async fn get_scheduled_task_runs( + state: &Arc<AppState>, + schedule_id: Option<&str>, + limit: i64, +) -> anyhow::Result<Vec<Value>> { + let client = state.pool.get().await?; + let rows = if let Some(schedule_id) = schedule_id { + client.query("SELECT * FROM unidesk_scheduled_task_runs WHERE schedule_id = $1 ORDER BY updated_at DESC LIMIT $2", &[&schedule_id, &limit]).await? + } else { + client + .query( + "SELECT * FROM unidesk_scheduled_task_runs ORDER BY updated_at DESC LIMIT $1", + &[&limit], + ) + .await? + }; + Ok(rows + .into_iter() + .map(schedule_run_from_row) + .map(|run| schedule_run_view(&run)) + .collect()) +} + +async fn upsert_scheduled_task( + state: &Arc<AppState>, + body: Value, + schedule_id_from_path: Option<&str>, +) -> anyhow::Result<crate::http::HttpResponse> { + let id_value = schedule_id_from_path + .map(|value| Value::String(value.to_string())) + .or_else(|| body.get("id").cloned()); + let id = normalize_schedule_id(id_value.as_ref())?; + let (schedule, schedule_json) = normalize_schedule_spec(body.get("schedule"))?; + let (_action, action_json) = normalize_schedule_action(state, body.get("action"))?; + let name = body + .get("name") + .and_then(Value::as_str) + .filter(|value| !value.trim().is_empty()) + .unwrap_or(&id) + .trim() + .to_string(); + let description = body + .get("description") + .and_then(Value::as_str) + .unwrap_or("") + .to_string(); + let enabled = body.get("enabled").and_then(Value::as_bool).unwrap_or(true); + let concurrency_policy = + if body.get("concurrencyPolicy").and_then(Value::as_str) == Some("parallel") { + "parallel" + } else { + "skip" + }; + let existing = get_scheduled_task_row(state, &id).await?; + let next_run_at = if let Some(existing) = &existing { + if existing.schedule_json == schedule_json { + existing + .next_run_at + .unwrap_or_else(|| compute_next_run_at(&schedule, Utc::now())) + } else { + compute_next_run_at(&schedule, Utc::now()) + } + } else { + compute_next_run_at(&schedule, Utc::now()) + }; + let client = state.pool.get().await?; + let row = client.query_one( + r#" + INSERT INTO unidesk_scheduled_tasks (id, name, description, enabled, schedule_json, action_json, concurrency_policy, next_run_at, updated_at) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, now()) + ON CONFLICT (id) DO UPDATE SET + name = EXCLUDED.name, + description = EXCLUDED.description, + enabled = EXCLUDED.enabled, + schedule_json = EXCLUDED.schedule_json, + action_json = EXCLUDED.action_json, + concurrency_policy = EXCLUDED.concurrency_policy, + next_run_at = EXCLUDED.next_run_at, + updated_at = now() + RETURNING * + "#, + &[&id, &name, &description, &enabled, &schedule_json, &action_json, &concurrency_policy, &next_run_at], + ).await?; + let row = schedule_task_from_row(row); + record_event(state, if existing.is_none() { "scheduled_task_created" } else { "scheduled_task_updated" }, "scheduler", json!({ "scheduleId": id, "name": name, "enabled": enabled, "schedule": schedule_json, "action": compact_json(&action_json) })).await; + Ok(json_response( + json!({ "ok": true, "schedule": scheduled_task_view(&row, &[]) }), + if existing.is_none() { 201 } else { 200 }, + )) +} + +async fn patch_scheduled_task( + state: &Arc<AppState>, + schedule_id: &str, + body: Value, +) -> anyhow::Result<crate::http::HttpResponse> { + let Some(current) = get_scheduled_task_row(state, schedule_id).await? else { + return Ok(json_response( + json!({ "ok": false, "error": format!("scheduled task not found: {schedule_id}") }), + 404, + )); + }; + let (schedule, schedule_json) = if body.get("schedule").is_some() { + normalize_schedule_spec(body.get("schedule"))? + } else { + normalize_schedule_spec(Some(¤t.schedule_json))? + }; + let (_, action_json) = if body.get("action").is_some() { + normalize_schedule_action(state, body.get("action"))? + } else { + normalize_schedule_action(state, Some(¤t.action_json))? + }; + let name = body + .get("name") + .and_then(Value::as_str) + .filter(|value| !value.trim().is_empty()) + .unwrap_or(¤t.name) + .trim() + .to_string(); + let description = body + .get("description") + .and_then(Value::as_str) + .unwrap_or(¤t.description) + .to_string(); + let enabled = body + .get("enabled") + .and_then(Value::as_bool) + .unwrap_or(current.enabled); + let concurrency_policy = match body.get("concurrencyPolicy").and_then(Value::as_str) { + Some("parallel") => "parallel", + Some("skip") => "skip", + _ => ¤t.concurrency_policy, + }; + let schedule_changed = body.get("schedule").is_some() && schedule_json != current.schedule_json; + let next_run_at = if schedule_changed || current.next_run_at.is_none() { + compute_next_run_at(&schedule, Utc::now()) + } else { + current.next_run_at.unwrap() + }; + let client = state.pool.get().await?; + let row = client.query_one( + "UPDATE unidesk_scheduled_tasks SET name=$2, description=$3, enabled=$4, schedule_json=$5, action_json=$6, concurrency_policy=$7, next_run_at=$8, updated_at=now() WHERE id=$1 RETURNING *", + &[&schedule_id, &name, &description, &enabled, &schedule_json, &action_json, &concurrency_policy, &next_run_at], + ).await?; + let row = schedule_task_from_row(row); + record_event(state, "scheduled_task_updated", "scheduler", json!({ "scheduleId": schedule_id, "name": name, "enabled": enabled, "schedule": schedule_json, "action": compact_json(&action_json) })).await; + Ok(json_response( + json!({ "ok": true, "schedule": scheduled_task_view(&row, &[]) }), + 200, + )) +} + +async fn delete_scheduled_task( + state: &Arc<AppState>, + schedule_id: &str, +) -> anyhow::Result<crate::http::HttpResponse> { + let client = state.pool.get().await?; + let rows = client + .query( + "DELETE FROM unidesk_scheduled_tasks WHERE id = $1 RETURNING id", + &[&schedule_id], + ) + .await?; + if rows.is_empty() { + return Ok(json_response( + json!({ "ok": false, "error": format!("scheduled task not found: {schedule_id}") }), + 404, + )); + } + record_event( + state, + "scheduled_task_deleted", + "scheduler", + json!({ "scheduleId": schedule_id }), + ) + .await; + Ok(json_response( + json!({ "ok": true, "deleted": schedule_id }), + 200, + )) +} + +async fn trigger_scheduled_task( + state: &Arc<AppState>, + schedule_id: &str, + trigger_type: &str, +) -> anyhow::Result<Option<Value>> { + let run_id = format!( + "schedrun_{}_{}", + Utc::now().timestamp_millis(), + format!("{:x}", rand::thread_rng().gen::<u32>()) + ); + let client = state.pool.get().await?; + let row = client + .query_opt( + "SELECT * FROM unidesk_scheduled_tasks WHERE id = $1", + &[&schedule_id], + ) + .await?; + let Some(row) = row else { + anyhow::bail!("scheduled task not found: {schedule_id}"); + }; + let schedule_row = schedule_task_from_row(row); + if trigger_type == "schedule" && !schedule_row.enabled { + return Ok(None); + } + let running_count: i64 = client.query_one("SELECT count(*)::bigint AS count FROM unidesk_scheduled_task_runs WHERE schedule_id = $1 AND status IN ('queued', 'running')", &[&schedule_id]).await?.get("count"); + let (schedule, _) = normalize_schedule_spec(Some(&schedule_row.schedule_json))?; + if trigger_type == "schedule" { + let next_run_at = compute_next_run_at(&schedule, Utc::now()); + client.execute("UPDATE unidesk_scheduled_tasks SET next_run_at = $2, updated_at = now() WHERE id = $1", &[&schedule_id, &next_run_at]).await?; + } + let run_row = if schedule_row.concurrency_policy != "parallel" && running_count > 0 { + client.query_one("INSERT INTO unidesk_scheduled_task_runs (id, schedule_id, trigger_type, status, result, error, finished_at, duration_ms) VALUES ($1, $2, $3, 'skipped', $4, 'previous run still active', now(), 0) RETURNING *", &[&run_id, &schedule_id, &trigger_type, &json!({ "reason": "previous run still active", "runningCount": running_count })]).await? + } else { + client.query_one("INSERT INTO unidesk_scheduled_task_runs (id, schedule_id, trigger_type, status) VALUES ($1, $2, $3, 'queued') RETURNING *", &[&run_id, &schedule_id, &trigger_type]).await? + }; + let run = schedule_run_from_row(run_row); + if run.status == "queued" { + tokio::spawn(execute_scheduled_run(state.clone(), run_id.clone())); + } + record_event(state, "scheduled_task_triggered", "scheduler", json!({ "scheduleId": schedule_id, "runId": run_id, "triggerType": trigger_type, "status": run.status })).await; + Ok(Some(schedule_run_view(&run))) +} + +pub async fn scheduled_task_route( + state: &Arc<AppState>, + method: Method, + path: &str, + query: &str, + body: Bytes, +) -> anyhow::Result<crate::http::HttpResponse> { + let prefix = "/api/schedules"; + let rest = if path == prefix { + "" + } else { + path.strip_prefix(&format!("{prefix}/")).unwrap_or("") + }; + let segments = rest + .split('/') + .filter(|part| !part.is_empty()) + .map(percent_decode) + .collect::<Vec<_>>(); + let url = ParsedUrl::new(path, query); + match (segments.as_slice(), method.as_str()) { + ([], "GET") => Ok(json_response( + json!({ "ok": true, "schedules": get_scheduled_tasks(state, read_limit(&url, 100)).await? }), + 200, + )), + ([], "POST") => { + let body = serde_json::from_slice::<Value>(&body).unwrap_or_else(|_| json!({})); + upsert_scheduled_task(state, body, None).await + } + ([only], "GET") if only == "runs" => Ok(json_response( + json!({ "ok": true, "runs": get_scheduled_task_runs(state, None, read_limit(&url, 100)).await? }), + 200, + )), + ([schedule_id], "GET") => { + let Some(schedule) = get_scheduled_task_row(state, schedule_id).await? else { + return Ok(json_response( + json!({ "ok": false, "error": format!("scheduled task not found: {schedule_id}") }), + 404, + )); + }; + let runs = get_scheduled_task_runs_raw(state, schedule_id, 20).await?; + Ok(json_response( + json!({ "ok": true, "schedule": scheduled_task_view(&schedule, &runs) }), + 200, + )) + } + ([schedule_id], "PUT") => { + let body = serde_json::from_slice::<Value>(&body).unwrap_or_else(|_| json!({})); + upsert_scheduled_task(state, body, Some(schedule_id)).await + } + ([schedule_id], "PATCH") => { + let body = serde_json::from_slice::<Value>(&body).unwrap_or_else(|_| json!({})); + patch_scheduled_task(state, schedule_id, body).await + } + ([schedule_id], "DELETE") => delete_scheduled_task(state, schedule_id).await, + ([schedule_id, action], "POST") if action == "run" => { + match trigger_scheduled_task(state, schedule_id, "manual").await? { + Some(run) => Ok(json_response(json!({ "ok": true, "run": run }), 200)), + None => Ok(json_response( + json!({ "ok": false, "error": format!("scheduled task was not triggered: {schedule_id}") }), + 409, + )), + } + } + ([schedule_id, action], "GET") if action == "runs" => Ok(json_response( + json!({ "ok": true, "runs": get_scheduled_task_runs(state, Some(schedule_id), read_limit(&url, 100)).await? }), + 200, + )), + _ => Ok(json_response( + json!({ "ok": false, "error": "scheduled task route not found", "path": path }), + 404, + )), + } +} + +async fn get_scheduled_task_runs_raw( + state: &Arc<AppState>, + schedule_id: &str, + limit: i64, +) -> anyhow::Result<Vec<ScheduledTaskRunRow>> { + let client = state.pool.get().await?; + Ok(client.query("SELECT * FROM unidesk_scheduled_task_runs WHERE schedule_id = $1 ORDER BY updated_at DESC LIMIT $2", &[&schedule_id, &limit]).await?.into_iter().map(schedule_run_from_row).collect()) +} + +pub async fn recover_scheduled_runs(state: &Arc<AppState>) -> anyhow::Result<()> { + let client = state.pool.get().await?; + let rows = client.query( + r#" + UPDATE unidesk_scheduled_task_runs + SET status = 'failed', + error = 'backend-core restarted before scheduled run completed', + result = jsonb_build_object('error', 'backend-core restarted before scheduled run completed'), + finished_at = now(), + updated_at = now() + WHERE status IN ('queued', 'running') + RETURNING * + "#, + &[], + ).await?; + if !rows.is_empty() { + record_event( + state, + "scheduled_runs_recovered", + "scheduler", + json!({ "failedRunCount": rows.len() }), + ) + .await; + } + let schedules = client + .query( + "SELECT * FROM unidesk_scheduled_tasks WHERE next_run_at IS NULL LIMIT 200", + &[], + ) + .await? + .into_iter() + .map(schedule_task_from_row) + .collect::<Vec<_>>(); + for schedule in schedules { + let (spec, _) = normalize_schedule_spec(Some(&schedule.schedule_json))?; + let next_run_at = compute_next_run_at(&spec, Utc::now()); + client.execute("UPDATE unidesk_scheduled_tasks SET next_run_at = $2, updated_at = now() WHERE id = $1", &[&schedule.id, &next_run_at]).await?; + } + Ok(()) +} + +pub async fn run_due_scheduled_tasks(state: &Arc<AppState>) -> anyhow::Result<()> { + if !state.db_ready() { + return Ok(()); + } + let client = state.pool.get().await?; + let rows = client.query( + "SELECT * FROM unidesk_scheduled_tasks WHERE enabled = true AND (next_run_at IS NULL OR next_run_at <= now()) ORDER BY next_run_at ASC NULLS FIRST LIMIT 5", + &[], + ).await?; + for row in rows { + let schedule = schedule_task_from_row(row); + if let Err(error) = trigger_scheduled_task(state, &schedule.id, "schedule").await { + state.log( + "error", + "scheduled_task_due_trigger_failed", + Some(json!({ "scheduleId": schedule.id, "error": error.to_string() })), + ); + } + } + Ok(()) +} diff --git a/src/components/backend-core/src/ssh_bridge.rs b/src/components/backend-core/src/ssh_bridge.rs new file mode 100644 index 00000000..6346588b --- /dev/null +++ b/src/components/backend-core/src/ssh_bridge.rs @@ -0,0 +1,290 @@ +use std::sync::Arc; + +use axum::extract::ws::{Message, WebSocket, WebSocketUpgrade}; +use axum::http::{HeaderMap, Uri}; +use axum::response::Response; +use futures_util::{SinkExt, StreamExt}; +use rand::Rng; +use serde_json::{json, Value}; +use tokio::sync::mpsc; + +use crate::db::provider_supports; +use crate::http::json_response; +use crate::state::{AppState, SshClientConnection}; + +pub async fn ssh_ws( + state: Arc<AppState>, + ws: WebSocketUpgrade, + headers: HeaderMap, + uri: Uri, +) -> Response { + let query_token = uri.query().and_then(|query| { + url::form_urlencoded::parse(query.as_bytes()) + .find(|(key, _)| key == "token") + .map(|(_, value)| value.into_owned()) + }); + let token = query_token.or_else(|| { + headers + .get("x-provider-token") + .and_then(|value| value.to_str().ok()) + .map(ToOwned::to_owned) + }); + if token.as_deref() != Some(&state.config.provider_token) { + return json_response( + json!({ "ok": false, "error": "invalid ssh bridge token" }), + 401, + ); + } + ws.on_upgrade(move |socket| ssh_socket_task(state, socket)) +} + +async fn ssh_socket_task(state: Arc<AppState>, socket: WebSocket) { + let (mut sender_ws, mut receiver_ws) = socket.split(); + let (tx, mut rx) = mpsc::unbounded_channel::<Message>(); + let send_task = tokio::spawn(async move { + while let Some(message) = rx.recv().await { + if sender_ws.send(message).await.is_err() { + break; + } + } + }); + let mut session_id: Option<String> = None; + while let Some(message) = receiver_ws.next().await { + match message { + Ok(Message::Text(text)) => { + if let Err(error) = + handle_ssh_client_text(&state, tx.clone(), &mut session_id, text).await + { + let _ = tx.send(Message::Text( + json!({ "type": "ssh.error", "message": error.to_string() }).to_string(), + )); + } + } + Ok(Message::Close(_)) | Err(_) => break, + _ => {} + } + } + if let Some(session_id) = session_id { + state.active_ssh_clients.write().await.remove(&session_id); + } + send_task.abort(); +} + +async fn handle_ssh_client_text( + state: &Arc<AppState>, + client_sender: mpsc::UnboundedSender<Message>, + session_id: &mut Option<String>, + text: String, +) -> anyhow::Result<()> { + let message: Value = serde_json::from_str(&text)?; + let message_type = message.get("type").and_then(Value::as_str).unwrap_or(""); + if message_type == "ssh.open" { + let provider_id = message + .get("providerId") + .and_then(Value::as_str) + .unwrap_or(""); + if provider_id.is_empty() { + let _ = client_sender.send(Message::Text( + json!({ "type": "ssh.error", "message": "providerId is required" }).to_string(), + )); + return Ok(()); + } + let provider = { + let providers = state.active_providers.read().await; + providers.get(provider_id).cloned() + }; + let Some(provider) = provider else { + let _ = client_sender.send(Message::Text(json!({ "type": "ssh.error", "message": format!("provider is not online: {provider_id}") }).to_string())); + return Ok(()); + }; + if !provider_supports(state, provider_id, "host.ssh") + .await + .unwrap_or(false) + { + let _ = client_sender.send(Message::Text(json!({ "type": "ssh.error", "message": format!("provider does not declare host.ssh capability: {provider_id}") }).to_string())); + return Ok(()); + } + let new_session_id = safe_session_id(); + *session_id = Some(new_session_id.clone()); + state.active_ssh_clients.write().await.insert( + new_session_id.clone(), + SshClientConnection { + provider_id: provider_id.to_string(), + sender: client_sender.clone(), + }, + ); + let mut open_message = json!({ + "type": "host_ssh_open", + "sessionId": new_session_id, + "cols": number_from_unknown(message.get("cols"), 100, 20, 300), + "rows": number_from_unknown(message.get("rows"), 30, 8, 120), + }); + if let Some(cwd) = message + .get("cwd") + .and_then(Value::as_str) + .filter(|value| !value.is_empty()) + { + open_message["cwd"] = Value::String(cwd.to_string()); + } + if let Some(command) = message + .get("command") + .and_then(Value::as_str) + .filter(|value| !value.is_empty()) + { + open_message["command"] = Value::String(command.to_string()); + } + if let Some(tty) = message.get("tty").and_then(Value::as_bool) { + open_message["tty"] = Value::Bool(tty); + } + let _ = provider + .sender + .send(Message::Text(open_message.to_string())); + let _ = client_sender.send(Message::Text(json!({ "type": "ssh.dispatched", "providerId": provider_id, "sessionId": new_session_id }).to_string())); + state.log("info", "ssh_session_dispatched", Some(json!({ "providerId": provider_id, "sessionId": session_id, "hasCommand": open_message.get("command").is_some() }))); + return Ok(()); + } + + let Some(current_session_id) = session_id.clone() else { + if message_type == "ssh.eof" || message_type == "ssh.close" { + return Ok(()); + } + let _ = client_sender.send(Message::Text( + json!({ "type": "ssh.error", "message": "ssh session is not open" }).to_string(), + )); + return Ok(()); + }; + let provider_id = { + let clients = state.active_ssh_clients.read().await; + clients + .get(¤t_session_id) + .map(|client| client.provider_id.clone()) + }; + let provider_id = provider_id.unwrap_or_else(|| { + message + .get("providerId") + .and_then(Value::as_str) + .unwrap_or("") + .to_string() + }); + let provider = state + .active_providers + .read() + .await + .get(&provider_id) + .cloned(); + let Some(provider) = provider else { + let _ = client_sender.send(Message::Text( + json!({ "type": "ssh.error", "message": "provider went offline" }).to_string(), + )); + return Ok(()); + }; + + match message_type { + "ssh.input" => { + let data = message.get("data").and_then(Value::as_str).unwrap_or(""); + if message.get("encoding").and_then(Value::as_str) != Some("base64") || data.is_empty() + { + let _ = client_sender.send(Message::Text( + json!({ "type": "ssh.error", "message": "ssh.input requires base64 data" }) + .to_string(), + )); + return Ok(()); + } + let _ = provider.sender.send(Message::Text(json!({ "type": "host_ssh_input", "sessionId": current_session_id, "data": data, "encoding": "base64" }).to_string())); + } + "ssh.resize" => { + let _ = provider.sender.send(Message::Text( + json!({ + "type": "host_ssh_resize", + "sessionId": current_session_id, + "cols": number_from_unknown(message.get("cols"), 100, 20, 300), + "rows": number_from_unknown(message.get("rows"), 30, 8, 120), + }) + .to_string(), + )); + } + "ssh.eof" => { + let _ = provider.sender.send(Message::Text( + json!({ "type": "host_ssh_eof", "sessionId": current_session_id }).to_string(), + )); + } + "ssh.close" => { + let _ = provider.sender.send(Message::Text( + json!({ "type": "host_ssh_close", "sessionId": current_session_id }).to_string(), + )); + close_ssh_client(state, ¤t_session_id, None).await; + } + _ => { + let _ = client_sender.send(Message::Text(json!({ "type": "ssh.error", "message": format!("unsupported ssh client message: {message_type}") }).to_string())); + } + } + Ok(()) +} + +pub async fn forward_ssh_provider_message(state: &Arc<AppState>, message: &Value) { + let session_id = message + .get("sessionId") + .and_then(Value::as_str) + .unwrap_or(""); + let clients = state.active_ssh_clients.read().await; + let Some(client) = clients.get(session_id) else { + state.log("warn", "ssh_client_missing", Some(json!({ "providerId": message.get("providerId").cloned().unwrap_or(Value::Null), "sessionId": session_id, "type": message.get("type").cloned().unwrap_or(Value::Null) }))); + return; + }; + let message_type = message.get("type").and_then(Value::as_str).unwrap_or(""); + let outbound = match message_type { + "host_ssh_opened" => { + json!({ "type": "ssh.opened", "providerId": message.get("providerId").cloned().unwrap_or(Value::Null), "sessionId": session_id }) + } + "host_ssh_data" => { + json!({ "type": "ssh.data", "stream": message.get("stream").cloned().unwrap_or(Value::Null), "data": message.get("data").cloned().unwrap_or(Value::Null), "encoding": message.get("encoding").cloned().unwrap_or(Value::Null) }) + } + "host_ssh_exit" => { + json!({ "type": "ssh.exit", "exitCode": message.get("exitCode").cloned().unwrap_or(Value::Null), "signal": message.get("signal").cloned().unwrap_or(Value::Null) }) + } + _ => { + json!({ "type": "ssh.error", "message": message.get("message").and_then(Value::as_str).unwrap_or("ssh session error") }) + } + }; + let _ = client.sender.send(Message::Text(outbound.to_string())); + drop(clients); + if message_type == "host_ssh_exit" || message_type == "host_ssh_error" { + let state = state.clone(); + let session_id = session_id.to_string(); + tokio::spawn(async move { + tokio::time::sleep(std::time::Duration::from_millis(50)).await; + close_ssh_client(&state, &session_id, None).await; + }); + } +} + +async fn close_ssh_client(state: &Arc<AppState>, session_id: &str, message: Option<&str>) { + let client = state.active_ssh_clients.write().await.remove(session_id); + if let Some(client) = client { + let close = Message::Close(Some(axum::extract::ws::CloseFrame { + code: axum::extract::ws::close_code::NORMAL, + reason: message.unwrap_or("ssh session closed").to_string().into(), + })); + let _ = client.sender.send(close); + } +} + +fn safe_session_id() -> String { + let suffix: u64 = rand::thread_rng().gen(); + format!( + "ssh_{}_{}", + chrono::Utc::now().timestamp_millis(), + format!("{suffix:x}") + ) +} + +fn number_from_unknown(value: Option<&Value>, fallback: i64, min: i64, max: i64) -> i64 { + let parsed = value + .and_then(|value| { + value + .as_i64() + .or_else(|| value.as_str().and_then(|text| text.parse::<i64>().ok())) + }) + .unwrap_or(fallback); + parsed.clamp(min, max) +} diff --git a/src/components/backend-core/src/state.rs b/src/components/backend-core/src/state.rs new file mode 100644 index 00000000..393fe8a7 --- /dev/null +++ b/src/components/backend-core/src/state.rs @@ -0,0 +1,115 @@ +use std::collections::HashMap; +use std::str::FromStr; +use std::sync::atomic::{AtomicBool, Ordering}; +use std::sync::Arc; + +use anyhow::Context; +use axum::extract::ws::Message; +use deadpool_postgres::{Manager, ManagerConfig, Pool, RecyclingMethod}; +use tokio::sync::{mpsc, oneshot, Mutex, RwLock}; +use tokio_postgres::NoTls; + +use crate::logger::Logger; +use crate::types::{ + JsonValue, MicroserviceProxyCacheEntry, OperationPerformanceSample, RequestPerformanceSample, + RuntimeConfig, +}; + +pub struct ProviderConnection { + pub provider_id: Mutex<Option<String>>, + pub sender: mpsc::UnboundedSender<Message>, +} + +pub struct SshClientConnection { + pub provider_id: String, + pub sender: mpsc::UnboundedSender<Message>, +} + +pub struct HttpTunnelResponse { + pub provider_id: String, + pub request_id: String, + pub ok: bool, + pub result: JsonValue, +} + +pub type HttpTunnelWaiter = oneshot::Sender<Result<HttpTunnelResponse, String>>; +pub type TaskTerminalWaiter = oneshot::Sender<Option<crate::types::RawTaskRow>>; + +pub struct AppState { + pub config: RuntimeConfig, + pub pool: Pool, + pub logger: Logger, + db_ready: AtomicBool, + pub service_started_at: chrono::DateTime<chrono::Utc>, + pub active_providers: RwLock<HashMap<String, Arc<ProviderConnection>>>, + pub active_ssh_clients: RwLock<HashMap<String, SshClientConnection>>, + pub http_tunnel_waiters: Mutex<HashMap<String, HttpTunnelWaiter>>, + pub task_terminal_waiters: Mutex<HashMap<String, Vec<TaskTerminalWaiter>>>, + pub active_egress_tcp: Mutex<HashMap<String, crate::egress_tcp::EgressTcpConnection>>, + pub microservice_proxy_cache: Mutex<HashMap<String, MicroserviceProxyCacheEntry>>, + pub microservice_availability_cache: Mutex<HashMap<String, (i64, JsonValue)>>, + pub request_performance_samples: Mutex<Vec<RequestPerformanceSample>>, + pub operation_performance_samples: Mutex<Vec<OperationPerformanceSample>>, + pub active_scheduled_runs: Mutex<std::collections::HashSet<String>>, +} + +impl AppState { + pub async fn new(config: RuntimeConfig, logger: Logger) -> anyhow::Result<Self> { + let pg_config = + tokio_postgres::Config::from_str(&config.database_url).context("parse DATABASE_URL")?; + let manager = Manager::from_config( + pg_config, + NoTls, + ManagerConfig { + recycling_method: RecyclingMethod::Fast, + }, + ); + let pool = Pool::builder(manager) + .max_size(config.database_pool_max) + .build() + .context("build PostgreSQL pool")?; + Ok(Self { + config, + pool, + logger, + db_ready: AtomicBool::new(false), + service_started_at: chrono::Utc::now(), + active_providers: RwLock::new(HashMap::new()), + active_ssh_clients: RwLock::new(HashMap::new()), + http_tunnel_waiters: Mutex::new(HashMap::new()), + task_terminal_waiters: Mutex::new(HashMap::new()), + active_egress_tcp: Mutex::new(HashMap::new()), + microservice_proxy_cache: Mutex::new(HashMap::new()), + microservice_availability_cache: Mutex::new(HashMap::new()), + request_performance_samples: Mutex::new(Vec::new()), + operation_performance_samples: Mutex::new(Vec::new()), + active_scheduled_runs: Mutex::new(std::collections::HashSet::new()), + }) + } + + pub fn set_db_ready(&self, ready: bool) { + self.db_ready.store(ready, Ordering::SeqCst); + } + + pub fn db_ready(&self) -> bool { + self.db_ready.load(Ordering::SeqCst) + } + + pub fn log(&self, level: &str, message: &str, data: Option<JsonValue>) { + self.logger.log(level, message, data); + } + + pub async fn active_provider_count(&self) -> usize { + self.active_providers.read().await.len() + } + + pub async fn send_provider(&self, provider_id: &str, value: JsonValue) -> bool { + let providers = self.active_providers.read().await; + providers.get(provider_id).is_some_and(|provider| { + provider + .sender + .send(Message::Text(value.to_string())) + .is_ok() + }) + } +} diff --git a/src/components/backend-core/src/task_dispatcher.rs b/src/components/backend-core/src/task_dispatcher.rs new file mode 100644 index 00000000..e0e62b68 --- /dev/null +++ b/src/components/backend-core/src/task_dispatcher.rs @@ -0,0 +1,247 @@ +use std::sync::Arc; +use std::time::Duration; + +use anyhow::bail; +use axum::body::Bytes; +use rand::Rng; +use serde_json::{json, Value}; + +use crate::db::{provider_supports, raw_task, record_event}; +use crate::json_util::{compact_json_for_storage, now_iso}; +use crate::state::AppState; +use crate::types::{JsonValue, RawTaskRow}; + +pub fn is_terminal_task_status(status: &str) -> bool { + status == "succeeded" || status == "failed" +} + +pub fn is_provider_dispatch_command(command: &str) -> bool { + matches!( + command, + "docker.ps" | "provider.upgrade" | "host.ssh" | "microservice.http" | "echo" + ) +} + +pub async fn notify_task_terminal(state: &Arc<AppState>, task_id: &str) -> anyhow::Result<()> { + let waiters = { + let mut waiters_by_task = state.task_terminal_waiters.lock().await; + waiters_by_task.remove(task_id) + }; + let Some(waiters) = waiters else { + return Ok(()); + }; + let task = raw_task(state, task_id).await?; + if task + .as_ref() + .is_none_or(|task| !is_terminal_task_status(&task.status)) + { + return Ok(()); + } + for waiter in waiters { + let _ = waiter.send(task.clone()); + } + Ok(()) +} + +pub async fn wait_for_task_terminal( + state: &Arc<AppState>, + task_id: &str, + timeout_ms: u64, +) -> anyhow::Result<Option<RawTaskRow>> { + let latest = raw_task(state, task_id).await?; + if latest + .as_ref() + .is_some_and(|task| is_terminal_task_status(&task.status)) + { + return Ok(latest); + } + let (tx, rx) = tokio::sync::oneshot::channel(); + { + let mut waiters = state.task_terminal_waiters.lock().await; + waiters.entry(task_id.to_string()).or_default().push(tx); + } + match tokio::time::timeout(Duration::from_millis(timeout_ms.max(1)), rx).await { + Ok(Ok(task)) => Ok(task), + _ => { + let task = raw_task(state, task_id).await?; + Ok(task.or(latest)) + } + } +} + +pub async fn create_and_send_task( + state: &Arc<AppState>, + provider_id: &str, + command: &str, + payload: JsonValue, +) -> anyhow::Result<(String, bool)> { + if !is_provider_dispatch_command(command) { + bail!("unsupported dispatch command: {command}"); + } + let suffix: u64 = rand::thread_rng().gen(); + let task_id = format!( + "task_{}_{}", + chrono::Utc::now().timestamp_millis(), + format!("{suffix:x}") + ); + let client = state.pool.get().await?; + client.execute( + "INSERT INTO unidesk_tasks (id, provider_id, command, status, payload, result) VALUES ($1, $2, $3, 'queued', $4, NULL)", + &[&task_id, &provider_id, &command, &payload], + ).await?; + let dispatch = + json!({ "type": "dispatch", "taskId": task_id, "command": command, "payload": payload }); + let provider_online = state.send_provider(provider_id, dispatch).await; + if !provider_online { + record_event( + state, + "task_queued_provider_offline", + provider_id, + json!({ "taskId": task_id, "providerId": provider_id, "command": command }), + ) + .await; + return Ok((task_id, false)); + } + client.execute("UPDATE unidesk_tasks SET status = 'dispatched', updated_at = now() WHERE id = $1 AND status = 'queued'", &[&task_id]).await?; + record_event( + state, + "task_dispatched", + provider_id, + json!({ "taskId": task_id, "providerId": provider_id, "command": command }), + ) + .await; + Ok((task_id, true)) +} + +pub async fn mark_stale_tasks_failed(state: &Arc<AppState>) -> anyhow::Result<()> { + if !state.db_ready() { + return Ok(()); + } + let timeout_ms = state.config.task_pending_timeout_ms as i64; + let client = state.pool.get().await?; + let rows = client + .query( + r#" + WITH stale AS ( + SELECT id, provider_id, command, status AS previous_status, updated_at + FROM unidesk_tasks + WHERE status IN ('queued', 'dispatched', 'running') + AND updated_at < now() - ($1::bigint * interval '1 millisecond') + FOR UPDATE + ), + updated AS ( + UPDATE unidesk_tasks task + SET + status = 'failed', + result = jsonb_build_object( + 'error', 'task timed out without terminal provider status', + 'timeoutMs', $1::bigint, + 'previousStatus', stale.previous_status, + 'previousResult', task.result, + 'timedOutAt', now() + ), + updated_at = now() + FROM stale + WHERE task.id = stale.id + RETURNING task.id, task.provider_id, task.command, stale.previous_status, stale.updated_at + ) + SELECT * FROM updated + "#, + &[&timeout_ms], + ) + .await?; + for row in rows { + let task_id: String = row.get("id"); + let provider_id: String = row.get("provider_id"); + let previous_status: String = row.get("previous_status"); + let updated_at: chrono::DateTime<chrono::Utc> = row.get("updated_at"); + record_event(state, "task_timeout", &provider_id, json!({ + "taskId": task_id, + "providerId": provider_id, + "command": row.get::<_, String>("command"), + "previousStatus": previous_status, + "previousUpdatedAt": updated_at.to_rfc3339_opts(chrono::SecondsFormat::Millis, true), + "timeoutMs": timeout_ms, + })).await; + if let Err(error) = notify_task_terminal(state, &task_id).await { + state.log( + "error", + "task_waiter_notify_failed", + Some(json!({ "taskId": task_id, "error": error.to_string() })), + ); + } + } + Ok(()) +} + +pub async fn maybe_mark_stale_tasks_failed( + state: &Arc<AppState>, + _min_interval_ms: u64, +) -> anyhow::Result<()> { + // The Rust implementation keeps this simple; the periodic sweeper handles + // the steady-state load, and list calls may also opportunistically mark. + mark_stale_tasks_failed(state).await +} + +pub async fn dispatch_task(state: &Arc<AppState>, body: Bytes) -> crate::http::HttpResponse { + let parsed: Value = serde_json::from_slice(&body).unwrap_or_else(|_| json!({})); + let provider_id = parsed + .get("providerId") + .and_then(Value::as_str) + .unwrap_or(""); + let command = parsed.get("command").and_then(Value::as_str).unwrap_or(""); + let payload = parsed + .get("payload") + .filter(|value| value.is_object()) + .cloned() + .unwrap_or_else(|| json!({})); + if provider_id.is_empty() { + return crate::http::json_response( + json!({ "ok": false, "error": "providerId is required" }), + 400, + ); + } + if !is_provider_dispatch_command(command) { + return crate::http::json_response( + json!({ "ok": false, "error": "command must be one of docker.ps, provider.upgrade, host.ssh, microservice.http, echo" }), + 400, + ); + } + if command == "host.ssh" + && !provider_supports(state, provider_id, "host.ssh") + .await + .unwrap_or(false) + { + return crate::http::json_response( + json!({ "ok": false, "error": format!("provider does not declare host.ssh capability: {provider_id}") }), + 409, + ); + } + if command == "microservice.http" + && !provider_supports(state, provider_id, "microservice.http") + .await + .unwrap_or(false) + { + return crate::http::json_response( + json!({ "ok": false, "error": format!("provider does not declare microservice.http capability: {provider_id}") }), + 409, + ); + } + match create_and_send_task( + state, + provider_id, + command, + compact_json_for_storage(&payload), + ) + .await + { + Ok((task_id, provider_online)) => crate::http::json_response( + json!({ "ok": true, "taskId": task_id, "status": if provider_online { "dispatched" } else { "queued" }, "providerOnline": provider_online }), + 200, + ), + Err(error) => crate::http::json_response( + json!({ "ok": false, "error": error.to_string(), "at": now_iso() }), + 500, + ), + } +} diff --git a/src/components/backend-core/src/types.rs b/src/components/backend-core/src/types.rs new file mode 100644 index 00000000..2efc32c1 --- /dev/null +++ b/src/components/backend-core/src/types.rs @@ -0,0 +1,197 @@ +use chrono::{DateTime, Utc}; +use serde::{Deserialize, Serialize}; +use serde_json::Value; + +pub type JsonValue = Value; + +#[derive(Debug, Clone, Deserialize, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct RuntimeConfig { + pub port: u16, + pub provider_port: u16, + pub database_url: String, + pub identity: RuntimeIdentity, + pub provider_token: String, + pub heartbeat_timeout_ms: u64, + pub task_pending_timeout_ms: u64, + pub database_volume_name: String, + pub database_volume_size: String, + pub pgdata_backup_staging_dir: String, + pub baidu_netdisk_internal_url: String, + pub microservices: Vec<MicroserviceConfig>, + pub log_file: String, + pub database_pool_max: usize, +} + +#[derive(Debug, Clone, Deserialize, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct RuntimeIdentity { + pub environment: String, + pub namespace: String, + pub database_name: String, + pub deploy_ref: String, + pub service_id: String, + pub repo: String, + pub commit: String, + pub requested_commit: String, +} + +#[derive(Debug, Clone, Deserialize, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct MicroserviceConfig { + pub id: String, + pub name: String, + pub provider_id: String, + pub description: String, + pub repository: MicroserviceRepositoryConfig, + pub backend: MicroserviceBackendConfig, + #[serde(default)] + pub deployment: MicroserviceDeploymentConfig, + pub development: MicroserviceDevelopmentConfig, + pub frontend: MicroserviceFrontendConfig, +} + +#[derive(Debug, Clone, Deserialize, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct MicroserviceRepositoryConfig { + pub url: String, + pub commit_id: String, + pub dockerfile: String, + pub compose_file: String, + pub compose_service: String, + pub container_name: String, +} + +#[derive(Debug, Clone, Deserialize, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct MicroserviceBackendConfig { + pub node_base_url: String, + pub node_bind_host: String, + pub node_port: u16, + pub proxy_mode: String, + pub frontend_only: bool, + pub public: bool, + pub allowed_methods: Vec<String>, + pub allowed_path_prefixes: Vec<String>, + pub health_path: String, + pub timeout_ms: u64, +} + +#[derive(Debug, Clone, Deserialize, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct MicroserviceDeploymentConfig { + pub mode: String, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub adapter_service_id: Option<String>, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub k3s_service_id: Option<String>, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub namespace: Option<String>, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub expected_node_ids: Option<Vec<String>>, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub active_node_id: Option<String>, +} + +impl Default for MicroserviceDeploymentConfig { + fn default() -> Self { + Self { + mode: "unidesk-direct".to_string(), + adapter_service_id: None, + k3s_service_id: None, + namespace: None, + expected_node_ids: None, + active_node_id: None, + } + } +} + +#[derive(Debug, Clone, Deserialize, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct MicroserviceDevelopmentConfig { + pub provider_id: String, + pub ssh_passthrough: bool, + pub worktree_path: String, +} + +#[derive(Debug, Clone, Deserialize, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct MicroserviceFrontendConfig { + pub route: String, + pub integrated: bool, +} + +#[derive(Debug, Clone)] +pub struct RawTaskRow { + pub id: String, + pub provider_id: String, + pub command: String, + pub status: String, + pub payload: JsonValue, + pub result: Option<JsonValue>, + pub updated_at: DateTime<Utc>, +} + +#[derive(Debug, Clone)] +pub struct ScheduledTaskRow { + pub id: String, + pub name: String, + pub description: String, + pub enabled: bool, + pub schedule_json: JsonValue, + pub action_json: JsonValue, + pub concurrency_policy: String, + pub next_run_at: Option<DateTime<Utc>>, + pub last_run_at: Option<DateTime<Utc>>, + pub last_run_id: Option<String>, + pub created_at: DateTime<Utc>, + pub updated_at: DateTime<Utc>, +} + +#[derive(Debug, Clone)] +pub struct ScheduledTaskRunRow { + pub id: String, + pub schedule_id: String, + pub trigger_type: String, + pub status: String, + pub task_id: Option<String>, + pub result: Option<JsonValue>, + pub error: Option<String>, + pub started_at: Option<DateTime<Utc>>, + pub finished_at: Option<DateTime<Utc>>, + pub duration_ms: Option<i64>, + pub created_at: DateTime<Utc>, + pub updated_at: DateTime<Utc>, +} + +#[derive(Debug, Clone, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct RequestPerformanceSample { + pub at: String, + pub component: String, + pub method: String, + pub path: String, + pub status: u16, + pub duration_ms: f64, + pub ok: bool, +} + +#[derive(Debug, Clone, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct OperationPerformanceSample { + pub at: String, + pub service: String, + pub operation: String, + pub duration_ms: f64, + pub ok: bool, + pub detail: String, +} + +#[derive(Debug, Clone)] +pub struct MicroserviceProxyCacheEntry { + pub expires_at_ms: i64, + pub stale_expires_at_ms: i64, + pub status: u16, + pub content_type: String, + pub body_text: String, +} diff --git a/src/components/dev-frontend-proxy/Dockerfile b/src/components/dev-frontend-proxy/Dockerfile new file mode 100644 index 00000000..828767d5 --- /dev/null +++ b/src/components/dev-frontend-proxy/Dockerfile @@ -0,0 +1,2 @@ +FROM nginx:1.27-alpine +COPY src/components/dev-frontend-proxy/nginx.conf /etc/nginx/conf.d/default.conf diff --git a/src/components/dev-frontend-proxy/nginx.conf b/src/components/dev-frontend-proxy/nginx.conf new file mode 100644 index 00000000..048270f1 --- /dev/null +++ b/src/components/dev-frontend-proxy/nginx.conf @@ -0,0 +1,20 @@ +server { + listen 8080; + server_name _; + + # The dev frontend is intentionally reached through the existing backend-core + # microservice proxy so the public port does not need direct access to D601. + location / { + proxy_http_version 1.1; + proxy_set_header Host $host; + proxy_set_header X-Forwarded-Host $host; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection "upgrade"; + proxy_buffering off; + proxy_read_timeout 300s; + proxy_pass http://backend-core:8080/api/microservices/k3sctl-adapter/proxy/api/services/frontend-dev/proxy$request_uri; + } +} diff --git a/src/components/microservices/code-queue/Dockerfile b/src/components/microservices/code-queue/Dockerfile index 21f81936..5014a4f5 100644 --- a/src/components/microservices/code-queue/Dockerfile +++ b/src/components/microservices/code-queue/Dockerfile @@ -2,8 +2,9 @@ ARG CODE_QUEUE_BASE_IMAGE=oven/bun:1-debian FROM ${CODE_QUEUE_BASE_IMAGE} ENV PLAYWRIGHT_BROWSERS_PATH=/ms-playwright +ENV PATH="/root/.cargo/bin:${PATH}" -RUN (command -v codex >/dev/null 2>&1 && command -v opencode >/dev/null 2>&1 && command -v docker >/dev/null 2>&1 && command -v rg >/dev/null 2>&1) \ +RUN (command -v codex >/dev/null 2>&1 && command -v opencode >/dev/null 2>&1 && command -v docker >/dev/null 2>&1 && command -v rg >/dev/null 2>&1 && command -v cargo >/dev/null 2>&1 && command -v rustc >/dev/null 2>&1 && command -v rustfmt >/dev/null 2>&1) \ || (apt-get update \ && apt-get install -y --no-install-recommends \ bash \ @@ -30,6 +31,7 @@ RUN (command -v codex >/dev/null 2>&1 && command -v opencode >/dev/null 2>&1 && python3 \ python3-pip \ ripgrep \ + rustup \ rsync \ tar \ tini \ @@ -38,6 +40,8 @@ RUN (command -v codex >/dev/null 2>&1 && command -v opencode >/dev/null 2>&1 && && mkdir -p /usr/local/lib/docker/cli-plugins /root/.docker/cli-plugins \ && ln -sf /usr/bin/docker-compose /usr/local/lib/docker/cli-plugins/docker-compose \ && ln -sf /usr/bin/docker-compose /root/.docker/cli-plugins/docker-compose \ + && rustup default stable \ + && rustup component add rustfmt \ && npm install -g @openai/codex@0.128.0 opencode-ai@1.14.48 playwright@1.59.1 \ && playwright install --with-deps chromium \ && apt-get clean \ diff --git a/src/components/microservices/k3sctl-adapter/k3s/ci/unidesk-ci.pipeline.yaml b/src/components/microservices/k3sctl-adapter/k3s/ci/unidesk-ci.pipeline.yaml index eebfd8d6..86dc5763 100644 --- a/src/components/microservices/k3sctl-adapter/k3s/ci/unidesk-ci.pipeline.yaml +++ b/src/components/microservices/k3sctl-adapter/k3s/ci/unidesk-ci.pipeline.yaml @@ -254,7 +254,7 @@ spec: docker compose version >/dev/null bun install --frozen-lockfile (cd src && bun install --frozen-lockfile) - bun scripts/cli.ts check + UNIDESK_D601_RUST_CHECK=1 bun scripts/cli.ts check --full --rust --- apiVersion: tekton.dev/v1 kind: Task