Add remote gc for provider hosts

This commit is contained in:
Codex
2026-05-29 01:43:14 +00:00
parent 3d5e613674
commit 3b9c8d8ddb
6 changed files with 1506 additions and 6 deletions
+1 -1
View File
@@ -146,7 +146,7 @@ UniDesk 是一个以主 server 为统一入口的分布式工作平台;本文
- `bun scripts/cli.ts server swap status|ensure [--path /swapfile] [--size 2GiB] [--dry-run]`:以 JSON 查看或幂等创建主 server swapfile`ensure` 输出 before/after、动作、持久化状态和 degraded/failed 详情,规则见 `docs/reference/deployment.md`
- `bun scripts/cli.ts server logs [--tail-bytes N]`:分页返回文件日志与 Docker 日志尾部并带截断元数据,日志规则见 `docs/reference/observability.md`
- `bun scripts/cli.ts server cleanup plan [--min-age-hours N] [--limit N]`:只读/干跑生成主 server Docker 镜像清理计划,默认只列出至少 24 小时前创建的非保护镜像,输出 active/protected images、stale candidates、预计释放空间、风险等级和必须人工确认的 `docker image rm` 命令;禁止默认删除、禁止 prune、禁止触碰 database volume、registry storage 或 Baidu Netdisk 状态。
- `bun scripts/cli.ts gc plan|run|db-trace|policy`:主 server 磁盘高水位一次性缓解和低风险防膨胀入口,覆盖日志、journald、Docker BuildKit cache、allowlisted `/tmp` 诊断目录、显式 trace 遥测留存和 systemd 定时策略;`gc run``gc db-trace run` 需要显式确认,规则见 `docs/reference/cli.md`
- `bun scripts/cli.ts gc plan|run|db-trace|policy|remote`:主 server 或受控 provider 磁盘高水位一次性缓解和低风险防膨胀入口,覆盖日志、journald、Docker BuildKit cache、allowlisted `/tmp` 诊断目录、受限 core dump、显式 trace 遥测留存和 systemd 定时策略;`gc run``gc remote ... run``gc db-trace run` 需要显式确认,规则见 `docs/reference/cli.md`
- `bun scripts/cli.ts server rebuild <backend-core|frontend|dev-frontend-proxy|provider-gateway|todo-note|code-queue-mgr|project-manager|baidu-netdisk|oa-event-flow>`:以 build-first、Compose lock、no-deps force-recreate 和 post-up validation 的异步 job 重建主 server Compose 内单个服务;对 database、File Browser、Code Queue 执行面、k3sctl-adapter 或未知对象返回结构化 `unsupported-server-rebuild`,规则见 `docs/reference/deployment.md``docs/reference/cicd-standardization.md`
- `bun scripts/cli.ts provider attach <providerId> [--master-server URL] [--up] [--force]` / `bun scripts/cli.ts provider triage <providerId> [--observed-error text] [--observed-scope scope] [--microservice id ...] [--full|--raw]`:前者在新增计算节点上生成两项配置的 provider-gateway 挂载包;后者是只读多信号健康裁决入口,默认低噪声输出 `decision``healthyScopes``failedScopes``retryable` 和异常信号摘要,用来把单路径 `provider is not online`、SSH 超时、registry 失败或 proxy 失败归类为 `retryable-transient``service-degraded``global-offline`,完整 evidence 需显式 `--full|--raw`,规则见 `docs/reference/provider-gateway.md``docs/reference/code-queue-supervision.md`
- `bun scripts/cli.ts ssh <route> [operation args...]` / `tran <route> [operation args...]`:通过 provider-gateway 的 Host SSH / WSL SSH 维护桥进入 provider、host workspace、Windows cmd route、k3s 控制面或 pod workspace,并提供带 SHA-256 校验的 `upload`/`download` 文件传输;主 server 人工/Codex 分布式操作必须用本机 `tran` wrapper,CLI 参考和可移植脚本可保留完整命令,细则见 `docs/reference/cli.md``docs/reference/windows-passthrough.md``docs/reference/provider-gateway.md`
+1 -1
View File
@@ -25,7 +25,7 @@ CI/CD、GitOps、rollout、artifact 发布、PR 合并后的 DEV/PROD 滚动、P
- `server swap status|ensure [--path /swapfile] [--size 2GiB] [--dry-run]` 是主 server swap 管理入口。`status` 仅读 `/proc/meminfo``/proc/swaps``/etc/fstab` 并返回 JSON`ensure` 在已有任何 active swap 时只报告 no-op,在无 active swap 时创建固定 swapfile、`chmod 600``mkswap``swapon` 并尽量写入 `/etc/fstab`。输出必须包含 `before``after`、total memory、active swap、持久化状态、关键动作和错误详情;若 swap 已启用但 fstab 写入失败,状态为 `degraded`,调用者需按返回的 detail 修复持久化。
- `server logs` 返回 `logs/` 文件日志和 Docker 容器日志的尾部,默认限制输出大小,避免日志爆炸。实现必须只读取文件末尾字节,不得为了 tail 先把巨大日志完整读入 CLI 内存。
- `server cleanup plan [--min-age-hours N] [--limit N]` 只生成主 server Docker 镜像清理 dry-run 计划,不执行删除;默认 `--min-age-hours 24`,避免把刚发布或刚验证的镜像列为 stale。输出必须包含 `dryRun=true``mutation=false``policy.deletionExecuted=false`、active containers/images、受保护镜像、candidate stale images、估算释放空间、风险等级、`commandsToReview` 和人工审批清单。计划必须保守白名单:保留 running containers 使用的 image ID,保留 stopped containers 引用的 image ID 直到人工先复核容器,保留 `deploy.json`/`CI.json` 当前 commit-pinned artifact、Compose stable image、上游 digest pin 和 provider-gateway runner image`protectedStorage` 必须显式列出 PostgreSQL named volume、Baidu Netdisk `.state`、D601 registry storage 和 Docker volumes/host data policy。该入口禁止生成或执行 `docker system prune``docker image prune``docker builder prune``docker volume rm``docker compose down -v`、数据库清理或 host data `rm` 命令;未来若增加真实删除,必须另设显式审批参数并先复核 dry-run 输出。
- `gc plan|run --confirm` 是主 server 磁盘高水位的一次性短期缓解入口,覆盖 UniDesk 文件日志、Docker `json-file` 容器日志、systemd journal、Docker BuildKit cache 和 allowlisted `/tmp` 诊断目录。`gc plan` 默认只读并返回候选、风险、估算释放空间、受保护对象和数据库只读摘要;候选列表默认按释放空间排序限制为 50 条,`--limit N` 调整页大小,`--full|--raw` 才展开全部候选。输出中的 `estimatedReclaim` 表示全量候选估算,`returnedEstimatedReclaim` 表示当前输出页估算;`gc run` 默认只返回前 50 条结果,`--result-limit N` 调整结果页大小,避免 GC 自身输出膨胀。`gc run` 必须显式 `--confirm`,只执行当前输出候选中的短期清理动作,因此默认不会执行被分页隐藏的候选。`gc` 不删除 Docker image/container/volume,不触碰 PostgreSQL PGDATA,不删除 Baidu Netdisk staging/backups 或 D601 registry storage;默认路径中数据库只做诊断摘要。`gc policy plan|install` 渲染或安装低风险长期策略:journald 512MiB 上限和每日 `unidesk-gc.timer`,该 timer 只运行文件日志与 allowlisted `/tmp` 低风险 GC,不主动 vacuum journal,不触碰数据库、Docker image/volume 或 Baidu staging,输出写入 `.state/gc/``gc db-trace plan|run --confirm --before-date YYYY-MM-DD --vacuum-full` 是显式数据库 trace 遥测留存入口,只删除 `oa_events` 中默认 trace 高频事件类型在指定日期前的数据,并在 `--vacuum-full` 下重写 `public.oa_events``df` 真正回收磁盘;执行前必须确认近期 PostgreSQL basebackup,且应视为维护窗口操作。常用参数包括 `--logs-keep-days N``--file-log-max-bytes SIZE``--file-log-tail-bytes SIZE``--docker-log-max-bytes SIZE``--journal-target-size SIZE``--build-cache-until 24h`、显式清空全部 BuildKit cache 的 `--build-cache-all``--tmp-min-age-hours N`显式 `--include-browser-cache`
- `gc plan|run --confirm` 是主 server 磁盘高水位的一次性短期缓解入口,覆盖 UniDesk 文件日志、Docker `json-file` 容器日志、systemd journal、Docker BuildKit cache 和 allowlisted `/tmp` 诊断目录。`gc plan` 默认只读并返回候选、风险、估算释放空间、受保护对象和数据库只读摘要;候选列表默认按释放空间排序限制为 50 条,`--limit N` 调整页大小,`--full|--raw` 才展开全部候选。输出中的 `estimatedReclaim` 表示全量候选估算,`returnedEstimatedReclaim` 表示当前输出页估算;`gc run` 默认只返回前 50 条结果,`--result-limit N` 调整结果页大小,避免 GC 自身输出膨胀。`gc run` 必须显式 `--confirm`,只执行当前输出候选中的短期清理动作,因此默认不会执行被分页隐藏的候选。`gc` 不删除 Docker image/container/volume,不触碰 PostgreSQL PGDATA,不删除 Baidu Netdisk staging/backups 或 D601 registry storage;默认路径中数据库只做诊断摘要。`gc remote <providerId> plan|run --confirm|status --job-id <id>` 通过 UniDesk SSH 透传在 provider host 上执行同类低风险 GC;G14 路径会先确认原生 k3s node 身份,并显式保护 `/var/lib/rancher/k3s``/var/lib/rancher/k3s/storage``/var/lib/kubelet``/var/lib/containerd`、HWLAB 固定 workspaces、Kubernetes workload/Secret/PVC/PV/Argo/Tekton 对象以及 Docker image/container/volume,默认只允许 journal vacuum、Docker json log truncate、Docker BuildKit cache prune、apt archive clean、allowlisted `/tmp` 临时目录删除,以及受限 core dump 清理;core dump 仅匹配 `/root/unidesk/core.<pid>` 这类普通文件,并在执行前重新校验路径 allowlist、Git 未跟踪和无 `fuser` 活跃引用。HWLAB registry 清理必须显式加 `--include-hwlab-registry` 才会进入,策略不是只留 latest:默认保留当前 workload 引用的 tag/digest、cache/latest、基础镜像 tag、48 小时内所有 tag,以及每个业务 repo 至少 20 个最新 tag;执行时会以远端异步 job 暂停 G14/v0.2 branch poller CronJob、等待 hwlab-ci PipelineRun/TaskRun/Job 空闲、通过 registry API 删除不再被保留 tag 引用的 manifest,再缩容 registry pod 后运行官方 `registry garbage-collect`,最后恢复 registry 和 CronJob;中断恢复可显式使用 `--registry-gc-only` 生成只跑官方 GC 的收尾候选,该候选不删除任何 tag。长任务状态通过 `gc remote <providerId> status --job-id <id>` 查询。`gc policy plan|install` 渲染或安装低风险长期策略:journald 512MiB 上限和每日 `unidesk-gc.timer`,该 timer 只运行文件日志与 allowlisted `/tmp` 低风险 GC,不主动 vacuum journal,不触碰数据库、Docker image/volume 或 Baidu staging,输出写入 `.state/gc/``gc db-trace plan|run --confirm --before-date YYYY-MM-DD --vacuum-full` 是显式数据库 trace 遥测留存入口,只删除 `oa_events` 中默认 trace 高频事件类型在指定日期前的数据,并在 `--vacuum-full` 下重写 `public.oa_events``df` 真正回收磁盘;执行前必须确认近期 PostgreSQL basebackup,且应视为维护窗口操作。常用参数包括 `--logs-keep-days N``--file-log-max-bytes SIZE``--file-log-tail-bytes SIZE``--docker-log-max-bytes SIZE``--journal-target-size SIZE``--build-cache-until 24h`、显式清空全部 BuildKit cache 的 `--build-cache-all``--tmp-min-age-hours N`显式 `--include-browser-cache`、远端 registry 的 `--registry-keep-per-repo N``--registry-min-age-hours N``--registry-gc-only``--job-id ID`
- `server rebuild <backend-core|frontend|dev-frontend-proxy|provider-gateway|todo-note|code-queue-mgr|project-manager|baidu-netdisk|oa-event-flow>` 创建异步 job,先构建目标服务镜像,随后在 `.state/locks/server-compose.lock` 串行保护下用 `--no-deps --force-recreate` 替换目标 service 并等待容器 `healthy/running`;该命令用于替代手工删除容器的兜底流程,其中 `dev-frontend-proxy` 只更新主 server dev 入口薄代理,`todo-note``code-queue-mgr``project-manager``baidu-netdisk``oa-event-flow` 只重建主 server 承载的对应后端,不会重建或删除 database 命名卷。D601 Code Queue 执行面不由 `server rebuild` 管理,Rust backend-core 迭代不得用 `server rebuild backend-core` 在 master server 编译,规则见 `docs/reference/dev-environment.md`
- `provider attach <providerId> [--master-server URL] [--up] [--force]` 在新计算节点生成两项配置的 provider-gateway 挂载包:`.state/provider-<ID>.env` 默认只包含 `UNIDESK_MASTER_SERVER``PROVIDER_ID``provider-<ID>.yml` 固定 Docker socket、`pid: "host"``restart: always`、只读 `/workspace` 和 SSH 维护私钥挂载;`--up` 会立即执行生成的 `docker compose up -d --build``provider triage <providerId> [--observed-error text] [--observed-scope scope] [--microservice id ...] [--full|--raw]` 是只读多信号健康裁决入口,会把单路径 `provider is not online`、SSH 超时、registry 失败和 service proxy 失败归类成 `runner-local-observation-gap``service-degraded``provider-degraded``global-blocker`。默认输出只返回裁决、scope、失败/降级/未知信号和有界 evidence 摘要,完整 evidence 必须显式加 `--full``--raw`;推荐交叉验证命令仍包含 `debug health``debug dispatch <providerId> host.ssh --wait-ms 15000``ssh <providerId> argv true``artifact-registry health --provider-id <providerId>``microservice health k3sctl-adapter``microservice health code-queue``codex tasks --view supervisor --limit 20`
- `ssh <route> [operation args...]` / `tran <route> [operation args...]` 通过 backend-core 内网 WebSocket broker 和 provider-gateway 的 Host SSH / WSL SSH 维护桥连接目标节点;`route` 基础形态是 provider id,例如 `D601``G14`,也可以扩展为纯定位路径 `provider:plane[:namespace:resource[:container]]`,例如 `D601:win``D601:win/c/test``G14:k3s``D601:k3s``G14:k3s:<namespace>:<workload>`。WSL provider 的 Windows cmd 入口固定写 `tran D601:win cmd <command-line>`,需要 Windows cwd 时用 `tran D601:win/c/test cmd cd`,由 CLI 自动设置 `chcp 65001``PYTHONUTF8=1``PYTHONIOENCODING=utf-8`;命名只允许 `win`,不得使用 `win32`。非交互远端命令优先使用 `ssh <providerId> argv ...`;需要 shell 脚本、管道、变量或循环时优先使用 quoted heredoc 单步传输,例如 `tran G14 script <<'SCRIPT'``tran G14:k3s script <<'SCRIPT'``tran G14:k3s:<namespace>:<workload> script <<'SCRIPT'`,把脚本走 stdin。`script -- '<单个字符串>'` 是无需 stdin 的远端 shell one-liner,例如 `tran G14:/root/hwlab script -- 'cd /root/hwlab && git status --short --branch'``script -- <多个 argv>` 才是 direct argv,适合 `tran D601:/path script -- sed -n '1,20p' file` 这类带短横线的单进程命令。顶层 remote option parser 必须保留命令已经开始后的 `--`,不得把它吞成全局选项结束符。需要远端改文本文件时默认优先使用 `<route> apply-patch < patch.diff`;需要可靠传输非文本或整文件时使用 `<route> upload <local-file> <remote-file>``<route> download <remote-file> <local-file>`CLI 会按字节数与 SHA-256 自动校验并在 provider-gateway stdin/argv 限制下切换客户端分块策略;需要旧 helper 时显式使用 `<provider>:k3s:<namespace>:<workload> apply-patch-v1``<providerId> apply-patch-v1`。ssh-like 命令遇到 timeout/kex/255 类失败时,CLI 会在 stderr 追加一行 `UNIDESK_SSH_HINT` JSON,提示 stdin script/argv 重试和 provider triage 交叉验证。
File diff suppressed because it is too large Load Diff
+6 -1
View File
@@ -3,6 +3,7 @@ import { closeSync, existsSync, ftruncateSync, lstatSync, mkdirSync, openSync, r
import { basename, join, resolve } from "node:path";
import { type UniDeskConfig, repoRoot, rootPath } from "./config";
import { runRemoteGcCommand } from "./gc-remote";
type GcRisk = "low" | "medium" | "high" | "blocked";
type GcItemKind =
@@ -183,6 +184,10 @@ const TMP_EXACT_PROTECT = new Set([
export async function runGcCommand(config: UniDeskConfig, args: string[]): Promise<unknown> {
const [action = "plan", ...rest] = args;
if (action === "remote") {
const [providerId, subaction = "plan", ...remoteArgs] = rest;
return await runRemoteGcCommand(config, providerId, subaction, remoteArgs);
}
if (action === "policy") {
const [subaction = "plan", ...policyArgs] = rest;
const options = parseGcPolicyOptions(policyArgs);
@@ -243,7 +248,7 @@ export async function runGcCommand(config: UniDeskConfig, args: string[]): Promi
ok: false,
error: "unsupported-gc-action",
action,
supportedActions: ["plan", "run", "db-trace", "policy"],
supportedActions: ["plan", "run", "db-trace", "policy", "remote"],
};
}
+14 -3
View File
@@ -18,7 +18,7 @@ export function rootHelp(): unknown {
{ command: "server swap status|ensure [--path /swapfile] [--size 2GiB] [--dry-run]", description: "Inspect or idempotently create host swap for low-memory main-server operation." },
{ command: "server logs [--tail-bytes N]", description: "Return bounded tails from file logs and docker logs." },
{ command: "server cleanup plan [--min-age-hours N] [--limit N]", description: "Dry-run Docker image cleanup plan only: list active/protected images, stale candidates older than the default 24h threshold, risk, estimated reclaim, and manual review commands without deleting anything." },
{ command: "gc plan|run|db-trace|policy [--confirm] [--logs-keep-days N] [--include-browser-cache]", description: "One-time main-server disk relief and low-risk anti-bloat policy for logs, journald, allowlisted /tmp artifacts and explicit trace telemetry retention; plan is read-only and run requires --confirm." },
{ command: "gc plan|run|db-trace|policy|remote [--confirm] [--logs-keep-days N] [--include-browser-cache]", description: "One-time main-server or remote provider disk relief and low-risk anti-bloat policy for logs, journald, allowlisted /tmp artifacts, scoped core dumps and explicit trace telemetry retention; plan is read-only and run requires --confirm." },
{ command: "server rebuild <backend-core|frontend|dev-frontend-proxy|provider-gateway|todo-note|code-queue-mgr|project-manager|baidu-netdisk|oa-event-flow>", description: "Maintenance-only local Compose rebuild for reviewed main-server services; frontend standard release must use CI artifact plus deploy apply dev/prod artifact consumers." },
{ command: "provider attach <providerId> [--master-server URL] [--up] [--force] | provider triage <providerId> [--observed-error text] [--observed-scope scope] [--microservice id ...] [--full|--raw]", description: "Generate the minimal external provider-gateway env/compose bundle or run the low-noise read-only provider health triage contract." },
{ command: "ssh <route> [operation args...]", description: "Open a Host SSH / WSL SSH maintenance session through the provider-gateway bridge; route syntax such as `G14:k3s` or `D601:win/c/test` only locates distributed targets." },
@@ -263,7 +263,7 @@ function providerHelp(): unknown {
function gcHelp(): unknown {
return {
command: "gc plan|run|db-trace|policy",
command: "gc plan|run|db-trace|policy|remote",
output: "json",
usage: [
"bun scripts/cli.ts gc plan",
@@ -275,9 +275,12 @@ function gcHelp(): unknown {
"bun scripts/cli.ts gc db-trace run --confirm --before-date 2026-05-25 --vacuum-full",
"bun scripts/cli.ts gc policy plan",
"bun scripts/cli.ts gc policy install",
"bun scripts/cli.ts gc remote G14 plan",
"bun scripts/cli.ts gc remote G14 run --confirm",
"bun scripts/cli.ts gc remote G14 status --job-id <id>",
"bun scripts/cli.ts gc plan --full",
],
description: "Plan or execute bounded one-time main-server disk relief for file logs, Docker json logs, systemd journal, Docker BuildKit cache, allowlisted /tmp artifacts and explicitly scoped database trace telemetry retention.",
description: "Plan or execute bounded one-time disk relief for file logs, Docker json logs, systemd journal, Docker BuildKit cache, allowlisted /tmp artifacts, scoped remote core dumps and explicitly scoped database trace telemetry retention.",
safety: {
default: "plan is read-only and mutation=false",
runGuard: "run requires --confirm",
@@ -293,6 +296,13 @@ function gcHelp(): unknown {
"--build-cache-until DURATION": "prune Docker builder cache unused for duration; default 24h",
"--build-cache-all": "prune all Docker builder cache without an until filter",
"--tmp-min-age-hours N": "delete allowlisted /tmp artifacts older than N hours; default 24",
"--core-dump-min-age-hours N": "remote only: delete untracked allowlisted core.<pid> dumps older than N hours; default 1",
"--no-core-dumps": "remote only: do not include scoped core dump cleanup candidates",
"--include-hwlab-registry": "remote G14 only: opt in to conservative HWLAB registry tag retention plus official registry garbage-collect",
"--registry-gc-only": "remote G14 only: run official registry garbage-collect without deleting additional tags; intended for interrupted registry retention recovery",
"--registry-keep-per-repo N": "remote registry only: keep at least N newest tags per service repo; default 20, minimum 10",
"--registry-min-age-hours N": "remote registry only: keep all tags newer than N hours; default 48, minimum 24",
"--job-id ID": "remote status only: inspect a long-running remote gc job",
"--limit N": "number of candidates returned and executed by run when --full is not set; default 50",
"--result-limit N": "number of per-candidate run results returned when --full is not set; default 50",
"--full|--raw": "return and run against all candidates rather than the default bounded page",
@@ -300,6 +310,7 @@ function gcHelp(): unknown {
"db-trace --before-date YYYY-MM-DD": "plan or delete default trace telemetry event types before the date",
"db-trace run --vacuum-full": "rewrite public.oa_events after deletion so df can reclaim disk; requires maintenance window",
"policy plan|install": "render or install journald caps and a daily file-log plus allowlisted /tmp low-risk gc systemd timer",
"remote <providerId> plan|run": "run bounded GC through UniDesk SSH passthrough on a provider host; G14 protects HWLAB k3s/runtime/PVC/workspace paths, and HWLAB registry retention is explicit opt-in with workload-ref, recent-tag and per-repo tag protection",
"--no-file-logs|--no-docker-logs|--no-journal|--no-build-cache|--no-tmp|--no-db-summary": "disable one collector",
},
reference: "docs/reference/cli.md",
+10
View File
@@ -2309,6 +2309,16 @@ async function runSshCaptureRemoteCommand(config: UniDeskConfig, invocation: Par
});
}
export async function runSshCommandCapture(config: UniDeskConfig, target: string, args: string[], input?: string): Promise<SshCaptureResult> {
const invocation = parseSshInvocation(target, args);
const parsed = invocation.parsed;
if (parsed.remoteCommand === null) throw new Error(`ssh ${target} capture requires a non-interactive operation`);
const stdin = parsed.stdinPrefix !== undefined || parsed.stdinSuffix !== undefined
? `${parsed.stdinPrefix ?? ""}${input ?? ""}${parsed.stdinSuffix ?? ""}`
: input;
return await runSshCaptureRemoteCommand(config, invocation, parsed.remoteCommand, stdin);
}
function writeChunkedStdin(stdin: NodeJS.WritableStream, input: string): void {
const buffer = Buffer.from(input, "utf8");
const chunkSize = 32 * 1024;