diff --git a/AGENTS.md b/AGENTS.md index 0bfa7449..5a19029a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -45,7 +45,7 @@ UniDesk 是一个以主 server 为统一入口的分布式工作平台；本文 - `bun scripts/cli.ts help`：输出所有可用命令的 JSON 索引，详细规范见 `docs/reference/cli.md`。 - `bun scripts/cli.ts --main-server-ip `：默认通过公网 frontend 登录态远程执行调试、用户服务（底层命令名 `microservice`）、Code Queue 查询与节点自测命令，不要求主 server SSH key，详细规范见 `docs/reference/cli.md`。 - `bun scripts/cli.ts config show`：校验并展示根目录 `config.json`，配置来源规则见 `docs/reference/config.md`。 -- `bun scripts/cli.ts check [--full|--files|--scripts-typecheck|--components|--compose|--logs|--rust]`：默认只运行轻量配置和 TypeScript 语法检查；Rust backend-core 检查只能在 D601 CI/dev execution 中用 `UNIDESK_D601_RUST_CHECK=1` 开启，规则见 `docs/reference/dev-environment.md`。 +- `bun scripts/cli.ts check [--full|--files|--scripts-typecheck|--components|--compose|--logs|--recovery-guardrails|--rust]` / `bun scripts/cli.ts check recovery-guardrails`：默认只运行轻量配置和 TypeScript 语法检查；`check recovery-guardrails` 只读低噪声报告 D601 reboot 后 k3s/Code Queue hostPath、`/proc/mounts`、CRI sandbox 和 ContainerCreating 风险；Rust backend-core 检查只能在 D601 CI/dev execution 中用 `UNIDESK_D601_RUST_CHECK=1` 开启，规则见 `docs/reference/dev-environment.md` 和 `docs/reference/devops-hygiene.md`。 - `bun scripts/cli.ts server start`：以异步 job 启动 database、backend-core、frontend、provider-gateway、code-queue-mgr 和主 server 用户服务，部署规则见 `docs/reference/deployment.md`。 - `bun scripts/cli.ts server status`：查询固定端口、swap 摘要、容器状态、健康检查和访问 URL，包含生产 frontend、dev frontend proxy 和 provider ingress，判定标准见 `docs/reference/deployment.md` 与 `docs/reference/dev-environment.md`。 - `bun scripts/cli.ts server swap status|ensure [--path /swapfile] [--size 2GiB] [--dry-run]`：以 JSON 查看或幂等创建主 server swapfile，`ensure` 输出 before/after、动作、持久化状态和 degraded/failed 详情，规则见 `docs/reference/deployment.md`。 diff --git a/docs/reference/cli.md b/docs/reference/cli.md index 6bcec6e4..bb0a56e3 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -10,7 +10,7 @@ CLI 可以从 `master` 快速演进，但必须兼容 `deploy.json` 固定的 CI - 每个 CLI 命名空间必须支持 `help`、`--help` 或 `-h` 并返回 JSON，不得为了打印帮助而访问 runtime 服务、拉起交互会话或执行长时任务。 - `--main-server-ip ` 默认通过公网 frontend 登录态调用主 server 的同源 API 代理，不要求计算节点持有主 server SSH key；显式提供 `--main-server-key` 或 `--main-server-transport ssh` 时才使用旧 SSH 传输。 - `config show` 读取并校验根目录 `config.json`，不从环境变量、默认值或隐藏文件静默补配置。 -- `check` 默认只执行轻量配置校验、Bun 版本检查和 Bun Transpiler 语法解析（覆盖 CLI 入口、主要 `scripts/` 模块和核心组件入口，不做类型推导）；关键文件存在性、`scripts/` TypeScript 类型检查、`src/components/` TypeScript 类型检查、Docker Compose config 和日志轮转策略扫描默认不启用，分别通过 `--files`、`--scripts-typecheck`、`--components`、`--compose`、`--logs` 开启，或用 `--full` 一次性开启。`--rust` 只允许在 D601 CI/dev execution 中配合 `UNIDESK_D601_RUST_CHECK=1` 使用，长期规则见 `docs/reference/dev-environment.md`。 +- `check` 默认只执行轻量配置校验、Bun 版本检查和 Bun Transpiler 语法解析（覆盖 CLI 入口、主要 `scripts/` 模块和核心组件入口，不做类型推导）；关键文件存在性、`scripts/` TypeScript 类型检查、`src/components/` TypeScript 类型检查、Docker Compose config、日志轮转策略扫描和 D601 recovery guardrails 默认不启用，分别通过 `--files`、`--scripts-typecheck`、`--components`、`--compose`、`--logs`、`--recovery-guardrails` 开启，或用 `--full` 一次性开启。`check recovery-guardrails` 是同一诊断的低噪声直接入口，报告 malformed `/proc/mounts`、kubelet validation risk、stale CRI sandbox count、Code Queue worktree/symlink、Code Queue/MDTODO hostPath 和 `ContainerCreating` 分类；它不得重启 k3s、删除 CRI sandbox、修改 hostPath、deploy/rollout 或 prune/reset。`--rust` 只允许在 D601 CI/dev execution 中配合 `UNIDESK_D601_RUST_CHECK=1` 使用，长期规则见 `docs/reference/dev-environment.md` 和 `docs/reference/devops-hygiene.md`。 - `server start` 创建异步 job，在后台执行 Docker 构建和启动；命令本身只负责返回 job id、日志路径和启动命令。 - `server stop` 创建异步 job，在后台停止固定 Compose project 中的全部 UniDesk 服务。 - `server status` 查询公开端口、受限宿主端口、内部端口、主机 swap 摘要、Compose 容器、core/frontend/dev-frontend/provider/database 健康检查和访问 URL；D601 Code Queue 使用的 PostgreSQL/OA Event Flow host mapping 必须出现在受限宿主端口而不是无条件公开入口中。低内存主 server 上 `swap.warning` 非空时，先执行 `server swap status` 或 `server swap ensure`。 diff --git a/docs/reference/devops-hygiene.md b/docs/reference/devops-hygiene.md index fa533c41..32732aa4 100644 --- a/docs/reference/devops-hygiene.md +++ b/docs/reference/devops-hygiene.md @@ -35,6 +35,7 @@ Manual operations are allowed only when they are narrow, visible and followed by - `deploy apply --service k3sctl-adapter` may update the k3s control bridge catalog through the documented local manifest exception in `docs/reference/deploy.md`. - Host SSH/provider-gateway dispatch may start the `ci run-dev-e2e` short launcher described in `docs/reference/dev-ci-runner.md`; it must not carry large shell bodies or become a general deployment path. - Read-only smoke checks such as `curl http://74.48.78.17:18083/health`, `server status`, `microservice proxy .../health` and `kubectl get` may validate state, but they do not replace desired-state and live-commit verification. +- `bun scripts/cli.ts check recovery-guardrails` and the equivalent `check --recovery-guardrails` gate are commander-safe read-only recovery diagnostic surfaces for D601 reboot incidents. They may read `/proc/mounts`, inspect local path metadata, parse committed k3s manifests, and run bounded read-only `kubectl get pods -A -o json` / `crictl pods -o json` probes when the tools are present. They must not restart k3s, delete pods or CRI sandboxes, apply manifests, rollout workloads, mutate hostPath directories, prune Docker state, or repair symlinks automatically. - File Browser recovery may use the existing provider-local image-only/docker-run path only as a bounded repair path. Standardization requires first resolving `docker.io/filebrowser/filebrowser:v2.63.3` to an upstream manifest digest or a digest-verified local mirror, then validating the running container through the UniDesk private proxy. Manual Secret/env/rollout repair is allowed only as a bounded runtime recovery path. It must have explicit authorization for the target environment and service, a narrow object scope, redacted evidence, an issue review trail and a durable source fix. Acceptable evidence includes object names, revision changes, health status, rollout status and redacted key presence; it must not include secret values, tokens, full env dumps or copy-pastable sensitive mutation commands. @@ -43,6 +44,28 @@ Any manual repair that changes live credentials, env wiring, DNS/egress assumpti If a manual repair is needed to unblock the platform, the durable fix must be committed and pushed, then redeployed or revalidated through the normal path. Do not preserve the repair only as hidden runtime state. +## D601 Recovery Hotfix Exception + +D601 reboot recovery has a narrow hotfix exception because k3s, Code Queue and hostPath readiness can fail before normal UniDesk proxy/CD surfaces are healthy. The exception authorizes diagnosis and carefully scoped host repair only; it does not make live host edits a new deployment path. + +Allowed read-only recovery checks: + +- `bun scripts/cli.ts check recovery-guardrails` or `bun scripts/cli.ts check --recovery-guardrails` on the host or runner environment. +- `/proc/mounts` inspection for malformed Docker Desktop `/Docker/host` 9p rows that may break kubelet mount-table validation. +- `kubectl get/describe/logs/events` and `crictl pods -o json` as bounded observation. +- Path metadata checks for `/home/ubuntu/unidesk-code-queue-deploy`, `/home/ubuntu/cq-deploy`, Code Queue hostPath directories, `.codex` files, `.ssh`, `.agents/skills`, and MDTODO workspace/log paths. + +Manual host hotfix may be considered only after the read-only output identifies a concrete redline and the operator has reviewed whether the target is source checkout, credential material, log/cache state, or user data. Examples include restoring a missing Git worktree from pushed remote state, recreating an intended compatibility symlink after confirming its target, restoring a missing runtime Secret source, or fixing a Docker Desktop/WSL mount-table condition. The repair must be recorded in the relevant issue or commander brief and followed by a source or runbook update when the root cause is durable. + +Forbidden automatic recovery actions: + +- `systemctl restart k3s`, `service k3s restart`, `kubectl delete pod`, `crictl rmp`, `crictl rm`, `docker system prune`, `docker volume prune`, recursive chmod/chown/rm under `/home/ubuntu`, and `git reset --hard` of a live worktree. +- Deleting CRI sandboxes or Kubernetes Pods because a diagnostic counted stale sandboxes. +- Creating, deleting or replacing MDTODO workspace content from Code Queue or the generic CLI. MDTODO hostPaths contain user-authored Markdown data. +- Treating `DirectoryOrCreate` as permission to mass-create parent trees or credential directories. Kubelet may create the final directory only when mount validation and parent permissions are already healthy; humans still decide user-data boundaries. + +ClaudeQQ or direct user approval is required before any high-risk host action that restarts k3s/kubelet/Docker Desktop, deletes pods/sandboxes, changes credential or SSH paths, touches MDTODO workspace data, force-resets a worktree, changes production rollout state, or could interrupt active Code Queue tasks. If ClaudeQQ is unavailable, the operator must stop at the written plan and record the approval gap instead of silently executing the action. + ## CI And Private Source Auth Private repository access is part of the CI contract. `ci run` must not rely on unauthenticated HTTPS clone, an operator's local dirty worktree or an ad-hoc secret copied by hand into one PipelineRun. diff --git a/docs/reference/microservices.md b/docs/reference/microservices.md index cc118228..15079531 100644 --- a/docs/reference/microservices.md +++ b/docs/reference/microservices.md @@ -296,6 +296,7 @@ D601 上必须显式使用原生 k3s kubeconfig：`KUBECONFIG=/etc/rancher/k3s/k D601 是 Windows + WSL Ubuntu + Docker Desktop 节点，Docker Desktop 当前 `LiveRestore=false` 时，机器或 Docker daemon 重启会停止容器，恢复链路必须同时覆盖 Windows 登录、WSL keepalive、Docker daemon ready、provider-gateway 和业务用户服务： +- 首选只读诊断入口：`bun scripts/cli.ts check recovery-guardrails`。该入口报告 malformed `/proc/mounts` Docker Desktop `/Docker/host` 9p 行、kubelet mount-table validation risk、stale CRI sandbox count、Code Queue deploy worktree/`/home/ubuntu/cq-deploy` symlink readiness、Code Queue/MDTODO k3s hostPath readiness、MDTODO adjacent hostPath 透明度和 `ContainerCreating` hostPath 分类。输出只给 safe read-only 证据、manual host hotfix redlines 和禁止自动动作；不得据此自动重启 k3s、删除 CRI sandbox、删除 Pod、修改 live hostPath、执行 deploy/rollout 或 destructive prune/reset。完整 hotfix exception、用户数据边界和 ClaudeQQ/用户请示条件见 `docs/reference/devops-hygiene.md`。 - Windows 登录任务：计划任务 `UniDesk-D601-Autostart` 在用户 `DESKTOP-1MHOD9I\liang` 登录时运行 `C:\WINDOWS\System32\cmd.exe /c ""C:\Users\liang\AppData\Local\UniDesk\d601-autostart.cmd""`，工作目录为 `C:\Users\liang\AppData\Local\UniDesk`。 - Windows launcher：`C:\Users\liang\AppData\Local\UniDesk\d601-autostart.cmd` 先启动 `%ProgramFiles%\Docker\Docker\Docker Desktop.exe`，再执行 `C:\Windows\System32\wsl.exe -d Ubuntu -u ubuntu -- /bin/bash -lc "/home/ubuntu/.local/bin/unidesk-d601-autostart task"`；D601 的 WSL distro 名必须写 `Ubuntu`，不能写成未验证的 `Ubuntu-22.04`。 - WSL keepalive：`/home/ubuntu/.local/bin/unidesk-d601-autostart` 使用 `~/.state/unidesk/d601-autostart.lock` 防重复，启动 WSL `sshd`，等待 Docker Desktop daemon 和原生 k3s 就绪，把 `unidesk-provider-gateway-D601` 修正为 `restart always` 且 running，然后调用 `/home/ubuntu/.local/bin/unidesk-microservice-autorecover boot`；进入常驻 watchdog 后每 300 秒重复检查 provider-gateway、Docker 直管服务和 k3s 代管服务。 diff --git a/scripts/cli.ts b/scripts/cli.ts index 28e91eac..828191b1 100644 --- a/scripts/cli.ts +++ b/scripts/cli.ts @@ -4,7 +4,7 @@ import { isRebuildableService, rebuildService, stackLogs, stackStatus, startStac import { parseE2ERunOptions, runE2E } from "./src/e2e"; import { emitError, emitJson } from "./src/output"; import { jobWithTail, listJobs, listJobsSummary, readJob, runJob } from "./src/jobs"; -import { checkHelp, parseCheckOptions, runChecks } from "./src/check"; +import { checkHelp, parseCheckOptions, runChecks, runRecoveryGuardrailsCheck } from "./src/check"; import { runSsh } from "./src/ssh"; import { autoRemoteCiPublishUserServiceDryRunPlan, extractRemoteCliOptions, runRemoteCli } from "./src/remote"; import { runMicroserviceCommand } from "./src/microservices"; @@ -291,6 +291,12 @@ async function main(): Promise { emitJson(commandName, checkHelp()); return; } + if (sub === "recovery-guardrails") { + const result = runRecoveryGuardrailsCheck(config); + emitJson(commandName, result, result.ok); + if (!result.ok) process.exitCode = 1; + return; + } const result = runChecks(config, parseCheckOptions(args.slice(1))); emitJson(commandName, result, result.ok); if (!result.ok) process.exitCode = 1; diff --git a/scripts/d601-recovery-guardrails-contract-test.ts b/scripts/d601-recovery-guardrails-contract-test.ts new file mode 100644 index 00000000..76a2b698 --- /dev/null +++ b/scripts/d601-recovery-guardrails-contract-test.ts @@ -0,0 +1,157 @@ +import { readConfig } from "./src/config"; +import { extractHostPathEntries, parseContainerCreatingReport, parseProcMounts, runD601RecoveryGuardrails, type RecoveryGuardrailsFixture } from "./src/recovery-guardrails"; + +type JsonRecord = Record; + +function assertCondition(condition: unknown, message: string, detail: unknown = {}): void { + if (!condition) throw new Error(`${message}: ${JSON.stringify(detail)}`); +} + +const malformedProcMounts = [ + "proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0", + "drvfs /Docker/host 9p rw,dirsync,aname=drvfs;path=C:\\Program Files\\Docker\\Docker Desktop\\host 0 0", +].join("\n"); + +const manifest = ` +apiVersion: apps/v1 +kind: Deployment +metadata: + name: code-queue-scheduler-dev + namespace: unidesk-dev +spec: + template: + spec: + volumes: + - name: repo + hostPath: + path: /home/ubuntu/unidesk-dev-code-queue-deploy/code-queue + type: Directory + - name: state + hostPath: + path: /home/ubuntu/unidesk-dev-code-queue-deploy/state/code-queue + type: DirectoryOrCreate +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: mdtodo-dev + namespace: unidesk-dev +spec: + template: + spec: + volumes: + - name: workspace + hostPath: + path: /home/ubuntu/unidesk-dev-mdtodo-workspace + type: Directory + - name: logs + hostPath: + path: /home/ubuntu/cq-deploy/.state/mdtodo-dev/logs + type: DirectoryOrCreate +`; + +const kubectlPods = { + items: [ + { + metadata: { namespace: "unidesk-dev", name: "code-queue-scheduler-dev-abc" }, + status: { + containerStatuses: [{ + name: "code-queue", + state: { + waiting: { + reason: "ContainerCreating", + message: "MountVolume.SetUp failed for volume \"repo\": hostPath type check failed: /home/ubuntu/unidesk-dev-code-queue-deploy/code-queue is not a directory", + }, + }, + }], + }, + }, + { + metadata: { namespace: "unidesk-dev", name: "mdtodo-dev-def" }, + status: { + containerStatuses: [{ + name: "mdtodo", + state: { waiting: { reason: "ContainerCreating", message: "" } }, + }], + }, + }, + ], +}; + +const staleCriPods = { + items: [ + { + id: "sandbox-stale", + metadata: { name: "code-queue-scheduler-dev-abc", namespace: "unidesk-dev" }, + state: "SANDBOX_NOTREADY", + createdAt: "2026-05-23T09:00:00.000Z", + }, + ], +}; + +export function runD601RecoveryGuardrailsContract(): JsonRecord { + const procMounts = parseProcMounts(malformedProcMounts, "fixture:/proc/mounts"); + assertCondition(procMounts.ok === false, "malformed /proc/mounts fixture should fail", procMounts); + assertCondition(procMounts.malformedLines.length === 1, "one malformed mount line expected", procMounts); + assertCondition(procMounts.malformedLines[0]?.dockerDesktopHost9p === true, "malformed line should be classified as Docker Desktop /Docker/host 9p", procMounts); + + const entries = extractHostPathEntries("fixture.k8s.yaml", manifest); + assertCondition(entries.some((entry) => entry.hostPath === "/home/ubuntu/unidesk-dev-code-queue-deploy/code-queue"), "hostPath parser should extract repo path", entries); + assertCondition(entries.some((entry) => entry.hostPath === "/home/ubuntu/unidesk-dev-mdtodo-workspace"), "hostPath parser should extract MDTODO workspace path", entries); + + const fixture: RecoveryGuardrailsFixture = { + observedAt: "2026-05-23T10:00:00.000Z", + procMountsText: malformedProcMounts, + criPodsJsonText: JSON.stringify(staleCriPods), + kubectlPodsJsonText: JSON.stringify(kubectlPods), + deployWorktreePath: "/home/ubuntu/unidesk-code-queue-deploy", + compatibilitySymlinkPath: "/home/ubuntu/cq-deploy", + manifestPaths: ["fixture.k8s.yaml"], + manifestTexts: { "fixture.k8s.yaml": manifest }, + pathStates: { + "/home/ubuntu/unidesk-code-queue-deploy": { kind: "missing" }, + "/home/ubuntu/cq-deploy": { kind: "symlink", target: "/home/ubuntu/unidesk-code-queue-deploy", targetExists: false }, + "/home/ubuntu/unidesk-dev-code-queue-deploy/code-queue": { kind: "missing" }, + "/home/ubuntu/unidesk-dev-code-queue-deploy/state/code-queue": { kind: "missing" }, + "/home/ubuntu/unidesk-dev-mdtodo-workspace": { kind: "missing" }, + "/home/ubuntu/cq-deploy/.state/mdtodo-dev/logs": { kind: "missing" }, + }, + }; + const result = runD601RecoveryGuardrails(readConfig(), fixture); + assertCondition(result.scope.liveMutationAllowed === false && result.mutation === false, "guardrail result must be read-only", result.scope); + assertCondition(result.ok === false, "fixture should produce red guardrails", result.redlineSummary); + assertCondition(result.redlineSummary.requiresManualHostHotfix.includes("proc-mounts-malformed-kubelet-risk"), "malformed mount risk should require manual host hotfix", result.redlineSummary); + assertCondition(result.redlineSummary.requiresManualHostHotfix.includes("code-queue-deploy-worktree-not-ready"), "missing worktree/symlink target should require manual host hotfix", result.redlineSummary); + assertCondition(result.redlineSummary.requiresManualHostHotfix.includes("required-hostpaths-not-ready"), "missing hostPath should require manual host hotfix", result.redlineSummary); + assertCondition(result.redlineSummary.requiresManualHostHotfix.includes("opaque-containercreating-hostpath-risk"), "ContainerCreating hostPath risk should require manual host hotfix", result.redlineSummary); + assertCondition(result.checks.codeQueueDeployWorktree.canonicalPath.exists === false, "canonical deploy worktree should report missing", result.checks.codeQueueDeployWorktree); + assertCondition(result.checks.codeQueueDeployWorktree.compatibilitySymlink.isSymlink === true, "compat symlink should be visible", result.checks.codeQueueDeployWorktree); + assertCondition(result.checks.codeQueueDeployWorktree.compatibilitySymlink.targetExists === false, "compat symlink target should report missing", result.checks.codeQueueDeployWorktree); + assertCondition(result.checks.hostPaths.missing.some((problem) => problem.entry.hostPath === "/home/ubuntu/unidesk-dev-code-queue-deploy/code-queue"), "missing Directory hostPath should be red", result.checks.hostPaths.missing); + assertCondition(result.checks.hostPaths.directoryOrCreateMissing.some((problem) => problem.entry.hostPath === "/home/ubuntu/unidesk-dev-code-queue-deploy/state/code-queue"), "DirectoryOrCreate missing should be surfaced separately", result.checks.hostPaths.directoryOrCreateMissing); + assertCondition(result.checks.mdtodoAdjacentHostPaths.opaqueRisk === true, "MDTODO adjacent hostPath readiness should be opaque when workspace/logs missing", result.checks.mdtodoAdjacentHostPaths); + assertCondition(result.checks.criSandboxes.staleNotReadyCount === 1, "stale CRI sandbox should be counted", result.checks.criSandboxes); + assertCondition(result.checks.containerCreating.hostPathContainerCreating.length === 1, "explicit hostPath ContainerCreating should be classified", result.checks.containerCreating); + assertCondition(result.checks.containerCreating.opaqueContainerCreating.length === 1, "empty-message ContainerCreating should be classified as opaque when hostPaths are missing", result.checks.containerCreating); + assertCondition(result.redlineSummary.forbiddenAutomaticActions.includes("crictl rmp"), "CRI deletion should be explicitly forbidden", result.redlineSummary); + assertCondition(result.redlineSummary.forbiddenAutomaticActions.includes("systemctl restart k3s"), "k3s restart should be explicitly forbidden", result.redlineSummary); + + const containerCreatingOnly = parseContainerCreatingReport(JSON.stringify(kubectlPods), result.checks.hostPaths, "fixture", null); + assertCondition(containerCreatingOnly.ok === false, "ContainerCreating parser should fail when opaque/hostPath findings exist", containerCreatingOnly); + + return { + ok: true, + checks: [ + "malformed /proc/mounts Docker Desktop /Docker/host 9p line detected", + "missing Code Queue worktree symlink and target reported", + "missing hostPath and DirectoryOrCreate readiness reported", + "opaque and explicit hostPath ContainerCreating classified", + "stale CRI sandbox count reported without cleanup", + "destructive prune/reset/restart/delete actions forbidden", + ], + }; +} + +if (import.meta.main) { + process.stdout.write(`${JSON.stringify(runD601RecoveryGuardrailsContract(), null, 2)}\n`); +} diff --git a/scripts/src/check.ts b/scripts/src/check.ts index a71cc949..ae11ec83 100644 --- a/scripts/src/check.ts +++ b/scripts/src/check.ts @@ -3,6 +3,7 @@ import { extname } from "node:path"; import { runCommand } from "./command"; import { type UniDeskConfig, repoRoot, rootPath } from "./config"; import { composeConfig } from "./docker"; +import { compactD601RecoveryGuardrails, runD601RecoveryGuardrails } from "./recovery-guardrails"; interface CheckItem { name: string; @@ -27,6 +28,7 @@ const syntaxFiles = [ "scripts/src/e2e.ts", "scripts/src/help.ts", "scripts/src/commander.ts", + "scripts/src/recovery-guardrails.ts", "scripts/src/server-cleanup.ts", "scripts/src/remote.ts", "scripts/host-codex-commander-contract-test.ts", @@ -42,6 +44,7 @@ const syntaxFiles = [ "scripts/code-queue-submit-summary-contract-test.ts", "scripts/code-queue-cli-read-terminal-contract-test.ts", "scripts/code-queue-gh-auth-redaction-contract-test.ts", + "scripts/d601-recovery-guardrails-contract-test.ts", "scripts/microservice-health-output-contract-test.ts", "scripts/code-queue-supervisor-disclosure-contract-test.ts", "scripts/code-queue-commander-view-contract-test.ts", @@ -70,6 +73,7 @@ export interface CheckOptions { components: boolean; compose: boolean; logs: boolean; + recoveryGuardrails: boolean; rust: boolean; } @@ -80,6 +84,7 @@ const defaultCheckOptions: CheckOptions = { components: false, compose: false, logs: false, + recoveryGuardrails: false, rust: false, }; @@ -87,7 +92,10 @@ export function checkHelp(): Record { return { ok: true, command: "check", - usage: "bun scripts/cli.ts check [--syntax-only|--full|--files|--scripts-typecheck|--components|--compose|--logs|--rust]", + usage: [ + "bun scripts/cli.ts check [--syntax-only|--full|--files|--scripts-typecheck|--components|--compose|--logs|--recovery-guardrails|--rust]", + "bun scripts/cli.ts check recovery-guardrails", + ], defaultMode: "syntax/config only; Rust is never compiled on the master server by default", options: [ { name: "--syntax-only|--basic", description: "Run only config validation, Bun version and TypeScript syntax transpile." }, @@ -97,12 +105,18 @@ export function checkHelp(): Record { { name: "--components", description: "Run component TypeScript typecheck." }, { name: "--compose", description: "Render Docker Compose config." }, { name: "--logs", description: "Check unified log rotation policy." }, + { name: "--recovery-guardrails", description: "Run D601 k3s/Code Queue reboot recovery diagnostics in read-only mode." }, { name: "--rust", description: "Run cargo check only when UNIDESK_D601_RUST_CHECK=1 is set inside D601 CI/dev execution." }, ], rustBoundary: { masterServer: "do not run cargo check/build here", d601: "use deploy apply --env dev --service backend-core and CI with UNIDESK_D601_RUST_CHECK=1", }, + recoveryGuardrailsBoundary: { + command: "bun scripts/cli.ts check recovery-guardrails", + mutation: false, + forbidden: ["restart k3s", "delete CRI sandboxes or pods", "modify hostPath directories", "deploy/rollout", "destructive prune/reset"], + }, }; } @@ -116,6 +130,7 @@ export function parseCheckOptions(args: string[]): CheckOptions { options.components = true; options.compose = true; options.logs = true; + options.recoveryGuardrails = true; } else if (arg === "--files") { options.files = true; } else if (arg === "--scripts-typecheck") { @@ -126,6 +141,8 @@ export function parseCheckOptions(args: string[]): CheckOptions { options.compose = true; } else if (arg === "--logs") { options.logs = true; + } else if (arg === "--recovery-guardrails") { + options.recoveryGuardrails = true; } else if (arg === "--rust") { options.rust = true; } else if (arg === "--basic" || arg === "--syntax-only") { @@ -277,6 +294,10 @@ function skippedItem(name: string, reason: string, enableWith: string): CheckIte return { name, ok: true, detail: { skipped: true, reason, enableWith } }; } +export function runRecoveryGuardrailsCheck(config: UniDeskConfig): ReturnType { + return compactD601RecoveryGuardrails(runD601RecoveryGuardrails(config)); +} + export function runChecks(config: UniDeskConfig, options: CheckOptions = defaultCheckOptions): { ok: boolean; mode: string; options: CheckOptions; items: CheckItem[] } { const items: CheckItem[] = [ { name: "config:validated", ok: true, detail: { project: config.project.name, runtime: config.runtime } }, @@ -352,8 +373,10 @@ export function runChecks(config: UniDeskConfig, options: CheckOptions = default fileItem("scripts/server-cleanup-plan-contract-test.ts"), fileItem("scripts/src/artifact-registry.ts"), fileItem("scripts/src/server-cleanup.ts"), + fileItem("scripts/src/recovery-guardrails.ts"), fileItem("scripts/src/auth-broker.ts"), fileItem("scripts/auth-broker-contract-test.ts"), + fileItem("scripts/d601-recovery-guardrails-contract-test.ts"), fileItem("src/components/microservices/auth-broker/Cargo.toml"), fileItem("src/components/microservices/auth-broker/Dockerfile"), fileItem("src/components/microservices/auth-broker/src/main.rs"), @@ -403,6 +426,7 @@ export function runChecks(config: UniDeskConfig, options: CheckOptions = default items.push(commandItem("gh:pr-contract", ["bun", "scripts/gh-cli-pr-contract-test.ts"], 30_000)); items.push(commandItem("playwright:cli-wrapper-contract", ["bun", "scripts/playwright-cli-contract-test.ts"], 30_000)); items.push(commandItem("auth-broker:p0-contract", ["bun", "scripts/auth-broker-contract-test.ts"], 30_000)); + items.push(commandItem("d601:recovery-guardrails-contract", ["bun", "scripts/d601-recovery-guardrails-contract-test.ts"], 30_000)); } else { items.push(skippedItem("typescript:scripts", "scripts TypeScript typecheck is opt-in", "--scripts-typecheck or --full")); items.push(skippedItem("code-queue:prompt-observation-contract", "prompt observation contract is opt-in with script checks", "--scripts-typecheck or --full")); @@ -439,12 +463,23 @@ export function runChecks(config: UniDeskConfig, options: CheckOptions = default items.push(skippedItem("gh:pr-contract", "GitHub PR CLI contract is opt-in with script checks", "--scripts-typecheck or --full")); items.push(skippedItem("playwright:cli-wrapper-contract", "Playwright wrapper/headless/session contract is opt-in with script checks", "--scripts-typecheck or --full")); items.push(skippedItem("auth-broker:p0-contract", "Auth Broker P0 skeleton and CLI adapter contract is opt-in with script checks", "--scripts-typecheck or --full")); + items.push(skippedItem("d601:recovery-guardrails-contract", "D601 recovery guardrails fixture contract is opt-in with script checks", "--scripts-typecheck or --full")); } if (options.logs) { items.push(unifiedLogRotationItem()); } else { items.push(skippedItem("logs:unified-hourly-rotation", "policy scan is opt-in", "--logs or --full")); } + if (options.recoveryGuardrails) { + const recovery = runRecoveryGuardrailsCheck(config); + items.push({ + name: "d601:recovery-guardrails", + ok: recovery.ok, + detail: recovery, + }); + } else { + items.push(skippedItem("d601:recovery-guardrails", "D601 reboot recovery diagnostics are opt-in and read-only", "--recovery-guardrails or --full")); + } if (options.components) { items.push(commandItem("typescript:components", ["bunx", "tsc", "-p", "src/tsconfig.check.json", "--pretty", "false"], 180_000)); } else { diff --git a/scripts/src/help.ts b/scripts/src/help.ts index e2fadd89..63fbe990 100644 --- a/scripts/src/help.ts +++ b/scripts/src/help.ts @@ -9,7 +9,7 @@ export function rootHelp(): unknown { { command: "help", description: "List supported commands." }, { command: "--main-server-ip ", description: "Run selected commands through the public frontend API; use --main-server-key only for legacy SSH transport." }, { command: "config show", description: "Validate and print config.json as the single source of truth." }, - { command: "check [--full|--files|--scripts-typecheck|--components|--compose|--logs|--rust]", description: "Run the lightweight default syntax/config gate; Rust is opt-in and only allowed from D601 CI/dev execution." }, + { command: "check [--full|--files|--scripts-typecheck|--components|--compose|--logs|--recovery-guardrails|--rust] | check recovery-guardrails", description: "Run the lightweight default syntax/config gate or the low-noise read-only D601 recovery guardrails; Rust is opt-in and only allowed from D601 CI/dev execution." }, { command: "server start", description: "Fire-and-forget build/start for database, backend-core, frontend, provider gateway, and managed main-server user services." }, { command: "server stop", description: "Fire-and-forget docker-compose down for the fixed UniDesk stack." }, { command: "server status", description: "Show fixed ports, containers, service health, and public URLs." }, diff --git a/scripts/src/recovery-guardrails.ts b/scripts/src/recovery-guardrails.ts new file mode 100644 index 00000000..09c7bd3f --- /dev/null +++ b/scripts/src/recovery-guardrails.ts @@ -0,0 +1,1179 @@ +import { lstatSync, readFileSync, readlinkSync, statSync } from "node:fs"; +import { dirname, isAbsolute, resolve } from "node:path"; +import { runCommand, type CommandResult } from "./command"; +import { repoRoot, rootPath, type UniDeskConfig } from "./config"; + +type Severity = "info" | "yellow" | "red"; +type Disposition = "safe-read-only" | "requires-manual-host-hotfix" | "forbidden-automatic-action"; +type PathKind = "missing" | "directory" | "file" | "socket" | "symlink" | "other"; + +export interface RecoveryGuardrailRedline { + id: string; + severity: Severity; + disposition: Disposition; + summary: string; + evidence: Record; + safeReadOnly: boolean; + requiresManualHostHotfix: boolean; + forbiddenAutomaticActions: string[]; + recommendedNext: string[]; +} + +export interface RecoveryGuardrailsResult { + ok: boolean; + surface: "d601-recovery-guardrails"; + mutation: false; + observedAt: string; + scope: { + nodeId: "D601"; + environment: "host-read-only"; + liveMutationAllowed: false; + }; + safeReadOnly: { + filesRead: string[]; + commandsAttempted: string[][]; + note: string; + }; + redlineSummary: { + red: number; + yellow: number; + safeReadOnly: boolean; + requiresManualHostHotfix: string[]; + forbiddenAutomaticActions: string[]; + }; + redlines: RecoveryGuardrailRedline[]; + checks: { + procMounts: ProcMountsReport; + kubeletValidationRisk: KubeletValidationRisk; + criSandboxes: CriSandboxReport; + codeQueueDeployWorktree: WorktreeReadinessReport; + hostPaths: HostPathReadinessReport; + mdtodoAdjacentHostPaths: MdtodoAdjacentReadiness; + containerCreating: ContainerCreatingReport; + }; +} + +export interface RecoveryGuardrailsFixture { + observedAt?: string; + procMountsText?: string; + criPodsJsonText?: string; + kubectlPodsJsonText?: string; + pathStates?: Record; + manifestTexts?: Record; + deployWorktreePath?: string; + compatibilitySymlinkPath?: string; + manifestPaths?: string[]; +} + +export interface PathStateFixture { + kind: PathKind; + target?: string; + targetExists?: boolean; +} + +export interface CompactRecoveryGuardrailsResult { + ok: boolean; + surface: RecoveryGuardrailsResult["surface"]; + mutation: false; + observedAt: string; + scope: RecoveryGuardrailsResult["scope"]; + safeReadOnly: RecoveryGuardrailsResult["safeReadOnly"]; + redlineSummary: RecoveryGuardrailsResult["redlineSummary"]; + redlines: Array>; + checks: { + procMounts: Pick; + kubeletValidationRisk: KubeletValidationRisk; + criSandboxes: Pick; + codeQueueDeployWorktree: WorktreeReadinessReport; + hostPaths: { + ok: boolean; + manifests: CompactHostPathManifest[]; + totalEntries: number; + checkedCount: number; + missing: CompactHostPathProblem[]; + typeMismatches: CompactHostPathProblem[]; + directoryOrCreateMissing: CompactHostPathProblem[]; + sensitiveUserDataBoundaries: string[]; + }; + mdtodoAdjacentHostPaths: MdtodoAdjacentReadiness; + containerCreating: Pick; + }; +} + +interface CompactHostPathProblem { + hostPath: string; + hostPathType: string; + manifest: string; + workload: string; + volumeName: string | null; + problem: string | null; + severity: Severity; + disposition: Disposition; + reason: string; +} + +interface CompactHostPathManifest { + path: string; + readError: string | null; + textBytes: number; +} + +interface ProcMountsReport { + ok: boolean; + source: string; + totalLines: number; + malformedLines: MountLineFinding[]; + dockerDesktopHost9pLines: MountLineFinding[]; + readError: string | null; +} + +interface MountLineFinding { + lineNumber: number; + fieldCount: number; + dockerDesktopHost9p: boolean; + textPreview: string; + reason: string; +} + +interface KubeletValidationRisk { + ok: boolean; + risk: "none" | "kubelet-mount-table-validation-risk"; + severity: Severity; + reason: string; + evidenceLineNumbers: number[]; +} + +interface CriSandboxReport { + ok: boolean; + source: "crictl" | "fixture" | "unavailable"; + command: string[] | null; + commandResult: CommandResultSummary | null; + total: number; + notReadyCount: number; + staleNotReadyCount: number; + unknownAgeCount: number; + staleAfterMinutes: number; + staleSandboxes: CriSandboxSummary[]; + parseError: string | null; +} + +interface CommandResultSummary { + exitCode: number | null; + signal: NodeJS.Signals | null; + timedOut: boolean; + stdoutTail: string; + stderrTail: string; +} + +interface CriSandboxSummary { + id: string; + name: string | null; + namespace: string | null; + state: string; + ageMinutes: number | null; +} + +interface WorktreeReadinessReport { + ok: boolean; + canonicalPath: PathCheck; + compatibilitySymlink: SymlinkCheck; + requiredFor: string[]; +} + +interface SymlinkCheck extends PathCheck { + expectedTarget: string; + targetExists: boolean; + targetKind: PathKind; +} + +interface PathCheck { + path: string; + exists: boolean; + kind: PathKind; + isSymlink: boolean; + symlinkTarget: string | null; + resolvedPath: string | null; + matchesExpectedType: boolean; + expectedType: string; + problem: string | null; +} + +interface HostPathManifest { + path: string; + text: string; + readError: string | null; +} + +interface HostPathEntry { + manifest: string; + documentIndex: number; + kind: string | null; + name: string | null; + namespace: string | null; + volumeName: string | null; + hostPath: string; + hostPathType: string; +} + +interface HostPathProblem { + entry: HostPathEntry; + check: PathCheck; + severity: Severity; + disposition: Disposition; + reason: string; +} + +interface HostPathReadinessReport { + ok: boolean; + manifests: HostPathManifest[]; + totalEntries: number; + checkedEntries: HostPathEntry[]; + missing: HostPathProblem[]; + typeMismatches: HostPathProblem[]; + directoryOrCreateMissing: HostPathProblem[]; + sensitiveUserDataBoundaries: string[]; +} + +interface MdtodoAdjacentReadiness { + ok: boolean; + workspacePaths: string[]; + logPaths: string[]; + missingPaths: string[]; + opaqueRisk: boolean; + userDataBoundary: string; +} + +interface ContainerCreatingReport { + ok: boolean; + source: "kubectl" | "fixture" | "unavailable"; + command: string[] | null; + commandResult: CommandResultSummary | null; + totalContainerCreating: number; + opaqueContainerCreating: ContainerCreatingFinding[]; + hostPathContainerCreating: ContainerCreatingFinding[]; + parseError: string | null; +} + +interface ContainerCreatingFinding { + namespace: string; + pod: string; + container: string; + reason: string; + messagePreview: string; + classification: "opaque-containercreating-hostpath-risk" | "hostpath-mount-failure" | "containercreating-observed"; +} + +const staleSandboxAfterMinutes = 30; +const defaultCompatibilitySymlinkPath = "/home/ubuntu/cq-deploy"; +const defaultManifestPaths = [ + "src/components/microservices/k3sctl-adapter/k3s/code-queue.k8s.yaml", + "src/components/microservices/k3sctl-adapter/k3s/dev/unidesk-dev-code-queue.k8s.yaml", + "src/components/microservices/k3sctl-adapter/k3s/mdtodo.k8s.yaml", + "src/components/microservices/k3sctl-adapter/k3s/dev/unidesk-dev-mdtodo.k8s.yaml", +]; +const forbiddenAutomaticActions = [ + "systemctl restart k3s", + "service k3s restart", + "kubectl delete pod", + "crictl rmp", + "crictl rm", + "docker system prune", + "docker volume prune", + "rm -rf /home/ubuntu", + "git reset --hard live worktree", +]; + +export function runD601RecoveryGuardrails(config: UniDeskConfig, fixture: RecoveryGuardrailsFixture = {}): RecoveryGuardrailsResult { + const observedAt = fixture.observedAt ?? new Date().toISOString(); + const filesRead: string[] = []; + const commandsAttempted: string[][] = []; + const codeQueueService = config.microservices.find((service) => service.id === "code-queue"); + const deployWorktreePath = fixture.deployWorktreePath ?? codeQueueService?.development.worktreePath ?? "/home/ubuntu/unidesk-code-queue-deploy"; + const compatibilitySymlinkPath = fixture.compatibilitySymlinkPath ?? defaultCompatibilitySymlinkPath; + const manifestPaths = fixture.manifestPaths ?? defaultManifestPaths; + + const procMounts = fixture.procMountsText === undefined + ? collectProcMounts("/proc/mounts", filesRead) + : parseProcMounts(fixture.procMountsText, "fixture:/proc/mounts"); + const kubeletValidationRisk = classifyKubeletValidationRisk(procMounts); + const manifests = manifestPaths.map((manifestPath) => collectManifest(manifestPath, fixture, filesRead)); + const hostPathEntries = manifests.flatMap((manifest) => manifest.readError === null ? extractHostPathEntries(manifest.path, manifest.text) : []); + const hostPaths = evaluateHostPaths(hostPathEntries, manifests, fixture.pathStates); + const worktree = evaluateWorktreeReadiness(deployWorktreePath, compatibilitySymlinkPath, fixture.pathStates); + const mdtodo = evaluateMdtodoAdjacentHostPaths(hostPaths); + const criSandboxes = fixture.criPodsJsonText === undefined + ? collectCriSandboxes(observedAt, commandsAttempted) + : parseCriSandboxReport(fixture.criPodsJsonText, observedAt, "fixture", null); + const containerCreating = fixture.kubectlPodsJsonText === undefined + ? collectContainerCreating(hostPaths, commandsAttempted) + : parseContainerCreatingReport(fixture.kubectlPodsJsonText, hostPaths, "fixture", null); + const redlines = buildRedlines({ + procMounts, + kubeletValidationRisk, + criSandboxes, + worktree, + hostPaths, + mdtodo, + containerCreating, + }); + const redlineSummary = { + red: redlines.filter((item) => item.severity === "red").length, + yellow: redlines.filter((item) => item.severity === "yellow").length, + safeReadOnly: true, + requiresManualHostHotfix: redlines.filter((item) => item.requiresManualHostHotfix).map((item) => item.id), + forbiddenAutomaticActions: Array.from(new Set(redlines.flatMap((item) => item.forbiddenAutomaticActions))), + }; + return { + ok: redlineSummary.red === 0, + surface: "d601-recovery-guardrails", + mutation: false, + observedAt, + scope: { + nodeId: "D601", + environment: "host-read-only", + liveMutationAllowed: false, + }, + safeReadOnly: { + filesRead, + commandsAttempted, + note: "Diagnostics only read files, inspect path metadata, and run optional bounded read-only kubectl/crictl probes. They never delete CRI sandboxes, prune Docker state, edit hostPath directories, restart k3s, apply manifests, or rollout workloads.", + }, + redlineSummary, + redlines, + checks: { + procMounts, + kubeletValidationRisk, + criSandboxes, + codeQueueDeployWorktree: worktree, + hostPaths, + mdtodoAdjacentHostPaths: mdtodo, + containerCreating, + }, + }; +} + +export function compactD601RecoveryGuardrails(result: RecoveryGuardrailsResult): CompactRecoveryGuardrailsResult { + return { + ok: result.ok, + surface: result.surface, + mutation: false, + observedAt: result.observedAt, + scope: result.scope, + safeReadOnly: result.safeReadOnly, + redlineSummary: result.redlineSummary, + redlines: result.redlines.map((item) => ({ + id: item.id, + severity: item.severity, + disposition: item.disposition, + summary: item.summary, + recommendedNext: item.recommendedNext, + })), + checks: { + procMounts: result.checks.procMounts, + kubeletValidationRisk: result.checks.kubeletValidationRisk, + criSandboxes: { + ok: result.checks.criSandboxes.ok, + source: result.checks.criSandboxes.source, + command: result.checks.criSandboxes.command, + total: result.checks.criSandboxes.total, + notReadyCount: result.checks.criSandboxes.notReadyCount, + staleNotReadyCount: result.checks.criSandboxes.staleNotReadyCount, + unknownAgeCount: result.checks.criSandboxes.unknownAgeCount, + staleAfterMinutes: result.checks.criSandboxes.staleAfterMinutes, + staleSandboxes: result.checks.criSandboxes.staleSandboxes, + parseError: result.checks.criSandboxes.parseError, + }, + codeQueueDeployWorktree: result.checks.codeQueueDeployWorktree, + hostPaths: { + ok: result.checks.hostPaths.ok, + manifests: result.checks.hostPaths.manifests.map((manifest) => ({ + path: manifest.path, + readError: manifest.readError, + textBytes: Buffer.byteLength(manifest.text, "utf8"), + })), + totalEntries: result.checks.hostPaths.totalEntries, + checkedCount: result.checks.hostPaths.checkedEntries.length, + missing: result.checks.hostPaths.missing.map(compactHostPathProblem), + typeMismatches: result.checks.hostPaths.typeMismatches.map(compactHostPathProblem), + directoryOrCreateMissing: result.checks.hostPaths.directoryOrCreateMissing.map(compactHostPathProblem), + sensitiveUserDataBoundaries: result.checks.hostPaths.sensitiveUserDataBoundaries, + }, + mdtodoAdjacentHostPaths: result.checks.mdtodoAdjacentHostPaths, + containerCreating: { + ok: result.checks.containerCreating.ok, + source: result.checks.containerCreating.source, + command: result.checks.containerCreating.command, + totalContainerCreating: result.checks.containerCreating.totalContainerCreating, + opaqueContainerCreating: result.checks.containerCreating.opaqueContainerCreating, + hostPathContainerCreating: result.checks.containerCreating.hostPathContainerCreating, + parseError: result.checks.containerCreating.parseError, + }, + }, + }; +} + +export function parseProcMounts(text: string, source: string): ProcMountsReport { + const lines = text.split(/\r?\n/u).filter((line) => line.trim().length > 0); + const malformedLines: MountLineFinding[] = []; + const dockerDesktopHost9pLines: MountLineFinding[] = []; + lines.forEach((line, index) => { + const tokens = line.trim().split(/\s+/u); + const dockerDesktopHost9p = line.includes("/Docker/host") && /(?:^|\s)9p(?:\s|$)/u.test(line); + const finding: MountLineFinding = { + lineNumber: index + 1, + fieldCount: tokens.length, + dockerDesktopHost9p, + textPreview: safePreview(line, 260), + reason: tokens.length === 6 ? "docker-desktop-host-9p-line-observed" : "proc-mounts-line-field-count-not-six", + }; + if (dockerDesktopHost9p) dockerDesktopHost9pLines.push(finding); + if (tokens.length !== 6) malformedLines.push(finding); + }); + return { + ok: malformedLines.length === 0, + source, + totalLines: lines.length, + malformedLines, + dockerDesktopHost9pLines, + readError: null, + }; +} + +export function classifyKubeletValidationRisk(report: ProcMountsReport): KubeletValidationRisk { + const malformedDockerHost = report.malformedLines.filter((line) => line.dockerDesktopHost9p); + if (report.malformedLines.length === 0) { + return { + ok: true, + risk: "none", + severity: "info", + reason: report.dockerDesktopHost9pLines.length === 0 + ? "No malformed /proc/mounts lines were detected." + : "Docker Desktop /Docker/host 9p lines were observed but field counts are valid.", + evidenceLineNumbers: [], + }; + } + return { + ok: false, + risk: "kubelet-mount-table-validation-risk", + severity: "red", + reason: malformedDockerHost.length > 0 + ? "Malformed Docker Desktop /Docker/host 9p mount rows can make kubelet mount-table validation reject hostPath setup after reboot." + : "Malformed /proc/mounts rows can make kubelet mount-table validation unreliable.", + evidenceLineNumbers: report.malformedLines.map((line) => line.lineNumber), + }; +} + +export function parseCriSandboxReport( + jsonText: string, + observedAt: string, + source: "crictl" | "fixture", + commandResult: CommandResultSummary | null, +): CriSandboxReport { + try { + const parsed = JSON.parse(jsonText) as unknown; + const items = sandboxItems(parsed); + const observedMs = Date.parse(observedAt); + const notReady = items.filter((item) => { + const state = String(recordField(item, "state") ?? "").toUpperCase(); + return state.length === 0 || (!state.includes("READY") && state !== "SANDBOX_READY" && state !== "READY") || state.includes("NOTREADY"); + }); + const summaries = notReady.map((item) => sandboxSummary(item, observedMs)); + const staleSandboxes = summaries.filter((item) => item.ageMinutes === null || item.ageMinutes >= staleSandboxAfterMinutes); + return { + ok: staleSandboxes.length === 0, + source, + command: source === "crictl" ? ["crictl", "pods", "-o", "json"] : null, + commandResult, + total: items.length, + notReadyCount: notReady.length, + staleNotReadyCount: staleSandboxes.length, + unknownAgeCount: summaries.filter((item) => item.ageMinutes === null).length, + staleAfterMinutes: staleSandboxAfterMinutes, + staleSandboxes, + parseError: null, + }; + } catch (error) { + return { + ok: false, + source, + command: source === "crictl" ? ["crictl", "pods", "-o", "json"] : null, + commandResult, + total: 0, + notReadyCount: 0, + staleNotReadyCount: 0, + unknownAgeCount: 0, + staleAfterMinutes: staleSandboxAfterMinutes, + staleSandboxes: [], + parseError: errorMessage(error), + }; + } +} + +export function extractHostPathEntries(manifestPath: string, text: string): HostPathEntry[] { + return splitManifestDocuments(text).flatMap((doc) => { + const lines = doc.raw.split(/\r?\n/u); + const entries: HostPathEntry[] = []; + for (let index = 0; index < lines.length; index += 1) { + if (!/^\s*hostPath:\s*$/u.test(lines[index] ?? "")) continue; + const volumeName = findVolumeName(lines, index); + const path = findScalarAfter(lines, index, "path"); + if (path === null) continue; + entries.push({ + manifest: manifestPath, + documentIndex: doc.index, + kind: doc.kind, + name: doc.name, + namespace: doc.namespace, + volumeName, + hostPath: path, + hostPathType: findScalarAfter(lines, index, "type") ?? "Unset", + }); + } + return entries; + }); +} + +export function evaluateHostPaths( + entries: HostPathEntry[], + manifests: HostPathManifest[], + fixtures: Record | undefined = undefined, +): HostPathReadinessReport { + const checkedEntries = dedupeHostPathEntries(entries); + const problems = checkedEntries.map((entry) => hostPathProblem(entry, checkPath(entry.hostPath, entry.hostPathType, fixtures))).filter((item): item is HostPathProblem => item !== null); + return { + ok: problems.filter((problem) => problem.severity === "red").length === 0, + manifests, + totalEntries: entries.length, + checkedEntries, + missing: problems.filter((problem) => problem.check.problem === "missing"), + typeMismatches: problems.filter((problem) => problem.check.problem === "type-mismatch"), + directoryOrCreateMissing: problems.filter((problem) => problem.check.problem === "directory-or-create-missing"), + sensitiveUserDataBoundaries: [ + "/home/ubuntu/.codex/auth.json", + "/home/ubuntu/.ssh", + "/home/ubuntu/.agents/skills", + "/home/ubuntu/cq-deploy/.state/mdtodo-workspace", + "/home/ubuntu/unidesk-dev-mdtodo-workspace", + ], + }; +} + +export function evaluateWorktreeReadiness( + canonicalPath: string, + compatibilitySymlinkPath = defaultCompatibilitySymlinkPath, + fixtures: Record | undefined = undefined, +): WorktreeReadinessReport { + const canonicalPathCheck = checkPath(canonicalPath, "Directory", fixtures); + const symlink = checkPath(compatibilitySymlinkPath, "Directory", fixtures); + const expectedTarget = canonicalPath; + const targetPath = symlink.resolvedPath ?? resolveSymlinkTarget(compatibilitySymlinkPath, symlink.symlinkTarget); + const targetCheck = targetPath === null ? null : checkPath(targetPath, "Directory", fixtures); + return { + ok: canonicalPathCheck.matchesExpectedType && symlink.matchesExpectedType && (symlink.isSymlink ? targetCheck?.matchesExpectedType === true : true), + canonicalPath: canonicalPathCheck, + compatibilitySymlink: { + ...symlink, + expectedTarget, + targetExists: targetCheck?.exists ?? false, + targetKind: targetCheck?.kind ?? "missing", + }, + requiredFor: [ + "config.json microservices.code-queue.development.worktreePath", + "production code-queue /root/unidesk and /app hostPath mapping through /home/ubuntu/cq-deploy", + "MDTODO production workspace/log hostPath adjacency under /home/ubuntu/cq-deploy/.state", + ], + }; +} + +export function parseContainerCreatingReport( + jsonText: string, + hostPaths: HostPathReadinessReport, + source: "kubectl" | "fixture", + commandResult: CommandResultSummary | null, +): ContainerCreatingReport { + try { + const parsed = JSON.parse(jsonText) as unknown; + const pods = podItems(parsed); + const hostPathNeedles = missingHostPathNeedles(hostPaths); + const findings = pods.flatMap((pod) => containerCreatingFindings(pod, hostPathNeedles)); + const opaque = findings.filter((item) => item.classification === "opaque-containercreating-hostpath-risk"); + const hostPathFailures = findings.filter((item) => item.classification === "hostpath-mount-failure"); + return { + ok: opaque.length === 0 && hostPathFailures.length === 0, + source, + command: source === "kubectl" ? ["kubectl", "get", "pods", "-A", "-o", "json"] : null, + commandResult, + totalContainerCreating: findings.length, + opaqueContainerCreating: opaque, + hostPathContainerCreating: hostPathFailures, + parseError: null, + }; + } catch (error) { + return { + ok: false, + source, + command: source === "kubectl" ? ["kubectl", "get", "pods", "-A", "-o", "json"] : null, + commandResult, + totalContainerCreating: 0, + opaqueContainerCreating: [], + hostPathContainerCreating: [], + parseError: errorMessage(error), + }; + } +} + +function collectProcMounts(path: string, filesRead: string[]): ProcMountsReport { + filesRead.push(path); + try { + return parseProcMounts(readFileSync(path, "utf8"), path); + } catch (error) { + return { + ok: false, + source: path, + totalLines: 0, + malformedLines: [], + dockerDesktopHost9pLines: [], + readError: errorMessage(error), + }; + } +} + +function collectManifest(manifestPath: string, fixture: RecoveryGuardrailsFixture, filesRead: string[]): HostPathManifest { + if (fixture.manifestTexts?.[manifestPath] !== undefined) { + return { path: manifestPath, text: fixture.manifestTexts[manifestPath] ?? "", readError: null }; + } + const absolute = rootPath(manifestPath); + filesRead.push(absolute); + try { + return { path: manifestPath, text: readFileSync(absolute, "utf8"), readError: null }; + } catch (error) { + return { path: manifestPath, text: "", readError: errorMessage(error) }; + } +} + +function collectCriSandboxes(observedAt: string, commandsAttempted: string[][]): CriSandboxReport { + const probe = runCommand(["sh", "-lc", "command -v crictl >/dev/null 2>&1"], repoRoot, { timeoutMs: 5_000 }); + if (probe.exitCode !== 0) { + return unavailableCriSandboxReport("crictl binary is not available in this environment."); + } + const command = ["crictl", "pods", "-o", "json"]; + commandsAttempted.push(command); + const result = runCommand(command, repoRoot, { timeoutMs: 10_000 }); + const summary = summarizeCommandResult(result); + if (result.exitCode !== 0) { + return { + ...unavailableCriSandboxReport("crictl read-only pod sandbox probe failed."), + command, + commandResult: summary, + }; + } + return parseCriSandboxReport(result.stdout, observedAt, "crictl", summary); +} + +function collectContainerCreating(hostPaths: HostPathReadinessReport, commandsAttempted: string[][]): ContainerCreatingReport { + const probe = runCommand(["sh", "-lc", "command -v kubectl >/dev/null 2>&1"], repoRoot, { timeoutMs: 5_000 }); + if (probe.exitCode !== 0) { + return unavailableContainerCreatingReport("kubectl binary is not available in this environment."); + } + const command = ["kubectl", "get", "pods", "-A", "-o", "json"]; + commandsAttempted.push(command); + const result = runCommand(command, repoRoot, { timeoutMs: 10_000 }); + const summary = summarizeCommandResult(result); + if (result.exitCode !== 0) { + return { + ...unavailableContainerCreatingReport("kubectl read-only pod status probe failed."), + command, + commandResult: summary, + }; + } + return parseContainerCreatingReport(result.stdout, hostPaths, "kubectl", summary); +} + +function unavailableCriSandboxReport(reason: string): CriSandboxReport { + return { + ok: true, + source: "unavailable", + command: null, + commandResult: null, + total: 0, + notReadyCount: 0, + staleNotReadyCount: 0, + unknownAgeCount: 0, + staleAfterMinutes: staleSandboxAfterMinutes, + staleSandboxes: [], + parseError: reason, + }; +} + +function unavailableContainerCreatingReport(reason: string): ContainerCreatingReport { + return { + ok: true, + source: "unavailable", + command: null, + commandResult: null, + totalContainerCreating: 0, + opaqueContainerCreating: [], + hostPathContainerCreating: [], + parseError: reason, + }; +} + +function evaluateMdtodoAdjacentHostPaths(hostPaths: HostPathReadinessReport): MdtodoAdjacentReadiness { + const mdtodoEntries = hostPaths.checkedEntries.filter((entry) => entry.hostPath.includes("mdtodo")); + const workspacePaths = mdtodoEntries.filter((entry) => entry.volumeName === "workspace" || entry.hostPath.includes("workspace")).map((entry) => entry.hostPath); + const logPaths = mdtodoEntries.filter((entry) => entry.volumeName === "logs" || entry.hostPath.includes("/logs")).map((entry) => entry.hostPath); + const missingPaths = Array.from(new Set([ + ...hostPaths.missing.filter((problem) => problem.entry.hostPath.includes("mdtodo")).map((problem) => problem.entry.hostPath), + ...hostPaths.directoryOrCreateMissing.filter((problem) => problem.entry.hostPath.includes("mdtodo")).map((problem) => problem.entry.hostPath), + ])); + return { + ok: missingPaths.length === 0 && workspacePaths.length > 0, + workspacePaths: Array.from(new Set(workspacePaths)), + logPaths: Array.from(new Set(logPaths)), + missingPaths, + opaqueRisk: missingPaths.length > 0, + userDataBoundary: "MDTODO workspace hostPaths are user data. Recovery diagnostics may report readiness, but must not create, delete, prune, reset, or replace the workspace automatically.", + }; +} + +function buildRedlines(reports: { + procMounts: ProcMountsReport; + kubeletValidationRisk: KubeletValidationRisk; + criSandboxes: CriSandboxReport; + worktree: WorktreeReadinessReport; + hostPaths: HostPathReadinessReport; + mdtodo: MdtodoAdjacentReadiness; + containerCreating: ContainerCreatingReport; +}): RecoveryGuardrailRedline[] { + const redlines: RecoveryGuardrailRedline[] = []; + if (!reports.procMounts.ok || !reports.kubeletValidationRisk.ok) { + redlines.push(redline( + "proc-mounts-malformed-kubelet-risk", + "red", + "requires-manual-host-hotfix", + "Malformed /proc/mounts lines create kubelet hostPath validation risk.", + { + source: reports.procMounts.source, + malformedLines: reports.procMounts.malformedLines, + kubeletValidationRisk: reports.kubeletValidationRisk, + }, + [ + "Host commander or user should repair the Docker Desktop/WSL mount-table condition and then run this read-only check again.", + "Do not restart k3s from Code Queue or this CLI without explicit host authorization.", + ], + )); + } + if (reports.criSandboxes.staleNotReadyCount > 0 || reports.criSandboxes.parseError !== null && reports.criSandboxes.source !== "unavailable") { + redlines.push(redline( + "stale-cri-sandboxes-observed", + reports.criSandboxes.staleNotReadyCount > 0 ? "red" : "yellow", + "requires-manual-host-hotfix", + "Stale or unreadable CRI pod sandbox state needs host-side review; automatic deletion is forbidden.", + { + source: reports.criSandboxes.source, + staleNotReadyCount: reports.criSandboxes.staleNotReadyCount, + unknownAgeCount: reports.criSandboxes.unknownAgeCount, + staleSandboxes: reports.criSandboxes.staleSandboxes, + parseError: reports.criSandboxes.parseError, + }, + [ + "Review CRI sandbox state on D601 host with bounded read-only crictl/kubectl commands.", + "Escalate to host commander/user before any sandbox cleanup.", + ], + )); + } + if (!reports.worktree.ok) { + redlines.push(redline( + "code-queue-deploy-worktree-not-ready", + "red", + "requires-manual-host-hotfix", + "Code Queue deploy worktree or compatibility symlink is missing or unresolved.", + reports.worktree as unknown as Record, + [ + "Restore the Git worktree/symlink on D601 host from pushed Git state.", + "Do not git reset --hard or replace a dirty live worktree without explicit user/host approval.", + ], + )); + } + if (!reports.hostPaths.ok || reports.hostPaths.directoryOrCreateMissing.length > 0) { + redlines.push(redline( + "required-hostpaths-not-ready", + reports.hostPaths.ok ? "yellow" : "red", + "requires-manual-host-hotfix", + "One or more k3s hostPath mounts required by Code Queue or MDTODO are missing or have the wrong type.", + { + missing: reports.hostPaths.missing, + typeMismatches: reports.hostPaths.typeMismatches, + directoryOrCreateMissing: reports.hostPaths.directoryOrCreateMissing, + sensitiveUserDataBoundaries: reports.hostPaths.sensitiveUserDataBoundaries, + }, + [ + "Repair only the specific missing hostPath on D601 host after reviewing whether it is user data, credential material, or cache/log state.", + "Do not mass-create, delete, prune, chmod recursively, or replace hostPath directories from Code Queue.", + ], + )); + } + if (!reports.mdtodo.ok) { + redlines.push(redline( + "mdtodo-adjacent-hostpaths-opaque", + "red", + "requires-manual-host-hotfix", + "MDTODO workspace/log hostPaths are not transparently ready, making ContainerCreating recovery ambiguous.", + reports.mdtodo as unknown as Record, + [ + "Verify the MDTODO workspace source and logs path on D601 host before restarting or rolling pods.", + "Ask the user before touching the MDTODO workspace because it contains user-authored Markdown data.", + ], + )); + } + if (reports.containerCreating.opaqueContainerCreating.length > 0 || reports.containerCreating.hostPathContainerCreating.length > 0) { + redlines.push(redline( + "opaque-containercreating-hostpath-risk", + "red", + "requires-manual-host-hotfix", + "Pods stuck in ContainerCreating have opaque or hostPath-looking evidence.", + { + opaqueContainerCreating: reports.containerCreating.opaqueContainerCreating, + hostPathContainerCreating: reports.containerCreating.hostPathContainerCreating, + }, + [ + "Use kubectl describe/get events on D601 host to confirm the exact mount failure.", + "Do not delete pods or sandboxes automatically; first fix the hostPath or kubelet mount-table condition.", + ], + )); + } + redlines.push(redline( + "destructive-recovery-actions-forbidden", + "yellow", + "forbidden-automatic-action", + "Recovery diagnostics must never perform destructive cleanup or rollout as an automatic follow-up.", + { forbiddenAutomaticActions }, + [ + "Keep this CLI read-only.", + "Use ClaudeQQ/user approval for high-risk host actions such as k3s restart, CRI cleanup, user-data directory repair, or Code Queue runtime intervention.", + ], + )); + return redlines; +} + +function redline( + id: string, + severity: Severity, + disposition: Disposition, + summary: string, + evidence: Record, + recommendedNext: string[], +): RecoveryGuardrailRedline { + return { + id, + severity, + disposition, + summary, + evidence, + safeReadOnly: true, + requiresManualHostHotfix: disposition === "requires-manual-host-hotfix", + forbiddenAutomaticActions, + recommendedNext, + }; +} + +function hostPathProblem(entry: HostPathEntry, check: PathCheck): HostPathProblem | null { + if (check.problem === null) return null; + if (check.problem === "directory-or-create-missing") { + return { + entry, + check, + severity: "yellow", + disposition: "requires-manual-host-hotfix", + reason: "DirectoryOrCreate hostPath is missing; kubelet may create it only if mount-table validation and parent permissions are healthy.", + }; + } + return { + entry, + check, + severity: "red", + disposition: "requires-manual-host-hotfix", + reason: check.problem, + }; +} + +function checkPath(path: string, expectedType: string, fixtures: Record | undefined): PathCheck { + const fixture = fixtures?.[path]; + if (fixture !== undefined) return checkFixturePath(path, expectedType, fixture); + try { + const lstat = lstatSync(path); + const isSymlink = lstat.isSymbolicLink(); + const symlinkTarget = isSymlink ? readlinkSync(path) : null; + const resolvedPath = isSymlink ? resolveSymlinkTarget(path, symlinkTarget) : path; + const stat = statSync(resolvedPath ?? path); + const kind = stat.isDirectory() ? "directory" : stat.isFile() ? "file" : stat.isSocket() ? "socket" : isSymlink ? "symlink" : "other"; + const matchesExpectedType = matchesType(kind, expectedType); + return { + path, + exists: true, + kind, + isSymlink, + symlinkTarget, + resolvedPath, + matchesExpectedType, + expectedType, + problem: matchesExpectedType ? null : "type-mismatch", + }; + } catch (error) { + try { + const lstat = lstatSync(path); + if (lstat.isSymbolicLink()) { + const symlinkTarget = readlinkSync(path); + return { + path, + exists: true, + kind: "symlink", + isSymlink: true, + symlinkTarget, + resolvedPath: resolveSymlinkTarget(path, symlinkTarget), + matchesExpectedType: false, + expectedType, + problem: "symlink-target-missing", + }; + } + } catch { + // Fall through to the generic missing/error report. + } + if (isMissingPathError(error)) return missingPathCheck(path, expectedType); + return { + path, + exists: false, + kind: "missing", + isSymlink: false, + symlinkTarget: null, + resolvedPath: null, + matchesExpectedType: false, + expectedType, + problem: errorMessage(error), + }; + } +} + +function checkFixturePath(path: string, expectedType: string, fixture: PathStateFixture): PathCheck { + const exists = fixture.kind !== "missing"; + const resolvedPath = fixture.kind === "symlink" && fixture.target !== undefined ? resolveSymlinkTarget(path, fixture.target) : exists ? path : null; + const targetExists = fixture.kind !== "symlink" ? exists : fixture.targetExists === true; + const effectiveKind = fixture.kind === "symlink" && targetExists ? "directory" : fixture.kind; + const matchesExpectedType = exists && matchesType(effectiveKind, expectedType) && (fixture.kind !== "symlink" || targetExists); + return { + path, + exists, + kind: fixture.kind, + isSymlink: fixture.kind === "symlink", + symlinkTarget: fixture.target ?? null, + resolvedPath, + matchesExpectedType, + expectedType, + problem: matchesExpectedType ? null : expectedType === "DirectoryOrCreate" && !exists ? "directory-or-create-missing" : exists ? "type-mismatch" : "missing", + }; +} + +function missingPathCheck(path: string, expectedType: string): PathCheck { + return { + path, + exists: false, + kind: "missing", + isSymlink: false, + symlinkTarget: null, + resolvedPath: null, + matchesExpectedType: expectedType === "DirectoryOrCreate", + expectedType, + problem: expectedType === "DirectoryOrCreate" ? "directory-or-create-missing" : "missing", + }; +} + +function matchesType(kind: PathKind, expectedType: string): boolean { + if (expectedType === "Unset" || expectedType.length === 0) return kind !== "missing"; + if (expectedType === "Directory" || expectedType === "DirectoryOrCreate") return kind === "directory"; + if (expectedType === "File" || expectedType === "FileOrCreate") return kind === "file"; + if (expectedType === "Socket") return kind === "socket"; + return kind !== "missing"; +} + +function resolveSymlinkTarget(path: string, target: string | null): string | null { + if (target === null) return null; + return isAbsolute(target) ? target : resolve(dirname(path), target); +} + +function dedupeHostPathEntries(entries: HostPathEntry[]): HostPathEntry[] { + const seen = new Set(); + const result: HostPathEntry[] = []; + for (const entry of entries) { + const key = `${entry.hostPath}\0${entry.hostPathType}`; + if (seen.has(key)) continue; + seen.add(key); + result.push(entry); + } + return result; +} + +function splitManifestDocuments(text: string): Array<{ index: number; raw: string; kind: string | null; name: string | null; namespace: string | null }> { + return text.split(/^---\s*$/mu) + .map((raw, index) => ({ index, raw: raw.trim() })) + .filter((doc) => doc.raw.length > 0) + .map((doc) => ({ + ...doc, + kind: scalarAfter(doc.raw, "kind"), + name: metadataScalarAfter(doc.raw, "name"), + namespace: metadataScalarAfter(doc.raw, "namespace"), + })); +} + +function scalarAfter(text: string, key: string): string | null { + const match = text.match(new RegExp(`^\\s*${key}:\\s*"?([^"\\n#]+)"?\\s*(?:#.*)?$`, "mu")); + return match?.[1]?.trim() ?? null; +} + +function metadataScalarAfter(text: string, key: string): string | null { + const metadataIndex = text.search(/^metadata:\s*$/mu); + if (metadataIndex < 0) return null; + return scalarAfter(text.slice(metadataIndex), key); +} + +function findVolumeName(lines: string[], hostPathIndex: number): string | null { + for (let index = hostPathIndex - 1; index >= Math.max(0, hostPathIndex - 8); index -= 1) { + const match = (lines[index] ?? "").match(/^\s*-\s+name:\s*"?([^"\n#]+)"?/u); + if (match?.[1] !== undefined) return match[1].trim(); + } + return null; +} + +function findScalarAfter(lines: string[], startIndex: number, key: string): string | null { + for (let index = startIndex + 1; index < Math.min(lines.length, startIndex + 8); index += 1) { + const match = (lines[index] ?? "").match(new RegExp(`^\\s*${key}:\\s*"?([^"\\n#]+)"?\\s*(?:#.*)?$`, "u")); + if (match?.[1] !== undefined) return match[1].trim(); + } + return null; +} + +function sandboxItems(parsed: unknown): Record[] { + const record = asRecord(parsed); + const items = record.items ?? record.sandboxes ?? record.pods ?? []; + return Array.isArray(items) ? items.flatMap((item) => typeof item === "object" && item !== null ? [item as Record] : []) : []; +} + +function sandboxSummary(item: Record, observedMs: number): CriSandboxSummary { + const metadata = asRecord(recordField(item, "metadata")); + const createdAt = recordField(item, "createdAt") ?? recordField(item, "created_at") ?? recordField(item, "created"); + const createdMs = typeof createdAt === "string" || typeof createdAt === "number" ? Number(createdAt) > 10_000_000_000_000 ? Math.floor(Number(createdAt) / 1_000_000) : Date.parse(String(createdAt)) : NaN; + const ageMinutes = Number.isFinite(createdMs) && Number.isFinite(observedMs) ? Math.max(0, Math.floor((observedMs - createdMs) / 60_000)) : null; + return { + id: String(recordField(item, "id") ?? recordField(item, "podSandboxId") ?? "unknown"), + name: stringOrNull(recordField(metadata, "name")), + namespace: stringOrNull(recordField(metadata, "namespace")), + state: String(recordField(item, "state") ?? "unknown"), + ageMinutes, + }; +} + +function podItems(parsed: unknown): Record[] { + const record = asRecord(parsed); + const items = record.items ?? []; + return Array.isArray(items) ? items.flatMap((item) => typeof item === "object" && item !== null ? [item as Record] : []) : []; +} + +function containerCreatingFindings(pod: Record, hostPathNeedles: string[]): ContainerCreatingFinding[] { + const metadata = asRecord(recordField(pod, "metadata")); + const status = asRecord(recordField(pod, "status")); + const namespace = stringOrNull(recordField(metadata, "namespace")) ?? "default"; + const podName = stringOrNull(recordField(metadata, "name")) ?? "unknown"; + const statuses = [ + ...statusArray(status, "initContainerStatuses"), + ...statusArray(status, "containerStatuses"), + ]; + return statuses.flatMap((container) => { + const state = asRecord(recordField(container, "state")); + const waiting = asRecord(recordField(state, "waiting")); + const reason = stringOrNull(recordField(waiting, "reason")) ?? ""; + if (reason !== "ContainerCreating") return []; + const message = stringOrNull(recordField(waiting, "message")) ?? ""; + const classification = classifyContainerCreating(message, hostPathNeedles); + return [{ + namespace, + pod: podName, + container: stringOrNull(recordField(container, "name")) ?? "unknown", + reason, + messagePreview: safePreview(message, 260), + classification, + }]; + }); +} + +function classifyContainerCreating(message: string, hostPathNeedles: string[]): ContainerCreatingFinding["classification"] { + const lower = message.toLowerCase(); + if (hostPathNeedles.some((needle) => needle.length > 0 && message.includes(needle)) || lower.includes("hostpath") || lower.includes("mountvolume") || lower.includes("not a directory") || lower.includes("no such file or directory")) { + return "hostpath-mount-failure"; + } + if (message.trim().length === 0 && hostPathNeedles.length > 0) return "opaque-containercreating-hostpath-risk"; + return "containercreating-observed"; +} + +function missingHostPathNeedles(hostPaths: HostPathReadinessReport): string[] { + return Array.from(new Set([ + ...hostPaths.missing.map((problem) => problem.entry.hostPath), + ...hostPaths.typeMismatches.map((problem) => problem.entry.hostPath), + ...hostPaths.directoryOrCreateMissing.map((problem) => problem.entry.hostPath), + ])); +} + +function statusArray(status: Record, key: string): Record[] { + const value = recordField(status, key); + return Array.isArray(value) ? value.flatMap((item) => typeof item === "object" && item !== null ? [item as Record] : []) : []; +} + +function summarizeCommandResult(result: CommandResult): CommandResultSummary { + return { + exitCode: result.exitCode, + signal: result.signal, + timedOut: result.timedOut, + stdoutTail: result.stdout.length === 0 ? "" : ``, + stderrTail: result.stderr.slice(-2000), + }; +} + +function compactHostPathProblem(problem: HostPathProblem): CompactHostPathProblem { + return { + hostPath: problem.entry.hostPath, + hostPathType: problem.entry.hostPathType, + manifest: problem.entry.manifest, + workload: `${problem.entry.namespace ?? "default"}/${problem.entry.kind ?? "Unknown"}/${problem.entry.name ?? "unknown"}`, + volumeName: problem.entry.volumeName, + problem: problem.check.problem, + severity: problem.severity, + disposition: problem.disposition, + reason: problem.reason, + }; +} + +function recordField(record: Record, key: string): unknown; +function recordField(record: unknown, key: string): unknown; +function recordField(record: unknown, key: string): unknown { + return typeof record === "object" && record !== null && key in record ? (record as Record)[key] : undefined; +} + +function asRecord(value: unknown): Record { + return typeof value === "object" && value !== null && !Array.isArray(value) ? value as Record : {}; +} + +function stringOrNull(value: unknown): string | null { + return typeof value === "string" && value.length > 0 ? value : null; +} + +function errorMessage(error: unknown): string { + return error instanceof Error ? error.message : String(error); +} + +function isMissingPathError(error: unknown): boolean { + if (typeof error !== "object" || error === null || !("code" in error)) return false; + const code = (error as { code?: unknown }).code; + return code === "ENOENT" || code === "ENOTDIR"; +} + +function safePreview(text: string, maxChars: number): string { + return text.length <= maxChars ? text : `${text.slice(0, maxChars)}...`; +}