pikasTech-unidesk/docs/reference/devops-hygiene.md

# DevOps Hygiene

This document is the authoritative source for UniDesk deployment hygiene: Git-backed deployment truth, dirty-environment boundaries, bounded manual operations and CI source-auth rules. Release-line and CI/CD runtime-version governance is owned by `docs/reference/release-governance.md` and [GitHub issue #6](https://github.com/pikasTech/unidesk/issues/6). If the same hygiene rule would need edits in `docs/reference/dev-environment.md`, `docs/reference/deploy.md`, `docs/reference/ci.md`, `docs/reference/dev-ci-runner.md`, `docs/reference/deployment.md`, `AGENTS.md` or `TEST.md`, keep the detailed rule here and leave only a cross-reference elsewhere.

## Source Of Truth

UniDesk deployment state is healthy only when all three layers point back to pushed Git commits:

- Desired state: `origin/master:deploy.json`, `config.json` and committed service manifests.
- Runtime state: live service metadata, image tags, Deployment annotations/env stamps or Compose labels that identify the deployed commit.
- Verification state: `deploy plan`, health checks, `server status`, service proxy checks and CI/e2e results that agree with the desired commit.

Local worktrees, D601 runtime files, copied scripts, copied images, ad-hoc Kubernetes objects and one-off curl results are never deployment truth.

When stable release lanes such as `release/v1` are enabled, the desired-state ref must be explicit in the command, job log and deploy output. Until that support exists, commands that are documented to read `origin/master:deploy.json` must keep doing so and must not silently switch to another branch or a dirty manifest.

Source and CLI files must not be kept near the 3000-line split boundary. Once a file exceeds 3000 lines, split it by responsibility until the original file is below 2000 lines before continuing feature or fix work. Do not make token-preserving micro-edits that leave the file just under or exactly at 3000 lines; that only guarantees the next small change will trigger the same split problem again.

Large shell, Node, Python or other script bodies must live in files with their native suffix, such as `.sh`, `.mjs` or `.py`, and be loaded or streamed from those files. Do not embed operational scripts as large string literals in TypeScript, shell wrappers, YAML or another host language; small command fragments and bounded quoted heredocs remain acceptable when they are the native interface of a controlled command.

When declared validation dependencies are missing, install them with the repository's package manager and lockfile, then continue the original validation. Do not skip, downgrade or terminate validation merely because the local environment is incomplete. If the required dependency is undeclared or installation would introduce a lockfile change, first add the dependency through the repository's normal declaration workflow and keep that change in the task's reviewed scope.

## Prohibited Deployment Truth

The following practices are not acceptable as the long-term or hidden source of a working environment:

- 微服务线上契约与版本校验必须非阻塞：
  - 服务间的契约版本、源码版本、镜像版本、`commitId`、provenance 或部署对齐检查只能产生结构化 warning、OTel 事件和运维诊断；
  - 这些检查不得拒绝、暂停、降级或延迟用户业务请求，也不得成为 admission、readiness、dispatch、CI/CD public gate 或 Web/CLI 可用性门禁；
  - 接收方必须按当前业务语义解析请求，并对可安全归一化的旧字段、冗余字段和版本差异做兼容归一化，同时记录 warning；
  - 只有业务数据本身无法解释、违反安全/权限边界或会造成不可逆损害时才允许拒绝，并返回与具体业务字段相关的 typed error；
  - 禁止把版本、commit 或契约漂移包装成 `schema-invalid`、`503` 或其他用户阻塞错误；
  - CI/CD、status 和 observability 可以报告版本漂移，但只能作为修复提示，不能压过真实业务成功证据。

- Hand editing D601 runtime files, k3s manifests, ConfigMaps, Deployments or container env values and treating the live result as source of truth.
- Rebuilding backend-core, frontend, k3sctl-adapter or other managed services from a dirty worktree on the master server, D601 or an operator machine.
- Copying large local shell scripts, generated manifests, Docker images or application source to D601 as the main deployment mechanism.
- Fixing dev or production reachability by adding direct D601 public ports, NodePorts or backend-core hardcoded service entries instead of updating the proper catalog/control bridge.
- Treating `server rebuild backend-core` as a Rust backend-core iteration path; Rust build/check belongs to D601 CI and CD must consume the published artifact.
- Using local-manifest production deploy for services that already have artifact consumers. `backend-core`, `frontend`, `baidu-netdisk` and `decision-center` production deploys must enter through `deploy apply --env prod` so CD consumes a commit-pinned registry artifact instead of silently falling back to target-side source build.
- Treating upstream images as UniDesk source-build services. `filebrowser` and `filebrowser-d601` are upstream-image consumers; they require digest pin or digest-verified mirror governance and must not be added to Dockerfile CI artifacts.
- Considering manual curl, kubectl or Docker checks sufficient when live commit metadata, deploy plan, health checks and CI/e2e disagree.

## Bounded Manual Operations

Manual operations are allowed only when they are narrow, visible and followed by commit-based verification:

- `server rebuild dev-frontend-proxy` may update the thin main-server nginx proxy for the public dev UI port; it must not build backend/frontend application code.
- `deploy apply --service k3sctl-adapter` may update the k3s control bridge catalog through the documented local manifest exception in `docs/reference/deploy.md`.
- Host SSH/provider-gateway dispatch may start the `ci run-dev-e2e` short launcher described in `docs/reference/dev-ci-runner.md`; it must not carry large shell bodies or become a general deployment path.
- Read-only smoke checks such as `curl http://74.48.78.17:18083/health`, `server status`, `microservice proxy .../health` and `kubectl get` may validate state, but they do not replace desired-state and live-commit verification.
- `bun scripts/cli.ts check recovery-guardrails` and the equivalent `check --recovery-guardrails` gate are commander-safe read-only recovery diagnostic surfaces for D601 reboot incidents. They may read `/proc/mounts`, inspect local path metadata, parse committed k3s manifests, and run bounded read-only `kubectl get pods -A -o json` / `crictl pods -o json` probes when the tools are present. They must not restart k3s, delete pods or CRI sandboxes, apply manifests, rollout workloads, mutate hostPath directories, prune Docker state, or repair symlinks automatically.
- File Browser recovery may use the existing provider-local image-only/docker-run path only as a bounded repair path. Standardization requires first resolving `docker.io/filebrowser/filebrowser:v2.63.3` to an upstream manifest digest or a digest-verified local mirror, then validating the running container through the UniDesk private proxy.

Manual Secret/env/rollout repair is allowed only as a bounded runtime recovery path. It must have explicit authorization for the target environment and service, a narrow object scope, redacted evidence, an issue review trail and a durable source fix. Acceptable evidence includes object names, revision changes, health status, rollout status and redacted key presence; it must not include secret values, tokens, full env dumps or copy-pastable sensitive mutation commands.

Any manual repair that changes live credentials, env wiring, DNS/egress assumptions, ConfigMaps, Deployments or rollout state must be followed by a source-of-truth update in the owning repository, secret management path, manifest, deployment catalog or a tracked remediation issue. The live environment is recovery evidence, not deployment truth.

If a manual repair is needed to unblock the platform, the durable fix must be committed and pushed, then redeployed or revalidated through the normal path. Do not preserve the repair only as hidden runtime state.

## 分布式敏捷流程

“分布式敏捷”是 UniDesk 对 distributed agile field repair 的固定流程名；通用 P1/P2/P3/P4 阶段、禁止行为和证据边界由 `$dad-dev` skill 维护，本参考不再重复展开。UniDesk 项目内只保留下面的特有约束：必须使用结构化 `trans`/UniDesk CLI 进入真实 provider、pod、host bridge 或 service port；运行面热补只能证明方向或临时恢复，不能成为隐藏部署真相；持久化完成必须回到 Git/PR/CI/CD 后原入口复测。

固定主 repo 是 source truth anchor，不是源码/运行面 scratch 区。会产生源码、配置、issue closeout、部署脚本、验收产物或高风险 dad-dev / post-task 交付的工作，执行前必须先从目标 fixed repo 的最新 remote/base 创建任务专属 `.worktree/<task>`，后续编辑、验证、提交、push 和受控 CLI 写操作都在该 worktree 内完成。UniDesk 任务 worktree 必须通过 `bun scripts/cli.ts dev-env worktree add .worktree/<task> --branch <branch> [--from origin/master]` 创建，让 `.worktreecopy` 白名单内的本地配置自动复制到新 worktree；fixed repo 只用于 `git fetch`、`git status`、快进同步、读取规则和这个受控创建入口。其中已有的并行未提交修改默认保持不动，不纳入当前任务，也不要用 reset、checkout 或删除来“清理”。

The root UniDesk checkout at `/root/unidesk` is the fixed main worktree and must stay on `master`. Non-`master` work belongs in a task-specific `.worktree/<task>`. Unexpected local changes are presumed to belong to another concurrent task: preserve them, do not reset, checkout, delete or rewrite them, and scope the current task's commit to its own files.

## Worktree-Independent Local State

Tracked source, YAML and scripts belong to the selected checkout; operator credentials and mutable runtime state do not. UniDesk's canonical owner-level roots are `/root/.unidesk/.env` for credentials and `/root/.unidesk/.state` for mutable state, logs, caches, generated artifacts and Secret source files. YAML `sourceRef`, CLI path configuration and source defaults that consume these materials must use those fixed absolute roots. They must not derive the path from `process.cwd()`, the module's task-worktree root, or a relative `.env`/`.state`/`logs` path.

`/root/unidesk/.env`, `/root/unidesk/.state` and `/root/unidesk/logs` are compatibility symlinks to the canonical owner-level roots. They preserve old main-worktree entrypoints but are not source authority. `.worktreecopy` must not copy these trees, and task worktrees must not receive private copies or task-specific compatibility links. A command that fails only because a task worktree lacks `.env`, `.state`, `logs`, `secrets`, cache or generated state has a path-contract defect: fix the owning config/helper to consume the canonical absolute path before continuing the original operation.

When diagnosing a missing worktree file, first classify ownership. A tracked file is repaired through remote/base synchronization; an ephemeral test fixture uses an explicit temporary directory; durable local credentials or state move to the canonical owner-level root and are referenced absolutely. Never repair the latter by copying secrets/state into every worktree, silently falling back to the current directory, embedding values in Git, or maintaining divergent per-worktree data. A migration must preserve existing contents, keep the canonical directories owner-only, verify the resolved symlink target and data size, and leave the main-repo compatibility symlink in place.

开始使用任何固定主/目标 worktree 时，如果发现它落后 remote/base，必须先完成 source-truth 同步再继续分析或创建任务 worktree：若工作区有脏改，先用 `git stash push -u` 保存当前脏改（包括 untracked），再执行 `git pull --ff-only <remote> <branch>` 快进，随后 `git stash apply` 并按快进后的语义合并冲突或重复内容；若工作区 clean，则直接 `git pull --ff-only`。这个 stash 是保护并行修改以便固定主 repo 回到最新 source truth，不是清理、丢弃或隐式接管并行任务；apply 后只提交当前任务明确相关的文件，其他并行修改继续保留原状。不得在落后的固定 worktree 上继续源码分析、创建任务分支、修改代码或给 issue 写基于旧源码的结论。

固定 worktree 对齐 remote 是硬前置，不是建议项。`git status` clean、运行面 healthy、CI/CD snapshot 对齐或历史 PipelineRun 通过，都不能替代固定工作区自身的 remote/base 对齐检查；只要 HEAD 与声明 remote/base 不一致、`remoteUpToDate=false`、ahead/behind/diverged 或受控 source-workspace 状态返回 not-ready，就必须先修复固定工作区或改用从最新 remote/base 创建的干净 anchor。禁止从未对齐的固定 worktree 创建任务 worktree、开测试 PR、运行 repo 内验证或写 closeout 结论。

所有源码、配置、部署脚本和运行面修复都必须在从最新 remote/base 创建的独立 `.worktree/<task>` 中完成；固定主 worktree 不直接承载代码修改。`.worktreecopy` 只能声明可复制的本地文件路径模式，支持空行、注释、glob 和 `!pattern` 排除/重包含语义，不得写入 secret 值；复制结果默认不覆盖目标文件，只有显式 `--overwrite-local` 才允许覆盖。PR 合并或等价集成进入 remote base 后，应及时回到对应固定主 worktree 执行 `git fetch` 与 `git pull --ff-only`，让下一轮 source-truth 预检、web-probe 复核、issue closeout 和新任务 worktree 都从已合并的最新源码开始。若固定主 worktree 因并行脏改不能安全快进，保留脏改并改用干净的最新 remote/base worktree 继续，不能 reset、checkout 或删除他人修改。

任务 worktree 清理前必须做语义合并核查。最低要求是：worktree clean；相关提交已经是 `origin/master` 祖先，或 `git log --left-right --cherry-pick <worktree-head>...origin/master` 没有 left-only 未吸收 patch；必要时再核对关键文件 diff、PR merge commit、issue closeout 和运行面验证是否对应最新 `master`。只有确认当前任务语义已经进入 `master` 或被更新实现等价替代后，才允许 `git worktree remove <path>`；不得只因为分支落后、PR 已关闭、文件看起来相似或本地空间紧张就删除。若发现未提交文件、未推送提交、left-only patch 或语义不确定，先把应保留内容提交/合并/推送到 `master`，或记录阻塞并保留 worktree。

文档治理是固定主 repo 保护规则的轻量例外。单纯文档、`AGENTS.md`、`docs/reference/*.md`、skill 规则、runbook、过程文档蒸馏和其他长期参考收敛不需要创建新 `.worktree` 或短生命周期 PR；应在当前主 worktree 按上面的 stash-if-dirty + `git pull --ff-only` 对齐最新 remote 后再直接修改、做最小语法/diff 检查、提交并 push。该例外只覆盖文档/规则本身，不得夹带源码、配置、部署、运行面或 issue lifecycle 写操作；若主 worktree 已有并行文档修改，只提交本次明确相关文件，不能 reset、drop stash 或顺手合并他人修改。

允许不创建新 `.worktree` 的场景包括 P1 只读探测、运行面临时热补、上述文档/skill/长期参考轻量修改，或目标项目长期参考明确声明的直接修改例外。例外必须能解释为什么不会污染 fixed repo source truth，并且不得触碰无关并行修改；一旦需要写源码、配置、issue closeout、部署脚本、验收产物或其他高风险交付记录，立即切回独立 `.worktree`。

CONSTAR、控之星、71-FILTER、71-FREQ、PLC 项目和 PLC 控制器任务是目标项目明确声明的直接修改例外。固定入口为 `G14-WSL:win/d/Work/CONSTAR_workspace`，Windows 路径为 `D:\Work\CONSTAR_workspace`；这些任务直接在该固定主 worktree 及其嵌套项目仓库工作，不创建任务 `.worktree`。开始前仍须核对目标仓库分支和 remote 对齐状态，保留全部既有并行修改，只精确提交当前任务文件，禁止用 reset、覆盖式 checkout 或删除清理他人改动。该例外不扩展到 `/root/unidesk` 或其他未明确列出的固定仓库。

When UniDesk's own CLI, wrapper or controlled toolchain is repaired, merge the durable fix into `master` before relying on it for the original operation. The checkout that actually executes the tool must then contain that merged revision. If a fixed checkout cannot fast-forward because of unrelated changes, preserve it and run the tool from a clean worktree based on current `origin/master`; never reset or overwrite concurrent work to update the executable path.

在模型 provider、API provider、硬件链路、跨平台 bridge、CLI/trans/tran 或高频工具链问题上，判定外部 blocker 前仍需完成 UniDesk 的防误判核查：确认当前 runtime config / Secret key presence / env / proxy / NO_PROXY / endpoint / args，使用实际目标运行面复现，并尽量与 UniDesk/HWLAB 成熟实现对照。用户反馈或新证据推翻 blocker 判断时，立即切回 `$dad-dev` 的现场修复闭环。

如果某个现场步骤因为 quoting、route 定位、kubeconfig、输出体积或缺少 helper 而反复痛苦，优先改进 UniDesk passthrough / CLI 并在本文件的 `Distributed Command Passthrough` 或 `docs/reference/cli.md` 中记录稳定入口，不要沉淀一批一次性 shell 菜谱。

## Distributed Command Passthrough

Distributed runtime work should prefer structured CLI passthrough over ad-hoc nested shell strings. The standard escalation order is:

1. Use a purpose-built UniDesk route plus operation or helper such as `trans D601:k3s kubectl ...`, `trans D601:k3s sh`, `trans D601:k3s:<namespace>:<workload> logs`, `trans D601:k3s:<namespace>:<workload> sh`, `trans D601:k3s:<namespace>:<workload>[:<container>] apply-patch --cwd /workspace`, `trans <providerId>:/absolute/workspace apply-patch`, `trans <providerId> py`, `trans <providerId> find`, `trans <providerId> glob` or `trans <providerId> skills`. Use legacy `apply-patch-v1` only when the old remote helper is explicitly required.
2. If no helper exists, use `trans <providerId> argv <command> [args...]` so the CLI quotes each argv token once.
3. If shell features such as pipes, redirects, loops or variable expansion are required, use a single quoted heredoc with explicit `trans <providerId> sh|bash` or `trans D601:k3s:<namespace>:<workload> sh|bash` so the script body travels over stdin instead of through shell command-string arguments.
4. Treat free-form ssh-like command strings as an interactive compatibility path, not as the default automation surface.

For D601 Kubernetes work, route syntax is preferred over positional shell recipes, but the route must stay a pure locator. `D601:k3s` means the native k3s control plane, and `D601:k3s:<namespace>:<workload>[:container]` means a namespaced workload or pod/container. `:` is the distributed route separator; `/` is only an in-container filesystem cwd, so container selection must use `:<container>` or `--container <container>`, not `pod/<pod>/<container>`. Operations come after the route: `kubectl` runs on the control plane, `logs` reads bounded workload logs, `sh`/`bash` stream a local heredoc/stdin script into the host or target pod with an explicit shell dialect, and `apply-patch --cwd /workspace` is the default remote text patch operation for pod workspaces. The route-operation split keeps distributed location and execution behavior independently extensible, fixes `KUBECONFIG=/etc/rancher/k3s/k3s.yaml`, refuses long-follow logs, and assembles common `kubectl exec` / `kubectl logs` / stdin shell / pod patch target arguments without adding a provider-gateway protocol change. This prevents the common failure mode where a command crosses local shell, UniDesk SSH broker, remote shell command strings, `kubectl exec`, and container shell quoting layers before reaching the process that should run it.

Longer scripts should move across stdin (`trans py`, explicit `trans sh|bash`, or k3s `sh|bash` operation), and remote text patches should default to `apply-patch` with a host or pod workspace route. Legacy `apply-patch-v1` remains available as the explicit fallback and uses the injected `sh` helper path instead of assuming target containers have `python3`, `node` or repository-local tools. Avoid heredocs nested inside remote command strings, `python - <<EOF` inside SSH strings, or JSON/Markdown bodies passed through shell arguments. These patterns often bind stdin to the wrong process, strip quotes, or leave a half-open provider SSH session that looks like a platform outage.

When structured passthrough is missing for a recurring workflow, fix the CLI first and then document the durable helper. Do not preserve a growing collection of one-off shell recipes as the long-term runbook.

`trans`/`tran` and non-interactive `ssh` are short-operation tools. Their outer runtime limit is intentionally bounded, so long builds, downloads, Tekton/Argo observations, device operations and Code Agent trace waits must use short-start plus poll semantics: start or observe a named job, write logs/state on the target, then return; follow-up commands read bounded status, log tails and terminal evidence. A `UNIDESK_TRAN_TIMEOUT_HINT` means the caller held the transport too long, not that the target task necessarily failed.

## D601 Recovery Hotfix Exception

D601 reboot recovery has a narrow hotfix exception because k3s, Code Queue and hostPath readiness can fail before normal UniDesk proxy/CD surfaces are healthy. The exception authorizes diagnosis and carefully scoped host repair only; it does not make live host edits a new deployment path.

Allowed read-only recovery checks:

- `bun scripts/cli.ts check recovery-guardrails` or `bun scripts/cli.ts check --recovery-guardrails` on the host or runner environment.
- `/proc/mounts` inspection for malformed Docker Desktop `/Docker/host` 9p rows that may break kubelet mount-table validation.
- `kubectl get/describe/logs/events` and `crictl pods -o json` as bounded observation.
- Path metadata checks for `/home/ubuntu/unidesk-code-queue-deploy`, `/home/ubuntu/cq-deploy`, Code Queue hostPath directories, `.codex` files, `.ssh`, `.agents/skills`, and MDTODO workspace/log paths.

Manual host hotfix may be considered only after the read-only output identifies a concrete redline and the operator has reviewed whether the target is source checkout, credential material, log/cache state, or user data. Examples include restoring a missing Git worktree from pushed remote state, recreating an intended compatibility symlink after confirming its target, restoring a missing runtime Secret source, or fixing a Docker Desktop/WSL mount-table condition. The repair must be recorded in the relevant issue or commander brief and followed by a source or runbook update when the root cause is durable.

Forbidden automatic recovery actions:

- `systemctl restart k3s`, `service k3s restart`, `kubectl delete pod`, `crictl rmp`, `crictl rm`, `docker system prune`, `docker volume prune`, recursive chmod/chown/rm under `/home/ubuntu`, and `git reset --hard` of a live worktree.
- Deleting CRI sandboxes or Kubernetes Pods because a diagnostic counted stale sandboxes.
- Creating, deleting or replacing MDTODO workspace content from Code Queue or the generic CLI. MDTODO hostPaths contain user-authored Markdown data.
- Treating `DirectoryOrCreate` as permission to mass-create parent trees or credential directories. Kubelet may create the final directory only when mount validation and parent permissions are already healthy; humans still decide user-data boundaries.

ClaudeQQ or direct user approval is required before any high-risk host action that restarts k3s/kubelet/Docker Desktop, deletes pods/sandboxes, changes credential or SSH paths, touches MDTODO workspace data, force-resets a worktree, changes production rollout state, or could interrupt active Code Queue tasks. If ClaudeQQ is unavailable, the operator must stop at the written plan and record the approval gap instead of silently executing the action.

## CI And Private Source Auth

Private repository access is part of the CI contract. `ci run` must not rely on unauthenticated HTTPS clone, an operator's local dirty worktree or an ad-hoc secret copied by hand into one PipelineRun.

Acceptable source access implementations are:

- a first-class CI Git credential installed by `ci install` and referenced by the Tekton Pipeline;
- an in-cluster Git mirror managed by UniDesk; or
- the same commit-pinned host-fetch boundary used by `ci run-dev-e2e`, where D601 fetches the manifest commit and passes only verified inputs into Tekton.

For backend-core artifact publication, the required implementation is the host-fetch boundary: D601 uses the existing GitHub SSH deploy identity plus the node-local provider-gateway WS egress proxy, exports the requested commit to `/home/ubuntu/.unidesk/ci/backend-core-artifacts/<commit>`, and passes only that verified source directory into Tekton. This path must not be replaced by an in-cluster Git mirror, a third-party source mirror, an operator local checkout, or Tekton-mounted GitHub credentials.

If a CI repo-check task fails at `git clone` because credentials are unavailable, classify it as a CI infrastructure/auth gap, not as an application test failure.

## 用户交付 CHANGELOG 写法

面向用户、交付方或评审方的 CHANGELOG 必须先说明用户可感知的功能改进或问题修复，再在同一条目的后半句说明交付文档同步情况。不要把 commit、内部函数、调试注入、测试过程、构建下载过程或治理动作写成用户更新；测试只用于证明条目成立，测试结果应留在测试报告或任务记录中。

CHANGELOG 的文档交叉引用必须同时满足以下要求：

- 精确到文件名和文档内稳定编号或章节，例如 `jsonrpc_api_design.md` A14、`03_input_filter_output_mainline.md` T3.5。
- 对每个被引用的文档分别用一个短语说明具体同步内容，不能只罗列一串文件名和编号。
- API 文档说明字段、接口或状态语义的变化；配置文档说明字段默认值、范围和生效规则；测试文档说明新增或修改的验收场景与判定标准。
- 同一功能涉及多个提交时合并成一个用户条目；纯内部治理只并入对应功能条目，不单独包装成功能更新。
- 只有说明补充、没有代码功能变化时，明确写“补充文档说明”，不能表述为功能已经修复。
- 用语必须直接描述最终行为，例如“输入无效后自动停止硬件输出并归零”，避免“受控处理”“优化逻辑”等不能回答实际结果的含混措辞。

推荐格式：

```markdown
- <用户可感知的功能或问题修复>。文档同步：`<API 文档>` <编号> <接口或字段变化>；`<配置文档>` <编号> <默认值、范围或生效规则变化>；`<测试文档>` <编号> <验收场景或判定变化>。
```

## Verification Priority

The user's latest explicit objective takes precedence over stale tests, guards, preflights and assertions. When an old gate blocks the current requirement without protecting a still-valid high-value risk, remove it from the active path instead of preserving a fallback, legacy mode or compatibility branch around it.

When checks disagree, use this priority order:

1. The pushed desired commit and `deploy.json`/manifest contract.
2. Live runtime metadata proving which commit is deployed.
3. Controlled health checks and service proxy checks for the same runtime object.
4. CI/e2e results tied to the same commit.
5. Manual curl/kubectl output as supporting evidence only.

A passing lower-priority manual check cannot override a higher-priority mismatch. Fix the desired state, deploy path, runtime metadata or CI infrastructure until all layers agree.