diff --git a/AGENTS.md b/AGENTS.md index 756d9cff..a48591c3 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -33,7 +33,7 @@ UniDesk 是一个以主 server 为统一入口的分布式工作平台；本文 - `bun scripts/cli.ts server swap status|ensure [--path /swapfile] [--size 2GiB] [--dry-run]`：以 JSON 查看或幂等创建主 server swapfile，`ensure` 输出 before/after、动作、持久化状态和 degraded/failed 详情，规则见 `docs/reference/deployment.md`。 - `bun scripts/cli.ts server logs [--tail-bytes N]`：分页返回文件日志与 Docker 日志尾部并带截断元数据，日志规则见 `docs/reference/observability.md`。 - `bun scripts/cli.ts server rebuild `：以 build-first、Compose lock、no-deps force-recreate 和 post-up validation 的异步 job 重建主 server Compose 内单个服务；对 database、File Browser、Code Queue 执行面、k3sctl-adapter 或未知对象返回结构化 `unsupported-server-rebuild`，规则见 `docs/reference/deployment.md` 与 `docs/reference/cicd-standardization.md`。 -- `bun scripts/cli.ts provider attach [--master-server URL] [--up] [--force]` / `bun scripts/cli.ts provider triage [--observed-error text] [--observed-scope scope] [--microservice id ...]`：前者在新增计算节点上生成两项配置的 provider-gateway 挂载包；后者是只读多信号健康裁决入口，用来把单路径 `provider is not online`、SSH 超时、registry 失败或 proxy 失败归类为 runner-local、service-degraded、provider-degraded 或 global-blocker，规则见 `docs/reference/provider-gateway.md` 和 `docs/reference/code-queue-supervision.md`。 +- `bun scripts/cli.ts provider attach [--master-server URL] [--up] [--force]` / `bun scripts/cli.ts provider triage [--observed-error text] [--observed-scope scope] [--microservice id ...]`：前者在新增计算节点上生成两项配置的 provider-gateway 挂载包；后者是只读多信号健康裁决入口，输出 `decision`、`healthyScopes`、`failedScopes` 和 `retryable`，用来把单路径 `provider is not online`、SSH 超时、registry 失败或 proxy 失败归类为 `retryable-transient`、`service-degraded` 或 `global-offline`，规则见 `docs/reference/provider-gateway.md` 和 `docs/reference/code-queue-supervision.md`。 - `bun scripts/cli.ts ssh [ssh-like args...]`：通过 provider-gateway 的 Host SSH / WSL SSH 维护桥打开近似原生 ssh 的交互会话或远端命令，并在远端 PATH 注入 `apply_patch`、`glob` 与 `skill-discover`；`apply-patch`、`py`、`skills`、结构化 `find`、`glob` 和 `argv` 子命令用于避免远端补丁、Python stdin、skill 发现与常用只读命令的嵌套转义问题，使用规则见 `docs/reference/cli.md` 和 `docs/reference/provider-gateway.md`。 - `bun scripts/cli.ts microservice list/status/health/diagnostics/tunnel-self-test/proxy`：管理和验证挂载在主 server、计算节点 Docker 或 k3s 控制面上的用户服务，`proxy` 支持受控 JSON body，OA Event Flow/Todo Note/Baidu Netdisk/Code Queue Manager on main-server、k3s Control/Code Queue 执行面/MDTODO/Decision Center/FindJob/Pipeline/MET Nonlinear on D601 的规则见 `docs/reference/microservices.md`。 - `bun scripts/cli.ts microservice health/diagnostics/proxy code-agent-sandbox`：验证独立 Code Agent Sandbox 的 health、只读 diagnostics、trace 和 adapter/mode/credential boundary 契约，规则见 `docs/reference/code-agent-sandbox.md`。 diff --git a/docs/reference/artifact-registry.md b/docs/reference/artifact-registry.md index 77538094..4041aa9d 100644 --- a/docs/reference/artifact-registry.md +++ b/docs/reference/artifact-registry.md @@ -122,7 +122,9 @@ bun scripts/cli.ts ssh D601 argv bash -lc '' - 远端 config、compose、unit 的 hash 是否匹配 CLI 渲染结果。 - 运行镜像是否匹配期望版本。 -`status` 表示只读查询是否成功；未安装时仍可 `ok=true` 并报告 `installed=false`。`health` 表示 registry 是否已按期望运行；未安装或不健康时返回 `ok=false`。 +`status` 表示只读查询是否成功；未安装时仍可 `ok=true` 并报告 `installed=false`。`health` 表示 registry 是否已按期望运行；未安装或不健康时返回 `ok=false`。两个只读命令都应输出 `decision`、`retryable`、`healthyScopes`、`failedScopes` 和 `runtimeApiHealthy`，方便上层 provider triage 判断局部退化范围。 + +registry health 的 `decision=service-degraded` 不等同于 D601 全局离线。特别是当 systemd unit inactive 或 unit hash drift，但 Docker container running、loopback listener 正常、`/v2/` 返回 200 时，runtime registry API 仍可用；这种状态应作为 registry 服务治理问题处理，不能覆盖 provider heartbeat、Host SSH、k3sctl-adapter、Code Queue scheduler 或业务 API 的健康证据。 ## Manual Maintenance diff --git a/docs/reference/code-queue-supervision.md b/docs/reference/code-queue-supervision.md index e3c9ff33..77716c82 100644 --- a/docs/reference/code-queue-supervision.md +++ b/docs/reference/code-queue-supervision.md @@ -150,7 +150,7 @@ bun scripts/code-queue-pr-preflight-example.ts --repo pikasTech/unidesk --base m - Code Queue scheduler heartbeat、任务 heartbeat、trace/output 是否持续入库； - 当前 runner 容器内 CLI/proxy 路径是否只是局部不可达。 -只有多个独立观察面同时失败，或同一关键路径在明确时间窗口内持续失败，才能把问题判为全局阻塞。否则应记录为 transient 或 runner-local observation gap，优先重试、steer 任务纠偏或拆出基础设施 follow-up；不得让业务 worker 把单次局部失败作为最终 blocker。CLI 和 runtime 后续应把错误输出结构化为 `scope=runner-local|provider-gateway|ssh|registry|k3s|scheduler|service-proxy`、`observedAt`、`retryable` 和建议的交叉验证命令。 +只有多个独立观察面同时失败，或同一关键路径在明确时间窗口内持续失败，才能把问题判为全局阻塞。否则应记录为 transient 或 runner-local observation gap，优先重试、steer 任务纠偏或拆出基础设施 follow-up；不得让业务 worker 把单次局部失败作为最终 blocker。CLI 和 runtime 必须把错误输出结构化为 `scope=runner-local|provider-gateway|ssh|registry|k3s|scheduler|service-proxy`、`observedAt`、`retryable`、`decision`、`healthyScopes`、`failedScopes` 和建议的交叉验证命令。 ClaudeQQ 是面向用户的主动提醒通道，不是 #24 简报更新的自动转发器。指挥官只应在三类情况下自主发送 ClaudeQQ 消息：核心服务或关键执行面宕机且需要用户知情，高风险决策需要用户请示，或出现里程碑式进展值得同步。消息必须简明扼要，一次不超过 200 个中文字符，写成一段话，不使用 Markdown 语法。普通轮询、普通 issue 更新、普通 #24 简报追加、外部 token provider 正常限流、以及无用户动作要求的中间状态，不发送 ClaudeQQ。发送失败只记录到 #24 或对应 blocker issue，不回滚已经完成的 GitHub issue 更新。 @@ -158,7 +158,9 @@ ClaudeQQ 是面向用户的主动提醒通道，不是 #24 简报更新的自动 当多信号裁决显示 provider 服务器、D601 执行面或关键维护桥疑似需要人工检查时，指挥官可以在更新 #24/#40 等记录之外，通过 ClaudeQQ 额外提醒用户检查 provider 服务器状态。提醒只在首次确认、状态恶化、恢复或需要用户介入时发送，不能在每轮轮询中重复轰炸。ClaudeQQ 提醒是 best-effort：若 ClaudeQQ 本身依赖同一条故障 provider/k3sctl 链路而不可达，指挥官应把通知失败的原因写入 #24 或对应 blocker issue，并继续按轮询和恢复规则推进。 -在 UniDesk CLI 中，`bun scripts/cli.ts provider triage ` 是只读多信号裁决入口，适合作为 worker 和指挥官的统一健康判断前置。它必须至少保留这些合同：`provider is not online` 这类单路径失败只应落到 `runner-local-observation-gap`，不得直接输出 `global-blocker`；只有 provider-gateway/SSH/k3s/scheduler 等多个独立关键路径同时失败，才允许输出 `global-blocker`；registry 或单个 service proxy 失败但 heartbeat、SSH 或节点视图仍健康时，应输出 `service-degraded` 或 `provider-degraded`。`recommendedCrossChecks` 必须包含 `debug health`、`debug dispatch host.ssh --wait-ms 15000`、`ssh argv true`、`artifact-registry health --provider-id `、`microservice health k3sctl-adapter`、`microservice health code-queue` 与 `codex tasks --view supervisor --limit 20`。 +在 UniDesk CLI 中，`bun scripts/cli.ts provider triage ` 是只读多信号裁决入口，适合作为 worker 和指挥官的统一健康判断前置。它必须至少保留这些合同：`provider is not online` 这类单路径失败只应落到 `decision=retryable-transient` / `blockingDisposition=runner-local-observation-gap`，不得直接输出 `global-offline`；只有 provider-gateway/SSH/k3s/scheduler 等多个独立关键路径同时失败且缺少健康交叉证据，才允许输出 `decision=global-offline`；registry 或单个 service proxy 失败但 heartbeat、SSH 或节点视图仍健康时，应输出 `decision=service-degraded`。`recommendedCrossChecks` 必须包含 `debug health`、`debug dispatch host.ssh --wait-ms 15000`、`ssh argv true`、`artifact-registry health --provider-id `、`microservice health k3sctl-adapter`、`microservice health code-queue` 与 `codex tasks --view supervisor --limit 20`。 + +D601 artifact registry 的 systemd unit inactive 不等于 D601 全局离线。如果 `artifact-registry health` 或 `provider triage D601` 同时看到 registry container running、loopback listener healthy、`/v2/` 返回 200，且 provider heartbeat、Host SSH、k3sctl-adapter、Code Queue scheduler 或业务 API 有健康信号，这只能判为 `service-degraded`，不得写成 provider offline、D601 offline 或 CI/CD 全局不可推进。只有这些健康面也同时失败，才进入 `global-offline` 判断。 对于 trace 或 heartbeat 新鲜的长任务，通常应保持运行。每几分钟轮询一次优于反复 interrupt/retry。 diff --git a/docs/reference/provider-gateway.md b/docs/reference/provider-gateway.md index eb687be9..bc204e08 100644 --- a/docs/reference/provider-gateway.md +++ b/docs/reference/provider-gateway.md @@ -138,6 +138,12 @@ backend-core 可以通过真实 WebSocket 调度向在线 provider 下发 `provi 旧版 provider-gateway 如果只能返回 plan 或因为旧环境中的 `PROVIDER_UPGRADE_ENABLED=false` 拒绝 schedule，需要先通过任意现有维护通道手动 bootstrap 一次。bootstrap 的目标不是长期流程，而是把节点更新到支持 always-enabled 远程升级和 Host SSH / WSL SSH 维护桥的版本；完成后必须立刻用 `bun scripts/cli.ts debug dispatch provider.upgrade --mode schedule --wait-ms 15000` 做一次真实一键升级验证，再用 `bun scripts/cli.ts debug health` 或公网 frontend 确认该节点仍在线、`unideskCapabilities` 包含 `provider.upgrade`，需要 SSH 维护的 WSL 节点还必须包含 `host.ssh`。 +## Provider Triage + +`bun scripts/cli.ts provider triage ` 是 provider 运行状态的只读多信号裁决入口。输出必须包含 `decision`、`retryable`、`healthyScopes`、`failedScopes`、`degradedScopes`、`blockingDisposition`、`rationale`、`signals` 和 `recommendedCrossChecks`。`decision` 的长期语义是：`global-offline` 表示 provider heartbeat、Host SSH、k3s 或 scheduler 等多个独立关键面同时失败且没有健康交叉证据；`service-degraded` 表示 registry、service proxy 或单个用户服务局部退化但仍存在 provider 级健康信号；`retryable-transient` 表示单次 runner-local、SSH、proxy 或 API timeout 证据不足，应重试或补交叉验证；`healthy` 表示未观察到失败或退化信号。 + +D601 这类长期 WSL provider 不得因为单一路径失败被直接写成全局离线。典型局部退化包括 artifact registry 的 `unidesk-artifact-registry.service` inactive，但 registry container 仍 running、listener 仍绑定 loopback、`http://127.0.0.1:5000/v2/` 返回 200；这种状态应在 registry scope 内显示 degraded，并在 provider triage 中落到 `decision=service-degraded`，只提示修复 systemd drift，不阻断所有 D601 上的 Code Queue、k3sctl-adapter 或业务 API 判断。 + ## Manual Upgrade Maintenance 手动升级只用于把旧节点 bootstrap 到支持 always-enabled 远程升级的版本；bootstrap 完成后，常规重建/升级必须回到 `provider.upgrade mode=schedule`，不得再用 SSH 透传同步重建 `provider-gateway`。节点侧维护步骤是：进入节点本地 UniDesk 仓库，确认 GitHub 访问走本机 provider-gateway WS egress proxy，例如 `git config --local http.proxy http://127.0.0.1:18789` 和 `git config --local https.proxy http://127.0.0.1:18789` 后再执行 `git pull --ff-only` 获取主 server 已推送版本；不得让 provider 侧 Git 拉取退回直连公网、SSH SOCKS 或公开 master proxy。随后确认 `.state/provider-.env` 中存在 `PROVIDER_SERVER_URL=ws://74.48.78.17:18082/ws/provider`、`PROVIDER_ID=`、`PROVIDER_NAME=`、`PROVIDER_TOKEN`、`PROVIDER_LABELS_JSON`、`PROVIDER_UPGRADE_HOST_PROJECT_ROOT=/home/ubuntu/unidesk`、`PROVIDER_UPGRADE_WORKSPACE_PATH=/workspace`、`PROVIDER_UPGRADE_COMPOSE_FILE`、`PROVIDER_UPGRADE_ENV_FILE`、`PROVIDER_UPGRADE_COMPOSE_PROJECT`、`PROVIDER_UPGRADE_SERVICE=provider-gateway`、`PROVIDER_UPGRADE_RUNNER_IMAGE=unidesk_provider-gateway:`、`DOCKER_SOCKET_PATH=/var/run/docker.sock`、`MONITOR_DISK_PATH=/`、心跳和重连参数。旧 env 文件中如果还残留 `PROVIDER_UPGRADE_ENABLED`，新版 provider-gateway 会忽略它；长期文档和新部署不得再依赖这个键。 diff --git a/scripts/src/artifact-registry.ts b/scripts/src/artifact-registry.ts index 4db093d9..f3105fe4 100644 --- a/scripts/src/artifact-registry.ts +++ b/scripts/src/artifact-registry.ts @@ -1080,6 +1080,29 @@ function asBool(value: string | undefined): boolean { return value === "true"; } +function registryHealthDecision(checks: Record, commandOk: boolean): Record { + const scopeChecks: Array<[string, boolean]> = [ + ["systemd", checks.systemctlAvailable === true && checks.unitExists === true && checks.unitActive === true], + ["docker", checks.dockerAvailable === true], + ["registry-container", checks.containerRunning === true], + ["loopback-listener", checks.loopbackOnly === true], + ["registry-api", checks.v2Ok === true], + ["storage", checks.storageExists === true], + ["rendered-config", checks.configHashMatches === true && checks.composeHashMatches === true && checks.unitHashMatches === true], + ["registry-image", checks.imageMatches === true], + ]; + const healthyScopes = scopeChecks.filter(([, ok]) => ok).map(([scope]) => scope); + const failedScopes = scopeChecks.filter(([, ok]) => !ok).map(([scope]) => scope); + const runtimeApiHealthy = checks.containerRunning === true && checks.loopbackOnly === true && checks.v2Ok === true; + return { + decision: commandOk && failedScopes.length === 0 ? "healthy" : commandOk || runtimeApiHealthy ? "service-degraded" : "retryable-transient", + retryable: true, + healthyScopes, + failedScopes, + runtimeApiHealthy, + }; +} + function commandTail(result: CommandResult): Record { return { command: result.command.length > 7 ? [...result.command.slice(0, 7), ""] : result.command, @@ -1247,11 +1270,13 @@ function statusFromValues(options: ArtifactRegistryOptions, values: Record { ], "2026-05-20T00:00:00.000Z"); expect(result.blockingDisposition).toBe("runner-local-observation-gap"); + expect(result.decision).toBe("retryable-transient"); expect(result.retryable).toBe(true); expect(result.scope).toBe("runner-local"); + expect(result.failedScopes).toContain("runner-local"); expect(result.contract.singlePathProviderOfflineIsGlobalBlocker).toBe(false); }); @@ -38,6 +40,7 @@ describe("provider triage contract", () => { ], "2026-05-20T00:00:00.000Z"); expect(result.blockingDisposition).toBe("global-blocker"); + expect(result.decision).toBe("global-offline"); expect(result.retryable).toBe(false); expect(result.failedIndependentScopes).toContain("provider-gateway"); expect(result.failedIndependentScopes).toContain("ssh"); @@ -52,8 +55,26 @@ describe("provider triage contract", () => { ], "2026-05-20T00:00:00.000Z"); expect(result.blockingDisposition).toBe("service-degraded"); + expect(result.decision).toBe("service-degraded"); expect(result.retryable).toBe(true); + expect(result.failedScopes).toContain("registry"); + expect(result.healthyScopes).toContain("provider-gateway"); expect(result.healthyIndependentScopes).toContain("provider-gateway"); expect(result.healthyIndependentScopes).toContain("ssh"); }); + + test("registry systemd inactive with listener and api healthy is service degraded", () => { + const result = buildProviderTriageResult("D601", [ + signal("backend-core-node", "provider-gateway", "ok"), + signal("host-ssh-probe", "ssh", "ok"), + signal("artifact-registry-health", "registry", "degraded"), + signal("k3sctl-adapter-health", "k3s", "ok"), + ], "2026-05-20T00:00:00.000Z"); + + expect(result.decision).toBe("service-degraded"); + expect(result.blockingDisposition).toBe("service-degraded"); + expect(result.retryable).toBe(true); + expect(result.degradedScopes).toContain("registry"); + expect(result.healthyScopes).toEqual(expect.arrayContaining(["k3s", "provider-gateway", "ssh"])); + }); }); diff --git a/scripts/src/provider-triage.ts b/scripts/src/provider-triage.ts index e436bce0..d2b3d264 100644 --- a/scripts/src/provider-triage.ts +++ b/scripts/src/provider-triage.ts @@ -24,6 +24,12 @@ export type ProviderBlockingDisposition = | "service-degraded" | "global-blocker"; +export type ProviderTriageDecision = + | "healthy" + | "retryable-transient" + | "service-degraded" + | "global-offline"; + export interface ProviderTriageSignal { id: string; scope: ProviderSignalScope; @@ -36,11 +42,15 @@ export interface ProviderTriageSignal { export interface ProviderTriageClassification { scope: ProviderSignalScope; + decision: ProviderTriageDecision; observedAt: string; retryable: boolean; recommendedCrossChecks: string[]; blockingDisposition: ProviderBlockingDisposition; rationale: string[]; + failedScopes: ProviderSignalScope[]; + degradedScopes: ProviderSignalScope[]; + healthyScopes: ProviderSignalScope[]; failedIndependentScopes: ProviderSignalScope[]; healthyIndependentScopes: ProviderSignalScope[]; } @@ -193,8 +203,23 @@ function sshSignal(result: unknown, providerId: string): ProviderTriageSignal { function registrySignal(result: unknown): ProviderTriageSignal { const record = asRecord(result); if (record === null) return signal("artifact-registry-health", "registry", "unknown", "artifact registry health returned non-object", result); - const status: ProviderSignalStatus = record.ok === true && record.healthy !== false ? "ok" : record.ok === false ? "failed" : "degraded"; - return signal("artifact-registry-health", "registry", status, `artifact registry health ok=${String(record.ok)} healthy=${String(record.healthy)}`, { + const checks = asRecord(record.checks) ?? {}; + const runtimeApiHealthy = checks.containerRunning === true && checks.loopbackOnly === true && checks.v2Ok === true; + const status: ProviderSignalStatus = record.ok === true && record.healthy !== false + ? "ok" + : runtimeApiHealthy + ? "degraded" + : record.ok === false + ? "failed" + : "degraded"; + return signal("artifact-registry-health", "registry", status, [ + `artifact registry health ok=${String(record.ok)}`, + `healthy=${String(record.healthy)}`, + `unitActive=${String(checks.unitActive)}`, + `containerRunning=${String(checks.containerRunning)}`, + `loopbackOnly=${String(checks.loopbackOnly)}`, + `v2Ok=${String(checks.v2Ok)}`, + ].join(" "), { ok: record.ok, installed: record.installed ?? null, healthy: record.healthy ?? null, @@ -273,9 +298,9 @@ export function providerTriageRecommendedCrossChecks(providerId: string): string ]; } -function uniqueScopes(signals: ProviderTriageSignal[], statuses: ProviderSignalStatus[]): ProviderSignalScope[] { +function uniqueScopes(signals: ProviderTriageSignal[], statuses: ProviderSignalStatus[], independentOnly = true): ProviderSignalScope[] { return Array.from(new Set(signals - .filter((item) => item.independentPath) + .filter((item) => !independentOnly || item.independentPath) .filter((item) => statuses.includes(item.status)) .map((item) => item.scope))) .sort(); @@ -292,10 +317,11 @@ function primaryScope(signals: ProviderTriageSignal[]): ProviderSignalScope { } export function classifyProviderTriage(providerId: string, signals: ProviderTriageSignal[], observedAt = isoNow()): ProviderTriageClassification { - const failedScopes = uniqueScopes(signals, ["failed"]); - const degradedScopes = uniqueScopes(signals, ["degraded"]); + const failedScopes = uniqueScopes(signals, ["failed"], false); + const degradedScopes = uniqueScopes(signals, ["degraded"], false); const healthyScopes = uniqueScopes(signals, ["ok"]); - const independentFailedScopes = failedScopes.filter((scope) => scope !== "runner-local"); + const independentFailedScopes = uniqueScopes(signals, ["failed"]).filter((scope) => scope !== "runner-local"); + const independentDegradedScopes = uniqueScopes(signals, ["degraded"]); const failedCriticalScopes = independentFailedScopes.filter((scope) => criticalScopes.has(scope)); const runnerLocalObservedFailure = signals.some((signal) => signal.scope === "runner-local" && signal.status === "failed"); const serviceOnlyFailure = independentFailedScopes.length > 0 && independentFailedScopes.every((scope) => scope === "registry" || scope === "service-proxy" || scope === "microservice" || scope === "k3s"); @@ -312,7 +338,7 @@ export function classifyProviderTriage(providerId: string, signals: ProviderTria } else if (serviceOnlyFailure && hasIndependentHealthy) { blockingDisposition = "service-degraded"; rationale.push("service-scoped path failed while at least one provider-level path remains healthy"); - } else if (failedCriticalScopes.length > 0 || degradedScopes.some((scope) => criticalScopes.has(scope))) { + } else if (failedCriticalScopes.length > 0 || independentDegradedScopes.some((scope) => criticalScopes.has(scope))) { blockingDisposition = hasIndependentHealthy ? "provider-degraded" : "transient"; rationale.push(hasIndependentHealthy ? "provider-critical path is degraded but cross-checks still show independent healthy evidence" @@ -327,15 +353,27 @@ export function classifyProviderTriage(providerId: string, signals: ProviderTria if (runnerLocalObservedFailure) rationale.push("runner-local observation failed but is not counted as an independent global blocker by contract"); if (hasIndependentHealthy) rationale.push(`healthy independent scopes: ${healthyScopes.join(", ")}`); - if (failedScopes.length > 0) rationale.push(`failed independent scopes: ${failedScopes.join(", ")}`); + if (failedScopes.length > 0) rationale.push(`failed scopes: ${failedScopes.join(", ")}`); + const hasAnyIssueSignal = signals.some((item) => item.status === "failed" || item.status === "degraded"); + const decision: ProviderTriageDecision = blockingDisposition === "global-blocker" + ? "global-offline" + : blockingDisposition === "service-degraded" + ? "service-degraded" + : hasAnyIssueSignal + ? "retryable-transient" + : "healthy"; return { - scope: runnerLocalObservedFailure && failedScopes.length === 0 && degradedScopes.length === 0 ? "runner-local" : primaryScope(signals), + scope: runnerLocalObservedFailure && independentFailedScopes.length === 0 && independentDegradedScopes.length === 0 ? "runner-local" : primaryScope(signals), + decision, observedAt, retryable: blockingDisposition !== "global-blocker", recommendedCrossChecks: providerTriageRecommendedCrossChecks(providerId), blockingDisposition, rationale, + failedScopes, + degradedScopes, + healthyScopes, failedIndependentScopes: independentFailedScopes, healthyIndependentScopes: healthyScopes, };