Merge pull request #73 from pikasTech/fix/code-queue-active-run-diagnostics
fix(code-queue): clarify active-run diagnostics
This commit is contained in:
Lyon
@@ -163,7 +163,7 @@ bun scripts/cli.ts codex pr-preflight --remote --issue 20
|
||||
|
||||
`codex pr-preflight --remote` 的 `auth-missing` 只表示 scheduler/runtime preflight surface（`scheduler-runner-env`）没有看到 `GH_TOKEN/GITHUB_TOKEN` 或 auth-broker，不得被简化成“当前 active runner/dev container 不能创建 PR”。Code Queue 输出必须同时给出 `scopeBoundary` 和 `activeRunnerDevContainer`：前者说明 scheduler env 与当前 CLI/dev container 是独立 scope，后者只报告当前 CLI 进程是否看见 token，且不打印 token 值。指挥官看到 remote preflight `auth-missing` 时，应继续用当前 runner 内的 `bun scripts/cli.ts gh auth status --repo pikasTech/unidesk`、`gh pr create --dry-run`、`gh pr comment create --dry-run` 验证实际 PR 能力；只有这些 active runner 检查也失败时，才能把它判成当前 turn 不能 PR。
|
||||
|
||||
该命令经 backend-core 稳定 `code-queue` proxy 访问 D601 scheduler 的 `/api/runtime-preflight`，报告 scheduler/runner 环境里的 `GH_TOKEN`/`GITHUB_TOKEN` 覆盖、工具、Git worktree、GitHub egress、repo/issue/PR 只读探测和可选 push dry-run。需要复核 PR body/创建命令 guard 时追加 `--pr-create-dry-run --pr-create-dry-run-head <head>`；该 guard 只执行 dry-run，不创建 PR。缺少 env token 时必须返回 `ok=false`、`runnerDisposition=infra-blocked`、`tokenCoverage.missing=["GH_TOKEN","GITHUB_TOKEN"]` 和 `authBroker.source="broker/auth-broker-needed"`，因为 provider dev container 只能转发 scheduler 已经拥有的 token，除非后续接入 broker-held GitHub credential。系统 `gh` binary 缺失只能作为 `tools.systemGhBinary.ok=false` 观测，不得把它误判为 UniDesk REST `bun scripts/cli.ts gh` 不可用。`--remote` 在 runner-like 环境里不再要求本地 `unidesk-backend-core`、`unidesk-database`、`baidu-netdisk-backend` 容器存在；这些本地 target stack 缺失只作为证据，不是最终主阻塞。若远程控制面可达，输出继续保留 ready preflight；若远程控制面不可达，结构化失败归类为 `failureKind=control-plane-missing` / `degradedReason=remote-control-plane-unreachable`。输出中的 `prCapabilityContract` 用于指挥官快速审查 runner handoff：目标分支固定显示、push/PR create dry-run 标记为不写远端、系统 `gh` binary 与 UniDesk REST `bun scripts/cli.ts gh` 可用性分开报告，且 merge 明确保持 `unsupported-command`。
|
||||
该命令经 backend-core 稳定 `code-queue` proxy 访问 D601 scheduler 的 `/api/runtime-preflight`，报告 scheduler/runner 环境里的 `GH_TOKEN`/`GITHUB_TOKEN` 覆盖、工具、Git worktree、GitHub egress、repo/issue/PR 只读探测和可选 push dry-run。需要复核 PR body/创建命令 guard 时追加 `--pr-create-dry-run --pr-create-dry-run-head <head>`；该 guard 只执行 dry-run，不创建 PR。缺少 env token 时必须返回 `ok=false`、`runnerDisposition=infra-blocked`、`tokenCoverage.missing=["GH_TOKEN","GITHUB_TOKEN"]` 和 `authBroker.source="broker/auth-broker-needed"`，因为 provider dev container 只能转发 scheduler 已经拥有的 token，除非后续接入 broker-held GitHub credential。系统 `gh` binary 缺失只能作为 `tools.systemGhBinary.ok=false` 观测，不得把它误判为 UniDesk REST `bun scripts/cli.ts gh` 不可用。`--remote` 在 runner-like 环境里不再要求本地 `unidesk-backend-core`、`unidesk-database`、`baidu-netdisk-backend` 容器存在；这些本地 target stack 缺失只作为证据，不是最终主阻塞，并应额外标成 `blockingDisposition=runner-local-observation-gap` 或 `localObservationGap.kind=runner-local-observation-gap`。若远程控制面可达，输出继续保留 ready preflight；若远程控制面不可达，结构化失败归类为 `failureKind=control-plane-missing` / `degradedReason=remote-control-plane-unreachable`，并额外标成 `blockingDisposition=control-plane-observation-gap`。`runnerDisposition` 可以为了旧调用方兼容继续保持 `infra-blocked`，但 observation-gap 字段才是判断“观测路径缺口，不是 scheduler 停摆”的稳定口径。输出中的 `prCapabilityContract` 用于指挥官快速审查 runner handoff：目标分支固定显示、push/PR create dry-run 标记为不写远端、系统 `gh` binary 与 UniDesk REST `bun scripts/cli.ts gh` 可用性分开报告，且 merge 明确保持 `unsupported-command`。
|
||||
|
||||
本地 runner preflight 示例：
|
||||
|
||||
@@ -215,7 +215,7 @@ bun scripts/cli.ts codex pr-preflight --remote --issue <issue-number>
|
||||
|
||||
完成未读任务的审阅也必须遵循渐进披露。指挥官默认只拉取原始 prompt 和最终 response，用它判断任务是否声称完成、是否有明显越界、是否缺少验收证据；不要默认拉完整 trace、全量 tool summary 或 raw output。只有当 final response 与目标不一致、证据不足、远端 commit 无法验证、任务疑似造假、或需要追溯失败原因时，才继续展开 `--detail`、分页 `--trace`、或按 seq 读取 `codex output`。这条规则的目标是降低上下文压力，同时保留通过多步查询拿到完整证据的能力。
|
||||
|
||||
队列诊断中的 `split-brain` 表示控制面/执行面观测分裂，不自动证明任务已经死亡。只要任务 heartbeat 还在刷新、trace 仍在推进，就不能把它判成服务中断或要求立刻 stop；应把它视为 `splitBrainLive=true` 的 live 任务，继续监督并推进 #20 里的已排任务，而不是 interrupt、替换或把 backend 当成已经挂掉。队列摘要应显示 `effectiveLiveness=live`、`splitBrainLive=true` 和 `recommendedAction=continue-supervision`；只有 heartbeat expired/missing 或满足 stale-recovery 条件时，才应显示 `effectiveLiveness=at-risk` 并进入恢复判断。
|
||||
队列诊断中的 `split-brain` 表示控制面/执行面观测分裂，不自动证明任务已经死亡。只要任务 heartbeat 还在刷新、trace 仍在推进，就不能把它判成服务中断或要求立刻 stop；应把它视为 `splitBrainLive=true` 的 live 任务，继续监督并推进 #20 里的已排任务，而不是 interrupt、替换或把 backend 当成已经挂掉。队列摘要应显示 `effectiveLiveness=live`、`splitBrainLive=true` 和 `recommendedAction=continue-supervision`；compact 输出还应在 `executionDiagnostics.liveness` 中重复这些低噪声字段，并突出 `activeHeartbeatCount`、有界 `heartbeatFreshTaskIds`、`databaseActiveTaskCount` 和 `schedulerActiveRunSlotCount`。当 master/control-plane 的 `schedulerActiveRunSlotCount=0` 但 `heartbeatFreshTaskIds` 非空时，active 数应优先按 scheduler heartbeat 摘要解释为 live，而不是按 master 本地 slot 0 解释为执行停摆。只有 heartbeat expired/missing 或满足 stale-recovery 条件时，才应显示 `effectiveLiveness=at-risk` 并进入恢复判断。
|
||||
|
||||
单次 `provider is not online`、SSH 超时、proxy 超时或 registry 请求失败只能证明“当前观察路径失败”，不能单独升级为 D601 全局离线、CI/CD 全局阻塞或业务任务不可推进。指挥官和 runner 必须用多信号裁决运行面状态，至少区分以下观察面：
|
||||
|
||||
@@ -225,7 +225,7 @@ bun scripts/cli.ts codex pr-preflight --remote --issue <issue-number>
|
||||
- Code Queue scheduler heartbeat、任务 heartbeat、trace/output 是否持续入库；
|
||||
- 当前 runner 容器内 CLI/proxy 路径是否只是局部不可达。
|
||||
|
||||
只有多个独立观察面同时失败，或同一关键路径在明确时间窗口内持续失败，才能把问题判为全局阻塞。否则应记录为 transient 或 runner-local observation gap，优先重试、steer 任务纠偏或拆出基础设施 follow-up；不得让业务 worker 把单次局部失败作为最终 blocker。CLI 和 runtime 必须把错误输出结构化为 `scope=runner-local|provider-gateway|ssh|registry|k3s|scheduler|service-proxy`、`observedAt`、`retryable`、`decision`、`healthyScopes`、`failedScopes` 和建议的交叉验证命令。
|
||||
只有多个独立观察面同时失败，或同一关键路径在明确时间窗口内持续失败，才能把问题判为全局阻塞。否则应记录为 transient、`runner-local-observation-gap` 或 `control-plane-observation-gap`，优先重试、steer 任务纠偏或拆出基础设施 follow-up；不得让业务 worker 把单次局部失败作为最终 blocker。CLI 和 runtime 必须把错误输出结构化为 `scope=runner-local|control-plane|provider-gateway|ssh|registry|k3s|scheduler|service-proxy`、`observedAt`、`retryable`、`decision` 或 `blockingDisposition`、`healthyScopes`、`failedScopes` 和建议的交叉验证命令。当前 runner/local backend-core 容器缺失属于 runner-local observation gap；远程控制面也不可达属于 control-plane observation gap；两者都不能单独写成 active runner 数归零或 scheduler 停摆。
|
||||
|
||||
ClaudeQQ 是面向用户的主动提醒通道，不是 #24 简报更新的自动转发器。指挥官只应在三类情况下自主发送 ClaudeQQ 消息：核心服务或关键执行面宕机且需要用户知情，高风险决策需要用户请示，或出现里程碑式进展值得同步。消息必须简明扼要，一次不超过 200 个中文字符，写成一段话，不使用 Markdown 语法。普通轮询、普通 issue 更新、普通 #24 简报追加、外部 token provider 正常限流、以及无用户动作要求的中间状态，不发送 ClaudeQQ。发送失败只记录到 #24 或对应 blocker issue，不回滚已经完成的 GitHub issue 更新。
|
||||
|
||||
|
||||
@@ -339,6 +339,9 @@ async function main(): Promise<void> {
|
||||
assertCondition(fallback.runnerDisposition === "ready", "remote fallback should stay ready", fallback);
|
||||
assertCondition(fallback.controlPlane && asRecord(fallback.controlPlane).remoteFallbackUsed === true, "remote fallback should be marked", fallback.controlPlane);
|
||||
assertCondition(fallback.failureKind === null, "remote fallback should not invent a failure kind when remote control plane is healthy", fallback);
|
||||
const fallbackLocalGap = asRecord(fallback.localObservationGap);
|
||||
assertCondition(fallbackLocalGap.kind === "runner-local-observation-gap", "healthy remote fallback should classify local backend-core absence as runner-local observation gap", fallbackLocalGap);
|
||||
assertCondition(fallbackLocalGap.schedulerStoppage === false, "local observation gap must not imply scheduler stoppage", fallbackLocalGap);
|
||||
const fallbackPreflight = asRecord(fallback.preflight);
|
||||
assertCondition(fallbackPreflight.ok === true, "remote fallback preflight should stay ready", fallbackPreflight);
|
||||
assertCondition(asRecord(fallbackPreflight.tokenCoverage).source === "GH_TOKEN", "token source should be GH_TOKEN", fallbackPreflight.tokenCoverage);
|
||||
@@ -352,6 +355,11 @@ async function main(): Promise<void> {
|
||||
assertCondition(remoteControlPlaneMissingRecord.ok === false, "missing control plane should fail", remoteControlPlaneMissingRecord);
|
||||
assertCondition(remoteControlPlaneMissingRecord.failureKind === "control-plane-missing", "missing control plane should classify as control-plane-missing", remoteControlPlaneMissingRecord);
|
||||
assertCondition(remoteControlPlaneMissingRecord.degradedReason === "remote-control-plane-unreachable", "missing control plane should classify as remote-control-plane-unreachable", remoteControlPlaneMissingRecord);
|
||||
assertCondition(remoteControlPlaneMissingRecord.runnerDisposition === "infra-blocked", "missing remote control plane keeps legacy runnerDisposition compatibility", remoteControlPlaneMissingRecord);
|
||||
assertCondition(remoteControlPlaneMissingRecord.blockingDisposition === "control-plane-observation-gap", "missing remote control plane should expose observation gap blocking disposition", remoteControlPlaneMissingRecord);
|
||||
const remoteControlPlaneGap = asRecord(remoteControlPlaneMissingRecord.observationGap);
|
||||
assertCondition(remoteControlPlaneGap.kind === "control-plane-observation-gap", "missing remote control plane should expose control-plane observation gap", remoteControlPlaneGap);
|
||||
assertCondition(remoteControlPlaneGap.schedulerStoppage === false, "control-plane observation gap must not imply scheduler stoppage", remoteControlPlaneGap);
|
||||
assertCondition(asRecord(remoteControlPlaneMissingRecord.controlPlane).localBackendCoreMissing === true, "local backend-core absence should remain evidence only", remoteControlPlaneMissingRecord.controlPlane);
|
||||
|
||||
const directAuthMissing = await codexPrPreflightQueryForTest(["--remote"], {
|
||||
@@ -389,6 +397,8 @@ async function main(): Promise<void> {
|
||||
assertCondition(directAuthMissingRecord.ok === false, "auth-missing remote result should fail", directAuthMissingRecord);
|
||||
assertCondition(directAuthMissingRecord.failureKind === "auth-missing", "missing token should classify as auth-missing", directAuthMissingRecord);
|
||||
assertCondition(directAuthMissingRecord.degradedReason === "GH_TOKEN/GITHUB_TOKEN missing", "auth missing should state token gap", directAuthMissingRecord);
|
||||
const directAuthObservationGap = asRecord(directAuthMissingRecord.observationGap);
|
||||
assertCondition(directAuthObservationGap.kind === "runner-local-observation-gap", "auth missing after remote fallback should keep local backend-core absence scoped as runner-local observation gap", directAuthObservationGap);
|
||||
const directAuthScopeBoundary = asRecord(directAuthMissingRecord.scopeBoundary);
|
||||
const directAuthActiveRunner = asRecord(directAuthMissingRecord.activeRunnerDevContainer);
|
||||
assertCondition(directAuthScopeBoundary.scopesAreIndependent === true, "remote auth-missing must distinguish scheduler env from active runner dev container", directAuthScopeBoundary);
|
||||
@@ -405,6 +415,19 @@ async function main(): Promise<void> {
|
||||
const gitRemoteGapRecord = asRecord(gitRemoteGap);
|
||||
assertCondition(gitRemoteGapRecord.failureKind === "git-remote-gap", "git probe failures should stay structured", gitRemoteGapRecord);
|
||||
|
||||
const localOnlyObservationGap = await codexPrPreflightQueryForTest([], {
|
||||
config: null,
|
||||
coreFetch: () => localBackendCoreMissingFixture(),
|
||||
});
|
||||
const localOnlyObservationGapRecord = asRecord(localOnlyObservationGap);
|
||||
assertCondition(localOnlyObservationGapRecord.ok === false, "local-only backend-core absence should fail the preflight", localOnlyObservationGapRecord);
|
||||
assertCondition(localOnlyObservationGapRecord.failureKind === "target-stack-not-running", "local-only backend-core absence should preserve target-stack evidence", localOnlyObservationGapRecord);
|
||||
assertCondition(localOnlyObservationGapRecord.runnerDisposition === "infra-blocked", "local-only backend-core absence keeps legacy runnerDisposition compatibility", localOnlyObservationGapRecord);
|
||||
assertCondition(localOnlyObservationGapRecord.blockingDisposition === "runner-local-observation-gap", "local-only backend-core absence should expose runner-local blocking disposition", localOnlyObservationGapRecord);
|
||||
const localOnlyGap = asRecord(localOnlyObservationGapRecord.observationGap);
|
||||
assertCondition(localOnlyGap.kind === "runner-local-observation-gap", "local-only backend-core absence should include observationGap detail", localOnlyGap);
|
||||
assertCondition(localOnlyGap.schedulerStoppage === false, "local-only backend-core absence must not imply scheduler stoppage", localOnlyGap);
|
||||
|
||||
const proxyGap = await codexPrPreflightQueryForTest(["--remote"], {
|
||||
config: null,
|
||||
coreFetch: () => ({
|
||||
@@ -741,7 +764,8 @@ async function main(): Promise<void> {
|
||||
checks: [
|
||||
"runner-like local target-stack absence does not block remote fallback",
|
||||
"remote control plane fallback preserves ready preflight",
|
||||
"missing remote control plane returns control-plane-missing",
|
||||
"missing remote control plane returns control-plane-observation-gap",
|
||||
"local backend-core absence returns runner-local-observation-gap",
|
||||
"auth missing returns auth-missing with broker/auth-broker-needed",
|
||||
"proxy failures return proxy-gap",
|
||||
"git remote failures return git-remote-gap",
|
||||
|
||||
@@ -36,8 +36,8 @@ function fixtureResponse(): JsonRecord {
|
||||
heartbeatFreshTaskIds: ["task-running"],
|
||||
databaseActiveTaskCount: 1,
|
||||
databaseActiveTaskIds: ["task-running"],
|
||||
schedulerActiveRunSlotCount: 1,
|
||||
schedulerActiveTaskIds: ["task-running"],
|
||||
schedulerActiveRunSlotCount: 0,
|
||||
schedulerActiveTaskIds: [],
|
||||
},
|
||||
},
|
||||
queues: [
|
||||
@@ -93,6 +93,13 @@ function assertQueuesShape(label: string, result: unknown, expectedView: string)
|
||||
assertCondition(diagnostics.splitBrainLive === true, `${label} split-brain live should remain explicitly true`, diagnostics);
|
||||
assertCondition(diagnostics.effectiveLiveness === "live", `${label} diagnostics should retain derived liveness`, diagnostics);
|
||||
assertCondition(diagnostics.recommendedAction === "continue-supervision", `${label} split-brain live should continue supervision`, diagnostics);
|
||||
const liveness = asRecord(diagnostics.liveness);
|
||||
assertCondition(liveness.effectiveLiveness === "live", `${label} liveness summary should foreground effective live state`, liveness);
|
||||
assertCondition(liveness.recommendedAction === "continue-supervision", `${label} liveness summary should foreground recommended action`, liveness);
|
||||
assertCondition(liveness.activeHeartbeatCount === 1, `${label} liveness summary should derive active heartbeat count from fresh heartbeat ids`, liveness);
|
||||
assertCondition(liveness.schedulerActiveRunSlotCount === 0, `${label} liveness summary should keep master active slot zero visible`, liveness);
|
||||
assertCondition(asArray(liveness.heartbeatFreshTaskIds).length === 1, `${label} liveness summary should include bounded fresh heartbeat task ids`, liveness);
|
||||
assertCondition(String(liveness.interpretation ?? "").includes("heartbeat is fresh"), `${label} liveness interpretation should explain slot-zero split-brain`, liveness);
|
||||
assertCondition(Array.isArray(queues.activeTaskIds), `${label} activeTaskIds should be present`, queues);
|
||||
assertCondition(Array.isArray(queues.queuedTaskIds), `${label} queuedTaskIds should be present`, queues);
|
||||
}
|
||||
|
||||
@@ -157,6 +157,7 @@ export function runCodeQueueSupervisorDisclosureContract(): JsonRecord {
|
||||
const completedUnread = asRecord(supervisorView.completedUnread);
|
||||
const fullTasks = asRecord(asRecord(full).tasks);
|
||||
const diagnostics = asRecord(supervisorView.executionDiagnostics);
|
||||
const liveness = asRecord(diagnostics.liveness);
|
||||
const listBudget = asRecord(diagnostics.listBudget);
|
||||
const omittedCounts = asRecord(listBudget.omittedCounts);
|
||||
|
||||
@@ -167,6 +168,12 @@ export function runCodeQueueSupervisorDisclosureContract(): JsonRecord {
|
||||
assertCondition(recentItems.every((item) => asRecord(item).unreadTerminal === false), "recentCompleted should not duplicate unread terminal tasks", { recentItems });
|
||||
assertCondition(asArray(diagnostics.databaseActiveTaskIds).length === 12, "diagnostic task id lists should be capped", diagnostics);
|
||||
assertCondition(omittedCounts.databaseActiveTaskIds === 68, "diagnostic omitted counts should preserve full visibility metadata", omittedCounts);
|
||||
assertCondition(liveness.effectiveLiveness === "live", "supervisor liveness summary should keep split-brain live explicit", liveness);
|
||||
assertCondition(liveness.recommendedAction === "continue-supervision", "supervisor liveness summary should recommend continued supervision", liveness);
|
||||
assertCondition(liveness.splitBrainLive === true, "supervisor liveness summary should mark splitBrainLive", liveness);
|
||||
assertCondition(liveness.activeHeartbeatCount === 80, "supervisor liveness summary should foreground active heartbeat count", liveness);
|
||||
assertCondition(asArray(liveness.heartbeatFreshTaskIds).length === 12, "supervisor liveness summary should keep heartbeatFreshTaskIds bounded", liveness);
|
||||
assertCondition(String(liveness.interpretation ?? "").includes("continue supervision"), "supervisor liveness interpretation should not imply scheduler stoppage", liveness);
|
||||
assertCondition(asArray(diagnostics.reasons).length === 6, "diagnostic reasons should be capped", diagnostics);
|
||||
assertCondition(diagnostics.livenessSummaryTruncated === true, "long diagnostic liveness summary should be previewed", diagnostics);
|
||||
assertCondition(listBudget.truncated === true && typeof listBudget.rawCommand === "string", "diagnostic list budget should disclose raw command", listBudget);
|
||||
|
||||
+122
-4
@@ -260,6 +260,7 @@ interface CodexPrPreflightOptions {
|
||||
}
|
||||
|
||||
type CodeQueuePrPreflightFailureKind = "auth-missing" | "proxy-gap" | "git-remote-gap" | "control-plane-missing" | "target-stack-not-running";
|
||||
type CodeQueueObservationGapKind = "runner-local-observation-gap" | "control-plane-observation-gap" | null;
|
||||
|
||||
interface CodeQueuePrPreflightTransport {
|
||||
config?: UniDeskConfig | null;
|
||||
@@ -1015,6 +1016,42 @@ function recommendedActionFromDiagnostics(record: Record<string, unknown>): stri
|
||||
return "none";
|
||||
}
|
||||
|
||||
function activeHeartbeatCountFromDiagnostics(record: Record<string, unknown>, activeHeartbeatTaskIds: { count: number }, heartbeatFreshTaskIds: { count: number }): number {
|
||||
const explicit = asNumber(record.activeHeartbeatCount, Number.NaN);
|
||||
return Number.isFinite(explicit) ? explicit : Math.max(activeHeartbeatTaskIds.count, heartbeatFreshTaskIds.count);
|
||||
}
|
||||
|
||||
function compactLivenessDecision(record: Record<string, unknown>, lists: {
|
||||
activeHeartbeatTaskIds: { items: string[]; count: number; truncated: boolean; omitted: number };
|
||||
heartbeatFreshTaskIds: { items: string[]; count: number; truncated: boolean; omitted: number };
|
||||
heartbeatRiskTaskIds: { items: string[]; count: number };
|
||||
databaseActiveTaskIds: { count: number };
|
||||
}): Record<string, unknown> {
|
||||
const splitBrainLive = splitBrainLiveFromDiagnostics(record);
|
||||
const effectiveLiveness = effectiveLivenessFromDiagnostics(record);
|
||||
const recommendedAction = recommendedActionFromDiagnostics(record);
|
||||
const activeHeartbeatCount = activeHeartbeatCountFromDiagnostics(record, lists.activeHeartbeatTaskIds, lists.heartbeatFreshTaskIds);
|
||||
return {
|
||||
effectiveLiveness,
|
||||
recommendedAction,
|
||||
splitBrainLive,
|
||||
activeHeartbeatCount,
|
||||
heartbeatFreshTaskCount: lists.heartbeatFreshTaskIds.count,
|
||||
heartbeatFreshTaskIds: lists.heartbeatFreshTaskIds.items,
|
||||
heartbeatFreshTaskIdsTruncated: lists.heartbeatFreshTaskIds.truncated,
|
||||
databaseActiveTaskCount: asNumber(record.databaseActiveTaskCount, lists.databaseActiveTaskIds.count),
|
||||
schedulerActiveRunSlotCount: record.schedulerActiveRunSlotCount ?? null,
|
||||
heartbeatRiskTaskCount: lists.heartbeatRiskTaskIds.count,
|
||||
interpretation: splitBrainLive
|
||||
? "scheduler heartbeat is fresh; treat active task count from heartbeat as live and continue supervision"
|
||||
: effectiveLiveness === "at-risk"
|
||||
? "heartbeat risk is present; investigate heartbeat freshness before recovery"
|
||||
: effectiveLiveness === "degraded"
|
||||
? "diagnostics are degraded; cross-check heartbeat, trace and control-plane sources"
|
||||
: "diagnostics indicate healthy liveness",
|
||||
};
|
||||
}
|
||||
|
||||
function boundedUniqueStringList(value: unknown, limit = diagnosticsIdPreviewLimit): { items: string[]; count: number; omitted: number; truncated: boolean } {
|
||||
const all = Array.from(new Set(stringList(value))).sort();
|
||||
const items = all.slice(0, limit);
|
||||
@@ -1059,6 +1096,12 @@ function compactExecutionDiagnostics(value: unknown): Record<string, unknown> |
|
||||
const allReasons = stringList(record.reasons);
|
||||
const reasons = allReasons.slice(0, diagnosticsReasonPreviewLimit).map((reason) => boundedInlineString(reason, 240).text).filter((reason): reason is string => reason !== null);
|
||||
const livenessSummary = boundedInlineString(record.livenessSummary, 420);
|
||||
const liveness = compactLivenessDecision(record, {
|
||||
activeHeartbeatTaskIds,
|
||||
heartbeatFreshTaskIds,
|
||||
heartbeatRiskTaskIds,
|
||||
databaseActiveTaskIds,
|
||||
});
|
||||
const omittedCounts = {
|
||||
databaseActiveTaskIds: databaseActiveTaskIds.omitted,
|
||||
schedulerActiveTaskIds: schedulerActiveTaskIds.omitted,
|
||||
@@ -1078,8 +1121,9 @@ function compactExecutionDiagnostics(value: unknown): Record<string, unknown> |
|
||||
degraded: record.degraded ?? null,
|
||||
splitBrain: record.splitBrain ?? null,
|
||||
splitBrainLive: splitBrainLiveFromDiagnostics(record),
|
||||
effectiveLiveness: effectiveLivenessFromDiagnostics({ ...record, heartbeatRiskTaskIds: fullHeartbeatRiskTaskIds }),
|
||||
recommendedAction: recommendedActionFromDiagnostics({ ...record, heartbeatRiskTaskIds: fullHeartbeatRiskTaskIds }),
|
||||
effectiveLiveness: liveness.effectiveLiveness,
|
||||
recommendedAction: liveness.recommendedAction,
|
||||
liveness,
|
||||
livenessSummary: livenessSummary.text,
|
||||
livenessSummaryChars: livenessSummary.chars,
|
||||
livenessSummaryTruncated: livenessSummary.truncated,
|
||||
@@ -1089,7 +1133,7 @@ function compactExecutionDiagnostics(value: unknown): Record<string, unknown> |
|
||||
databaseActiveTaskIds: databaseActiveTaskIds.items,
|
||||
schedulerActiveRunSlotCount: record.schedulerActiveRunSlotCount ?? null,
|
||||
schedulerActiveTaskIds: schedulerActiveTaskIds.items,
|
||||
activeHeartbeatCount: record.activeHeartbeatCount ?? activeHeartbeatTaskIds.count,
|
||||
activeHeartbeatCount: liveness.activeHeartbeatCount,
|
||||
activeHeartbeatTaskIds: activeHeartbeatTaskIds.items,
|
||||
heartbeatFreshTaskIds: heartbeatFreshTaskIds.items,
|
||||
heartbeatExpiredTaskIds: heartbeatExpiredTaskIds.items,
|
||||
@@ -2875,6 +2919,30 @@ function decoratePrPreflightScopeBoundary(record: Record<string, unknown>): Reco
|
||||
};
|
||||
}
|
||||
|
||||
function prPreflightObservationGap(kind: Exclude<CodeQueueObservationGapKind, null>, detail: {
|
||||
reason: string;
|
||||
localBackendCoreMissing?: boolean;
|
||||
remoteFallbackUsed?: boolean;
|
||||
remoteControlPlaneReachable?: boolean | null;
|
||||
}): Record<string, unknown> {
|
||||
const controlPlaneGap = kind === "control-plane-observation-gap";
|
||||
return {
|
||||
kind,
|
||||
blockingDisposition: kind,
|
||||
scope: controlPlaneGap ? "control-plane" : "runner-local",
|
||||
reason: detail.reason,
|
||||
localBackendCoreMissing: detail.localBackendCoreMissing === true,
|
||||
remoteFallbackUsed: detail.remoteFallbackUsed === true,
|
||||
remoteControlPlaneReachable: detail.remoteControlPlaneReachable ?? null,
|
||||
schedulerStoppage: false,
|
||||
schedulerStateMachineChanged: false,
|
||||
recommendedAction: controlPlaneGap ? "cross-check-control-plane" : "retry-from-control-plane-or-remote-fallback",
|
||||
note: controlPlaneGap
|
||||
? "The control-plane observation path is unavailable; this is not evidence that the scheduler stopped executing active tasks."
|
||||
: "The current runner/local backend-core observation path is unavailable; this is not evidence that active Code Queue execution stopped.",
|
||||
};
|
||||
}
|
||||
|
||||
function compactPrRuntimePreflight(preflight: Record<string, unknown>, options: CodexPrPreflightOptions): Record<string, unknown> {
|
||||
const pull = asRecord(preflight.pullRequestDelivery) ?? {};
|
||||
const tools = asRecord(pull.tools) ?? {};
|
||||
@@ -3056,9 +3124,16 @@ function queryRemoteMainServerPrPreflight(optionArgs: string[], config: UniDeskC
|
||||
return {
|
||||
ok: false,
|
||||
runnerDisposition: "infra-blocked",
|
||||
blockingDisposition: "control-plane-observation-gap",
|
||||
failureKind: "control-plane-missing",
|
||||
degradedReason: "remote-control-plane-unreachable",
|
||||
message,
|
||||
observationGap: prPreflightObservationGap("control-plane-observation-gap", {
|
||||
reason: "remote control plane CLI returned a structured error",
|
||||
localBackendCoreMissing: false,
|
||||
remoteFallbackUsed: true,
|
||||
remoteControlPlaneReachable: false,
|
||||
}),
|
||||
controlPlane: {
|
||||
mode: "remote-frontend",
|
||||
host: config.network.publicHost,
|
||||
@@ -3081,9 +3156,16 @@ function queryRemoteMainServerPrPreflight(optionArgs: string[], config: UniDeskC
|
||||
return {
|
||||
ok: false,
|
||||
runnerDisposition: "infra-blocked",
|
||||
blockingDisposition: "control-plane-observation-gap",
|
||||
failureKind: "control-plane-missing",
|
||||
degradedReason: "remote-control-plane-unreachable",
|
||||
message,
|
||||
observationGap: prPreflightObservationGap("control-plane-observation-gap", {
|
||||
reason: "remote control plane CLI could not be reached or returned non-JSON output",
|
||||
localBackendCoreMissing: false,
|
||||
remoteFallbackUsed: true,
|
||||
remoteControlPlaneReachable: false,
|
||||
}),
|
||||
controlPlane: {
|
||||
mode: "remote-frontend",
|
||||
host: config.network.publicHost,
|
||||
@@ -3128,6 +3210,18 @@ function codeQueuePrPreflight(optionArgs: string[] = [], transport: CodeQueuePrP
|
||||
if (remoteRecord.ok === false) {
|
||||
return decoratePrPreflightScopeBoundary({
|
||||
...remoteRecord,
|
||||
observationGap: prPreflightObservationGap(
|
||||
remoteRecord.failureKind === "control-plane-missing" ? "control-plane-observation-gap" : "runner-local-observation-gap",
|
||||
{
|
||||
reason: remoteRecord.failureKind === "control-plane-missing"
|
||||
? "remote control plane could not be observed after local backend-core target-stack absence"
|
||||
: "local backend-core target-stack absence was bypassed by remote fallback, but the remote result still failed",
|
||||
localBackendCoreMissing: true,
|
||||
remoteFallbackUsed: true,
|
||||
remoteControlPlaneReachable: remoteRecord.failureKind === "control-plane-missing" ? false : true,
|
||||
},
|
||||
),
|
||||
blockingDisposition: remoteRecord.failureKind === "control-plane-missing" ? "control-plane-observation-gap" : remoteRecord.blockingDisposition ?? remoteRecord.runnerDisposition ?? "infra-blocked",
|
||||
controlPlane: {
|
||||
...(asRecord(remoteRecord.controlPlane) ?? {}),
|
||||
mode: "remote-frontend",
|
||||
@@ -3142,6 +3236,12 @@ function codeQueuePrPreflight(optionArgs: string[] = [], transport: CodeQueuePrP
|
||||
}
|
||||
return decoratePrPreflightScopeBoundary({
|
||||
...remoteRecord,
|
||||
localObservationGap: prPreflightObservationGap("runner-local-observation-gap", {
|
||||
reason: "local backend-core target-stack absence was bypassed by healthy remote control-plane fallback",
|
||||
localBackendCoreMissing: true,
|
||||
remoteFallbackUsed: true,
|
||||
remoteControlPlaneReachable: true,
|
||||
}),
|
||||
controlPlane: {
|
||||
...(asRecord(remoteRecord.controlPlane) ?? {}),
|
||||
mode: "remote-frontend",
|
||||
@@ -3163,9 +3263,16 @@ function codeQueuePrPreflight(optionArgs: string[] = [], transport: CodeQueuePrP
|
||||
...(localRecord ?? {}),
|
||||
ok: false,
|
||||
runnerDisposition: "infra-blocked",
|
||||
blockingDisposition: "control-plane-observation-gap",
|
||||
failureKind,
|
||||
degradedReason,
|
||||
message: "remote control plane unreachable; local backend-core target-stack absence is evidence only",
|
||||
message: "remote control plane unreachable; local backend-core target-stack absence is observation-gap evidence only",
|
||||
observationGap: prPreflightObservationGap("control-plane-observation-gap", {
|
||||
reason: "local backend-core target stack is missing and no remote control plane could be observed",
|
||||
localBackendCoreMissing: true,
|
||||
remoteFallbackUsed: false,
|
||||
remoteControlPlaneReachable: false,
|
||||
}),
|
||||
controlPlane: {
|
||||
mode: "local-backend-core",
|
||||
localBackendCoreMissing: true,
|
||||
@@ -3183,9 +3290,20 @@ function codeQueuePrPreflight(optionArgs: string[] = [], transport: CodeQueuePrP
|
||||
...(localRecord ?? {}),
|
||||
ok: false,
|
||||
runnerDisposition: localRecord?.runnerDisposition ?? "infra-blocked",
|
||||
blockingDisposition: localTargetStackMissing ? "runner-local-observation-gap" : localRecord?.blockingDisposition ?? localRecord?.runnerDisposition ?? "infra-blocked",
|
||||
failureKind: (localRecord?.failureKind as CodeQueuePrPreflightFailureKind | undefined) ?? "proxy-gap",
|
||||
degradedReason: localRecord?.degradedReason ?? "backend-core-proxy-unavailable",
|
||||
message: localRecord?.message ?? localRecord?.stderrTail ?? localRecord?.stdoutTail ?? "Code Queue runtime preflight could not be observed",
|
||||
...(localTargetStackMissing
|
||||
? {
|
||||
observationGap: prPreflightObservationGap("runner-local-observation-gap", {
|
||||
reason: "local backend-core target stack is missing in this runner observation path",
|
||||
localBackendCoreMissing: true,
|
||||
remoteFallbackUsed: false,
|
||||
remoteControlPlaneReachable: null,
|
||||
}),
|
||||
}
|
||||
: {}),
|
||||
controlPlane: {
|
||||
mode: "local-backend-core",
|
||||
localBackendCoreMissing: localTargetStackMissing,
|
||||
|
||||
Reference in New Issue
Block a user