diff --git a/docs/reference/cli.md b/docs/reference/cli.md index 7d6e7101..94023a35 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -47,7 +47,7 @@ CLI 可以从 `master` 快速演进，但必须兼容 `deploy.json` 固定的 CI - `codex submit [prompt] [--prompt-file path|--prompt-stdin] [--queue queueId] [--provider-id id] [--cwd path] [--model model] [--reasoning-effort effort] [--execution-mode mode] [--max-attempts N] [--reference-task-id id] [--dry-run]` 通过 backend-core 私有代理向稳定 `code-queue` 用户服务路径提交任务；prompt 必须且只能来自位置参数、文件或 stdin 之一，`--dry-run` 只返回结构化请求且不实际入队。长 prompt、多行 prompt、含引号/反引号/Markdown 表格/JSON/反斜杠的 prompt 必须优先用 `--prompt-stdin` 或 `--prompt-file`，不要拼进 shell 单个参数；位置参数只适合短单行 smoke prompt。stdin 推荐用 quoted heredoc：`cat <<'PROMPT' | bun scripts/cli.ts codex submit --prompt-stdin --queue --dry-run`，文件路径推荐 `bun scripts/cli.ts codex submit --prompt-file /tmp/code-queue-prompt.md --queue --dry-run`，确认 dry-run 后移除 `--dry-run` 提交同一 payload。dry-run 会额外输出 `routingRecommendation`，包含推荐 route、runner、model、风险信号、prompt 自包含/issue 非唯一来源/prod-secret-DB 禁止/运行态或 release 禁止/证据要求/中等复杂度候选等 guard 状态；同时输出 `policyContract`，固定暴露 GPT-5.5、DeepSeek、MiniMax 的风险分层、并发上限和外部 provider 429 退避处置。该建议只用于指挥官 preflight，不会改写 payload，不改变 runtime admission，也不假设生产 MiniMax 或 DeepSeek 可用。`--dry-run` 必须返回完整 prompt、字符数和 `truncated=false` 用于人工验收；真实提交是写入操作，默认只返回 `accepted=true`、task id、队列、写入保护摘要和后续查看命令，必须标记 `promptOmitted=true` 且不得回显 prompt 或 promptPreview。真实提交会经过本机本地串行化保护和短节流，避免同一指挥端并发 submit 把低内存主机或 `code-queue-mgr` 控制面打抖；返回值会附带低噪声 `submitConcurrencyGuard` 说明本次提交的锁与等待信息。backend-core 默认把提交、队列 CRUD、已读状态、历史摘要和轻量 Trace 读取分流到主 server `code-queue-mgr`，由它写入主 PostgreSQL；D601 scheduler 只轮询并执行已入库任务。 - `codex pr-preflight [--remote] [--push-dry-run --push-dry-run-ref refs/heads/probe/] [--pr-create-dry-run --pr-create-dry-run-head ] [--issue N] [--full]` 通过稳定 `code-queue` proxy 请求 D601 scheduler `/api/runtime-preflight`，用于 PR 型派单 admission。输出会压缩展示 scheduler/runner 的 token 覆盖、Auth Broker source/capability/nextAction、工具、agent port、Git worktree、GitHub egress、repo/issue/PR 只读探测、可选 push dry-run，以及可选 PR body/create dry-run guard；只报告 `GH_TOKEN`/`GITHUB_TOKEN` 是否存在和来源 key，不打印值。当 auth-broker 配置存在时，`tokenCoverage.source="auth-broker"`、`credentialSource="broker-issued-token"` 且 runner env token 不是成功前提；当仅 env token 存在时，`credentialSource="env-token"` 且 `preflight.authBroker.nextAction="use-env-token-until-auth-broker-live"`；两者都缺失时顶层 `ok=false`、`runnerDisposition=infra-blocked`、`degradedReason=auth-broker-needed`，`tokenCoverage.missing` 同时列出 `GH_TOKEN` 与 `GITHUB_TOKEN`，并输出 `preflight.authBroker.source="broker/auth-broker-needed"`、`capability.source="missing-token"`。该 `auth-missing` 的 scope 是 `scheduler-runner-env`，不能简化成“当前 active runner/dev container 不能创建 PR”；输出必须带 `scopeBoundary` 和 `activeRunnerDevContainer`，要求调用方另跑 `bun scripts/cli.ts gh auth status --repo pikasTech/unidesk` 与 PR dry-run 来确认当前 dev container 能力。`preflight.prCapabilityContract` 是 runner-facing 合同摘要，必须包含目标分支、token/auth 来源、`systemGhBinaryRequiredForWrites=false`、UniDesk REST `bun scripts/cli.ts gh` 可用性、push dry-run/PR create dry-run 的 `writesRemote=false`、expected PR handoff、真实 PR 创建需要 commander 授权和 `gh pr merge` 的 `unsupported-command` 边界；系统 `gh` binary 缺失只进入 `tools.systemGhBinary`，不得误判为 UniDesk REST `gh` CLI 不可用。`--remote` 在 runner-like 环境里不再依赖本地 `unidesk-backend-core`、`unidesk-database`、`baidu-netdisk-backend` 容器存在；这些缺失只作为本地观测证据。若远程控制面可达，则继续走远程控制面结果；若远程控制面不可达，则结构化返回 `failureKind=control-plane-missing` / `degradedReason=remote-control-plane-unreachable`，而不是把本地 `backend-core-container-missing` 当作最终阻塞。`--pr-create-dry-run` 不 POST GitHub，只证明 runner 内 PR body 生成、`scripts/cli.ts gh pr create --dry-run` 和 branch 参数形态可用；服务端创建权限仍以 token/auth broker、repo/issue/PR read、push dry-run 和最终授权后的真实 PR 创建结果为准。 - `codex task ` 通过 Code Queue 私有代理按任务 ID 查询结构化审阅摘要；默认只返回任务身份、执行 Provider、工作目录、attempt 计数、原始 prompt、最终 response、最后错误和渐进披露命令，适合指挥官审阅完成未读任务且避免上下文爆炸。`--detail` 仍是有界详细摘要：默认只返回少量 attempt/tool 行、短 prompt/response/stderr/feedback 预览和 omitted/truncated 元数据；需要完整 prompt/response 文本或更多 tool/attempt 细节时再显式加 `--full`、`--tool-limit N`、`--trace` 或 `codex output`。该摘要读取默认由主 server `code-queue-mgr` 从 PostgreSQL 返回，不依赖 D601 `code-queue-read` Service 可用。 -- `codex tasks [--view supervisor|full] [--queue id] [--status succeeded|running|queued|failed|canceled|judging|retry_wait[,..]] [--unread|--unread-only] [--limit N] [--before-id id]` 通过同一私有代理输出渐进式披露视图。默认 `supervisor` 是低噪声指挥官视图，只返回 `running`、`completedUnread`、`recentCompleted`、`queued` 和 `executionDiagnostics` 的紧凑行；prompt/body 只给短预览和原始字符数，`running`/`completedUnread`/`queued` 默认只返回一个很小的有界页并通过 section `commands.next` 继续分页，`recentCompleted` 默认也限量且不重复 `completedUnread` 未读终态，不嵌入完整 Trace、final response 或全量 overview。`--limit` 在 supervisor 中主要是扫描/分页预算，不是返回几十条肥行的开关；需要更详细当前页任务行时显式使用 `--view full` 或 `--full`，仍受 `--limit` 和 `--before-id` 分页约束。 +- `codex tasks [--view supervisor|full] [--queue id] [--status succeeded|running|queued|failed|canceled|judging|retry_wait[,..]] [--unread|--unread-only] [--limit N] [--before-id id]` 通过同一私有代理输出渐进式披露视图。默认 `supervisor` 是低噪声指挥官视图，只返回 `activeRunning`、`running`、`completedUnread`、`recentCompleted`、`queued`、`activity`、`commanderConcurrency` 和 `executionDiagnostics` 的紧凑行；`activeRunning.count` 是 running+judging 的状态计数，`exact=true` 时来自 queue summary counts，`running.returned` 和 `activeRunning.rowPage.returned` 只是本次返回的紧凑行数。`commanderConcurrency.activeRunnerCount` 是并发策略应使用的 active/running 计数，等于 `activity.effectiveActiveTaskCount`；15 并发策略按 `15 - activeRunnerCount` 计算剩余窗口。`commanderConcurrency.splitBrainDisposition=live-count-as-active` 表示 split-brain 有 fresh heartbeat 证据，应继续监督并计入 active；`interventionRequired=true` 才提示介入。prompt/body 只给短预览和原始字符数，`running`/`completedUnread`/`queued` 默认只返回一个有界小页并通过 section `commands.next` 继续分页，`recentCompleted` 默认限量且不重复 `completedUnread` 未读终态，不嵌入完整 Trace、final response 或全量 overview。`--limit` 在 supervisor 中主要是扫描/分页预算，不是返回几十条肥行的开关；CLI 安全上限是 100，输出会在 `filters.requestedLimit`、`filters.effectiveLimit`、`filters.limitCapped` 和 `disclosure.limitPolicy` 说明显式请求是否被 capped；底层 overview 拉取预算独立显示在 `source.requestedLimit` / `source.effectiveLimit`，所以 `--limit 260` 应显示 requested=260、effective=100、source requested/effective=200，而不是只露出一个含糊的 `limit`。`--unread` 是 `--unread-only` 的别名，必须只保留未读终态；`--status` 必须真实过滤支持的状态，未知参数或未知状态必须结构化失败。需要更详细当前页任务行时显式使用 `--view full` 或 `--full`，仍受 `--limit` 和 `--before-id` 分页约束。 - `codex task --trace --tail|--from-start|--after-seq N|--before-seq N --limit N` 按页拉取 Code Queue 的逻辑 trace；响应会返回 `nextAfterSeq`、`previousBeforeSeq`、`hasMore`、`hasBefore` 和下一页/上一页命令，默认 `--trace` 取最新一页，且仍以分页 trace 为主；需要完整 prompt/最终 response 时加 `--full`，需要详细 task 摘要时加 `--detail`。 - `codex output --tail|--from-start|--after-seq N|--before-seq N --limit N [--full-text]` 按原始 output seq 分页读取底层记录；当 trace 行提示 `commandOmittedLines`、`bodyOmittedLines` 或 `rawSeqs` 时，用该命令按 seq 补取信息。默认是低噪声 raw-output 摘要：即使传入很大的 `--limit`，非 `--full-text` 也会限制返回行数和单条文本预览，并在 `disclosure.limitCapped`、`requestedLimit`、`effectiveLimit` 和 `commands.fullText` 中说明如何继续展开；显式 `--full-text` 才返回该页全文。 - `codex read ` 在人工审阅后标记单个终态任务已读；列表、overview 和 supervisor 视图只返回这个命令字段，不得自动执行，也不得批量清空未读状态。 @@ -56,7 +56,7 @@ CLI 可以从 `master` 快速演进，但必须兼容 `deploy.json` 固定的 CI - `codex steer [prompt|--prompt-file path|--prompt-stdin] [--dry-run] [--no-retry|--retry-attempts N]` 通过 Code Queue 私有代理向正在运行的 task 注入纠偏提示，正式替代底层 `microservice proxy code-queue /api/tasks//steer` 调用。prompt 必须且只能来自位置参数、文件或 stdin 之一；`--dry-run` 只输出 `method`、`path`、`stableProxyPath`、retry policy、prompt 字符数、截断预览和 raw proxy 等价命令，不触碰运行中 session，也不得泄露超长 prompt 全文。真实执行是写入操作，成功只返回 `accepted=true`、task id、prompt 字符数、`promptOmitted=true`、有界 task/queue 确认、attempt summary 和后续查看命令，不回显 prompt 或完整 task state；路径固定为 `/api/microservices/code-queue/proxy/api/tasks//steer`，只能作用于 D601 scheduler 上存在 active steerable turn 的 running task。默认对 `stable-proxy-failed` 和 `backend-core-unreachable` 这类 retryable control-plane failures 做一次有界重试；`--retry-attempts N` 最大为 3，`--retry-delay-ms N` 最大为 5000，`--no-retry` 用于复现单次失败。 - `codex steer` 非 dry-run 失败仍输出 JSON 且退出非零；`.data.diagnostics.reason` 用于 runner 分流，当前包括 `backend-core-unreachable`、`code-queue-microservice-unregistered`、`proxy-unauthorized`、`proxy-404`、`steer-endpoint-404`、`upstream-runtime-rejected`、`stable-proxy-failed` 和 `invalid-proxy-response`。`scope` 区分 `backend-core`、`stable-proxy`、`code-queue-runtime` 或 `unknown`，并带 `status`、`exitCode`、`retryable`、有界 `upstreamBodyPreview`、`attempts`、`retryPolicy` 和推荐交叉验证命令；若任务不在 running/active-turn 状态，通常归类为 `upstream-runtime-rejected`，不得静默成功。`502 provider HTTP tunnel failed`、`provider-gateway-http-fetch`、`The operation was aborted` 或约 30 秒 tunnel wait abort 会归类为 `stable-proxy-failed`，CLI 会先按 retry policy 重试；如果仍失败，`.data.diagnostics.operatorGuidance.rawProxyEquivalentIsFallback=false` 表示 raw proxy 等价命令走同一条 tunnel，只能用于对照诊断，不应被当作更低噪声 fallback。此时 `.data.steer.deliveryUnconfirmed=true`，指挥官应先看 `codex tasks --view supervisor`、`codex task ` 和 `microservice health code-queue`，再从主 server CLI 或显式 SSH transport 重试同一个 `codex steer`。 - `codex interrupt|cancel ` 通过 Code Queue 私有代理请求中断；running/judging 任务会请求 D601 当前 agent run 停止，queued/retry_wait 任务的取消也必须保持与 WebUI 相同代理路径，返回有界 task 摘要和后续查询命令。任何需要接触 active run 的动作仍属于 D601 执行面。 -- Code Queue 多队列 lane 由 `codex` 命令命名空间管理：`queues [--full|--all] [--limit N] [--page N|--offset N]` 列表、`queue create ` 创建、`queue merge --into ` 合并、`move --queue ` 迁移；这些队列管理入口默认由主 server `code-queue-mgr` 直管 PostgreSQL，仍通过稳定 `code-queue` 用户服务代理路径访问。`codex queues` 默认只返回 active/nonempty/unread/runnable queue 摘要、全局 counts 和 execution diagnostics；`--full` 或 `--all` 只切换为完整队列行视图的一页，仍受 `--limit`/`--page`/`--offset` 分页约束，不再默认携带 deprecated full array。summary 和 full 的稳定机读路径都是 `.data.queues.items[]`，全局元数据固定在 `.data.queues.counts`、`.data.queues.executionDiagnostics`、`.data.queues.activeTaskIds` 和 `.data.queues.queuedTaskIds`；需要完整 upstream 时使用输出中的 raw command。旧 full 顶层数组语义已作为 deprecated 兼容信息记录，不再作为 `.data.queues` 主形态。同一个 queue 内部串行执行，不同 queue 之间并行执行。迁移只允许尚未被 scheduler claim 的 `queued`/`retry_wait` 任务，必须满足 `startedAt=null`、`currentAttempt=0` 且没有 active thread/turn；已进入 `running`/`judging` 或已有 claim 标记的任务返回 409，不得被 move/merge 回写成 queued。合并会移动可迁移任务归属并自动删除源 queue 记录，只保留合并后的目标 queue；若 source 或 target queue 存在 active/claimed 任务，合并整体返回 409。合并后的目标 queue 按任务原 `queueEnteredAt`/`createdAt` 时间顺序串行，成功迁移 queued/retry_wait 任务后由 D601 scheduler 轮询推进。 +- Code Queue 多队列 lane 由 `codex` 命令命名空间管理：`queues [--full|--all] [--limit N] [--page N|--offset N]` 列表、`queue create ` 创建、`queue merge --into ` 合并、`move --queue ` 迁移；这些队列管理入口默认由主 server `code-queue-mgr` 直管 PostgreSQL，仍通过稳定 `code-queue` 用户服务代理路径访问。`codex queues` 默认只返回 active/nonempty/unread/runnable queue 摘要、activity、commanderConcurrency、全局 counts 和 execution diagnostics；`--full` 或 `--all` 只切换为完整队列行视图的一页，仍受 `--limit`/`--page`/`--offset` 分页约束，不再默认携带 deprecated full array。summary 和 full 的稳定机读路径都是 `.data.queues.items[]`，全局元数据固定在 `.data.queues.commanderConcurrency`、`.data.queues.activity`、`.data.queues.counts`、`.data.queues.executionDiagnostics`、`.data.queues.activeTaskIds` 和 `.data.queues.queuedTaskIds`；需要完整 upstream 时使用输出中的 raw command。`commanderConcurrency.activeRunnerCount` / `activity.effectiveActiveTaskCount` 是指挥官并发判断的有效活跃数，`schedulerLocalActiveQueueCount`/`activeQueueIds` 只描述本地 scheduler active-run slots，不能覆盖数据库 running 计数或 heartbeat-fresh runner 计数。旧 full 顶层数组语义已作为 deprecated 兼容信息记录，不再作为 `.data.queues` 主形态。同一个 queue 内部串行执行，不同 queue 之间并行执行。迁移只允许尚未被 scheduler claim 的 `queued`/`retry_wait` 任务，必须满足 `startedAt=null`、`currentAttempt=0` 且没有 active thread/turn；已进入 `running`/`judging` 或已有 claim 标记的任务返回 409，不得被 move/merge 回写成 queued。合并会移动可迁移任务归属并自动删除源 queue 记录，只保留合并后的目标 queue；若 source 或 target queue 存在 active/claimed 任务，合并整体返回 409。合并后的目标 queue 按任务原 `queueEnteredAt`/`createdAt` 时间顺序串行，成功迁移 queued/retry_wait 任务后由 D601 scheduler 轮询推进。 - 所有 `codex` 查询和管理命令必须走与 WebUI 相同的 backend-core 私有代理路径 `/api/microservices/code-queue/proxy/...`；CLI 不得为了提交、移动、中断、取消或队列管理直接调用 D601 内部 Service、数据库、pod curl 或 k3sctl scheduler 子服务。若该路径失败，应先修复 CLI/backend/provider tunnel 链路，而不是绕过控制面。 - `job list [--limit N] [--include-command]` 与 `job status [--tail-bytes N]` 查询 `.state/jobs/` 文件系统状态，是异步命令的可观测入口。`job list` 默认只返回最新 50 条摘要；`job status` 默认只返回 stdout/stderr 末尾 12000 字节，并带 `tailPolicy` 与完整日志路径。 - `debug health`、`debug dispatch` 与 `debug task` 走真实内部 core、WebSocket、数据库、provider、系统指标、Docker 状态和 Host SSH 维护桥流程，只用于开发调试，不写入 `TEST.md` 的正式验收步骤。 diff --git a/docs/reference/code-queue-supervision.md b/docs/reference/code-queue-supervision.md index 207a63f8..e5b7a97c 100644 --- a/docs/reference/code-queue-supervision.md +++ b/docs/reference/code-queue-supervision.md @@ -220,15 +220,15 @@ bun scripts/cli.ts codex pr-preflight --remote --issue 常用入口： -- `bun scripts/cli.ts codex tasks --view supervisor --limit N`：查看默认低噪声监督视图，包括 running、完成未读、少量最近完成、queued/runnable、execution diagnostics、任务分类和下一步 drill-down 命令。默认行只保留 task id、队列、短 prompt/body 预览和原始字符数；`--limit` 是扫描/分页预算，不是返回几十条肥行的开关；`show/detail/trace/output/full/read` 放在 section template 中，避免每条任务重复刷屏，需要更多内容再按 taskId 展开。 -- `bun scripts/cli.ts codex queues`：查看低噪声队列计数、active task id、完成未读队列、runnable 队列和控制面诊断；需要完整队列行视图时加 `--full`，但 `--full` 仍默认分页，继续用 `--limit N`、`--page N` 或 `--offset N` 渐进展开。summary 和 full 都使用稳定 JSON path `.data.queues.items[]` 读取队列行，并从 `.data.queues.counts` 与 `.data.queues.executionDiagnostics` 读取全局计数和执行诊断；完整 upstream 只通过输出中的 raw command 显式获取。 +- `bun scripts/cli.ts codex tasks --view supervisor --limit N`：查看默认低噪声监督视图，包括 `activeRunning`、running、完成未读、少量最近完成、queued/runnable、activity、commanderConcurrency、execution diagnostics、任务分类和下一步 drill-down 命令。默认行只保留 task id、队列、短 prompt/body 预览和原始字符数；`--limit` 是扫描/分页预算，不是返回几十条肥行的开关，CLI effective limit 安全上限为 100，输出必须用 `filters.requestedLimit`、`filters.effectiveLimit`、`filters.limitCapped`、`source.requestedLimit` 和 `source.effectiveLimit` 区分用户请求、CLI cap 和 overview 源拉取预算；例如 `--limit 260` 应明确显示 requested=260、effective=100、source=200，`running.returned` 只是低噪声返回行数。`show/detail/trace/output/full/read` 放在 section template 中，避免每条任务重复刷屏，需要更多内容再按 taskId 展开。 +- `bun scripts/cli.ts codex queues`：查看低噪声队列计数、activity、commanderConcurrency、active task id、完成未读队列、runnable 队列和控制面诊断；需要完整队列行视图时加 `--full`，但 `--full` 仍默认分页，继续用 `--limit N`、`--page N` 或 `--offset N` 渐进展开。summary 和 full 都使用稳定 JSON path `.data.queues.items[]` 读取队列行，并从 `.data.queues.commanderConcurrency`、`.data.queues.activity`、`.data.queues.counts` 与 `.data.queues.executionDiagnostics` 读取全局活跃计数和执行诊断；完整 upstream 只通过输出中的 raw command 显式获取。 - `bun scripts/cli.ts codex tasks --unread --limit N`：查看完成未读审阅积压；`--unread` 与 `--unread-only` 等价，不能被静默忽略。 - `bun scripts/cli.ts codex tasks --status succeeded --unread --limit N`：按具体终态过滤监督结果；不支持的 status filter 必须显式失败，不能扩大为未过滤结果。 - `bun scripts/cli.ts codex task `：默认只查看原始 prompt、最终 response、最后错误和 drill-down 命令，这是完成未读任务审阅的第一步。 - 当默认审阅摘要不足时，再逐级使用 `bun scripts/cli.ts codex task --detail`、`bun scripts/cli.ts codex task --trace --limit N` 或 `codex output`。 - 当 master 控制面状态和 D601 scheduler 状态看起来分裂时，使用 `docs/reference/observability.md` 中的活性规则判断。 -默认 supervisor 视图必须保持低噪声。`running`、`completedUnread` 和 `queued` 即使传入较大的 `--limit`，默认也只返回一个很小的有界页，并通过 section `commands.next` 继续分页；`--limit` 保留为扫描/分页预算和 full view 返回预算，不得让一次 supervisor 调用输出几十条肥行。每个任务行只应带 task id 和必要摘要，`show`、`detail`、`trace`、`output`、`full`、`read` 使用 section template 表达，让下一步渐进披露动作明确且不重复；默认不得嵌入完整 queue 列表、完整 final response、raw output 页或完整 trace 行。`recentCompleted` 必须默认限量，且不得重复 `completedUnread` 里的未读终态，避免完成历史把当前 running、阻塞和未读审阅挤出视野；需要完整当前页时显式使用 `--view full`。`executionDiagnostics` 只能展示有界 task-id/reason 预览、总数、截断标记和 omitted counts；需要全量诊断时使用输出中的 raw command。`commands.read` 只是在人工审阅后的建议命令，listing 命令绝不能自动执行。 +默认 supervisor 视图必须保持低噪声。`activeRunning.count` 是指挥官 active running 计数，来源是 queue summary 的 status counts 时 `activeRunning.exact=true`，用于 redline 判断；`activeRunning.rowPage.returned` / `running.returned` 只表示本次返回的紧凑任务行。`activeRunning.redline` 必须写明 `countField`、routine target、burst redline、hard redline、`state` 和 `decisionReady`；只有 `decisionReady=true` 时，才能直接用该 count 做红线/补派判断。`running`、`completedUnread` 和 `queued` 即使传入较大的 `--limit`，默认也只返回一个很小的有界页，并通过 section `commands.next` 继续分页；`--limit` 保留为扫描/分页预算和 full view 返回预算，不得让一次 supervisor 调用输出几十条肥行。每个任务行只应带 task id 和必要摘要，`show`、`detail`、`trace`、`output`、`full`、`read` 使用 section template 表达，让下一步渐进披露动作明确且不重复；默认不得嵌入完整 queue 列表、完整 final response、raw output 页或完整 trace 行。`recentCompleted` 必须默认限量，且不得重复 `completedUnread` 里的未读终态，避免完成历史把当前 running、阻塞和未读审阅挤出视野；需要完整当前页时显式使用 `--view full`。`executionDiagnostics` 只能展示有界 task-id/reason 预览、总数、截断标记和 omitted counts；需要全量诊断时使用输出中的 raw command。`commands.read` 只是在人工审阅后的建议命令，listing 命令绝不能自动执行。 `codex tasks` 中的 `status` 永远是 scheduler/control-plane 原始状态，不因为看到 worker final response 而改写。若某个非终态任务的最后 assistant 文本来自 `finalResponse`，CLI 会额外显示 `statusLabel`、`awaitingTerminalJudge=true`、`closeoutState=awaiting-terminal-or-judge` 或 `awaiting-judge`，并附带 closeout hint。指挥官应把这类行理解为“worker 已经产出最终回复文本，但 Code Queue 还在等待 agent terminal event、scheduler 写回或 judge 结果”；它仍占用 active/running 监督窗口，不能按完成任务 `read` 或验收，直到 `status` 进入 `succeeded`、`failed` 或 `canceled` 并可审阅 judge/terminal 记录。 @@ -238,6 +238,8 @@ bun scripts/cli.ts codex pr-preflight --remote --issue 队列诊断中的 `split-brain` 表示控制面/执行面观测分裂，不自动证明任务已经死亡。只要任务 heartbeat 还在刷新、trace 仍在推进，就不能把它判成服务中断或要求立刻 stop；应把它视为 `splitBrainLive=true` 的 live 任务，继续监督并推进 #20 里的已排任务，而不是 interrupt、替换或把 backend 当成已经挂掉。队列摘要应显示 `effectiveLiveness=live`、`splitBrainLive=true` 和 `recommendedAction=continue-supervision`；compact 输出还应在 `executionDiagnostics.liveness` 中重复这些低噪声字段，并突出 `activeHeartbeatCount`、有界 `heartbeatFreshTaskIds`、`databaseActiveTaskCount` 和 `schedulerActiveRunSlotCount`。当 master/control-plane 的 `schedulerActiveRunSlotCount=0` 但 `heartbeatFreshTaskIds` 非空时，active 数应优先按 scheduler heartbeat 摘要解释为 live，而不是按 master 本地 slot 0 解释为执行停摆。只有 heartbeat expired/missing 或满足 stale-recovery 条件时，才应显示 `effectiveLiveness=at-risk` 并进入恢复判断。 +`codex queues` 和默认 supervisor 视图的 `activity` / `commanderConcurrency` 是指挥官并发治理的主读数。并发决策固定使用 `commanderConcurrency.activeRunnerCount`，它等于 `activity.effectiveActiveTaskCount`；15 并发策略的可补窗口按 `15 - activeRunnerCount` 计算，不能用 `activeQueueIds.length` 或 scheduler-local slot 数替代。`effectiveActiveTaskCount` 表示用于调度判断的有效活跃任务数；`databaseRunningTaskCount` 来自 PostgreSQL 中 `running` 状态计数；`databaseActiveTaskCount` 覆盖 running/judging 等数据库活跃任务；`heartbeatFreshActiveTaskCount` 表示 heartbeat-fresh 的有效 runner 数；`schedulerLocalActiveQueueCount` 和 `schedulerLocalActiveRunSlotCount` 只表示当前控制面本地可见 active run slots。`activeQueueIds` 与 `activeQueueCount` 是 scheduler-local 字段，可能在 `counts.running>0` 且 heartbeat 新鲜时为 0；看到这种组合时应按 `activity.effectiveActiveTaskCount`、`activity.heartbeatFreshActiveTaskCount` 和 `splitBrainLive` 决策，不得把空 `activeQueueIds` 当作零并发或停摆证据。`commanderConcurrency.splitBrainDisposition=live-count-as-active` 表示 split-brain 仍是 live 且应计入 active runner；`interventionRequired=true`、heartbeat risk、stale recovery candidates，或非 `continue-supervision` 的 recommended action 才进入人工介入/恢复判断。 + 单次 `provider is not online`、SSH 超时、proxy 超时或 registry 请求失败只能证明“当前观察路径失败”，不能单独升级为 D601 全局离线、CI/CD 全局阻塞或业务任务不可推进。指挥官和 runner 必须用多信号裁决运行面状态，至少区分以下观察面： - backend-core 节点视图和 provider heartbeat，例如 `debug health` 中的 D601 `status`、`lastHeartbeat` 和 gateway 版本； diff --git a/scripts/code-queue-queues-shape-contract-test.ts b/scripts/code-queue-queues-shape-contract-test.ts index a414c971..484eecd0 100644 --- a/scripts/code-queue-queues-shape-contract-test.ts +++ b/scripts/code-queue-queues-shape-contract-test.ts @@ -16,6 +16,10 @@ function asArray(value: unknown): unknown[] { return value as unknown[]; } +function manyIds(prefix: string, count: number): string[] { + return Array.from({ length: count }, (_, index) => `${prefix}-${String(index + 1).padStart(2, "0")}`); +} + function fixtureResponse(): JsonRecord { return { ok: true, @@ -76,6 +80,66 @@ function fixtureResponse(): JsonRecord { }; } +function splitBrainLiveResponse(): JsonRecord { + const liveTaskIds = manyIds("task-running", 8); + return { + ok: true, + status: 200, + body: { + ok: true, + queue: { + total: 8, + queueCount: 2, + activeQueueIds: [], + activeTaskIds: [], + queuedTaskIds: [], + counts: { running: 8 }, + unreadTerminal: 0, + executionDiagnostics: { + state: "split-brain", + splitBrain: true, + splitBrainLive: true, + effectiveLiveness: "live", + recommendedAction: "continue-supervision", + databaseActiveTaskCount: 8, + databaseActiveTaskIds: liveTaskIds, + schedulerActiveRunSlotCount: 0, + schedulerActiveTaskIds: [], + activeHeartbeatCount: 8, + activeHeartbeatTaskIds: liveTaskIds, + heartbeatFreshTaskIds: liveTaskIds, + heartbeatExpiredTaskIds: [], + heartbeatMissingTaskIds: [], + staleRecoveryCandidateTaskIds: [], + heartbeatRiskTaskIds: [], + }, + }, + queues: [ + { + id: "alpha", + name: "Alpha", + total: 4, + counts: { running: 4 }, + unreadTerminal: 0, + activeTaskId: null, + runnableTaskId: null, + updatedAt: "2026-05-20T00:00:00.000Z", + }, + { + id: "beta", + name: "Beta", + total: 4, + counts: { running: 4 }, + unreadTerminal: 0, + activeTaskId: null, + runnableTaskId: null, + updatedAt: "2026-05-20T00:01:00.000Z", + }, + ], + }, + }; +} + function assertQueuesShape(label: string, result: unknown, expectedView: string): void { const data = asRecord(result); const queues = asRecord(data.queues); @@ -100,15 +164,62 @@ function assertQueuesShape(label: string, result: unknown, expectedView: string) assertCondition(liveness.schedulerActiveRunSlotCount === 0, `${label} liveness summary should keep master active slot zero visible`, liveness); assertCondition(asArray(liveness.heartbeatFreshTaskIds).length === 1, `${label} liveness summary should include bounded fresh heartbeat task ids`, liveness); assertCondition(String(liveness.interpretation ?? "").includes("heartbeat is fresh"), `${label} liveness interpretation should explain slot-zero split-brain`, liveness); + const activity = asRecord(queues.activity); + assertCondition(activity.effectiveActiveTaskCount === 1, `${label} activity should foreground effective active task count`, activity); + assertCondition(activity.databaseRunningTaskCount === 1, `${label} activity should distinguish database running tasks`, activity); + assertCondition(activity.heartbeatFreshActiveTaskCount === 1, `${label} activity should distinguish heartbeat-fresh active runners`, activity); + assertCondition(activity.schedulerLocalActiveQueueCount === 1, `${label} activity should distinguish scheduler-local active queues`, activity); + const commanderConcurrency = asRecord(activity.commanderConcurrency); + assertCondition(commanderConcurrency.activeRunnerCount === 1, `${label} activity should expose commander-facing active runner count`, commanderConcurrency); + assertCondition(commanderConcurrency.activeRunnerCountField === "activity.effectiveActiveTaskCount", `${label} activity should name the active runner count field`, commanderConcurrency); + assertCondition(activity.activeQueueIdsScope === "scheduler-local-active-run-slots", `${label} activity should label activeQueueIds scope`, activity); assertCondition(Array.isArray(queues.activeTaskIds), `${label} activeTaskIds should be present`, queues); assertCondition(Array.isArray(queues.queuedTaskIds), `${label} queuedTaskIds should be present`, queues); } +function assertSplitBrainLiveActivity(label: string, result: unknown): void { + const queues = asRecord(asRecord(result).queues); + const totals = asRecord(queues.totals); + assertCondition(totals.activeQueueCount === 0, `${label} scheduler-local active queue count should be zero`, totals); + assertCondition(totals.schedulerLocalActiveQueueCount === 0, `${label} should preserve zero scheduler-local active queues`, totals); + assertCondition(totals.runnableQueueCount === 0, `${label} runnable queue count should be zero`, totals); + assertCondition(totals.databaseRunningTaskCount === 8, `${label} should foreground DB running task count`, totals); + assertCondition(totals.databaseActiveTaskCount === 8, `${label} should foreground DB active task count`, totals); + assertCondition(totals.heartbeatFreshActiveTaskCount === 8, `${label} should foreground heartbeat-effective active runners`, totals); + assertCondition(totals.commanderActiveRunnerCount === 8, `${label} should mirror commander active count in totals`, totals); + assertCondition(totals.effectiveActiveTaskCount === 8, `${label} should foreground effective active task count`, totals); + assertCondition(asArray(queues.activeQueueIds).length === 0, `${label} activeQueueIds should remain the scheduler-local list`, queues); + assertCondition(queues.activeQueueIdsScope === "scheduler-local-active-run-slots", `${label} activeQueueIds should be scoped`, queues); + assertCondition(String(queues.activeQueueIdsNote ?? "").includes("scheduler-local only"), `${label} activeQueueIds note should explain local-only semantics`, queues); + const activity = asRecord(queues.activity); + assertCondition(activity.effectiveActiveTaskCount === 8, `${label} activity should expose effective active count`, activity); + assertCondition(activity.effectiveActiveSource === "heartbeat-fresh", `${label} activity should choose heartbeat-fresh source`, activity); + assertCondition(activity.databaseRunningTaskCount === 8, `${label} activity should expose DB running count`, activity); + assertCondition(activity.heartbeatFreshActiveTaskCount === 8, `${label} activity should expose heartbeat-effective active count`, activity); + assertCondition(activity.schedulerLocalActiveQueueCount === 0, `${label} activity should expose scheduler-local queue count`, activity); + assertCondition(activity.schedulerLocalActiveRunSlotCount === 0, `${label} activity should expose scheduler-local slot count`, activity); + assertCondition(activity.runnableQueueCount === 0, `${label} activity should expose runnable queue count`, activity); + assertCondition(activity.splitBrainLive === true, `${label} activity should preserve split-brain live`, activity); + assertCondition(activity.splitBrainDisposition === "live-count-as-active", `${label} activity should count live split-brain as active`, activity); + const commanderConcurrency = asRecord(queues.commanderConcurrency); + assertCondition(commanderConcurrency.activeRunnerCount === 8, `${label} should expose commander-facing active runner count`, commanderConcurrency); + assertCondition(commanderConcurrency.activeRunnerCountField === "activity.effectiveActiveTaskCount", `${label} should name the active runner count field`, commanderConcurrency); + assertCondition(commanderConcurrency.splitBrainDisposition === "live-count-as-active", `${label} should classify live split-brain capacity`, commanderConcurrency); + assertCondition(commanderConcurrency.interventionRequired === false, `${label} should not require intervention for fresh split-brain`, commanderConcurrency); + assertCondition(String(commanderConcurrency.decisionRule ?? "").includes("15 - activeRunnerCount"), `${label} should give 15-concurrency arithmetic`, commanderConcurrency); + assertCondition(String(activity.activeQueueIdsNote ?? "").includes("zero local queue ids does not mean zero active runners"), `${label} activity note should prevent zero-active misread`, activity); + assertCondition(String(activity.interpretation ?? "").includes("continue supervision"), `${label} activity interpretation should keep supervision action`, activity); +} + export function runCodeQueueQueuesShapeContract(): JsonRecord { const fetcher = (path: string): JsonRecord => { assertCondition(path === "/api/microservices/code-queue/proxy/api/queues", "codex queues should use stable proxy path", { path }); return fixtureResponse(); }; + const splitBrainFetcher = (path: string): JsonRecord => { + assertCondition(path === "/api/microservices/code-queue/proxy/api/queues", "codex queues should use stable proxy path", { path }); + return splitBrainLiveResponse(); + }; const summary = codexQueuesQueryForTest([], fetcher); assertQueuesShape("summary", summary, "summary"); @@ -140,6 +251,11 @@ export function runCodeQueueQueuesShapeContract(): JsonRecord { assertCondition(offsetFullQueues.hasPrevious === true, "offset full should expose previous page", offsetFullQueues); assertCondition(asRecord(asArray(offsetFullQueues.items)[0]).id === "gamma", "offset full should return second page rows", offsetFullQueues); + const splitSummary = codexQueuesQueryForTest([], splitBrainFetcher); + assertSplitBrainLiveActivity("split-brain summary", splitSummary); + const splitFull = codexQueuesQueryForTest(["--full"], splitBrainFetcher); + assertSplitBrainLiveActivity("split-brain full", splitFull); + return { ok: true, checks: [ @@ -150,6 +266,8 @@ export function runCodeQueueQueuesShapeContract(): JsonRecord { "deprecated full array omitted from default output", "full explicit limit remains bounded and paged", "offset pagination", + "split-brain live activity counts distinguish scheduler-local queues, DB running tasks, and heartbeat-fresh runners", + "commander concurrency block names the active runner count and 15-concurrency rule", ], }; } diff --git a/scripts/code-queue-supervisor-disclosure-contract-test.ts b/scripts/code-queue-supervisor-disclosure-contract-test.ts index e6f87b14..865e414d 100644 --- a/scripts/code-queue-supervisor-disclosure-contract-test.ts +++ b/scripts/code-queue-supervisor-disclosure-contract-test.ts @@ -82,6 +82,14 @@ function fixtureResponse(path: string): JsonRecord { body: { ok: true, queue: { + counts: { + running: 15, + judging: 0, + queued: 4, + retry_wait: 1, + succeeded: 13, + }, + maxActiveQueues: 12, executionDiagnostics: { state: "split-brain", splitBrain: true, @@ -112,7 +120,7 @@ function fixtureResponse(path: string): JsonRecord { pagination: { limit: 200, returned: 15, - total: 15, + total: 33, hasMore: false, nextBeforeId: null, includeActive: true, @@ -152,6 +160,13 @@ function manyRunningFixtureResponse(path: string): JsonRecord { body: { ok: true, queue: { + counts: { + running: 40, + judging: 0, + queued: 7, + retry_wait: 2, + }, + maxActiveQueues: 50, executionDiagnostics: { state: "healthy", databaseActiveTaskCount: 40, @@ -174,11 +189,66 @@ function manyRunningFixtureResponse(path: string): JsonRecord { }; } +function splitBrainLiveSupervisorFixtureResponse(path: string): JsonRecord { + if (path.includes("/summary")) return fixtureResponse(path); + assertCondition(path.startsWith("/api/microservices/code-queue/proxy/api/tasks/overview"), "unexpected path", { path }); + const liveTaskIds = manyIds("split-live", 8); + const tasks = liveTaskIds.map((taskId, index) => task( + taskId, + "running", + `2026-05-22T01:${String(50 - index).padStart(2, "0")}:00.000Z`, + )); + return { + ok: true, + status: 200, + body: { + ok: true, + queue: { + counts: { running: 8 }, + activeQueueIds: [], + activeTaskIds: [], + activeRunSlotCount: 0, + databaseActiveTaskCount: 8, + executionDiagnostics: { + state: "split-brain", + splitBrain: true, + splitBrainLive: true, + effectiveLiveness: "live", + recommendedAction: "continue-supervision", + databaseActiveTaskCount: 8, + databaseActiveTaskIds: liveTaskIds, + schedulerActiveRunSlotCount: 0, + schedulerActiveTaskIds: [], + activeHeartbeatCount: 8, + activeHeartbeatTaskIds: liveTaskIds, + heartbeatFreshTaskIds: liveTaskIds, + heartbeatExpiredTaskIds: [], + heartbeatMissingTaskIds: [], + staleRecoveryCandidateTaskIds: [], + heartbeatRiskTaskIds: [], + }, + }, + pagination: { + limit: 200, + returned: 8, + total: 8, + hasMore: false, + nextBeforeId: null, + includeActive: true, + }, + tasks, + }, + }; +} + export function runCodeQueueSupervisorDisclosureContract(): JsonRecord { const supervisor = codexTasksQueryForTest(["--view", "supervisor", "--limit", "20"], fixtureResponse); + const cappedLimit = codexTasksQueryForTest(["--view", "supervisor", "--limit", "260"], fixtureResponse); const full = codexTasksQueryForTest(["--view", "full", "--limit", "20"], fixtureResponse); + const cappedFull = codexTasksQueryForTest(["--view", "full", "--limit", "260"], fixtureResponse); const runningFiltered = codexTasksQueryForTest(["--status", "running", "--limit", "40"], manyRunningFixtureResponse); const unreadFiltered = codexTasksQueryForTest(["--unread", "--limit", "20"], fixtureResponse); + const splitBrainLive = codexTasksQueryForTest(["--view", "supervisor", "--limit", "20"], splitBrainLiveSupervisorFixtureResponse); const supervisorBody = JSON.stringify(supervisor); const fullBody = JSON.stringify(full); @@ -186,6 +256,7 @@ export function runCodeQueueSupervisorDisclosureContract(): JsonRecord { const unreadFilteredBody = JSON.stringify(unreadFiltered); const supervisorData = asRecord(supervisor); const supervisorView = asRecord(supervisorData.supervisor); + const cappedSupervisorView = asRecord(asRecord(cappedLimit).supervisor); const runningFilteredView = asRecord(asRecord(runningFiltered).supervisor); const runningFilteredSection = asRecord(runningFilteredView.running); const unreadFilteredView = asRecord(asRecord(unreadFiltered).supervisor); @@ -197,12 +268,36 @@ export function runCodeQueueSupervisorDisclosureContract(): JsonRecord { const fullItem = asRecord(asArray(asRecord(asRecord(full).tasks).items)[0]); const completedUnread = asRecord(supervisorView.completedUnread); const fullTasks = asRecord(asRecord(full).tasks); + const cappedFullTasks = asRecord(asRecord(cappedFull).tasks); const diagnostics = asRecord(supervisorView.executionDiagnostics); + const filters = asRecord(supervisorView.filters); + const activeRunning = asRecord(supervisorView.activeRunning); + const activeRunningRowPage = asRecord(activeRunning.rowPage); + const activeRunningRedline = asRecord(activeRunning.redline); + const activeRunningCommands = asRecord(activeRunning.commands); + const counts = asRecord(supervisorView.counts); + const outputBudget = asRecord(asRecord(disclosure.outputBudget)); const listBudget = asRecord(diagnostics.listBudget); const omittedCounts = asRecord(listBudget.omittedCounts); + const splitBrainLiveView = asRecord(asRecord(splitBrainLive).supervisor); + const splitBrainLiveActivity = asRecord(splitBrainLiveView.activity); + const splitBrainLiveConcurrency = asRecord(splitBrainLiveView.commanderConcurrency); + const splitBrainLiveCounts = asRecord(splitBrainLiveView.counts); + const cappedFilters = asRecord(cappedSupervisorView.filters); + const cappedSource = asRecord(cappedSupervisorView.source); + const cappedLimitPolicy = asRecord(asRecord(cappedSupervisorView.disclosure).limitPolicy); + const cappedCommands = asRecord(cappedSupervisorView.commands); + const cappedFullFilters = asRecord(cappedFullTasks.filters); + const cappedFullSource = asRecord(cappedFullTasks.source); assertCondition(supervisorBody.length < fullBody.length * 0.55, "supervisor output should be materially smaller than full output", { supervisorChars: supervisorBody.length, fullChars: fullBody.length }); assertCondition(supervisorBody.length < 45_000, "supervisor output should remain bounded even with large diagnostics", { supervisorChars: supervisorBody.length }); + assertCondition(cappedFilters.requestedLimit === 260 && cappedFilters.effectiveLimit === 100 && cappedFilters.limit === 100 && cappedFilters.limitCapped === true, "supervisor filters should disclose requested and capped effective limit", cappedFilters); + assertCondition(cappedSource.requestedLimit === 200 && cappedSource.effectiveLimit === 200 && cappedSource.limit === 200 && cappedSource.returned === 15, "supervisor source should disclose independent overview fetch limit", cappedSource); + assertCondition(cappedLimitPolicy.requestedLimit === 260 && cappedLimitPolicy.effectiveLimit === 100 && cappedLimitPolicy.sourceFetchLimit === 200 && cappedLimitPolicy.sourceEffectiveLimit === 200, "supervisor disclosure should summarize requested/effective/source limits", cappedLimitPolicy); + assertCondition(String(cappedCommands.refresh ?? "").includes("--limit 260") && String(cappedCommands.byStatus ?? "").includes("--limit 260"), "supervisor follow-up commands should preserve requested limit", cappedCommands); + assertCondition(cappedFullFilters.requestedLimit === 260 && cappedFullFilters.effectiveLimit === 100 && cappedFullFilters.limitCapped === true, "full view filters should disclose capped requested limit", cappedFullFilters); + assertCondition(cappedFullSource.requestedLimit === 200 && cappedFullSource.effectiveLimit === 200, "full view source should disclose independent overview fetch limit", cappedFullSource); assertCondition(recentItems.length === 3, "recentCompleted should be capped below --limit by default", { returned: recentItems.length }); assertCondition(asArray(completedUnread.items).length === 3, "completedUnread should be locally paged and kept separate from recentCompleted", completedUnread); assertCondition(recentItems.every((item) => asRecord(item).unreadTerminal === false), "recentCompleted should not duplicate unread terminal tasks", { recentItems }); @@ -229,26 +324,58 @@ export function runCodeQueueSupervisorDisclosureContract(): JsonRecord { assertCondition(fullItem.status === "running" && String(fullItem.statusLabel ?? "").includes("awaiting terminal/judge"), "full view should keep raw status while exposing derived closeout label", fullItem); assertCondition(fullItem.awaitingTerminalJudge === true && fullItem.closeoutState === "awaiting-terminal-or-judge", "full view should expose awaiting terminal/judge state", fullItem); assertCondition(fullTasks.returned === 15, "full view must not inherit supervisor recentCompleted cap", fullTasks); - const budget = asRecord(disclosure.outputBudget); - assertCondition(budget.recentCompletedReturnedLimit === 3 && budget.sectionReturnedLimit === 3, "supervisor must expose output budget metadata", disclosure); + assertCondition(filters.requestedLimit === 20 && filters.limit === 20 && filters.limitCapped === false, "supervisor filters should disclose requested vs effective limit", filters); + assertCondition(outputBudget.requestedLimit === 20 && outputBudget.effectiveLimit === 20 && outputBudget.sectionReturnedLimit === 3, "supervisor must expose output budget metadata", outputBudget); + assertCondition(activeRunning.count === 15 && activeRunning.exact === true && activeRunning.source === "queue-summary-counts", "activeRunning should expose exact running+judging count from queue summary", activeRunning); + assertCondition(activeRunningRowPage.returned === 1 && activeRunningRowPage.returnedLimit === 3 && activeRunningRowPage.distinction.includes("row page"), "activeRunning row page should distinguish returned rows from active count", activeRunningRowPage); + assertCondition(activeRunningRedline.countField === "supervisor.activeRunning.count" && activeRunningRedline.hardRedline === 15 && activeRunningRedline.state === "at-or-over-hard-redline", "activeRunning redline should name count field and interpretation", activeRunningRedline); + assertCondition(counts.activeRunningCount === 15 && counts.activeRunningExact === true && counts.activeRunningRowsReturned === 1, "supervisor counts should separate active count from returned running rows", counts); + assertCondition(String(activeRunningCommands.running ?? "").includes("--status running,judging"), "activeRunning should provide running drilldown", activeRunningCommands); assertCondition(asArray(runningFilteredSection.items).length === 3, "running status filter should be locally paged below --limit", runningFilteredSection); assertCondition(runningFilteredSection.count === 40 && runningFilteredSection.hasMore === true, "running status filter should preserve count and hasMore", runningFilteredSection); assertCondition(String(asRecord(runningFilteredSection.commands).next ?? "").includes("--before-id task-running-03"), "running status filter should provide next page command", runningFilteredSection); assertCondition(runningFilteredBody.length < 14_000, "running status filter output should remain bounded", { chars: runningFilteredBody.length }); assertCondition(asArray(unreadFilteredSection.items).length <= 3, "unread list should be locally paged below --limit", unreadFilteredSection); assertCondition(unreadFilteredBody.length < 14_000, "unread output should remain bounded", { chars: unreadFilteredBody.length }); + assertCondition(splitBrainLiveCounts.running === 8, "split-brain supervisor should preserve DB running task count", splitBrainLiveCounts); + assertCondition(splitBrainLiveCounts.commanderActiveRunnerCount === 8, "split-brain supervisor should mirror commander active count in counts", splitBrainLiveCounts); + assertCondition(splitBrainLiveCounts.effectiveActive === 8, "split-brain supervisor should foreground effective active count", splitBrainLiveCounts); + assertCondition(splitBrainLiveCounts.databaseRunning === 8, "split-brain supervisor should distinguish database running tasks", splitBrainLiveCounts); + assertCondition(splitBrainLiveCounts.heartbeatFreshActive === 8, "split-brain supervisor should distinguish heartbeat-effective active runners", splitBrainLiveCounts); + assertCondition(splitBrainLiveCounts.schedulerLocalActiveQueues === 0, "split-brain supervisor should preserve zero scheduler-local active queues", splitBrainLiveCounts); + assertCondition(splitBrainLiveActivity.effectiveActiveTaskCount === 8, "split-brain supervisor activity should expose effective active count", splitBrainLiveActivity); + assertCondition(splitBrainLiveActivity.effectiveActiveSource === "heartbeat-fresh", "split-brain supervisor activity should prefer heartbeat-fresh source", splitBrainLiveActivity); + assertCondition(splitBrainLiveActivity.databaseRunningTaskCount === 8, "split-brain supervisor activity should expose DB running count", splitBrainLiveActivity); + assertCondition(splitBrainLiveActivity.heartbeatFreshActiveTaskCount === 8, "split-brain supervisor activity should expose heartbeat-fresh active count", splitBrainLiveActivity); + assertCondition(splitBrainLiveActivity.schedulerLocalActiveQueueCount === 0, "split-brain supervisor activity should expose scheduler-local queue count", splitBrainLiveActivity); + assertCondition(splitBrainLiveActivity.schedulerLocalActiveRunSlotCount === 0, "split-brain supervisor activity should expose scheduler-local slot count", splitBrainLiveActivity); + assertCondition(splitBrainLiveActivity.splitBrainLive === true, "split-brain supervisor activity should mark live split-brain", splitBrainLiveActivity); + assertCondition(splitBrainLiveActivity.splitBrainDisposition === "live-count-as-active", "split-brain supervisor activity should classify live split-brain as active capacity", splitBrainLiveActivity); + assertCondition(splitBrainLiveActivity.commanderConcurrency !== undefined, "split-brain supervisor activity should include commander concurrency guidance", splitBrainLiveActivity); + assertCondition(splitBrainLiveConcurrency.activeRunnerCount === 8, "split-brain supervisor should expose commander-facing active runner count", splitBrainLiveConcurrency); + assertCondition(splitBrainLiveConcurrency.activeRunnerCountField === "activity.effectiveActiveTaskCount", "split-brain supervisor should name the field to use", splitBrainLiveConcurrency); + assertCondition(splitBrainLiveConcurrency.splitBrainDisposition === "live-count-as-active", "split-brain supervisor should explain live split-brain disposition", splitBrainLiveConcurrency); + assertCondition(splitBrainLiveConcurrency.interventionRequired === false, "fresh split-brain supervisor should not require intervention", splitBrainLiveConcurrency); + assertCondition(String(splitBrainLiveConcurrency.decisionRule ?? "").includes("15 - activeRunnerCount"), "split-brain supervisor should give 15-concurrency arithmetic", splitBrainLiveConcurrency); + assertCondition(String(splitBrainLiveActivity.activeQueueIdsNote ?? "").includes("zero local queue ids does not mean zero active runners"), "split-brain supervisor activity should explain activeQueueIds are local-only", splitBrainLiveActivity); + assertCondition(String(splitBrainLiveActivity.interpretation ?? "").includes("continue supervision"), "split-brain supervisor activity should not imply scheduler stoppage", splitBrainLiveActivity); return { ok: true, checks: [ "supervisor output materially smaller than full", "recentCompleted capped", + "explicit --limit cap disclosed", "running/unread locally paged", "split-brain diagnostics capped", + "active running exact count exposed", + "requested/effective/returned limits disclosed", "prompt/body previews bounded", "running finalResponse rows labeled awaiting terminal/judge", "drill-down commands preserved", "full view remains detailed", + "split-brain live supervisor activity distinguishes scheduler-local, database, and heartbeat counts", + "commander concurrency block names the active runner count and 15-concurrency rule", ], supervisorChars: supervisorBody.length, fullChars: fullBody.length, diff --git a/scripts/src/code-queue.ts b/scripts/src/code-queue.ts index abbff91d..ecb62d87 100644 --- a/scripts/src/code-queue.ts +++ b/scripts/src/code-queue.ts @@ -14,6 +14,7 @@ const maxOutputPreviewChars = 1200; const defaultTasksLimit = 20; const defaultQueuesLimit = 8; const maxTasksLimit = 100; +const tasksOverviewSourceFetchLimit = 200; const supervisorSectionReturnedLimit = 3; const supervisorRecentCompletedLimit = 3; const supervisorPromptPreviewChars = 70; @@ -189,6 +190,7 @@ interface CompactTaskMutationResponseOptions { interface CodexTasksOptions { queueId: string | undefined; + requestedLimit: number; limit: number; beforeId: string | undefined; unreadOnly: boolean; @@ -288,6 +290,14 @@ interface CodexTasksDegraded { reason: string; } +interface SupervisorStatusCounts { + counts: Record; + exact: boolean; + source: "queue-summary-counts" | "overview-page-fallback"; + scope: "all-queues" | "queue"; + queueId: string | null; +} + interface CodexQueuesOptions { full: boolean; limit: number; @@ -433,6 +443,10 @@ function asNumber(value: unknown, fallback = 0): number { return typeof value === "number" && Number.isFinite(value) ? value : fallback; } +function finiteNumber(value: unknown): number | null { + return typeof value === "number" && Number.isFinite(value) ? value : null; +} + function asBoolean(value: unknown): boolean { return value === true || value === 1 || value === "1" || value === "true"; } @@ -1304,6 +1318,149 @@ function compactQueueExecutionDiagnostics(value: unknown): Record, + diagnostics: Record | null, + options: { schedulerLocalActiveQueueIds?: string[]; runnableQueueCount?: number | null } = {}, +): Record { + const rawDiagnostics = asRecord(queue.executionDiagnostics) ?? {}; + const compactDiagnostics = diagnostics ?? {}; + const rawLiveness = asRecord(rawDiagnostics.liveness) ?? {}; + const compactLiveness = asRecord(compactDiagnostics.liveness) ?? {}; + const counts = asRecord(queue.counts) ?? {}; + const schedulerLocalActiveQueueIds = options.schedulerLocalActiveQueueIds ?? stringList(queue.activeQueueIds); + const databaseRunningTaskCount = firstFiniteNumber(counts.running) ?? 0; + const databaseJudgingTaskCount = firstFiniteNumber(counts.judging) ?? 0; + const databaseActiveTaskCount = firstFiniteNumber( + rawDiagnostics.databaseActiveTaskCount, + queue.databaseActiveTaskCount, + compactDiagnostics.databaseActiveTaskCount, + stringListCount(rawDiagnostics.databaseActiveTaskIds), + stringListCount(queue.databaseActiveTaskIds), + databaseRunningTaskCount + databaseJudgingTaskCount, + ) ?? 0; + const heartbeatFreshActiveTaskCount = firstFiniteNumber( + rawLiveness.heartbeatFreshTaskCount, + compactLiveness.heartbeatFreshTaskCount, + stringListCount(rawDiagnostics.heartbeatFreshTaskIds), + stringListCount(compactDiagnostics.heartbeatFreshTaskIds), + ) ?? 0; + const activeHeartbeatTaskCount = firstFiniteNumber( + rawDiagnostics.activeHeartbeatCount, + compactDiagnostics.activeHeartbeatCount, + stringListCount(rawDiagnostics.activeHeartbeatTaskIds), + heartbeatFreshActiveTaskCount, + ) ?? 0; + const heartbeatRiskTaskCount = firstFiniteNumber( + rawLiveness.heartbeatRiskTaskCount, + compactLiveness.heartbeatRiskTaskCount, + stringListCount(rawDiagnostics.heartbeatRiskTaskIds), + stringListCount(compactDiagnostics.heartbeatRiskTaskIds), + ) ?? 0; + const staleRecoveryCandidateTaskCount = firstFiniteNumber( + rawLiveness.staleRecoveryCandidateTaskCount, + compactLiveness.staleRecoveryCandidateTaskCount, + stringListCount(rawDiagnostics.staleRecoveryCandidateTaskIds), + stringListCount(compactDiagnostics.staleRecoveryCandidateTaskIds), + ) ?? 0; + const schedulerLocalActiveRunSlotCount = firstFiniteNumber( + rawDiagnostics.schedulerActiveRunSlotCount, + queue.schedulerActiveRunSlotCount, + queue.activeRunSlotCount, + compactDiagnostics.schedulerActiveRunSlotCount, + ); + const runnableQueueCount = firstFiniteNumber(options.runnableQueueCount, queue.runnableQueueCount); + const effectiveActiveTaskCount = Math.max(databaseActiveTaskCount, databaseRunningTaskCount, heartbeatFreshActiveTaskCount); + const executionState = asString(rawDiagnostics.state ?? compactDiagnostics.state); + const effectiveLiveness = asString(rawDiagnostics.effectiveLiveness ?? compactDiagnostics.effectiveLiveness ?? rawLiveness.effectiveLiveness ?? compactLiveness.effectiveLiveness); + const recommendedAction = asString(rawDiagnostics.recommendedAction ?? compactDiagnostics.recommendedAction ?? rawLiveness.recommendedAction ?? compactLiveness.recommendedAction); + const splitBrain = asBoolean(rawDiagnostics.splitBrain) || asBoolean(compactDiagnostics.splitBrain) || executionState === "split-brain"; + const splitBrainLive = splitBrainLiveFromDiagnostics(rawDiagnostics) || splitBrainLiveFromDiagnostics(compactDiagnostics); + const effectiveActiveSource = heartbeatFreshActiveTaskCount > 0 && heartbeatFreshActiveTaskCount >= databaseActiveTaskCount + ? "heartbeat-fresh" + : databaseActiveTaskCount > 0 + ? "database-active" + : schedulerLocalActiveQueueIds.length > 0 || (schedulerLocalActiveRunSlotCount ?? 0) > 0 + ? "scheduler-local" + : "none"; + const activeQueueIdsNote = schedulerLocalActiveQueueIds.length === 0 && effectiveActiveTaskCount > 0 + ? "activeQueueIds are scheduler-local only; zero local queue ids does not mean zero active runners when database or heartbeat counts are nonzero." + : "activeQueueIds are scheduler-local active-run slots; use effectiveActiveTaskCount for commander concurrency decisions."; + const recommendedActionIntervenes = recommendedAction.length > 0 && recommendedAction !== "continue-supervision"; + const interventionRequired = heartbeatRiskTaskCount > 0 || staleRecoveryCandidateTaskCount > 0 || recommendedActionIntervenes || (splitBrain && !splitBrainLive); + const splitBrainDisposition = splitBrainLive + ? "live-count-as-active" + : splitBrain + ? "risk-investigate-before-new-work" + : "not-split-brain"; + const splitBrainReason = splitBrainLive + ? "database active/running tasks have fresh heartbeat evidence and no heartbeat-risk candidates in the compact summary." + : splitBrain + ? "split-brain without fresh-heartbeat live evidence is risky; inspect heartbeat risk, stale recovery candidates, and raw diagnostics before changing capacity." + : "control-plane and execution-plane activity signals are not split-brain."; + const interventionReason = splitBrainLive + ? "fresh heartbeat makes split-brain live; count these runners as active and continue supervision." + : heartbeatRiskTaskCount > 0 + ? "heartbeat risk is present; inspect before adding replacement work or recovering tasks." + : staleRecoveryCandidateTaskCount > 0 + ? "stale recovery candidates are present; follow the recovery runbook before changing concurrency." + : recommendedActionIntervenes + ? `execution diagnostics recommend ${recommendedAction}; intervene before adding work.` + : splitBrain + ? "split-brain is not proven live; inspect raw diagnostics before treating capacity as available." + : "no intervention signal in compact activity summary."; + const commanderConcurrency = { + activeRunnerCount: effectiveActiveTaskCount, + activeRunnerCountField: "activity.effectiveActiveTaskCount", + activeRunnerCountSource: effectiveActiveSource, + decisionRule: "subtract activeRunnerCount from the commander concurrency target; for a 15-runner policy, remaining slots = 15 - activeRunnerCount.", + splitBrainDisposition, + splitBrainReason, + interventionRequired, + interventionReason, + }; + return { + effectiveActiveTaskCount, + effectiveActiveSource, + databaseRunningTaskCount, + databaseActiveTaskCount, + heartbeatFreshActiveTaskCount, + activeHeartbeatTaskCount, + heartbeatRiskTaskCount, + staleRecoveryCandidateTaskCount, + schedulerLocalActiveQueueCount: schedulerLocalActiveQueueIds.length, + schedulerLocalActiveRunSlotCount, + runnableQueueCount, + effectiveLiveness: effectiveLiveness || null, + recommendedAction: recommendedAction || null, + splitBrainLive, + splitBrainDisposition, + splitBrainReason, + commanderConcurrency, + activeQueueIdsScope: "scheduler-local-active-run-slots", + activeQueueIdsNote, + interpretation: splitBrainLive + ? "split-brain live: database-active tasks have fresh scheduler heartbeat; continue supervision." + : heartbeatRiskTaskCount > 0 + ? "heartbeat risk is present; investigate before retry or recovery." + : effectiveActiveTaskCount > 0 + ? "active work is present; compare database, heartbeat, and scheduler-local counts before changing concurrency." + : "no active work observed by database, heartbeat, or scheduler-local signals.", + }; +} + function supervisorExecutionDiagnostics(value: unknown): Record | null { const diagnostics = compactExecutionDiagnostics(value); if (diagnostics === null) return null; @@ -1704,9 +1861,11 @@ function parseTasksOptions(args: string[]): CodexTasksOptions { const unsupported = statusFilter.filter((status) => !supported.has(status)); if (unsupported.length > 0) throw new Error(`unsupported --status value: ${unsupported.join(", ")}; supported: ${Array.from(supported).join(", ")}`); } + const requestedLimit = positiveIntegerOption(args, ["--limit"], defaultTasksLimit); return { queueId: optionValue(args, ["--queue", "--queue-id"]), - limit: positiveIntegerOption(args, ["--limit"], defaultTasksLimit, maxTasksLimit), + requestedLimit, + limit: Math.min(requestedLimit, maxTasksLimit), beforeId: optionValue(args, ["--before-id", "--beforeId"]), unreadOnly: hasFlag(args, "--unread-only") || hasFlag(args, "--unread"), statusFilter, @@ -2229,18 +2388,56 @@ function filterTasksForOptions(tasks: Record[], options: CodexT .filter((task) => !options.unreadOnly || taskUnreadTerminal(task)); } +function baseTaskListOptions(options: CodexTasksOptions): CodexTasksOptions { + return { + queueId: options.queueId, + requestedLimit: options.requestedLimit, + limit: options.limit, + beforeId: options.beforeId, + unreadOnly: options.unreadOnly, + statusFilter: options.statusFilter, + view: options.view, + }; +} + function taskListCommand(options: CodexTasksOptions, extra: string[] = []): string { const args = ["codex", "tasks"]; if (options.view !== "supervisor") args.push("--view", options.view); if (options.queueId !== undefined) args.push("--queue", options.queueId); if (options.unreadOnly) args.push("--unread"); if (options.statusFilter !== null) args.push("--status", options.statusFilter.join(",")); - if (options.limit !== defaultTasksLimit) args.push("--limit", String(options.limit)); + if (options.requestedLimit !== defaultTasksLimit) args.push("--limit", String(options.requestedLimit)); if (options.beforeId !== undefined) args.push("--before-id", options.beforeId); args.push(...extra); return `bun scripts/cli.ts ${args.join(" ")}`; } +function codexTasksLimitDisclosure(options: CodexTasksOptions): Record { + return { + requestedLimit: options.requestedLimit, + effectiveLimit: options.limit, + maxLimit: maxTasksLimit, + limitCapped: options.requestedLimit > options.limit, + }; +} + +function codexTasksSourceDisclosure(pagination: Record): Record { + const effectiveLimit = asNumber(pagination.limit, 0) || null; + return { + endpoint: "/api/tasks/overview", + requestedLimit: tasksOverviewSourceFetchLimit, + fetchLimit: tasksOverviewSourceFetchLimit, + limit: effectiveLimit, + effectiveLimit, + limitCapped: effectiveLimit !== null && effectiveLimit < tasksOverviewSourceFetchLimit, + returned: asNumber(pagination.returned, 0) || null, + total: asNumber(pagination.total, 0) || null, + hasMore: asBoolean(pagination.hasMore), + nextBeforeId: asString(pagination.nextBeforeId) || null, + includeActive: asBoolean(pagination.includeActive), + }; +} + function taskSectionLimit(options: CodexTasksOptions, maxReturned = supervisorSectionReturnedLimit): number { return Math.min(options.limit, maxReturned); } @@ -2259,6 +2456,155 @@ function sectionNextCommand( return sourceNextCommand; } +function countRecordValues(value: unknown): Record { + const record = asRecord(value); + const counts: Record = {}; + if (record === null) return counts; + for (const [status, rawCount] of Object.entries(record)) { + const count = typeof rawCount === "number" ? rawCount : typeof rawCount === "string" ? Number(rawCount) : Number.NaN; + if (Number.isFinite(count) && count >= 0) counts[status] = Math.floor(count); + } + return counts; +} + +function queueSummaryCountsForOptions(taskPage: CodexTasksTaskPage, options: CodexTasksOptions): SupervisorStatusCounts | null { + const queue = taskPage.queue; + if (queue === null) return null; + if (options.queueId !== undefined) { + for (const row of asArray(queue.queues)) { + const record = asRecord(row); + if (record === null || asString(record.id) !== options.queueId) continue; + if (asRecord(record.counts) === null) return null; + return { + counts: countRecordValues(record.counts), + exact: true, + source: "queue-summary-counts", + scope: "queue", + queueId: options.queueId, + }; + } + return null; + } + if (asRecord(queue.counts) === null) return null; + return { + counts: countRecordValues(queue.counts), + exact: true, + source: "queue-summary-counts", + scope: "all-queues", + queueId: null, + }; +} + +function fallbackStatusCountsForOptions(taskPage: CodexTasksTaskPage, options: CodexTasksOptions): SupervisorStatusCounts { + const counts: Record = {}; + for (const task of taskPage.tasks) { + if (options.queueId !== undefined && asString(task.queueId) !== options.queueId) continue; + const status = asString(task.status); + if (status.length === 0) continue; + counts[status] = (counts[status] ?? 0) + 1; + } + return { + counts, + exact: false, + source: "overview-page-fallback", + scope: options.queueId === undefined ? "all-queues" : "queue", + queueId: options.queueId ?? null, + }; +} + +function statusCountsForOptions(taskPage: CodexTasksTaskPage, options: CodexTasksOptions): SupervisorStatusCounts { + return queueSummaryCountsForOptions(taskPage, options) ?? fallbackStatusCountsForOptions(taskPage, options); +} + +function positiveCount(value: unknown): number | null { + const count = asNumber(value, Number.NaN); + return Number.isFinite(count) && count >= 0 ? Math.floor(count) : null; +} + +function runtimeMaxActiveQueues(taskPage: CodexTasksTaskPage): number | null { + const value = positiveCount(taskPage.queue?.maxActiveQueues); + return value !== null && value > 0 ? value : null; +} + +function activeRunningRedlineState(activeCount: number, hardRedline: number, burstRedline: number, routineTarget: number): string { + if (activeCount >= hardRedline) return "at-or-over-hard-redline"; + if (activeCount >= burstRedline) return "burst-redline"; + if (activeCount >= routineTarget) return "above-routine-target"; + return "below-routine-target"; +} + +function supervisorActiveRunningSummary( + taskPage: CodexTasksTaskPage, + options: CodexTasksOptions, + runningSection: CodexTasksSection, + diagnostics: Record | null, +): Record { + const statusCounts = statusCountsForOptions(taskPage, options); + const runningCount = statusCounts.counts.running ?? 0; + const judgingCount = statusCounts.counts.judging ?? 0; + const activeStatusCount = runningCount + judgingCount; + const heartbeatActiveCount = positiveCount(diagnostics?.activeHeartbeatCount); + const schedulerLocalActiveRunSlotCount = positiveCount(diagnostics?.schedulerActiveRunSlotCount); + const effectiveActiveRunnerCount = Math.max(activeStatusCount, heartbeatActiveCount ?? 0, schedulerLocalActiveRunSlotCount ?? 0); + const policy = submitPolicyContract().concurrency; + const routineTarget = policy.gpt55Routine; + const burstRedline = Math.max(policy.gpt55BurstMax, policy.minimaxSimpleMax); + const hardRedline = 15; + const runtimeLimit = runtimeMaxActiveQueues(taskPage); + const redlineState = activeRunningRedlineState(activeStatusCount, hardRedline, burstRedline, routineTarget); + const exactInterpretation = statusCounts.exact + ? "activeRunning.count is exact from queue status counts; row pagination does not change this count" + : "activeRunning.count is page-scoped because queue status counts were absent; do not use it for commander redline decisions without a queue summary/raw cross-check"; + const baseOptions = baseTaskListOptions({ ...options, beforeId: undefined, unreadOnly: false, statusFilter: null }); + const runningOptions: CodexTasksOptions = { ...baseOptions, statusFilter: ["running", "judging"] }; + const queuedOptions: CodexTasksOptions = { ...baseOptions, statusFilter: ["queued", "retry_wait"] }; + return { + count: activeStatusCount, + exact: statusCounts.exact, + source: statusCounts.source, + scope: statusCounts.scope, + queueId: statusCounts.queueId, + statuses: { + running: runningCount, + judging: judgingCount, + }, + effectiveActiveRunnerCount, + effectiveActiveRunnerCountSources: { + activeStatusCount, + heartbeatActiveCount, + schedulerLocalActiveRunSlotCount, + }, + rowPage: { + returned: runningSection.returned, + availableInCurrentOverview: runningSection.count, + returnedLimit: taskSectionLimit(options), + truncated: runningSection.truncated, + hasMore: runningSection.hasMore, + distinction: "returned is the compact row page; count is the active running total used for supervision", + }, + redline: { + countField: "supervisor.activeRunning.count", + routineTarget, + burstRedline, + hardRedline, + runtimeMaxActiveQueues: runtimeLimit, + state: redlineState, + remainingToRoutineTarget: Math.max(0, routineTarget - activeStatusCount), + remainingToBurstRedline: Math.max(0, burstRedline - activeStatusCount), + remainingToHardRedline: Math.max(0, hardRedline - activeStatusCount), + decisionReady: statusCounts.exact, + interpretation: exactInterpretation, + }, + commands: { + running: taskListCommand(runningOptions), + runningFull: taskListCommand({ ...runningOptions, view: "full" }), + queued: taskListCommand(queuedOptions), + queues: "bun scripts/cli.ts codex queues", + nextRunningPage: runningSection.commands.next, + }, + }; +} + function fetchTaskSummaries(taskIds: string[], fetcher: CodexResponseFetcher): { summaries: Map>; degraded: CodexTasksDegraded | null } { const boundedIds = taskIds.slice(0, maxTasksLimit); const summaries = new Map>(); @@ -2299,8 +2645,8 @@ type CodexTasksTaskPage = { function tasksListQueryString(options: CodexTasksOptions): string { return queryString({ - limit: 200, - priorityLimit: 200, + limit: tasksOverviewSourceFetchLimit, + priorityLimit: tasksOverviewSourceFetchLimit, queueId: options.queueId, beforeId: options.beforeId, includeActive: 1, @@ -2355,6 +2701,9 @@ function codexTasksOverviewResult( const queuedSection = buildSupervisorTaskSection(queuedTasks, summaries, sectionLimit, sectionNextCommand(queuedTasks, sectionLimit, options, nextCommand), fullCommand); const pagination = taskPage.pagination; const diagnostics = supervisorExecutionDiagnostics(asRecord(taskPage.queue)?.executionDiagnostics); + const activity = compactCodeQueueActivity(asRecord(taskPage.queue) ?? {}, diagnostics); + const commanderConcurrency = asRecord(activity.commanderConcurrency) ?? {}; + const activeRunning = supervisorActiveRunningSummary(taskPage, options, runningSection, diagnostics); const visibleSupervisorItems = [...runningSection.items, ...unreadSection.items, ...recentSection.items, ...queuedSection.items]; const classifierCounts = visibleSupervisorItems.reduce((counts, item) => { const key = item.kind; @@ -2368,28 +2717,32 @@ function codexTasksOverviewResult( filters: { view: options.view, queueId: options.queueId ?? null, + requestedLimit: options.requestedLimit, limit: options.limit, + ...codexTasksLimitDisclosure(options), unreadOnly: options.unreadOnly, status: options.statusFilter, beforeId: options.beforeId ?? null, }, - source: { - endpoint: "/api/tasks/overview", - queueId: options.queueId ?? null, - limit: asNumber(pagination.limit, 0) || null, - returned: asNumber(pagination.returned, 0) || null, - total: asNumber(pagination.total, 0) || null, - hasMore: asBoolean(pagination.hasMore), - nextBeforeId: asString(pagination.nextBeforeId) || null, - includeActive: asBoolean(pagination.includeActive), - }, + source: { queueId: options.queueId ?? null, ...codexTasksSourceDisclosure(pagination) }, bounded: true, disclosure: { defaultView: "supervisor", policy: "bounded summary rows only; prompt/body are short previews and raw detail requires explicit task/full/trace/output commands", + limitSemantics: "--limit is request/scanning/page budget; supervisor sections still return compact bounded row pages and expose counts separately", + limitPolicy: { + ...codexTasksLimitDisclosure(options), + sourceFetchLimit: tasksOverviewSourceFetchLimit, + sourceEffectiveLimit: asNumber(pagination.limit, 0) || null, + sourceReturned: asNumber(pagination.returned, 0) || null, + note: "codex tasks --limit is capped for CLI output safety; supervisor sections also keep small row caps and expose commands.next for pagination", + }, outputBudget: { promptPreviewChars: supervisorPromptPreviewChars, bodyPreviewChars: supervisorBodyPreviewChars, + requestedLimit: options.requestedLimit, + effectiveLimit: options.limit, + limitCapped: options.limit < options.requestedLimit, sectionReturnedLimit: sectionLimit, recentCompletedReturnedLimit: recentLimit, recentCompletedBodyPreviewChars: supervisorRecentBodyPreviewChars, @@ -2402,17 +2755,28 @@ function codexTasksOverviewResult( counts: { scanned: allTasks.length, running: runningSection.count, + activeRunningCount: activeRunning.count, + activeRunningExact: activeRunning.exact, + activeRunningRowsReturned: runningSection.returned, completedUnread: unreadSection.count, recentCompleted: recentSection.count, queued: queuedSection.count, + commanderActiveRunnerCount: asNumber(commanderConcurrency.activeRunnerCount, 0), + effectiveActive: asNumber(activity.effectiveActiveTaskCount, 0), + databaseRunning: asNumber(activity.databaseRunningTaskCount, 0), + heartbeatFreshActive: asNumber(activity.heartbeatFreshActiveTaskCount, 0), + schedulerLocalActiveQueues: asNumber(activity.schedulerLocalActiveQueueCount, 0), }, classifierCounts, + commanderConcurrency, + activity, + activeRunning, executionDiagnostics: diagnostics, degraded, commands: { refresh: taskListCommand(options), unread: taskListCommand({ ...options, unreadOnly: true }), - byStatus: `bun scripts/cli.ts codex tasks --status ${options.queueId === undefined ? "" : ` --queue ${options.queueId}`}${options.limit === defaultTasksLimit ? "" : ` --limit ${options.limit}`}`, + byStatus: `bun scripts/cli.ts codex tasks --status ${options.queueId === undefined ? "" : ` --queue ${options.queueId}`}${options.requestedLimit === defaultTasksLimit ? "" : ` --limit ${options.requestedLimit}`}`, full: fullCommand, next: nextCommand, }, @@ -2443,18 +2807,14 @@ function codexTasksFullResult( filters: { view: "full", queueId: options.queueId ?? null, + requestedLimit: options.requestedLimit, limit: options.limit, + ...codexTasksLimitDisclosure(options), unreadOnly: options.unreadOnly, status: options.statusFilter, beforeId: options.beforeId ?? null, }, - source: { - endpoint: "/api/tasks/overview", - returned: asNumber(pagination.returned, 0) || null, - total: asNumber(pagination.total, 0) || null, - hasMore: asBoolean(pagination.hasMore), - nextBeforeId, - }, + source: codexTasksSourceDisclosure(pagination), count: filtered.length, returned: visible.length, hasMore: filtered.length > visible.length || sourceHasMore, @@ -2698,6 +3058,8 @@ function compactQueuesResponse(body: Record, options: CodexQueu const selected = options.full ? queues : Array.from(new Map([...activeQueues, ...unreadQueues, ...runnableQueues, ...nonemptyQueues].map((row) => [String(row.id), row])).values()); const visible = selected.slice(options.offset, options.offset + options.limit); const diagnostics = compactQueueExecutionDiagnostics(queue.executionDiagnostics); + const activity = compactCodeQueueActivity(queue, diagnostics, { schedulerLocalActiveQueueIds: activeIds, runnableQueueCount: runnableQueues.length }); + const commanderConcurrency = asRecord(activity.commanderConcurrency) ?? {}; const activeTaskIds = boundedUniqueStringList(queue.activeTaskIds, Math.min(options.limit, maxTasksLimit)); const queuedTaskIds = boundedUniqueStringList(queue.queuedTaskIds, Math.min(options.limit, maxTasksLimit)); const nextOffset = options.offset + visible.length; @@ -2725,11 +3087,22 @@ function compactQueuesResponse(body: Record, options: CodexQueu totalTasks: queue.total ?? null, queueCount: queue.queueCount ?? queues.length, activeQueueCount: activeIds.length, + activeQueueCountScope: "scheduler-local-active-run-slots", + schedulerLocalActiveQueueCount: activeIds.length, + commanderActiveRunnerCount: commanderConcurrency.activeRunnerCount, + effectiveActiveTaskCount: activity.effectiveActiveTaskCount, + databaseRunningTaskCount: activity.databaseRunningTaskCount, + databaseActiveTaskCount: activity.databaseActiveTaskCount, + heartbeatFreshActiveTaskCount: activity.heartbeatFreshActiveTaskCount, nonemptyQueueCount: nonemptyQueues.length, unreadQueueCount: unreadQueues.length, runnableQueueCount: runnableQueues.length, }, + commanderConcurrency, + activity, activeQueueIds: queue.activeQueueIds ?? [], + activeQueueIdsScope: "scheduler-local-active-run-slots", + activeQueueIdsNote: activity.activeQueueIdsNote, activeTaskIds: activeTaskIds.items, activeTaskIdsCount: activeTaskIds.count, activeTaskIdsTruncated: activeTaskIds.truncated, diff --git a/scripts/src/help.ts b/scripts/src/help.ts index 7cbc163e..6c85fb58 100644 --- a/scripts/src/help.ts +++ b/scripts/src/help.ts @@ -56,14 +56,14 @@ export function rootHelp(): unknown { { command: "codex skills-sync --dry-run [--full]", description: "Inspect the controlled runner skills hostPath lifecycle contract without copying files, restarting services, reading secrets, or mutating live runner paths." }, { command: "codex pr-preflight [--remote] [--push-dry-run --push-dry-run-ref refs/heads/probe/] [--pr-create-dry-run --pr-create-dry-run-head ] [--issue N]", description: "Read-only PR admission check against the D601 scheduler/runner token, GitHub egress, repo visibility, skills lifecycle health, optional push dry-run, and PR body/create dry-run guard." }, { command: "codex task [--detail] [--trace --tail|--from-start|--after-seq N|--before-seq N --limit N] [--full]", description: "Fetch the bounded review view by default; --detail is still capped, while --full/trace/output explicitly expand evidence." }, - { command: "codex tasks [--view supervisor|full] [--queue id] [--status status[,status]] [--unread|--unread-only] [--limit N] [--before-id id]", description: "Show the low-noise supervisor view by default: compact task rows, tiny local sections, diagnostics, and drill-down commands; use --view full for detailed rows." }, + { command: "codex tasks [--view supervisor|full] [--queue id] [--status status[,status]] [--unread|--unread-only] [--limit N] [--before-id id]", description: "Show the low-noise supervisor view by default: compact task rows, tiny local sections, activity counts, diagnostics, and drill-down commands; use --view full for detailed rows." }, { command: "codex output [--tail|--from-start|--after-seq N|--before-seq N --limit N] [--full-text]", description: "Fetch paged raw Code Queue output records; default caps large limits/text previews, --full-text explicitly expands one seq window." }, { command: "codex read ", description: "Mark one reviewed terminal task read; never run automatically as part of listing." }, { command: "codex dev-ready", description: "Fetch execution-container readiness, including sanitized skill injection status from /api/dev-ready." }, { command: "codex judge --attempt N [--dry-run] [--include-prompt]", description: "Replay one stored Code Queue attempt through the same judge context builder and MiniMax judge call path used by the live queue worker." }, { command: "codex steer [prompt|--prompt-file path|--prompt-stdin] [--dry-run] [--no-retry|--retry-attempts N]", description: "Push a corrective prompt into a running Code Queue task; retryable tunnel aborts get bounded retry diagnostics, and real success does not echo prompt text." }, { command: "codex interrupt|cancel ", description: "Request interrupt for a running Code Queue task, or cancel a queued/retry_wait task, through the same private proxy." }, - { command: "codex (queues [--full|--all] | queue create | queue merge --into | move --queue )", description: "List low-noise queue summaries by default; full queue rows require --full/--all." }, + { command: "codex (queues [--full|--all] | queue create | queue merge --into | move --queue )", description: "List low-noise queue summaries by default, including effective activity counts that distinguish scheduler-local queues, DB running tasks, and heartbeat-fresh runners; full queue rows require --full/--all." }, { command: "job list [--limit N] [--include-command]", description: "List async jobs from .state/jobs with a bounded default page." }, { command: "job status [--tail-bytes N]", description: "Show job state with bounded stdout/stderr tails." }, { command: "debug health", description: "Probe internal core, nodes, system/Docker status, frontend, provider ingress, and public boundary." }, @@ -284,7 +284,19 @@ function codexHelp(): unknown { defaultPolicy: "low-noise JSON by default; write commands confirm persistence, list/detail/output commands return bounded summaries with drill-down commands", expand: ["codex task --full", "codex task --trace --limit N", "codex output --after-seq N --limit N --full-text", "codex tasks --view full --limit N", "codex skills-sync --dry-run --full"], }, - description: "Operate Code Queue through the stable backend-core private proxy path. Real submit/steer success is a low-noise write confirmation and does not echo prompt text.", + activityFields: { + path: "data.queues.activity and data.supervisor.activity", + effectiveActiveTaskCount: "Commander-facing active count derived from database active/running tasks and heartbeat-fresh runners.", + commanderConcurrency: "Use data.supervisor.commanderConcurrency.activeRunnerCount or data.queues.commanderConcurrency.activeRunnerCount for concurrency decisions; the block states the 15-runner arithmetic and intervention signal.", + schedulerLocalActiveQueueCount: "Only queues currently visible in this scheduler-local active-run slot view; zero does not override DB or heartbeat activity.", + heartbeatFreshActiveTaskCount: "Heartbeat-effective active runner count used to avoid split-brain zero-active mistakes.", + }, + supervisorView: { + activeRunning: "Use data.supervisor.activeRunning.count for exact active running+judging count when exact=true; data.supervisor.running.returned is only the compact row page.", + redline: "data.supervisor.activeRunning.redline names the count field, routine target, burst redline, hard redline, and decisionReady flag.", + limitSemantics: "filters.requestedLimit preserves the user input; filters.limit/effectiveLimit shows the capped query budget; section outputBudget/rowPage show returned-row caps.", + }, + description: "Operate Code Queue through the stable backend-core private proxy path with bounded activity summaries for queue and supervisor views. Real submit/steer success is a low-noise write confirmation and does not echo prompt text.", }; }