From 6c02cb165554629f9b105dbc987990bd4ef0ddd2 Mon Sep 17 00:00:00 2001 From: unidesk-code-queue-runner Date: Sat, 23 May 2026 06:28:13 +0000 Subject: [PATCH] fix(cli): retry codex steer tunnel aborts --- AGENTS.md | 2 +- docs/reference/cli.md | 4 +- docs/reference/code-queue-supervision.md | 2 +- scripts/cli.ts | 3 +- scripts/code-queue-cli-steer-test.ts | 60 ++++++++-- scripts/src/code-queue.ts | 144 +++++++++++++++++++++-- scripts/src/help.ts | 4 +- 7 files changed, 196 insertions(+), 23 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 457acfeb..04389b58 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -52,7 +52,7 @@ UniDesk 是一个以主 server 为统一入口的分布式工作平台；本文 - `bun scripts/cli.ts codex submit [prompt] [--prompt-file path|--prompt-stdin] [--queue ]` / `codex pr-preflight [--remote]`：前者通过 backend-core 私有代理提交 Code Queue 任务，`--dry-run` 会给出 MiniMax/GPT/人工路由建议但不改写 payload，真实提交成功只返回写入确认、task id 和后续查看命令，不回显 prompt；后者只读检查 D601 scheduler/runner 的 GitHub token、egress 和 PR 能力，PR 型派单前必须使用，规则见 `docs/reference/cli.md` 和 `docs/reference/code-queue-supervision.md`。 - `bun scripts/cli.ts codex task `：按 Code Queue 任务 ID 查询默认审阅摘要，只返回原始 prompt、最终 response、最后错误和渐进披露命令；`--detail`、`codex output` 和 supervisor 大 `--limit` 仍默认有界，完整内容需显式 `--full`/`--full-text`/分页展开；`codex queues [--full] [--limit N] [--page N|--offset N]` 默认分页低噪声输出队列摘要，完整 upstream 只通过 raw command 显式获取。 - `bun scripts/cli.ts codex judge --attempt [--dry-run]`：按指定 task/attempt 用与队列 worker 相同的上下文构建和 MiniMax judge 调用路径单步复现完成判定；`--dry-run` 只输出 prompt/payload 诊断。 -- `bun scripts/cli.ts codex steer [prompt|--prompt-file path|--prompt-stdin] [--dry-run]`：通过 Code Queue 私有代理向运行中的 active turn 注入纠偏提示，真实成功只确认写入并返回后续查看命令，不回显 prompt 或完整 task state。 +- `bun scripts/cli.ts codex steer [prompt|--prompt-file path|--prompt-stdin] [--dry-run] [--no-retry|--retry-attempts N]`：通过 Code Queue 私有代理向运行中的 active turn 注入纠偏提示，对 retryable tunnel abort 做有界重试诊断，真实成功只确认写入并返回后续查看命令，不回显 prompt 或完整 task state。 - `bun scripts/cli.ts codex interrupt|cancel `：通过 Code Queue 私有代理中断运行任务或取消 queued/retry_wait 任务，规则见 `docs/reference/cli.md`。 - `bun scripts/cli.ts server stop`：以异步 job 停止固定 Compose 项目中的全部 UniDesk 服务，停止后用 `server status` 复核。 - `bun scripts/cli.ts job list [--limit N]` / `bun scripts/cli.ts job status latest [--tail-bytes N]`：分页查询 `.state/jobs/` 中的异步任务状态，状态输出只读日志尾部并保留完整日志路径，job 机制见 `docs/reference/cli.md`。 diff --git a/docs/reference/cli.md b/docs/reference/cli.md index 0a4ece6d..76b41edb 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -53,8 +53,8 @@ CLI 可以从 `master` 快速演进，但必须兼容 `deploy.json` 固定的 CI - `codex read ` 在人工审阅后标记单个终态任务已读；列表、overview 和 supervisor 视图只返回这个命令字段，不得自动执行，也不得批量清空未读状态。 - `codex dev-ready` 查询 Code Queue `/api/dev-ready` 并返回有界 readiness 摘要，包括工具、Docker、Codex config、SSH 和 `devReady.skills`。`devReady.skills` 只暴露 `UNIDESK_SKILLS_PATH`、是否存在、是否只读、skillCount、`cli-spec` 是否可见和修复建议，不输出宿主 auth/token 文件内容。 - `codex judge --attempt N [--dry-run] [--include-prompt]` 通过 Code Queue 私有代理按指定 attempt 单步复现 judge；这是执行面诊断入口，仍依赖 D601 scheduler/runner 侧的真实 judge builder、MiniMax 调用路径和执行环境。默认会真实调用 MiniMax，`--dry-run` 只返回 prompt/payload 大小、attempt 窗口和重建来源诊断，`--include-prompt` 仅用于本地深度排查。 -- `codex steer [prompt|--prompt-file path|--prompt-stdin] [--dry-run]` 通过 Code Queue 私有代理向正在运行的 task 注入纠偏提示，正式替代底层 `microservice proxy code-queue /api/tasks//steer` 调用。prompt 必须且只能来自位置参数、文件或 stdin 之一；`--dry-run` 只输出 `method`、`path`、`stableProxyPath`、prompt 字符数、截断预览和 raw proxy 等价命令，不触碰运行中 session，也不得泄露超长 prompt 全文。真实执行是写入操作，成功只返回 `accepted=true`、task id、prompt 字符数、`promptOmitted=true`、有界 task/queue 确认和后续查看命令，不回显 prompt 或完整 task state；路径固定为 `/api/microservices/code-queue/proxy/api/tasks//steer`，只能作用于 D601 scheduler 上存在 active steerable turn 的 running task。 -- `codex steer` 非 dry-run 失败仍输出 JSON 且退出非零；`.data.diagnostics.reason` 用于 runner 分流，当前包括 `backend-core-unreachable`、`code-queue-microservice-unregistered`、`proxy-unauthorized`、`proxy-404`、`steer-endpoint-404`、`upstream-runtime-rejected`、`stable-proxy-failed` 和 `invalid-proxy-response`。`scope` 区分 `backend-core`、`stable-proxy`、`code-queue-runtime` 或 `unknown`，并带 `status`、`exitCode`、`retryable`、有界 `upstreamBodyPreview` 和推荐交叉验证命令；若任务不在 running/active-turn 状态，通常归类为 `upstream-runtime-rejected`，不得静默成功。 +- `codex steer [prompt|--prompt-file path|--prompt-stdin] [--dry-run] [--no-retry|--retry-attempts N]` 通过 Code Queue 私有代理向正在运行的 task 注入纠偏提示，正式替代底层 `microservice proxy code-queue /api/tasks//steer` 调用。prompt 必须且只能来自位置参数、文件或 stdin 之一；`--dry-run` 只输出 `method`、`path`、`stableProxyPath`、retry policy、prompt 字符数、截断预览和 raw proxy 等价命令，不触碰运行中 session，也不得泄露超长 prompt 全文。真实执行是写入操作，成功只返回 `accepted=true`、task id、prompt 字符数、`promptOmitted=true`、有界 task/queue 确认、attempt summary 和后续查看命令，不回显 prompt 或完整 task state；路径固定为 `/api/microservices/code-queue/proxy/api/tasks//steer`，只能作用于 D601 scheduler 上存在 active steerable turn 的 running task。默认对 `stable-proxy-failed` 和 `backend-core-unreachable` 这类 retryable control-plane failures 做一次有界重试；`--retry-attempts N` 最大为 3，`--retry-delay-ms N` 最大为 5000，`--no-retry` 用于复现单次失败。 +- `codex steer` 非 dry-run 失败仍输出 JSON 且退出非零；`.data.diagnostics.reason` 用于 runner 分流，当前包括 `backend-core-unreachable`、`code-queue-microservice-unregistered`、`proxy-unauthorized`、`proxy-404`、`steer-endpoint-404`、`upstream-runtime-rejected`、`stable-proxy-failed` 和 `invalid-proxy-response`。`scope` 区分 `backend-core`、`stable-proxy`、`code-queue-runtime` 或 `unknown`，并带 `status`、`exitCode`、`retryable`、有界 `upstreamBodyPreview`、`attempts`、`retryPolicy` 和推荐交叉验证命令；若任务不在 running/active-turn 状态，通常归类为 `upstream-runtime-rejected`，不得静默成功。`502 provider HTTP tunnel failed`、`provider-gateway-http-fetch`、`The operation was aborted` 或约 30 秒 tunnel wait abort 会归类为 `stable-proxy-failed`，CLI 会先按 retry policy 重试；如果仍失败，`.data.diagnostics.operatorGuidance.rawProxyEquivalentIsFallback=false` 表示 raw proxy 等价命令走同一条 tunnel，只能用于对照诊断，不应被当作更低噪声 fallback。此时 `.data.steer.deliveryUnconfirmed=true`，指挥官应先看 `codex tasks --view supervisor`、`codex task ` 和 `microservice health code-queue`，再从主 server CLI 或显式 SSH transport 重试同一个 `codex steer`。 - `codex interrupt|cancel ` 通过 Code Queue 私有代理请求中断；running/judging 任务会请求 D601 当前 agent run 停止，queued/retry_wait 任务的取消也必须保持与 WebUI 相同代理路径，返回有界 task 摘要和后续查询命令。任何需要接触 active run 的动作仍属于 D601 执行面。 - Code Queue 多队列 lane 由 `codex` 命令命名空间管理：`queues [--full|--all] [--limit N] [--page N|--offset N]` 列表、`queue create ` 创建、`queue merge --into ` 合并、`move --queue ` 迁移；这些队列管理入口默认由主 server `code-queue-mgr` 直管 PostgreSQL，仍通过稳定 `code-queue` 用户服务代理路径访问。`codex queues` 默认只返回 active/nonempty/unread/runnable queue 摘要、全局 counts 和 execution diagnostics；`--full` 或 `--all` 只切换为完整队列行视图的一页，仍受 `--limit`/`--page`/`--offset` 分页约束，不再默认携带 deprecated full array。summary 和 full 的稳定机读路径都是 `.data.queues.items[]`，全局元数据固定在 `.data.queues.counts`、`.data.queues.executionDiagnostics`、`.data.queues.activeTaskIds` 和 `.data.queues.queuedTaskIds`；需要完整 upstream 时使用输出中的 raw command。旧 full 顶层数组语义已作为 deprecated 兼容信息记录，不再作为 `.data.queues` 主形态。同一个 queue 内部串行执行，不同 queue 之间并行执行。迁移只允许尚未被 scheduler claim 的 `queued`/`retry_wait` 任务，必须满足 `startedAt=null`、`currentAttempt=0` 且没有 active thread/turn；已进入 `running`/`judging` 或已有 claim 标记的任务返回 409，不得被 move/merge 回写成 queued。合并会移动可迁移任务归属并自动删除源 queue 记录，只保留合并后的目标 queue；若 source 或 target queue 存在 active/claimed 任务，合并整体返回 409。合并后的目标 queue 按任务原 `queueEnteredAt`/`createdAt` 时间顺序串行，成功迁移 queued/retry_wait 任务后由 D601 scheduler 轮询推进。 - 所有 `codex` 查询和管理命令必须走与 WebUI 相同的 backend-core 私有代理路径 `/api/microservices/code-queue/proxy/...`；CLI 不得为了提交、移动、中断、取消或队列管理直接调用 D601 内部 Service、数据库、pod curl 或 k3sctl scheduler 子服务。若该路径失败，应先修复 CLI/backend/provider tunnel 链路，而不是绕过控制面。 diff --git a/docs/reference/code-queue-supervision.md b/docs/reference/code-queue-supervision.md index 23d2d9d7..8a3b4e23 100644 --- a/docs/reference/code-queue-supervision.md +++ b/docs/reference/code-queue-supervision.md @@ -283,7 +283,7 @@ D601 artifact registry 的 systemd unit inactive 不等于 D601 全局离线。 只有存在明确理由时才干预。 - 如果任务还在运行且 trace 或 scheduler heartbeat 新鲜，应引导而不是 interrupt。 -- 对运行中任务的引导应优先使用正式 CLI：`bun scripts/cli.ts codex steer --prompt-file `。该命令和 `codex task/tasks/read` 复用同一个 backend-core stable proxy helper；`--dry-run` 会显示 `method/path/stableProxyPath`、prompt 摘要和 raw proxy 等价命令但不发送。非 dry-run 失败时先看 `.data.diagnostics.reason`：`backend-core-unreachable` 属于本机到 core 的观察路径，`code-queue-microservice-unregistered`/`proxy-unauthorized`/`proxy-404` 属于 stable proxy 配置或权限，`steer-endpoint-404`/`upstream-runtime-rejected` 属于 D601 runtime 或任务状态，`stable-proxy-failed` 多为 provider/k3s/tunnel 链路问题。若正式 CLI 自身不可用，临时通过受控 microservice proxy 调用只能作为现场恢复手段；这类绕行必须记录到指挥简报 issue #24 主体的常驻观察，并创建正式 issue 补齐 CLI 能力，避免长期依赖隐式 API。 +- 对运行中任务的引导应优先使用正式 CLI：`bun scripts/cli.ts codex steer --prompt-file `。该命令和 `codex task/tasks/read` 复用同一个 backend-core stable proxy helper；`--dry-run` 会显示 `method/path/stableProxyPath`、retry policy、prompt 摘要和 raw proxy 等价命令但不发送。非 dry-run 默认对 `stable-proxy-failed` 和 `backend-core-unreachable` 做一次有界重试，失败时先看 `.data.diagnostics.reason`、`.data.diagnostics.attempts` 和 `.data.diagnostics.operatorGuidance`：`backend-core-unreachable` 属于本机到 core 的观察路径，`code-queue-microservice-unregistered`/`proxy-unauthorized`/`proxy-404` 属于 stable proxy 配置或权限，`steer-endpoint-404`/`upstream-runtime-rejected` 属于 D601 runtime 或任务状态，`stable-proxy-failed` 多为 provider/k3s/tunnel 链路问题。`502 provider HTTP tunnel failed`、`The operation was aborted`、约 30 秒 provider tunnel wait abort 仍失败时，`.data.steer.deliveryUnconfirmed=true`；指挥官应先用 `codex tasks --view supervisor --limit 20`、`codex task ` 和 `microservice health code-queue` 交叉确认任务活性，再从主 server CLI 或显式 SSH transport 重试同一个 steer。raw proxy 等价命令走同一条 tunnel，`rawProxyEquivalentIsFallback=false`，只能做诊断对照，不应作为正式 fallback。若正式 CLI 自身不可用，临时通过受控 microservice proxy 调用只能作为现场恢复手段；这类绕行必须记录到指挥简报 issue #24 主体的常驻观察，并创建正式 issue 补齐 CLI 能力，避免长期依赖隐式 API。 - 如果任务进入终态但缺少必要验收证据，应使用聚焦 continuation prompt retry 同一任务。 - 如果任务被可复用基础设施缺陷阻塞，应把该缺陷分配给合适的空闲或低风险队列，让原业务任务等待，或在修复后 retry。 - 如果基础设施缺陷影响 Code Queue 控制面可用性，指挥官可以执行恢复队列所需的最小受控部署，然后验证原任务能继续。 diff --git a/scripts/cli.ts b/scripts/cli.ts index 802db025..126d4bb5 100644 --- a/scripts/cli.ts +++ b/scripts/cli.ts @@ -67,6 +67,7 @@ function displayCommandName(parts: string[]): string { } if (parts[0] === "codex" && parts[1] === "steer" && parts[2] !== undefined) { const shown = ["codex", "steer", parts[2]]; + const shownValueOptions = new Set(["--prompt-file", "--file", "--retry-attempts", "--retry-delay-ms"]); const hasPromptFile = parts.includes("--prompt-file") || parts.includes("--file"); const hasPromptStdin = parts.includes("--prompt-stdin") || parts.includes("--stdin"); const hasHelp = parts.slice(3).some(isHelpToken); @@ -75,7 +76,7 @@ function displayCommandName(parts: string[]): string { const part = parts[index] ?? ""; if (!part.startsWith("--")) continue; shown.push(part); - if (part === "--prompt-file" || part === "--file") { + if (shownValueOptions.has(part)) { shown.push(parts[index + 1] ?? ""); index += 1; } diff --git a/scripts/code-queue-cli-steer-test.ts b/scripts/code-queue-cli-steer-test.ts index 89ee48c2..2b41fa72 100644 --- a/scripts/code-queue-cli-steer-test.ts +++ b/scripts/code-queue-cli-steer-test.ts @@ -147,13 +147,58 @@ export function runCodeQueueCliSteerContract(): JsonRecord { assertCondition(!successJson.includes("send this"), "successful steer must not echo prompt text", success); assertCondition(!successJson.includes("promptPreview"), "successful steer must not include promptPreview", success); - assertReason(codexSteerTaskForTest("direct_task", ["p"], () => ({ ok: false, exitCode: 1, stderrTail: "Cannot connect to the Docker daemon" })), "backend-core-unreachable", null); - assertReason(codexSteerTaskForTest("direct_task", ["p"], () => ({ ok: false, status: 404, body: { ok: false, error: "microservice not found: code-queue" } })), "code-queue-microservice-unregistered", 404); - assertReason(codexSteerTaskForTest("direct_task", ["p"], () => ({ ok: false, status: 401, body: { ok: false, error: "unauthorized" } })), "proxy-unauthorized", 401); - assertReason(codexSteerTaskForTest("direct_task", ["p"], () => ({ ok: false, status: 404, body: { ok: false, error: "proxy route not found", path: "/api/microservices/code-queue/proxy/api/tasks/direct_task/steer" } })), "proxy-404", 404); - assertReason(codexSteerTaskForTest("direct_task", ["p"], () => ({ ok: false, status: 404, body: { ok: false, error: "task not found" } })), "steer-endpoint-404", 404); - assertReason(codexSteerTaskForTest("direct_task", ["p"], () => ({ ok: false, status: 409, body: { ok: false, error: "task does not have an active steerable turn" } })), "upstream-runtime-rejected", 409); - assertReason(codexSteerTaskForTest("direct_task", ["p"], () => ({ ok: false, status: 504, body: { ok: false, error: "provider HTTP tunnel timed out or disconnected", stage: "http-tunnel-wait" } })), "stable-proxy-failed", 504); + assertReason(codexSteerTaskForTest("direct_task", ["p", "--no-retry"], () => ({ ok: false, exitCode: 1, stderrTail: "Cannot connect to the Docker daemon" })), "backend-core-unreachable", null); + assertReason(codexSteerTaskForTest("direct_task", ["p", "--no-retry"], () => ({ ok: false, status: 404, body: { ok: false, error: "microservice not found: code-queue" } })), "code-queue-microservice-unregistered", 404); + assertReason(codexSteerTaskForTest("direct_task", ["p", "--no-retry"], () => ({ ok: false, status: 401, body: { ok: false, error: "unauthorized" } })), "proxy-unauthorized", 401); + assertReason(codexSteerTaskForTest("direct_task", ["p", "--no-retry"], () => ({ ok: false, status: 404, body: { ok: false, error: "proxy route not found", path: "/api/microservices/code-queue/proxy/api/tasks/direct_task/steer" } })), "proxy-404", 404); + assertReason(codexSteerTaskForTest("direct_task", ["p", "--no-retry"], () => ({ ok: false, status: 404, body: { ok: false, error: "task not found" } })), "steer-endpoint-404", 404); + assertReason(codexSteerTaskForTest("direct_task", ["p", "--no-retry"], () => ({ ok: false, status: 409, body: { ok: false, error: "task does not have an active steerable turn" } })), "upstream-runtime-rejected", 409); + assertReason(codexSteerTaskForTest("direct_task", ["p", "--no-retry"], () => ({ ok: false, status: 504, body: { ok: false, error: "provider HTTP tunnel timed out or disconnected", stage: "http-tunnel-wait" } })), "stable-proxy-failed", 504); + + const abortedTunnelBody = { + ok: false, + error: "provider HTTP tunnel failed", + stage: "provider-gateway-http-fetch", + providerId: "D601", + serviceId: "code-queue", + providerError: "The operation was aborted", + retryable: false, + attempts: [{ attempt: 1, ok: false, durationMs: 30003, timeoutMs: 30000, result: { ok: false, error: "The operation was aborted" } }], + }; + let retryCalls = 0; + const retryThenSuccess = codexSteerTaskForTest("direct_task", ["transient correction", "--retry-delay-ms", "0"], () => { + retryCalls += 1; + if (retryCalls === 1) return { ok: false, status: 502, body: abortedTunnelBody }; + return { + ok: true, + status: 200, + body: { + ok: true, + task: { id: "direct_task", status: "running", prompt: "hidden" }, + queue: { activeTaskIds: ["direct_task"] }, + }, + }; + }) as JsonRecord; + assertCondition(retryCalls === 2, "retryable 502 tunnel abort should be retried once by default", { retryCalls, retryThenSuccess }); + assertCondition(nestedRecord(retryThenSuccess, ["steer"]).accepted === true, "retry success should accept steer", retryThenSuccess); + const retrySuccessAttempts = nestedRecord(retryThenSuccess, ["steer"]).attempts; + assertCondition(Array.isArray(retrySuccessAttempts) && retrySuccessAttempts.length === 2, "retry success should expose both attempts", retryThenSuccess); + assertCondition(String(JSON.stringify(retryThenSuccess)).includes("The operation was aborted"), "retry attempts should preserve aborted tunnel evidence", retryThenSuccess); + assertCondition(!String(JSON.stringify(retryThenSuccess)).includes("transient correction"), "retry success must not echo steer prompt", retryThenSuccess); + + let exhaustedCalls = 0; + const exhausted = codexSteerTaskForTest("direct_task", ["final correction", "--retry-attempts", "2", "--retry-delay-ms", "0"], () => { + exhaustedCalls += 1; + return { ok: false, status: 502, body: abortedTunnelBody }; + }) as JsonRecord; + assertCondition(exhaustedCalls === 2, "retryable 502 tunnel abort should honor retry-attempts", { exhaustedCalls, exhausted }); + assertReason(exhausted, "stable-proxy-failed", 502); + const exhaustedDiagnostics = nestedRecord(exhausted, ["diagnostics"]); + const exhaustedAttempts = exhaustedDiagnostics.attempts; + assertCondition(Array.isArray(exhaustedAttempts) && exhaustedAttempts.length === 2, "exhausted retry diagnostics should expose attempts", exhaustedDiagnostics); + assertCondition(String(exhaustedDiagnostics.message || "").includes("The operation was aborted"), "diagnostics should include provider abort message", exhaustedDiagnostics); + assertCondition(nestedRecord(exhaustedDiagnostics, ["operatorGuidance"]).rawProxyEquivalentIsFallback === false, "raw proxy equivalent should be diagnostic, not fallback", exhaustedDiagnostics); + assertCondition(String(nestedRecord(exhausted, ["commands"]).rawProxy || "").includes("microservice proxy code-queue /api/tasks/direct_task/steer"), "failure should still expose raw proxy diagnostic command", exhausted); return { ok: true, @@ -170,6 +215,7 @@ export function runCodeQueueCliSteerContract(): JsonRecord { "non-dry-run uses stable proxy helper", "successful steer confirms write without echoing prompt", "steer failure classification is JSON-consumable", + "retryable tunnel aborts are retried with bounded diagnostics", ], }; } diff --git a/scripts/src/code-queue.ts b/scripts/src/code-queue.ts index 6ed22c89..ea22f23e 100644 --- a/scripts/src/code-queue.ts +++ b/scripts/src/code-queue.ts @@ -33,6 +33,10 @@ const submitLockWaitMs = 60_000; const submitLockPollMs = 250; const submitLockStaleMs = 120_000; const submitThrottleMs = nonNegativeIntegerEnv("UNIDESK_CODEX_SUBMIT_THROTTLE_MS", 2000); +const defaultSteerRetryAttempts = 2; +const maxSteerRetryAttempts = 3; +const defaultSteerRetryDelayMs = 750; +const maxSteerRetryDelayMs = 5_000; interface CodexTaskOptions { detail: boolean; @@ -138,6 +142,8 @@ interface SubmitRoutingRecommendation { interface CodexSteerOptions { prompt: string; dryRun: boolean; + retryAttempts: number; + retryDelayMs: number; } type CodexSteerFailureReason = @@ -164,6 +170,19 @@ interface ClassifiedCodexSteerError { recommendedCrossChecks: string[]; } +interface CodexSteerAttemptSummary { + attempt: number; + ok: boolean; + durationMs: number; + status: number | null; + exitCode: number | null; + reason: CodexSteerFailureReason | null; + scope: ClassifiedCodexSteerError["scope"] | null; + retryable: boolean; + message: string | null; + upstreamBodyPreview?: unknown; +} + interface CompactTaskMutationResponseOptions { fullPrompt?: boolean; } @@ -497,7 +516,10 @@ function responseBody(response: Record | null): Record | null): string { const body = responseBody(response); const bodyError = body?.error; - if (typeof bodyError === "string" && bodyError.length > 0) return bodyError; + const providerError = body?.providerError; + if (typeof bodyError === "string" && bodyError.length > 0) { + return typeof providerError === "string" && providerError.length > 0 ? `${bodyError}: ${providerError}` : bodyError; + } if (typeof response?.codeQueueObservationNote === "string") return response.codeQueueObservationNote; if (typeof response?.stderrTail === "string" && response.stderrTail.length > 0) return response.stderrTail; if (typeof response?.commandStderrTail === "string" && response.commandStderrTail.length > 0) return response.commandStderrTail; @@ -577,6 +599,41 @@ function unwrapSteerResponse(response: unknown, targetPath: string, stableProxyP return { ok: false, diagnostics: classifySteerFailure(response, targetPath, stableProxyPath, rawProxyEquivalent) }; } +function steerSuccessAttempt(attempt: number, durationMs: number, upstream: { ok: unknown; status: unknown }): CodexSteerAttemptSummary { + const status = typeof upstream.status === "number" && Number.isFinite(upstream.status) ? upstream.status : null; + return { + attempt, + ok: true, + durationMs, + status, + exitCode: null, + reason: null, + scope: null, + retryable: false, + message: "steer accepted by Code Queue", + }; +} + +function steerFailureAttempt(attempt: number, durationMs: number, diagnostics: ClassifiedCodexSteerError): CodexSteerAttemptSummary { + return { + attempt, + ok: false, + durationMs, + status: diagnostics.status, + exitCode: diagnostics.exitCode, + reason: diagnostics.reason, + scope: diagnostics.scope, + retryable: diagnostics.retryable, + message: diagnostics.message, + upstreamBodyPreview: diagnostics.upstreamBodyPreview, + }; +} + +function shouldRetrySteerFailure(diagnostics: ClassifiedCodexSteerError, attempt: number, maxAttempts: number): boolean { + if (attempt >= maxAttempts) return false; + return diagnostics.retryable === true && (diagnostics.reason === "stable-proxy-failed" || diagnostics.reason === "backend-core-unreachable"); +} + function positiveIntegerOption(args: string[], names: string[], defaultValue: number, maxValue = Number.MAX_SAFE_INTEGER): number { for (const name of names) { const index = args.indexOf(name); @@ -2705,6 +2762,8 @@ const submitPromptValueOptions = new Set([ const steerPromptValueOptions = new Set([ "--prompt-file", "--file", + "--retry-attempts", + "--retry-delay-ms", ]); function referenceTaskIdsFromOptions(args: string[]): string[] { @@ -2758,12 +2817,17 @@ function parseSubmitOptions(args: string[]): CodexSubmitOptions { function parseSteerOptions(args: string[]): CodexSteerOptions { assertKnownOptions(args, { - flags: ["--prompt-stdin", "--stdin", "--dry-run"], - valueOptions: ["--prompt-file", "--file"], + flags: ["--prompt-stdin", "--stdin", "--dry-run", "--no-retry"], + valueOptions: ["--prompt-file", "--file", "--retry-attempts", "--retry-delay-ms"], }, "codex steer"); + const retryAttempts = hasFlag(args, "--no-retry") + ? 1 + : positiveIntegerOption(args, ["--retry-attempts"], defaultSteerRetryAttempts, maxSteerRetryAttempts); return { prompt: promptFromArgs(args, "codex steer", steerPromptValueOptions), dryRun: hasFlag(args, "--dry-run"), + retryAttempts, + retryDelayMs: nonNegativeIntegerOption(args, ["--retry-delay-ms"], defaultSteerRetryDelayMs, maxSteerRetryDelayMs), }; } @@ -3887,6 +3951,13 @@ function codexSteerTask(taskId: string, args: string[], fetcher: CodexResponseFe path: targetPath, stableProxyPath, method: "POST", + retryPolicy: { + defaultAttempts: defaultSteerRetryAttempts, + maxAttempts: options.retryAttempts, + delayMs: options.retryDelayMs, + retryableReasons: ["stable-proxy-failed", "backend-core-unreachable"], + deliveryConfirmation: "success confirms Code Queue accepted the steer request; repeated stable-proxy failures mean delivery is unconfirmed", + }, bodySummary: { promptChars: options.prompt.length, promptPreviewChars: steerPromptPreviewChars, @@ -3901,22 +3972,70 @@ function codexSteerTask(taskId: string, args: string[], fetcher: CodexResponseFe request, commands: { run: `bun scripts/cli.ts codex steer ${taskId} --prompt-file `, + noRetry: `bun scripts/cli.ts codex steer ${taskId} --prompt-file --no-retry`, rawProxy: rawProxyEquivalent, }, }; } - const response = unwrapSteerResponse(fetcher(stableProxyPath, { method: "POST", body: { prompt: options.prompt } }), targetPath, stableProxyPath, rawProxyEquivalent); - if (!response.ok) { + const attempts: CodexSteerAttemptSummary[] = []; + let failedResponse: { ok: false; diagnostics: ClassifiedCodexSteerError } | null = null; + let successfulResponse: { ok: true; upstream: { ok: unknown; status: unknown }; body: Record } | null = null; + for (let attempt = 1; attempt <= options.retryAttempts; attempt += 1) { + const startedAt = Date.now(); + const response = unwrapSteerResponse(fetcher(stableProxyPath, { method: "POST", body: { prompt: options.prompt } }), targetPath, stableProxyPath, rawProxyEquivalent); + const durationMs = Date.now() - startedAt; + if (response.ok) { + attempts.push(steerSuccessAttempt(attempt, durationMs, response.upstream)); + successfulResponse = response; + break; + } + attempts.push(steerFailureAttempt(attempt, durationMs, response.diagnostics)); + failedResponse = response; + if (!shouldRetrySteerFailure(response.diagnostics, attempt, options.retryAttempts)) break; + sleepSync(options.retryDelayMs); + } + if (successfulResponse === null) { + const diagnostics = failedResponse?.diagnostics ?? classifySteerFailure(null, targetPath, stableProxyPath, rawProxyEquivalent); + const transportDeliveryUnconfirmed = diagnostics.reason === "stable-proxy-failed" || diagnostics.reason === "backend-core-unreachable"; return { ok: false, steer: { accepted: false, prompt, + attempts, + deliveryUnconfirmed: transportDeliveryUnconfirmed, + retryPolicy: { + attempted: attempts.length, + maxAttempts: options.retryAttempts, + retryDelayMs: options.retryDelayMs, + retried: attempts.length > 1, + exhausted: transportDeliveryUnconfirmed && attempts.length >= options.retryAttempts, + note: "codex steer retried only retryable stable-proxy/backend-core failures; raw microservice proxy uses the same tunnel and is diagnostic, not a lower-noise fallback.", + }, }, request, - diagnostics: response.diagnostics, + diagnostics: { + ...diagnostics, + attempts, + retryPolicy: { + attempted: attempts.length, + maxAttempts: options.retryAttempts, + retryDelayMs: options.retryDelayMs, + retried: attempts.length > 1, + exhausted: transportDeliveryUnconfirmed && attempts.length >= options.retryAttempts, + }, + operatorGuidance: { + rawProxyEquivalentIsFallback: false, + deliveryUnconfirmed: transportDeliveryUnconfirmed, + nextStep: transportDeliveryUnconfirmed + ? "Check task liveness and retry codex steer from the main-server CLI or explicit SSH transport; do not treat a raw proxy failure as separate evidence that the task rejected the correction." + : "Inspect the non-retryable reason before resubmitting the correction.", + }, + }, commands: { dryRun: `bun scripts/cli.ts codex steer ${taskId} --prompt-file --dry-run`, + retry: `bun scripts/cli.ts codex steer ${taskId} --prompt-file `, + noRetry: `bun scripts/cli.ts codex steer ${taskId} --prompt-file --no-retry`, rawProxy: rawProxyEquivalent, tasks: "bun scripts/cli.ts codex tasks --view supervisor --limit 20", health: "bun scripts/cli.ts microservice health code-queue", @@ -3925,12 +4044,19 @@ function codexSteerTask(taskId: string, args: string[], fetcher: CodexResponseFe } return { ok: true, - upstream: response.upstream, + upstream: successfulResponse.upstream, steer: { accepted: true, taskId, promptChars: options.prompt.length, promptOmitted: true, + attempts, + retryPolicy: { + attempted: attempts.length, + maxAttempts: options.retryAttempts, + retryDelayMs: options.retryDelayMs, + retried: attempts.length > 1, + }, outputPolicy: { default: "write-confirmation", promptEchoed: false, @@ -3938,8 +4064,8 @@ function codexSteerTask(taskId: string, args: string[], fetcher: CodexResponseFe reason: "codex steer is a write operation; default output confirms delivery and provides drill-down commands without echoing prompt text or full task state.", }, }, - task: compactSubmitTaskConfirmation(response.body.task), - queue: compactSubmitQueueConfirmation(response.body.queue), + task: compactSubmitTaskConfirmation(successfulResponse.body.task), + queue: compactSubmitQueueConfirmation(successfulResponse.body.queue), commands: { show: `bun scripts/cli.ts codex task ${taskId}`, detail: `bun scripts/cli.ts codex task ${taskId} --detail`, diff --git a/scripts/src/help.ts b/scripts/src/help.ts index ce2d0b2e..5b60e84c 100644 --- a/scripts/src/help.ts +++ b/scripts/src/help.ts @@ -60,7 +60,7 @@ export function rootHelp(): unknown { { command: "codex read ", description: "Mark one reviewed terminal task read; never run automatically as part of listing." }, { command: "codex dev-ready", description: "Fetch execution-container readiness, including sanitized skill injection status from /api/dev-ready." }, { command: "codex judge --attempt N [--dry-run] [--include-prompt]", description: "Replay one stored Code Queue attempt through the same judge context builder and MiniMax judge call path used by the live queue worker." }, - { command: "codex steer [prompt|--prompt-file path|--prompt-stdin] [--dry-run]", description: "Push a corrective prompt into a running Code Queue task; real success only confirms the write and does not echo prompt text." }, + { command: "codex steer [prompt|--prompt-file path|--prompt-stdin] [--dry-run] [--no-retry|--retry-attempts N]", description: "Push a corrective prompt into a running Code Queue task; retryable tunnel aborts get bounded retry diagnostics, and real success does not echo prompt text." }, { command: "codex interrupt|cancel ", description: "Request interrupt for a running Code Queue task, or cancel a queued/retry_wait task, through the same private proxy." }, { command: "codex (queues [--full|--all] | queue create | queue merge --into | move --queue )", description: "List low-noise queue summaries by default; full queue rows require --full/--all." }, { command: "job list [--limit N] [--include-command]", description: "List async jobs from .state/jobs with a bounded default page." }, @@ -258,7 +258,7 @@ function codexHelp(): unknown { "bun scripts/cli.ts codex dev-ready", "bun scripts/cli.ts codex pr-preflight [--remote] [--push-dry-run --push-dry-run-ref refs/heads/probe/] [--pr-create-dry-run --pr-create-dry-run-head ] [--issue N]", "bun scripts/cli.ts codex judge --attempt N [--dry-run] [--include-prompt]", - "bun scripts/cli.ts codex steer [prompt|--prompt-file path|--prompt-stdin] [--dry-run]", + "bun scripts/cli.ts codex steer [prompt|--prompt-file path|--prompt-stdin] [--dry-run] [--no-retry|--retry-attempts N]", "bun scripts/cli.ts codex interrupt|cancel ", "bun scripts/cli.ts codex queues [--full|--all] [--limit N] [--page N|--offset N] | queue create | queue merge --into | move --queue ", ],