fix: clarify code queue split-brain liveness
This commit is contained in:
@@ -50,7 +50,7 @@ Use:
|
||||
- `bun scripts/cli.ts codex task <taskId> --trace --limit N` or `codex output` only when the summary is insufficient.
|
||||
- The liveness rules in `docs/reference/observability.md` when master control-plane state and D601 scheduler state appear split.
|
||||
|
||||
`split-brain` in queue diagnostics is a control-plane/execution-plane divergence signal, not automatic evidence that the work is dead. If the task heartbeats are fresh and the trace is still advancing, treat the task as live and keep supervising it rather than interrupting or replacing it.
|
||||
`split-brain` in queue diagnostics is a control-plane/execution-plane divergence signal, not automatic evidence that the work is dead. If the task heartbeats are fresh and the trace is still advancing, treat the task as live and keep supervising it rather than interrupting or replacing it. The queue summary should expose this as `effectiveLiveness=live`, `splitBrainLive=true`, and `recommendedAction=continue-supervision`; expired, missing, or stale-recovery heartbeat evidence should instead surface `effectiveLiveness=at-risk`.
|
||||
|
||||
Long-running tasks with fresh trace or heartbeat evidence should normally be left alone. Polling every few minutes is preferred over repeated interrupt/retry cycles.
|
||||
|
||||
|
||||
@@ -50,6 +50,6 @@ Code Queue task 明明产出最终回复却反复 `retry_wait` 时,应优先
|
||||
|
||||
### Code Queue Liveness
|
||||
|
||||
Code Queue 的“任务是否卡死”不能由单一控制面字段判断。排障必须同时看 PostgreSQL 中的 `running`/`judging` 任务、D601 scheduler 本地 active run/active slot/active queue、scheduler-owned heartbeat、Trace/OA 持久化进度和 OA publisher pending/lastError。master `code-queue-mgr` 的 `postgres-control-plane` 视图只证明数据库行存在;当它显示 `activeRunSlotCount=0` 但 D601 heartbeat 仍新鲜时,正确结论是 control-plane/execution-plane 分裂,diagnostics 应显示 `split-brain` 或 `degraded`,不能宣称任务未执行或卡死。
|
||||
Code Queue 的“任务是否卡死”不能由单一控制面字段判断。排障必须同时看 PostgreSQL 中的 `running`/`judging` 任务、D601 scheduler 本地 active run/active slot/active queue、scheduler-owned heartbeat、Trace/OA 持久化进度和 OA publisher pending/lastError。master `code-queue-mgr` 的 `postgres-control-plane` 视图只证明数据库行存在;当它显示 `activeRunSlotCount=0` 但 D601 heartbeat 仍新鲜时,正确结论是 control-plane/execution-plane 分裂,diagnostics 应显示 `split-brain` 或 `degraded`,不能宣称任务未执行或卡死。诊断输出中的 `effectiveLiveness=live`、`splitBrainLive=true` 和 `recommendedAction=continue-supervision` 表示这是 heartbeat 新鲜的观测分裂,应继续监督;`effectiveLiveness=at-risk` 或 `recommendedAction=investigate-heartbeat-risk` 表示存在 expired/missing/stale heartbeat 风险,需要优先人工确认。
|
||||
|
||||
Trace/OA 长时间没有新 seq 但 scheduler heartbeat 正常时,应归类为 trace gap 或 publisher degraded,不得自动 retry。只有 scheduler 本地没有 active run,且对应 owner heartbeat 已过期时,才允许进入 stale recovery candidate;缺失 heartbeat 只能触发 degraded 诊断和人工确认。任何恢复入口都必须由 scheduler 执行,使用条件更新和审计事件区分 user interrupt、admin stale recovery 与 service restart recovery;禁止直接修改 production PostgreSQL 任务状态来“修复” active run。
|
||||
|
||||
Reference in New Issue
Block a user