feat: add Sub2API runtime configuration CRUD

.agents/skills/unidesk-sub2api/references/codex-pool.md

@@ -1,11 +1,14 @@
 ## Codex Pool
 
-- `codex-pool plan|sync|validate|runtime-state` 同时覆盖 k3s target 和 PK01 host-Docker target：
+- `codex-pool plan|sync|validate|runtime` 同时覆盖 k3s target 和 PK01 host-Docker target：
   - PK01 host-Docker 的 `sync --confirm` 通过 Sub2API admin API 对齐 group、YAML-managed accounts、统一消费 API key、capacity/loadFactor、WebSocket v2 标记和内置 `temp_unschedulable` 规则；
   - 统一消费 key 写入 YAML 声明的 `targets[PK01].hostDocker.envPath`；
   - PK01 路径不创建 k8s Secret、不部署 sentinel 资源，也不触发 `sub2api apply`、Docker compose、Caddy reload 或容器重启；
-  - `runtime-state` 只读列出 pool group 内全部账号，包括手工账号，并披露临时不可调度规则、当前冻结状态、`pool_mode`、代理绑定和相对 YAML 基线的差异；
-  - `runtime-state` 不输出 API key 或 credentials 正文，默认返回紧凑 JSON，使用 `--full` 或 `--raw` 下钻；
+  - `runtime list|get` 查询 pool group 内全部账号或指定账号的实际状态；
+  - `runtime errors` 按账号汇总 Sub2API 日志中的真实 `account_upstream_error` 状态码，并用 `openai.forward_failed` 提供脱敏错误摘要；
+  - `runtime apply|delete` 只修改显式 `--account` 指定账号的 `temp_unschedulable` 配置；
+  - 写操作不带 `--confirm` 时只返回 dry-run，带 `--confirm` 才通过 Sub2API admin API 写入；
+  - runtime 命令不输出 API key 或完整 credentials，默认返回紧凑 JSON，使用 `get`、`--full` 或 `--raw` 下钻；
   - `sentinel-report`、`sentinel-probe`、`sentinel-image` 和部分 `trace` 能力仍以 k3s target 为主，需要这些能力时显式选择对应 k3s target。
 
 k3s 账号池操作示例：
@@ -14,7 +17,12 @@ k3s 账号池操作示例：
 bun scripts/cli.ts platform-infra sub2api codex-pool plan --target D601
 bun scripts/cli.ts platform-infra sub2api codex-pool sync --target D601 --confirm
 bun scripts/cli.ts platform-infra sub2api codex-pool validate --target D601
-bun scripts/cli.ts platform-infra sub2api codex-pool runtime-state --target PK01
+bun scripts/cli.ts platform-infra sub2api codex-pool runtime list --target PK01
+bun scripts/cli.ts platform-infra sub2api codex-pool runtime get --target PK01 --account <name-or-id>
+bun scripts/cli.ts platform-infra sub2api codex-pool runtime errors --target PK01 --since 24h
+bun scripts/cli.ts platform-infra sub2api codex-pool runtime apply --target PK01 --account <name-or-id> --template <template-id>
+bun scripts/cli.ts platform-infra sub2api codex-pool runtime apply --target PK01 --account <name-or-id> --template <template-id> --confirm
+bun scripts/cli.ts platform-infra sub2api codex-pool runtime delete --target PK01 --account <name-or-id> --kind temp-unschedulable
 bun scripts/cli.ts platform-infra sub2api codex-pool trace --request-id <requestId>
 bun scripts/cli.ts platform-infra sub2api codex-pool sentinel-image status --target D601
 bun scripts/cli.ts platform-infra sub2api codex-pool sentinel-image build --target D601 --confirm
@@ -29,8 +37,12 @@ bun scripts/cli.ts platform-infra sub2api codex-pool cleanup-probes --target D60
 - `pool.apiKeySecretName` / `pool.apiKeySecretKey`: 统一消费 API key 的 key 名。k3s target 写入对应 namespace 下的 k3s Secret；PK01 host-Docker target 写入 `config/platform-infra/sub2api.yaml` 中 `targets[PK01].hostDocker.envPath` 声明的 env 文件。
 - `pool.minOwnerBalanceUsd`: pool key owner 最低余额，sync/validate 会补齐。
 - `pool.minOwnerConcurrency`: 可选统一消费 API key owner 最低并发；省略时 CLI 自动使用所有已解析账号 capacity 的总和，sync/validate 会补齐。显式 YAML 值只作为 override，仍必须不小于账号 capacity 总和；未显式写 `profiles.entries[].capacity` 的账号会使用 `pool.defaultAccountCapacity` 参与求和，不要用提高某个 provider capacity 来掩盖用户并发层 WS 1013。
-- `pool.defaultTempUnschedulable`: Sub2API 内置请求路径临时不可调度开关和 YAML 规则列表。当前要求是按 YAML 开启通用规则；sync 把 `temp_unschedulable_enabled` / `temp_unschedulable_rules` 渲染到 managed accounts，让匹配的 400/5xx/超时/模型路由/加密内容错误短暂冷却当前账号并触发同组 failover。
-- `pool.defaultTempUnschedulable` 与外部 `sentinel.*` 分开配置、互不驱动。内置规则负责 near-real-time request-path cooling/failover；哨兵负责 marker health、账号级隔离/恢复和 probe 退避。
+- `runtime.templates`: 可复用 runtime 规则模板；模板存在不表示账号已绑定，也不自动接管手工账号。
+- `pool.defaultTempUnschedulableTemplate`: YAML-managed 账号默认引用的临时不可调度模板。`sync --confirm` 只把模板渲染到 YAML-managed 账号。
+- `profiles.entries[].tempUnschedulableTemplate`: 可选账号级模板引用，只用于明确偏离默认模板。
+- YAML 未声明的账号标记为 `runtime-manual`；它可以显示 `matchingTemplate`，但模板差异不是 drift，也不能由 `sync` 自动改写。
+- 手工账号需要采用模板时，必须通过 `runtime apply --account ... --template ... --confirm` 显式写入；需要删除该类配置时使用 `runtime delete --account ... --kind temp-unschedulable --confirm`。
+- runtime 模板与外部 `sentinel.*` 分开配置、互不驱动。内置规则负责请求路径即时冷却和 failover；哨兵负责 marker health、账号级隔离、恢复和 probe 退避。
 - 外部 sentinel 的写入面只允许通过 Sub2API admin `schedulable` 接口冻结/恢复账号；不能写入、清理或间接清理 `temp_unschedulable_until` / `temp_unschedulable_reason`、rate-limit、overload、model-rate-limit 等 Sub2API 请求路径 runtime 状态，也不能调用 `recover-state` 作为恢复动作。看到 UI 里的“触发时间/解除时间/规则序号/匹配关键词”临时不可调度状态时，默认先归因到 Sub2API 内置 request-path temp-unschedulable，而不是 sentinel。
 - YAML 只选择和配置 Codex 上游，不声明 `schedulable` 长期字段；`codex-pool sync --confirm` 不负责把既有账号恢复为 `schedulable=true`。既有账号的 `schedulable=false` 必须由哨兵先同步 Sub2API runtime 状态，再在 marker probe 命中后恢复。
 - `profiles.entries`: 从 master `~/.codex/` 选择上游 profile 并映射到 Sub2API account。
@@ -41,7 +53,6 @@ bun scripts/cli.ts platform-infra sub2api codex-pool cleanup-probes --target D60
 - `profiles.entries[].sentinelProtect`: 可选账号级哨兵保护覆盖；只用于明确偏离 pool 默认策略。它只影响哨兵冻结判定和 `sentinel-report` 可见性，不改变 Sub2API account priority/capacity/loadFactor。
 - 除非用户明确要求修改配置，不要仅凭推断改账号 membership、priority、capacity、loadFactor、WebSocket mode 或其他调度策略；先保留 YAML，完成 provenance/runtime evidence 溯源，并把结论写回相关 issue 或 runbook 后再提出变更。
 - Sub2API 是 UniDesk 可读源码和可观测运行面的受控组件；排查 Sub2API 调度、failover、错误传播、临时不可调度或 account selection 时，默认先读当前 Sub2API 源码实现，再用真实 request id、Sub2API 日志和原入口流量验证。不要用 mock upstream、临时 probe account 或测试桩作为默认结论来源；这类探针最多是显式 debug 辅助，不能替代源码链路和真实运行证据。
-- `profiles.entries[].tempUnschedulable`: 可选 per-account Sub2API 内置临时不可调度覆盖；只用于明确偏离 pool 默认规则，不用它给某个账号特殊优先级或临时绕过通用 failover。
 - `profiles.entries[].openaiResponsesWebSocketsV2Mode`: 需要 Responses WebSocket v2 的上游才设置，值为 `off`、`ctx_pool` 或 `passthrough`。
 - `profiles.entries[].upstreamUserAgent`: 少数要求 Codex CLI User-Agent 的上游才设置，不能含换行。
 - `manualAccounts.protected`: 已在 Sub2API 手动创建/维护、且必须排除在 UniDesk-managed Codex pool credentials 和 sentinel 控制之外的账号。默认不得改 credentials/status/schedulable/priority/capacity/loadFactor；只有显式声明 `proxyBinding` 时，`sync --confirm` 才允许把该账号的 `proxy_id` 对齐到 YAML 目标的 egress proxy；只有显式声明 `groupBinding.source: pool-group` 时，才允许把该账号加入统一消费 API key 使用的 pool group。`targetIds` 可选；省略表示所有 target 都保护该账号，设置后只在匹配 target 上纳入 proxy/group 窄同步和 sentinel-probe 拒绝列表，避免 PK01-only 手动账号漂移卡住 JD01 pool。

config/platform-infra/sub2api-codex-pool.yaml

@@ -8,6 +8,27 @@ metadata:
     - 339
     - 340
 
+runtime:
+  templates:
+    - id: codex-upstream-failover
+      kind: temp-unschedulable
+      description: Reusable Sub2API request-path cooling and failover policy for Codex-compatible API-key accounts.
+      spec:
+        enabled: true
+        rules:
+          - statusCode: 502
+            keywords: [upstream service temporarily unavailable, upstream request failed, context window, stream usage incomplete, missing terminal event]
+            durationMinutes: 1
+            description: Observed 502 upstream availability, context-window, and incomplete-stream failures briefly cool down the selected account.
+          - statusCode: 503
+            keywords: [upstream service temporarily unavailable, upstream request failed, temporarily unavailable]
+            durationMinutes: 1
+            description: Observed 503 temporary upstream availability failures briefly cool down the selected account.
+          - statusCode: 524
+            keywords: [upstream request failed, missing_required_parameter, encrypted_content]
+            durationMinutes: 1
+            description: Observed 524 upstream request and encrypted-content failures briefly cool down the selected account.
+
 pool:
   groupName: unidesk-codex-pool
   groupDescription: UniDesk-managed Codex API-key pool for YAML-selected Sub2API clients.
@@ -25,53 +46,7 @@ pool:
     initialRetryDelaySeconds: 2
     maxRetryDelaySeconds: 60
     backoffMultiplier: 2
-  defaultTempUnschedulable:
-    enabled: true
-    rules:
-      - statusCode: 400
-        keywords: [invalid_encrypted_content, encrypted content, could not be verified, could not be decrypted, bad_response_status_code, model_not_found, no available channel for model, unsupported, not supported, not support, 暂不支持, 可用模型]
-        durationMinutes: 1
-        description: Stable upstream 400 model-routing or Responses encrypted-content compatibility failures should use another account.
-      - statusCode: 401
-        keywords: [unauthorized, invalid api key, invalid_api_key, authentication, recovered upstream error]
-        durationMinutes: 1
-        description: Credential/auth failures should briefly cool down this account and use another account.
-      - statusCode: 403
-        keywords: [forbidden, access denied, quota, billing, capacity, weekly limit, less than 10% of your weekly limit left, run /status for a breakdown, recovered upstream error]
-        durationMinutes: 1
-        description: Permission, quota, or account-state failures should briefly cool down this account and use another account.
-      - statusCode: 429
-        keywords: [capacity, rate limit, rate_limit, quota, weekly limit, less than 10% of your weekly limit left, run /status for a breakdown, too many requests, overloaded, resource_exhausted, recovered upstream error]
-        durationMinutes: 1
-        description: Capacity and rate-limit responses are often temporary; briefly cool down this account and use another account.
-      - statusCode: 500
-        keywords: [capacity, overloaded, temporarily unavailable, temporary, upstream, recovered upstream error]
-        durationMinutes: 1
-        description: Transient upstream server failures should briefly cool down this account and prefer another account.
-      - statusCode: 502
-        keywords: [capacity, overloaded, temporarily unavailable, temporary, upstream, bad gateway, upstream request failed, unknown error, context deadline exceeded, context canceled, websocket dial, handshake response, recovered upstream error]
-        durationMinutes: 1
-        description: Gateway upstream failures, including recovered upstream error wrappers, should briefly cool down this account.
-      - statusCode: 413
-        keywords: [openai_error, payload too large, request too large, context length, context window, maximum context]
-        durationMinutes: 1
-        description: Large-context upstream failures should cool down the selected account so a larger-context channel can handle the request.
-      - statusCode: 503
-        keywords: [capacity, overloaded, temporarily unavailable, temporary, upstream, recovered upstream error, model_not_found, no available channel for model]
-        durationMinutes: 1
-        description: Service unavailable and upstream model-routing failures should briefly cool down this account.
-      - statusCode: 504
-        keywords: [gateway timeout, timeout, upstream, upstream request failed, unknown error, context deadline exceeded, context canceled, recovered upstream error]
-        durationMinutes: 1
-        description: Gateway timeout responses should cool down the selected account so another account can handle the next request.
-      - statusCode: 524
-        keywords: [timeout, a timeout occurred, cloudflare, gateway timeout, upstream, upstream request failed, unknown error, context deadline exceeded, context canceled, recovered upstream error]
-        durationMinutes: 1
-        description: Cloudflare 524 timeout responses should cool down the selected account so another account can handle the next request.
-      - statusCode: 529
-        keywords: [capacity, overloaded, temporarily unavailable, temporary, recovered upstream error]
-        durationMinutes: 1
-        description: Provider overloaded responses should briefly cool down this account and use another account.
+  defaultTempUnschedulableTemplate: codex-upstream-failover
 profiles:
   entries:
     - profile: yjxm1221

docs/reference/platform-infra.md

@@ -153,8 +153,12 @@
 - `pool.groupName` names the Sub2API group that represents the pool.
 - `pool.apiKeySecretName` and `pool.apiKeySecretKey` name the single consumer API key. k3s targets store it in a k3s Secret; PK01 host-Docker stores the same key in the YAML-declared `hostDocker.envPath`.
 - `pool.minOwnerConcurrency` is optional; when omitted, the CLI automatically uses the sum of all resolved account capacities as the minimum concurrency for the Sub2API user that owns the unified consumer API key. A YAML value is only an explicit override and must still be at least that capacity sum, so the shared key does not fail requests or WS sessions at the user-concurrency layer. "Resolved" means each account's explicit `profiles.entries[].capacity` or, when omitted, `pool.defaultAccountCapacity`. Do not compensate for owner-concurrency 1013 errors by pinning capacity to one provider.
-- `pool.defaultTempUnschedulable` is the Sub2API built-in request-path temporary-unschedulable switch plus its YAML rule list. When enabled, `codex-pool sync --confirm` renders `temp_unschedulable_enabled` and `temp_unschedulable_rules` into every managed account unless an account-level override says otherwise. This is the generic same-request recovery path for selected-account upstream failures: a matching upstream error briefly cools the selected account so Sub2API's existing failover loop can select another account in the same group.
-- The built-in temporary-unschedulable configuration and external `sentinel.*` configuration are separate control surfaces. `pool.defaultTempUnschedulable` handles near-real-time request-path cooling and failover; `sentinel.*` handles account-level marker health, quarantine, restore, and probe cadence. Changing one surface must not silently rewrite the other surface's cadence, marker semantics, quarantine state, or rule list.
+- `runtime.templates` 保存可复用的 Sub2API runtime 规则模板。模板本身不是账号绑定，不会自动接管未在 YAML 中声明的手工账号。
+- `pool.defaultTempUnschedulableTemplate` 是 YAML-managed 账号的默认临时不可调度模板引用；`profiles.entries[].tempUnschedulableTemplate` 可以为明确声明的账号选择其他模板。`codex-pool sync --confirm` 只向这些 managed 账号渲染 `temp_unschedulable_enabled` 和 `temp_unschedulable_rules`。
+- YAML 未声明的账号属于 `runtime-manual`。`codex-pool runtime list|get` 可以显示其实际状态和 `matchingTemplate`，但与模板不同不构成 drift，也不得由 `sync` 自动纳管。
+- `codex-pool runtime errors` 从实际 Sub2API 日志按账号聚合错误。规则状态码只能取 `account_upstream_error` 的观测值；`openai.forward_failed` 只作为脱敏错误摘要证据，不能凭摘要推测未观测状态码。
+- 手工账号的 runtime 配置必须通过 `codex-pool runtime apply|delete` 受控修改。写操作必须显式指定 `--account`，不带 `--confirm` 时只生成 dry-run，带 `--confirm` 才通过 Sub2API admin API 写入。
+- 内置临时不可调度配置与外部 `sentinel.*` 是独立控制面。runtime 模板负责请求路径即时冷却与 failover；`sentinel.*` 负责账号 marker health、隔离、恢复和 probe 周期，任一控制面都不得静默重写另一控制面的状态或规则。
 - The external sentinel write surface is intentionally limited to the Sub2API admin `schedulable` action. Sentinel freeze/restore may set `schedulable=false|true`, but must not write, clear, or indirectly clear Sub2API request-path runtime state such as `temp_unschedulable_until`, `temp_unschedulable_reason`, rate-limit, overload, or model-rate-limit state. In particular, sentinel restore must not call Sub2API `recover-state`, because that endpoint is a broader runtime-state recovery operation rather than a pure schedulability restore.
 - Codex accounts selected by YAML do not declare `schedulable` as durable configuration. `codex-pool sync --confirm` must not restore existing account schedulability merely because YAML selects the account or sentinel state lacks an active quarantine. Existing `schedulable=false` is runtime state: the sentinel first reads Sub2API's actual account state, schedules a recovery probe for unschedulable managed accounts, and restores `schedulable=true` only after the marker probe matches.
 - `codex-pool sync --confirm` preserves UniDesk-managed accounts that are absent from YAML by default; explicit upstream retirement requires `codex-pool sync --confirm --prune-removed`. This keeps account deletion out of the normal availability-recovery path and prevents temporary YAML edits from becoming destructive runtime changes.
@@ -165,7 +169,6 @@
 - `profiles.entries[].loadFactor` optionally overrides `pool.defaultAccountLoadFactor` for one account and is rendered to Sub2API `load_factor`. Treat it as routing policy: values belong in YAML and `codex-pool validate` output, not code constants, Secrets, or ad-hoc runtime patches.
 - Do not change account membership, priority, capacity, load factor, WebSocket mode, or other routing policy from inference alone. Unless the user explicitly asks for a configuration change, first preserve the current YAML, collect provenance and runtime evidence, and write the finding to the relevant issue or runbook before proposing a change.
 - Sub2API is a source-available UniDesk-operated runtime component. For Sub2API scheduling, failover, temporary-unschedulable behavior, error propagation, and account selection, the default investigation path is to read the current Sub2API source implementation and then verify it with real request ids, gateway logs, and original-entry traffic. Do not use mock upstreams, temporary probe accounts, or test stubs as the default proof for Sub2API behavior; those are explicit debug aids only and do not replace source-path review plus runtime evidence.
-- `profiles.entries[].tempUnschedulable` may override the pool default for one account. When enabled, the CLI renders it into Sub2API credentials as `temp_unschedulable_enabled` and `temp_unschedulable_rules`; when disabled, runtime credentials omit both fields. Use account-level override only for an explicit deviation from the pool policy, not as an availability workaround for a named account.
 - Codex account-state, quota prompts, model-routing failures, encrypted-content affinity failures, gateway wrappers, and timeout-like upstream errors must be handled by the generic temporary-unschedulable/failover path plus the external marker sentinel. Do not change membership, priority, capacity, load factor, WebSocket mode, `pool_mode`, or a specific provider's status merely to work around those errors. If a matching upstream failure still logs `openai.forward_failed` without `openai.upstream_failover_switching`, the missing fix is in Sub2API's HTTP `/responses` failover classification/error propagation, not in account pinning.
 - `profiles.entries[].openaiResponsesWebSocketsV2Mode` is the account-level Responses WebSocket v2 switch for OpenAI-compatible upstreams that require WebSocket transport. Allowed values are `off`, `ctx_pool`, and `passthrough`; omit the field unless that upstream needs it.
 - `profiles.entries[].upstreamUserAgent` is an optional account-level upstream request User-Agent override. Use it only for upstreams that require a Codex CLI compatible User-Agent; keep the value YAML-controlled and newline-free.

scripts/src/platform-infra-sub2api-codex/actions.ts

@@ -27,9 +27,9 @@ import { collectCodexProfiles, readCodexPoolConfig } from "./config";
 import { codexPoolSentinelProbeConfigFingerprint, fingerprint } from "./config-utils";
 import { apiKeyPreview, codexConsumerBaseUrl, fetchPoolApiKey, probePublicModels, validatePublicGatewayWithKey, writeLocalCodexConfig } from "./local-codex";
 import { manualBindingSourcePlan, poolTarget, prepareTargetPublicExposureSecret, protectedManualAccountNamesForTarget, resolvedManualAccountProtections, secretMaterialSummary, sentinelProfileSecrets, targetFrpPublicExposure, targetPublicExposureApplyScript, targetPublicExposureSummary } from "./public-exposure";
-import { codexPoolConfigSummary, compactProfile, compactRuntimeAccountState, compactSentinelProbeResult, redactProfile, renderSub2ApiTempUnschedulableCredentials } from "./redaction";
+import { codexPoolConfigSummary, compactProfile, compactSentinelProbeResult, redactProfile, renderSub2ApiTempUnschedulableCredentials } from "./redaction";
 import { boolField, capture, compactCapture, parseJsonOutput, runRemoteCodexPoolScript } from "./remote";
-import { cleanupProbesScript, runtimeStateScript, sentinelImageBuildScript, sentinelImageStatusScript, sentinelProbeScript, sentinelReportScript, syncScript, traceScript, validateScript } from "./remote-scripts";
+import { cleanupProbesScript, sentinelImageBuildScript, sentinelImageStatusScript, sentinelProbeScript, sentinelReportScript, syncScript, traceScript, validateScript } from "./remote-scripts";
 import { codexPoolSyncSummary, codexPoolValidationSummary, renderCodexPoolPlan, renderCodexPoolSentinelProbeResult, renderCodexPoolSyncResult, renderCodexPoolValidateResult, renderSentinelReport, renderTraceReport, renderedCliResult } from "./render";
 import { codexPoolRuntimeTarget, defaultCodexPoolRuntimeTargetId, targetFlag } from "./runtime-target";
 import { codexPoolConfigPath, serviceName, sub2apiConfigPath } from "./types";
@@ -317,35 +317,6 @@ export async function codexPoolValidate(config: UniDeskConfig, options: Disclosu
   return options.full ? response : renderCodexPoolValidateResult(response);
 }
 
-export async function codexPoolRuntimeState(config: UniDeskConfig, options: DisclosureOptions): Promise<Record<string, unknown>> {
-  const pool = readCodexPoolConfig();
-  const runtimeTarget = codexPoolRuntimeTarget(options.targetId);
-  const result = await runRemoteCodexPoolScript(config, "runtime-state", runtimeStateScript(pool, runtimeTarget), runtimeTarget);
-  const parsed = parseJsonOutput(result.stdout);
-  if (options.raw) {
-    return {
-      ok: result.exitCode === 0 && boolField(parsed, "ok", false),
-      action: "platform-infra-sub2api-codex-pool-runtime-state",
-      remote: compactCapture(result, { full: true }),
-      parsed,
-      valuesPrinted: false,
-    };
-  }
-  return {
-    ok: result.exitCode === 0 && boolField(parsed, "ok", false),
-    action: "platform-infra-sub2api-codex-pool-runtime-state",
-    target: {
-      id: runtimeTarget.id,
-      route: runtimeTarget.route,
-      namespace: runtimeTarget.namespace,
-      runtimeMode: runtimeTarget.runtimeMode,
-    },
-    state: options.full ? parsed : compactRuntimeAccountState(parsed),
-    remote: compactCapture(result, { full: result.exitCode !== 0 || parsed === null }),
-    valuesPrinted: false,
-  };
-}
-
 export async function codexPoolTrace(config: UniDeskConfig, options: TraceOptions): Promise<Record<string, unknown> | RenderedCliResult> {
   const pool = readCodexPoolConfig();
   const runtimeTarget = codexPoolRuntimeTarget(options.targetId);

scripts/src/platform-infra-sub2api-codex/config.ts

@@ -21,7 +21,7 @@ import {
 import { parseEnvFile, readTextFile, redactRepoPath, requiredEnvValue } from "../secrets";
 import { runSshCommandCapture, type SshCaptureResult } from "../ssh";
 
-import type { CodexPoolCaddyEdgeRetryConfig, CodexPoolConfig, CodexPoolLocalCodexConfig, CodexPoolManualAccountGroupBinding, CodexPoolManualAccountProtection, CodexPoolManualAccountProxyBinding, CodexPoolManualAccountsConfig, CodexPoolManualBindingKind, CodexPoolManualBindingProvider, CodexPoolManualBindingSource, CodexPoolManualBindingSourcesConfig, CodexPoolProfileConfig, CodexPoolPublicExposureConfig, CodexProfile, CodexSentinelProtectPolicy, CodexTempUnschedulablePolicy, CodexTempUnschedulableRule, OpenAIResponsesWebSocketsV2Mode } from "./types";
+import type { CodexPoolCaddyEdgeRetryConfig, CodexPoolConfig, CodexPoolLocalCodexConfig, CodexPoolManualAccountGroupBinding, CodexPoolManualAccountProtection, CodexPoolManualAccountProxyBinding, CodexPoolManualAccountsConfig, CodexPoolManualBindingKind, CodexPoolManualBindingProvider, CodexPoolManualBindingSource, CodexPoolManualBindingSourcesConfig, CodexPoolProfileConfig, CodexPoolPublicExposureConfig, CodexProfile, CodexRuntimeConfig, CodexRuntimeTemplate, CodexSentinelProtectPolicy, CodexTempUnschedulablePolicy, CodexTempUnschedulableRule, OpenAIResponsesWebSocketsV2Mode } from "./types";
 import { booleanValue, integerArrayConfigField, integerConfigField, isRecord, normalizeBaseUrl, numberValue, readRequiredBaseUrl, readRequiredDescription, readRequiredEmail, readRequiredPort, readRequiredPositiveNumber, requiredRecordConfigField, requiredStringConfigField, stringValue, validateKubernetesName } from "./config-utils";
 import { readAuthAPIKey, uniqueAccountName } from "./redaction";
 import { codexPoolConfigPath } from "./types";
@@ -59,6 +59,7 @@ export function collectCodexProfiles(): CodexProfile[] {
       priority: entry.priority,
       capacity: entry.capacity ?? pool.defaultAccountCapacity,
       loadFactor: entry.loadFactor ?? pool.defaultAccountLoadFactor,
+      tempUnschedulableTemplate: entry.tempUnschedulableTemplate,
       tempUnschedulable: entry.tempUnschedulable,
       authOpenAIKeyShape: existsSync(authPath) ? "unknown" : "missing",
       ok: false,
@@ -127,13 +128,21 @@ export function readCodexPoolConfig(): CodexPoolConfig {
   if (!isRecord(pool)) throw new Error(`${codexPoolConfigPath}.pool must be a YAML object`);
   if (!isRecord(parsed.profiles)) throw new Error(`${codexPoolConfigPath}.profiles must be a YAML object`);
   rejectSchedulableYamlField(pool, "pool");
-  const defaultTempUnschedulable = readTempUnschedulablePolicy(pool.defaultTempUnschedulable, "pool.defaultTempUnschedulable");
+  const runtime = readRuntimeConfig(parsed.runtime);
+  const defaultTempUnschedulableTemplate = readRuntimeTemplateRef(
+    pool.defaultTempUnschedulableTemplate,
+    "pool.defaultTempUnschedulableTemplate",
+    runtime,
+  );
+  const defaultTempUnschedulable = runtime.templatesById[defaultTempUnschedulableTemplate]!.tempUnschedulable;
   const defaultAccountPriorityValue = readAccountPriority(pool.defaultAccountPriority, "pool.defaultAccountPriority");
   const defaultAccountCapacityValue = readAccountCapacity(pool.defaultAccountCapacity, "pool.defaultAccountCapacity");
   const defaultAccountLoadFactorValue = readAccountLoadFactor(pool.defaultAccountLoadFactor, "pool.defaultAccountLoadFactor");
   const defaultSentinelProtect = readSentinelProtectPolicy(pool.defaultSentinelProtect, "pool.defaultSentinelProtect");
   const profiles = readProfileConfig(
     parsed.profiles,
+    runtime,
+    defaultTempUnschedulableTemplate,
     defaultTempUnschedulable,
     defaultSentinelProtect,
     defaultAccountPriorityValue,
@@ -165,9 +174,11 @@ export function readCodexPoolConfig(): CodexPoolConfig {
     defaultAccountPriority: defaultAccountPriorityValue,
     defaultAccountCapacity: defaultAccountCapacityValue,
     defaultAccountLoadFactor: defaultAccountLoadFactorValue,
+    defaultTempUnschedulableTemplate,
     defaultTempUnschedulable,
     defaultSentinelProtect,
     profiles,
+    runtime,
     manualAccounts,
     publicExposure: readPublicExposureConfig(parsed.publicExposure),
     localCodex: readLocalCodexConfig(parsed.localCodex),
@@ -188,8 +199,46 @@ export function readCodexPoolConfig(): CodexPoolConfig {
   return config;
 }
 
+export function readRuntimeConfig(value: unknown): CodexRuntimeConfig {
+  if (!isRecord(value)) throw new Error(`${codexPoolConfigPath}.runtime must be a YAML object`);
+  if (!Array.isArray(value.templates)) throw new Error(`${codexPoolConfigPath}.runtime.templates must be a YAML array`);
+  const templates: CodexRuntimeTemplate[] = value.templates.map((item, index) => {
+    if (!isRecord(item)) throw new Error(`${codexPoolConfigPath}.runtime.templates[${index}] must be a YAML object`);
+    const id = requiredStringConfigField(item, "id", `runtime.templates[${index}]`);
+    if (!/^[a-z0-9](?:[a-z0-9._-]*[a-z0-9])?$/u.test(id)) {
+      throw new Error(`${codexPoolConfigPath}.runtime.templates[${index}].id has an unsupported format`);
+    }
+    const kind = requiredStringConfigField(item, "kind", `runtime.templates[${index}]`);
+    if (kind !== "temp-unschedulable") {
+      throw new Error(`${codexPoolConfigPath}.runtime.templates[${index}].kind must be temp-unschedulable`);
+    }
+    return {
+      id,
+      kind,
+      description: readRequiredDescription(item.description, `runtime.templates[${index}].description`, 300),
+      tempUnschedulable: readTempUnschedulablePolicy(item.spec, `runtime.templates[${index}].spec`),
+    };
+  });
+  if (templates.length === 0) throw new Error(`${codexPoolConfigPath}.runtime.templates must not be empty`);
+  const templatesById: Record<string, CodexRuntimeTemplate> = {};
+  for (const template of templates) {
+    if (templatesById[template.id] !== undefined) throw new Error(`${codexPoolConfigPath}.runtime.templates duplicates id ${template.id}`);
+    templatesById[template.id] = template;
+  }
+  return { templates, templatesById };
+}
+
+export function readRuntimeTemplateRef(value: unknown, key: string, runtime: CodexRuntimeConfig): string {
+  const id = stringValue(value);
+  if (id === null) throw new Error(`${codexPoolConfigPath}.${key} must be a non-empty template id`);
+  if (runtime.templatesById[id] === undefined) throw new Error(`${codexPoolConfigPath}.${key} references missing runtime template ${id}`);
+  return id;
+}
+
 export function readProfileConfig(
   value: unknown,
+  runtime: CodexRuntimeConfig,
+  defaultTempUnschedulableTemplate: string,
   defaultTempUnschedulable: CodexTempUnschedulablePolicy,
   defaultSentinelProtect: CodexSentinelProtectPolicy,
   defaultPriority: number,
@@ -221,7 +270,13 @@ export function readProfileConfig(
     const priority = readAccountPriority(entry.priority, `profiles.entries[${index}].priority`, defaultPriority);
     const capacity = entry.capacity === undefined || entry.capacity === null ? null : readAccountCapacity(entry.capacity, `profiles.entries[${index}].capacity`);
     const loadFactor = entry.loadFactor === undefined || entry.loadFactor === null ? null : readAccountLoadFactor(entry.loadFactor, `profiles.entries[${index}].loadFactor`);
-    const tempUnschedulable = readTempUnschedulablePolicy(entry.tempUnschedulable, `profiles.entries[${index}].tempUnschedulable`, defaultTempUnschedulable);
+    if (entry.tempUnschedulable !== undefined) {
+      throw new Error(`${codexPoolConfigPath}.profiles.entries[${index}].tempUnschedulable was replaced by tempUnschedulableTemplate`);
+    }
+    const tempUnschedulableTemplate = entry.tempUnschedulableTemplate === undefined || entry.tempUnschedulableTemplate === null
+      ? defaultTempUnschedulableTemplate
+      : readRuntimeTemplateRef(entry.tempUnschedulableTemplate, `profiles.entries[${index}].tempUnschedulableTemplate`, runtime);
+    const tempUnschedulable = runtime.templatesById[tempUnschedulableTemplate]?.tempUnschedulable ?? defaultTempUnschedulable;
     return {
       profile,
       accountName,
@@ -236,6 +291,7 @@ export function readProfileConfig(
       priority,
       capacity,
       loadFactor,
+      tempUnschedulableTemplate,
       tempUnschedulable,
     };
   });

scripts/src/platform-infra-sub2api-codex/options.ts

@@ -22,8 +22,9 @@ import { parseEnvFile, readTextFile, redactRepoPath, requiredEnvValue } from "..
 import { runSshCommandCapture, type SshCaptureResult } from "../ssh";
 
 import type { ConfirmOptions, DisclosureOptions, SentinelImageOptions, SentinelProbeOptions, SentinelReportOptions, SyncOptions, TraceOptions } from "./types";
-import { codexPoolCleanupProbes, codexPoolConfigureLocal, codexPoolExpose, codexPoolPlan, codexPoolRuntimeState, codexPoolSentinelImage, codexPoolSentinelProbe, codexPoolSentinelReport, codexPoolSync, codexPoolTrace, codexPoolValidate } from "./actions";
+import { codexPoolCleanupProbes, codexPoolConfigureLocal, codexPoolExpose, codexPoolPlan, codexPoolSentinelImage, codexPoolSentinelProbe, codexPoolSentinelReport, codexPoolSync, codexPoolTrace, codexPoolValidate } from "./actions";
 import { renderCodexPoolPlan } from "./render";
+import { codexPoolRuntime } from "./runtime";
 import { defaultCodexPoolRuntimeTargetId } from "./runtime-target";
 import { codexPoolHelp } from "./types";
 
@@ -36,7 +37,7 @@ export async function runCodexPoolCommand(config: UniDeskConfig, args: string[])
   }
   if (action === "sync") return await codexPoolSync(config, parseSyncOptions(args.slice(1)));
   if (action === "validate") return await codexPoolValidate(config, parseDisclosureOptions(args.slice(1)));
-  if (action === "runtime-state") return await codexPoolRuntimeState(config, parseDisclosureOptions(args.slice(1)));
+  if (action === "runtime") return await codexPoolRuntime(config, args.slice(1));
   if (action === "trace") return await codexPoolTrace(config, parseTraceOptions(args.slice(1)));
   if (action === "sentinel-image") return await codexPoolSentinelImage(config, parseSentinelImageOptions(args.slice(1)));
   if (action === "sentinel-probe") return await codexPoolSentinelProbe(config, parseSentinelProbeOptions(args.slice(1)));

scripts/src/platform-infra-sub2api-codex/redaction.ts

@@ -70,6 +70,7 @@ export function redactProfile(profile: CodexProfile): Record<string, unknown> {
     priority: profile.priority,
     capacity: profile.capacity,
     loadFactor: profile.loadFactor,
+    tempUnschedulableTemplate: profile.tempUnschedulableTemplate,
     tempUnschedulable: tempUnschedulableSummary(profile.tempUnschedulable),
     apiKeyPresent: profile.apiKey !== null && profile.apiKey.length > 0,
     apiKeyBytes: profile.apiKey === null ? 0 : Buffer.byteLength(profile.apiKey, "utf8"),
@@ -93,6 +94,7 @@ export function compactProfile(profile: CodexProfile): Record<string, unknown> {
     sentinelProtectConsecutiveFailures: profile.sentinelProtect.enabled ? profile.sentinelProtect.consecutiveFailures : undefined,
     capacity: profile.capacity,
     loadFactor: profile.loadFactor,
+    tempUnschedulableTemplate: profile.tempUnschedulableTemplate,
     tempUnschedulableEnabled: profile.tempUnschedulable.enabled,
     tempUnschedulableRuleCount: profile.tempUnschedulable.rules.length,
     apiKeyPresent: profile.apiKey !== null && profile.apiKey.length > 0,
@@ -142,7 +144,14 @@ export function codexPoolConfigSummary(pool: CodexPoolConfig, target: CodexPoolR
     defaultAccountPriority: pool.defaultAccountPriority,
     defaultAccountCapacity: pool.defaultAccountCapacity,
     defaultAccountLoadFactor: pool.defaultAccountLoadFactor,
+    defaultTempUnschedulableTemplate: pool.defaultTempUnschedulableTemplate,
     defaultTempUnschedulable: tempUnschedulableSummary(pool.defaultTempUnschedulable),
+    runtimeTemplates: pool.runtime.templates.map((template) => ({
+      id: template.id,
+      kind: template.kind,
+      description: template.description,
+      tempUnschedulable: tempUnschedulableSummary(template.tempUnschedulable),
+    })),
     defaultSentinelProtect: pool.defaultSentinelProtect,
     profileCount: pool.profiles.length,
     manualAccounts: {
@@ -188,43 +197,6 @@ export function pickSummaryFields(item: Record<string, unknown>, keys: string[])
   return result;
 }
 
-export function compactRuntimeAccountState(parsed: Record<string, unknown> | null): Record<string, unknown> | null {
-  if (parsed === null) return null;
-  const state = isRecord(parsed.state) ? parsed.state : {};
-  const accounts = recordArray(state.accounts).map((item) => {
-    const temp = isRecord(item.tempUnschedulable) ? item.tempUnschedulable : {};
-    const poolMode = isRecord(item.poolMode) ? item.poolMode : {};
-    const optimization = isRecord(item.optimization) ? item.optimization : {};
-    return {
-      ...pickSummaryFields(item, ["accountName", "accountId", "type", "status", "schedulable", "proxyId"]),
-      tempEnabled: temp.enabled,
-      tempStatusCodes: temp.statusCodes,
-      tempActive: temp.active,
-      poolModeEnabled: poolMode.enabled,
-      tempUnschedulableNeedsAlignment: optimization.tempUnschedulableNeedsAlignment,
-      credentialsPrinted: false,
-    };
-  });
-  const runtimeImage = isRecord(parsed.runtimeImage) ? parsed.runtimeImage : {};
-  return {
-    ok: parsed.ok,
-    mode: parsed.mode,
-    namespace: parsed.namespace,
-    serviceDns: parsed.serviceDns,
-    appPod: parsed.appPod,
-    runtimeImage: pickSummaryFields(runtimeImage, ["container", "image", "ready", "restartCount", "startedAt", "health"]),
-    group: state.group,
-    yamlBaseline: state.yamlBaseline,
-    summary: state.summary,
-    accounts,
-    disclosure: {
-      full: "bun scripts/cli.ts platform-infra sub2api codex-pool runtime-state --full",
-      raw: "bun scripts/cli.ts platform-infra sub2api codex-pool runtime-state --raw",
-    },
-    valuesPrinted: false,
-  };
-}
-
 export function compactStatusBlock(block: unknown, keys: string[]): Record<string, unknown> | null {
   if (!isRecord(block)) return null;
   const items = recordArray(block.items);

scripts/src/platform-infra-sub2api-codex/remote-python-sync.ts

@@ -30,7 +30,7 @@ function pyJson(value: unknown): string {
   return `json.loads(${JSON.stringify(JSON.stringify(value))})`;
 }
 
-export function remotePythonScript(mode: "sync" | "validate" | "runtime-state" | "trace" | "cleanup-probes" | "sentinel-probe", encodedPayload: string, pool: CodexPoolConfig, target: CodexPoolRuntimeTarget): string {
+export function remotePythonScript(mode: "sync" | "validate" | "trace" | "cleanup-probes" | "sentinel-probe", encodedPayload: string, pool: CodexPoolConfig, target: CodexPoolRuntimeTarget): string {
   const hostDockerEnvPath = target.runtimeMode === "host-docker" ? target.hostDockerEnvPath : null;
   return `
 set -u
@@ -2437,134 +2437,6 @@ def account_temp_unschedulable_status(token):
         "valuesPrinted": False,
     }
 
-def normalized_int(value):
-    if isinstance(value, bool):
-        return None
-    if isinstance(value, int):
-        return value
-    if isinstance(value, float):
-        return int(value)
-    if isinstance(value, str):
-        try:
-            return int(value.strip())
-        except ValueError:
-            return None
-    return None
-
-def normalized_status_codes(value):
-    if not isinstance(value, list):
-        return None
-    codes = []
-    for item in value:
-        code = normalized_int(item)
-        if code is not None and 100 <= code <= 599 and code not in codes:
-            codes.append(code)
-    return sorted(codes)
-
-def temp_unschedulable_active(until):
-    if not isinstance(until, str) or not until.strip():
-        return False
-    try:
-        parsed = datetime.fromisoformat(until.replace("Z", "+00:00"))
-        if parsed.tzinfo is None:
-            parsed = parsed.replace(tzinfo=timezone.utc)
-        return parsed > datetime.now(timezone.utc)
-    except ValueError:
-        return False
-
-def account_runtime_state(token):
-    group = next((item for item in list_groups(token) if item.get("name") == POOL_GROUP_NAME), None)
-    if not isinstance(group, dict) or group.get("id") is None:
-        return {
-            "ok": False,
-            "group": {"name": POOL_GROUP_NAME, "id": None, "found": False},
-            "accounts": [],
-            "error": "pool-group-not-found",
-            "valuesPrinted": False,
-        }
-    baseline = normalize_temp_unschedulable_credentials(EXPECTED_DEFAULT_TEMP_UNSCHEDULABLE)
-    baseline_codes = sorted(set(rule.get("error_code") for rule in baseline["rules"] if isinstance(rule.get("error_code"), int)))
-    protected_names = set(item.get("accountName") for item in MANUAL_ACCOUNT_PROTECTIONS if isinstance(item, dict) and isinstance(item.get("accountName"), str))
-    accounts = list_accounts_for_group(token, group["id"])
-    items = []
-    for account in sorted(accounts, key=lambda item: str(item.get("name") or "")):
-        detail = get_account_detail(token, account)
-        credentials = runtime_account_credentials(detail)
-        runtime_policy = normalize_temp_unschedulable_credentials(credentials)
-        runtime_codes = sorted(set(rule.get("error_code") for rule in runtime_policy["rules"] if isinstance(rule.get("error_code"), int)))
-        name = detail.get("name") or account.get("name")
-        pool_mode = bool_value(credentials.get("pool_mode"))
-        temp_until = detail.get("temp_unschedulable_until") or detail.get("tempUnschedulableUntil")
-        temp_reason = detail.get("temp_unschedulable_reason") or detail.get("tempUnschedulableReason") or ""
-        origin = "yaml-managed" if name in EXPECTED_ACCOUNT_TEMP_UNSCHEDULABLE else ("yaml-protected-manual" if name in protected_names else "runtime-manual")
-        supports_policy = (detail.get("type") or account.get("type")) == "apikey"
-        policy_matches = runtime_policy == baseline if supports_policy else None
-        missing_codes = [code for code in baseline_codes if code not in runtime_codes] if supports_policy else []
-        items.append({
-            "accountName": name,
-            "accountId": detail.get("id") or account.get("id"),
-            "origin": origin,
-            "platform": detail.get("platform") or account.get("platform"),
-            "type": detail.get("type") or account.get("type"),
-            "status": detail.get("status") or account.get("status"),
-            "schedulable": detail.get("schedulable") if detail.get("schedulable") is not None else account.get("schedulable"),
-            "proxyId": detail.get("proxy_id") if detail.get("proxy_id") is not None else account.get("proxy_id"),
-            "priority": detail.get("priority") if detail.get("priority") is not None else account.get("priority"),
-            "concurrency": detail.get("concurrency") if detail.get("concurrency") is not None else account.get("concurrency"),
-            "currentConcurrency": account.get("current_concurrency"),
-            "baseUrl": normalize_runtime_base_url(credentials.get("base_url")),
-            "tempUnschedulable": {
-                "enabled": runtime_policy["enabled"],
-                "ruleCount": len(runtime_policy["rules"]),
-                "statusCodes": runtime_codes,
-                "rules": summarize_temp_unschedulable_rules(runtime_policy["rules"]),
-                "until": temp_until,
-                "reasonPreview": text(str(temp_reason), 500) if temp_reason else "",
-                "stateRecorded": temp_until is not None or bool(temp_reason),
-                "active": temp_unschedulable_active(temp_until),
-                "matchesYamlBaseline": policy_matches,
-                "missingYamlBaselineStatusCodes": missing_codes,
-            },
-            "poolMode": {
-                "enabled": pool_mode,
-                "retryCountConfigured": normalized_int(credentials.get("pool_mode_retry_count")) if "pool_mode_retry_count" in credentials else None,
-                "retryStatusCodesConfigured": normalized_status_codes(credentials.get("pool_mode_retry_status_codes")) if "pool_mode_retry_status_codes" in credentials else None,
-            },
-            "optimization": {
-                "tempUnschedulableNeedsAlignment": supports_policy and runtime_policy != baseline,
-                "poolModeShouldBeDisabled": pool_mode,
-            },
-            "credentialsPrinted": False,
-        })
-    temp_alignment = [item["accountName"] for item in items if item["optimization"]["tempUnschedulableNeedsAlignment"]]
-    pool_mode_enabled = [item["accountName"] for item in items if item["poolMode"]["enabled"]]
-    return {
-        "ok": True,
-        "group": {
-            "name": POOL_GROUP_NAME,
-            "id": group.get("id"),
-            "found": True,
-            "platform": group.get("platform"),
-            "status": group.get("status"),
-        },
-        "yamlBaseline": {
-            "enabled": baseline["enabled"],
-            "ruleCount": len(baseline["rules"]),
-            "statusCodes": baseline_codes,
-        },
-        "summary": {
-            "accountCount": len(items),
-            "tempUnschedulableEnabledCount": sum(1 for item in items if item["tempUnschedulable"]["enabled"]),
-            "tempUnschedulableNeedsAlignmentCount": len(temp_alignment),
-            "tempUnschedulableNeedsAlignment": temp_alignment,
-            "poolModeEnabledCount": len(pool_mode_enabled),
-            "poolModeEnabled": pool_mode_enabled,
-            "optimizationAvailable": bool(temp_alignment or pool_mode_enabled),
-        },
-        "accounts": items,
-        "valuesPrinted": False,
-    }
-
 def account_capacity_status(token):
     accounts = list_accounts(token)
     by_name = {item.get("name"): item for item in accounts if isinstance(item.get("name"), str)}
@@ -2905,21 +2777,6 @@ def run_validate():
         "validation": {"gatewayModels": gateway, "gatewayResponses": responses_smoke, "gatewayResponsesRecent": responses_evidence, "gatewayCompactRecent": compact_evidence},
     }
 
-def run_runtime_state():
-    admin_email, token, admin_compliance = login()
-    state = account_runtime_state(token)
-    return {
-        "ok": state.get("ok") is True,
-        "mode": "runtime-state",
-        "namespace": NAMESPACE,
-        "serviceDns": SERVICE_DNS,
-        "appPod": APP_POD,
-        "admin": {"email": admin_email, "tokenPrinted": False, "compliance": admin_compliance},
-        "runtimeImage": app_pod_runtime_image(),
-        "state": state,
-        "valuesPrinted": False,
-    }
-
 def parse_log_line(line):
     json_start = line.find("{")
     if json_start < 0:
@@ -3430,8 +3287,6 @@ def run_cleanup_probes():
 try:
     if MODE == "sync":
         result = run_sync()
-    elif MODE == "runtime-state":
-        result = run_runtime_state()
     elif MODE == "trace":
         result = run_trace()
     elif MODE == "cleanup-probes":

scripts/src/platform-infra-sub2api-codex/remote-scripts.ts

@@ -35,10 +35,6 @@ export function validateScript(pool: CodexPoolConfig, target: CodexPoolRuntimeTa
   return remotePythonScript("validate", "", pool, target);
 }
 
-export function runtimeStateScript(pool: CodexPoolConfig, target: CodexPoolRuntimeTarget): string {
-  return remotePythonScript("runtime-state", "", pool, target);
-}
-
 export function traceScript(pool: CodexPoolConfig, options: TraceOptions, target: CodexPoolRuntimeTarget): string {
   const encoded = Buffer.from(JSON.stringify({
     requestId: options.requestId,

scripts/src/platform-infra-sub2api-codex/remote.ts

@@ -28,7 +28,7 @@ export async function capture(config: UniDeskConfig, target: string, args: strin
   return await runSshCommandCapture(config, target, args, input);
 }
 
-export type RemoteCodexPoolMode = "sync" | "validate" | "runtime-state" | "sentinel-probe" | "sentinel-image-status" | "sentinel-image-build";
+export type RemoteCodexPoolMode = "sync" | "validate" | "runtime" | "sentinel-probe" | "sentinel-image-status" | "sentinel-image-build";
 
 export async function runRemoteCodexPoolScript(config: UniDeskConfig, mode: RemoteCodexPoolMode, script: string, target = codexPoolRuntimeTarget()): Promise<SshCaptureResult> {
   const jobName = `codex-pool-${mode}-${Date.now().toString(36)}`.slice(0, 63);
@@ -83,7 +83,7 @@ export async function runRemoteCodexPoolScript(config: UniDeskConfig, mode: Remo
 
 export function codexPoolModeCommand(mode: RemoteCodexPoolMode): string {
   if (mode === "sync") return "bun scripts/cli.ts platform-infra sub2api codex-pool sync --confirm";
-  if (mode === "runtime-state") return "bun scripts/cli.ts platform-infra sub2api codex-pool runtime-state";
+  if (mode === "runtime") return "bun scripts/cli.ts platform-infra sub2api codex-pool runtime list";
   if (mode === "sentinel-probe") return "bun scripts/cli.ts platform-infra sub2api codex-pool sentinel-probe --account <accountName> --confirm";
   if (mode === "sentinel-image-status") return "bun scripts/cli.ts platform-infra sub2api codex-pool sentinel-image status";
   if (mode === "sentinel-image-build") return "bun scripts/cli.ts platform-infra sub2api codex-pool sentinel-image build --confirm";

scripts/src/platform-infra-sub2api-codex/runtime.ts

@@ -0,0 +1,552 @@
+import type { UniDeskConfig } from "../config";
+
+import { desiredAccountNames } from "./accounts";
+import { readCodexPoolConfig } from "./config";
+import { protectedManualAccountsForTarget } from "./public-exposure";
+import { renderSub2ApiTempUnschedulableCredentials } from "./redaction";
+import { boolField, compactCapture, parseJsonOutput, runRemoteCodexPoolScript } from "./remote";
+import { codexPoolRuntimeTarget, defaultCodexPoolRuntimeTargetId } from "./runtime-target";
+
+type RuntimeAction = "list" | "get" | "errors" | "apply" | "delete";
+
+interface RuntimeOptions {
+  action: RuntimeAction;
+  account: string | null;
+  template: string | null;
+  kind: "temp-unschedulable";
+  confirm: boolean;
+  full: boolean;
+  raw: boolean;
+  since: string;
+  tail: number;
+  targetId: string;
+}
+
+export async function codexPoolRuntime(config: UniDeskConfig, args: string[]): Promise<Record<string, unknown>> {
+  const options = parseRuntimeOptions(args);
+  const pool = readCodexPoolConfig();
+  const target = codexPoolRuntimeTarget(options.targetId);
+  const selectedTemplate = options.template === null ? null : pool.runtime.templatesById[options.template];
+  if (options.template !== null && selectedTemplate === undefined) {
+    throw new Error(`unknown runtime template ${options.template}; available: ${pool.runtime.templates.map((item) => item.id).join(", ")}`);
+  }
+  const payload = {
+    action: options.action,
+    account: options.account,
+    kind: options.kind,
+    confirm: options.confirm,
+    full: options.full,
+    since: options.since,
+    tail: options.tail,
+    groupName: pool.groupName,
+    adminEmailDefault: pool.adminEmailDefault,
+    managedAccountNames: desiredAccountNames(pool),
+    protectedManualAccountNames: protectedManualAccountsForTarget(pool, target).map((item) => item.accountName),
+    templates: pool.runtime.templates.map((template) => ({
+      id: template.id,
+      kind: template.kind,
+      description: template.description,
+      credentials: renderSub2ApiTempUnschedulableCredentials(template.tempUnschedulable),
+    })),
+    selectedTemplate: selectedTemplate === null || selectedTemplate === undefined ? null : {
+      id: selectedTemplate.id,
+      kind: selectedTemplate.kind,
+      credentials: renderSub2ApiTempUnschedulableCredentials(selectedTemplate.tempUnschedulable),
+    },
+  };
+  const result = await runRemoteCodexPoolScript(config, "runtime", runtimeScript(payload, target), target);
+  const parsed = parseJsonOutput(result.stdout);
+  const ok = result.exitCode === 0 && boolField(parsed, "ok", false);
+  if (options.raw) {
+    return {
+      ok,
+      action: "platform-infra-sub2api-codex-pool-runtime",
+      remote: compactCapture(result, { full: true }),
+      parsed,
+      valuesPrinted: false,
+    };
+  }
+  return {
+    ok,
+    action: "platform-infra-sub2api-codex-pool-runtime",
+    target: {
+      id: target.id,
+      route: target.route,
+      namespace: target.namespace,
+      runtimeMode: target.runtimeMode,
+      endpoint: target.serviceDns,
+    },
+    request: {
+      operation: options.action,
+      account: options.account,
+      template: options.template,
+      kind: options.kind,
+      confirm: options.confirm,
+      since: options.action === "errors" ? options.since : undefined,
+      tail: options.action === "errors" ? options.tail : undefined,
+    },
+    runtime: parsed,
+    remote: compactCapture(result, { full: options.full || result.exitCode !== 0 || parsed === null }),
+    disclosure: options.action === "list"
+      ? {
+        get: "bun scripts/cli.ts platform-infra sub2api codex-pool runtime get --account <name-or-id>",
+        errors: "bun scripts/cli.ts platform-infra sub2api codex-pool runtime errors [--account <name-or-id>] [--since 24h]",
+      }
+      : undefined,
+    valuesPrinted: false,
+  };
+}
+
+export function parseRuntimeOptions(args: string[]): RuntimeOptions {
+  const [actionRaw = "list", ...rest] = args;
+  if (actionRaw !== "list" && actionRaw !== "get" && actionRaw !== "errors" && actionRaw !== "apply" && actionRaw !== "delete") {
+    throw new Error("runtime usage: list|get|errors|apply|delete [--account <name-or-id>] [--since 24h] [--tail 50000] [--template <id>] [--kind temp-unschedulable] [--confirm] [--target <id>] [--full|--raw]");
+  }
+  let account: string | null = null;
+  let template: string | null = null;
+  let kind = "temp-unschedulable" as const;
+  let confirm = false;
+  let full = false;
+  let raw = false;
+  let since = "24h";
+  let tail = 50_000;
+  let targetId = defaultCodexPoolRuntimeTargetId();
+  for (let index = 0; index < rest.length; index += 1) {
+    const arg = rest[index]!;
+    const readValue = (name: string): string => {
+      const value = rest[index + 1];
+      if (value === undefined || value.startsWith("--")) throw new Error(`${name} requires a value`);
+      index += 1;
+      return value.trim();
+    };
+    if (arg === "--confirm") confirm = true;
+    else if (arg === "--full") full = true;
+    else if (arg === "--raw") raw = true;
+    else if (arg === "--account") account = readValue("--account");
+    else if (arg.startsWith("--account=")) account = arg.slice("--account=".length).trim();
+    else if (arg === "--template") template = readValue("--template");
+    else if (arg.startsWith("--template=")) template = arg.slice("--template=".length).trim();
+    else if (arg === "--kind") kind = parseRuntimeKind(readValue("--kind"));
+    else if (arg.startsWith("--kind=")) kind = parseRuntimeKind(arg.slice("--kind=".length));
+    else if (arg === "--target") targetId = readValue("--target");
+    else if (arg.startsWith("--target=")) targetId = arg.slice("--target=".length).trim();
+    else if (arg === "--since") since = readValue("--since");
+    else if (arg.startsWith("--since=")) since = arg.slice("--since=".length).trim();
+    else if (arg === "--tail") tail = parseRuntimeTail(readValue("--tail"));
+    else if (arg.startsWith("--tail=")) tail = parseRuntimeTail(arg.slice("--tail=".length));
+    else throw new Error(`unsupported runtime option: ${arg}`);
+  }
+  if ((actionRaw === "get" || actionRaw === "apply" || actionRaw === "delete") && !account) {
+    throw new Error(`runtime ${actionRaw} requires --account <name-or-id>`);
+  }
+  if (account !== null && (account.length > 256 || /[\r\n]/u.test(account))) throw new Error("--account has an unsupported format");
+  if (actionRaw === "apply" && !template) throw new Error("runtime apply requires --template <id>");
+  if (actionRaw !== "apply" && template !== null) throw new Error(`runtime ${actionRaw} does not accept --template`);
+  if (actionRaw !== "errors" && (since !== "24h" || tail !== 50_000)) throw new Error(`runtime ${actionRaw} does not accept --since or --tail`);
+  if (!/^\d+[mhd]$/u.test(since)) throw new Error("--since must use <number>m, <number>h, or <number>d");
+  if ((actionRaw === "list" || actionRaw === "get" || actionRaw === "errors") && confirm) throw new Error(`runtime ${actionRaw} does not accept --confirm`);
+  if (full && raw) throw new Error("use only one of --full or --raw");
+  return { action: actionRaw, account, template, kind, confirm, full, raw, since, tail, targetId };
+}
+
+function parseRuntimeKind(value: string): "temp-unschedulable" {
+  if (value !== "temp-unschedulable") throw new Error("--kind must be temp-unschedulable");
+  return value;
+}
+
+function parseRuntimeTail(value: string): number {
+  const parsed = Number(value);
+  if (!Number.isInteger(parsed) || parsed < 100 || parsed > 200_000) throw new Error("--tail must be an integer from 100 to 200000");
+  return parsed;
+}
+
+function pyJson(value: unknown): string {
+  return `json.loads(${JSON.stringify(JSON.stringify(value))})`;
+}
+
+function runtimeScript(payload: unknown, target: ReturnType<typeof codexPoolRuntimeTarget>): string {
+  return `
+set -u
+python3 - <<'PY'
+import json
+import re
+import subprocess
+import sys
+from urllib.parse import quote
+
+RUNTIME_MODE = ${pyJson(target.runtimeMode)}
+NAMESPACE = ${pyJson(target.namespace)}
+HOST_DOCKER_APP_PORT = ${pyJson(target.hostDockerAppPort)}
+HOST_DOCKER_ENV_PATH = ${pyJson(target.hostDockerEnvPath)}
+HOST_DOCKER_APP_CONTAINER = "sub2api-app"
+APP_SECRET_NAME = ${pyJson(target.appSecretName)}
+PAYLOAD = ${pyJson(payload)}
+
+def run(cmd, input_bytes=None):
+    return subprocess.run(cmd, input=input_bytes, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+
+def text(value, limit=1000):
+    if isinstance(value, bytes):
+        value = value.decode("utf-8", errors="replace")
+    return str(value)[-limit:]
+
+def docker(args):
+    proc = run(["docker", *args])
+    if proc.returncode == 0:
+        return proc
+    sudo_proc = run(["sudo", "-n", "docker", *args])
+    return sudo_proc if sudo_proc.returncode == 0 else proc
+
+def kubectl(args, input_bytes=None):
+    return run(["kubectl", *args], input_bytes)
+
+def kube_json(args, label):
+    proc = kubectl([*args, "-o", "json"])
+    if proc.returncode != 0:
+        raise RuntimeError(f"{label} failed: {text(proc.stderr)}")
+    return json.loads(proc.stdout.decode("utf-8"))
+
+def read_host_env():
+    try:
+        with open(HOST_DOCKER_ENV_PATH, "r", encoding="utf-8") as handle:
+            lines = handle.read().splitlines()
+    except PermissionError:
+        proc = run(["sudo", "-n", "cat", HOST_DOCKER_ENV_PATH])
+        if proc.returncode != 0:
+            raise RuntimeError("read host env failed: " + text(proc.stderr))
+        lines = proc.stdout.decode("utf-8", errors="replace").splitlines()
+    values = {}
+    for line in lines:
+        stripped = line.strip()
+        if not stripped or stripped.startswith("#") or "=" not in stripped:
+            continue
+        key, value = stripped.split("=", 1)
+        value = value.strip()
+        if len(value) >= 2 and value[0] == value[-1] and value[0] in ("'", '"'):
+            value = value[1:-1]
+        values[key.strip()] = value
+    return values
+
+def select_app_pod():
+    if RUNTIME_MODE == "host-docker":
+        return HOST_DOCKER_APP_CONTAINER
+    pods = kube_json(["-n", NAMESPACE, "get", "pods", "-l", "app.kubernetes.io/name=sub2api"], "sub2api pods").get("items") or []
+    running = [item for item in pods if (item.get("status") or {}).get("phase") == "Running"]
+    if not running:
+        raise RuntimeError("sub2api app pod not found")
+    return running[0]["metadata"]["name"]
+
+APP_POD = select_app_pod()
+
+def config_value(name, key, fallback=None):
+    if RUNTIME_MODE == "host-docker":
+        return read_host_env().get(key) or fallback
+    data = kube_json(["-n", NAMESPACE, "get", "configmap", name], f"configmap/{name}").get("data") or {}
+    return data.get(key) or fallback
+
+def secret_value(name, key):
+    if RUNTIME_MODE == "host-docker":
+        return read_host_env().get(key)
+    import base64
+    data = kube_json(["-n", NAMESPACE, "get", "secret", name], f"secret/{name}").get("data") or {}
+    return base64.b64decode(data[key]).decode("utf-8") if key in data else None
+
+def parse_curl(proc):
+    output = proc.stdout.decode("utf-8", errors="replace")
+    marker = "\\n__HTTP_CODE__:"
+    pos = output.rfind(marker)
+    if pos < 0:
+        return {"ok": False, "httpStatus": 0, "json": None, "body": output, "stderr": text(proc.stderr)}
+    body = output[:pos]
+    try:
+        status = int(output[pos + len(marker):].strip()[-3:])
+    except ValueError:
+        status = 0
+    try:
+        parsed = json.loads(body) if body.strip() else None
+    except json.JSONDecodeError:
+        parsed = None
+    return {"ok": proc.returncode == 0 and 200 <= status < 300, "httpStatus": status, "json": parsed, "body": body, "stderr": text(proc.stderr)}
+
+def curl_api(method, path, bearer=None, payload=None):
+    body = b"" if payload is None else json.dumps(payload, separators=(",", ":")).encode("utf-8")
+    script = r'''
+set -eu
+method="$1"
+url="$2"
+token="\${3:-}"
+tmp="$(mktemp)"
+trap 'rm -f "$tmp"' EXIT
+cat > "$tmp"
+if [ -n "$token" ] && [ "$method" = GET ] && [ ! -s "$tmp" ]; then
+  curl -sS -w '\n__HTTP_CODE__:%{http_code}' -X "$method" -H "Authorization: Bearer $token" "$url"
+elif [ -n "$token" ]; then
+  curl -sS -w '\n__HTTP_CODE__:%{http_code}' -X "$method" -H 'Content-Type: application/json' -H "Authorization: Bearer $token" --data-binary @"$tmp" "$url"
+elif [ "$method" = GET ] && [ ! -s "$tmp" ]; then
+  curl -sS -w '\n__HTTP_CODE__:%{http_code}' -X "$method" "$url"
+else
+  curl -sS -w '\n__HTTP_CODE__:%{http_code}' -X "$method" -H 'Content-Type: application/json' --data-binary @"$tmp" "$url"
+fi
+'''
+    if RUNTIME_MODE == "host-docker":
+        proc = run(["sh", "-c", script, "sh", method, f"http://127.0.0.1:{HOST_DOCKER_APP_PORT}{path}", bearer or ""], body)
+    else:
+        proc = run(["kubectl", "-n", NAMESPACE, "exec", "-i", APP_POD, "--", "sh", "-c", script, "sh", method, f"http://127.0.0.1:8080{path}", bearer or ""], body)
+    return parse_curl(proc)
+
+def data_of(response, label):
+    parsed = response.get("json")
+    code = parsed.get("code") if isinstance(parsed, dict) else None
+    if response.get("ok") is not True or (code is not None and code != 0):
+        message = parsed.get("message") if isinstance(parsed, dict) else text(response.get("body"), 500)
+        raise RuntimeError(f"{label} failed: http={response.get('httpStatus')} message={message}")
+    return parsed.get("data") if isinstance(parsed, dict) and "data" in parsed else parsed
+
+def items_of(data):
+    if isinstance(data, list):
+        return data
+    if isinstance(data, dict):
+        for key in ("items", "groups", "accounts"):
+            if isinstance(data.get(key), list):
+                return data[key]
+    return []
+
+def login():
+    email = config_value("sub2api-config", "ADMIN_EMAIL", PAYLOAD["adminEmailDefault"])
+    password = secret_value(APP_SECRET_NAME, "ADMIN_PASSWORD")
+    if not password:
+        raise RuntimeError("ADMIN_PASSWORD missing")
+    data = data_of(curl_api("POST", "/api/v1/auth/login", payload={"email": email, "password": password}), "admin login")
+    token = None
+    if isinstance(data, dict):
+        token = data.get("access_token") or data.get("token")
+    if not token:
+        raise RuntimeError("admin login response has no token")
+    return token
+
+def list_groups(token):
+    return items_of(data_of(curl_api("GET", "/api/v1/admin/groups/all?platform=openai", bearer=token), "list groups"))
+
+def list_group_accounts(token, group_id):
+    path = f"/api/v1/admin/accounts?group_id={group_id}&page=1&page_size=500&platform=openai"
+    return items_of(data_of(curl_api("GET", path, bearer=token), "list group accounts"))
+
+def account_detail(token, account_id):
+    return data_of(curl_api("GET", f"/api/v1/admin/accounts/{account_id}", bearer=token), "get account")
+
+def normalize_policy(credentials):
+    if not isinstance(credentials, dict):
+        credentials = {}
+    enabled = credentials.get("temp_unschedulable_enabled") is True
+    raw = credentials.get("temp_unschedulable_rules")
+    if isinstance(raw, str):
+        try:
+            raw = json.loads(raw)
+        except json.JSONDecodeError:
+            raw = []
+    rules = raw if isinstance(raw, list) else []
+    codes = sorted(set(item.get("error_code") for item in rules if isinstance(item, dict) and isinstance(item.get("error_code"), int)))
+    return {"enabled": enabled, "ruleCount": len(rules), "statusCodes": codes, "rules": rules}
+
+def template_summary(template):
+    policy = normalize_policy(template.get("credentials"))
+    return {"id": template.get("id"), "kind": template.get("kind"), "description": template.get("description"), **{key: policy[key] for key in ("enabled", "ruleCount", "statusCodes")}}
+
+def matching_template(credentials):
+    current = normalize_policy(credentials)
+    for template in PAYLOAD["templates"]:
+        if normalize_policy(template.get("credentials")) == current:
+            return template.get("id")
+    return None
+
+def account_summary(detail, include_rules=False):
+    credentials = detail.get("credentials") if isinstance(detail.get("credentials"), dict) else {}
+    policy = normalize_policy(credentials)
+    name = detail.get("name")
+    management = "yaml-managed" if name in PAYLOAD["managedAccountNames"] else ("yaml-protected-manual" if name in PAYLOAD["protectedManualAccountNames"] else "runtime-manual")
+    result = {
+        "accountName": name,
+        "accountId": detail.get("id"),
+        "management": management,
+        "type": detail.get("type"),
+        "status": detail.get("status"),
+        "schedulable": detail.get("schedulable"),
+        "proxyId": detail.get("proxy_id"),
+        "tempUnschedulable": {key: policy[key] for key in ("enabled", "ruleCount", "statusCodes")},
+        "matchingTemplate": matching_template(credentials),
+        "poolModeEnabled": credentials.get("pool_mode") is True,
+        "credentialsPrinted": False,
+    }
+    if include_rules:
+        result["tempUnschedulable"]["rules"] = policy["rules"]
+    return result
+
+def policy_summary(policy, include_rules=False):
+    keys = ("enabled", "ruleCount", "statusCodes", "rules") if include_rules else ("enabled", "ruleCount", "statusCodes")
+    return {key: policy[key] for key in keys}
+
+def find_account(token, accounts, selector):
+    exact = [item for item in accounts if str(item.get("id")) == selector or item.get("name") == selector]
+    if len(exact) != 1:
+        raise RuntimeError("account-not-found-or-ambiguous: " + selector)
+    return account_detail(token, exact[0]["id"])
+
+def runtime_log_lines():
+    since = PAYLOAD.get("since", "24h")
+    tail = str(PAYLOAD.get("tail", 50000))
+    if RUNTIME_MODE == "host-docker":
+        proc = docker(["logs", "--since", since, "--tail", tail, APP_POD])
+    else:
+        proc = kubectl(["-n", NAMESPACE, "logs", APP_POD, "--since=" + since, "--tail=" + tail])
+    if proc.returncode != 0:
+        raise RuntimeError("read Sub2API runtime logs failed: " + text(proc.stderr))
+    output = proc.stdout + b"\\n" + proc.stderr
+    return output.decode("utf-8", errors="replace").splitlines()
+
+def compact_error(value):
+    if not isinstance(value, str):
+        return ""
+    return " ".join(value.replace("\x00", "").split())[:240]
+
+def observed_runtime_errors(accounts):
+    selector = PAYLOAD.get("account")
+    selected = [item for item in accounts if selector is None or str(item.get("id")) == str(selector) or item.get("name") == selector]
+    if selector is not None and len(selected) != 1:
+        raise RuntimeError("account-not-found-or-ambiguous: " + str(selector))
+    selected_ids = set(item.get("id") for item in selected)
+    stats = {}
+    for item in selected:
+        stats[item.get("id")] = {
+            "accountName": item.get("name"),
+            "accountId": item.get("id"),
+            "upstreamEventCount": 0,
+            "observedStatusCodes": {},
+            "forwardFailureCount": 0,
+            "forwardFailureStatusCodes": {},
+            "lastSeenAt": None,
+            "errorSamples": [],
+        }
+    scanned = 0
+    for line in runtime_log_lines():
+        if "account_upstream_error" not in line and "openai.forward_failed" not in line:
+            continue
+        json_start = line.find("{")
+        if json_start < 0:
+            continue
+        try:
+            item = json.loads(line[json_start:])
+        except json.JSONDecodeError:
+            continue
+        account_id = item.get("account_id") if isinstance(item, dict) else None
+        if account_id not in selected_ids:
+            continue
+        scanned += 1
+        at = line.split("\t", 1)[0].strip() or None
+        current = stats[account_id]
+        if at and (current["lastSeenAt"] is None or at > current["lastSeenAt"]):
+            current["lastSeenAt"] = at
+        if "account_upstream_error" in line:
+            code = item.get("status_code")
+            if isinstance(code, int):
+                key = str(code)
+                current["observedStatusCodes"][key] = current["observedStatusCodes"].get(key, 0) + 1
+            current["upstreamEventCount"] += 1
+            continue
+        current["forwardFailureCount"] += 1
+        error = compact_error(item.get("error"))
+        match = re.search(r"(?:upstream error:\\s*|status(?:_code)?[=:]\\s*)(\\d{3})", error, re.IGNORECASE)
+        if match:
+            key = match.group(1)
+            current["forwardFailureStatusCodes"][key] = current["forwardFailureStatusCodes"].get(key, 0) + 1
+        if error and error not in current["errorSamples"] and len(current["errorSamples"]) < 3:
+            current["errorSamples"].append(error)
+    rows = []
+    aggregate = {}
+    for item in selected:
+        current = stats[item.get("id")]
+        observed = [{"statusCode": int(code), "count": count} for code, count in sorted(current["observedStatusCodes"].items(), key=lambda pair: int(pair[0]))]
+        forwarded = [{"statusCode": int(code), "count": count} for code, count in sorted(current["forwardFailureStatusCodes"].items(), key=lambda pair: int(pair[0]))]
+        for entry in observed:
+            key = str(entry["statusCode"])
+            aggregate[key] = aggregate.get(key, 0) + entry["count"]
+        current["observedStatusCodes"] = observed
+        current["forwardFailureStatusCodes"] = forwarded
+        if PAYLOAD.get("full") is not True:
+            current["errorSamples"] = current["errorSamples"][:1]
+        if current["upstreamEventCount"] > 0 or current["forwardFailureCount"] > 0 or selector is not None:
+            rows.append(current)
+    return {
+        "window": {"since": PAYLOAD.get("since"), "tail": PAYLOAD.get("tail"), "matchedLogEvents": scanned},
+        "aggregateObservedStatusCodes": [{"statusCode": int(code), "count": count} for code, count in sorted(aggregate.items(), key=lambda pair: int(pair[0]))],
+        "accountCountWithErrors": sum(1 for item in rows if item["upstreamEventCount"] > 0 or item["forwardFailureCount"] > 0),
+        "accounts": rows,
+        "message": "observedStatusCodes comes only from account_upstream_error; forward failures are supplemental message evidence",
+    }
+
+def runtime_result():
+    token = login()
+    group = next((item for item in list_groups(token) if item.get("name") == PAYLOAD["groupName"]), None)
+    if not isinstance(group, dict) or group.get("id") is None:
+        raise RuntimeError("pool-group-not-found")
+    accounts = list_group_accounts(token, group["id"])
+    base = {
+        "group": {"id": group.get("id"), "name": group.get("name"), "status": group.get("status")},
+        "templates": [template_summary(item) for item in PAYLOAD["templates"]],
+        "endpoint": "host-docker" if RUNTIME_MODE == "host-docker" else "k3s-service",
+        "valuesPrinted": False,
+    }
+    action = PAYLOAD["action"]
+    if action == "list":
+        details = [account_detail(token, item["id"]) for item in accounts]
+        return {**base, "ok": True, "operation": "list", "mutation": False, "accountCount": len(details), "accounts": [account_summary(item) for item in sorted(details, key=lambda value: value.get("name") or "")]}
+    if action == "errors":
+        errors = observed_runtime_errors(accounts)
+        return {**base, "ok": True, "operation": "errors", "mutation": False, "errors": errors}
+    detail = find_account(token, accounts, str(PAYLOAD["account"]))
+    if action == "get":
+        return {**base, "ok": True, "operation": "get", "mutation": False, "account": account_summary(detail, True)}
+    if detail.get("type") != "apikey":
+        raise RuntimeError("runtime temp-unschedulable policy requires an apikey account")
+    credentials = dict(detail.get("credentials") or {})
+    before = normalize_policy(credentials)
+    if action == "apply":
+        template = PAYLOAD.get("selectedTemplate")
+        if not isinstance(template, dict):
+            raise RuntimeError("selected runtime template missing")
+        desired_fields = template.get("credentials") or {}
+        desired = dict(credentials)
+        desired.update(desired_fields)
+        after_plan = normalize_policy(desired)
+        exists = "temp_unschedulable_enabled" in credentials or "temp_unschedulable_rules" in credentials
+        change = "noop" if before == after_plan else ("update" if exists else "create")
+    else:
+        desired = dict(credentials)
+        desired.pop("temp_unschedulable_enabled", None)
+        desired.pop("temp_unschedulable_rules", None)
+        after_plan = normalize_policy(desired)
+        change = "delete" if before != after_plan or "temp_unschedulable_enabled" in credentials or "temp_unschedulable_rules" in credentials else "noop"
+    include_plan_rules = PAYLOAD.get("full") is True
+    plan = {
+        "change": change,
+        "account": account_summary(detail, include_plan_rules),
+        "before": policy_summary(before, include_plan_rules),
+        "after": policy_summary(after_plan, include_plan_rules),
+        "template": PAYLOAD.get("selectedTemplate", {}).get("id") if isinstance(PAYLOAD.get("selectedTemplate"), dict) else None,
+        "credentialsPrinted": False,
+    }
+    if PAYLOAD.get("confirm") is not True or change == "noop":
+        return {**base, "ok": True, "operation": action, "mode": "dry-run" if PAYLOAD.get("confirm") is not True else "confirmed-noop", "mutation": False, "plan": plan}
+    updated = data_of(curl_api("PUT", f"/api/v1/admin/accounts/{detail['id']}", bearer=token, payload={"credentials": desired}), "update account runtime config")
+    after_detail = updated if isinstance(updated, dict) else account_detail(token, detail["id"])
+    return {**base, "ok": True, "operation": action, "mode": "confirmed", "mutation": True, "plan": plan, "account": account_summary(after_detail, include_plan_rules)}
+
+try:
+    result = runtime_result()
+except Exception as exc:
+    result = {"ok": False, "operation": PAYLOAD.get("action"), "mutation": False, "error": str(exc), "valuesPrinted": False}
+
+print(json.dumps(result, ensure_ascii=False, indent=2))
+sys.exit(0 if result.get("ok") else 1)
+PY
+`;
+}

scripts/src/platform-infra-sub2api-codex/types.ts

@@ -139,6 +139,7 @@ export interface CodexProfile {
   priority: number;
   capacity: number;
   loadFactor: number;
+  tempUnschedulableTemplate: string;
   tempUnschedulable: CodexTempUnschedulablePolicy;
   authOpenAIKeyShape: string;
   ok: boolean;
@@ -159,6 +160,18 @@ export interface CodexTempUnschedulablePolicy {
   rules: CodexTempUnschedulableRule[];
 }
 
+export interface CodexRuntimeTemplate {
+  id: string;
+  kind: "temp-unschedulable";
+  description: string;
+  tempUnschedulable: CodexTempUnschedulablePolicy;
+}
+
+export interface CodexRuntimeConfig {
+  templates: CodexRuntimeTemplate[];
+  templatesById: Record<string, CodexRuntimeTemplate>;
+}
+
 export interface CodexPoolConfig {
   version: number;
   kind: string;
@@ -179,9 +192,11 @@ export interface CodexPoolConfig {
   defaultAccountPriority: number;
   defaultAccountCapacity: number;
   defaultAccountLoadFactor: number;
+  defaultTempUnschedulableTemplate: string;
   defaultTempUnschedulable: CodexTempUnschedulablePolicy;
   defaultSentinelProtect: CodexSentinelProtectPolicy;
   profiles: CodexPoolProfileConfig[];
+  runtime: CodexRuntimeConfig;
   manualAccounts: CodexPoolManualAccountsConfig;
   publicExposure: CodexPoolPublicExposureConfig;
   localCodex: CodexPoolLocalCodexConfig;
@@ -251,6 +266,7 @@ export interface CodexPoolProfileConfig {
   priority: number;
   capacity: number | null;
   loadFactor: number | null;
+  tempUnschedulableTemplate: string;
   tempUnschedulable: CodexTempUnschedulablePolicy;
 }
 
@@ -325,14 +341,18 @@ export function codexPoolHelp(): unknown {
   const pool = readCodexPoolConfig();
   const runtimeTarget = codexPoolRuntimeTarget();
   return {
-    command: "platform-infra sub2api codex-pool plan|sync|validate|runtime-state|trace|sentinel-image|sentinel-probe|sentinel-report|cleanup-probes|expose|configure-local",
+    command: "platform-infra sub2api codex-pool plan|sync|validate|runtime|trace|sentinel-image|sentinel-probe|sentinel-report|cleanup-probes|expose|configure-local",
     output: "json, except trace and sentinel-report default to low-noise text tables",
     usage: [
       "bun scripts/cli.ts platform-infra sub2api codex-pool plan",
       "bun scripts/cli.ts platform-infra sub2api codex-pool plan --target D601",
       "bun scripts/cli.ts platform-infra sub2api codex-pool sync [--target D601] --confirm [--prune-removed]",
       "bun scripts/cli.ts platform-infra sub2api codex-pool validate [--target D601] [--full|--raw]",
-      "bun scripts/cli.ts platform-infra sub2api codex-pool runtime-state [--target PK01] [--full|--raw]",
+      "bun scripts/cli.ts platform-infra sub2api codex-pool runtime list [--target PK01] [--full|--raw]",
+      "bun scripts/cli.ts platform-infra sub2api codex-pool runtime get --account <name-or-id> [--target PK01] [--full|--raw]",
+      "bun scripts/cli.ts platform-infra sub2api codex-pool runtime errors [--account <name-or-id>] [--since 24h] [--tail 50000] [--target PK01] [--full|--raw]",
+      "bun scripts/cli.ts platform-infra sub2api codex-pool runtime apply --account <name-or-id> --template <id> [--target PK01] [--confirm]",
+      "bun scripts/cli.ts platform-infra sub2api codex-pool runtime delete --account <name-or-id> --kind temp-unschedulable [--target PK01] [--confirm]",
       "bun scripts/cli.ts platform-infra sub2api codex-pool trace [--target D601] --request-id <requestId> [--since 24h|--tail 20000|--context-seconds 300|--show-lines|--raw]",
       "bun scripts/cli.ts platform-infra sub2api codex-pool sentinel-image status [--target D601]",
       "bun scripts/cli.ts platform-infra sub2api codex-pool sentinel-image build [--target D601] --confirm",