Merge remote-tracking branch 'origin/master' into feat/hwlab-1119-d601-external-db
# Conflicts: # config/platform-db/postgres-pk01.yaml # scripts/src/platform-db.ts
This commit is contained in:
Codex
@@ -1,6 +1,6 @@
|
||||
---
|
||||
name: unidesk-ops
|
||||
description: UniDesk 手动运维 CLI — `server`、`gc` 和 PK01 `platform-db postgres` 子命令，覆盖主 server 启停、健康检查、swap、日志、Docker 镜像清理、磁盘 GC、服务重建和 PK01 host PostgreSQL 运维。用户提到 server start、server status、server swap、server rebuild、gc、磁盘清理、platform-db、PK01 PostgreSQL、运维时使用。
|
||||
description: UniDesk 手动运维 CLI — `server`、`gc` 和 PK01 `platform-db postgres` 子命令，覆盖主 server 启停、健康检查、swap、日志、Docker 镜像清理、磁盘 GC、服务重建/重启和 PK01 host PostgreSQL 运维。用户提到 server start、server status、server swap、server rebuild、server restart、gc、磁盘清理、platform-db、PK01 PostgreSQL、运维时使用。
|
||||
---
|
||||
|
||||
# UniDesk 手动运维 CLI
|
||||
@@ -16,9 +16,11 @@ description: UniDesk 手动运维 CLI — `server`、`gc` 和 PK01 `platform-db
|
||||
```bash
|
||||
bun scripts/cli.ts server start
|
||||
bun scripts/cli.ts server stop
|
||||
bun scripts/cli.ts server restart <service>
|
||||
```
|
||||
|
||||
异步 job 模式，返回 `job.id`、日志路径。`start` 执行 Docker 构建+启动，`stop` 停止 Compose project 全部服务。
|
||||
`restart` 是无构建单服务维护重启，使用现有镜像执行 `--no-build --no-deps --force-recreate` 并等待容器 `healthy/running`，适合刷新 provider-gateway 这类运行态异常，不能替代镜像发布或源码构建。
|
||||
|
||||
---
|
||||
|
||||
@@ -32,6 +34,18 @@ bun scripts/cli.ts server status
|
||||
|
||||
低内存时 `swap.warning` 非空，先执行 `server swap ensure`。
|
||||
|
||||
## 节点资源指标同步
|
||||
|
||||
资源面板显示旧 CPU/内存/磁盘值时，先对照 provider 上报、backend-core 落库和内部 API 三段：
|
||||
|
||||
```bash
|
||||
docker logs --tail 120 unidesk-provider-gateway-main | rg 'system_status_sent|docker_status_sent|error'
|
||||
docker logs --tail 200 unidesk-backend-core | rg 'provider_system_status|provider_docker_status|provider_message_failed|serializing parameter'
|
||||
docker exec unidesk-backend-core sh -lc 'backend-core --fetch-json http://127.0.0.1:8080/api/nodes/system-status?limit=5'
|
||||
```
|
||||
|
||||
provider 已发送但 backend-core 没有 `provider_system_status` / `provider_docker_status`，优先查 backend-core ingest/DB 写入错误；provider 自身停发时才用 `server restart provider-gateway` 做无构建维护重启。修复 backend-core 后必须用 `server rebuild backend-core` 上线并验证 `stale=false`、`currentCollectedAt` 为当前采样。长期语义见 `docs/reference/observability.md#node-resource-status`。
|
||||
|
||||
---
|
||||
|
||||
## Swap 管理
|
||||
@@ -141,9 +155,11 @@ bun scripts/cli.ts platform-db postgres apply --config config/platform-db/postgr
|
||||
|
||||
- `plan` / `status` 只读；`apply --confirm` 默认创建本地异步 job；`apply --confirm --wait` 会启动 PK01 侧 root-owned job 并短轮询。
|
||||
- 输出只显示 Secret key 名、presence、fingerprint、连接 host、SSL 状态和状态摘要；禁止打印密码或完整 `DATABASE_URL`。
|
||||
- 同一个 PK01 PostgreSQL 实例可承载多个 YAML 声明的 role/database；新增消费者按 `secrets.entries`、`objects.roles`、`objects.databases`、`postgres.auth.pgHba` 和 `exports.connectionStrings` 成套声明，不新开 PostgreSQL 实例，也不默认用 schema 隔离应用状态。
|
||||
- 跨节点消费者必须直连 YAML 的 `postgres.network.connectionHost`，当前是 PK01 公网 endpoint；不要让 D601/G14/Sub2API/HWLAB/AgentRun 通过 master server 中转 PostgreSQL。
|
||||
- 当前 TLS 口径是 PostgreSQL native TLS + `sslmode=require`。`publicDns` 只是可选 alias；只要 `connectionHost` 是可达 IP，DNS 未解析不作为切库 blocker。
|
||||
- 远端 PostgreSQL 配置或 `pg_hba` 来源 CIDR 变化后，先跑 `apply --confirm --wait`，再跑 `status`；若消费者公网出口 IP 变化，必须先更新 YAML `allowSources` 和对应 `pg_hba`。
|
||||
- `status` 验收要看 `roles[]`、`databases[]` 和 `appConnections[]`；不要只看旧的 `roleExists` / `databaseExists` 标量。
|
||||
|
||||
日常复验建议：
|
||||
|
||||
|
||||
@@ -189,6 +189,7 @@ bun scripts/cli.ts platform-infra sub2api codex-pool configure-local --confirm
|
||||
- 从 `platform-infra/<apiKeySecretName>.<apiKeySecretKey>` 读取统一 API key。
|
||||
- 把当前 `~/.codex/config.toml` 和 `~/.codex/auth.json` 备份为 `.<backupSuffix>`，默认 `.pre-sub2api`。
|
||||
- 重写默认 `~/.codex` 消费端，固定指向 `https://sub2api.74-48-78-17.nip.io/`，provider 名称和 wire API 来自 `localCodex`。
|
||||
- 按 `localCodex.modelContextWindow` / `localCodex.modelAutoCompactTokenLimit` 写入 `model_context_window` / `model_auto_compact_token_limit`，用于统一控制 Codex auto compact 触发窗口，避免 GPT-5.5 消费端生成过大的 `/responses/compact` 长请求。
|
||||
- 按 `localCodex` 写入 Codex transport 标记：`supports_websockets` 与 `[features] responses_websockets_v2` 必须同开同关。只有至少一个上游通过 direct Codex WSv2 probe 时才启用；否则保持 HTTP Responses，避免每次原入口先经历无效 WS reconnect。
|
||||
- 用统一 key 做一次 gateway 验证。
|
||||
|
||||
@@ -209,6 +210,7 @@ bun scripts/cli.ts platform-infra sub2api codex-pool configure-local --confirm
|
||||
## 排障
|
||||
|
||||
- Codex pool 哨兵、账号冻结/恢复、marker-only 判断或 probe 周期看不清：第一步跑 `bun scripts/cli.ts platform-infra sub2api codex-pool sentinel-report`。这个报表是主观察面；只有报表缺字段或需要底层证据时，才继续看 `--raw`、CronJob log、state ConfigMap 或 Sub2API 管理 UI。若看到“临时不可调度状态”且包含规则序号/匹配关键词，检查 Sub2API `account_temp_unschedulable` 日志和账号 `temp_unschedulable_*` 字段；sentinel 只解释 `schedulable=false` 的 active quarantine，不解释这类内置临时冷却。
|
||||
- 只加强监控、不让哨兵自动冻结账号时，把 YAML `sentinel.actions.enabled=false` 后 `codex-pool sync --confirm`。此时 marker probe 和 gateway failure monitor 仍记录 `would-freeze` / observe-only 证据，但不会通过 Sub2API admin 写 `schedulable=false`；`/responses/compact` 的 `codex.remote_compact.failed` 和 compact 上游 5xx failover 只作为 `gateway-compact-*` 观察事件记录，不作为哨兵自动切换触发器。
|
||||
- 单个 request id 报 502/503/中断/没有自动切号：第一步跑 `bun scripts/cli.ts platform-infra sub2api codex-pool trace --request-id <requestId>`。先看 `outcome`、`reason`、`FAILOVER`、`SELECT-FAILED`、`ACCOUNT SIGNALS` 和 `WINDOW STATS`；只有 trace 报表缺字段或需要审计原始日志时，才加 `--show-lines` 或 `--raw`。若 `reason=failover-attempted-no-candidate`，说明切号动作已发生，但 scheduler 在排除失败账号后没有可用候选；继续用 `sentinel-report` 和 `validate --full` 区分 sentinel quarantine、request-path temp-unschedulable、账号 status 或容量耗尽。
|
||||
- profile invalid：先修 `~/.codex/config.toml.<profile>` 的 `base_url`、`wire_api`、`model` 或 `auth.json.<profile>` 的 API key；不要在 YAML 中写密钥。
|
||||
- Sub2API 卡在 `wait-postgres` / `wait-redis` 或服务内大量 `context deadline exceeded`：先跑 `sub2api status` 看 `networkPolicy.ok`，再跑 `sub2api validate` 看 `postgresCrossPodPgIsReady` / `redisCrossPodPing`；缺失或异常时用 `sub2api apply --confirm` 恢复受控 `NetworkPolicy/allow-all`，不要保留手工 iptables bypass 作为长期修复。
|
||||
|
||||
@@ -9,6 +9,8 @@ metadata:
|
||||
- 280
|
||||
- 281
|
||||
- 1119
|
||||
- 297
|
||||
- 300
|
||||
|
||||
cluster:
|
||||
role: primary
|
||||
@@ -86,6 +88,9 @@ postgres:
|
||||
- id: D601-public
|
||||
cidr: 36.49.29.73/32
|
||||
purpose: platform-infra-and-hwlab-v03-app
|
||||
- id: G14-public
|
||||
cidr: 202.98.17.68/32
|
||||
purpose: platform-infra-runtime
|
||||
tuning:
|
||||
maxConnections: 50
|
||||
sharedBuffers: 512MB
|
||||
@@ -141,6 +146,66 @@ postgres:
|
||||
user: hwlab_d601_v03_app
|
||||
address: 36.49.29.73/32
|
||||
method: scram-sha-256
|
||||
- type: hostssl
|
||||
database: langbot
|
||||
user: langbot
|
||||
address: 10.0.8.0/22
|
||||
method: scram-sha-256
|
||||
- type: hostssl
|
||||
database: postgres
|
||||
user: langbot
|
||||
address: 10.0.8.0/22
|
||||
method: scram-sha-256
|
||||
- type: hostssl
|
||||
database: langbot
|
||||
user: langbot
|
||||
address: 74.48.78.17/32
|
||||
method: scram-sha-256
|
||||
- type: hostssl
|
||||
database: postgres
|
||||
user: langbot
|
||||
address: 74.48.78.17/32
|
||||
method: scram-sha-256
|
||||
- type: hostssl
|
||||
database: langbot
|
||||
user: langbot
|
||||
address: 202.98.17.68/32
|
||||
method: scram-sha-256
|
||||
- type: hostssl
|
||||
database: postgres
|
||||
user: langbot
|
||||
address: 202.98.17.68/32
|
||||
method: scram-sha-256
|
||||
- type: hostssl
|
||||
database: n8n
|
||||
user: n8n
|
||||
address: 10.0.8.0/22
|
||||
method: scram-sha-256
|
||||
- type: hostssl
|
||||
database: postgres
|
||||
user: n8n
|
||||
address: 10.0.8.0/22
|
||||
method: scram-sha-256
|
||||
- type: hostssl
|
||||
database: n8n
|
||||
user: n8n
|
||||
address: 74.48.78.17/32
|
||||
method: scram-sha-256
|
||||
- type: hostssl
|
||||
database: postgres
|
||||
user: n8n
|
||||
address: 74.48.78.17/32
|
||||
method: scram-sha-256
|
||||
- type: hostssl
|
||||
database: n8n
|
||||
user: n8n
|
||||
address: 202.98.17.68/32
|
||||
method: scram-sha-256
|
||||
- type: hostssl
|
||||
database: postgres
|
||||
user: n8n
|
||||
address: 202.98.17.68/32
|
||||
method: scram-sha-256
|
||||
|
||||
secrets:
|
||||
source: master-local
|
||||
@@ -174,6 +239,34 @@ secrets:
|
||||
HWLAB_D601_V03_DB_NAME: hwlab_d601_v03
|
||||
randomHex:
|
||||
HWLAB_D601_V03_DB_PASSWORD: 32
|
||||
- name: langbot-db-credentials
|
||||
sourceRef: platform-db/langbot-db.env
|
||||
type: env
|
||||
requiredKeys:
|
||||
- LANGBOT_DB_USER
|
||||
- LANGBOT_DB_PASSWORD
|
||||
- LANGBOT_DB_NAME
|
||||
createIfMissing:
|
||||
enabled: true
|
||||
values:
|
||||
LANGBOT_DB_USER: langbot
|
||||
LANGBOT_DB_NAME: langbot
|
||||
randomHex:
|
||||
LANGBOT_DB_PASSWORD: 32
|
||||
- name: n8n-db-credentials
|
||||
sourceRef: platform-db/n8n-db.env
|
||||
type: env
|
||||
requiredKeys:
|
||||
- N8N_DB_USER
|
||||
- N8N_DB_PASSWORD
|
||||
- N8N_DB_NAME
|
||||
createIfMissing:
|
||||
enabled: true
|
||||
values:
|
||||
N8N_DB_USER: n8n
|
||||
N8N_DB_NAME: n8n
|
||||
randomHex:
|
||||
N8N_DB_PASSWORD: 32
|
||||
|
||||
objects:
|
||||
roles:
|
||||
@@ -195,6 +288,24 @@ objects:
|
||||
createdb: false
|
||||
createrole: false
|
||||
superuser: false
|
||||
- name: langbot
|
||||
passwordRef:
|
||||
sourceRef: platform-db/langbot-db.env
|
||||
key: LANGBOT_DB_PASSWORD
|
||||
login: true
|
||||
attributes:
|
||||
createdb: false
|
||||
createrole: false
|
||||
superuser: false
|
||||
- name: n8n
|
||||
passwordRef:
|
||||
sourceRef: platform-db/n8n-db.env
|
||||
key: N8N_DB_PASSWORD
|
||||
login: true
|
||||
attributes:
|
||||
createdb: false
|
||||
createrole: false
|
||||
superuser: false
|
||||
databases:
|
||||
- name: sub2api
|
||||
owner: sub2api
|
||||
@@ -206,6 +317,16 @@ objects:
|
||||
encoding: UTF8
|
||||
locale: C.UTF-8
|
||||
extensions: []
|
||||
- name: langbot
|
||||
owner: langbot
|
||||
encoding: UTF8
|
||||
locale: C.UTF-8
|
||||
extensions: []
|
||||
- name: n8n
|
||||
owner: n8n
|
||||
encoding: UTF8
|
||||
locale: C.UTF-8
|
||||
extensions: []
|
||||
|
||||
exports:
|
||||
connectionStrings:
|
||||
@@ -254,6 +375,36 @@ exports:
|
||||
- scope: hwlab-v03
|
||||
secret: hwlab-v03-openfga
|
||||
key: datastore-uri
|
||||
- name: langbot-database-url
|
||||
sourceSecretRef: platform-db/langbot-db.env
|
||||
render:
|
||||
envKey: DATABASE_URL
|
||||
format: postgresql://$(LANGBOT_DB_USER):$(LANGBOT_DB_PASSWORD)@$(PGHOST):5432/$(LANGBOT_DB_NAME)?sslmode=require
|
||||
variables:
|
||||
PGHOST: 82.156.23.220
|
||||
writeToSecretSource:
|
||||
sourceRef: platform-infra/langbot.env
|
||||
key: DATABASE_URL
|
||||
mode: update-or-insert
|
||||
consumers:
|
||||
- scope: platform-infra
|
||||
secret: langbot-secrets
|
||||
key: DATABASE_URL
|
||||
- name: n8n-database-url
|
||||
sourceSecretRef: platform-db/n8n-db.env
|
||||
render:
|
||||
envKey: DATABASE_URL
|
||||
format: postgresql://$(N8N_DB_USER):$(N8N_DB_PASSWORD)@$(PGHOST):5432/$(N8N_DB_NAME)?sslmode=require
|
||||
variables:
|
||||
PGHOST: 82.156.23.220
|
||||
writeToSecretSource:
|
||||
sourceRef: platform-infra/n8n.env
|
||||
key: DATABASE_URL
|
||||
mode: update-or-insert
|
||||
consumers:
|
||||
- scope: platform-infra
|
||||
secret: n8n-secrets
|
||||
key: DATABASE_URL
|
||||
|
||||
backup:
|
||||
phase: minimum-restoreable
|
||||
@@ -284,6 +435,15 @@ observability:
|
||||
- kind: psql-app-role
|
||||
database: sub2api
|
||||
user: sub2api
|
||||
- kind: psql-app-role
|
||||
database: hwlab_d601_v03
|
||||
user: hwlab_d601_v03_app
|
||||
- kind: psql-app-role
|
||||
database: langbot
|
||||
user: langbot
|
||||
- kind: psql-app-role
|
||||
database: n8n
|
||||
user: n8n
|
||||
- kind: disk-free
|
||||
path: /var/lib/postgresql/16/main
|
||||
minFreeGiB: 10
|
||||
|
||||
@@ -0,0 +1,98 @@
|
||||
version: 1
|
||||
kind: platform-infra-langbot
|
||||
|
||||
metadata:
|
||||
id: langbot-public
|
||||
owner: unidesk
|
||||
relatedIssues:
|
||||
- 297
|
||||
|
||||
image:
|
||||
repository: rockchin/langbot
|
||||
tag: v4.10.1
|
||||
pullPolicy: IfNotPresent
|
||||
|
||||
dependencyImages:
|
||||
postgresClient: docker.io/library/postgres:16-alpine
|
||||
|
||||
targets:
|
||||
- id: G14
|
||||
route: G14:k3s
|
||||
namespace: platform-infra
|
||||
enabled: true
|
||||
replicas: 1
|
||||
publicExposure:
|
||||
enabled: true
|
||||
publicBaseUrl: https://langbot.pikapython.com
|
||||
dns:
|
||||
hostname: langbot.pikapython.com
|
||||
expectedA: 82.156.23.220
|
||||
resolvers: [1.1.1.1, 8.8.8.8, 223.5.5.5, 114.114.114.114]
|
||||
frpc:
|
||||
deploymentName: langbot-frpc
|
||||
secretName: langbot-frpc-secrets
|
||||
secretKey: frpc.toml
|
||||
image: fatedier/frpc:v0.68.1
|
||||
serverAddr: 82.156.23.220
|
||||
serverPort: 22000
|
||||
proxyName: platform-infra-langbot-g14-web
|
||||
remotePort: 22097
|
||||
localIP: langbot.platform-infra.svc.cluster.local
|
||||
localPort: 5300
|
||||
tokenSourceRef: platform-infra/pk01-frp.env
|
||||
tokenSourceKey: FRP_TOKEN
|
||||
pk01:
|
||||
route: PK01
|
||||
caddyConfigPath: /etc/caddy/Caddyfile
|
||||
caddyServiceName: caddy
|
||||
responseHeaderTimeoutSeconds: 600
|
||||
|
||||
runtime:
|
||||
timezone: Asia/Shanghai
|
||||
database:
|
||||
mode: external
|
||||
sourceRef: platform-db/langbot-db.env
|
||||
sourceKeys:
|
||||
user: LANGBOT_DB_USER
|
||||
password: LANGBOT_DB_PASSWORD
|
||||
dbName: LANGBOT_DB_NAME
|
||||
secretName: langbot-secrets
|
||||
host: 82.156.23.220
|
||||
port: 5432
|
||||
user: langbot
|
||||
dbName: langbot
|
||||
sslMode: require
|
||||
appDatabaseValue: langbot?ssl=require
|
||||
secrets:
|
||||
root: /root/unidesk/.state/secrets
|
||||
appSourceRef: platform-infra/langbot.env
|
||||
storage:
|
||||
data:
|
||||
name: langbot-data
|
||||
size: 10Gi
|
||||
plugins:
|
||||
name: langbot-plugins
|
||||
size: 5Gi
|
||||
pluginRuntime:
|
||||
name: langbot-plugin-runtime-data
|
||||
size: 5Gi
|
||||
box:
|
||||
enabled: false
|
||||
reason: LangBot Box mounts Docker socket in the official manifest; public platform service keeps it disabled by default.
|
||||
|
||||
apiKey:
|
||||
sourceRef: platform-infra/langbot.env
|
||||
key: LANGBOT_API_KEY
|
||||
name: unidesk-langbot-cli
|
||||
description: UniDesk controlled CLI API key for langbot.pikapython.com
|
||||
|
||||
officialWechat:
|
||||
supportedAdapters:
|
||||
- officialaccount
|
||||
- wecom
|
||||
- wecomcs
|
||||
webhookBaseUrl: https://langbot.pikapython.com
|
||||
defaultCallbackPath: /callback/command
|
||||
notes:
|
||||
- Official account, WeCom, and WeCom customer service adapters are supported through LangBot official platform sources.
|
||||
- Personal WeChat/OpenClaw is not enabled by default for this public service.
|
||||
@@ -0,0 +1,83 @@
|
||||
version: 1
|
||||
kind: platform-infra-n8n
|
||||
|
||||
metadata:
|
||||
id: n8n-public
|
||||
owner: unidesk
|
||||
relatedIssues:
|
||||
- 300
|
||||
|
||||
image:
|
||||
repository: n8nio/n8n
|
||||
tag: "1.123.55"
|
||||
pullPolicy: IfNotPresent
|
||||
|
||||
dependencyImages:
|
||||
postgresClient: docker.io/library/postgres:16-alpine
|
||||
|
||||
targets:
|
||||
- id: G14
|
||||
route: G14:k3s
|
||||
namespace: platform-infra
|
||||
enabled: true
|
||||
replicas: 1
|
||||
publicExposure:
|
||||
enabled: true
|
||||
publicBaseUrl: https://n8n.pikapython.com
|
||||
dns:
|
||||
hostname: n8n.pikapython.com
|
||||
expectedA: 82.156.23.220
|
||||
resolvers: [1.1.1.1, 8.8.8.8, 223.5.5.5, 114.114.114.114]
|
||||
frpc:
|
||||
deploymentName: n8n-frpc
|
||||
secretName: n8n-frpc-secrets
|
||||
secretKey: frpc.toml
|
||||
image: fatedier/frpc:v0.68.1
|
||||
serverAddr: 82.156.23.220
|
||||
serverPort: 22000
|
||||
proxyName: platform-infra-n8n-g14-web
|
||||
remotePort: 22099
|
||||
localIP: n8n.platform-infra.svc.cluster.local
|
||||
localPort: 5678
|
||||
tokenSourceRef: platform-infra/pk01-frp.env
|
||||
tokenSourceKey: FRP_TOKEN
|
||||
pk01:
|
||||
route: PK01
|
||||
caddyConfigPath: /etc/caddy/Caddyfile
|
||||
caddyServiceName: caddy
|
||||
responseHeaderTimeoutSeconds: 600
|
||||
|
||||
runtime:
|
||||
timezone: Asia/Shanghai
|
||||
database:
|
||||
mode: external
|
||||
sourceRef: platform-db/n8n-db.env
|
||||
sourceKeys:
|
||||
user: N8N_DB_USER
|
||||
password: N8N_DB_PASSWORD
|
||||
dbName: N8N_DB_NAME
|
||||
secretName: n8n-secrets
|
||||
host: 82.156.23.220
|
||||
port: 5432
|
||||
user: n8n
|
||||
dbName: n8n
|
||||
sslMode: require
|
||||
sslRejectUnauthorized: false
|
||||
secrets:
|
||||
root: /root/unidesk/.state/secrets
|
||||
appSourceRef: platform-infra/n8n.env
|
||||
encryptionKey: N8N_ENCRYPTION_KEY
|
||||
storage:
|
||||
data:
|
||||
name: n8n-data
|
||||
size: 10Gi
|
||||
public:
|
||||
host: n8n.pikapython.com
|
||||
protocol: https
|
||||
editorBaseUrl: https://n8n.pikapython.com
|
||||
webhookUrl: https://n8n.pikapython.com
|
||||
proxyHops: 1
|
||||
security:
|
||||
diagnosticsEnabled: false
|
||||
personalisationEnabled: false
|
||||
secureCookie: true
|
||||
@@ -152,6 +152,8 @@ localCodex:
|
||||
backupSuffix: pre-sub2api
|
||||
providerName: OpenAI
|
||||
wireApi: responses
|
||||
modelContextWindow: 272000
|
||||
modelAutoCompactTokenLimit: 240000
|
||||
supportsWebSockets: false
|
||||
responsesWebSocketsV2: false
|
||||
responsesSmokeModel: gpt-5.5
|
||||
@@ -159,7 +161,7 @@ sentinel:
|
||||
monitor:
|
||||
enabled: true
|
||||
actions:
|
||||
enabled: true
|
||||
enabled: false
|
||||
schedule: "*/1 * * * *"
|
||||
image: python:3.12-alpine
|
||||
sdk:
|
||||
|
||||
@@ -73,7 +73,7 @@ CI/CD、GitOps、rollout、artifact 发布、PR 合并后的 runtime lane 滚动
|
||||
- `hwlab g14 git-mirror status|apply|sync|flush [--dry-run|--confirm]` 是 `devops-infra` git mirror/relay 的受控维护入口：`apply` 渲染并 server-side apply `devops-infra/git-mirror.yaml`，同时删除遗留 `git-mirror-hwlab-sync` CronJob；`sync` 创建一次性 manual Job，把 GitHub allowlist refs 拉入本地 mirror；`flush` 创建一次性 manual Job，把本地 `v0.2-gitops` 快进推回 GitHub。
|
||||
`status` 返回 read/write URL、last sync/write/flush、本地 ref、GitHub staging ref 和 pending flush 状态，并在 `cache.summary` 给出 `localV02`、`localGitops`、`githubGitops`、`pendingFlush`、`flushNeeded`、`githubInSync` 和下一条受控 `flushCommand`。confirmed `sync` 和 `flush` 默认创建 `.state/jobs/` 异步 job 并立刻返回可查询状态，只有现场同步调试才显式加 `--wait`；mirror 不设置 CronJob。
|
||||
如果 PipelineRun 的 `gitops-promote` 阶段报 git mirror 控制面漂移或 refs 不一致，先执行 `hwlab g14 git-mirror apply --confirm` 重新应用当前 `devops-infra/git-mirror.yaml` hook/ConfigMap，再执行 `hwlab g14 git-mirror sync --confirm --wait` 复核 refs；失败的同名 PipelineRun 只能通过 `hwlab g14 control-plane cleanup-runs --lane <lane> --pipeline-run <name> --confirm` 受控清理后重试，不要用原生 `kubectl delete` 或手工改 mirror hook。修复后仍必须用 `control-plane status --pipeline-run <name>` 和 `git-mirror status` 分别确认 runtime closeout 与 GitHub flush。
|
||||
- `platform-infra sub2api plan|apply|status|validate|codex-pool` 是 `platform-infra` namespace 内 Sub2API 的受控入口；`--target` 选择运行目标，默认 `G14` 为 active runtime，`D601` 为同一 YAML 控制的 standby predeploy target。镜像版本和 target 边界由 `config/platform-infra/sub2api.yaml` 控制，Codex 上游池、统一 API key Secret、FRP 公网端口和 master `~/.codex` 消费端由 `config/platform-infra/sub2api-codex-pool.yaml` 控制；完整日常部署、上游增删、FRP 暴露、local Codex 配置、验收和排障步骤统一见 `$unidesk-sub2api`（`.agents/skills/unidesk-sub2api/SKILL.md`）。`docs/reference/platform-infra.md` 只保留 namespace、YAML-first、路由、Secret 脱敏和探针开发边界。
|
||||
- `platform-infra sub2api|langbot|n8n ...` 是 `platform-infra` namespace 内平台公共服务的受控入口；`sub2api` 支持 `plan|apply|status|validate|codex-pool`，`langbot` 和 `n8n` 支持各自 YAML-controlled `plan|apply|status|logs|validate` 等公共服务操作。`--target` 选择运行目标，默认 `G14` 为 active runtime，`D601` 为同一 YAML 控制的 standby predeploy target。镜像版本和 target 边界由 `config/platform-infra/*.yaml` 控制，Codex 上游池、统一 API key Secret、FRP 公网端口和 master `~/.codex` 消费端由 `config/platform-infra/sub2api-codex-pool.yaml` 控制；完整 Sub2API 日常部署、上游增删、FRP 暴露、local Codex 配置、验收和排障步骤统一见 `$unidesk-sub2api`（`.agents/skills/unidesk-sub2api/SKILL.md`）。`docs/reference/platform-infra.md` 保留 namespace、YAML-first、路由、Secret 脱敏、PK01 Caddy+FRP 和探针开发边界。
|
||||
- `hwlab g14 observability status|apply|query|targets|boundary|closeout [--lane v02] [--promql <expr>] [--expect-count N] [--expect-value V] [--dry-run|--confirm]` 是 G14 `devops-infra` 共享监控基础设施和 HWLAB v0.2 监控 closeout 的受控入口。`apply` 固定安装 Prometheus Operator `v0.91.0`、Prometheus `v3.12.0`、Prometheus 发现 RBAC、`devops-infra` 内 Prometheus 实例和 ClusterIP query Service，并给被允许发现的 workload namespace 打低风险 label；它不把 Prometheus、Grafana 或 Alertmanager 部署到 `hwlab-v02`，也不接管 HWLAB runtime Deployment/Service。`status` 只读汇总 CRD、operator Deployment、Prometheus CR/pod/service、`hwlab-v02` ServiceMonitor/PrometheusRule 和 bounded `up` 查询；`query` 只通过 Kubernetes service proxy 查询 Prometheus，支持 `--expect-count` / `--expect-value` 输出 `assertion`、bad values 和 missing/extra series；`targets` 汇总 ServiceMonitor/PrometheusRule、metrics sidecar readiness/restart、三层指标值和 `metrics.k8s.io` 当前 CPU/内存资源快照；`boundary` 验证 workload namespace 没有 Prometheus/Alertmanager，并对 `19666/19667` 公网 `/metrics` 做负向验证；`closeout` 聚合平台 ready、scrape reachable、sidecar serving、business health probe、resource snapshot、namespace boundary 和 public metrics exposure 语义结论。长期边界见 `docs/reference/g14-observability-infra.md`。
|
||||
- `hwlab g14 tools-image status|build --name ci-node-tools --tag <tag> [--dockerfile deploy/ci/hwlab-ci-node-tools.Dockerfile] [--dry-run|--confirm]` 是 G14 固定 HWLAB CI tools image 的受控 host build/push 入口；构建和 push 只发生在 G14 host 与本地 registry，不在 master server 构建，也不把 `apk add`/runtime install 塞进 Tekton PipelineRun。
|
||||
- `trans gh:/owner/repo ...` 把 GitHub issue/PR 映射成只读/受控写入的虚拟文本目录，适合日报、PR 正文和 issue 正文的小补丁维护：`trans gh:/pikasTech/HWLAB ls` 展示 `pr/` 与 `issue/`，`trans gh:/pikasTech/HWLAB/pr ls [--limit N] [--full]` 和 `trans gh:/pikasTech/HWLAB/issue ls [--limit N] [--full]` 展示条目状态、楼层数、正文长度和标题，`trans gh:/pikasTech/HWLAB/pr/507 ls` 展示单个 PR 的一楼正文文件，`trans gh:/pikasTech/HWLAB/505/1 cat|rg|patch-apply` 兼容旧式 issue/PR number route。`patch-apply` 使用 UniDesk 默认 apply-patch v2 的虚拟文件 executor，把正文一楼映射为 `body.md`，写回仍走 `bun scripts/cli.ts gh issue/pr update` 的 guard/concurrency 规则；`rm` 对正文一楼结构化拒绝，避免误删 issue/PR 正文。大正文读取必须展开 UniDesk gh dump 文件，否则 `cat/rg/patch-apply` 会误读为空，这是 `gh:` 虚拟文件接口的 P0 可见性契约。
|
||||
|
||||
@@ -217,6 +217,7 @@ The G14 host persists this proxy configuration in these local files:
|
||||
- `/root/.gitconfig` pins root Git HTTP/HTTPS proxy settings.
|
||||
- `/root/.docker/config.json` pins Docker client proxy settings for commands and build contexts that honor Docker client proxy configuration.
|
||||
- `/etc/systemd/system/docker.service.d/proxy.conf` pins Docker daemon pull proxy settings. Updating this drop-in requires `systemctl daemon-reload` and a Docker restart before the active daemon sees the new `NO_PROXY`; do not restart Docker while G14 provider-gateway, k3s bootstrap or image builds are in flight unless that interruption is intentional.
|
||||
- `/etc/systemd/system/k3s.service.env` pins k3s/containerd image-pull proxy settings. If Pod events show image pulls failing through a stale loopback proxy such as `127.0.0.1:10808` while Hysteria is the active listener, update this file through `trans G14:/etc/systemd/system apply-patch`, run `systemctl daemon-reload`, restart k3s, then verify node readiness and the target image pull. Do not hand-import images as the long-term fix for a stale k3s proxy env.
|
||||
|
||||
The `NO_PROXY` list must include localhost, the main server, private LAN ranges, k3s pod/service CIDRs, Kubernetes service domains and the loopback registry so that k3s, `127.0.0.1:5000`, Kubernetes API access and UniDesk control paths do not route through the VPN proxy.
|
||||
|
||||
@@ -237,7 +238,7 @@ docker build --network host \
|
||||
|
||||
The backup proxy uses `HTTP_PROXY=http://127.0.0.1:11809`, `HTTPS_PROXY=http://127.0.0.1:11809` and `ALL_PROXY=socks5h://127.0.0.1:11808`.
|
||||
|
||||
This proxy is not a replacement for UniDesk runtime egress. k3s workloads such as Code Queue must still use the cataloged `g14-provider-egress-proxy` Kubernetes Service and `g14-tcp-egress-gateway` for normal runtime access to PostgreSQL, OA Event Flow and external APIs. The node-local VPN proxy is allowed only for G14 host-side bootstrap, image build, cache prewarm or recovery steps, and those steps should record the proxy choice in issue or deployment evidence.
|
||||
This proxy is not a replacement for UniDesk runtime egress. k3s workloads such as Code Queue must still use the cataloged `g14-provider-egress-proxy` Kubernetes Service and `g14-tcp-egress-gateway` for normal runtime access to PostgreSQL, OA Event Flow and external APIs. The node-local VPN proxy is allowed only for G14 host-side bootstrap, k3s image pulls, image build, cache prewarm or recovery steps, and those steps should record the proxy choice in issue or deployment evidence.
|
||||
|
||||
Runtime egress readiness is not proven by Service DNS alone. Before closing HWLAB, AgentRun, Code Queue, CI/CD or user-service issues that require GitHub/codeload/npm/pip/API access through `g14-provider-egress-proxy`, validate that the Deployment is Ready and the Service has at least one ready endpoint. If the Service resolves but `curl` through `http://g14-provider-egress-proxy.unidesk.svc.cluster.local:18789` fails immediately with connection refused, inspect `unidesk/g14-provider-egress-proxy` and `unidesk/g14-tcp-egress-gateway` pods for `ImagePullBackOff`, `ContainerStatusUnknown` or notReady endpoints. A pod trying to pull `unidesk-code-queue:g14` from Docker Hub as `docker.io/library/unidesk-code-queue:g14` is an invalid runtime egress deployment state; restore the controlled image source/import path instead of redirecting long-lived workload proxy env to the node-local bootstrap proxy.
|
||||
|
||||
|
||||
@@ -48,6 +48,14 @@ backend-core 必须提供 `/api/performance`，返回滚动窗口内的 HTTP 组
|
||||
|
||||
frontend Bun server 必须提供同源 `/api/frontend-performance`，记录 webui 静态资源、登录/session、API 代理和 frontend->core 代理操作耗时。浏览器中的 `运行总览 / 性能面板` 必须把 frontend 与 backend-core 指标合并展示为 Bwebui 曲线、组件汇总、最近失败请求、内部操作汇总和最近慢操作；完整性能 JSON 只能通过显式 `查看原始JSON` 打开。
|
||||
|
||||
## Node Resource Status
|
||||
|
||||
节点 CPU、内存、磁盘和进程资源指标的实时真相来自 provider-gateway 上报的 `system_status`，Docker 资源摘要来自 `docker_status`；backend-core 负责落库、历史采样和 `/api/nodes/system-status` 聚合。排查资源面板不同步时，必须同时对照 provider-gateway 日志里的 `system_status_sent` / `docker_status_sent`、backend-core 日志里的 `provider_system_status` / `provider_docker_status`、以及 backend-core 内部 API，而不能只看前端旧值或浏览器缓存。
|
||||
|
||||
provider 上报的 `collectedAt` 在 backend-core 入库前必须解析为 typed timestamp。Rust backend-core 使用 PostgreSQL 参数时不得把 RFC3339 字符串传给 `$n::timestamptz` 这类 SQL cast；`tokio-postgres` 会按目标 PostgreSQL 类型序列化参数，字符串参数可能在序列化阶段失败，导致 provider 已发送但数据库仍停在旧采样。等价实现应先解析为 `DateTime<Utc>` 或目标驱动支持的时间类型，再传入 SQL 参数。
|
||||
|
||||
`/api/nodes/system-status` 必须区分当前指标和最后已知指标。超过 backend stale window 的采样不得继续作为 `current` 冒充实时状态；接口和前端应暴露 `stale`、`currentCollectedAt` 和 `lastKnown`，并在指标过期时显示过期状态。运维验证优先使用 backend-core 内部只读入口：`docker exec unidesk-backend-core sh -lc 'backend-core --fetch-json http://127.0.0.1:8080/api/nodes/system-status?limit=5'`。外部 frontend 同源 API 可能受登录 session 保护，不应把未登录请求的 401 当成资源同步失败。
|
||||
|
||||
## Low-Memory Diagnostics
|
||||
|
||||
主 server 是低资源、低抖动控制面，排查内存时必须先区分共享内存、容器 cgroup 占用和进程私有占用。PostgreSQL 后端进程的 RSS 会重复显示 `shared_buffers` 等共享映射，不能把多个 `postgres` 进程 RSS 简单相加当成真实内存消耗；优先看 `docker stats unidesk-database`、cgroup memory、`/proc/<pid>/smaps_rollup` 的 PSS/USS、`pg_stat_activity` 连接数和 `pg_settings` 中的 `shared_buffers`/`work_mem`。
|
||||
|
||||
@@ -72,11 +72,15 @@ The public certificate depends on DNS. The `api.pikapython.com` record must reso
|
||||
|
||||
PK01 host-native PostgreSQL is declared by `config/platform-db/postgres-pk01.yaml` and managed through `bun scripts/cli.ts platform-db postgres plan|status|apply`; daily operation commands live in `$unidesk-ops` at `.agents/skills/unidesk-ops/SKILL.md`. It is a host systemd service, not a Docker container or k3s workload. The YAML is the source of truth for PostgreSQL version, TLS mode, listening addresses, `pg_hba` source CIDRs, generated Secret source files, exported `DATABASE_URL`, and backup timer settings.
|
||||
|
||||
The PK01 PostgreSQL cluster is a shared platform database instance. Multiple UniDesk services may be declared as separate `objects.roles[]` and `objects.databases[]` entries, with matching Secret sources, `pg_hba` hostssl rules and optional connection-string exports. A new platform consumer should use its own database and role inside this existing instance; do not create a second PostgreSQL instance or use one database with multiple schemas as the default isolation boundary for application state.
|
||||
|
||||
Cross-node platform consumers must connect directly to the YAML-declared `postgres.network.connectionHost`. For consumers outside the PK01 private VPC, that value must be PK01's public endpoint, not the private `10.0.8.3` address and not a master-server tunnel. The master server may run control-plane CLI operations and secret sync, but it must not become the data-plane relay for D601, G14, Sub2API, HWLAB, AgentRun, or other PostgreSQL clients.
|
||||
|
||||
`postgres.network.publicDns` is an optional alias for operator readability and future `sslmode=verify-full` work. With the current PostgreSQL native TLS posture, clients use `sslmode=require`; DNS resolution is therefore not a cutover blocker when `connectionHost` is a reachable IP endpoint. If `publicDns` later becomes the connection host or `verify-full` is enabled, certificate common name/SAN and DNS must be promoted back into the cutover criteria.
|
||||
|
||||
The exported Sub2API connection string is written under the configured ignored Secret root and must never be committed or printed in full. CLI output may show key names, presence, fingerprints, selected host, SSL status, and source/target Secret references only. If a consumer's public egress IP changes, update the YAML `allowSources` and matching `pg_hba` rules, then use the `$unidesk-ops` PK01 Host PostgreSQL workflow to apply and recheck status.
|
||||
Exported connection strings are written under the configured ignored Secret root and must never be committed or printed in full. CLI output may show key names, presence, fingerprints, selected host, SSL status, app connection status, and source/target Secret references only. If a consumer's public egress IP changes, update the YAML `allowSources` and matching `pg_hba` rules, then use the `$unidesk-ops` PK01 Host PostgreSQL workflow to apply and recheck status.
|
||||
|
||||
Backup coverage is explicit YAML state. Do not assume every declared database is backed up unless the backup configuration names it directly or the CLI reports multi-database coverage.
|
||||
|
||||
## Disk GC Policy
|
||||
|
||||
|
||||
@@ -22,6 +22,29 @@
|
||||
- External platform PostgreSQL endpoints for Sub2API are produced by the platform DB YAML and its `platform-db postgres` CLI. Cross-node Sub2API consumers connect directly to that endpoint; the master server is not a PostgreSQL data-plane relay. DNS aliases are optional when the exported `DATABASE_URL` uses a reachable IP with `sslmode=require`; current PK01-specific rules live in `docs/reference/pk01.md`.
|
||||
- Sub2API account sentinel and public exposure are target-scoped YAML decisions. Do not create a second sentinel, FRP client, public management surface, or edge proxy by hand; enable or move those resources only through the target YAML and the `platform-infra sub2api` / `codex-pool --target` CLI paths.
|
||||
|
||||
## LangBot Deployment Boundary
|
||||
|
||||
- LangBot is a UniDesk-operated public platform service in namespace `platform-infra`. The canonical entrypoint is `bun scripts/cli.ts platform-infra langbot plan|apply|status|logs|validate|bootstrap-api-key|query`; G14 is the default runtime target.
|
||||
- LangBot configuration is YAML-first in `config/platform-infra/langbot.yaml`. Image tag, target namespace, PVCs, PK01 Caddy/FRP exposure, API key seed source, and official WeChat adapter metadata must stay in YAML rather than helper constants or manual runtime patches.
|
||||
- LangBot uses the existing PK01 host-native PostgreSQL instance through `config/platform-db/postgres-pk01.yaml` and `platform-db postgres`. Adding LangBot state means adding a dedicated database and role inside that existing instance; do not deploy a second PostgreSQL StatefulSet, container, or external DB instance for LangBot.
|
||||
- Public exposure uses PK01 Caddy plus FRP to the G14 ClusterIP service. Do not add Kubernetes Ingress, NodePort, LoadBalancer, host networking, or host ports for LangBot unless a later YAML-controlled platform decision changes the exposure model.
|
||||
- LangBot's built-in Web frontend and API share the same public HTTPS origin. CLI queries must use the YAML-declared API key source and must report key names/fingerprints only, never the API key value.
|
||||
- `bootstrap-api-key` writes the YAML-declared key into LangBot's `api_keys` table after the app has initialized its schema. If the table is absent, start LangBot first and let its migrations run; do not create a parallel auth table or print the key while seeding it.
|
||||
- LangBot startup logs may include upstream env override values. `platform-infra langbot logs` must redact env keys containing `PASSWORD`, `SECRET`, `TOKEN`, `API_KEY`, or `DATABASE_URL`; any leaked DB password, JWT secret, or API key must be rotated through YAML/Secret sources and rolled out through the controlled `apply` path.
|
||||
- LangBot Secret material changes must update the app Deployment template with a Secret fingerprint annotation so `apply` rolls the Pod. Manual Pod deletion is only a temporary recovery action, not the durable rotation mechanism.
|
||||
- Closeout for public LangBot changes requires `platform-infra langbot status`, `platform-infra langbot validate`, and an API-key-backed `platform-infra langbot query`; frontend exposure is proved by the same public origin returning the built-in Web UI.
|
||||
- LangBot Box is disabled by default for the public service because the official Box deployment needs Docker socket access. Enabling Box requires a separate explicit platform decision and YAML-controlled security boundary.
|
||||
- Official WeChat support is through LangBot's official platform adapters such as `officialaccount`, `wecom`, and `wecomcs`; real AppID, token, EncodingAESKey and channel credentials are bound in LangBot after deployment. Personal WeChat or OpenClaw-style adapters are not part of the default public-service boundary.
|
||||
|
||||
## n8n Workflow Boundary
|
||||
|
||||
- n8n is the UniDesk-operated workflow/automation layer for LangBot and platform service integration. It is a workflow bridge for webhook orchestration, service calls, manual approval flows and external integrations; it does not replace LangBot or become the chat runtime.
|
||||
- The canonical entrypoint is `bun scripts/cli.ts platform-infra n8n plan|apply|status|logs|validate`; G14 is the default runtime target and `config/platform-infra/n8n.yaml` is the YAML source of truth.
|
||||
- n8n uses the existing Pika01/PK01 host-native PostgreSQL instance through `config/platform-db/postgres-pk01.yaml` and `platform-db postgres`. Adding n8n state means adding a dedicated `n8n` database and role inside that single external PostgreSQL instance; do not deploy an in-cluster PostgreSQL StatefulSet, a second PostgreSQL instance, or long-term SQLite state for n8n.
|
||||
- Public exposure uses PK01 Caddy plus FRP to the G14 ClusterIP service at `https://n8n.pikapython.com`. Do not add Kubernetes Ingress, NodePort, LoadBalancer, host networking, or host ports for n8n unless a later YAML-controlled platform decision changes the exposure model.
|
||||
- n8n reverse-proxy and webhook settings such as public base URL, `WEBHOOK_URL`, proxy hop trust and PostgreSQL connection fields must be rendered from YAML. Secret output may show key names, presence and fingerprints only; it must not print the database password, `N8N_ENCRYPTION_KEY`, or full `DATABASE_URL`.
|
||||
- Closeout for public n8n changes requires `platform-infra n8n status` and `platform-infra n8n validate --full`, proving both in-cluster HTTP and public HTTPS. Actual LangBot workflows, credentials and business automations are separate follow-up scope after the base n8n service is healthy.
|
||||
|
||||
## Codex Pool Routing
|
||||
|
||||
`config/platform-infra/sub2api-codex-pool.yaml` controls the Codex-facing OpenAI-compatible pool:
|
||||
|
||||
+11
-1
@@ -1,6 +1,6 @@
|
||||
import { readConfig } from "./src/config";
|
||||
import { debugDispatch, debugHealth, debugTask, isDebugDispatchCommand, type DebugDispatchCommand } from "./src/debug";
|
||||
import { isRebuildableService, rebuildService, stackLogs, stackStatus, startStack, stopStack, unsupportedRebuildService } from "./src/docker";
|
||||
import { isRebuildableService, rebuildService, restartService, stackLogs, stackStatus, startStack, stopStack, unsupportedRebuildService, unsupportedRestartService } from "./src/docker";
|
||||
import { emitError, emitJson, emitText, isRenderedCliResult } from "./src/output";
|
||||
import { cancelJob, jobWithTail, listJobs, listJobsSummary, readJob, runJob } from "./src/jobs";
|
||||
import { checkHelp, parseCheckOptions, runChecks, runRecoveryGuardrailsCheck } from "./src/check";
|
||||
@@ -439,6 +439,16 @@ async function main(): Promise<void> {
|
||||
emitJson(commandName, rebuildService(config, third));
|
||||
return;
|
||||
}
|
||||
if (sub === "restart") {
|
||||
if (!isRebuildableService(third)) {
|
||||
const result = unsupportedRestartService(third);
|
||||
emitJson(commandName, result, false);
|
||||
process.exitCode = 1;
|
||||
return;
|
||||
}
|
||||
emitJson(commandName, restartService(config, third));
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
if (top === "gc") {
|
||||
|
||||
@@ -100,6 +100,17 @@ export function unsupportedRebuildService(value: string | undefined): Record<str
|
||||
};
|
||||
}
|
||||
|
||||
export function unsupportedRestartService(value: string | undefined): Record<string, unknown> {
|
||||
const base = unsupportedRebuildService(value);
|
||||
return {
|
||||
...base,
|
||||
error: "unsupported-server-restart",
|
||||
reason: typeof base.reason === "string" ? base.reason.replace(/server rebuild/g, "server restart") : base.reason,
|
||||
replacement: typeof base.replacement === "string" ? base.replacement.replace(/server rebuild/g, "server restart") : base.replacement,
|
||||
policy: "server restart is a no-build maintenance entrypoint and must not silently mutate upstream images, D601 services, database, or unknown objects",
|
||||
};
|
||||
}
|
||||
|
||||
export function resolveComposeCommand(config: UniDeskConfig, envFile: string): string[] {
|
||||
const composeFile = rootPath(config.docker.composeFile);
|
||||
if (commandOk(["docker", "compose", "version"], repoRoot)) {
|
||||
@@ -431,6 +442,73 @@ export function rebuildService(config: UniDeskConfig, service: RebuildableServic
|
||||
};
|
||||
}
|
||||
|
||||
export function restartService(config: UniDeskConfig, service: RebuildableService): unknown {
|
||||
const runtimeEnv = writeComposeEnv(config, false);
|
||||
const compose = resolveComposeCommand(config, runtimeEnv.envFile);
|
||||
const listServiceContainersCommand = [
|
||||
"docker",
|
||||
"ps",
|
||||
"-q",
|
||||
"--filter",
|
||||
`label=com.docker.compose.project=${config.docker.projectName}`,
|
||||
"--filter",
|
||||
`label=com.docker.compose.service=${service}`,
|
||||
"--filter",
|
||||
"label=com.docker.compose.oneoff=False",
|
||||
];
|
||||
const upCommand = [...compose, "up", "-d", "--no-build", "--no-deps", "--force-recreate", service];
|
||||
const listAllServiceContainersCommand = [...listServiceContainersCommand];
|
||||
listAllServiceContainersCommand[2] = "-a";
|
||||
const validateScript = [
|
||||
"ready=0",
|
||||
"for attempt in $(seq 1 60); do",
|
||||
`cid=$(${shellJoin(listServiceContainersCommand)} || true)`,
|
||||
"if [ -n \"$cid\" ]; then",
|
||||
"health=$(docker inspect -f '{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}' $cid 2>/dev/null | head -1 || true)",
|
||||
`echo "service_container_probe service=${service} attempt=$attempt cid=$cid health=$health"`,
|
||||
"if [ \"$health\" = \"healthy\" ] || [ \"$health\" = \"running\" ]; then ready=1; break; fi",
|
||||
"else",
|
||||
`echo "service_container_probe service=${service} attempt=$attempt cid=missing"`,
|
||||
"fi",
|
||||
"sleep 1",
|
||||
"done",
|
||||
"if [ \"$ready\" != \"1\" ]; then",
|
||||
`echo "service_container_not_ready service=${service}" >&2`,
|
||||
`${shellJoin(listAllServiceContainersCommand)} --format '{{.ID}} {{.Names}} {{.Status}}' >&2 || true`,
|
||||
"exit 1",
|
||||
"fi",
|
||||
].join("\n");
|
||||
const script = [
|
||||
"set -euo pipefail",
|
||||
`echo ${shellJoin(["restart_service", service, "no_build_force_recreate_with_validation"])}`,
|
||||
restrictedHostAccessScript(config),
|
||||
shellJoin(upCommand),
|
||||
validateScript,
|
||||
].join("\n");
|
||||
const command = ["bash", "-lc", composeLockedScript(script)];
|
||||
const job = startJob("server_restart", command, `Restart and validate UniDesk ${service} without building images`);
|
||||
return {
|
||||
job,
|
||||
runtimeEnv,
|
||||
service,
|
||||
command,
|
||||
strategy: {
|
||||
buildBeforeReplace: false,
|
||||
noBuild: true,
|
||||
replaceScope: {
|
||||
projectLabel: config.docker.projectName,
|
||||
serviceLabel: service,
|
||||
},
|
||||
noDeps: true,
|
||||
forceRecreate: true,
|
||||
composeMutationLock: rootPath(".state", "locks", "server-compose.lock"),
|
||||
jobRunner: "local",
|
||||
postUpValidation: true,
|
||||
namedVolumesPreserved: true,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
function fixedPorts(config: UniDeskConfig): Array<{ name: string; port: number; listening: boolean }> {
|
||||
return [
|
||||
{ name: "frontend", port: config.network.frontend.port, listening: isPortListening(config.network.frontend.port) },
|
||||
|
||||
+15
-4
@@ -19,6 +19,7 @@ export function rootHelp(): unknown {
|
||||
{ command: "server cleanup plan [--min-age-hours N] [--limit N]", description: "Dry-run Docker image cleanup plan only: list active/protected images, stale candidates older than the default 24h threshold, risk, estimated reclaim, and manual review commands without deleting anything." },
|
||||
{ command: "gc plan|run|db-trace|policy|remote [--confirm] [--logs-keep-days N] [--include-browser-cache]", description: "One-time main-server or remote provider disk relief and low-risk anti-bloat policy for logs, journald, allowlisted /tmp artifacts, scoped core dumps and explicit trace telemetry retention; plan is read-only and run requires --confirm." },
|
||||
{ command: "server rebuild <backend-core|frontend|dev-frontend-proxy|provider-gateway|todo-note|code-queue-mgr|project-manager|baidu-netdisk|oa-event-flow>", description: "Maintenance-only local Compose rebuild for reviewed main-server services; frontend standard release must use CI artifact plus deploy apply dev/prod artifact consumers." },
|
||||
{ command: "server restart <backend-core|frontend|dev-frontend-proxy|provider-gateway|todo-note|code-queue-mgr|project-manager|baidu-netdisk|oa-event-flow>", description: "No-build single-service Compose restart for reviewed main-server maintenance recovery; returns an async job and validates the recreated container." },
|
||||
{ command: "provider attach <providerId> [--master-server URL] [--up] [--force] | provider triage <providerId> [--observed-error text] [--observed-scope scope] [--microservice id ...] [--full|--raw]", description: "Generate the minimal external provider-gateway env/compose bundle or run the low-noise read-only provider health triage contract." },
|
||||
{ command: "trans <route> [operation args...] (alias of ssh <route> ...)", description: "Open a Host SSH / WSL SSH maintenance session; provider WebSocket carries control and host.ssh.tcp-pool carries stdin/stdout/stderr data." },
|
||||
{ command: "trans gh:/owner/repo[/pr|/issue][/number[/1]] ls|cat|rg|patch-apply", description: "Treat GitHub PRs/issues as virtual text directories; `ls --full` shows state/floors/body length, and `patch-apply` updates first-floor `body.md` through UniDesk gh plus apply-patch v2." },
|
||||
@@ -59,7 +60,7 @@ export function rootHelp(): unknown {
|
||||
{ command: "hwlab nodes control-plane|git-mirror|secret --node <node> --lane <lane>", description: "Manage HWLAB node/lane runtime prerequisites, including D601 YAML-declared infra/tools-image/Argo bootstrap and G14 v0.3+ runtime lanes, with the node identity passed as data." },
|
||||
{ command: "hwlab g14 monitor-prs | hwlab g14 control-plane status|apply|trigger-current|runtime-migration|cleanup-runs|cleanup-released-pvs | hwlab g14 git-mirror status|apply|sync|flush | hwlab g14 tools-image status|build", description: "Start the legacy G14 PR monitor, run bounded v0.2 Tekton/Argo control-plane, manual PipelineRun trigger, runtime migration, CI workspace retention, manual devops-infra git mirror/relay maintenance, or fixed HWLAB CI tools image actions; long confirmed trigger/sync/flush actions return async jobs by default." },
|
||||
{ command: "agentrun get|describe|events|logs|result|ack|cancel|dispatch|create|apply|send|control-plane|git-mirror", description: "Use AgentRun v0.1 resource primitives with low-noise human output by default; session follow-up uses send only and the server decides internal steer vs turn." },
|
||||
{ command: "platform-infra sub2api plan|apply|status|validate|codex-pool", description: "Deploy Sub2API in G14 platform-infra, manage the YAML-controlled Codex upstream pool, expose the unified API, and inspect marker sentinel state with low-noise reports without printing API keys." },
|
||||
{ command: "platform-infra sub2api|langbot|n8n ...", description: "Deploy platform-infra services such as Sub2API, LangBot and n8n, manage YAML-controlled public FRP/Caddy exposure, and inspect status/logs without printing secrets." },
|
||||
{ command: "platform-db postgres plan|status|apply", description: "Manage YAML-declared host-native PostgreSQL 16 on PK01 for Sub2API/platform state, with PostgreSQL native TLS and redacted secret exports." },
|
||||
{ command: "hwlab cd audit --env dev | hwlab cd status --env dev | hwlab cd apply --env dev --dry-run", description: "Legacy D601 HWLAB DEV CD wrapper kept for explicit old-path diagnostics; current HWLAB rollout uses G14 GitOps." },
|
||||
{ command: "code-agent-sandbox", description: "Independent Code Agent Sandbox service skeleton for adapter, mode, and credential-boundary diagnostics." },
|
||||
@@ -101,7 +102,7 @@ export function isHelpToken(value: string | undefined): boolean {
|
||||
|
||||
export function serverHelp(action: string | undefined = undefined): unknown {
|
||||
return {
|
||||
command: action === undefined || isHelpToken(action) ? "server start|stop|status|swap|logs|cleanup|rebuild" : `server ${action}`,
|
||||
command: action === undefined || isHelpToken(action) ? "server start|stop|status|swap|logs|cleanup|rebuild|restart" : `server ${action}`,
|
||||
output: "json",
|
||||
description: "Manage the fixed main-server Docker Compose stack without exposing backend-core REST publicly.",
|
||||
usage: {
|
||||
@@ -112,6 +113,7 @@ export function serverHelp(action: string | undefined = undefined): unknown {
|
||||
logs: "bun scripts/cli.ts server logs [--tail-bytes N]",
|
||||
cleanup: "bun scripts/cli.ts server cleanup plan [--min-age-hours N] [--limit N]",
|
||||
rebuild: "bun scripts/cli.ts server rebuild <backend-core|frontend|dev-frontend-proxy|provider-gateway|todo-note|code-queue-mgr|project-manager|baidu-netdisk|oa-event-flow>",
|
||||
restart: "bun scripts/cli.ts server restart <backend-core|frontend|dev-frontend-proxy|provider-gateway|todo-note|code-queue-mgr|project-manager|baidu-netdisk|oa-event-flow>",
|
||||
},
|
||||
cleanupPlan: {
|
||||
dryRunOnly: true,
|
||||
@@ -138,6 +140,7 @@ export function serverHelp(action: string | undefined = undefined): unknown {
|
||||
maintenanceOnly: {
|
||||
frontend: "server rebuild frontend is not standard release evidence; publish 127.0.0.1:5000/unidesk/frontend:<commit> with CI, then consume it with deploy apply --env dev/prod --service frontend --commit <full-sha>",
|
||||
userServices: "server rebuild for main-server user services is reserved for local maintenance, bootstrap or recovery and must not replace commit-pinned artifact CD",
|
||||
restart: "server restart is for no-build single-service recovery when the existing image is already present and the goal is to refresh runtime state",
|
||||
},
|
||||
nonRebuildableBoundary: {
|
||||
upstreamImages: ["filebrowser", "filebrowser-d601"],
|
||||
@@ -602,7 +605,7 @@ function agentRunHelpSummary(): unknown {
|
||||
|
||||
function platformInfraHelpSummary(): unknown {
|
||||
return {
|
||||
command: "platform-infra sub2api plan|apply|status|validate|codex-pool",
|
||||
command: "platform-infra sub2api|langbot|n8n ...",
|
||||
output: "json",
|
||||
usage: [
|
||||
"bun scripts/cli.ts platform-infra sub2api plan",
|
||||
@@ -612,8 +615,16 @@ function platformInfraHelpSummary(): unknown {
|
||||
"bun scripts/cli.ts platform-infra sub2api codex-pool sentinel-image status",
|
||||
"bun scripts/cli.ts platform-infra sub2api codex-pool sentinel-probe --account unidesk-codex-hy --confirm",
|
||||
"bun scripts/cli.ts platform-infra sub2api codex-pool sentinel-report",
|
||||
"bun scripts/cli.ts platform-infra langbot plan",
|
||||
"bun scripts/cli.ts platform-infra langbot apply --confirm",
|
||||
"bun scripts/cli.ts platform-infra langbot status",
|
||||
"bun scripts/cli.ts platform-infra langbot query --path /api/v1/platform/bots",
|
||||
"bun scripts/cli.ts platform-infra n8n plan",
|
||||
"bun scripts/cli.ts platform-infra n8n apply --confirm",
|
||||
"bun scripts/cli.ts platform-infra n8n status",
|
||||
"bun scripts/cli.ts platform-infra n8n validate --full",
|
||||
],
|
||||
description: "Operate G14 platform-infra services such as Sub2API and the YAML-controlled Codex pool.",
|
||||
description: "Operate G14 platform-infra services such as Sub2API, LangBot, n8n, and the YAML-controlled Codex pool.",
|
||||
};
|
||||
}
|
||||
|
||||
|
||||
+191
-117
@@ -242,8 +242,8 @@ interface RemoteFacts {
|
||||
logDirExists: boolean;
|
||||
roleExists: boolean;
|
||||
databaseExists: boolean;
|
||||
roleExistsByName?: Record<string, boolean>;
|
||||
databaseExistsByName?: Record<string, boolean>;
|
||||
roles: Array<{ name: string; exists: boolean }>;
|
||||
databases: Array<{ name: string; owner: string; exists: boolean }>;
|
||||
serverVersion: string | null;
|
||||
sslOn: boolean;
|
||||
sslCertFile: string | null;
|
||||
@@ -252,6 +252,14 @@ interface RemoteFacts {
|
||||
appConnectionSsl: boolean | null;
|
||||
appConnectionHost: string | null;
|
||||
appConnectionError: string | null;
|
||||
appConnections: Array<{
|
||||
user: string;
|
||||
database: string;
|
||||
ok: boolean | null;
|
||||
ssl: boolean | null;
|
||||
host: string | null;
|
||||
error: string | null;
|
||||
}>;
|
||||
};
|
||||
raw?: Record<string, unknown>;
|
||||
}
|
||||
@@ -260,6 +268,8 @@ interface ControllerConnectionProbe {
|
||||
attempted: boolean;
|
||||
ok: boolean | null;
|
||||
ssl: boolean | null;
|
||||
user: string | null;
|
||||
database: string | null;
|
||||
host: string;
|
||||
port: number;
|
||||
clientAddr: string | null;
|
||||
@@ -371,7 +381,7 @@ async function plan(config: UniDeskConfig, options: PlatformDbOptions): Promise<
|
||||
next: {
|
||||
apply: `bun scripts/cli.ts platform-db postgres apply --config ${pg.configPath} --confirm`,
|
||||
status: `bun scripts/cli.ts platform-db postgres status --config ${pg.configPath}`,
|
||||
issue: "https://github.com/pikasTech/unidesk/issues/281",
|
||||
issues: pg.metadata.relatedIssues.map((issue) => `https://github.com/pikasTech/unidesk/issues/${issue}`),
|
||||
},
|
||||
remote: compactCapture(remote.capture, { full: options.full || remote.capture.exitCode !== 0 }),
|
||||
...(options.raw ? { raw: remote.capture } : {}),
|
||||
@@ -385,17 +395,18 @@ async function status(config: UniDeskConfig, options: PlatformDbOptions): Promis
|
||||
const facts = remote.parsed;
|
||||
const controllerConnection = controllerConnectionProbe(pg, secrets);
|
||||
const endpointHealthy = controllerConnection.ok === true && controllerConnection.ssl === true;
|
||||
const rolesReady = facts !== null && pg.objects.roles.every((role, index) => facts.postgres.roleExistsByName?.[role.name] ?? (index === 0 ? facts.postgres.roleExists : false));
|
||||
const databasesReady = facts !== null && pg.objects.databases.every((database, index) => facts.postgres.databaseExistsByName?.[database.name] ?? (index === 0 ? facts.postgres.databaseExists : false));
|
||||
const rolesHealthy = facts !== null && pg.objects.roles.every((role) => facts.postgres.roles.some((item) => item.name === role.name && item.exists));
|
||||
const databasesHealthy = facts !== null && pg.objects.databases.every((database) => facts.postgres.databases.some((item) => item.name === database.name && item.exists));
|
||||
const appConnectionsHealthy = facts !== null
|
||||
&& appProbes(pg, secrets).every((probe) => facts.postgres.appConnections.some((item) => item.user === probe.user && item.database === probe.database && item.ok === true && item.ssl === true));
|
||||
const deploymentHealthy = facts !== null
|
||||
&& facts.postgres.packageInstalled
|
||||
&& facts.postgres.serviceActive
|
||||
&& facts.postgres.sslOn
|
||||
&& rolesReady
|
||||
&& databasesReady
|
||||
&& rolesHealthy
|
||||
&& databasesHealthy
|
||||
&& facts.network.port5432Listening
|
||||
&& facts.postgres.appConnectionOk === true
|
||||
&& facts.postgres.appConnectionSsl === true;
|
||||
&& appConnectionsHealthy;
|
||||
const cutoverReady = deploymentHealthy && secrets.ok && endpointHealthy;
|
||||
const cutoverBlockers = cutoverReady
|
||||
? []
|
||||
@@ -425,14 +436,13 @@ async function status(config: UniDeskConfig, options: PlatformDbOptions): Promis
|
||||
dnsAddresses: facts.network.dns.addresses,
|
||||
roleExists: facts.postgres.roleExists,
|
||||
databaseExists: facts.postgres.databaseExists,
|
||||
roleExistsByName: facts.postgres.roleExistsByName ?? {},
|
||||
databaseExistsByName: facts.postgres.databaseExistsByName ?? {},
|
||||
rolesReady,
|
||||
databasesReady,
|
||||
roles: facts.postgres.roles,
|
||||
databases: facts.postgres.databases,
|
||||
appConnectionOk: facts.postgres.appConnectionOk,
|
||||
appConnectionSsl: facts.postgres.appConnectionSsl,
|
||||
appConnectionHost: facts.postgres.appConnectionHost,
|
||||
appConnectionError: facts.postgres.appConnectionError,
|
||||
appConnections: facts.postgres.appConnections,
|
||||
connectionHostProbe: controllerConnection,
|
||||
port5432Listening: facts.network.port5432Listening,
|
||||
serverVersion: facts.postgres.serverVersion,
|
||||
@@ -556,7 +566,18 @@ function readPostgresHostConfig(pathArg: string): PostgresHostConfig {
|
||||
const entries = arrayOfRecords(secrets.entries, `${configPath}.secrets.entries`).map((entry, index) => secretEntry(entry, `${configPath}.secrets.entries[${index}]`));
|
||||
const roles = arrayOfRecords(objects.roles, `${configPath}.objects.roles`).map((role, index) => roleConfig(role, `${configPath}.objects.roles[${index}]`));
|
||||
const databases = arrayOfRecords(objects.databases, `${configPath}.objects.databases`).map((database, index) => databaseConfig(database, `${configPath}.objects.databases[${index}]`));
|
||||
if (roles.length === 0 || databases.length === 0) throw new Error(`${configPath}.objects.roles/databases must each contain at least one item`);
|
||||
if (roles.length === 0 || databases.length === 0) throw new Error(`${configPath}.objects must declare at least one role and one database`);
|
||||
const roleNames = new Set<string>();
|
||||
for (const role of roles) {
|
||||
if (roleNames.has(role.name)) throw new Error(`${configPath}.objects.roles contains duplicate role ${role.name}`);
|
||||
roleNames.add(role.name);
|
||||
}
|
||||
const databaseNames = new Set<string>();
|
||||
for (const database of databases) {
|
||||
if (databaseNames.has(database.name)) throw new Error(`${configPath}.objects.databases contains duplicate database ${database.name}`);
|
||||
if (!roleNames.has(database.owner)) throw new Error(`${configPath}.objects.databases.${database.name}.owner must reference a declared role`);
|
||||
databaseNames.add(database.name);
|
||||
}
|
||||
const connectionStrings = arrayOfRecords(exportsConfig.connectionStrings, `${configPath}.exports.connectionStrings`).map((item, index) => connectionStringExport(item, `${configPath}.exports.connectionStrings[${index}]`));
|
||||
return {
|
||||
configPath: relativeConfigPath(configPath),
|
||||
@@ -925,6 +946,10 @@ function configSummary(pg: PostgresHostConfig): Record<string, unknown> {
|
||||
futureClientSslmode: pg.postgres.network.tls.futureClientSslmode,
|
||||
},
|
||||
haPhase: pg.cluster.haPhase,
|
||||
objects: {
|
||||
roles: pg.objects.roles.map((role) => ({ name: role.name, login: role.login })),
|
||||
databases: pg.objects.databases.map((database) => ({ name: database.name, owner: database.owner })),
|
||||
},
|
||||
noK3s: true,
|
||||
};
|
||||
}
|
||||
@@ -987,7 +1012,7 @@ function inspectSecrets(pg: PostgresHostConfig, materialize: boolean): SecretIns
|
||||
function ensureLocalSecretState(pg: PostgresHostConfig): { inspection: SecretInspection; summary: Record<string, unknown>; exports: Array<Record<string, unknown>> } {
|
||||
const inspection = inspectSecrets(pg, true);
|
||||
if (!inspection.ok) throw new Error("required secret source keys are missing and cannot be generated from YAML");
|
||||
validateSub2ApiSecretConsistency(pg, inspection);
|
||||
validateObjectSecretConsistency(pg, inspection);
|
||||
const exports = pg.exports.connectionStrings.map((item) => writeConnectionStringExport(pg, inspection, item));
|
||||
return {
|
||||
inspection,
|
||||
@@ -1002,13 +1027,17 @@ function ensureLocalSecretState(pg: PostgresHostConfig): { inspection: SecretIns
|
||||
};
|
||||
}
|
||||
|
||||
function validateSub2ApiSecretConsistency(pg: PostgresHostConfig, inspection: SecretInspection): void {
|
||||
const role = pg.objects.roles[0];
|
||||
const database = pg.objects.databases[0];
|
||||
const material = inspection.materials.get(role.passwordRef.sourceRef);
|
||||
if (material === undefined) throw new Error(`secret source not found for role passwordRef: ${role.passwordRef.sourceRef}`);
|
||||
if (material.values.SUB2API_DB_USER !== undefined && material.values.SUB2API_DB_USER !== role.name) throw new Error("SUB2API_DB_USER must match YAML objects.roles[0].name");
|
||||
if (material.values.SUB2API_DB_NAME !== undefined && material.values.SUB2API_DB_NAME !== database.name) throw new Error("SUB2API_DB_NAME must match YAML objects.databases[0].name");
|
||||
function validateObjectSecretConsistency(pg: PostgresHostConfig, inspection: SecretInspection): void {
|
||||
const roleNames = new Set(pg.objects.roles.map((role) => role.name));
|
||||
for (const role of pg.objects.roles) {
|
||||
const material = inspection.materials.get(role.passwordRef.sourceRef);
|
||||
if (material === undefined) throw new Error(`secret source not found for role passwordRef: ${role.passwordRef.sourceRef}`);
|
||||
const password = material.values[role.passwordRef.key];
|
||||
if (password === undefined || password.length === 0) throw new Error(`secret source ${role.passwordRef.sourceRef} is missing ${role.passwordRef.key}`);
|
||||
}
|
||||
for (const database of pg.objects.databases) {
|
||||
if (!roleNames.has(database.owner)) throw new Error(`database ${database.name} owner ${database.owner} is not declared in objects.roles`);
|
||||
}
|
||||
}
|
||||
|
||||
function writeConnectionStringExport(pg: PostgresHostConfig, inspection: SecretInspection, item: ConnectionStringExportConfig): Record<string, unknown> {
|
||||
@@ -1106,8 +1135,10 @@ function secretSummary(secrets: SecretInspection): Record<string, unknown> {
|
||||
}
|
||||
|
||||
function desiredSummary(pg: PostgresHostConfig, secrets: SecretInspection, facts: RemoteFacts | null): Record<string, unknown> {
|
||||
const roleExistsByName = facts?.postgres.roleExistsByName ?? {};
|
||||
const databaseExistsByName = facts?.postgres.databaseExistsByName ?? {};
|
||||
const role = pg.objects.roles[0];
|
||||
const database = pg.objects.databases[0];
|
||||
const observedRoles = facts?.postgres.roles ?? [];
|
||||
const observedDatabases = facts?.postgres.databases ?? [];
|
||||
return {
|
||||
package: {
|
||||
name: `postgresql-${pg.postgres.package.version}`,
|
||||
@@ -1144,14 +1175,17 @@ function desiredSummary(pg: PostgresHostConfig, secrets: SecretInspection, facts
|
||||
},
|
||||
pgHbaManagedBlock: { start: managedHbaStart, end: managedHbaEnd, rules: pg.postgres.auth.pgHba.length },
|
||||
pgHbaRemoteTransport: "hostssl",
|
||||
roles: pg.objects.roles.map((role, index) => ({
|
||||
name: role.name,
|
||||
action: (facts === null ? false : roleExistsByName[role.name] ?? (index === 0 ? facts.postgres.roleExists : false)) ? "none" : "create-or-update",
|
||||
role: { name: role.name, action: facts?.postgres.roleExists ? "none" : "create-or-update" },
|
||||
database: { name: database.name, owner: database.owner, action: facts?.postgres.databaseExists ? "none" : "create" },
|
||||
roles: pg.objects.roles.map((item) => ({
|
||||
name: item.name,
|
||||
login: item.login,
|
||||
action: observedRoles.some((observed) => observed.name === item.name && observed.exists) ? "none" : "create-or-update",
|
||||
})),
|
||||
databases: pg.objects.databases.map((database, index) => ({
|
||||
name: database.name,
|
||||
owner: database.owner,
|
||||
action: (facts === null ? false : databaseExistsByName[database.name] ?? (index === 0 ? facts.postgres.databaseExists : false)) ? "none" : "create",
|
||||
databases: pg.objects.databases.map((item) => ({
|
||||
name: item.name,
|
||||
owner: item.owner,
|
||||
action: observedDatabases.some((observed) => observed.name === item.name && observed.exists) ? "none" : "create",
|
||||
})),
|
||||
},
|
||||
secrets: secretSummary(secrets),
|
||||
@@ -1188,29 +1222,34 @@ function desiredChecks(pg: PostgresHostConfig, facts: RemoteFacts, secrets: Secr
|
||||
}
|
||||
|
||||
async function remoteFacts(config: UniDeskConfig, pg: PostgresHostConfig, secrets: SecretInspection | null): Promise<{ capture: SshCaptureResult; parsed: RemoteFacts | null }> {
|
||||
const capture = await runSshCommandCapture(config, pg.node.route, ["script"], factsScript(pg, appProbe(pg, secrets)));
|
||||
const capture = await runSshCommandCapture(config, pg.node.route, ["script"], factsScript(pg, appProbes(pg, secrets)));
|
||||
const parsed = parseJsonOutput(capture.stdout) as RemoteFacts | null;
|
||||
return { capture, parsed };
|
||||
}
|
||||
|
||||
function appProbe(pg: PostgresHostConfig, secrets: SecretInspection | null): { user: string; password: string; database: string } | null {
|
||||
if (secrets === null || !secrets.ok) return null;
|
||||
const role = pg.objects.roles[0];
|
||||
const database = pg.objects.databases[0];
|
||||
const material = secrets.materials.get(role.passwordRef.sourceRef);
|
||||
const password = material?.values[role.passwordRef.key];
|
||||
if (password === undefined || password.length === 0) return null;
|
||||
return { user: role.name, password, database: database.name };
|
||||
function appProbes(pg: PostgresHostConfig, secrets: SecretInspection | null): Array<{ user: string; password: string; database: string }> {
|
||||
if (secrets === null || !secrets.ok) return [];
|
||||
const rolesByName = new Map(pg.objects.roles.map((role) => [role.name, role]));
|
||||
const probes: Array<{ user: string; password: string; database: string }> = [];
|
||||
for (const database of pg.objects.databases) {
|
||||
const role = rolesByName.get(database.owner);
|
||||
if (role === undefined) continue;
|
||||
const material = secrets.materials.get(role.passwordRef.sourceRef);
|
||||
const password = material?.values[role.passwordRef.key];
|
||||
if (password === undefined || password.length === 0) continue;
|
||||
probes.push({ user: role.name, password, database: database.name });
|
||||
}
|
||||
return probes;
|
||||
}
|
||||
|
||||
function controllerConnectionProbe(pg: PostgresHostConfig, secrets: SecretInspection): ControllerConnectionProbe {
|
||||
const host = pg.postgres.network.connectionHost;
|
||||
const port = pg.postgres.network.port;
|
||||
const base = { host, port, clientAddr: null, serverAddr: null };
|
||||
const base = { host, port, user: null, database: null, clientAddr: null, serverAddr: null };
|
||||
if (!secrets.ok) {
|
||||
return { ...base, attempted: false, ok: null, ssl: null, psqlVersion: null, error: "secrets-unhealthy" };
|
||||
}
|
||||
const probe = appProbe(pg, secrets);
|
||||
const probe = appProbes(pg, secrets)[0] ?? null;
|
||||
if (probe === null) {
|
||||
return { ...base, attempted: false, ok: null, ssl: null, psqlVersion: null, error: "probe-secret-material-unavailable" };
|
||||
}
|
||||
@@ -1237,6 +1276,8 @@ function controllerConnectionProbe(pg: PostgresHostConfig, secrets: SecretInspec
|
||||
const ssl = ["t", "true", "1", "on"].includes(sslText) ? true : ["f", "false", "0", "off"].includes(sslText) ? false : null;
|
||||
return {
|
||||
...base,
|
||||
user: probe.user,
|
||||
database: probe.database,
|
||||
attempted: true,
|
||||
ok: result.status === 0 && ssl === true,
|
||||
ssl,
|
||||
@@ -1247,25 +1288,34 @@ function controllerConnectionProbe(pg: PostgresHostConfig, secrets: SecretInspec
|
||||
};
|
||||
}
|
||||
|
||||
function factsScript(pg: PostgresHostConfig, probe: { user: string; password: string; database: string } | null): string {
|
||||
function factsScript(pg: PostgresHostConfig, probes: Array<{ user: string; password: string; database: string }>): string {
|
||||
const release = releaseUrl(pg);
|
||||
const dataDir = pg.postgres.paths.dataDir;
|
||||
const configDir = pg.postgres.paths.configDir;
|
||||
const logDir = pg.postgres.paths.logDir;
|
||||
const role = pg.objects.roles[0].name;
|
||||
const database = pg.objects.databases[0].name;
|
||||
const roleSqlList = pg.objects.roles.map((item) => `'${item.name.replace(/'/gu, "''")}'`).join(",");
|
||||
const databaseSqlList = pg.objects.databases.map((item) => `'${item.name.replace(/'/gu, "''")}'`).join(",");
|
||||
const roleNames = pg.objects.roles.map((role) => role.name);
|
||||
const databaseItems = pg.objects.databases.map((database) => ({ name: database.name, owner: database.owner }));
|
||||
const roleListSql = sqlStringList(roleNames);
|
||||
const databaseListSql = sqlStringList(databaseItems.map((database) => database.name));
|
||||
const rolesJsonB64 = Buffer.from(JSON.stringify(roleNames), "utf8").toString("base64");
|
||||
const databasesJsonB64 = Buffer.from(JSON.stringify(databaseItems), "utf8").toString("base64");
|
||||
const dnsHost = pg.postgres.network.publicDns;
|
||||
const appHost = pg.postgres.network.listenAddresses.find((item) => item !== "127.0.0.1" && item !== "0.0.0.0") ?? "127.0.0.1";
|
||||
const probeEnv = probe === null
|
||||
? ""
|
||||
: `APP_PROBE_USER=${shellQuote(probe.user)}\nAPP_PROBE_DATABASE=${shellQuote(probe.database)}\nAPP_PROBE_PASSWORD=${shellQuote(probe.password)}\nAPP_PROBE_HOST=${shellQuote(appHost)}\nAPP_PROBE_PORT=${shellQuote(String(pg.postgres.network.port))}\n`;
|
||||
const probeCommands = probes.map((probe, index) => {
|
||||
const encoded = Buffer.from(JSON.stringify({ user: probe.user, database: probe.database }), "utf8").toString("base64");
|
||||
return [
|
||||
`printf '%s' '${encoded}' | base64 -d >"$tmp/appProbe.${index}.json"`,
|
||||
`PGPASSWORD=${shellQuote(probe.password)} psql "host=${appHost} port=${pg.postgres.network.port} user=${probe.user} dbname=${probe.database} sslmode=require" -Atqc "SELECT ssl FROM pg_stat_ssl WHERE pid = pg_backend_pid();" >"$tmp/appSsl.${index}" 2>"$tmp/appConnErr.${index}"`,
|
||||
`printf '%s' "$?" >"$tmp/appConnRc.${index}"`,
|
||||
`printf '%s' ${shellQuote(appHost)} >"$tmp/appConnHost.${index}"`,
|
||||
].join("\n");
|
||||
}).join("\n");
|
||||
return `
|
||||
set +e
|
||||
tmp="$(mktemp -d)"
|
||||
trap 'rm -rf "$tmp"' EXIT
|
||||
${probeEnv}
|
||||
printf '%s' '${rolesJsonB64}' | base64 -d >"$tmp/expectedRoles.json"
|
||||
printf '%s' '${databasesJsonB64}' | base64 -d >"$tmp/expectedDatabases.json"
|
||||
date -Is >"$tmp/observedAt" 2>/dev/null
|
||||
hostname >"$tmp/hostname" 2>/dev/null
|
||||
if [ -r /etc/os-release ]; then . /etc/os-release; printf '%s' "$PRETTY_NAME" >"$tmp/osPrettyName"; printf '%s' "$VERSION_CODENAME" >"$tmp/osCodename"; fi
|
||||
@@ -1305,16 +1355,13 @@ if command -v runuser >/dev/null 2>&1 && command -v psql >/dev/null 2>&1 && id p
|
||||
$root_prefix runuser -u postgres -- psql -Atqc "SHOW ssl;" >"$tmp/sslOn" 2>/dev/null
|
||||
$root_prefix runuser -u postgres -- psql -Atqc "SHOW ssl_cert_file;" >"$tmp/sslCertFile" 2>/dev/null
|
||||
$root_prefix runuser -u postgres -- psql -Atqc "SHOW ssl_key_file;" >"$tmp/sslKeyFile" 2>/dev/null
|
||||
$root_prefix runuser -u postgres -- psql -Atqc "SELECT 1 FROM pg_roles WHERE rolname='${role}'" >"$tmp/roleExists" 2>/dev/null
|
||||
$root_prefix runuser -u postgres -- psql -Atqc "SELECT 1 FROM pg_database WHERE datname='${database}'" >"$tmp/databaseExists" 2>/dev/null
|
||||
$root_prefix runuser -u postgres -- psql -Atqc "SELECT rolname FROM pg_roles WHERE rolname IN (${roleSqlList})" >"$tmp/rolesExisting" 2>/dev/null
|
||||
$root_prefix runuser -u postgres -- psql -Atqc "SELECT datname FROM pg_database WHERE datname IN (${databaseSqlList})" >"$tmp/databasesExisting" 2>/dev/null
|
||||
$root_prefix runuser -u postgres -- psql -Atqc "SELECT rolname FROM pg_roles WHERE rolname IN (${roleListSql}) ORDER BY rolname;" >"$tmp/rolesExisting" 2>/dev/null
|
||||
$root_prefix runuser -u postgres -- psql -Atqc "SELECT datname FROM pg_database WHERE datname IN (${databaseListSql}) ORDER BY datname;" >"$tmp/databasesExisting" 2>/dev/null
|
||||
fi
|
||||
fi
|
||||
if [ -n "\${APP_PROBE_PASSWORD:-}" ] && command -v psql >/dev/null 2>&1; then
|
||||
PGPASSWORD="$APP_PROBE_PASSWORD" psql "host=$APP_PROBE_HOST port=$APP_PROBE_PORT user=$APP_PROBE_USER dbname=$APP_PROBE_DATABASE sslmode=require" -Atqc "SELECT ssl FROM pg_stat_ssl WHERE pid = pg_backend_pid();" >"$tmp/appSsl" 2>"$tmp/appConnErr"
|
||||
printf '%s' "$?" >"$tmp/appConnRc"
|
||||
printf '%s' "$APP_PROBE_HOST" >"$tmp/appConnHost"
|
||||
if command -v psql >/dev/null 2>&1; then
|
||||
:
|
||||
${probeCommands}
|
||||
fi
|
||||
python3 - "$tmp" "${release}" <<'PY'
|
||||
import json, os, sys
|
||||
@@ -1332,12 +1379,43 @@ def int_or_none(name):
|
||||
return None
|
||||
def yes_file(name, expected="0"):
|
||||
return text(name) == expected
|
||||
def set_lines(name):
|
||||
def load_json(name, fallback):
|
||||
try:
|
||||
return json.load(open(os.path.join(tmp, name), encoding="utf-8"))
|
||||
except Exception:
|
||||
return fallback
|
||||
def set_from_lines(name):
|
||||
return {line.strip() for line in text(name).splitlines() if line.strip()}
|
||||
expected_roles = ${JSON.stringify(pg.objects.roles.map((item) => item.name))}
|
||||
expected_databases = ${JSON.stringify(pg.objects.databases.map((item) => item.name))}
|
||||
existing_roles = set_lines("rolesExisting")
|
||||
existing_databases = set_lines("databasesExisting")
|
||||
expected_roles = load_json("expectedRoles.json", [])
|
||||
expected_databases = load_json("expectedDatabases.json", [])
|
||||
existing_roles = set_from_lines("rolesExisting")
|
||||
existing_databases = set_from_lines("databasesExisting")
|
||||
roles = [{"name": name, "exists": name in existing_roles} for name in expected_roles]
|
||||
databases = [{"name": item["name"], "owner": item["owner"], "exists": item["name"] in existing_databases} for item in expected_databases]
|
||||
app_connections = []
|
||||
index = 0
|
||||
while True:
|
||||
meta = load_json(f"appProbe.{index}.json", None)
|
||||
if meta is None:
|
||||
break
|
||||
ssl_raw = text(f"appSsl.{index}").lower()
|
||||
if ssl_raw in {"t", "true", "1", "on"}:
|
||||
ssl = True
|
||||
elif ssl_raw in {"f", "false", "0", "off"}:
|
||||
ssl = False
|
||||
else:
|
||||
ssl = None
|
||||
rc_raw = text(f"appConnRc.{index}")
|
||||
app_connections.append({
|
||||
"user": meta.get("user"),
|
||||
"database": meta.get("database"),
|
||||
"ok": None if rc_raw == "" else rc_raw == "0",
|
||||
"ssl": ssl,
|
||||
"host": text(f"appConnHost.{index}") or None,
|
||||
"error": text(f"appConnErr.{index}") or None,
|
||||
})
|
||||
index += 1
|
||||
primary_conn = app_connections[0] if app_connections else {}
|
||||
release_rc = text("releaserc")
|
||||
payload = {
|
||||
"ok": True,
|
||||
@@ -1377,18 +1455,19 @@ payload = {
|
||||
"dataDirExists": yes_file("dataDirRc"),
|
||||
"configDirExists": yes_file("configDirRc"),
|
||||
"logDirExists": yes_file("logDirRc"),
|
||||
"roleExists": text("roleExists") == "1",
|
||||
"databaseExists": text("databaseExists") == "1",
|
||||
"roleExistsByName": {name: name in existing_roles for name in expected_roles},
|
||||
"databaseExistsByName": {name: name in existing_databases for name in expected_databases},
|
||||
"roleExists": bool(roles and roles[0]["exists"]),
|
||||
"databaseExists": bool(databases and databases[0]["exists"]),
|
||||
"roles": roles,
|
||||
"databases": databases,
|
||||
"serverVersion": text("serverVersion") or None,
|
||||
"sslOn": text("sslOn").lower() in {"on", "true", "1"},
|
||||
"sslCertFile": text("sslCertFile") or None,
|
||||
"sslKeyFile": text("sslKeyFile") or None,
|
||||
"appConnectionOk": None if text("appConnRc") == "" else text("appConnRc") == "0",
|
||||
"appConnectionSsl": None if text("appSsl") == "" else text("appSsl").lower() in {"t", "true", "1", "on"},
|
||||
"appConnectionHost": text("appConnHost") or None,
|
||||
"appConnectionError": text("appConnErr") or None,
|
||||
"appConnectionOk": primary_conn.get("ok"),
|
||||
"appConnectionSsl": primary_conn.get("ssl"),
|
||||
"appConnectionHost": primary_conn.get("host"),
|
||||
"appConnectionError": primary_conn.get("error"),
|
||||
"appConnections": app_connections,
|
||||
},
|
||||
}
|
||||
print(json.dumps(payload, ensure_ascii=False, indent=2))
|
||||
@@ -1416,18 +1495,16 @@ async function startRemoteApplyJob(config: UniDeskConfig, pg: PostgresHostConfig
|
||||
}
|
||||
|
||||
function remoteApplyPayload(pg: PostgresHostConfig, secrets: SecretInspection): Record<string, unknown> {
|
||||
const role = pg.objects.roles[0];
|
||||
const database = pg.objects.databases[0];
|
||||
const roles = pg.objects.roles.map((item) => {
|
||||
const material = secrets.materials.get(item.passwordRef.sourceRef);
|
||||
if (material === undefined) throw new Error(`missing material for ${item.passwordRef.sourceRef}`);
|
||||
const password = material.values[item.passwordRef.key];
|
||||
if (password === undefined || password.length === 0) throw new Error(`missing ${item.passwordRef.key}`);
|
||||
const roles = pg.objects.roles.map((role) => {
|
||||
const material = secrets.materials.get(role.passwordRef.sourceRef);
|
||||
if (material === undefined) throw new Error(`missing material for ${role.passwordRef.sourceRef}`);
|
||||
const password = material.values[role.passwordRef.key];
|
||||
if (password === undefined || password.length === 0) throw new Error(`missing ${role.passwordRef.key}`);
|
||||
return {
|
||||
name: item.name,
|
||||
name: role.name,
|
||||
password,
|
||||
login: item.login,
|
||||
attributes: item.attributes,
|
||||
login: role.login,
|
||||
attributes: role.attributes,
|
||||
};
|
||||
});
|
||||
return {
|
||||
@@ -1442,15 +1519,10 @@ function remoteApplyPayload(pg: PostgresHostConfig, secrets: SecretInspection):
|
||||
tuning: pg.postgres.tuning,
|
||||
pgHba: pg.postgres.auth.pgHba,
|
||||
passwordEncryption: pg.postgres.auth.passwordEncryption,
|
||||
role: {
|
||||
name: role.name,
|
||||
password: roles[0].password,
|
||||
login: role.login,
|
||||
attributes: role.attributes,
|
||||
},
|
||||
roles,
|
||||
database,
|
||||
databases: pg.objects.databases,
|
||||
role: roles[0],
|
||||
database: pg.objects.databases[0],
|
||||
backup: pg.backup.logicalDump,
|
||||
hbaMarkers: { start: managedHbaStart, end: managedHbaEnd },
|
||||
};
|
||||
@@ -1598,7 +1670,7 @@ if name == "pg_hba":
|
||||
print(f'{rule["type"]} {rule["database"]} {rule["user"]} {rule["address"]} {rule["method"]}')
|
||||
print(markers["end"])
|
||||
elif name == "sql":
|
||||
for role in data.get("roles", [data["role"]]):
|
||||
for role in data["roles"]:
|
||||
name = role["name"]
|
||||
password = role["password"].replace("'", "''")
|
||||
attrs = []
|
||||
@@ -1616,23 +1688,29 @@ elif name == "sql":
|
||||
print(" END IF;")
|
||||
print("END")
|
||||
print("$unidesk$;")
|
||||
elif name == "databases":
|
||||
for db in data.get("databases", [data["database"]]):
|
||||
print("\t".join([db["name"], db["owner"], db["encoding"], db["locale"]]))
|
||||
elif name == "databases_tsv":
|
||||
for database in data["databases"]:
|
||||
print("\t".join([database["name"], database["owner"], database["encoding"], database["locale"]]))
|
||||
elif name == "backup_script":
|
||||
backup = data["backup"]
|
||||
role = data["role"]["name"]
|
||||
db = data["database"]["name"]
|
||||
db = backup["database"]
|
||||
database = next((item for item in data["databases"] if item["name"] == db), None)
|
||||
if database is None:
|
||||
raise SystemExit(f"backup database {db} is not declared")
|
||||
role = next((item for item in data["roles"] if item["name"] == database["owner"]), None)
|
||||
if role is None:
|
||||
raise SystemExit(f"backup owner {database['owner']} has no declared role")
|
||||
role_name = role["name"]
|
||||
port = data["network"]["port"]
|
||||
dest = backup["destination"]["path"]
|
||||
retention = backup["retentionDays"]
|
||||
password = data["role"]["password"].replace("'", "'\"'\"'")
|
||||
password = role["password"].replace("'", "'\"'\"'")
|
||||
print("#!/bin/sh")
|
||||
print("set -eu")
|
||||
print(f"dest='{dest}'")
|
||||
print("mkdir -p \"$dest\"")
|
||||
print("ts=$(date +%Y%m%dT%H%M%S)")
|
||||
print(f"PGPASSWORD='{password}' /usr/lib/postgresql/{data['pgVersion']}/bin/pg_dump -h 127.0.0.1 -p {port} -U {role} -d {db} --format=custom --file \"$dest/{db}-$ts.dump\"")
|
||||
print(f"PGPASSWORD='{password}' /usr/lib/postgresql/{data['pgVersion']}/bin/pg_dump -h 127.0.0.1 -p {port} -U {role_name} -d {db} --format=custom --file \"$dest/{db}-$ts.dump\"")
|
||||
print(f"find \"$dest\" -type f -name '{db}-*.dump' -mtime +{retention} -delete")
|
||||
PY
|
||||
}
|
||||
@@ -1655,10 +1733,6 @@ CONFIG_DIR="$(json_get paths.configDir)"
|
||||
DATA_DIR="$(json_get paths.dataDir)"
|
||||
BACKUP_ENABLED="$(json_get backup.enabled)"
|
||||
BACKUP_PATH="$(json_get backup.destination.path)"
|
||||
DB_NAME="$(json_get database.name)"
|
||||
DB_OWNER="$(json_get database.owner)"
|
||||
DB_ENCODING="$(json_get database.encoding)"
|
||||
DB_LOCALE="$(json_get database.locale)"
|
||||
TLS_ENABLED="$(json_get tls.enabled)"
|
||||
TLS_COMMON_NAME="$(json_get tls.commonName)"
|
||||
TLS_CERT_FILE="$(json_get tls.certFile)"
|
||||
@@ -1748,13 +1822,13 @@ sql="$job_dir/role.sql"
|
||||
render_file sql > "$sql"
|
||||
runuser -u postgres -- psql -v ON_ERROR_STOP=1 < "$sql"
|
||||
|
||||
write_state running create-database
|
||||
write_state running create-databases
|
||||
databases_tsv="$job_dir/databases.tsv"
|
||||
render_file databases > "$databases_tsv"
|
||||
while IFS=' ' read -r db_name db_owner db_encoding db_locale; do
|
||||
[ -n "$db_name" ] || continue
|
||||
if ! runuser -u postgres -- psql -Atqc "SELECT 1 FROM pg_database WHERE datname='$db_name'" | grep -q 1; then
|
||||
runuser -u postgres -- createdb -O "$db_owner" -E "$db_encoding" --locale="$db_locale" --template=template0 "$db_name"
|
||||
render_file databases_tsv > "$databases_tsv"
|
||||
while IFS="$(printf '\t')" read -r DB_NAME DB_OWNER DB_ENCODING DB_LOCALE; do
|
||||
[ -n "$DB_NAME" ] || continue
|
||||
if ! runuser -u postgres -- psql -Atqc "SELECT 1 FROM pg_database WHERE datname='$DB_NAME'" | grep -q 1; then
|
||||
runuser -u postgres -- createdb -O "$DB_OWNER" -E "$DB_ENCODING" --locale="$DB_LOCALE" --template=template0 "$DB_NAME"
|
||||
fi
|
||||
done < "$databases_tsv"
|
||||
|
||||
@@ -1789,16 +1863,11 @@ fi
|
||||
|
||||
write_state running final-check
|
||||
runuser -u postgres -- psql -Atqc "SHOW server_version;" >/dev/null
|
||||
python3 - "$payload" <<'PY' > "$job_dir/final-check.sql"
|
||||
import json, sys
|
||||
data=json.load(open(sys.argv[1], encoding="utf-8"))
|
||||
for role in data.get("roles", [data["role"]]):
|
||||
print("SELECT 'role', %r, EXISTS (SELECT 1 FROM pg_roles WHERE rolname = %r);" % (role["name"], role["name"]))
|
||||
for db in data.get("databases", [data["database"]]):
|
||||
print("SELECT 'database', %r, EXISTS (SELECT 1 FROM pg_database WHERE datname = %r);" % (db["name"], db["name"]))
|
||||
PY
|
||||
chmod 0644 "$job_dir/final-check.sql"
|
||||
runuser -u postgres -- psql -Atq < "$job_dir/final-check.sql" | awk -F'|' '{ if ($3 != "t") exit 1 }'
|
||||
while IFS="$(printf '\t')" read -r DB_NAME DB_OWNER DB_ENCODING DB_LOCALE; do
|
||||
[ -n "$DB_NAME" ] || continue
|
||||
runuser -u postgres -- psql -Atqc "SELECT 1 FROM pg_roles WHERE rolname='$DB_OWNER'" | grep -q 1
|
||||
runuser -u postgres -- psql -Atqc "SELECT 1 FROM pg_database WHERE datname='$DB_NAME'" | grep -q 1
|
||||
done < "$databases_tsv"
|
||||
write_state succeeded complete 0
|
||||
trap - EXIT
|
||||
exit 0
|
||||
@@ -1897,6 +1966,11 @@ function shellQuote(value: string): string {
|
||||
return `'${value.replaceAll("'", "'\"'\"'")}'`;
|
||||
}
|
||||
|
||||
function sqlStringList(values: string[]): string {
|
||||
if (values.length === 0) return "''";
|
||||
return values.map((value) => `'${value.replaceAll("'", "''")}'`).join(", ");
|
||||
}
|
||||
|
||||
function parseJsonOutput(stdout: string): Record<string, unknown> | null {
|
||||
const trimmed = stdout.trim();
|
||||
if (trimmed.length === 0) return null;
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,372 @@
|
||||
import { createHash } from "node:crypto";
|
||||
import { Buffer } from "node:buffer";
|
||||
import type { UniDeskConfig } from "./config";
|
||||
import { runSshCommandCapture, type SshCaptureResult } from "./ssh";
|
||||
|
||||
export interface PublicServiceExposure {
|
||||
enabled: boolean;
|
||||
publicBaseUrl: string;
|
||||
dns: { hostname: string; expectedA: string; resolvers: string[] };
|
||||
frpc: {
|
||||
deploymentName: string;
|
||||
secretName: string;
|
||||
secretKey: string;
|
||||
image: string;
|
||||
serverAddr: string;
|
||||
serverPort: number;
|
||||
proxyName: string;
|
||||
remotePort: number;
|
||||
localIP: string;
|
||||
localPort: number;
|
||||
tokenSourceRef: string;
|
||||
tokenSourceKey: string;
|
||||
};
|
||||
pk01: {
|
||||
route: string;
|
||||
caddyConfigPath: string;
|
||||
caddyServiceName: string;
|
||||
responseHeaderTimeoutSeconds: number;
|
||||
};
|
||||
}
|
||||
|
||||
export interface PublicServiceTarget {
|
||||
id: string;
|
||||
route: string;
|
||||
namespace: string;
|
||||
replicas: number;
|
||||
publicExposure: PublicServiceExposure;
|
||||
}
|
||||
|
||||
export interface FrpcSecretMaterial {
|
||||
sourceRef: string;
|
||||
sourcePath: string;
|
||||
secretName: string;
|
||||
secretKey: string;
|
||||
frpcToml: string;
|
||||
fingerprint: string;
|
||||
valuesPrinted: false;
|
||||
}
|
||||
|
||||
export async function capture(config: UniDeskConfig, route: string, args: string[], stdin: string): Promise<SshCaptureResult> {
|
||||
return await runSshCommandCapture(config, route, args, stdin);
|
||||
}
|
||||
|
||||
export async function applyPk01CaddyBlock(
|
||||
config: UniDeskConfig,
|
||||
serviceId: string,
|
||||
exposure: PublicServiceExposure,
|
||||
markers: { start: string; end: string },
|
||||
): Promise<Record<string, unknown>> {
|
||||
if (!exposure.enabled) return { ok: true, action: "not-enabled" };
|
||||
const block = `${markers.start}
|
||||
${exposure.dns.hostname} {
|
||||
reverse_proxy 127.0.0.1:${exposure.frpc.remotePort} {
|
||||
transport http {
|
||||
response_header_timeout ${exposure.pk01.responseHeaderTimeoutSeconds}s
|
||||
}
|
||||
}
|
||||
}
|
||||
${markers.end}
|
||||
`;
|
||||
const blockB64 = Buffer.from(block, "utf8").toString("base64");
|
||||
const script = `
|
||||
set -u
|
||||
tmp="$(mktemp -d)"
|
||||
trap 'rm -rf "$tmp"' EXIT
|
||||
block="$tmp/${serviceId}.caddy"
|
||||
printf '%s' '${blockB64}' | base64 -d >"$block"
|
||||
sudo python3 - ${shQuote(exposure.pk01.caddyConfigPath)} "$block" ${shQuote(markers.start)} ${shQuote(markers.end)} >"$tmp/update.out" 2>"$tmp/update.err" <<'PY'
|
||||
import pathlib
|
||||
import sys
|
||||
|
||||
config_path = pathlib.Path(sys.argv[1])
|
||||
block_path = pathlib.Path(sys.argv[2])
|
||||
start = sys.argv[3]
|
||||
end = sys.argv[4]
|
||||
text = config_path.read_text(encoding="utf-8") if config_path.exists() else ""
|
||||
block = block_path.read_text(encoding="utf-8").strip() + "\\n"
|
||||
if start in text and end in text:
|
||||
before = text.split(start, 1)[0].rstrip()
|
||||
tail = text.split(end, 1)[1].lstrip()
|
||||
next_text = before + "\\n\\n" + block + "\\n" + tail
|
||||
action = "update"
|
||||
else:
|
||||
next_text = text.rstrip() + "\\n\\n" + block
|
||||
action = "append"
|
||||
tmp = config_path.with_suffix(config_path.suffix + ".unidesk-${serviceId}.tmp")
|
||||
tmp.write_text(next_text, encoding="utf-8")
|
||||
tmp.replace(config_path)
|
||||
print(action)
|
||||
PY
|
||||
update_rc=$?
|
||||
if [ "$update_rc" -eq 0 ]; then
|
||||
sudo caddy fmt --overwrite ${shQuote(exposure.pk01.caddyConfigPath)} >"$tmp/fmt.out" 2>"$tmp/fmt.err"
|
||||
fmt_rc=$?
|
||||
else
|
||||
: >"$tmp/fmt.out"; : >"$tmp/fmt.err"; fmt_rc=1
|
||||
fi
|
||||
if [ "$fmt_rc" -eq 0 ]; then
|
||||
sudo caddy validate --config ${shQuote(exposure.pk01.caddyConfigPath)} >"$tmp/validate.out" 2>"$tmp/validate.err"
|
||||
validate_rc=$?
|
||||
else
|
||||
: >"$tmp/validate.out"; : >"$tmp/validate.err"; validate_rc=1
|
||||
fi
|
||||
if [ "$validate_rc" -eq 0 ]; then
|
||||
sudo systemctl reload ${shQuote(exposure.pk01.caddyServiceName)} >"$tmp/reload.out" 2>"$tmp/reload.err" || sudo systemctl restart ${shQuote(exposure.pk01.caddyServiceName)} >>"$tmp/reload.out" 2>>"$tmp/reload.err"
|
||||
reload_rc=$?
|
||||
else
|
||||
: >"$tmp/reload.out"; : >"$tmp/reload.err"; reload_rc=1
|
||||
fi
|
||||
python3 - "$update_rc" "$fmt_rc" "$validate_rc" "$reload_rc" "$tmp/update.out" "$tmp/update.err" "$tmp/fmt.out" "$tmp/fmt.err" "$tmp/validate.out" "$tmp/validate.err" "$tmp/reload.out" "$tmp/reload.err" <<'PY'
|
||||
import json, sys
|
||||
rcs = [int(value) for value in sys.argv[1:5]]
|
||||
def text(path):
|
||||
try:
|
||||
return open(path, encoding="utf-8", errors="replace").read()[-3000:]
|
||||
except FileNotFoundError:
|
||||
return ""
|
||||
payload = {
|
||||
"ok": all(rc == 0 for rc in rcs),
|
||||
"serviceId": "${serviceId}",
|
||||
"hostname": "${exposure.dns.hostname}",
|
||||
"remotePort": ${exposure.frpc.remotePort},
|
||||
"caddyConfigPath": "${exposure.pk01.caddyConfigPath}",
|
||||
"service": "${exposure.pk01.caddyServiceName}",
|
||||
"managedBlock": {"start": "${markers.start}", "end": "${markers.end}"},
|
||||
"steps": {
|
||||
"update": {"exitCode": rcs[0], "stdout": text(sys.argv[5]), "stderr": text(sys.argv[6])},
|
||||
"fmt": {"exitCode": rcs[1], "stdout": text(sys.argv[7]), "stderr": text(sys.argv[8])},
|
||||
"validate": {"exitCode": rcs[2], "stdout": text(sys.argv[9]), "stderr": text(sys.argv[10])},
|
||||
"reload": {"exitCode": rcs[3], "stdout": text(sys.argv[11]), "stderr": text(sys.argv[12])},
|
||||
},
|
||||
}
|
||||
print(json.dumps(payload, ensure_ascii=False, indent=2))
|
||||
sys.exit(0 if payload["ok"] else 1)
|
||||
PY
|
||||
`;
|
||||
const result = await capture(config, exposure.pk01.route, ["script"], script);
|
||||
const parsed = parseJsonOutput(result.stdout);
|
||||
return parsed ?? { ok: false, capture: compactCapture(result, { full: true }) };
|
||||
}
|
||||
|
||||
export function prepareFrpcSecret(params: {
|
||||
secretRoot: string;
|
||||
exposure: PublicServiceExposure;
|
||||
sourcePathRedactor: (path: string) => string;
|
||||
parseEnvFile: (text: string) => Record<string, string>;
|
||||
requiredEnvValue: (values: Record<string, string>, key: string, sourceRef: string) => string;
|
||||
readTextFile: (path: string) => string;
|
||||
}): FrpcSecretMaterial {
|
||||
const { exposure } = params;
|
||||
const sourcePath = `${params.secretRoot.replace(/\/+$/u, "")}/${exposure.frpc.tokenSourceRef}`;
|
||||
const values = params.parseEnvFile(params.readTextFile(sourcePath));
|
||||
const token = params.requiredEnvValue(values, exposure.frpc.tokenSourceKey, exposure.frpc.tokenSourceRef);
|
||||
const frpcToml = [
|
||||
`serverAddr = "${exposure.frpc.serverAddr}"`,
|
||||
`serverPort = ${exposure.frpc.serverPort}`,
|
||||
"loginFailExit = true",
|
||||
`auth.token = "${escapeTomlString(token)}"`,
|
||||
"",
|
||||
"[[proxies]]",
|
||||
`name = "${exposure.frpc.proxyName}"`,
|
||||
'type = "tcp"',
|
||||
`localIP = "${exposure.frpc.localIP}"`,
|
||||
`localPort = ${exposure.frpc.localPort}`,
|
||||
`remotePort = ${exposure.frpc.remotePort}`,
|
||||
"",
|
||||
].join("\n");
|
||||
return {
|
||||
sourceRef: exposure.frpc.tokenSourceRef,
|
||||
sourcePath: params.sourcePathRedactor(sourcePath),
|
||||
secretName: exposure.frpc.secretName,
|
||||
secretKey: exposure.frpc.secretKey,
|
||||
frpcToml,
|
||||
fingerprint: fingerprintValues({ token, frpcToml }, ["token", "frpcToml"]),
|
||||
valuesPrinted: false,
|
||||
};
|
||||
}
|
||||
|
||||
export function renderFrpcManifest(target: PublicServiceTarget): string {
|
||||
const exposure = target.publicExposure;
|
||||
if (!exposure.enabled) return "";
|
||||
return `---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: ${exposure.frpc.deploymentName}
|
||||
namespace: ${target.namespace}
|
||||
labels:
|
||||
app.kubernetes.io/name: ${exposure.frpc.deploymentName}
|
||||
app.kubernetes.io/component: tunnel
|
||||
app.kubernetes.io/part-of: platform-infra
|
||||
app.kubernetes.io/managed-by: unidesk
|
||||
unidesk.ai/runtime-node: ${target.id}
|
||||
unidesk.ai/public-hostname: ${exposure.dns.hostname}
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: ${exposure.frpc.deploymentName}
|
||||
app.kubernetes.io/component: tunnel
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: ${exposure.frpc.deploymentName}
|
||||
app.kubernetes.io/component: tunnel
|
||||
app.kubernetes.io/part-of: platform-infra
|
||||
annotations:
|
||||
unidesk.ai/public-base-url: "${exposure.publicBaseUrl}"
|
||||
unidesk.ai/frp-server: "${exposure.frpc.serverAddr}:${exposure.frpc.serverPort}"
|
||||
unidesk.ai/frp-remote-port: "${exposure.frpc.remotePort}"
|
||||
spec:
|
||||
containers:
|
||||
- name: frpc
|
||||
image: ${exposure.frpc.image}
|
||||
imagePullPolicy: IfNotPresent
|
||||
args:
|
||||
- -c
|
||||
- /etc/frp/frpc.toml
|
||||
volumeMounts:
|
||||
- name: frpc-config
|
||||
mountPath: /etc/frp/frpc.toml
|
||||
subPath: ${exposure.frpc.secretKey}
|
||||
readOnly: true
|
||||
volumes:
|
||||
- name: frpc-config
|
||||
secret:
|
||||
secretName: ${exposure.frpc.secretName}
|
||||
`;
|
||||
}
|
||||
|
||||
export function publicServicePolicyChecks(yaml: string, target: PublicServiceTarget, serviceName: string): Array<Record<string, unknown>> {
|
||||
return [
|
||||
{ name: "no-ingress", ok: !/^\s*kind:\s*Ingress\s*$/mu.test(yaml), detail: `${serviceName} public exposure must use PK01 Caddy+FRP, not Kubernetes Ingress.` },
|
||||
{ name: "no-nodeport-or-loadbalancer", ok: !/^\s*type:\s*(NodePort|LoadBalancer)\s*$/mu.test(yaml), detail: "Services must stay ClusterIP." },
|
||||
{ name: "no-host-network", ok: !/^\s*hostNetwork:\s*true\s*$/mu.test(yaml), detail: "Pods must not use host network." },
|
||||
{ name: "no-host-port", ok: !/^\s*hostPort:\s*[0-9]+\s*$/mu.test(yaml), detail: "Pods must not expose host ports." },
|
||||
{ name: "no-cpu-memory-resources", ok: !/^\s*(cpu|memory):\s*/mu.test(yaml), detail: "No CPU/memory request or limit objects are rendered." },
|
||||
{ name: "allow-all-network-policy", ok: hasAllowAllNetworkPolicy(yaml, target.namespace), detail: `NetworkPolicy/allow-all exists in ${target.namespace}.` },
|
||||
];
|
||||
}
|
||||
|
||||
export function dryRunManifestScript(params: { yaml: string; target: PublicServiceTarget; fieldManager: string; manifestName: string }): string {
|
||||
const encoded = Buffer.from(params.yaml, "utf8").toString("base64");
|
||||
return `
|
||||
set -u
|
||||
tmp="$(mktemp -d)"
|
||||
trap 'rm -rf "$tmp"' EXIT
|
||||
manifest="$tmp/${params.manifestName}.k8s.yaml"
|
||||
printf '%s' '${encoded}' | base64 -d > "$manifest"
|
||||
kubectl apply --dry-run=client -f "$manifest" >"$tmp/client.out" 2>"$tmp/client.err"
|
||||
client_rc=$?
|
||||
if kubectl get namespace ${params.target.namespace} >/dev/null 2>&1; then
|
||||
kubectl apply --server-side --dry-run=server --field-manager=${params.fieldManager} -f "$manifest" >"$tmp/server.out" 2>"$tmp/server.err"
|
||||
server_rc=$?
|
||||
server_disposition=executed
|
||||
else
|
||||
: >"$tmp/server.err"
|
||||
printf '%s\\n' 'server dry-run skipped because namespace does not exist yet' >"$tmp/server.out"
|
||||
server_rc=0
|
||||
server_disposition=skipped-namespace-missing
|
||||
fi
|
||||
python3 - "$client_rc" "$server_rc" "$server_disposition" "$tmp/client.out" "$tmp/client.err" "$tmp/server.out" "$tmp/server.err" <<'PY'
|
||||
import json, sys
|
||||
client_rc, server_rc = int(sys.argv[1]), int(sys.argv[2])
|
||||
def text(path):
|
||||
try:
|
||||
return open(path, encoding="utf-8", errors="replace").read()
|
||||
except FileNotFoundError:
|
||||
return ""
|
||||
payload = {
|
||||
"ok": client_rc == 0 and server_rc == 0,
|
||||
"target": "${params.target.id}",
|
||||
"namespace": "${params.target.namespace}",
|
||||
"clientDryRun": {"exitCode": client_rc, "stdout": text(sys.argv[4])[-4000:], "stderr": text(sys.argv[5])[-4000:]},
|
||||
"serverDryRun": {"exitCode": server_rc, "disposition": sys.argv[3], "stdout": text(sys.argv[6])[-4000:], "stderr": text(sys.argv[7])[-4000:]},
|
||||
}
|
||||
print(json.dumps(payload, ensure_ascii=False, indent=2))
|
||||
sys.exit(0 if payload["ok"] else 1)
|
||||
PY
|
||||
`;
|
||||
}
|
||||
|
||||
export function parseJsonOutput(stdout: string): Record<string, unknown> | null {
|
||||
const trimmed = stdout.trim();
|
||||
if (trimmed.length === 0) return null;
|
||||
const start = trimmed.indexOf("{");
|
||||
const end = trimmed.lastIndexOf("}");
|
||||
if (start === -1 || end === -1 || end <= start) return null;
|
||||
try {
|
||||
const parsed = JSON.parse(trimmed.slice(start, end + 1)) as unknown;
|
||||
return typeof parsed === "object" && parsed !== null && !Array.isArray(parsed) ? parsed as Record<string, unknown> : null;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
export function compactCapture(result: SshCaptureResult, options: { full?: boolean } = {}): Record<string, unknown> {
|
||||
const full = options.full ?? false;
|
||||
return {
|
||||
exitCode: result.exitCode,
|
||||
stdoutBytes: Buffer.byteLength(result.stdout, "utf8"),
|
||||
stderrBytes: Buffer.byteLength(result.stderr, "utf8"),
|
||||
stdoutTail: full || result.exitCode !== 0 ? redactText(result.stdout).slice(-8000) : "",
|
||||
stderrTail: full || result.exitCode !== 0 ? redactText(result.stderr).slice(-4000) : "",
|
||||
};
|
||||
}
|
||||
|
||||
export function publicHttpProbe(baseUrl: string, path: string): Record<string, unknown> {
|
||||
const url = `${baseUrl.replace(/\/+$/u, "")}${path}`;
|
||||
const result = Bun.spawnSync(["curl", "-fsS", "--connect-timeout", "10", "--max-time", "30", "-o", "-", "-w", "\n%{http_code}", url], { stdout: "pipe", stderr: "pipe" });
|
||||
const stdout = new TextDecoder().decode(result.stdout);
|
||||
const stderr = new TextDecoder().decode(result.stderr);
|
||||
const lines = stdout.split(/\r?\n/u);
|
||||
const statusText = lines.pop() ?? "";
|
||||
const status = Number(statusText);
|
||||
const body = lines.join("\n");
|
||||
return {
|
||||
ok: result.exitCode === 0 && Number.isInteger(status) && status >= 200 && status < 500,
|
||||
url,
|
||||
status: Number.isInteger(status) ? status : null,
|
||||
bodyBytes: Buffer.byteLength(body, "utf8"),
|
||||
bodyPreview: body.slice(0, 2000),
|
||||
stderrTail: redactText(stderr).slice(-2000),
|
||||
valuesPrinted: false,
|
||||
};
|
||||
}
|
||||
|
||||
export function redactText(text: string): string {
|
||||
return text
|
||||
.replace(/postgres(?:ql)?:\/\/[^@\s]+@/giu, "postgresql://<redacted>@")
|
||||
.replace(/(N8N_ENCRYPTION_KEY|PASSWORD|SECRET|TOKEN|API_KEY|DATABASE_URL)([=:]\s*)[^\s,;]+/giu, "$1$2<redacted>");
|
||||
}
|
||||
|
||||
export function fingerprintValues(values: Record<string, string>, keys: string[]): string {
|
||||
const hash = createHash("sha256");
|
||||
for (const key of keys.slice().sort()) {
|
||||
hash.update(key);
|
||||
hash.update("\0");
|
||||
hash.update(values[key] ?? "");
|
||||
hash.update("\0");
|
||||
}
|
||||
return `sha256:${hash.digest("hex")}`;
|
||||
}
|
||||
|
||||
export function shQuote(value: string): string {
|
||||
return `'${value.replaceAll("'", "'\"'\"'")}'`;
|
||||
}
|
||||
|
||||
export function escapeTomlString(value: string): string {
|
||||
return value.replaceAll("\\", "\\\\").replaceAll("\"", "\\\"");
|
||||
}
|
||||
|
||||
function hasAllowAllNetworkPolicy(yaml: string, namespaceName: string): boolean {
|
||||
return yaml.split(/^---\s*$/mu).some((document) => /^\s*kind:\s*NetworkPolicy\s*$/mu.test(document)
|
||||
&& /^\s*name:\s*allow-all\s*$/mu.test(document)
|
||||
&& new RegExp(`^\\s*namespace:\\s*${escapeRegExp(namespaceName)}\\s*$`, "mu").test(document)
|
||||
&& /^\s*podSelector:\s*\{\}\s*$/mu.test(document));
|
||||
}
|
||||
|
||||
function escapeRegExp(value: string): string {
|
||||
return value.replace(/[.*+?^${}()|[\]\\]/gu, "\\$&");
|
||||
}
|
||||
@@ -1346,13 +1346,25 @@ def gateway_monitor_paths(config):
|
||||
return {"/responses", "/v1/responses", "/responses/compact", "/v1/responses/compact"}
|
||||
|
||||
def gateway_failure_kind(message, payload, config):
|
||||
if "openai.forward_failed" not in message or not isinstance(payload, dict):
|
||||
if not isinstance(payload, dict):
|
||||
return None
|
||||
path = payload.get("path")
|
||||
if path not in gateway_monitor_paths(config):
|
||||
return None
|
||||
if payload.get("account_id") is None:
|
||||
return None
|
||||
if "codex.remote_compact.failed" in message:
|
||||
status = payload.get("status_code")
|
||||
if isinstance(status, int) and status >= 500:
|
||||
return "gateway-compact-final-failure"
|
||||
return None
|
||||
if "openai.upstream_failover_switching" in message and path in ("/responses/compact", "/v1/responses/compact"):
|
||||
upstream_status = payload.get("upstream_status")
|
||||
if isinstance(upstream_status, int) and upstream_status >= 500:
|
||||
return "gateway-compact-upstream-failover"
|
||||
return None
|
||||
if "openai.forward_failed" not in message:
|
||||
return None
|
||||
error_text = str(payload.get("error") or "").lower()
|
||||
fallback_written = payload.get("fallback_error_response_written") is True
|
||||
upstream_already_written = payload.get("upstream_error_response_already_written") is True
|
||||
@@ -1404,7 +1416,7 @@ def gateway_failure_kind(message, payload, config):
|
||||
return None
|
||||
|
||||
def gateway_failure_is_observe_only(failure_kind):
|
||||
return failure_kind in {"gateway-session-affinity-failure"}
|
||||
return failure_kind in {"gateway-session-affinity-failure", "gateway-compact-final-failure", "gateway-compact-upstream-failover"}
|
||||
|
||||
def gateway_failure_item(ts, pod_name, payload, failure_kind):
|
||||
request_id = payload.get("request_id") or sha(json.dumps(payload, sort_keys=True, ensure_ascii=False))
|
||||
|
||||
@@ -202,6 +202,8 @@ interface CodexPoolLocalCodexConfig {
|
||||
backupSuffix: string;
|
||||
providerName: string;
|
||||
wireApi: string;
|
||||
modelContextWindow: number;
|
||||
modelAutoCompactTokenLimit: number;
|
||||
supportsWebSockets: boolean;
|
||||
responsesWebSocketsV2: boolean;
|
||||
responsesSmokeModel: string;
|
||||
@@ -211,6 +213,8 @@ interface CodexLocalConsumerTomlOptions {
|
||||
providerName: string;
|
||||
baseUrl: string;
|
||||
wireApi: string;
|
||||
modelContextWindow: number;
|
||||
modelAutoCompactTokenLimit: number;
|
||||
supportsWebSockets: boolean;
|
||||
responsesWebSocketsV2: boolean;
|
||||
}
|
||||
@@ -1009,6 +1013,8 @@ async function codexPoolConfigureLocal(config: UniDeskConfig, options: ConfirmOp
|
||||
baseUrl: codexConsumerBaseUrl(pool),
|
||||
providerName: pool.localCodex.providerName,
|
||||
wireApi: pool.localCodex.wireApi,
|
||||
modelContextWindow: pool.localCodex.modelContextWindow,
|
||||
modelAutoCompactTokenLimit: pool.localCodex.modelAutoCompactTokenLimit,
|
||||
supportsWebSockets: pool.localCodex.supportsWebSockets,
|
||||
responsesWebSocketsV2: pool.localCodex.responsesWebSocketsV2,
|
||||
responsesSmokeModel: pool.localCodex.responsesSmokeModel,
|
||||
@@ -1274,6 +1280,8 @@ function defaultCodexPoolConfig(): CodexPoolConfig {
|
||||
backupSuffix: "pre-sub2api",
|
||||
providerName: "OpenAI",
|
||||
wireApi: "responses",
|
||||
modelContextWindow: 272000,
|
||||
modelAutoCompactTokenLimit: 240000,
|
||||
supportsWebSockets: true,
|
||||
responsesWebSocketsV2: true,
|
||||
responsesSmokeModel: "gpt-5.5",
|
||||
@@ -1444,6 +1452,11 @@ function readPositiveInteger(value: unknown, key: string): number {
|
||||
return parsed;
|
||||
}
|
||||
|
||||
function readPositiveIntegerConfig(value: unknown, key: string, fallback: number): number {
|
||||
if (value === undefined || value === null) return fallback;
|
||||
return readPositiveInteger(value, key);
|
||||
}
|
||||
|
||||
function readCaddyRetryMethods(value: unknown, key: string): string[] {
|
||||
if (value === undefined || value === null) return [];
|
||||
if (!Array.isArray(value)) throw new Error(`${codexPoolConfigPath}.${key} must be a YAML array`);
|
||||
@@ -1619,6 +1632,8 @@ function readLocalCodexConfig(value: unknown, defaults: CodexPoolLocalCodexConfi
|
||||
backupSuffix: stringValue(value.backupSuffix) ?? defaults.backupSuffix,
|
||||
providerName: stringValue(value.providerName) ?? defaults.providerName,
|
||||
wireApi: stringValue(value.wireApi) ?? defaults.wireApi,
|
||||
modelContextWindow: readPositiveIntegerConfig(value.modelContextWindow, "localCodex.modelContextWindow", defaults.modelContextWindow),
|
||||
modelAutoCompactTokenLimit: readPositiveIntegerConfig(value.modelAutoCompactTokenLimit, "localCodex.modelAutoCompactTokenLimit", defaults.modelAutoCompactTokenLimit),
|
||||
supportsWebSockets: readBooleanConfig(value.supportsWebSockets, "localCodex.supportsWebSockets", defaults.supportsWebSockets),
|
||||
responsesWebSocketsV2: readBooleanConfig(value.responsesWebSocketsV2, "localCodex.responsesWebSocketsV2", defaults.responsesWebSocketsV2),
|
||||
responsesSmokeModel: stringValue(value.responsesSmokeModel) ?? defaults.responsesSmokeModel,
|
||||
@@ -3097,6 +3112,8 @@ function writeLocalCodexConfig(pool: CodexPoolConfig, apiKey: string): Record<st
|
||||
name: pool.localCodex.providerName,
|
||||
baseUrl: codexConsumerBaseUrl(pool),
|
||||
wireApi: pool.localCodex.wireApi,
|
||||
modelContextWindow: pool.localCodex.modelContextWindow,
|
||||
modelAutoCompactTokenLimit: pool.localCodex.modelAutoCompactTokenLimit,
|
||||
supportsWebSockets: pool.localCodex.supportsWebSockets,
|
||||
responsesWebSocketsV2: pool.localCodex.responsesWebSocketsV2,
|
||||
},
|
||||
@@ -3116,6 +3133,8 @@ function updateCodexConfigToml(current: string, pool: CodexPoolConfig): string {
|
||||
providerName: pool.localCodex.providerName,
|
||||
baseUrl: codexConsumerBaseUrl(pool),
|
||||
wireApi: pool.localCodex.wireApi,
|
||||
modelContextWindow: pool.localCodex.modelContextWindow,
|
||||
modelAutoCompactTokenLimit: pool.localCodex.modelAutoCompactTokenLimit,
|
||||
supportsWebSockets: pool.localCodex.supportsWebSockets,
|
||||
responsesWebSocketsV2: pool.localCodex.responsesWebSocketsV2,
|
||||
});
|
||||
@@ -3127,6 +3146,8 @@ function codexConsumerBaseUrl(pool: CodexPoolConfig): string {
|
||||
|
||||
export function renderCodexLocalConsumerToml(current: string, options: CodexLocalConsumerTomlOptions): string {
|
||||
let next = upsertTopLevelTomlString(current, "model_provider", options.providerName);
|
||||
next = upsertTopLevelTomlInteger(next, "model_context_window", options.modelContextWindow);
|
||||
next = upsertTopLevelTomlInteger(next, "model_auto_compact_token_limit", options.modelAutoCompactTokenLimit);
|
||||
next = upsertProviderSection(next, options);
|
||||
next = upsertTomlSectionBoolean(next, "features", "responses_websockets_v2", options.responsesWebSocketsV2);
|
||||
return next.endsWith("\n") ? next : `${next}\n`;
|
||||
@@ -3146,6 +3167,21 @@ function upsertTopLevelTomlString(text: string, key: string, value: string): str
|
||||
return lines.join("\n");
|
||||
}
|
||||
|
||||
function upsertTopLevelTomlInteger(text: string, key: string, value: number): string {
|
||||
const lines = text.split(/\r?\n/u);
|
||||
const firstSection = lines.findIndex((line) => /^\s*\[/.test(line));
|
||||
const end = firstSection === -1 ? lines.length : firstSection;
|
||||
const assignment = `${key} = ${value}`;
|
||||
for (let index = 0; index < end; index += 1) {
|
||||
if (new RegExp(`^\\s*${escapeRegExp(key)}\\s*=`).test(lines[index])) {
|
||||
lines[index] = assignment;
|
||||
return lines.join("\n");
|
||||
}
|
||||
}
|
||||
lines.splice(end, 0, assignment);
|
||||
return lines.join("\n");
|
||||
}
|
||||
|
||||
function upsertProviderSection(text: string, options: CodexLocalConsumerTomlOptions): string {
|
||||
const { providerName, baseUrl, wireApi, supportsWebSockets } = options;
|
||||
const header = `[model_providers.${providerName}]`;
|
||||
|
||||
@@ -191,7 +191,7 @@ interface EgressProxySecretMaterial {
|
||||
|
||||
export function platformInfraHelp(): unknown {
|
||||
return {
|
||||
command: "platform-infra sub2api plan|apply|status|validate|codex-pool",
|
||||
command: "platform-infra sub2api|langbot|n8n ...",
|
||||
output: "json",
|
||||
usage: [
|
||||
"bun scripts/cli.ts platform-infra sub2api plan [--target G14|D601]",
|
||||
@@ -205,8 +205,19 @@ export function platformInfraHelp(): unknown {
|
||||
"bun scripts/cli.ts platform-infra sub2api codex-pool trace --request-id <requestId>",
|
||||
"bun scripts/cli.ts platform-infra sub2api codex-pool sentinel-image status",
|
||||
"bun scripts/cli.ts platform-infra sub2api codex-pool sentinel-probe --account unidesk-codex-hy --confirm",
|
||||
"bun scripts/cli.ts platform-infra langbot plan [--target G14]",
|
||||
"bun scripts/cli.ts platform-infra langbot apply [--target G14] --confirm",
|
||||
"bun scripts/cli.ts platform-infra langbot status [--target G14] [--full|--raw]",
|
||||
"bun scripts/cli.ts platform-infra langbot logs [--target G14] [--component app|plugin-runtime|frpc|all]",
|
||||
"bun scripts/cli.ts platform-infra langbot bootstrap-api-key --confirm",
|
||||
"bun scripts/cli.ts platform-infra langbot query --path /api/v1/platform/bots",
|
||||
"bun scripts/cli.ts platform-infra n8n plan [--target G14]",
|
||||
"bun scripts/cli.ts platform-infra n8n apply [--target G14] --confirm",
|
||||
"bun scripts/cli.ts platform-infra n8n status [--target G14] [--full|--raw]",
|
||||
"bun scripts/cli.ts platform-infra n8n logs [--target G14] [--component app|frpc|all]",
|
||||
"bun scripts/cli.ts platform-infra n8n validate [--target G14] [--full|--raw]",
|
||||
],
|
||||
description: "Operate YAML-controlled Sub2API platform-infra targets. G14 remains the active bundled deployment; D601 can be held at external-pending predeployment or activated against the external PK01 PostgreSQL runtime.",
|
||||
description: "Operate YAML-controlled platform-infra services such as Sub2API, LangBot and n8n. Public services use PK01 Caddy+FRP rather than Kubernetes Ingress, NodePort, or LoadBalancer.",
|
||||
target: {
|
||||
default: defaultTargetId,
|
||||
namespace,
|
||||
@@ -231,6 +242,14 @@ export function platformInfraHelp(): unknown {
|
||||
|
||||
export async function runPlatformInfraCommand(config: UniDeskConfig, args: string[]): Promise<Record<string, unknown> | RenderedCliResult> {
|
||||
const [target, action] = args;
|
||||
if (target === "langbot") {
|
||||
const { runLangBotCommand } = await import("./platform-infra-langbot");
|
||||
return await runLangBotCommand(config, args.slice(1));
|
||||
}
|
||||
if (target === "n8n") {
|
||||
const { runN8nCommand } = await import("./platform-infra-n8n");
|
||||
return await runN8nCommand(config, args.slice(1));
|
||||
}
|
||||
if (target !== "sub2api") return unsupported(args);
|
||||
if (action === "plan" || action === undefined) return plan(parseTargetOptions(args.slice(2)));
|
||||
if (action === "apply") return await apply(config, parseApplyOptions(args.slice(2)));
|
||||
|
||||
@@ -164,11 +164,15 @@ function systemStatusSignal(debug: unknown, providerId: string): ProviderTriageS
|
||||
if (item === null) return signal("backend-core-system-status", "provider-gateway", "unknown", `no system status sample for ${providerId}`);
|
||||
const current = asRecord(item.current);
|
||||
const currentOk = current === null ? null : current.ok;
|
||||
const status: ProviderSignalStatus = current === null ? "unknown" : currentOk === false ? "degraded" : "ok";
|
||||
return signal("backend-core-system-status", "provider-gateway", status, `system status current.ok=${String(currentOk)} updatedAt=${item.updatedAt ?? "null"}`, {
|
||||
const stale = item.stale === true;
|
||||
const status: ProviderSignalStatus = current === null ? stale ? "degraded" : "unknown" : currentOk === false ? "degraded" : "ok";
|
||||
return signal("backend-core-system-status", "provider-gateway", status, `system status current.ok=${String(currentOk)} stale=${String(stale)} updatedAt=${item.updatedAt ?? "null"}`, {
|
||||
providerId: item.providerId,
|
||||
nodeStatus: item.nodeStatus,
|
||||
updatedAt: item.updatedAt,
|
||||
currentCollectedAt: item.currentCollectedAt ?? null,
|
||||
stale,
|
||||
staleSeconds: item.staleSeconds ?? null,
|
||||
current: current === null ? null : {
|
||||
ok: current.ok,
|
||||
collectedAt: current.collectedAt,
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
use std::sync::Arc;
|
||||
|
||||
use anyhow::Context;
|
||||
use chrono::{DateTime, Utc};
|
||||
use chrono::{DateTime, Duration, Utc};
|
||||
use serde_json::{json, Value};
|
||||
use tokio_postgres::Row;
|
||||
|
||||
@@ -192,18 +192,43 @@ pub async fn update_provider_heartbeat(
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn provider_collected_at(
|
||||
state: &Arc<AppState>,
|
||||
provider_id: &str,
|
||||
source: &str,
|
||||
value: &str,
|
||||
) -> DateTime<Utc> {
|
||||
match DateTime::parse_from_rfc3339(value) {
|
||||
Ok(parsed) => parsed.with_timezone(&Utc),
|
||||
Err(error) => {
|
||||
state.log(
|
||||
"warn",
|
||||
"provider_status_collected_at_invalid",
|
||||
Some(json!({
|
||||
"providerId": provider_id,
|
||||
"source": source,
|
||||
"value": value,
|
||||
"error": error.to_string()
|
||||
})),
|
||||
);
|
||||
Utc::now()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub async fn upsert_docker_status(
|
||||
state: &Arc<AppState>,
|
||||
provider_id: &str,
|
||||
status: &JsonValue,
|
||||
collected_at: &str,
|
||||
) -> anyhow::Result<()> {
|
||||
let collected_at = provider_collected_at(state, provider_id, "docker_status", collected_at);
|
||||
let client = state.pool.get().await?;
|
||||
client
|
||||
.execute(
|
||||
r#"
|
||||
INSERT INTO unidesk_node_docker_status (provider_id, status, collected_at, updated_at)
|
||||
VALUES ($1, $2, $3::timestamptz, now())
|
||||
VALUES ($1, $2, $3, now())
|
||||
ON CONFLICT (provider_id) DO UPDATE SET
|
||||
status = EXCLUDED.status,
|
||||
collected_at = EXCLUDED.collected_at,
|
||||
@@ -224,12 +249,13 @@ pub async fn upsert_system_status(
|
||||
let cpu_percent = nested_number(status, "cpu", "percent");
|
||||
let memory_percent = nested_number(status, "memory", "percent");
|
||||
let disk_percent = nested_number(status, "disk", "percent");
|
||||
let collected_at = provider_collected_at(state, provider_id, "system_status", collected_at);
|
||||
let mut client = state.pool.get().await?;
|
||||
let tx = client.transaction().await?;
|
||||
tx.execute(
|
||||
r#"
|
||||
INSERT INTO unidesk_node_system_status (provider_id, status, collected_at, updated_at)
|
||||
VALUES ($1, $2, $3::timestamptz, now())
|
||||
VALUES ($1, $2, $3, now())
|
||||
ON CONFLICT (provider_id) DO UPDATE SET
|
||||
status = EXCLUDED.status,
|
||||
collected_at = EXCLUDED.collected_at,
|
||||
@@ -241,10 +267,18 @@ pub async fn upsert_system_status(
|
||||
tx.execute(
|
||||
r#"
|
||||
INSERT INTO unidesk_node_metric_samples (provider_id, collected_at, cpu_percent, memory_percent, disk_percent, sample)
|
||||
VALUES ($1, $2::timestamptz, $3, $4, $5, $6)
|
||||
VALUES ($1, $2, $3, $4, $5, $6)
|
||||
"#,
|
||||
&[&provider_id, &collected_at, &cpu_percent, &memory_percent, &disk_percent, status],
|
||||
).await?;
|
||||
&[
|
||||
&provider_id,
|
||||
&collected_at,
|
||||
&cpu_percent,
|
||||
&memory_percent,
|
||||
&disk_percent,
|
||||
status,
|
||||
],
|
||||
)
|
||||
.await?;
|
||||
tx.execute(
|
||||
r#"
|
||||
DELETE FROM unidesk_node_metric_samples
|
||||
@@ -321,6 +355,18 @@ fn metric_point_from_sample(sample: &JsonValue, collected_at: &str) -> JsonValue
|
||||
})
|
||||
}
|
||||
|
||||
const SYSTEM_STATUS_STALE_AFTER_SECONDS: i64 = 300;
|
||||
|
||||
fn system_status_is_stale(collected_at: Option<DateTime<Utc>>, now: DateTime<Utc>) -> bool {
|
||||
collected_at
|
||||
.map(|value| now.signed_duration_since(value) > Duration::seconds(SYSTEM_STATUS_STALE_AFTER_SECONDS))
|
||||
.unwrap_or(false)
|
||||
}
|
||||
|
||||
fn system_status_age_seconds(collected_at: Option<DateTime<Utc>>, now: DateTime<Utc>) -> Option<i64> {
|
||||
collected_at.map(|value| now.signed_duration_since(value).num_seconds().max(0))
|
||||
}
|
||||
|
||||
pub async fn get_node_system_statuses(
|
||||
state: &Arc<AppState>,
|
||||
limit: i64,
|
||||
@@ -328,7 +374,7 @@ pub async fn get_node_system_statuses(
|
||||
let client = state.pool.get().await?;
|
||||
let current_rows = client.query(
|
||||
r#"
|
||||
SELECT n.provider_id, n.name, n.status AS node_status, s.status AS system_status, s.updated_at
|
||||
SELECT n.provider_id, n.name, n.status AS node_status, s.status AS system_status, s.collected_at AS system_collected_at, s.updated_at
|
||||
FROM unidesk_nodes n
|
||||
LEFT JOIN unidesk_node_system_status s ON s.provider_id = n.provider_id
|
||||
ORDER BY n.status DESC, n.provider_id ASC
|
||||
@@ -366,15 +412,23 @@ pub async fn get_node_system_statuses(
|
||||
&collected_at.to_rfc3339_opts(chrono::SecondsFormat::Millis, true),
|
||||
));
|
||||
}
|
||||
let now = Utc::now();
|
||||
Ok(current_rows.into_iter().map(|row| {
|
||||
let provider_id: String = row.get("provider_id");
|
||||
let updated_at: Option<DateTime<Utc>> = row.get("updated_at");
|
||||
let system_collected_at: Option<DateTime<Utc>> = row.get("system_collected_at");
|
||||
let system_status: Option<JsonValue> = row.get("system_status");
|
||||
let stale = system_status_is_stale(system_collected_at, now);
|
||||
let current = if stale { None } else { system_status.clone() };
|
||||
json!({
|
||||
"providerId": provider_id,
|
||||
"name": row.get::<_, String>("name"),
|
||||
"nodeStatus": if row.get::<_, String>("node_status") == "online" { "online" } else { "offline" },
|
||||
"current": system_status,
|
||||
"current": current,
|
||||
"lastKnown": system_status,
|
||||
"currentCollectedAt": system_collected_at.map(|value| value.to_rfc3339_opts(chrono::SecondsFormat::Millis, true)),
|
||||
"stale": stale,
|
||||
"staleSeconds": system_status_age_seconds(system_collected_at, now),
|
||||
"history": history_by_provider.remove(&provider_id).unwrap_or_default(),
|
||||
"updatedAt": updated_at.map(|value| value.to_rfc3339_opts(chrono::SecondsFormat::Millis, true)),
|
||||
})
|
||||
|
||||
@@ -237,9 +237,22 @@ function metricPointFromSample(sample: unknown, collectedAt: string): JsonValue
|
||||
};
|
||||
}
|
||||
|
||||
const SYSTEM_STATUS_STALE_AFTER_SECONDS = 300;
|
||||
|
||||
function systemStatusAgeSeconds(collectedAt: unknown, now: Date): number | null {
|
||||
const date = collectedAt instanceof Date ? collectedAt : typeof collectedAt === "string" ? new Date(collectedAt) : null;
|
||||
if (date === null || Number.isNaN(date.getTime())) return null;
|
||||
return Math.max(0, Math.floor((now.getTime() - date.getTime()) / 1000));
|
||||
}
|
||||
|
||||
function systemStatusIsStale(collectedAt: unknown, now: Date): boolean {
|
||||
const ageSeconds = systemStatusAgeSeconds(collectedAt, now);
|
||||
return ageSeconds !== null && ageSeconds > SYSTEM_STATUS_STALE_AFTER_SECONDS;
|
||||
}
|
||||
|
||||
export async function getNodeSystemStatuses(limit: number): Promise<ApiNodeSystemStatus[]> {
|
||||
const currentRows = await sql()<Array<Record<string, unknown>>>`
|
||||
SELECT n.provider_id, n.name, n.status AS node_status, s.status AS system_status, s.updated_at
|
||||
SELECT n.provider_id, n.name, n.status AS node_status, s.status AS system_status, s.collected_at AS system_collected_at, s.updated_at
|
||||
FROM unidesk_nodes n
|
||||
LEFT JOIN unidesk_node_system_status s ON s.provider_id = n.provider_id
|
||||
ORDER BY n.status DESC, n.provider_id ASC
|
||||
@@ -265,13 +278,21 @@ export async function getNodeSystemStatuses(limit: number): Promise<ApiNodeSyste
|
||||
history.push(metricPointFromSample(row.sample ?? {}, collectedAt));
|
||||
historyByProvider.set(providerId, history);
|
||||
}
|
||||
const now = new Date();
|
||||
return currentRows.map((row) => {
|
||||
const providerId = String(row.provider_id);
|
||||
const lastKnown = row.system_status === null || row.system_status === undefined ? null : (row.system_status as JsonValue);
|
||||
const currentCollectedAt = row.system_collected_at instanceof Date ? row.system_collected_at.toISOString() : row.system_collected_at === null || row.system_collected_at === undefined ? null : String(row.system_collected_at);
|
||||
const stale = systemStatusIsStale(row.system_collected_at, now);
|
||||
return {
|
||||
providerId,
|
||||
name: String(row.name),
|
||||
nodeStatus: row.node_status === "online" ? "online" : "offline",
|
||||
current: row.system_status === null || row.system_status === undefined ? null : (row.system_status as JsonValue),
|
||||
current: stale ? null : lastKnown,
|
||||
lastKnown,
|
||||
currentCollectedAt,
|
||||
stale,
|
||||
staleSeconds: systemStatusAgeSeconds(row.system_collected_at, now),
|
||||
history: historyByProvider.get(providerId) ?? [],
|
||||
updatedAt: row.updated_at instanceof Date ? row.updated_at.toISOString() : row.updated_at === null || row.updated_at === undefined ? null : String(row.updated_at),
|
||||
};
|
||||
|
||||
@@ -571,22 +571,22 @@ async fn provider_socket_task(state: Arc<AppState>, socket: WebSocket) {
|
||||
while let Some(message) = receiver_ws.next().await {
|
||||
match message {
|
||||
Ok(Message::Text(text)) => {
|
||||
if let Err(error) = handle_provider_text(&state, &connection, text).await {
|
||||
if let Err(error) = handle_provider_text(&state, &connection, &text).await {
|
||||
state.log(
|
||||
"error",
|
||||
"provider_message_failed",
|
||||
Some(json!({ "error": error.to_string() })),
|
||||
Some(provider_message_error_context(&text, &error)),
|
||||
);
|
||||
let _ = tx.send(Message::Text(json!({ "type": "ack", "requestId": "message", "ok": false, "message": error.to_string() }).to_string()));
|
||||
}
|
||||
}
|
||||
Ok(Message::Binary(bytes)) => {
|
||||
if let Ok(text) = String::from_utf8(bytes.to_vec()) {
|
||||
if let Err(error) = handle_provider_text(&state, &connection, text).await {
|
||||
if let Err(error) = handle_provider_text(&state, &connection, &text).await {
|
||||
state.log(
|
||||
"error",
|
||||
"provider_message_failed",
|
||||
Some(json!({ "error": error.to_string() })),
|
||||
Some(provider_message_error_context(&text, &error)),
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -629,12 +629,36 @@ async fn provider_socket_task(state: Arc<AppState>, socket: WebSocket) {
|
||||
send_task.abort();
|
||||
}
|
||||
|
||||
fn provider_message_error_context(text: &str, error: &anyhow::Error) -> Value {
|
||||
let parsed = serde_json::from_str::<Value>(text).ok();
|
||||
json!({
|
||||
"error": error.to_string(),
|
||||
"messageType": parsed
|
||||
.as_ref()
|
||||
.and_then(|value| value.get("type"))
|
||||
.and_then(Value::as_str)
|
||||
.unwrap_or("unknown"),
|
||||
"providerId": parsed
|
||||
.as_ref()
|
||||
.and_then(|value| value.get("providerId"))
|
||||
.and_then(Value::as_str)
|
||||
.unwrap_or("unknown"),
|
||||
"requestId": parsed
|
||||
.as_ref()
|
||||
.and_then(|value| value.get("requestId"))
|
||||
.and_then(Value::as_str)
|
||||
.unwrap_or("unknown"),
|
||||
"textBytes": text.len(),
|
||||
"textPreview": text.chars().take(300).collect::<String>(),
|
||||
})
|
||||
}
|
||||
|
||||
async fn handle_provider_text(
|
||||
state: &Arc<AppState>,
|
||||
connection: &Arc<ProviderConnection>,
|
||||
text: String,
|
||||
text: &str,
|
||||
) -> anyhow::Result<()> {
|
||||
let message: Value = serde_json::from_str(&text)?;
|
||||
let message: Value = serde_json::from_str(text)?;
|
||||
let provider_id = message
|
||||
.get("providerId")
|
||||
.and_then(Value::as_str)
|
||||
@@ -845,6 +869,7 @@ pub async fn mark_stale_providers_offline(state: &Arc<AppState>) -> anyhow::Resu
|
||||
return Ok(());
|
||||
}
|
||||
let timeout_ms = state.config.heartbeat_timeout_ms as i64;
|
||||
let cutoff = Utc::now() - Duration::milliseconds(timeout_ms);
|
||||
let client = state.pool.get().await?;
|
||||
let rows = client
|
||||
.query(
|
||||
@@ -853,10 +878,10 @@ pub async fn mark_stale_providers_offline(state: &Arc<AppState>) -> anyhow::Resu
|
||||
SET status = 'offline', updated_at = now()
|
||||
WHERE status = 'online'
|
||||
AND last_heartbeat IS NOT NULL
|
||||
AND last_heartbeat < now() - ($1 * interval '1 millisecond')
|
||||
AND last_heartbeat < $1
|
||||
RETURNING provider_id
|
||||
"#,
|
||||
&[&timeout_ms],
|
||||
&[&cutoff],
|
||||
)
|
||||
.await?;
|
||||
for row in rows {
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -784,11 +784,50 @@ function HeartbeatPage({ nodes }: AnyRecord) {
|
||||
);
|
||||
}
|
||||
|
||||
function metricPointFromCurrent(current: AnyRecord): AnyRecord {
|
||||
const cpu = current?.cpu || {};
|
||||
const memory = current?.memory || {};
|
||||
const disk = current?.disk || {};
|
||||
return {
|
||||
at: current?.collectedAt,
|
||||
cpuPercent: asNumber(cpu.percent),
|
||||
memoryPercent: asNumber(memory.percent),
|
||||
diskPercent: asNumber(disk.percent),
|
||||
memoryUsedBytes: asNumber(memory.usedBytes),
|
||||
memoryTotalBytes: asNumber(memory.totalBytes),
|
||||
diskUsedBytes: asNumber(disk.usedBytes),
|
||||
diskTotalBytes: asNumber(disk.totalBytes),
|
||||
load1: asNumber(cpu.load1),
|
||||
};
|
||||
}
|
||||
|
||||
function synchronizedMetricPoints(history: any[], current: AnyRecord | null): AnyRecord[] {
|
||||
if (!current) return [];
|
||||
const currentPoint = metricPointFromCurrent(current);
|
||||
const currentAt = timeMs(currentPoint.at);
|
||||
const points = (Array.isArray(history) ? history : [])
|
||||
.filter((point) => point && typeof point === "object")
|
||||
.filter((point) => currentAt === null || timeMs(point.at) !== currentAt);
|
||||
points.push(currentPoint);
|
||||
return points
|
||||
.sort((left, right) => (timeMs(left.at) ?? 0) - (timeMs(right.at) ?? 0))
|
||||
.slice(-60);
|
||||
}
|
||||
|
||||
function NodeMonitorPage({ nodes, systemStatuses, tasks, onRaw, refresh }: AnyRecord) {
|
||||
const [selectedProvider, setSelectedProvider] = useState("");
|
||||
const merged = useMemo(() => nodes.map((node: any) => {
|
||||
const status = systemStatuses.find((item: any) => item.providerId === node.providerId);
|
||||
return { ...node, systemCurrent: status?.current || null, systemHistory: status?.history || [], systemUpdatedAt: status?.updatedAt || null };
|
||||
return {
|
||||
...node,
|
||||
systemCurrent: status?.current || null,
|
||||
systemLastKnown: status?.lastKnown || null,
|
||||
systemCurrentCollectedAt: status?.currentCollectedAt || null,
|
||||
systemStale: Boolean(status?.stale),
|
||||
systemStaleSeconds: status?.staleSeconds ?? null,
|
||||
systemHistory: status?.history || [],
|
||||
systemUpdatedAt: status?.updatedAt || null,
|
||||
};
|
||||
}), [nodes, systemStatuses]);
|
||||
const active = merged.find((node: any) => node.providerId === selectedProvider) || merged[0] || null;
|
||||
useEffect(() => {
|
||||
@@ -802,12 +841,10 @@ function NodeMonitorPage({ nodes, systemStatuses, tasks, onRaw, refresh }: AnyRe
|
||||
const cpu = current?.cpu || {};
|
||||
const memory = current?.memory || {};
|
||||
const disk = current?.disk || {};
|
||||
const points = history.length > 0 ? history : current ? [{
|
||||
at: current.collectedAt,
|
||||
cpuPercent: asNumber(cpu.percent),
|
||||
memoryPercent: asNumber(memory.percent),
|
||||
diskPercent: asNumber(disk.percent),
|
||||
}] : [];
|
||||
const points = synchronizedMetricPoints(history, current);
|
||||
const staleText = active.systemCurrentCollectedAt
|
||||
? `最后采样 ${fmtDate(active.systemCurrentCollectedAt)}，已过期 ${fmtDuration(asNumber(active.systemStaleSeconds))}`
|
||||
: "最后采样时间不可用";
|
||||
|
||||
return h("div", { className: "monitor-page", "data-testid": "node-monitor-page" },
|
||||
h("div", { className: "docker-node-strip" },
|
||||
@@ -820,7 +857,7 @@ function NodeMonitorPage({ nodes, systemStatuses, tasks, onRaw, refresh }: AnyRe
|
||||
h("span", { className: `pulse ${node.status}` }),
|
||||
h("strong", null, node.name),
|
||||
h("code", null, node.providerId),
|
||||
h("span", null, node.systemCurrent ? `CPU ${fmtPercent(node.systemCurrent.cpu?.percent)} / MEM ${fmtPercent(node.systemCurrent.memory?.percent)}` : "等待指标"),
|
||||
h("span", null, node.systemCurrent ? `CPU ${fmtPercent(node.systemCurrent.cpu?.percent)} / MEM ${fmtPercent(node.systemCurrent.memory?.percent)}` : node.systemStale ? "指标过期" : "等待指标"),
|
||||
)),
|
||||
),
|
||||
h("div", { className: "monitor-layout" },
|
||||
@@ -828,9 +865,9 @@ function NodeMonitorPage({ nodes, systemStatuses, tasks, onRaw, refresh }: AnyRe
|
||||
title: "任务管理器视图",
|
||||
eyebrow: active.name,
|
||||
className: "monitor-main-panel",
|
||||
actions: current ? h(RawButton, { title: `System ${active.providerId}`, data: { current, history }, onOpen: onRaw }) : null,
|
||||
actions: current || active.systemStale ? h(RawButton, { title: `System ${active.providerId}`, data: { current, lastKnown: active.systemLastKnown, currentCollectedAt: active.systemCurrentCollectedAt, stale: active.systemStale, staleSeconds: active.systemStaleSeconds, history }, onOpen: onRaw }) : null,
|
||||
},
|
||||
!current ? h(EmptyState, { title: "系统指标未上报", text: "provider-gateway 会周期性采集 /proc 与 df，并保存历史曲线" }) :
|
||||
!current ? h(EmptyState, { title: active.systemStale ? "系统指标已过期" : "系统指标未上报", text: active.systemStale ? `${staleText}；等待 provider-gateway 恢复 system.status 上报` : "provider-gateway 会周期性采集 /proc 与 df，并保存历史曲线" }) :
|
||||
h("div", null,
|
||||
h("div", { className: "monitor-hero" },
|
||||
h("div", null,
|
||||
@@ -854,7 +891,7 @@ function NodeMonitorPage({ nodes, systemStatuses, tasks, onRaw, refresh }: AnyRe
|
||||
h(MetricCard, { label: "CPU 当前", value: fmtPercent(cpu.percent), hint: `history ${points.length} samples`, tone: "ok" }),
|
||||
h(MetricCard, { label: "实际内存", value: fmtBytes(memory.usedBytes), hint: `${fmtPercent(memory.percent)} 不含缓存` }),
|
||||
h(MetricCard, { label: "硬盘已用", value: fmtBytes(disk.usedBytes), hint: fmtPercent(disk.percent) }),
|
||||
h(MetricCard, { label: "更新时间", value: fmtDate(active.systemUpdatedAt || current.collectedAt), hint: active.providerId }),
|
||||
h(MetricCard, { label: "更新时间", value: fmtDate(current.collectedAt || active.systemUpdatedAt), hint: active.providerId }),
|
||||
),
|
||||
h(ProcessResourceTable, { current, onRaw }),
|
||||
),
|
||||
|
||||
@@ -278,6 +278,10 @@ export interface ApiNodeSystemStatus {
|
||||
name: string;
|
||||
nodeStatus: "online" | "offline";
|
||||
current: JsonValue | null;
|
||||
lastKnown?: JsonValue | null;
|
||||
currentCollectedAt?: string | null;
|
||||
stale?: boolean;
|
||||
staleSeconds?: number | null;
|
||||
history: JsonValue[];
|
||||
updatedAt: string | null;
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user