diff --git a/docs/reference/cli.md b/docs/reference/cli.md index 721b72c2..6210dbcc 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -60,7 +60,7 @@ CI/CD、GitOps、rollout、artifact 发布、PR 合并后的 DEV/PROD 滚动、P - `hwlab g14 control-plane cleanup-released-pvs --lane all [--limit N] [--dry-run|--confirm]` 是 local-path 未自动回收后的补充 retention 入口；只列并删除 `Released`、`local-path`、`Delete`、`claimNamespace=hwlab-ci` 且 claim 名称形如 Tekton 临时 `pvc-*` 的 PV。 - `hwlab g14 git-mirror status|apply|sync|flush [--dry-run|--confirm]` 是 `devops-infra` git mirror/relay 的受控维护入口：`apply` 渲染并 server-side apply `devops-infra/git-mirror.yaml`，同时删除遗留 `git-mirror-hwlab-sync` CronJob；`sync` 创建一次性 manual Job，把 GitHub allowlist refs 拉入本地 mirror；`flush` 创建一次性 manual Job，把本地 `v0.2-gitops` 快进推回 GitHub。 `status` 返回 read/write URL、last sync/write/flush、本地 ref、GitHub staging ref 和 pending flush 状态，并在 `cache.summary` 给出 `localV02`、`localGitops`、`githubGitops`、`pendingFlush`、`flushNeeded`、`githubInSync` 和下一条受控 `flushCommand`。confirmed `sync` 和 `flush` 默认创建 `.state/jobs/` 异步 job 并立刻返回可查询状态，只有现场同步调试才显式加 `--wait`；mirror 不设置 CronJob。 -- `hwlab g14 observability status|apply|query [--promql ] [--dry-run|--confirm]` 是 G14 `devops-infra` 共享监控基础设施的受控入口。`apply` 固定安装 Prometheus Operator `v0.91.0`、Prometheus `v3.12.0`、Prometheus 发现 RBAC、`devops-infra` 内 Prometheus 实例和 ClusterIP query Service，并给被允许发现的 workload namespace 打低风险 label；它不把 Prometheus、Grafana 或 Alertmanager 部署到 `hwlab-v02`，也不接管 HWLAB runtime Deployment/Service。`status` 只读汇总 CRD、operator Deployment、Prometheus CR/pod/service、`hwlab-v02` ServiceMonitor/PrometheusRule 和 bounded `up` 查询；`query` 只通过 Kubernetes service proxy 查询 Prometheus，不开放 FRP、NodePort 或 LoadBalancer。长期边界见 `docs/reference/g14-observability-infra.md`。 +- `hwlab g14 observability status|apply|query|targets|boundary|closeout [--lane v02] [--promql ] [--expect-count N] [--expect-value V] [--dry-run|--confirm]` 是 G14 `devops-infra` 共享监控基础设施和 HWLAB v0.2 监控 closeout 的受控入口。`apply` 固定安装 Prometheus Operator `v0.91.0`、Prometheus `v3.12.0`、Prometheus 发现 RBAC、`devops-infra` 内 Prometheus 实例和 ClusterIP query Service，并给被允许发现的 workload namespace 打低风险 label；它不把 Prometheus、Grafana 或 Alertmanager 部署到 `hwlab-v02`，也不接管 HWLAB runtime Deployment/Service。`status` 只读汇总 CRD、operator Deployment、Prometheus CR/pod/service、`hwlab-v02` ServiceMonitor/PrometheusRule 和 bounded `up` 查询；`query` 只通过 Kubernetes service proxy 查询 Prometheus，支持 `--expect-count` / `--expect-value` 输出 `assertion`、bad values 和 missing/extra series；`targets` 汇总 ServiceMonitor/PrometheusRule、metrics sidecar readiness/restart 和三层指标值；`boundary` 验证 workload namespace 没有 Prometheus/Alertmanager，并对 `19666/19667` 公网 `/metrics` 做负向验证；`closeout` 聚合平台 ready、scrape reachable、sidecar serving、business health probe、namespace boundary 和 public metrics exposure 语义结论。长期边界见 `docs/reference/g14-observability-infra.md`。 - `hwlab g14 tools-image status|build --name ci-node-tools --tag [--dockerfile deploy/ci/hwlab-ci-node-tools.Dockerfile] [--dry-run|--confirm]` 是 G14 固定 HWLAB CI tools image 的受控 host build/push 入口；构建和 push 只发生在 G14 host 与本地 registry，不在 master server 构建，也不把 `apk add`/runtime install 塞进 Tekton PipelineRun。 - `trans gh:/owner/repo ...` 把 GitHub issue/PR 映射成只读/受控写入的虚拟文本目录，适合日报、PR 正文和 issue 正文的小补丁维护：`trans gh:/pikasTech/HWLAB ls` 展示 `pr/` 与 `issue/`，`trans gh:/pikasTech/HWLAB/pr ls [--limit N] [--full]` 和 `trans gh:/pikasTech/HWLAB/issue ls [--limit N] [--full]` 展示条目状态、楼层数、正文长度和标题，`trans gh:/pikasTech/HWLAB/pr/507 ls` 展示单个 PR 的一楼正文文件，`trans gh:/pikasTech/HWLAB/505/1 cat|rg|patch-apply` 兼容旧式 issue/PR number route。`patch-apply` 使用 UniDesk 默认 apply-patch v2 的虚拟文件 executor，把正文一楼映射为 `body.md`，写回仍走 `bun scripts/cli.ts gh issue/pr update` 的 guard/concurrency 规则；`rm` 对正文一楼结构化拒绝，避免误删 issue/PR 正文。大正文读取必须展开 UniDesk gh dump 文件，否则 `cat/rg/patch-apply` 会误读为空，这是 `gh:` 虚拟文件接口的 P0 可见性契约。 - `hwlab cd status|audit|preflight|apply --env dev [--dry-run]` 是旧 D601 HWLAB DEV CD 指挥侧 wrapper，仅用于显式 legacy 诊断和迁移对照。默认通过 UniDesk provider `host.ssh` 进入 D601，再调用 HWLAB repo-owned `scripts/dev-cd-apply.mjs`，不内嵌发布 kubectl 逻辑：`status` 汇总固定 CD mirror、Git clean/main/origin-main、`deploy/deploy.json`/artifact catalog/report、D601 native k3s guard 和 CD Lease lock，并用 `scripts/dev-cd-apply.mjs --status --skip-live-verify` 取得 target/promotion 摘要；`audit` 在 k3s/CD 恢复后做只读健康审计，返回有界 JSON 的 blocker 分类、D601 guard/node、SecretRef 存在性、registry 可达性、Lease phase/holder/staleness、deploy.json 与 artifact/workload image 收敛、current Deployment image/revision/rollout、16666/16667 public health commit/readiness 和 DB/runtime durability 摘要；`preflight` 进一步检查必需 SecretRef 对象/键存在性并运行 HWLAB `scripts/dev-cd-apply.mjs --dry-run --skip-live-verify` 受控事务摘要。完整远端 stdout/stderr 写入 D601 `~/.state/unidesk-hwlab-cd//` 和本地 `.state/hwlab-cd//` task dump，stdout 只返回有界摘要。默认 HWLAB CD repo 是 `/home/ubuntu/hwlab_cd`，`/home/ubuntu/hwlab` runner 历史目录不得作为发布真相。wrapper 强制 `KUBECONFIG=/etc/rancher/k3s/k3s.yaml` 并只以这个显式目标作为 gate；显式目标出现 `docker-desktop`、`desktop-control-plane` 或 `127.0.0.1:11700` 信号会结构化拒绝，audit/preflight/apply --dry-run 都必须观察到 node `d601`。真实 apply 只暴露 `scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report` 命令形状并标注 host-commander-only，本 runner 不执行 live apply、rollout、Lease mutation 或 DEV deploy apply。长期规则见 `docs/reference/hwlab.md`。 diff --git a/docs/reference/g14-observability-infra.md b/docs/reference/g14-observability-infra.md index ebcd2ab4..d911db34 100644 --- a/docs/reference/g14-observability-infra.md +++ b/docs/reference/g14-observability-infra.md @@ -101,13 +101,14 @@ Shared monitoring readiness is necessary but not sufficient for closing an appli A durable closeout must include: - `hwlab g14 observability status` or the equivalent controlled infrastructure status showing CRDs, Prometheus Operator and Prometheus Ready in `devops-infra`. -- Explicit PromQL queries for the workload namespace, not only the infrastructure status summary. The result count must match the application spec and every terminal health value must be checked. +- Explicit PromQL assertions for the workload namespace, not only the infrastructure status summary. Use `hwlab g14 observability query --promql --expect-count --expect-value ` so the CLI returns `assertion.ok`, actual count, bad values and missing/extra series instead of requiring manual vector inspection. - For HWLAB v0.2, the current application-owned PromQL checks are `up{namespace="hwlab-v02"}`, `hwlab_service_up{namespace="hwlab-v02"}` and `hwlab_service_health_probe_success{namespace="hwlab-v02"}`. `up=1` proves Prometheus can scrape the sidecar; it does not prove the sidecar can reach the business health endpoint. -- A namespace boundary check proving the workload namespace contains application `ServiceMonitor` / `PodMonitor` / `PrometheusRule` declarations only, with no shared Prometheus or Alertmanager control-plane instance. -- A public ingress negative check proving FRP, edge proxy or public Web/API ports do not expose raw Prometheus text or a Prometheus UI. +- `hwlab g14 observability targets --lane v02` for the high-level target view: discovered service/pod, metrics sidecar readiness and restart count, selected monitor declarations and the latest `up` / `hwlab_service_up` / `hwlab_service_health_probe_success` values. +- `hwlab g14 observability boundary --lane v02` for the namespace and public ingress boundary: the workload namespace may contain application `ServiceMonitor` / `PodMonitor` / `PrometheusRule` declarations only, must not contain shared Prometheus or Alertmanager instances, and public `19666/19667` `/metrics` must be denied or non-Prometheus text. +- `hwlab g14 observability closeout --lane v02` as the standard monitoring closeout summary. It should report semantic fields such as `platformReady`, `scrapeReachable`, `sidecarServing`, `businessHealthProbe`, `namespaceControlPlaneBoundary` and `publicMetricsExposure`, plus bounded drill-down evidence and next diagnostic commands on failure. Public `/metrics` denial is represented as `publicMetricsExposure=pass` with `publicMetricsExposureState=denied`. - CI/CD and GitOps provenance when the workload desired state changed. For HWLAB v0.2 this includes the target source commit, PipelineRun, Argo sync revision and git mirror `pendingFlush=false` / `githubInSync=true`. -Issue comments should lead with the semantic conclusion and then list the commands, result counts and target values. A raw metrics dump or a green `status` command alone is not a closeout. +Issue comments should lead with the semantic conclusion and then list the commands, result counts and target values. A raw metrics dump or a green `status` command alone is not a closeout, and CI/CD provenance still comes from `hwlab g14 control-plane closeout --lane v02 --source-commit ` or the equivalent high-level control-plane entry when runtime desired state changed. ## Failure Modes diff --git a/scripts/hwlab-g14-contract-test.ts b/scripts/hwlab-g14-contract-test.ts index bb913ae4..0846d96d 100644 --- a/scripts/hwlab-g14-contract-test.ts +++ b/scripts/hwlab-g14-contract-test.ts @@ -1,4 +1,5 @@ -import { activeV02PipelineRuns, gitMirrorFlushJobManifest, gitMirrorStatusSummary, gitMirrorSyncJobManifest, gitMirrorV02SyncRequirement, hwlabG14Help, hwlabG14MonitorStateFileName, parseGitMirrorStatusRefs, parsePipelineTaskRunMetrics, parseV02TriggerSnapshot, rolloutRecordBody, semanticChangelogBullets, summarizeV02CdStatus, v02CloseoutVerdict, v02CommitAlignment, v02ControlPlaneRefreshScriptHash, v02ControlPlaneRenderScript, v02ExistingPipelineRunReuseDecision, v02FalseGreenGuard, v02GitMirrorPreSyncWaitMs, v02LatestOnlyTargetValidation, v02PipelineServiceIds, v02PrAutomationCommentBody, v02ReusableGitMirrorPreSyncMarker, v02ReusableRefreshMarker, v02TaskRunPerformanceSummary } from "./src/hwlab-g14"; +import { activeV02PipelineRuns, g14ObservabilityQueryAssertion, gitMirrorFlushJobManifest, gitMirrorStatusSummary, gitMirrorSyncJobManifest, gitMirrorV02SyncRequirement, hwlabG14Help, hwlabG14MonitorStateFileName, parseGitMirrorStatusRefs, parsePipelineTaskRunMetrics, parseV02TriggerSnapshot, rolloutRecordBody, semanticChangelogBullets, summarizeV02CdStatus, v02CloseoutVerdict, v02CommitAlignment, v02ControlPlaneRefreshScriptHash, v02ControlPlaneRenderScript, v02ExistingPipelineRunReuseDecision, v02FalseGreenGuard, v02GitMirrorPreSyncWaitMs, v02LatestOnlyTargetValidation, v02PipelineServiceIds, v02PrAutomationCommentBody, v02ReusableGitMirrorPreSyncMarker, v02ReusableRefreshMarker, v02TaskRunPerformanceSummary } from "./src/hwlab-g14"; +import { runCommand } from "./src/command"; function assertCondition(condition: unknown, message: string, detail: unknown = {}): void { if (!condition) throw new Error(`${message}: ${JSON.stringify(detail)}`); @@ -60,6 +61,59 @@ assertCondition( "v0.2 help must expose the controlled OpenFGA upstream image mirroring path", hwlabHelpUsage, ); +assertCondition( + hwlabHelpUsage.some((line) => line.includes("observability query --promql 'up{namespace=\"hwlab-v02\"}' --expect-count 6 --expect-value 1")) + && hwlabHelpUsage.some((line) => line.includes("observability targets --lane v02")) + && hwlabHelpUsage.some((line) => line.includes("observability boundary --lane v02")) + && hwlabHelpUsage.some((line) => line.includes("observability closeout --lane v02")), + "observability help must expose assertion, target, boundary, and closeout entrypoints", + hwlabHelpUsage, +); + +const observabilityAssertionPass = g14ObservabilityQueryAssertion({ + status: "success", + data: { + resultType: "vector", + result: [ + { metric: { job: "hwlab-cloud-api", namespace: "hwlab-v02", pod: "api-pod", hwlab_pikastech_local_service_id: "hwlab-cloud-api" }, value: [1, "1"] }, + { metric: { job: "hwlab-cloud-web", namespace: "hwlab-v02", pod: "web-pod", hwlab_pikastech_local_service_id: "hwlab-cloud-web" }, value: [1, "1.0"] }, + ], + }, +}, 2, "1"); +const observabilityAssertionFail = g14ObservabilityQueryAssertion({ + status: "success", + data: { + resultType: "vector", + result: [ + { metric: { job: "hwlab-cloud-api", namespace: "hwlab-v02", pod: "api-pod", hwlab_pikastech_local_service_id: "hwlab-cloud-api" }, value: [1, "0"] }, + ], + }, +}, 2, "1"); +assertCondition( + record(observabilityAssertionPass).ok === true + && record(observabilityAssertionPass).actualCount === 2 + && record(observabilityAssertionPass).valueOk === true, + "observability query assertion must pass matching vector count and numeric string values", + observabilityAssertionPass, +); +assertCondition( + record(observabilityAssertionFail).ok === false + && record(observabilityAssertionFail).actualCount === 1 + && Array.isArray(record(observabilityAssertionFail).badValues) + && Array.isArray(record(observabilityAssertionFail).missingSeries) + && record(observabilityAssertionFail).badValueCount === 1, + "observability query assertion must report bad values and missing series instead of requiring manual vector inspection", + observabilityAssertionFail, +); +const unsupportedObservabilityOption = runCommand(["bun", "scripts/cli.ts", "hwlab", "g14", "observability", "query", "--not-a-real-option"], process.cwd(), { timeoutMs: 30_000 }); +const unsupportedObservabilityJson = JSON.parse(unsupportedObservabilityOption.stdout) as unknown; +assertCondition( + unsupportedObservabilityOption.exitCode !== 0 + && record(unsupportedObservabilityJson).ok === false + && String(record(record(unsupportedObservabilityJson).error).message ?? "").includes("unsupported observability option"), + "observability CLI must fail visibly on unsupported options instead of silently ignoring friction-prone flags", + unsupportedObservabilityJson, +); const v02CommentBody = v02PrAutomationCommentBody({ pr: { @@ -683,6 +737,10 @@ console.log(JSON.stringify({ "once jobs do not overwrite long-running monitor state", "dry-run jobs do not overwrite live monitor state", "once dry-run jobs have a distinct diagnostic state file", + "v0.2 PR monitor state pointers do not overwrite legacy G14 monitor pointers", + "observability help exposes assertion, target, boundary, and closeout entrypoints", + "observability query assertions report count and terminal value pass/fail", + "observability CLI rejects unsupported options with visible JSON errors", "git mirror sync is a manual devops-infra Job, not a CronJob", "git mirror flush is a manual devops-infra Job, not a CronJob", "trigger-current can decide whether v0.2 git mirror pre-sync is required", diff --git a/scripts/src/hwlab-g14.ts b/scripts/src/hwlab-g14.ts index 1af3426d..a3f461d5 100644 --- a/scripts/src/hwlab-g14.ts +++ b/scripts/src/hwlab-g14.ts @@ -62,6 +62,28 @@ const V02_SERVICE_IDS = [ ]; const V02_CLOUD_WEB_URL = "http://74.48.78.17:19666"; const V02_CLOUD_API_URL = "http://74.48.78.17:19667"; +const V02_OBSERVABILITY_SERVICE_IDS = [ + "hwlab-agent-skills", + "hwlab-cloud-api", + "hwlab-cloud-web", + "hwlab-deepseek-proxy", + "hwlab-device-pod", + "hwlab-edge-proxy", +]; +const V02_OBSERVABILITY_EXPECTED_TARGET_COUNT = V02_OBSERVABILITY_SERVICE_IDS.length; +const V02_OBSERVABILITY_QUERIES = { + scrapeReachable: 'up{namespace="hwlab-v02"}', + sidecarServing: 'hwlab_service_up{namespace="hwlab-v02"}', + businessHealthProbe: 'hwlab_service_health_probe_success{namespace="hwlab-v02"}', + healthProbeConfigured: 'hwlab_service_health_probe_configured{namespace="hwlab-v02"}', + healthProbeStatusCode: 'hwlab_service_health_probe_status_code{namespace="hwlab-v02"}', + healthProbeDurationSeconds: 'hwlab_service_health_probe_duration_seconds{namespace="hwlab-v02"}', + processUptimeSeconds: 'hwlab_service_process_uptime_seconds{namespace="hwlab-v02"}', + scrapeDurationSeconds: 'scrape_duration_seconds{namespace="hwlab-v02"}', + scrapeSamplesScraped: 'scrape_samples_scraped{namespace="hwlab-v02"}', +}; +const V02_OBSERVABILITY_CLOSEOUT_QUERY_NAMES = ["scrapeReachable", "sidecarServing", "businessHealthProbe"] as const; +const V02_OBSERVABILITY_BOOLEAN_QUERY_NAMES = [...V02_OBSERVABILITY_CLOSEOUT_QUERY_NAMES, "healthProbeConfigured"]; export function v02PipelineServiceIds(): string[] { return [...V02_SERVICE_IDS]; @@ -155,12 +177,15 @@ interface G14GitMirrorOptions { } interface G14ObservabilityOptions { - action: "status" | "apply" | "query"; + action: "status" | "apply" | "query" | "targets" | "boundary" | "closeout"; + lane: "v02"; dryRun: boolean; confirm: boolean; wait: boolean; timeoutSeconds: number; query: string; + expectCount?: number; + expectValue?: string; } interface G14SecretOptions { @@ -427,22 +452,55 @@ function parseGitMirrorOptions(args: string[]): G14GitMirrorOptions { function parseObservabilityOptions(args: string[]): G14ObservabilityOptions { const [actionRaw] = args; - if (actionRaw !== "status" && actionRaw !== "apply" && actionRaw !== "query") { - throw new Error("observability usage: status|apply|query [--promql ] [--dry-run|--confirm]"); + if ( + actionRaw !== "status" && + actionRaw !== "apply" && + actionRaw !== "query" && + actionRaw !== "targets" && + actionRaw !== "boundary" && + actionRaw !== "closeout" + ) { + throw new Error("observability usage: status|apply|query|targets|boundary|closeout [--lane v02] [--promql ] [--expect-count N] [--expect-value V] [--dry-run|--confirm]"); } + const valueOptions = new Set(["--lane", "--promql", "--query", "--expect-count", "--expect-value", "--timeout-seconds"]); + const booleanOptions = new Set(["--confirm", "--dry-run"]); + for (let i = 1; i < args.length; i += 1) { + const arg = args[i]; + if (valueOptions.has(arg)) { + i += 1; + if (i >= args.length || args[i].startsWith("--")) throw new Error(`${arg} requires a value`); + continue; + } + if (booleanOptions.has(arg)) continue; + if (arg === "--wait") throw new Error("observability does not support --wait; commands are bounded short CLI calls"); + if (arg.startsWith("--")) throw new Error(`unsupported observability option: ${arg}`); + throw new Error(`unexpected observability argument: ${arg}`); + } + const lane = optionValue(args, "--lane") ?? "v02"; + if (lane !== "v02") throw new Error("observability currently supports --lane v02"); const confirm = args.includes("--confirm"); const explicitDryRun = args.includes("--dry-run"); if (confirm && explicitDryRun) throw new Error("observability accepts only one of --confirm or --dry-run"); + if ((confirm || explicitDryRun) && actionRaw !== "apply") throw new Error("--confirm and --dry-run are only supported by observability apply"); const query = optionValue(args, "--promql") ?? optionValue(args, "--query") ?? 'up{namespace="hwlab-v02"}'; if (query.length > 500) throw new Error("--promql is limited to 500 characters"); if (query.includes("\n") || query.includes("\r")) throw new Error("--promql must be a single-line expression"); + const expectCountRaw = optionValue(args, "--expect-count"); + const expectCount = expectCountRaw === undefined ? undefined : Number(expectCountRaw); + if (expectCountRaw !== undefined && (!Number.isInteger(expectCount) || expectCount < 0)) throw new Error("--expect-count must be a non-negative integer"); + const expectValue = optionValue(args, "--expect-value"); + if ((expectCount !== undefined || expectValue !== undefined) && actionRaw !== "query") throw new Error("--expect-count and --expect-value are only supported by observability query"); + if (expectValue !== undefined && expectValue.trim().length === 0) throw new Error("--expect-value must not be blank"); return { action: actionRaw, + lane, confirm, wait: args.includes("--wait"), - dryRun: actionRaw === "status" || actionRaw === "query" ? true : explicitDryRun || !confirm, + dryRun: actionRaw === "apply" ? explicitDryRun || !confirm : true, timeoutSeconds: positiveIntegerOption(args, "--timeout-seconds", actionRaw === "apply" ? 240 : 120, 900), query, + expectCount, + expectValue, }; } @@ -556,6 +614,18 @@ function commandData(result: CommandJsonResult): Record { return record(sanitizeCliOutput(record(expandedParsedRoot(result).data))); } +function compactCommandMetadata(result: CommandJsonResult, includeTail = false, includeCommand = false): Record { + return { + ok: result.ok, + exitCode: result.exitCode, + command: includeCommand ? redactedCommand(result.command) : undefined, + stdoutBytes: Buffer.byteLength(result.stdout, "utf8"), + stderrBytes: Buffer.byteLength(result.stderr, "utf8"), + stdoutTail: includeTail ? compactInlineText(result.stdout, 4000) : undefined, + stderrTail: includeTail ? compactInlineText(result.stderr, 4000) : undefined, + }; +} + function textHash(value: string): string { return createHash("sha256").update(value).digest("hex").slice(0, 12); } @@ -4643,6 +4713,523 @@ function prometheusReady(prometheus: Record): boolean { return available === "True" || reconciled === "True"; } +function observabilityQueryPath(promql: string): string { + return `/api/v1/namespaces/${G14_OBSERVABILITY_NAMESPACE}/services/http:${G14_PROMETHEUS_SERVICE}:9090/proxy/api/v1/query?query=${encodeURIComponent(promql)}`; +} + +function arrayRecords(value: unknown): Record[] { + return Array.isArray(value) ? value.map((item) => record(item)) : []; +} + +function prometheusResultItems(parsed: unknown): Record[] { + return arrayRecords(nested(parsed, ["data", "result"])); +} + +function prometheusSeriesValue(series: Record): string | null { + const value = series.value; + if (Array.isArray(value) && value.length >= 2) return String(value[1]); + if (typeof value === "string" || typeof value === "number") return String(value); + return null; +} + +function prometheusSeriesTimestamp(series: Record): number | null { + const value = series.value; + if (!Array.isArray(value) || value.length === 0) return null; + return numericValue(value[0]); +} + +export function parseK8sCpuMillicores(value: unknown): number | null { + const raw = stringOrNull(value); + if (raw === null) return null; + const text = raw.trim(); + if (text.length === 0) return null; + const match = text.match(/^([0-9]+(?:\.[0-9]+)?)(n|u|m)?$/u); + if (match === null) return null; + const amount = Number(match[1]); + if (!Number.isFinite(amount)) return null; + const unit = match[2] ?? ""; + if (unit === "n") return amount / 1_000_000; + if (unit === "u") return amount / 1_000; + if (unit === "m") return amount; + return amount * 1000; +} + +export function parseK8sMemoryMiB(value: unknown): number | null { + const raw = stringOrNull(value); + if (raw === null) return null; + const text = raw.trim(); + if (text.length === 0) return null; + const match = text.match(/^([0-9]+(?:\.[0-9]+)?)(Ei|Pi|Ti|Gi|Mi|Ki|E|P|T|G|M|K)?$/u); + if (match === null) return null; + const amount = Number(match[1]); + if (!Number.isFinite(amount)) return null; + const unit = match[2] ?? ""; + const binaryFactors: Record = { + Ki: 1 / 1024, + Mi: 1, + Gi: 1024, + Ti: 1024 * 1024, + Pi: 1024 * 1024 * 1024, + Ei: 1024 * 1024 * 1024 * 1024, + }; + const decimalByteFactors: Record = { + "": 1, + K: 1000, + M: 1000 * 1000, + G: 1000 * 1000 * 1000, + T: 1000 * 1000 * 1000 * 1000, + P: 1000 * 1000 * 1000 * 1000 * 1000, + E: 1000 * 1000 * 1000 * 1000 * 1000 * 1000, + }; + if (binaryFactors[unit] !== undefined) return amount * binaryFactors[unit]; + const byteFactor = decimalByteFactors[unit]; + return byteFactor === undefined ? null : (amount * byteFactor) / (1024 * 1024); +} + +function prometheusSeriesSummary(series: Record): Record { + const metric = record(series.metric); + return { + metricName: stringOrNull(metric.__name__), + serviceId: stringOrNull(metric.hwlab_pikastech_local_service_id), + service: stringOrNull(metric.service), + job: stringOrNull(metric.job), + pod: stringOrNull(metric.pod), + namespace: stringOrNull(metric.namespace), + container: stringOrNull(metric.container), + instance: stringOrNull(metric.instance), + value: prometheusSeriesValue(series), + timestamp: prometheusSeriesTimestamp(series), + }; +} + +function prometheusSeriesSummaries(parsed: unknown, limit = 30): Record[] { + return prometheusResultItems(parsed).slice(0, limit).map(prometheusSeriesSummary); +} + +function metricValueMatches(actual: string | null, expected: string): boolean { + if (actual === expected) return true; + const actualNumber = actual === null ? Number.NaN : Number(actual); + const expectedNumber = Number(expected); + return Number.isFinite(actualNumber) && Number.isFinite(expectedNumber) && actualNumber === expectedNumber; +} + +export function g14ObservabilityQueryAssertion(parsed: unknown, expectedCount?: number, expectedValue?: string): Record | null { + if (expectedCount === undefined && expectedValue === undefined) return null; + const series = prometheusResultItems(parsed); + const actualCount = series.length; + const countOk = expectedCount === undefined || actualCount === expectedCount; + const badValues = expectedValue === undefined + ? [] + : series + .filter((item) => !metricValueMatches(prometheusSeriesValue(item), expectedValue)) + .map(prometheusSeriesSummary); + const missingCount = expectedCount === undefined ? 0 : Math.max(0, expectedCount - actualCount); + const extraCount = expectedCount === undefined ? 0 : Math.max(0, actualCount - expectedCount); + const queryStatus = stringOrNull(record(parsed).status); + return { + ok: queryStatus === "success" && countOk && badValues.length === 0, + queryStatus, + expectedCount: expectedCount ?? null, + actualCount, + expectedValue: expectedValue ?? null, + countOk, + valueOk: expectedValue === undefined ? null : badValues.length === 0, + badValueCount: badValues.length, + badValues: badValues.slice(0, 20), + missingSeries: missingCount > 0 ? [{ expectedCount, actualCount, missingCount }] : [], + extraSeries: extraCount > 0 ? prometheusSeriesSummaries(parsed, Math.min(extraCount, 20)) : [], + }; +} + +function observabilitySemanticName(promql: string): string | null { + if (promql === V02_OBSERVABILITY_QUERIES.scrapeReachable) return "scrapeReachable"; + if (promql === V02_OBSERVABILITY_QUERIES.sidecarServing) return "sidecarServing"; + if (promql === V02_OBSERVABILITY_QUERIES.businessHealthProbe) return "businessHealthProbe"; + return null; +} + +function k8sItems(parsed: unknown): Record[] { + const root = record(parsed); + return Array.isArray(root.items) ? root.items.map((item) => record(item)) : []; +} + +function k8sMetadataName(item: Record): string | null { + return stringOrNull(record(item.metadata).name); +} + +function k8sLabels(item: Record): Record { + return record(record(item.metadata).labels); +} + +function podReadyCondition(pod: Record): boolean | null { + const conditions = arrayRecords(record(pod.status).conditions); + const ready = conditions.find((item) => item.type === "Ready"); + return ready === undefined ? null : ready.status === "True"; +} + +function podMetricsSidecarSummary(pod: Record): Record { + const statuses = arrayRecords(record(pod.status).containerStatuses); + const metrics = statuses.find((item) => item.name === "hwlab-metrics") + ?? statuses.find((item) => String(item.name ?? "").includes("metrics")); + return { + pod: k8sMetadataName(pod), + namespace: stringOrNull(record(pod.metadata).namespace), + serviceId: stringOrNull(k8sLabels(pod)["hwlab.pikastech.local/service-id"]) + ?? stringOrNull(k8sLabels(pod)["app.kubernetes.io/name"]), + phase: stringOrNull(record(pod.status).phase), + podReady: podReadyCondition(pod), + sidecarFound: metrics !== undefined, + sidecarName: metrics === undefined ? null : stringOrNull(metrics.name), + sidecarReady: metrics === undefined ? null : metrics.ready === true, + restartCount: metrics === undefined ? null : numericValue(metrics.restartCount), + }; +} + +function resourceUsageSummary(containers: Record[]): Record { + const rows = containers.map((container) => { + const usage = record(container.usage); + return { + name: stringOrNull(container.name), + cpuMillicores: parseK8sCpuMillicores(usage.cpu), + memoryMiB: parseK8sMemoryMiB(usage.memory), + }; + }); + const sum = (key: "cpuMillicores" | "memoryMiB"): number | null => { + const values = rows.map((item) => item[key]).filter((item): item is number => item !== null); + return values.length === 0 ? null : values.reduce((acc, item) => acc + item, 0); + }; + return { + cpuMillicores: sum("cpuMillicores"), + memoryMiB: sum("memoryMiB"), + containers: rows, + }; +} + +function podResourceMetricSummary(item: Record): Record { + const containers = arrayRecords(item.containers); + const metricsSidecars = containers.filter((container) => stringOrNull(container.name) === "hwlab-metrics"); + const businessContainers = containers.filter((container) => stringOrNull(container.name) !== "hwlab-metrics"); + return { + pod: k8sMetadataName(item), + namespace: stringOrNull(record(item.metadata).namespace), + serviceId: stringOrNull(k8sLabels(item)["hwlab.pikastech.local/service-id"]) + ?? stringOrNull(k8sLabels(item)["app.kubernetes.io/name"]), + timestamp: stringOrNull(item.timestamp), + window: stringOrNull(item.window), + total: resourceUsageSummary(containers), + business: resourceUsageSummary(businessContainers), + metricsSidecar: resourceUsageSummary(metricsSidecars), + }; +} + +function monitorSummaries(items: Record[]): Record[] { + return items.map((item) => ({ + kind: stringOrNull(item.kind), + name: k8sMetadataName(item), + labels: { + monitoring: stringOrNull(k8sLabels(item)["hwlab.pikastech.local/monitoring"]), + serviceId: stringOrNull(k8sLabels(item)["hwlab.pikastech.local/service-id"]), + gitopsTarget: stringOrNull(k8sLabels(item)["hwlab.pikastech.local/gitops-target"]), + }, + })); +} + +function monitoredServiceIds(monitors: Record[], metricSeries: Record[]>): string[] { + const ids = new Set(); + for (const item of monitorSummaries(monitors)) { + const kind = stringOrNull(item.kind); + const serviceId = stringOrNull(record(item.labels).serviceId); + if ((kind === "ServiceMonitor" || kind === "PodMonitor") && serviceId !== null) ids.add(serviceId); + } + for (const seriesItems of Object.values(metricSeries)) { + for (const item of seriesItems.map(prometheusSeriesSummary)) { + const serviceId = stringOrNull(item.serviceId) ?? stringOrNull(item.service) ?? stringOrNull(item.job); + if (serviceId !== null) ids.add(serviceId); + } + } + return [...ids].sort(); +} + +function attachResourceMetricsToTargets(targets: Record[], resources: Record[]): Record[] { + const byPod = new Map(resources.map((item) => [String(item.pod ?? ""), item])); + const byServiceId = new Map(resources.map((item) => [String(item.serviceId ?? ""), item])); + return targets.map((target) => { + const resource = byPod.get(String(target.pod ?? "")) ?? byServiceId.get(String(target.serviceId ?? "")); + return resource === undefined ? target : { ...target, resource }; + }); +} + +function targetMetricValue(target: Record, metricName: string): number | null { + const metric = record(record(target.metrics)[metricName]); + return numericValue(metric.value); +} + +function targetMetricString(target: Record, metricName: string): string | null { + return stringOrNull(record(record(target.metrics)[metricName]).value); +} + +function targetMetricTimestamp(target: Record, metricName: string): number | null { + return numericValue(record(record(target.metrics)[metricName]).timestamp); +} + +function sortedNumeric(values: number[]): number[] { + return [...values].filter((item) => Number.isFinite(item)).sort((a, b) => a - b); +} + +function percentile(values: number[], p: number): number | null { + const sorted = sortedNumeric(values); + if (sorted.length === 0) return null; + if (sorted.length === 1) return sorted[0]; + const index = Math.min(sorted.length - 1, Math.max(0, Math.ceil((p / 100) * sorted.length) - 1)); + return sorted[index]; +} + +function secondsStats(values: number[]): Record { + const sorted = sortedNumeric(values); + if (sorted.length === 0) return { count: 0, minSeconds: null, maxSeconds: null, avgSeconds: null, p50Seconds: null, p95Seconds: null, p99Seconds: null }; + const sum = sorted.reduce((acc, item) => acc + item, 0); + return { + count: sorted.length, + minSeconds: sorted[0], + maxSeconds: sorted[sorted.length - 1], + avgSeconds: sum / sorted.length, + p50Seconds: percentile(sorted, 50), + p95Seconds: percentile(sorted, 95), + p99Seconds: percentile(sorted, 99), + }; +} + +function numberStats(values: number[]): Record { + const sorted = sortedNumeric(values); + if (sorted.length === 0) return { count: 0, min: null, max: null, avg: null, p50: null, p95: null, p99: null }; + const sum = sorted.reduce((acc, item) => acc + item, 0); + return { + count: sorted.length, + min: sorted[0], + max: sorted[sorted.length - 1], + avg: sum / sorted.length, + p50: percentile(sorted, 50), + p95: percentile(sorted, 95), + p99: percentile(sorted, 99), + }; +} + +function buildObservabilityLevelSummary(targets: Record[]): Record { + const observedTargets = targets.filter((target) => Object.keys(record(target.metrics)).length > 0); + const healthDurations = observedTargets.map((target) => targetMetricValue(target, "healthProbeDurationSeconds")).filter((item): item is number => item !== null); + const scrapeDurations = observedTargets.map((target) => targetMetricValue(target, "scrapeDurationSeconds")).filter((item): item is number => item !== null); + const samplesScraped = observedTargets.map((target) => targetMetricValue(target, "scrapeSamplesScraped")).filter((item): item is number => item !== null); + const resourceSnapshots = observedTargets.map((target) => record(target.resource)).filter((item) => Object.keys(item).length > 0); + const resourceVisible = resourceSnapshots.length === V02_OBSERVABILITY_EXPECTED_TARGET_COUNT; + const resourceValue = (snapshot: Record, path: string[]): number | null => numericValue(nested(snapshot, path)); + const resourceTotals = { + totalCpuMillicores: resourceSnapshots.map((item) => resourceValue(item, ["total", "cpuMillicores"])).filter((item): item is number => item !== null), + totalMemoryMiB: resourceSnapshots.map((item) => resourceValue(item, ["total", "memoryMiB"])).filter((item): item is number => item !== null), + businessCpuMillicores: resourceSnapshots.map((item) => resourceValue(item, ["business", "cpuMillicores"])).filter((item): item is number => item !== null), + businessMemoryMiB: resourceSnapshots.map((item) => resourceValue(item, ["business", "memoryMiB"])).filter((item): item is number => item !== null), + metricsSidecarCpuMillicores: resourceSnapshots.map((item) => resourceValue(item, ["metricsSidecar", "cpuMillicores"])).filter((item): item is number => item !== null), + metricsSidecarMemoryMiB: resourceSnapshots.map((item) => resourceValue(item, ["metricsSidecar", "memoryMiB"])).filter((item): item is number => item !== null), + }; + const healthRows = observedTargets.map((target) => ({ + serviceId: target.serviceId ?? target.service ?? target.job ?? target.key, + scrapeReachable: targetMetricString(target, "scrapeReachable"), + sidecarServing: targetMetricString(target, "sidecarServing"), + businessHealthProbe: targetMetricString(target, "businessHealthProbe"), + statusCode: targetMetricValue(target, "healthProbeStatusCode"), + healthProbeDurationSeconds: targetMetricValue(target, "healthProbeDurationSeconds"), + scrapeDurationSeconds: targetMetricValue(target, "scrapeDurationSeconds"), + scrapeSamplesScraped: targetMetricValue(target, "scrapeSamplesScraped"), + processUptimeSeconds: targetMetricValue(target, "processUptimeSeconds"), + totalCpuMillicores: resourceValue(record(target.resource), ["total", "cpuMillicores"]), + totalMemoryMiB: resourceValue(record(target.resource), ["total", "memoryMiB"]), + businessCpuMillicores: resourceValue(record(target.resource), ["business", "cpuMillicores"]), + businessMemoryMiB: resourceValue(record(target.resource), ["business", "memoryMiB"]), + metricsSidecarCpuMillicores: resourceValue(record(target.resource), ["metricsSidecar", "cpuMillicores"]), + metricsSidecarMemoryMiB: resourceValue(record(target.resource), ["metricsSidecar", "memoryMiB"]), + lastSampleTimestamp: targetMetricTimestamp(target, "scrapeReachable") ?? targetMetricTimestamp(target, "businessHealthProbe"), + })).sort((a, b) => String(a.serviceId).localeCompare(String(b.serviceId))); + const unsupportedSignals = [ + "request_rate", + "error_rate", + "route_latency_p95", + "route_latency_p99", + "business_operation_latency", + ...(resourceVisible ? [] : ["cpu_memory_node_pod_usage"]), + ]; + return { + observedTargetCount: observedTargets.length, + expectedTargetCount: V02_OBSERVABILITY_EXPECTED_TARGET_COUNT, + healthProbeDuration: secondsStats(healthDurations), + scrapeDuration: secondsStats(scrapeDurations), + scrapeSamplesScraped: { + count: samplesScraped.length, + min: sortedNumeric(samplesScraped)[0] ?? null, + max: sortedNumeric(samplesScraped)[sortedNumeric(samplesScraped).length - 1] ?? null, + values: samplesScraped, + }, + resourceUsage: { + source: "metrics.k8s.io/v1beta1", + observedTargetCount: resourceSnapshots.length, + expectedTargetCount: V02_OBSERVABILITY_EXPECTED_TARGET_COUNT, + totalCpuMillicores: numberStats(resourceTotals.totalCpuMillicores), + totalMemoryMiB: numberStats(resourceTotals.totalMemoryMiB), + businessCpuMillicores: numberStats(resourceTotals.businessCpuMillicores), + businessMemoryMiB: numberStats(resourceTotals.businessMemoryMiB), + metricsSidecarCpuMillicores: numberStats(resourceTotals.metricsSidecarCpuMillicores), + metricsSidecarMemoryMiB: numberStats(resourceTotals.metricsSidecarMemoryMiB), + }, + services: healthRows, + interpretation: { + visible: [ + "scrape availability", + "metrics sidecar availability", + "business health endpoint success", + "business health endpoint status code", + "business health probe duration", + "Prometheus scrape duration", + "metrics sidecar uptime", + ...(resourceVisible ? ["current pod/container CPU and memory snapshot from metrics.k8s.io"] : []), + ], + notYetVisible: unsupportedSignals, + conclusion: unsupportedSignals.length > 0 + ? "current metrics are sufficient for availability, synthetic health latency, and current resource snapshot when metrics.k8s.io is available, but not yet sufficient for real request throughput, route latency, or error rate analysis" + : "current metrics cover the requested performance surface", + }, + }; +} + +function compactObservabilityTarget(target: Record): Record { + const sidecar = record(target.sidecar); + return { + serviceId: target.serviceId ?? target.service ?? target.job ?? target.key, + pod: target.pod ?? sidecar.pod ?? null, + serviceMonitors: target.serviceMonitors ?? [], + sidecar: { + found: sidecar.sidecarFound ?? null, + ready: sidecar.sidecarReady ?? null, + podReady: sidecar.podReady ?? null, + restartCount: sidecar.restartCount ?? null, + }, + metrics: { + scrapeReachable: targetMetricString(target, "scrapeReachable"), + sidecarServing: targetMetricString(target, "sidecarServing"), + businessHealthProbe: targetMetricString(target, "businessHealthProbe"), + healthProbeConfigured: targetMetricString(target, "healthProbeConfigured"), + healthProbeStatusCode: targetMetricValue(target, "healthProbeStatusCode"), + healthProbeDurationSeconds: targetMetricValue(target, "healthProbeDurationSeconds"), + scrapeDurationSeconds: targetMetricValue(target, "scrapeDurationSeconds"), + scrapeSamplesScraped: targetMetricValue(target, "scrapeSamplesScraped"), + processUptimeSeconds: targetMetricValue(target, "processUptimeSeconds"), + lastSampleTimestamp: targetMetricTimestamp(target, "scrapeReachable") ?? targetMetricTimestamp(target, "businessHealthProbe"), + }, + resource: target.resource ?? null, + }; +} + +function mergeObservabilityTargets( + pods: Record[], + monitors: Record[], + metricSeries: Record[]>, +): Record[] { + const byKey = new Map>(); + const ensure = (key: string): Record => { + const existing = byKey.get(key); + if (existing !== undefined) return existing; + const next = { key, serviceId: null, service: null, job: null, pod: null, sidecar: null, metrics: {} as Record }; + byKey.set(key, next); + return next; + }; + for (const pod of pods.map(podMetricsSidecarSummary)) { + const key = String(pod.serviceId ?? pod.pod ?? "unknown"); + const target = ensure(key); + target.serviceId = target.serviceId ?? pod.serviceId; + target.pod = target.pod ?? pod.pod; + target.sidecar = pod; + } + for (const [metricName, seriesItems] of Object.entries(metricSeries)) { + for (const item of seriesItems.map(prometheusSeriesSummary)) { + const key = String(item.serviceId ?? item.service ?? item.job ?? item.pod ?? "unknown"); + const target = ensure(key); + target.serviceId = target.serviceId ?? item.serviceId; + target.service = target.service ?? item.service; + target.job = target.job ?? item.job; + target.pod = target.pod ?? item.pod; + record(target.metrics)[metricName] = { + value: item.value, + timestamp: item.timestamp, + instance: item.instance, + container: item.container, + }; + } + } + const serviceIds = new Set(monitoredServiceIds(monitors, metricSeries)); + const monitorNames = monitorSummaries(monitors); + return [...byKey.values()] + .filter((target) => { + const key = String(target.serviceId ?? target.service ?? target.job ?? target.key ?? ""); + return serviceIds.has(key); + }) + .map((target) => ({ + ...target, + serviceMonitors: monitorNames + .filter((item) => stringOrNull(record(item.labels).serviceId) === stringOrNull(target.serviceId)) + .map((item) => stringOrNull(item.name)) + .filter((name): name is string => name !== null) + .slice(0, 5), + })) + .sort((a, b) => String(a.serviceId ?? a.service ?? a.job ?? a.pod).localeCompare(String(b.serviceId ?? b.service ?? b.job ?? b.pod))); +} + +function publicMetricsProbe(name: string, url: string): Record { + const result = commandJson([ + "curl", + "-sS", + "-L", + "--max-time", + "20", + "--range", + "0-4095", + "-w", + "\n__UNIDESK_HTTP_CODE__:%{http_code}\n__UNIDESK_CONTENT_TYPE__:%{content_type}\n", + url, + ], 30_000); + const text = result.stdout; + const codeMatch = text.match(/\n__UNIDESK_HTTP_CODE__:(\d{3})\n/u); + const contentTypeMatch = text.match(/\n__UNIDESK_CONTENT_TYPE__:(.*)\n?$/u); + const body = codeMatch === null ? text : text.slice(0, codeMatch.index).trim(); + const httpCode = codeMatch === null ? null : Number(codeMatch[1]); + const contentType = contentTypeMatch === null ? "" : contentTypeMatch[1].trim(); + const prometheusText = looksLikePrometheusText(body, contentType); + const exposed = httpCode !== null && httpCode >= 200 && httpCode < 300 && prometheusText; + const denied = httpCode !== null && !exposed && (httpCode >= 400 || !prometheusText); + return { + ok: result.ok && denied, + name, + url, + httpCode, + contentType, + exposed, + denied, + reason: result.ok + ? exposed + ? "prometheus-text-exposed" + : httpCode !== null && httpCode >= 400 + ? "http-error-denied" + : "non-prometheus-body" + : "curl-failed", + bodyPreview: body.slice(0, 240), + commandResult: compactCommandMetadata(result, !result.ok || exposed), + }; +} + +function looksLikePrometheusText(body: string, contentType: string): boolean { + const type = contentType.toLowerCase(); + if (type.includes("openmetrics")) return true; + const hasMetricLine = /(?:^|\n)(?:#\s+(?:HELP|TYPE)\s+[A-Za-z_:][A-Za-z0-9_:]*|[A-Za-z_:][A-Za-z0-9_:]*(?:\{[^}\n]*\})?\s+[-+]?(?:\d+\.?\d*|\.\d+)(?:[eE][-+]?\d+)?)\b/u.test(body); + if (!hasMetricLine) return false; + return type.includes("text/plain") || type.length === 0; +} + function g14ObservabilityStatus(): Record { const startedAtMs = Date.now(); const queryPath = `/api/v1/namespaces/${G14_OBSERVABILITY_NAMESPACE}/services/http:${G14_PROMETHEUS_SERVICE}:9090/proxy/api/v1/query?query=${encodeURIComponent("up")}`; @@ -4696,8 +5283,9 @@ function g14ObservabilityStatus(): Record { || prometheusPods.some((pod) => pod.ready === true) ); const queryOk = sections.query?.exitCode === 0 && query.status === "success"; + const ok = isCommandSuccess(bundle) && requiredCrdsPresent && operatorIsReady && prometheusExists && prometheusIsReady && queryOk; return { - ok: isCommandSuccess(bundle) && requiredCrdsPresent && operatorIsReady && prometheusExists, + ok, command: "hwlab g14 observability status", namespace: G14_OBSERVABILITY_NAMESPACE, mode: "status", @@ -4767,7 +5355,7 @@ function g14ObservabilityStatus(): Record { sectionOk: shellSectionOk(sections.query), stderr: shellSectionOk(sections.query) ? "" : commandErrorSummary(bundle), }, - result: compactCommandResult(bundle), + result: compactCommandMetadata(bundle, !ok), next: requiredCrdsPresent && operatorIsReady && prometheusExists ? { query: 'bun scripts/cli.ts hwlab g14 observability query --promql \'up{namespace="hwlab-v02"}\'' } : { apply: "bun scripts/cli.ts hwlab g14 observability apply --confirm" }, @@ -4866,9 +5454,9 @@ function runG14ObservabilityApply(options: G14ObservabilityOptions): Record { - const serviceProxyPath = `/api/v1/namespaces/${G14_OBSERVABILITY_NAMESPACE}/services/http:${G14_PROMETHEUS_SERVICE}:9090/proxy/api/v1/query?query=${encodeURIComponent(options.query)}`; - const result = g14K3s(["kubectl", "get", "--raw", serviceProxyPath], options.timeoutSeconds * 1000); +function g14PrometheusQuery(promql: string, timeoutMs: number): { serviceProxyPath: string; result: CommandJsonResult; parsed: Record; ok: boolean } { + const serviceProxyPath = observabilityQueryPath(promql); + const result = g14K3s(["kubectl", "get", "--raw", serviceProxyPath], timeoutMs); const parsed = (() => { try { return record(JSON.parse(statusText(result)) as unknown); @@ -4876,25 +5464,318 @@ function runG14ObservabilityQuery(options: G14ObservabilityOptions): Record { + const query = g14PrometheusQuery(options.query, options.timeoutSeconds * 1000); + const assertion = g14ObservabilityQueryAssertion(query.parsed, options.expectCount, options.expectValue); + const semantic = observabilitySemanticName(options.query); + const resultCount = prometheusResultItems(query.parsed).length; + const hasAssertion = options.expectCount !== undefined || options.expectValue !== undefined; + const includeData = !hasAssertion && resultCount <= 20; + const ok = query.ok && (assertion === null || assertion.ok === true); return { - ok: isCommandSuccess(result) && parsed.status === "success", + ok, command: "hwlab g14 observability query", namespace: G14_OBSERVABILITY_NAMESPACE, service: G14_PROMETHEUS_SERVICE, promql: options.query, - serviceProxyPath, - status: parsed.status ?? null, - resultType: nested(parsed, ["data", "resultType"]) ?? null, - resultCount: Array.isArray(nested(parsed, ["data", "result"])) ? (nested(parsed, ["data", "result"]) as unknown[]).length : null, - data: parsed.data ?? null, - raw: Object.keys(parsed).length === 0 ? tailText(statusText(result), 4000) : undefined, - commandResult: compactCommandResult(result), + semantic, + serviceProxyPath: query.serviceProxyPath, + status: query.parsed.status ?? null, + resultType: nested(query.parsed, ["data", "resultType"]) ?? null, + resultCount, + series: prometheusSeriesSummaries(query.parsed, 50), + assertion, + data: includeData ? query.parsed.data ?? null : undefined, + dataOmitted: includeData ? undefined : { + reason: hasAssertion ? "assertion-mode-uses-series-summary" : "result-count-exceeds-default-inline-limit", + resultCount, + inlineLimit: 20, + summaryField: "series", + queryCommand: `bun scripts/cli.ts hwlab g14 observability query --promql ${shellQuote(options.query)}`, + }, + raw: Object.keys(query.parsed).length === 0 ? tailText(statusText(query.result), 4000) : undefined, + commandResult: compactCommandMetadata(query.result, !query.ok, !query.ok), + degradedReason: ok ? undefined : assertion !== null && assertion.ok !== true ? "assertion-failed" : "query-failed", + next: assertion !== null && assertion.ok !== true + ? { targets: "bun scripts/cli.ts hwlab g14 observability targets --lane v02", closeout: "bun scripts/cli.ts hwlab g14 observability closeout --lane v02" } + : undefined, + }; +} + +function runG14ObservabilityTargets(options: G14ObservabilityOptions): Record { + const startedAtMs = Date.now(); + const queryCommands = Object.entries(V02_OBSERVABILITY_QUERIES).map(([name, promql]) => `section query_${name} kubectl get --raw ${shellQuote(observabilityQueryPath(promql))}`); + const script = [ + "set +e", + "section() {", + " name=\"$1\"", + " shift", + " printf '__UNIDESK_SECTION_BEGIN__ %s\\n' \"$name\"", + " \"$@\"", + " code=$?", + " printf '\\n__UNIDESK_SECTION_END__ %s exit=%s\\n' \"$name\" \"$code\"", + "}", + `section pods kubectl get pods -n ${shellQuote(V02_RUNTIME_NAMESPACE)} -o json`, + `section monitors kubectl get servicemonitor,podmonitor,prometheusrule -n ${shellQuote(V02_RUNTIME_NAMESPACE)} -l hwlab.pikastech.local/monitoring=enabled -o json`, + `section podResourceMetrics kubectl get --raw ${shellQuote(`/apis/metrics.k8s.io/v1beta1/namespaces/${V02_RUNTIME_NAMESPACE}/pods`)}`, + ...queryCommands, + ].join("\n"); + const bundle = g14K3s(["script", "--", script], options.timeoutSeconds * 1000); + const sections = parseShellSections(statusText(bundle)); + const pods = k8sItems(parseSectionJson(sections.pods)); + const monitors = k8sItems(parseSectionJson(sections.monitors)); + const podResourceMetrics = k8sItems(parseSectionJson(sections.podResourceMetrics)).map(podResourceMetricSummary); + const parsedQueries = Object.fromEntries(Object.entries(V02_OBSERVABILITY_QUERIES).map(([name]) => [name, parseSectionJson(sections[`query_${name}`])])); + const metricSeries = Object.fromEntries(Object.entries(parsedQueries).map(([name, parsed]) => [name, prometheusResultItems(parsed)])); + const booleanQueryNames = new Set(V02_OBSERVABILITY_BOOLEAN_QUERY_NAMES); + const closeoutQueryNames = new Set(V02_OBSERVABILITY_CLOSEOUT_QUERY_NAMES); + const assertions = Object.fromEntries(Object.entries(parsedQueries).map(([name, parsed]) => [ + name, + g14ObservabilityQueryAssertion(parsed, V02_OBSERVABILITY_EXPECTED_TARGET_COUNT, booleanQueryNames.has(name) ? "1" : undefined), + ])); + const targets = attachResourceMetricsToTargets(mergeObservabilityTargets(pods, monitors, metricSeries), podResourceMetrics); + const compactTargets = targets.map(compactObservabilityTarget); + const levelSummary = buildObservabilityLevelSummary(targets); + const sidecars = targets + .map((target) => record(target.sidecar)) + .filter((sidecar) => Object.keys(sidecar).length > 0); + const notReadySidecars = sidecars.filter((sidecar) => sidecar.sidecarReady !== true || sidecar.podReady !== true); + const restartProblems = sidecars.filter((sidecar) => (numericValue(sidecar.restartCount) ?? 0) > 0); + const queryOk = V02_OBSERVABILITY_CLOSEOUT_QUERY_NAMES.every((name) => record(assertions[name]).ok === true); + const sidecarOk = targets.length === V02_OBSERVABILITY_EXPECTED_TARGET_COUNT + && sidecars.length === V02_OBSERVABILITY_EXPECTED_TARGET_COUNT + && notReadySidecars.length === 0; + const monitorsOk = shellSectionOk(sections.monitors) && monitors.length > 0; + const resourceOk = shellSectionOk(sections.podResourceMetrics) + && podResourceMetrics.filter((item) => { + const serviceId = stringOrNull(item.serviceId); + return serviceId !== null && V02_OBSERVABILITY_SERVICE_IDS.includes(serviceId); + }).length === V02_OBSERVABILITY_EXPECTED_TARGET_COUNT; + const ok = isCommandSuccess(bundle) && shellSectionOk(sections.pods) && monitorsOk && queryOk && sidecarOk && resourceOk; + return { + ok, + command: "hwlab g14 observability targets", + lane: options.lane, + namespace: V02_RUNTIME_NAMESPACE, + elapsedMs: Date.now() - startedAtMs, + expectedTargetCount: V02_OBSERVABILITY_EXPECTED_TARGET_COUNT, + targetCount: targets.length, + levelSummary, + targets: compactTargets, + sidecars: { + ok: sidecarOk, + observedCount: sidecars.length, + readyCount: sidecars.filter((sidecar) => sidecar.sidecarReady === true && sidecar.podReady === true).length, + notReady: notReadySidecars, + restartProblems, + }, + monitors: { + ok: monitorsOk, + count: monitors.length, + items: monitorSummaries(monitors), + sectionOk: shellSectionOk(sections.monitors), + }, + resourceSnapshot: { + ok: resourceOk, + source: "metrics.k8s.io/v1beta1", + observedTargetCount: podResourceMetrics.filter((item) => { + const serviceId = stringOrNull(item.serviceId); + return serviceId !== null && V02_OBSERVABILITY_SERVICE_IDS.includes(serviceId); + }).length, + expectedTargetCount: V02_OBSERVABILITY_EXPECTED_TARGET_COUNT, + sectionOk: shellSectionOk(sections.podResourceMetrics), + stderr: shellSectionOk(sections.podResourceMetrics) ? "" : commandErrorSummary(bundle), + }, + queries: Object.fromEntries(Object.entries(parsedQueries).map(([name, parsed]) => [name, { + promql: record(V02_OBSERVABILITY_QUERIES)[name], + resultType: nested(parsed, ["data", "resultType"]) ?? null, + resultCount: prometheusResultItems(parsed).length, + assertion: assertions[name], + seriesOmitted: { + reason: closeoutQueryNames.has(name) ? "target-summary-uses-assertion-and-target-rows" : "auxiliary-metric-summarized-in-levelSummary", + summaryField: closeoutQueryNames.has(name) ? "targets" : "levelSummary", + drillDownCommand: closeoutQueryNames.has(name) + ? `bun scripts/cli.ts hwlab g14 observability query --promql ${shellQuote(String(record(V02_OBSERVABILITY_QUERIES)[name]))} --expect-count ${V02_OBSERVABILITY_EXPECTED_TARGET_COUNT} --expect-value 1` + : undefined, + }, + sectionOk: shellSectionOk(sections[`query_${name}`]), + }])), + result: compactCommandMetadata(bundle, !ok), + degradedReason: ok ? undefined : "target-health-failed", + next: ok ? { closeout: "bun scripts/cli.ts hwlab g14 observability closeout --lane v02" } : { boundary: "bun scripts/cli.ts hwlab g14 observability boundary --lane v02" }, + }; +} + +function runG14ObservabilityBoundary(options: G14ObservabilityOptions): Record { + const startedAtMs = Date.now(); + const script = [ + "set +e", + "section() {", + " name=\"$1\"", + " shift", + " printf '__UNIDESK_SECTION_BEGIN__ %s\\n' \"$name\"", + " \"$@\"", + " code=$?", + " printf '\\n__UNIDESK_SECTION_END__ %s exit=%s\\n' \"$name\" \"$code\"", + "}", + `section workloadMonitoring kubectl get servicemonitor,podmonitor,prometheusrule,prometheus,alertmanager -n ${shellQuote(V02_RUNTIME_NAMESPACE)} -o json`, + `section infraControlPlane kubectl get prometheus,alertmanager -n ${shellQuote(G14_OBSERVABILITY_NAMESPACE)} -o json`, + ].join("\n"); + const bundle = g14K3s(["script", "--", script], options.timeoutSeconds * 1000); + const sections = parseShellSections(statusText(bundle)); + const workloadItems = k8sItems(parseSectionJson(sections.workloadMonitoring)); + const infraItems = k8sItems(parseSectionJson(sections.infraControlPlane)); + const forbiddenWorkloadControlPlane = workloadItems.filter((item) => item.kind === "Prometheus" || item.kind === "Alertmanager"); + const allowedWorkloadDeclarations = workloadItems.filter((item) => item.kind === "ServiceMonitor" || item.kind === "PodMonitor" || item.kind === "PrometheusRule"); + const namespaceBoundaryOk = shellSectionOk(sections.workloadMonitoring) && forbiddenWorkloadControlPlane.length === 0; + const infraControlPlaneOk = shellSectionOk(sections.infraControlPlane) && infraItems.some((item) => item.kind === "Prometheus" && k8sMetadataName(item) === G14_PROMETHEUS_NAME); + const publicProbes = [ + publicMetricsProbe("hwlab-v02-cloud-web", `${V02_CLOUD_WEB_URL}/metrics`), + publicMetricsProbe("hwlab-v02-cloud-api", `${V02_CLOUD_API_URL}/metrics`), + ]; + const publicMetricsExposureOk = publicProbes.every((probe) => probe.ok === true && probe.exposed !== true); + const ok = isCommandSuccess(bundle) && namespaceBoundaryOk && infraControlPlaneOk && publicMetricsExposureOk; + return { + ok, + command: "hwlab g14 observability boundary", + lane: options.lane, + elapsedMs: Date.now() - startedAtMs, + namespaceBoundary: { + ok: namespaceBoundaryOk, + workloadNamespace: V02_RUNTIME_NAMESPACE, + allowedKinds: ["ServiceMonitor", "PodMonitor", "PrometheusRule"], + allowedDeclarationCount: allowedWorkloadDeclarations.length, + allowedDeclarations: monitorSummaries(allowedWorkloadDeclarations), + forbiddenKinds: ["Prometheus", "Alertmanager"], + forbiddenControlPlaneCount: forbiddenWorkloadControlPlane.length, + forbiddenControlPlane: forbiddenWorkloadControlPlane.map((item) => ({ kind: item.kind, name: k8sMetadataName(item) })), + sectionOk: shellSectionOk(sections.workloadMonitoring), + }, + infraControlPlane: { + ok: infraControlPlaneOk, + namespace: G14_OBSERVABILITY_NAMESPACE, + items: infraItems.map((item) => ({ kind: item.kind, name: k8sMetadataName(item) })), + sectionOk: shellSectionOk(sections.infraControlPlane), + }, + publicMetricsExposure: { + ok: publicMetricsExposureOk, + expected: "denied", + probes: publicProbes, + }, + result: compactCommandMetadata(bundle, !ok), + degradedReason: ok ? undefined : "boundary-failed", + next: ok ? { closeout: "bun scripts/cli.ts hwlab g14 observability closeout --lane v02" } : { targets: "bun scripts/cli.ts hwlab g14 observability targets --lane v02" }, + }; +} + +function passFail(value: boolean): "pass" | "fail" { + return value ? "pass" : "fail"; +} + +function closeoutAdvice(summary: Record): string[] { + const advice: string[] = []; + if (summary.platformReady !== "pass") advice.push("platformReady failed -> run observability status/apply and inspect Prometheus Operator plus Prometheus readiness in devops-infra"); + if (summary.scrapeReachable !== "pass") advice.push("scrapeReachable failed -> check ServiceMonitor labels, metrics sidecar port name, and Prometheus target discovery"); + if (summary.sidecarServing !== "pass") advice.push("sidecarServing failed -> check hwlab-metrics sidecar readiness, restartCount, and metrics script/container logs"); + if (summary.businessHealthProbe !== "pass") advice.push("businessHealthProbe failed -> up=1 but health_probe=0 usually means sidecar can be scraped but cannot reach the business health endpoint"); + if (summary.namespaceControlPlaneBoundary !== "pass") advice.push("namespaceControlPlaneBoundary failed -> remove Prometheus/Alertmanager from workload namespace; shared control plane belongs in devops-infra"); + if (summary.publicMetricsExposure !== "pass") advice.push("publicMetricsExposure failed -> public /metrics returned Prometheus text; remove FRP/edge exposure or add an authenticated internal-only route"); + return advice; +} + +function runG14ObservabilityCloseout(options: G14ObservabilityOptions): Record { + const startedAtMs = Date.now(); + const status = g14ObservabilityStatus(); + const targets = runG14ObservabilityTargets(options); + const boundary = runG14ObservabilityBoundary(options); + const targetQueries = record(targets.queries); + const queryOk = (name: string): boolean => record(record(targetQueries[name]).assertion).ok === true; + const sidecarsOk = record(record(targets.sidecars)).ok === true; + const platformReady = record(status.crds).ok === true && record(status.operator).ok === true && record(status.prometheus).ok === true && record(status.query).ok === true; + const namespaceBoundaryOk = record(record(boundary.namespaceBoundary)).ok === true && record(record(boundary.infraControlPlane)).ok === true; + const publicDenied = record(record(boundary.publicMetricsExposure)).ok === true; + const summary = { + platformReady: passFail(platformReady), + workloadMonitorCount: numericValue(record(status.workloadMonitors).count) ?? numericValue(record(targets.monitors).count) ?? 0, + observedTargetCount: numericValue(targets.targetCount) ?? 0, + scrapeReachable: passFail(queryOk("scrapeReachable")), + sidecarServing: passFail(queryOk("sidecarServing") && sidecarsOk), + businessHealthProbe: passFail(queryOk("businessHealthProbe")), + sidecarReady: passFail(sidecarsOk), + namespaceControlPlaneBoundary: passFail(namespaceBoundaryOk), + publicMetricsExposure: passFail(publicDenied), + publicMetricsExposureState: publicDenied ? "denied" : "exposed-or-unknown", + }; + const ok = Object.entries(summary).every(([key, value]) => key === "workloadMonitorCount" + ? numericValue(value) !== null && Number(value) > 0 + : key === "observedTargetCount" + ? numericValue(value) === V02_OBSERVABILITY_EXPECTED_TARGET_COUNT + : key === "publicMetricsExposureState" + ? value === "denied" + : value === "pass"); + return { + ok, + command: "hwlab g14 observability closeout", + lane: options.lane, + elapsedMs: Date.now() - startedAtMs, + summary, + advice: closeoutAdvice(summary), + evidence: { + platform: { + ok: status.ok, + crdsOk: record(status.crds).ok === true, + operatorReady: record(status.operator).ok === true, + prometheusReady: record(status.prometheus).ok === true, + workloadMonitorCount: summary.workloadMonitorCount, + upResultCount: record(status.query).resultCount ?? null, + }, + targets: { + ok: targets.ok, + expectedTargetCount: targets.expectedTargetCount, + observedTargetCount: targets.targetCount, + readySidecarCount: record(targets.sidecars).readyCount ?? null, + healthProbeDuration: record(record(targets.levelSummary).healthProbeDuration), + scrapeDuration: record(record(targets.levelSummary).scrapeDuration), + services: arrayRecords(record(targets.levelSummary).services).map((service) => ({ + serviceId: service.serviceId ?? null, + scrapeReachable: service.scrapeReachable ?? null, + sidecarServing: service.sidecarServing ?? null, + businessHealthProbe: service.businessHealthProbe ?? null, + statusCode: service.statusCode ?? null, + })), + }, + boundary: { + ok: boundary.ok, + forbiddenControlPlaneCount: numericValue(record(boundary.namespaceBoundary).forbiddenControlPlaneCount) ?? 0, + publicMetricsExposure: record(boundary.publicMetricsExposure), + }, + drillDown: { + status: "bun scripts/cli.ts hwlab g14 observability status", + targets: "bun scripts/cli.ts hwlab g14 observability targets --lane v02", + boundary: "bun scripts/cli.ts hwlab g14 observability boundary --lane v02", + scrapeReachable: "bun scripts/cli.ts hwlab g14 observability query --promql 'up{namespace=\"hwlab-v02\"}' --expect-count 6 --expect-value 1", + sidecarServing: "bun scripts/cli.ts hwlab g14 observability query --promql 'hwlab_service_up{namespace=\"hwlab-v02\"}' --expect-count 6 --expect-value 1", + businessHealthProbe: "bun scripts/cli.ts hwlab g14 observability query --promql 'hwlab_service_health_probe_success{namespace=\"hwlab-v02\"}' --expect-count 6 --expect-value 1", + }, + }, + degradedReason: ok ? undefined : "observability-closeout-failed", + next: { + targets: "bun scripts/cli.ts hwlab g14 observability targets --lane v02", + boundary: "bun scripts/cli.ts hwlab g14 observability boundary --lane v02", + controlPlaneCloseout: "bun scripts/cli.ts hwlab g14 control-plane closeout --lane v02 --source-commit ", + }, }; } function runG14Observability(options: G14ObservabilityOptions): Record { if (options.action === "status") return g14ObservabilityStatus(); if (options.action === "query") return runG14ObservabilityQuery(options); + if (options.action === "targets") return runG14ObservabilityTargets(options); + if (options.action === "boundary") return runG14ObservabilityBoundary(options); + if (options.action === "closeout") return runG14ObservabilityCloseout(options); return runG14ObservabilityApply(options); } @@ -6552,7 +7433,12 @@ export function hwlabG14Help(): Record { "bun scripts/cli.ts hwlab g14 observability status", "bun scripts/cli.ts hwlab g14 observability apply --dry-run", "bun scripts/cli.ts hwlab g14 observability apply --confirm", - "bun scripts/cli.ts hwlab g14 observability query --promql 'up{namespace=\"hwlab-v02\"}'", + "bun scripts/cli.ts hwlab g14 observability query --promql 'up{namespace=\"hwlab-v02\"}' --expect-count 6 --expect-value 1", + "bun scripts/cli.ts hwlab g14 observability query --promql 'hwlab_service_up{namespace=\"hwlab-v02\"}' --expect-count 6 --expect-value 1", + "bun scripts/cli.ts hwlab g14 observability query --promql 'hwlab_service_health_probe_success{namespace=\"hwlab-v02\"}' --expect-count 6 --expect-value 1", + "bun scripts/cli.ts hwlab g14 observability targets --lane v02", + "bun scripts/cli.ts hwlab g14 observability boundary --lane v02", + "bun scripts/cli.ts hwlab g14 observability closeout --lane v02", "bun scripts/cli.ts hwlab g14 tools-image status --name ci-node-tools --tag node22-alpine-bun-v1", "bun scripts/cli.ts hwlab g14 tools-image build --name ci-node-tools --tag node22-alpine-bun-v1 --confirm", "bun scripts/cli.ts hwlab g14 upstream-image status --name openfga --tag v1.17.0", @@ -6560,7 +7446,7 @@ export function hwlabG14Help(): Record { "bun scripts/cli.ts hwlab g14 upstream-image ensure --name openfga --tag v1.17.0 --confirm", "bun scripts/cli.ts job status --tail-bytes 30000", ], - description: "G14 HWLAB PR monitor, DEV rollout command, bounded v0.2 control-plane bootstrap/cleanup/runtime-migration helper, v0.2 runtime SecretRef bootstrap, devops-infra git mirror and observability maintenance, controlled CI tools image build/status entry, and allowlisted upstream image mirroring. The public monitor starts a fire-and-forget job. Default monitor lane is base=G14; --lane v02 monitors base=v0.2 PRs, waits for GitHub preflight/CI readiness, automatically merges ready PRs without waiting for other active v0.2 PipelineRuns, triggers v0.2 CD with latest-only GitOps writeback, flushes the git mirror when needed, and posts deduplicated PR comments for pending, blocked/conflict, success, superseded, failure, or timeout states. confirmed control-plane trigger-current and git-mirror sync/flush also return async jobs by default, with --wait reserved for explicit synchronous debugging. control-plane status/closeout/apply/cleanup-runs/cleanup-released-pvs/runtime-migration uses UniDesk G14:k3s routes for v0.2 Tekton/Argo control resources, runtime migration, historical PipelineRun/source-commit closeout verdicts, GitOps mirror flush state, and completed CI workspace retention only. secret status/ensure is the standard v0.2 runtime SecretRef bootstrap path; it never reads or prints secret values. upstream-image status/ensure only mirrors allowlisted upstream runtime images into the G14 local registry. git-mirror status/apply/sync/flush is the manual devops-infra mirror/relay control path and does not install a CronJob. observability status/apply/query owns the shared Prometheus Operator and Prometheus instance in devops-infra, while HWLAB lane manifests own only ServiceMonitor and PrometheusRule objects.", + description: "G14 HWLAB PR monitor, DEV rollout command, bounded v0.2 control-plane bootstrap/cleanup/runtime-migration helper, v0.2 runtime SecretRef bootstrap, devops-infra git mirror and observability maintenance, controlled CI tools image build/status entry, and allowlisted upstream image mirroring. The public monitor starts a fire-and-forget job. Default monitor lane is base=G14; --lane v02 monitors base=v0.2 PRs, waits for GitHub preflight/CI readiness, automatically merges ready PRs without waiting for other active v0.2 PipelineRuns, triggers v0.2 CD with latest-only GitOps writeback, flushes the git mirror when needed, and posts deduplicated PR comments for pending, blocked/conflict, success, superseded, failure, or timeout states. confirmed control-plane trigger-current and git-mirror sync/flush also return async jobs by default, with --wait reserved for explicit synchronous debugging. control-plane status/closeout/apply/cleanup-runs/cleanup-released-pvs/runtime-migration uses UniDesk G14:k3s routes for v0.2 Tekton/Argo control resources, runtime migration, historical PipelineRun/source-commit closeout verdicts, GitOps mirror flush state, and completed CI workspace retention only. secret status/ensure is the standard v0.2 runtime SecretRef bootstrap path; it never reads or prints secret values. upstream-image status/ensure only mirrors allowlisted upstream runtime images into the G14 local registry. git-mirror status/apply/sync/flush is the manual devops-infra mirror/relay control path and does not install a CronJob. observability status/apply/query/targets/boundary/closeout owns the shared Prometheus Operator and Prometheus instance in devops-infra, adds bounded PromQL assertions and semantic closeout summaries, while HWLAB lane manifests own only ServiceMonitor and PrometheusRule objects.", defaults: { repo: HWLAB_REPO, base: G14_SOURCE_BRANCH,