# UniDesk Monitor Reference

## Current Web Sentinel Surface

Primary registry:

```bash
bun scripts/cli.ts web-probe sentinel status --node <node> --lane <lane>
```

Known sentinel ids vary by node/lane and must come from YAML. Common `v03` examples include:

- `workbench-dsflash-go-tool-call-10x`
- `workbench-auth-session-switch-2users`
- `workbench-fake-echo-session-invariance-10x`
- `mdtodo-visual-regression`
- `jd01-web-probe-sentinel`

Per-sentinel drill-down:

```bash
bun scripts/cli.ts web-probe sentinel status --node <node> --lane <lane> --sentinel <id>
bun scripts/cli.ts web-probe sentinel control-plane status --node <node> --lane <lane> --sentinel <id>
```

Freshness-only check:

```bash
bun scripts/web-probe-sentinel-scheduler.ts run --node <node> --lane <lane> --sentinel <id> --stale-multiplier 1 --dry-run
```

Cadence/runtime validation is k3s-first:

```bash
trans <node>:k3s kubectl -n <namespace> get cronjob -l app.kubernetes.io/component=cadence-scheduler
trans <node>:k3s kubectl -n <namespace> create job --from=cronjob/<quick-verify-cronjob> <manual-job-name>
```

Host `systemd` timer commands are legacy diagnostics only. Enabled HWLAB Web sentinels must run from target node/lane k3s CronJob/GitOps. If a sentinel's latest run is stale, first compare YAML cadence, latest run age and rendered CronJob state for that sentinel; a missing or stale CronJob is a runtime defect even if the dry-run scheduler can enumerate the sentinel and mark it due.

Dashboard render and screenshot verification:

```bash
bun scripts/cli.ts web-probe sentinel dashboard verify --node <node> --lane <lane> --sentinel <id>
bun scripts/cli.ts web-probe sentinel dashboard screenshot --node <node> --lane <lane> --sentinel <id>
```

The screenshot command runs through the selected node/lane remote browser and downloads the PNG artifact to the caller's `/tmp` by default. Closeout evidence should cite `localPath`, `sha256`, page HTTP status, selected DOM summary fields and `layout.horizontalOverflow` / `overflowCount`; do not replace this with a local browser screenshot or ad-hoc `web-probe script` when the sentinel command can cover the page.

`dashboard verify` is the canonical monitor-web DOM contract. It should assert that trend latest-point counts match the latest `/api/runs?sort=updated` run, and that the monitor-web check panel is scoped to the selected run by default. For a selected run, trend error/warning points, run-row tags and check table rows must all be derived from `/api/runs/{id}.findings` detail rows. Do not fall back to top-level sample counters such as `severityCounts` for user-visible run aggregates; historical `/api/findings` aggregates may be available only behind an explicit time-window scope and must be labeled as historical.

Use the freshness-only `--dry-run` scheduler command when the question is only "how old is the latest run?". It reads cadence, latest age, due status and latest run id without starting a new browser observation. If an enabled sentinel is `due` or stale while other sentinels are fresh, treat it as a sentinel-specific cadence or runtime issue and record the sentinel id, cadence, latest age and run id before starting a repair loop.

Report drill-down:

```bash
bun scripts/cli.ts web-probe sentinel report --node <node> --lane <lane> --sentinel <id> --latest --view summary
bun scripts/cli.ts web-probe sentinel report --node <node> --lane <lane> --sentinel <id> --latest --view findings
bun scripts/cli.ts web-probe sentinel report --node <node> --lane <lane> --sentinel <id> --run <runId> --view trace-frame
```

Public dashboard paths:

- `https://monitor.pikapython.com/`
- `https://monitor.pikapython.com/sentinels/d518-workbench-dsflash-go-tool-call-10x/`
- `https://monitor.pikapython.com/sentinels/workbench-auth-session-switch-2users/`
- `https://monitor.pikapython.com/sentinels/jd01-web-probe-sentinel/`

Direct API probes for shell/API/render separation:

```bash
curl -sS -D - https://monitor.pikapython.com/ -o /tmp/monitor-root.html
curl -sS -D - https://monitor.pikapython.com/api/overview -o /tmp/monitor-overview.json
curl -sS -D - https://monitor.pikapython.com/api/runs -o /tmp/monitor-runs.json
curl -sS -D - https://monitor.pikapython.com/sentinels/workbench-auth-session-switch-2users/api/overview -o /tmp/monitor-auth-overview.json
```

Use `web-probe script` with explicit `page.goto("https://monitor.pikapython.com/...")` only for one-off exploration that is not yet covered by `web-probe sentinel dashboard verify|screenshot`; repeated checks must be promoted into those sentinel dashboard commands.

## YAML Ownership

Root registry:

- `config/hwlab-node-lanes.yaml#lanes.v03.targets.D601.observability.webProbe.sentinels`
- `config/hwlab-node-lanes.yaml#lanes.v03.targets.D518.observability.webProbe.sentinels`
- `config/hwlab-node-lanes.yaml#lanes.v03.targets.JD01.observability.webProbe.sentinels`

Per-sentinel management YAML:

- `config/hwlab-web-probe-sentinels/d601-v03/workbench-dsflash-go-tool-call-10x.yaml#sentinel`
- `config/hwlab-web-probe-sentinels/d601-v03/workbench-auth-session-switch-2users.yaml#sentinel`
- `config/hwlab-web-probe-sentinels/d601-v03/mdtodo-visual-regression.yaml#sentinel`
- `config/hwlab-web-probe-sentinels/d518-v03/workbench-dsflash-go-tool-call-10x.yaml#sentinel`
- `config/hwlab-web-probe-sentinels/d518-v03/workbench-fake-echo-session-invariance-10x.yaml#sentinel`
- `config/hwlab-web-probe-sentinel/profiles.yaml#nodes.JD01.sentinels.jd01-web-probe-sentinel.sentinel`

Typical config refs:

- runtime: Deployment, Service, PVC, state root, SQLite, scheduler interval.
- workflow/scenarios: observed route, cadence, command sequence, prompt/report refs.
- promptSet: synthetic prompt ids and redaction policy.
- reportViews: default and drill-down views.
- publicExposure: `monitor.pikapython.com`, FRP, Caddy route prefix.
- cicd: source, image, GitOps, Argo and target validation.
- secrets: sourceRef and targetKey only; no values in stdout or issue bodies.

## Dashboard Triage Checklist

Classify a monitor page issue in this order:

1. HTML shell loads: root status, asset links, `data-sentinel-id`, `data-base-path`, contract version.
2. API returns data: `/api/overview`, `/api/runs`, `/api/findings`, selected `/api/runs/{id}`.
3. Count scopes are coherent: trend latest point is the selected/latest run's check row count by severity, not a raw sample count; selected run detail exposes row count and sample count separately; monitor-web checks default to selected-run scope and only show historical aggregates behind an explicit history/time-window selector.
4. Browser render executes: page errors, console errors, DOM rows, status summary, check cards, scope labels and overflow state.
5. Runtime service health: `/api/health`, `/metrics`, scheduler heartbeat, SQLite/PVC write probe.
6. Control-plane health: source, registry image, git-mirror, GitOps, Argo, Deployment/Service/PVC.

Do not treat public root/CSS/JS 200 as dashboard success. Browser console and DOM render evidence are required.

When a Pod and its ClusterIP Service return the expected monitor-web asset but `https://monitor.pikapython.com/monitor-web/assets/...` still returns an old file, check the public FRP hop before changing UI code again. On D518 this has happened when a stale legacy `hwlab-web-probe-sentinel-frpc` Argo app still owned PK01 remote port `22093`, while the active `hwlab-web-probe-sentinel-fake-echo-frpc` logged `start error: port already used`; Caddy then served the old runner even though the new Pod and runtime image were healthy. Retire the stale Argo/frpc owner through a controlled YAML/CLI path and then rerun `dashboard verify`.

## Rollout Closeout

For a Web sentinel fix, closeout needs four independent evidence surfaces:

1. `web-probe sentinel control-plane status --node <node> --lane <lane> --sentinel <id>` must be `ok`, with source, registry, GitOps, Argo and runtime image aligned. If runtime already has the expected image but Argo target/revision is stale, run the controlled `control-plane apply --confirm --wait` path once with a bounded timeout, then recheck status; do not declare the rollout complete on runtime readiness alone.
2. `web-probe sentinel validate --node <node> --lane <lane> --sentinel <id>` must pass `/api/health`, `/metrics`, indexed recent report, public exposure and public dashboard probes.
3. `web-probe sentinel dashboard screenshot --node <node> --lane <lane> --sentinel <id>` must pass through the remote browser and save a verified PNG to the caller `/tmp`; record the local path, hash and layout overflow result.
4. `web-probe sentinel report --node <node> --lane <lane> --sentinel <id> --latest --view summary` must show the latest business run, report hash, samples and finding severity. Auth/login failures, submit failures, missing samples, absent recent reports and Code Agent multi-round continuity breaks are blockers. YAML elapsed-budget or total-turn timing alerts are non-blocking warnings unless they coincide with a failed turn, broken continuity, missing report or unavailable user path.
5. For monitor-web count fixes, `dashboard verify` must include the latest run id, trend point row counts, selected-run check table counts, selected run-row red/yellow tags and any targeted run filter evidence. A pass requires all user-visible run aggregates to match `/api/runs/{id}.findings` detail rows, not just chart arithmetic.

Long `quick-verify` or CI/CD waits should be bounded by the YAML-declared budget and the operator's outer timeout. If a wait would exceed about two minutes during rollout, first inspect the visible stage and either optimize the slow path, defer the expensive quick verify to manual validation, or record it as a non-blocking timing warning; do not dead-wait without new evidence.

If `origin/master` advances while rolling out a sentinel, first classify the new commits. For unrelated parallel changes, finish the current bounded check, wait briefly for the branch head to stabilize, then perform one final rollout/status pass against the stable head. Do not loop forever chasing every concurrent merge, but do not call a rollout complete while source truth, internal mirror and runtime image point at different commits.

Source mirror readiness must be proven by the internal mirror object/read probe for the expected commit. A GitHub/source head check alone is not sufficient evidence to skip source sync, because it does not prove the k3s publish job can fetch the object from the node-local mirror.

If the internal mirror branch is ahead of the expected commit, status may still be ready only when the expected object exists and `expected` is an ancestor of the mirror branch tip. Treat that as `mirror-ahead`, not as a source blocker. An exact SHA match is sufficient but not required during parallel master advancement.

Registry readiness must follow the target node YAML. On nodes where `registry.mode` is `k8s-workload` but the effective registry endpoint is a loopback address, use the controlled `web-probe sentinel image status` / `control-plane status` registry probe rather than ad-hoc Service DNS curls; a host-network registry that binds `127.0.0.1` can be healthy for node-local publish and pulls while its ClusterIP Service is not a valid manifest probe path. OCI manifest probes must send an OCI/Docker manifest `Accept` header before treating a tag as missing.

When `trigger-current --confirm --wait` defers git-mirror flush into an async job, judge rollout readiness with a follow-up `web-probe sentinel control-plane status` and `hwlab nodes git-mirror status`. If the async flush job exits nonzero but the follow-up status shows source/registry/GitOps/Argo/runtime aligned and git mirror `pending=false` / `inSync=true`, record the async job failure as a closeout visibility issue rather than blocking the already-aligned sentinel rollout. If `pending=true` or GitOps is not in sync, continue through the controlled git-mirror flush path.

Dashboard aggregate counters may include historical runs only when the UI labels that scope explicitly. They must not sit beside a latest-run chart or selected-run check list without a scope label. If trend, run detail and check list disagree, first identify whether each number is a type count, sample count or historical aggregate before changing code.

For Code Agent multi-round quick-verify, accept the latest run's `turn-summary` / `trace-frame` plus `blockingFindingCount=0` and `controlFindingCount=0`. Analyzer red findings about hydration, API-to-DOM lag or timing drift are investigation evidence unless they coincide with missing durable turns/final responses, failed submit/login/auth, broken continuity, absent report or unavailable user path.

## Multi-Sentinel Target Architecture

Current implementation has independent sentinel runner services, but each runner still serves its own dashboard. The target architecture should split:

- `sentinel-runner`: one Deployment/PVC/Service per sentinel id; owns scheduler, observe wrapper, SQLite/report index, `/api/report`, `/api/overview`, `/metrics`.
- `monitor-web`: one Deployment/Service/public route per workspace or node/lane; owns dashboard shell, multi-sentinel registry view, aggregation/proxy APIs and public HTTPS exposure.

Recommended Kubernetes-native mechanisms:

- Label all runner Services and Pods with `unidesk.ai/web-probe-sentinel-id`, node, lane, workspace and component.
- Let monitor-web discover runners using Kubernetes Services or EndpointSlices with label selectors.
- Keep each runner Service `ClusterIP`; expose only monitor-web through declared `publicExposure`.
- Use runner Service DNS or Kubernetes discovery for aggregation, not hard-coded route prefixes.
- Keep per-runner PVCs for local report history unless a shared store is explicitly specified.
- Use CronJob only for short, stateless periodic probes. Keep Deployment for long-lived browser observers, maintenance API and progressive reports.

This keeps runner isolation while giving users one monitor page for all sentinels in a workspace.
