# UniDesk Monitor Reference

## Current Web Sentinel Surface

Primary registry:

```bash
bun scripts/cli.ts web-probe sentinel status --node D601 --lane v03
```

Known D601/v03 sentinel ids:

- `workbench-dsflash-go-tool-call-10x`
- `workbench-auth-session-switch-2users`
- `mdtodo-visual-regression`

Per-sentinel drill-down:

```bash
bun scripts/cli.ts web-probe sentinel status --node D601 --lane v03 --sentinel <id>
bun scripts/cli.ts web-probe sentinel control-plane status --node D601 --lane v03 --sentinel <id>
```

Freshness-only check:

```bash
bun scripts/web-probe-sentinel-scheduler.ts run --node D601 --lane v03 --sentinel <id> --stale-multiplier 1 --dry-run
```

Host timer installation/status:

```bash
bun scripts/web-probe-sentinel-scheduler.ts status-systemd --node D601 --lane v03
bun scripts/web-probe-sentinel-scheduler.ts install-systemd --node D601 --lane v03 --confirm
bun scripts/web-probe-sentinel-scheduler.ts status-systemd --node D601 --lane v03 --sentinel <id>
```

Without `--sentinel`, `status-systemd` and `install-systemd` enumerate every enabled sentinel from the YAML registry and manage independent per-sentinel timers. Use this when a sentinel's latest run is stale: a missing timer is a runtime defect even if `run --dry-run` can enumerate the sentinel and mark it due.

Dashboard render and screenshot verification:

```bash
bun scripts/cli.ts web-probe sentinel dashboard verify --node D601 --lane v03 --sentinel <id>
bun scripts/cli.ts web-probe sentinel dashboard screenshot --node D601 --lane v03 --sentinel <id>
```

The screenshot command runs through the selected node/lane remote browser and downloads the PNG artifact to the caller's `/tmp` by default. Closeout evidence should cite `localPath`, `sha256`, page HTTP status, selected DOM summary fields and `layout.horizontalOverflow` / `overflowCount`; do not replace this with a local browser screenshot or ad-hoc `web-probe script` when the sentinel command can cover the page.

`dashboard verify` is the canonical monitor-web DOM contract. It should assert that trend latest-point counts match the latest `/api/runs?sort=updated` run, and that the monitor-web check panel is scoped to the selected run by default. For a selected run, check type count, alert type count, error samples, warning samples and alert samples must match `/api/runs/{id}.findings`; historical `/api/findings` aggregates may be available only behind an explicit time-window scope and must be labeled as historical.

Use the freshness-only `--dry-run` scheduler command when the question is only "how old is the latest run?". It reads cadence, latest age, due status and latest run id without starting a new browser observation. If an enabled sentinel is `due` or stale while other sentinels are fresh, treat it as a sentinel-specific cadence or runtime issue and record the sentinel id, cadence, latest age and run id before starting a repair loop.

Report drill-down:

```bash
bun scripts/cli.ts web-probe sentinel report --node D601 --lane v03 --sentinel <id> --latest --view summary
bun scripts/cli.ts web-probe sentinel report --node D601 --lane v03 --sentinel <id> --latest --view findings
bun scripts/cli.ts web-probe sentinel report --node D601 --lane v03 --sentinel <id> --run <runId> --view trace-frame
```

Public dashboard paths:

- `https://monitor.pikapython.com/`
- `https://monitor.pikapython.com/sentinels/workbench-auth-session-switch-2users/`

Direct API probes for shell/API/render separation:

```bash
curl -sS -D - https://monitor.pikapython.com/ -o /tmp/monitor-root.html
curl -sS -D - https://monitor.pikapython.com/api/overview -o /tmp/monitor-overview.json
curl -sS -D - https://monitor.pikapython.com/api/runs -o /tmp/monitor-runs.json
curl -sS -D - https://monitor.pikapython.com/sentinels/workbench-auth-session-switch-2users/api/overview -o /tmp/monitor-auth-overview.json
```

Use `web-probe script` with explicit `page.goto("https://monitor.pikapython.com/...")` only for one-off exploration that is not yet covered by `web-probe sentinel dashboard verify|screenshot`; repeated checks must be promoted into those sentinel dashboard commands.

## YAML Ownership

Root registry:

- `config/hwlab-node-lanes.yaml#lanes.v03.targets.D601.observability.webProbe.sentinels`

Per-sentinel management YAML:

- `config/hwlab-web-probe-sentinels/d601-v03/workbench-dsflash-go-tool-call-10x.yaml#sentinel`
- `config/hwlab-web-probe-sentinels/d601-v03/workbench-auth-session-switch-2users.yaml#sentinel`
- `config/hwlab-web-probe-sentinels/d601-v03/mdtodo-visual-regression.yaml#sentinel`

Typical config refs:

- runtime: Deployment, Service, PVC, state root, SQLite, scheduler interval.
- workflow/scenarios: observed route, cadence, command sequence, prompt/report refs.
- promptSet: synthetic prompt ids and redaction policy.
- reportViews: default and drill-down views.
- publicExposure: `monitor.pikapython.com`, FRP, Caddy route prefix.
- cicd: source, image, GitOps, Argo and target validation.
- secrets: sourceRef and targetKey only; no values in stdout or issue bodies.

## Dashboard Triage Checklist

Classify a monitor page issue in this order:

1. HTML shell loads: root status, asset links, `data-sentinel-id`, `data-base-path`, contract version.
2. API returns data: `/api/overview`, `/api/runs`, `/api/findings`, selected `/api/runs/{id}`.
3. Count scopes are coherent: trend latest point is latest-run sample count; selected run detail exposes type count and sample count separately; monitor-web checks default to selected-run scope and only show historical aggregates behind an explicit history/time-window selector.
4. Browser render executes: page errors, console errors, DOM rows, status summary, check cards, scope labels and overflow state.
5. Runtime service health: `/api/health`, `/metrics`, scheduler heartbeat, SQLite/PVC write probe.
6. Control-plane health: source, registry image, git-mirror, GitOps, Argo, Deployment/Service/PVC.

Do not treat public root/CSS/JS 200 as dashboard success. Browser console and DOM render evidence are required.

## Rollout Closeout

For a Web sentinel fix, closeout needs four independent evidence surfaces:

1. `web-probe sentinel control-plane status --node <node> --lane <lane> --sentinel <id>` must be `ok`, with source, registry, GitOps, Argo and runtime image aligned. If runtime already has the expected image but Argo target/revision is stale, run the controlled `control-plane apply --confirm --wait` path once with a bounded timeout, then recheck status; do not declare the rollout complete on runtime readiness alone.
2. `web-probe sentinel validate --node <node> --lane <lane> --sentinel <id>` must pass `/api/health`, `/metrics`, indexed recent report, public exposure and public dashboard probes.
3. `web-probe sentinel dashboard screenshot --node <node> --lane <lane> --sentinel <id>` must pass through the remote browser and save a verified PNG to the caller `/tmp`; record the local path, hash and layout overflow result.
4. `web-probe sentinel report --node <node> --lane <lane> --sentinel <id> --latest --view summary` must show the latest business run, report hash, samples and finding severity. Auth/login failures, submit failures, missing samples, absent recent reports and Code Agent multi-round continuity breaks are blockers. YAML elapsed-budget or total-turn timing alerts are non-blocking warnings unless they coincide with a failed turn, broken continuity, missing report or unavailable user path.
5. For monitor-web count fixes, `dashboard verify` must include the latest run id, trend sample counts, selected-run check counts and any targeted run filter evidence. A pass requires check samples to match the selected run detail, not just chart arithmetic.

Long `quick-verify` or CI/CD waits should be bounded by the YAML-declared budget and the operator's outer timeout. If a wait would exceed about two minutes during rollout, first inspect the visible stage and either optimize the slow path, defer the expensive quick verify to manual validation, or record it as a non-blocking timing warning; do not dead-wait without new evidence.

If `origin/master` advances while rolling out a sentinel, first classify the new commits. For unrelated parallel changes, finish the current bounded check, wait briefly for the branch head to stabilize, then perform one final rollout/status pass against the stable head. Do not loop forever chasing every concurrent merge, but do not call a rollout complete while source truth, internal mirror and runtime image point at different commits.

Source mirror readiness must be proven by the internal mirror object/read probe for the expected commit. A GitHub/source head check alone is not sufficient evidence to skip source sync, because it does not prove the k3s publish job can fetch the object from the node-local mirror.

Dashboard aggregate counters may include historical runs only when the UI labels that scope explicitly. They must not sit beside a latest-run chart or selected-run check list without a scope label. If trend, run detail and check list disagree, first identify whether each number is a type count, sample count or historical aggregate before changing code.

For Code Agent multi-round quick-verify, accept the latest run's `turn-summary` / `trace-frame` plus `blockingFindingCount=0` and `controlFindingCount=0`. Analyzer red findings about hydration, API-to-DOM lag or timing drift are investigation evidence unless they coincide with missing durable turns/final responses, failed submit/login/auth, broken continuity, absent report or unavailable user path.

## Multi-Sentinel Target Architecture

Current implementation has independent sentinel runner services, but each runner still serves its own dashboard. The target architecture should split:

- `sentinel-runner`: one Deployment/PVC/Service per sentinel id; owns scheduler, observe wrapper, SQLite/report index, `/api/report`, `/api/overview`, `/metrics`.
- `monitor-web`: one Deployment/Service/public route per workspace or node/lane; owns dashboard shell, multi-sentinel registry view, aggregation/proxy APIs and public HTTPS exposure.

Recommended Kubernetes-native mechanisms:

- Label all runner Services and Pods with `unidesk.ai/web-probe-sentinel-id`, node, lane, workspace and component.
- Let monitor-web discover runners using Kubernetes Services or EndpointSlices with label selectors.
- Keep each runner Service `ClusterIP`; expose only monitor-web through declared `publicExposure`.
- Use runner Service DNS or Kubernetes discovery for aggregation, not hard-coded route prefixes.
- Keep per-runner PVCs for local report history unless a shared store is explicitly specified.
- Use CronJob only for short, stateless periodic probes. Keep Deployment for long-lived browser observers, maintenance API and progressive reports.

This keeps runner isolation while giving users one monitor page for all sentinels in a workspace.
