pikasTech-agentrun/docs/reference/architecture.md

# AgentRun Architecture Reference

AgentRun is a shared agent execution plane for UniDesk and HWLAB. It is not a rename of UniDesk Code Queue and must not replace existing Code Queue behavior by default. Code Queue remains the current UniDesk task queue; AgentRun is a new infrastructure line focused on agent run lifecycle, runner isolation and pluggable execution backends.

## Product Boundary

AgentRun owns generic execution infrastructure:

- creating and tracking runs;
- accepting durable commands such as `turn`, `steer`, `interrupt` and `resume`;
- assigning runs to short-lived runners;
- normalizing backend events, stdout/stderr, assistant messages, tool calls and terminal status;
- managing leases, heartbeats, retries caused by infrastructure recovery and run visibility;
- registering backend capabilities and credential injection boundaries.

UniDesk and HWLAB are tenants or clients. UniDesk owns platform entrypoints, provider inventory, CLI/frontend integration and existing Code Queue compatibility. HWLAB owns lab-specific task policy, device and hardware semantics, operation/audit/evidence models and HWLAB workspace rules. AgentRun must not decide whether a HWLAB live device mutation is authorized or whether a UniDesk production deployment may proceed; it executes runs whose tenant policy already grants that authority.

Every run must carry explicit isolation fields:

- `tenantId`, such as `unidesk` or `hwlab`;
- `projectId`, such as `pikasTech/unidesk` or `pikasTech/HWLAB`;
- `workspaceRef`, identifying the source/worktree/workspace target;
- `providerId`, such as `G14` or `D601`;
- `backendProfile`, such as `codex`, `opencode`, `claudecode`, `host-native` or `windows-native`;
- `executionPolicy`, including sandbox, approval, timeout, network and secret scope;
- `traceSink`, identifying where standardized events are mirrored.

## Service Shape

AgentRun should be built as a small service family:

```text
agentrun-mgr
  public RESTful API, durable facts, tenant/policy/idempotency checks

agentrun-runner
  short-lived per-run or per-attempt executor; claims a run, talks to one backend, writes events/status

agentrun-backend-*
  backend adapters for Codex, Claude Code, OpenCode, host-native or Windows-native execution

agentrun-scheduler
  later automatic dispatcher; scans pending runs, selects backend/profile/capacity, creates runner Jobs
```

The manager is the stable API and audit point. The runner is an executor and should not become a public API that business clients call directly. Operators may manually start a runner process or Kubernetes Job during MVP, but that runner still claims a run from `agentrun-mgr` and writes all facts back to the manager.

Backend adapters hide concrete tool protocols. Codex stdio JSON-RPC, OpenCode JSON events, Claude Code, host-native processes and Windows-native execution may use different internal protocols, but public AgentRun APIs must stay stable and backend-neutral.

## MVP Sequence

AgentRun should be built as a vertical slice rather than a broad parallel build.

### M0: Contract Skeleton

Define only the minimal resource model and state machine:

- `Run`
- `Command`
- `Event`
- `Runner`
- `Backend`

Only `turn`, `interrupt`, `status` and paged `events` are required for the first slice. Do not start with `steer`, `resume`, judge/retry, UI, multi-backend routing or automatic scheduling.

### M1: Minimal Runner Plus One Backend

The first executable proof should run without a manager or scheduler. A runner reads a local run spec, calls one backend and emits standardized events.

Acceptance:

- one `turn` executes through the backend;
- assistant/output/error events are normalized;
- terminal status is written;
- interrupt has at least a durable cancellation path, with real process interruption added when the backend supports it.

The first backend should be the narrowest backend that proves the real agent primitive. If Codex friction is high, a controlled process backend can be used briefly for contract shape, but the MVP must move to one real agent backend before higher layers are considered validated.

### M2: Manager Plus Runner Claim

Add `agentrun-mgr` as the durable fact store and public API. A client creates a run; an operator or CLI manually starts a runner with the run id; the runner claims, polls commands, appends events, heartbeats and exits.

Acceptance:

- run creation and query are durable;
- runner claim is idempotent and rejects double ownership;
- events are append-only and paged by sequence;
- command ack states are visible;
- heartbeat expiration is observable.

### M3: Manual Dispatch CLI

Add a CLI that starts a local runner process or Kubernetes Job for a selected run. This is manual dispatch, not manager-side synchronous orchestration. The manager still owns the facts; the runner still owns execution.

Acceptance:

- CLI returns quickly with JSON output;
- job/process identity and log path are visible;
- run status can be polled from the manager;
- failed runner startup is reported as infrastructure failure, not as silent task success.

### M4: Automatic Scheduler

Only after M1-M3 are stable, add `agentrun-scheduler`. The scheduler scans pending runs, applies policy/capacity/backend selection, creates runner Jobs and performs stale lease recovery. It must not directly execute backends.

Acceptance:

- pending runs become running automatically;
- scheduler restart does not interrupt already running runners;
- stale leases are recovered with explicit audit events;
- scheduler rollout does not imply active run failure.

### M5: Tenant Canary Integrations

UniDesk and HWLAB should integrate as canaries after core lifecycle proof:

- UniDesk may add `agentrun` CLI/API routes while existing Code Queue remains intact.
- HWLAB may route a narrow Code Agent canary through AgentRun.
- Tenant policy, workspace, secret scope and trace sinks must be explicit in every run.

## RESTful MVP Contract

The MVP uses only short RESTful HTTP/JSON requests. Long-running agent work is represented by durable command resources, run status and paged event polling. Do not keep an HTTP request open for a full model turn.

Public manager APIs:

```http
POST /api/v1/runs
GET  /api/v1/runs/:runId
GET  /api/v1/runs/:runId/events?afterSeq=0&limit=100
POST /api/v1/runs/:runId/commands
GET  /api/v1/runs/:runId/commands/:commandId
GET  /api/v1/backends
```

Runner-to-manager APIs:

```http
POST  /api/v1/runners/register
POST  /api/v1/runs/:runId/claim
PATCH /api/v1/runs/:runId/lease
GET   /api/v1/runs/:runId/commands?afterSeq=0&limit=20
POST  /api/v1/runs/:runId/events
PATCH /api/v1/runs/:runId/status
POST  /api/v1/commands/:commandId/ack
```

Runner inbound APIs should stay local/private and minimal:

```http
GET /health
GET /debug/status
```

Do not depend on clients calling transient runner Pod addresses. That breaks across Jobs, namespaces, host-native backends and restarts.

## Command State

Commands are durable resources. `turn`, `steer`, `interrupt` and `resume` must not be implemented as synchronous process calls from a client to a runner.

The initial command state machine is:

```text
accepted -> delivered -> confirmed
accepted -> delivered -> failed
accepted -> expired
```

All command writes should support idempotency keys. Duplicate commands with the same idempotency key and payload hash should return the existing command. Duplicate keys with different payload hashes should fail visibly.

## Event Model

Events are append-only and paged by sequence:

- `seq` is monotonic per run.
- `eventId` or `(runId, seq)` is idempotent.
- `GET /events?afterSeq=N&limit=M` is the first observation API.
- Later SSE may stream the same events, but must not replace the REST polling contract.

Minimum event categories:

- `system`
- `assistant_message`
- `tool_call`
- `command_output`
- `diff`
- `error`
- `backend_status`
- `terminal_status`

## Data Model Direction

The first implementation may use a compact schema, but it should not hide all facts in one JSON blob. The stable direction is:

- `agentrun_runs`: run identity, tenant/project/workspace/backend policy, status and timestamps;
- `agentrun_commands`: command type, idempotency key, payload hash, state and ack timestamps;
- `agentrun_events`: append-only event records keyed by run and sequence;
- `agentrun_runners`: registered runner identity, placement and heartbeat;
- `agentrun_backends`: backend profile, capabilities, capacity and health;
- `agentrun_leases`: current ownership and expiry.

## Deployment Direction

Default runtime targets are G14 native k3s namespaces `agentrun_dev` and `agentrun_prod`. Control-plane services should be long-lived; runners should be short-lived Jobs or controlled host-native processes. Backend adapters may run as pods or host-native services, but they must register capabilities and health through AgentRun rather than being called through ad hoc addresses.

Namespace isolation, RBAC, Secret scope, NetworkPolicy and ResourceQuota should be introduced before broad tenant use. A separate cluster is a later maturity option; the first implementation should prove the service inside G14 k3s namespaces unless a concrete isolation blocker requires a dedicated cluster.

## Non-Goals For MVP

Do not include these in the first MVP:

- migrating UniDesk Code Queue;
- replacing HWLAB Code Agent globally;
- multi-backend routing;
- UI beyond minimal diagnostics;
- judge/retry automation;
- automatic scaling;
- cross-cluster placement;
- SSE/WebSocket streaming;
- complete permission system;
- production rollout automation.

The first goal is one stable vertical run lifecycle: create run, manually start runner, execute one backend turn, append events, observe final status and issue a visible interrupt/cancel command.