pikasTech-unidesk/docs/reference/ci.md

UniDesk CI On D601 k3s

UniDesk CI is hosted on the D601 native k3s cluster with Tekton Pipelines and Tekton Triggers. It is CI only. CD remains separate from Tekton. No Tekton task may roll out production services. CI/CD runtime-version governance follows docs/reference/release-governance.md and GitHub issue #6. The default user-service release order is owned by docs/reference/user-service-delivery.md.

Components

• Tekton Pipelines: v1.12.0.
• Tekton Triggers: v0.34.0.
• UniDesk CI namespace: unidesk-ci.
• Manifests: src/components/microservices/k3sctl-adapter/k3s/ci/.
• Artifact catalog: root CI.json, which is CI artifact catalog only. It describes build inputs, image naming and summary fields; runtime topology, rollout target, ports, namespaces and desired service commits remain in config.json, service manifests and deploy.json.
• CLI entry: bun scripts/cli.ts ci install|status|run|publish-backend-core|publish-user-service|run-dev-e2e|logs.
• Dev namespace e2e runner: bun scripts/cli.ts ci run-dev-e2e; authoritative runner path, manifest contract and safety boundary are in docs/reference/dev-ci-runner.md.
• Rust backend-core check/build boundary: CI may run UNIDESK_D601_RUST_CHECK=1 bun scripts/cli.ts check --full --rust on D601; the master server must not compile Rust for backend-core iteration. The authoritative dev environment rule is docs/reference/dev-environment.md.

Pipeline Scope

Each commit CI run performs:

• git clone and checkout of the requested repository revision.
• bun install --frozen-lockfile at the repo root and src/, because bun scripts/cli.ts check compiles all src/components and needs the component workspace lockfile for frontend React dependencies.
• UNIDESK_D601_RUST_CHECK=1 bun scripts/cli.ts check --full --rust, so Rust backend-core is checked only inside the D601 CI execution boundary.
• Application contract checks such as Code Queue /api/workdirs, using ordinary app fixtures or E2E paths rather than CI/CD infrastructure self-tests.
• Temporary code-queue-ci-read Deployment and ClusterIP Service in unidesk-ci.
• Code Queue read performance checks against the production PostgreSQL through d601-tcp-egress-gateway.
• Manual dev desired-state smoke for Code Queue via ci run-dev-e2e, using the Git-pinned code-queue service commit from origin/master:deploy.json#environments.dev.

CI/CD bootstrap, repair and upgrade actions are infrastructure operations. They are manually tested and may be promoted directly to production when the infrastructure itself is the target; do not add CI jobs whose purpose is to prove that CI/CD can bootstrap or repair itself.

ci install also prewarms the D601 k3s containerd runtime with the Tekton entrypoint/workingdir helper images, oven/bun:1-debian, alpine/git:2.45.2 and unidesk-code-queue:dev. Missing images are pulled through the node-local provider-gateway WS egress proxy and then imported into native k3s containerd with digests preserved, so PipelineRun pods do not hang on external registry pulls. Sustained pull throughput below 1 MB/s is treated as a provider/main-server network or proxy degradation first, not as a Dockerfile or application failure.

Git clone and dependency downloads inside the repo check task use d601-provider-egress-proxy.unidesk.svc.cluster.local:18789; the NO_PROXY list keeps the in-cluster read service and D601 TCP egress gateway on the cluster network.

Private repository source authentication is part of the CI contract and follows docs/reference/devops-hygiene.md. If the repo-check task fails at git clone because credentials are unavailable, treat it as a CI infrastructure/auth gap, not as an application test result.

Artifact Catalog And Summary Contract

CI.json is the reusable CI artifact catalog. It must remain artifact-only: serviceId, artifact kind, producer command, source repository URL, optional repo root, repo-relative Dockerfile path, registry repository naming, upstream image digest/mirror metadata and summary-field semantics are allowed; provider ids, runtime namespaces, ports, compose services, Kubernetes Services, health paths, env, volumes and desired deploy commits are not allowed. deploy.json remains the version intent for deployments and must not be replaced by CI.json.

CI.json schema version 2 uses these artifact kinds:

• source-build: CI builds a Dockerfile from a pushed Git commit. UniDesk repo Dockerfiles, external Git repositories and Dockerfiles in repository subdirectories all use this kind.
• upstream-image: CI records an image-only service that comes from an upstream image digest and optional D601 mirror rule. It is not a Dockerfile build producer.

Each catalog artifact also has a status. supported means the matching producer command may start a dry-run or real CI producer action. blocked means the service is intentionally listed for coverage but the producer must return a structured blocked result instead of silently building, skipping or falling back. filebrowser and filebrowser-d601 are upstream-image blocked entries pinned to docker.io/filebrowser/filebrowser@sha256:289c5dd677c56662440f26eeb44266ed9746fe563d2e9100f546bff558534d70; they must not be represented as source-build services.

Current catalog coverage:

• source-build/supported: backend-core, frontend, baidu-netdisk, decision-center, project-manager, oa-event-flow, todo-note, code-queue-mgr, findjob, pipeline, met-nonlinear, k3sctl-adapter, mdtodo, claudeqq, and dev-only code-queue.
• upstream-image/blocked: filebrowser, filebrowser-d601.

publish-user-service reads source.repo and source.dockerfile from CI.json. The command rejects ad hoc --repo overrides; the catalog is the only source for producer build inputs. publish-backend-core also reads its producer inputs from CI.json, while preserving the dedicated backend-core command and Rust/D601 build boundary. For findjob, pipeline, met-nonlinear, and k3sctl-adapter, the catalog can also carry consumer-only notes so CI producers and deploy consumers stay aligned on the live contract.

Every successful image-producing CI task must expose a common artifactSummary contract:

• serviceId: stable UniDesk service id.
• sourceCommit: full 40-character Git commit used as source and tag.
• sourceRepo: Git repository URL used to materialize the source.
• dockerfile: repo-relative Dockerfile path.
• imageRef: commit-tagged image reference pushed by CI.
• tag: commit-pinned image tag; mutable latest is invalid.
• digest: registry manifest digest for the pushed image.
• digestRef: immutable repository@digest image reference.

Tekton artifact tasks write these values as TaskRun results and also print the legacy *_artifact_* log lines for operator diagnostics. The CLI must read TaskRun results first, fall back to pod logs only for older runs, derive imageRef/digestRef from repository, tag and digest where possible, and report exact missing fields such as digest or digestRef. It must not turn a succeeded PipelineRun into a generic incomplete failure.

CI/CD Runtime Governance

CI/CD server and control-plane runtime is production-like infrastructure. Its service version must be pinned by deploy.json and verified through runtime commit metadata; it must not float with the latest master just because the operator's CLI is newer.

The CLI may be run from master if it remains backward compatible with the pinned server version. When the CLI needs a newer server capability, it must detect that through a health or capability response and fail explicitly. It must not replace the missing server capability with raw SSH, direct kubectl, direct SQL, direct production namespace mutation, or another hidden deployment path.

CI/CD services should report their source commit, API/schema capability, supported environments and supported operations. CI diagnostics should include that information when rejecting an operation as unsupported.

During a release/v1 stabilization window, CI should continue using the implemented dev desired-state contract rather than adding split-lane infrastructure. The origin/master:deploy.json#environments.dev service pins may point at v1 stabilization commits for validation, but CI must print the manifest commit and service commits it used. Explicit dev-v1 and dev-master support is a later infrastructure change after v1 is stable.

When the broken component is CI/CD itself, use manual smoke checks, runtime health, logs, commit metadata and operator review as the acceptance path. Do not block the repair on a new CI self-test for the CI/CD bootstrap path.

Steps that call the Kubernetes API directly clear inherited proxy variables so service-account HTTPS calls to kubernetes.default.svc do not accidentally use the Code Queue image's Docker Compose proxy defaults. The rollout poll reads the Deployment main resource rather than the /status subresource, keeping CI RBAC limited to the same app/service resources it creates and deletes. The performance probe scans recent Code Queue tasks until it finds one with trace steps, so a newly selected task without persisted step detail does not make the whole gate fail before measuring the trace endpoints.

The temporary Code Queue service uses:

• CODE_QUEUE_SERVICE_ROLE=read.
• CODE_QUEUE_SCHEDULER_ENABLED=false.
• CODE_QUEUE_STARTUP_OA_BACKFILL_ENABLED=false.
• CODE_QUEUE_NOTIFY_CLAUDEQQ_ENABLED=false.
• CODE_QUEUE_CODEX_SQLITE_LOG_EXPORT_ENABLED=false.
• D601 k3s d601-provider-egress-proxy for external/OA Event Flow fetches, with d601-tcp-egress-gateway and the CI read service in NO_PROXY.
• EmptyDir state/log mounts.

This means the CI service can read existing tasks, Trace summaries, Trace steps and Trace step details from the main database, but it must not schedule, mutate, notify, backfill or become deployment truth.

Backend-Core Artifact Publication

backend-core production image creation belongs to a manual D601-side artifact producer action, not to master server CD and not to a CI/CD bootstrap self-test. The purpose is to keep Rust compilation, Docker build cache, dependency downloads and image push on the higher-resource D601 side while leaving production deployment with a small pull/recreate/verify surface.

The CI artifact task must follow these rules:

• Input revision comes from pushed Git and is resolved to a full 40-character commit. A dirty worktree or unpushed local tree must never be used as the image source.
• Source fetch for this artifact uses the existing D601 GitHub SSH deploy identity and the node-local provider-gateway WS egress proxy at http://127.0.0.1:18789. D601 prepares a commit-pinned source export under /home/ubuntu/.unidesk/ci/backend-core-artifacts/<commit> before creating the PipelineRun; Tekton consumes that prepared source through a read-only hostPath and must not clone GitHub itself, mount GitHub credentials, use an in-cluster Git mirror, or accept an operator-uploaded source tree.
• The source checkout, Rust build and Docker build run on D601 CI infrastructure. The master server must not run cargo build, docker compose build backend-core or server rebuild backend-core as part of production backend-core deployment.
• The image is tagged with the source commit, for example unidesk/backend-core:<commit>, and pushed to the D601 artifact registry as 127.0.0.1:5000/unidesk/backend-core:<commit>.
• The image must carry at least unidesk.ai/service-id=backend-core, unidesk.ai/source-repo, unidesk.ai/source-commit and unidesk.ai/dockerfile=src/components/backend-core/Dockerfile.
• Publication must fail if the D601 artifact registry is not healthy. It must not fall back to a third-party registry or a mutable latest tag.
• CI output must include the common artifactSummary fields defined above. artifactSummary.imageRef and artifactSummary.digestRef are deployment inputs for later CD, but CI must not restart production Compose services, call production deploy apply, mutate production namespaces, or change deploy.json.

The artifact registry contract and CD consumption path are defined in docs/reference/artifact-registry.md. CI is the producer of the backend-core image artifact; CD is only the consumer.

User-Service Artifact Publication

User-service image creation uses the same CI producer boundary as backend-core. Service identities, source repositories, Dockerfiles and image repositories come from root CI.json; runtime topology still comes from config.json, deploy.json and existing manifests. The reviewed sample services include baidu-netdisk, decision-center, frontend, mdtodo, claudeqq and code-queue, and the catalog also covers the other source-build services listed above. code-queue artifacts are allowed for dev validation only; production Code Queue artifact deploy remains unsupported.

The CI user-service artifact task must follow these rules:

• Inputs are a pushed full 40-character Git commit and a service id registered in CI.json. Dirty worktrees, operator-uploaded source trees, command-line repo overrides and local-only commits are not valid artifact sources.
• D601 prepares a commit-pinned source export under /home/ubuntu/.unidesk/ci/user-service-artifacts/<service-id>/<commit> using the existing GitHub SSH deploy identity and node-local provider-gateway WS egress proxy. Tekton consumes that export through a read-only hostPath.
• The image is tagged only with the source commit and pushed to the D601 registry as 127.0.0.1:5000/unidesk/<service-id>:<commit>. The producer must reject third-party registries and must not publish or consume a mutable latest tag.
• The image must carry unidesk.ai/service-id, unidesk.ai/source-repo, unidesk.ai/source-commit and unidesk.ai/dockerfile labels.
• The command output must include the common artifactSummary fields: serviceId, sourceCommit, sourceRepo, dockerfile, imageRef, tag, digest and digestRef. The digest ref is suitable as immutable input for later dev/prod deployment work.
• CI is an artifact producer only. It must not restart production services, call production deploy apply, mutate the production namespace, or change deploy.json.
• CI.json may also list downstream consumer-only catalog entries for D601 direct Compose services such as findjob, pipeline, met-nonlinear, and k3sctl-adapter; these entries describe the artifact contract and dry-run/support status, not new producer behavior.
• For D601 direct services, findjob and pipeline have reviewed dev/prod D601 Compose artifact consumers, met-nonlinear is dry-run only until the long-running service image contract matches the published artifact, and k3sctl-adapter is supervisor-only because it is the native k3s control bridge outside the k3s failure domain.
• ClaudeQQ source comes from https://gitee.com/lyon1998/agent_skills; the producer exports the claudeqq/ subtree and overlays the UniDesk Dockerfile plus API adapter from src/components/microservices/claudeqq/ before building. Runtime topology and deploy intent still live in manifests and deploy.json, not in CI.json.

The same command also has a read-only preflight mode: bun scripts/cli.ts ci publish-user-service --service <id> --commit <full-sha> --dry-run. That mode may be called from the main server or through remote frontend passthrough, and it must return runnerDisposition, missingChannels, missingControlChannels, channels, controlChannels, registry, artifactSummary, controlledPublish, boundary and next without creating a PipelineRun or pushing an image. missingChannels is the detailed probe list, while missingControlChannels is the runner-facing domain list using only backend-core, database, provider and registry. controlledPublish must point at the real producer boundary: D601, namespace unidesk-ci, PipelineRun unidesk-user-service-artifact-publish, and the non-dry-run ci publish-user-service command shape. If backend-core, database, provider or registry channels are missing, the result must be structured infra-blocked, not a bare container lookup failure.

Publish a Baidu Netdisk artifact:

bun scripts/cli.ts ci publish-user-service --service baidu-netdisk --commit <full-sha> --wait-ms 1200000

This command creates the unidesk-user-service-artifact-publish Tekton PipelineRun and pushes 127.0.0.1:5000/unidesk/baidu-netdisk:<commit>. It is only the CI producer step. Dev validation and production CD must consume that commit-pinned artifact with deploy apply --env dev --service baidu-netdisk and deploy apply --env prod --service baidu-netdisk; neither path may use server rebuild baidu-netdisk as release evidence.

Publish a Decision Center artifact:

bun scripts/cli.ts ci publish-user-service --service decision-center --commit <full-sha> --wait-ms 1200000

This command creates the unidesk-user-service-artifact-publish Tekton PipelineRun and pushes 127.0.0.1:5000/unidesk/decision-center:<commit>.

Publish a frontend artifact:

bun scripts/cli.ts ci publish-user-service --service frontend --commit <full-sha> --wait-ms 1200000

This command creates the unidesk-user-service-artifact-publish Tekton PipelineRun and pushes 127.0.0.1:5000/unidesk/frontend:<commit>. The next step is CD consumption, not a source rebuild: deploy apply --env dev --service frontend imports the artifact into D601 native k3s frontend-dev, and deploy apply --env prod --service frontend recreates the master-server Compose frontend service with --no-build and live /health.deploy.commit verification.

Publish k3s-managed service artifacts:

bun scripts/cli.ts ci publish-user-service --service mdtodo --commit <full-sha> --wait-ms 1200000
bun scripts/cli.ts ci publish-user-service --service claudeqq --commit <full-sha> --wait-ms 1200000
bun scripts/cli.ts ci publish-user-service --service code-queue --commit <full-sha> --wait-ms 1200000

MDTODO and ClaudeQQ artifacts are consumed first by dev CD and then by production CD through the D601 registry artifact consumer. Code Queue artifacts are consumed only by the dev artifact consumer; CI publication does not enable production Code Queue deployment.

Dev Namespace E2E

ci run-dev-e2e is the manual dev desired-state smoke flow. The single authoritative reference for its Git-controlled runner script, short launcher, result directory and no-CD boundary is docs/reference/dev-ci-runner.md.

The current dev namespace e2e is a harness and smoke gate, not a full frontend/backend stack rollout. It does include a controlled Code Queue slice: D601 builds or reuses the environments.dev.services[].id=code-queue commit, imports the image into native k3s containerd, starts temporary PostgreSQL plus Code Queue scheduler/read/write Services in unidesk-ci-e2e-<runId>, and verifies the HTTP API through the Kubernetes API service proxy. The stable frontend/backend path /api/microservices/code-queue/proxy/api/workdirs is covered by the normal UniDesk e2e check microservice:code-queue-workdirs. This remains CI-only and must not deploy persistent unidesk-dev or production resources.

Performance Gate

The initial budgets live in unidesk-ci/unidesk-ci-budgets:

• Code Queue first overview payload through the temporary read service, used as the service-side first-paint proxy: 10000ms.
• GET /api/tasks/{id}/trace-summary: 10000ms.
• GET /api/tasks/{id}/trace-steps: 20000ms diagnostic, reported but not blocking while the existing production TraceView step query is being optimized.
• GET /api/tasks/{id}/trace-step: 20000ms diagnostic, reported but not blocking while the existing production TraceView step query is being optimized.
• GET /api/tasks/overview p95 over 10 samples: 20000ms.

These are absolute budgets. Historical relative baselines can be added later by writing metrics to a dedicated CI table or object store; they should not be mixed into production task tables.

Commands

Install or refresh CI:

bun scripts/cli.ts ci install

Check status:

bun scripts/cli.ts ci status

Run CI manually for a commit:

bun scripts/cli.ts ci run --revision <commit>

Publish a backend-core artifact for production CD:

bun scripts/cli.ts ci publish-backend-core --commit <full-sha> --wait-ms 1200000

This command creates the unidesk-backend-core-artifact-publish Tekton PipelineRun. It is a CI producer action only: it may build and push 127.0.0.1:5000/unidesk/backend-core:<commit>, but it must not recreate the master server container. Production deployment is triggered separately with deploy apply --env prod --service backend-core --commit <full-sha>.

Publish a user-service artifact:

bun scripts/cli.ts ci publish-user-service --service baidu-netdisk --commit <full-sha> --wait-ms 1200000
bun scripts/cli.ts ci publish-user-service --service decision-center --commit <full-sha> --wait-ms 1200000
bun scripts/cli.ts ci publish-user-service --service mdtodo --commit <full-sha> --wait-ms 1200000
bun scripts/cli.ts ci publish-user-service --service claudeqq --commit <full-sha> --wait-ms 1200000
bun scripts/cli.ts ci publish-user-service --service code-queue --commit <full-sha> --wait-ms 1200000

This command is a CI producer action only. For reviewed user services, it builds and pushes 127.0.0.1:5000/unidesk/<service-id>:<commit> and reports the immutable digest without deploying production. For code-queue, the supported consumer is dev-only.

Run the dev namespace e2e harness manually:

bun scripts/cli.ts ci run-dev-e2e --wait-ms 600000

Inspect a run:

bun scripts/cli.ts ci logs <runId>

Trigger Boundary

unidesk-ci.triggers.yaml installs the EventListener, TriggerBinding and TriggerTemplate, but the EventListener remains a normal in-cluster Service. Do not expose it through NodePort, LoadBalancer or an unrestricted public ingress. If GitHub or another Git remote needs webhook delivery, add a UniDesk-controlled frontend/backend route with secret verification and then proxy to the EventListener; keep only the documented main-server public entrypoints: production frontend, dev frontend proxy and provider ingress. The dev frontend public port is defined in docs/reference/dev-environment.md.