feat: standardize user service artifact producer

This commit is contained in:
Codex
2026-05-19 10:05:15 +00:00
parent e866b0526a
commit 292d0cee60
4 changed files with 588 additions and 7 deletions
+30 -1
View File
@@ -8,7 +8,7 @@ UniDesk CI is hosted on the D601 native k3s cluster with Tekton Pipelines and Te
- Tekton Triggers: `v0.34.0`.
- UniDesk CI namespace: `unidesk-ci`.
- Manifests: `src/components/microservices/k3sctl-adapter/k3s/ci/`.
- CLI entry: `bun scripts/cli.ts ci install|status|run|run-dev-e2e|logs`.
- CLI entry: `bun scripts/cli.ts ci install|status|run|publish-backend-core|publish-user-service|run-dev-e2e|logs`.
- Dev namespace e2e runner: `bun scripts/cli.ts ci run-dev-e2e`; authoritative runner path, manifest contract and safety boundary are in `docs/reference/dev-ci-runner.md`.
- Rust backend-core check/build boundary: CI may run `UNIDESK_D601_RUST_CHECK=1 bun scripts/cli.ts check --full --rust` on D601; the master server must not compile Rust for backend-core iteration. The authoritative dev environment rule is `docs/reference/dev-environment.md`.
@@ -76,6 +76,27 @@ The CI artifact task must follow these rules:
The artifact registry contract and CD consumption path are defined in `docs/reference/artifact-registry.md`. CI is the producer of the backend-core image artifact; CD is only the consumer.
## User-Service Artifact Publication
User-service image creation uses the same CI producer boundary as backend-core, but the service identity and Dockerfile come from the registered `config.json.microservices[]` entry. The minimal supported service is `decision-center`.
The CI user-service artifact task must follow these rules:
- Inputs are a pushed full 40-character Git commit and a registered service id. Dirty worktrees, operator-uploaded source trees and local-only commits are not valid artifact sources.
- D601 prepares a commit-pinned source export under `/home/ubuntu/.unidesk/ci/user-service-artifacts/<service-id>/<commit>` using the existing GitHub SSH deploy identity and node-local provider-gateway WS egress proxy. Tekton consumes that export through a read-only hostPath.
- The image is tagged only with the source commit and pushed to the D601 registry as `127.0.0.1:5000/unidesk/<service-id>:<commit>`. The producer must reject third-party registries and must not publish or consume a mutable `latest` tag.
- The image must carry `unidesk.ai/service-id`, `unidesk.ai/source-repo`, `unidesk.ai/source-commit` and `unidesk.ai/dockerfile` labels.
- The command output must include the image ref, tag, digest, source commit and service id. The digest ref is suitable as immutable input for later dev/prod deployment work.
- CI is an artifact producer only. It must not restart production services, call production `deploy apply`, mutate the production namespace, or change `deploy.json`.
Publish a Decision Center artifact:
```bash
bun scripts/cli.ts ci publish-user-service --service decision-center --commit <full-sha> --wait-ms 1200000
```
This command creates the `unidesk-user-service-artifact-publish` Tekton PipelineRun and pushes `127.0.0.1:5000/unidesk/decision-center:<commit>`.
## Dev Namespace E2E
`ci run-dev-e2e` is the manual dev desired-state smoke flow. The single authoritative reference for its Git-controlled runner script, short launcher, result directory and no-CD boundary is `docs/reference/dev-ci-runner.md`.
@@ -122,6 +143,14 @@ bun scripts/cli.ts ci publish-backend-core --commit <full-sha> --wait-ms 1200000
This command creates the `unidesk-backend-core-artifact-publish` Tekton PipelineRun. It is a CI producer action only: it may build and push `127.0.0.1:5000/unidesk/backend-core:<commit>`, but it must not recreate the master server container. Production deployment is triggered separately with `artifact-registry deploy-backend-core`.
Publish a user-service artifact:
```bash
bun scripts/cli.ts ci publish-user-service --service decision-center --commit <full-sha> --wait-ms 1200000
```
This command is a CI producer action only. For Decision Center, it builds and pushes `127.0.0.1:5000/unidesk/decision-center:<commit>` and reports the immutable digest without deploying production.
Run the dev namespace e2e harness manually:
```bash
+6
View File
@@ -32,6 +32,8 @@ The default release flow for a user-service change is:
- No user-service artifact may rely on a third-party registry as source of truth.
- No production deploy may rebuild the source from a dirty worktree.
- Commit-pinned image tags are the deployment truth; mutable `latest` tags are not.
- The standard CI artifact producer is `bun scripts/cli.ts ci publish-user-service --service <id> --commit <full-sha>`. It accepts only a pushed Git commit and a registered service id, and reports image ref, tag, digest, source commit and service id.
- The CI artifact producer is not a deploy executor. It must not mutate the production namespace, restart production services, or update `deploy.json`.
- Every production release must finish with a manual acceptance step after the automated checks pass.
## Frontend Pairing
@@ -42,10 +44,14 @@ Many user services are surfaced through the UniDesk frontend rather than through
Decision Center is the canonical example of a user service that doubles as a product workflow for requirements, decisions, and daily work diaries.
- The minimal standard artifact command is `bun scripts/cli.ts ci publish-user-service --service decision-center --commit <full-sha> --wait-ms 1200000`.
- The expected artifact is `127.0.0.1:5000/unidesk/decision-center:<commit>` plus its registry digest from the CI output.
- Requirements and follow-up items should be represented with structured records such as `goal`, `decision`, `blocker`, `debt`, and `experiment`, with linked evidence and goal references.
- The service should act as a demand-management surface for external goals that need to be decomposed into internal tasks, blockers, and decisions.
- The work-diary surface should support creating today's diary entry automatically from the real current date.
- Historical diary entries should be editable by date, with PostgreSQL remaining the source of truth.
- The resulting product surface should stay structured and reviewable, not collapse into an untracked chat log.
- Before any production deploy, the dev gate must pass first with the focused Decision Center E2E set covering `microservice:decision-center-record-crud`, `microservice:decision-center-diary-lifecycle`, `frontend:decision-center-visible`, `frontend:decision-center-demand-management-visible`, and `frontend:decision-center-diary-visible`; this is an automated user-service validation gate, not a production deploy.
- Production acceptance is manual after the dev gate and must explicitly verify `health`, `records`, `diary editor`, the paired frontend page, no public business ports, and live commit / artifact information.
Detailed service-level API and UI contracts remain in `docs/reference/microservices.md`.
+341 -5
View File
@@ -1,7 +1,8 @@
import { randomUUID } from "node:crypto";
import { existsSync, readFileSync } from "node:fs";
import { posix as posixPath } from "node:path";
import { runCommand } from "./command";
import { type UniDeskConfig, repoRoot, rootPath } from "./config";
import { type UniDeskConfig, type UniDeskMicroserviceConfig, repoRoot, rootPath } from "./config";
import { ensureGithubSshIdentityForProvider, gitSshHttpConnectProxySource } from "./deploy-ssh-identity";
import { startJob } from "./jobs";
import { coreInternalFetch } from "./microservices";
@@ -42,6 +43,16 @@ interface CiPublishBackendCoreOptions {
sourceHostPath: string;
}
interface CiPublishUserServiceArtifactOptions {
repoUrl: string;
commit: string;
waitMs: number;
sourceHostPath: string;
serviceId: string;
dockerfile: string;
dryRun: boolean;
}
interface CiDevE2EOptions {
repoUrl: string;
desiredRef: string;
@@ -92,6 +103,19 @@ interface DeployDevManifestSummary {
services: Array<{ id: string; commitId: string; repo: string }>;
}
interface ArtifactSummary {
serviceId: string;
sourceCommit: string;
sourceRepo: string;
dockerfile: string;
registry: string;
repository: string;
tag: string;
imageRef: string;
digest: string | null;
digestRef: string | null;
}
function stringOption(args: string[], name: string): string | null {
const index = args.indexOf(name);
if (index === -1) return null;
@@ -120,6 +144,14 @@ function requireFullCommit(value: string | null, option = "--commit"): string {
return commit;
}
function requireServiceId(value: string | null): string {
const serviceId = value ?? "";
if (!/^[a-z0-9]([-a-z0-9]{0,62}[a-z0-9])?$/u.test(serviceId)) {
throw new Error("ci publish-user-service requires --service <dns-safe-user-service-id>");
}
return serviceId;
}
function requireDesiredRef(value: string | null): string {
const ref = value ?? "master";
if (!/^[A-Za-z0-9._/-]{1,160}$/u.test(ref) || ref.startsWith("-") || ref.includes("..")) {
@@ -149,6 +181,30 @@ function repoSshUrl(repoUrl: string): string {
return repoUrl;
}
function requireRepoRelativePath(path: string, label: string): string {
if (path.length === 0 || path.startsWith("/") || path.includes("\0") || path.split("/").includes("..")) {
throw new Error(`${label} must be a repo-relative path`);
}
const normalized = posixPath.normalize(path);
if (normalized === "." || normalized.startsWith("../")) throw new Error(`${label} must be a repo-relative path`);
return normalized;
}
function requireSupportedUserService(config: UniDeskConfig, serviceId: string): UniDeskMicroserviceConfig {
if (serviceId === "backend-core") {
throw new Error("backend-core uses ci publish-backend-core; publish-user-service is for registered user services");
}
const service = config.microservices.find((item) => item.id === serviceId);
if (service === undefined) throw new Error(`unknown user service: ${serviceId}`);
if (service.providerId !== d601ProviderId || service.development.providerId !== d601ProviderId) {
throw new Error(`ci publish-user-service currently supports only D601 user services; ${serviceId} is assigned to ${service.providerId}`);
}
if (service.deployment.mode !== "k3sctl-managed") {
throw new Error(`ci publish-user-service currently supports k3sctl-managed D601 user services; ${serviceId} uses ${service.deployment.mode}`);
}
return service;
}
function chunks(value: string, size: number): string[] {
const result: string[] = [];
for (let index = 0; index < value.length; index += size) {
@@ -542,10 +598,49 @@ spec:
`;
}
function userServiceArtifactPipelineRunManifest(options: CiPublishUserServiceArtifactOptions): string {
const safeSuffix = new Date().toISOString().replace(/[-:.TZ]/g, "").slice(0, 14).toLowerCase();
return `apiVersion: tekton.dev/v1
kind: PipelineRun
metadata:
generateName: user-service-artifact-${options.serviceId}-${safeSuffix}-
namespace: unidesk-ci
labels:
app.kubernetes.io/name: unidesk-user-service-artifact-publish
app.kubernetes.io/part-of: unidesk
unidesk.ai/service-id: ${JSON.stringify(options.serviceId)}
unidesk.ai/revision: ${JSON.stringify(options.commit)}
spec:
pipelineRef:
name: unidesk-user-service-artifact-publish
taskRunTemplate:
serviceAccountName: unidesk-ci-runner
params:
- name: repo-url
value: ${JSON.stringify(options.repoUrl)}
- name: revision
value: ${JSON.stringify(options.commit)}
- name: service-id
value: ${JSON.stringify(options.serviceId)}
- name: dockerfile
value: ${JSON.stringify(options.dockerfile)}
- name: source-host-path
value: ${JSON.stringify(options.sourceHostPath)}
workspaces:
- name: shared-workspace
persistentVolumeClaim:
claimName: unidesk-ci-cache
`;
}
function backendCoreArtifactSourceHostPath(commit: string): string {
return `/home/ubuntu/.unidesk/ci/backend-core-artifacts/${commit}`;
}
function userServiceArtifactSourceHostPath(serviceId: string, commit: string): string {
return `/home/ubuntu/.unidesk/ci/user-service-artifacts/${serviceId}/${commit}`;
}
async function prepareBackendCoreArtifactSource(config: UniDeskConfig, options: CiPublishBackendCoreOptions): Promise<Record<string, unknown>> {
const sshIdentity = await ensureGithubSshIdentityForProvider(config, d601ProviderId);
if (!sshIdentity.ok) throw new Error(sshIdentity.detail);
@@ -613,6 +708,82 @@ async function prepareBackendCoreArtifactSource(config: UniDeskConfig, options:
};
}
async function prepareUserServiceArtifactSource(config: UniDeskConfig, options: CiPublishUserServiceArtifactOptions): Promise<Record<string, unknown>> {
const sshIdentity = await ensureGithubSshIdentityForProvider(config, d601ProviderId);
if (!sshIdentity.ok) throw new Error(sshIdentity.detail);
const proxyPython = gitSshHttpConnectProxySource();
const sourceRoot = `/home/ubuntu/.unidesk/ci/user-service-artifacts/${options.serviceId}`;
const sourceHostPath = options.sourceHostPath;
const repoCache = "/home/ubuntu/.unidesk/ci/git/unidesk.git";
const repoFetchUrl = repoSshUrl(options.repoUrl);
const dockerfileDir = posixPath.dirname(options.dockerfile);
const script = [
"set -euo pipefail",
`service_id=${shellQuote(options.serviceId)}`,
`commit=${shellQuote(options.commit)}`,
`repo_url=${shellQuote(options.repoUrl)}`,
`repo_fetch_url=${shellQuote(repoFetchUrl)}`,
`dockerfile=${shellQuote(options.dockerfile)}`,
`dockerfile_dir=${shellQuote(dockerfileDir)}`,
`source_root=${shellQuote(sourceRoot)}`,
`source_dir=${shellQuote(sourceHostPath)}`,
`repo_cache=${shellQuote(repoCache)}`,
`proxy_url=${shellQuote(providerGatewayWsEgressProxyUrl)}`,
"mkdir -p \"$(dirname \"$repo_cache\")\" \"$source_root\"",
"export HTTP_PROXY=\"$proxy_url\" HTTPS_PROXY=\"$proxy_url\" ALL_PROXY=\"$proxy_url\"",
"export NO_PROXY=\"localhost,127.0.0.1,::1,host.docker.internal,.svc,.cluster.local,kubernetes.default.svc\"",
"curl -fsSI --max-time 20 -x \"$proxy_url\" https://github.com >/dev/null",
"git_ssh_proxy=/tmp/unidesk-git-ssh-http-connect.py",
"cat > \"$git_ssh_proxy\" <<'UNIDESK_GIT_SSH_PROXY'",
proxyPython,
"UNIDESK_GIT_SSH_PROXY",
"chmod 700 \"$git_ssh_proxy\"",
"export UNIDESK_GIT_SSH_HTTP_PROXY=\"$proxy_url\"",
"export GIT_SSH_COMMAND=\"ssh -o BatchMode=yes -o IdentitiesOnly=yes -o StrictHostKeyChecking=yes -o UserKnownHostsFile=$HOME/.ssh/known_hosts -i $HOME/.ssh/id_ed25519 -o 'ProxyCommand=$git_ssh_proxy %h %p'\"",
"echo user_service_artifact_source_proxy=provider-gateway-ws-egress:$proxy_url",
"echo user_service_artifact_repo_fetch_url=$repo_fetch_url",
"echo user_service_artifact_service_id=$service_id",
"if [ ! -d \"$repo_cache\" ]; then git clone --mirror \"$repo_fetch_url\" \"$repo_cache\"; fi",
"git -C \"$repo_cache\" remote set-url origin \"$repo_fetch_url\"",
"git -C \"$repo_cache\" fetch --no-tags origin \"$commit\" || git -C \"$repo_cache\" fetch --no-tags origin '+refs/heads/*:refs/remotes/origin/*'",
"resolved=$(git -C \"$repo_cache\" rev-parse --verify \"$commit^{commit}\")",
"test \"$resolved\" = \"$commit\" || { echo \"user_service_artifact_resolved_commit_mismatch=$resolved expected=$commit\" >&2; exit 1; }",
"git -C \"$repo_cache\" cat-file -e \"$commit:$dockerfile\"",
"git -C \"$repo_cache\" cat-file -e \"$commit:$dockerfile_dir/src\"",
"tmp_dir=\"$source_root/.tmp-$commit-$$\"",
"rm -rf \"$tmp_dir\"",
"mkdir -p \"$tmp_dir\"",
"git -C \"$repo_cache\" archive \"$commit\" | tar -x -C \"$tmp_dir\"",
"printf '%s\\n' \"$commit\" > \"$tmp_dir/.unidesk-source-commit\"",
"printf '%s\\n' \"$repo_url\" > \"$tmp_dir/.unidesk-source-repo\"",
"printf '%s\\n' \"$service_id\" > \"$tmp_dir/.unidesk-service-id\"",
"printf '%s\\n' \"$dockerfile\" > \"$tmp_dir/.unidesk-dockerfile\"",
"rm -rf \"$source_dir\"",
"mv \"$tmp_dir\" \"$source_dir\"",
"test -f \"$source_dir/$dockerfile\"",
"test -d \"$source_dir/$dockerfile_dir/src\"",
"echo user_service_artifact_source_host_path=$source_dir",
].join("\n");
const result = await runRemoteBackground(`prepare-${options.serviceId}-source`, script, 300_000);
if (!result.ok) throw new Error(`failed to prepare ${options.serviceId} source on D601: ${result.stderr || result.stdout || JSON.stringify(result.raw)}`);
return {
ok: true,
mode: "d601-host-github-ssh-export",
providerId: d601ProviderId,
repoUrl: options.repoUrl,
repoFetchUrl,
commit: options.commit,
serviceId: options.serviceId,
dockerfile: options.dockerfile,
sourceHostPath,
identity: {
fingerprint: sshIdentity.fingerprint,
seededFromLocal: sshIdentity.seededFromLocal,
},
stdoutTail: result.stdout.slice(-4000),
};
}
async function remoteCreatePipelineRun(manifest: string): Promise<string> {
const encoded = Buffer.from(manifest, "utf8").toString("base64");
const token = randomUUID().replace(/-/gu, "").slice(0, 12);
@@ -689,6 +860,46 @@ function pipelineRunWaitSucceeded(wait: DispatchResult | null, condition: Pipeli
return wait.ok || wait.exitCode === 0 || wait.stdout.includes("condition=True\tSucceeded\t");
}
function parseArtifactSummaryFromOutput(output: string, serviceId: string, commit: string, repoUrl: string, dockerfile: string): ArtifactSummary {
const fields = new Map<string, string>();
for (const line of output.split(/\r?\n/u)) {
const match = /^(user_service_artifact_[a-z_]+|backend_core_artifact_[a-z_]+)=(.*)$/u.exec(line.trim());
if (match !== null) fields.set(match[1], match[2]);
}
const digest = fields.get("user_service_artifact_digest") ?? fields.get("backend_core_artifact_digest") ?? null;
const imageRef = fields.get("user_service_artifact_image") ?? fields.get("backend_core_artifact_image") ?? `127.0.0.1:5000/unidesk/${serviceId}:${commit}`;
const repository = fields.get("user_service_artifact_repository") ?? `127.0.0.1:5000/unidesk/${serviceId}`;
const digestRef = fields.get("user_service_artifact_digest_ref") ?? fields.get("backend_core_artifact_digest_ref") ?? (digest === null || digest.length === 0 ? null : `${repository}@${digest}`);
return {
serviceId: fields.get("user_service_artifact_service_id") ?? serviceId,
sourceCommit: fields.get("user_service_artifact_source_commit") ?? fields.get("backend_core_artifact_source_commit") ?? commit,
sourceRepo: fields.get("user_service_artifact_source_repo") ?? repoUrl,
dockerfile: fields.get("user_service_artifact_dockerfile") ?? dockerfile,
registry: fields.get("user_service_artifact_registry") ?? "127.0.0.1:5000",
repository,
tag: fields.get("user_service_artifact_tag") ?? commit,
imageRef,
digest,
digestRef,
};
}
function assertArtifactSummaryComplete(artifact: ArtifactSummary, pipelineRun: string): void {
if (artifact.serviceId.length === 0 || artifact.sourceCommit.length !== 40 || artifact.imageRef.length === 0 || artifact.tag.length === 0 || artifact.digest === null || artifact.digest.length === 0 || artifact.digestRef === null || artifact.digestRef.length === 0) {
throw new Error(`artifact summary for ${pipelineRun} is incomplete`);
}
}
async function readPipelineRunLogText(name: string): Promise<string> {
const result = await runRemoteKubectlRaw([
"set -euo pipefail",
`kubectl get pipelinerun/${shellQuote(name)} -n unidesk-ci -o wide`,
`kubectl get taskrun -n unidesk-ci -l tekton.dev/pipelineRun=${shellQuote(name)} -o wide`,
`for pod in $(kubectl get pods -n unidesk-ci -l tekton.dev/pipelineRun=${shellQuote(name)} -o name); do echo "===== $pod"; kubectl logs -n unidesk-ci "$pod" --all-containers=true --tail=240 || true; done`,
].join("\n"), 60_000, 45_000);
return `${result.stdout}\n${result.stderr}`.trim();
}
async function run(options: CiOptions): Promise<Record<string, unknown>> {
const name = await remoteCreatePipelineRun(pipelineRunManifest(options));
const wait = await waitForPipelineRun(name, options.waitMs);
@@ -717,11 +928,27 @@ async function run(options: CiOptions): Promise<Record<string, unknown>> {
}
async function publishBackendCoreArtifact(config: UniDeskConfig, options: CiPublishBackendCoreOptions): Promise<Record<string, unknown>> {
const plannedArtifact: ArtifactSummary = {
serviceId: "backend-core",
sourceCommit: options.commit,
sourceRepo: options.repoUrl,
dockerfile: "src/components/backend-core/Dockerfile",
registry: "127.0.0.1:5000",
repository: "127.0.0.1:5000/unidesk/backend-core",
tag: options.commit,
imageRef: `127.0.0.1:5000/unidesk/backend-core:${options.commit}`,
digest: null,
digestRef: null,
};
const source = await prepareBackendCoreArtifactSource(config, options);
const name = await remoteCreatePipelineRun(backendCoreArtifactPipelineRunManifest(options));
const wait = await waitForPipelineRun(name, options.waitMs);
const condition = wait === null ? null : await readPipelineRunCondition(name);
const waitSucceeded = pipelineRunWaitSucceeded(wait, condition);
const artifact = waitSucceeded && wait !== null
? parseArtifactSummaryFromOutput(await readPipelineRunLogText(name), "backend-core", options.commit, options.repoUrl, "src/components/backend-core/Dockerfile")
: plannedArtifact;
if (waitSucceeded && wait !== null) assertArtifactSummaryComplete(artifact, name);
return {
ok: waitSucceeded,
pipelineRun: name,
@@ -729,7 +956,8 @@ async function publishBackendCoreArtifact(config: UniDeskConfig, options: CiPubl
repoUrl: options.repoUrl,
commit: options.commit,
source,
artifact: `127.0.0.1:5000/unidesk/backend-core:${options.commit}`,
artifact: artifact.imageRef,
artifactSummary: artifact,
boundary: "CI publishes the image to D601 registry; CD must pull it and must not build backend-core",
wait: wait === null ? null : {
ok: waitSucceeded,
@@ -747,6 +975,84 @@ async function publishBackendCoreArtifact(config: UniDeskConfig, options: CiPubl
};
}
async function publishUserServiceArtifact(config: UniDeskConfig, options: CiPublishUserServiceArtifactOptions): Promise<Record<string, unknown>> {
const plannedArtifact: ArtifactSummary = {
serviceId: options.serviceId,
sourceCommit: options.commit,
sourceRepo: options.repoUrl,
dockerfile: options.dockerfile,
registry: "127.0.0.1:5000",
repository: `127.0.0.1:5000/unidesk/${options.serviceId}`,
tag: options.commit,
imageRef: `127.0.0.1:5000/unidesk/${options.serviceId}:${options.commit}`,
digest: null,
digestRef: null,
} satisfies ArtifactSummary;
if (options.dryRun) {
return {
ok: true,
mode: "dry-run",
pipeline: "unidesk-user-service-artifact-publish",
namespace: "unidesk-ci",
repoUrl: options.repoUrl,
commit: options.commit,
serviceId: options.serviceId,
sourceHostPath: options.sourceHostPath,
source: {
ok: true,
mode: "planned-only",
providerId: d601ProviderId,
repoUrl: options.repoUrl,
repoFetchUrl: repoSshUrl(options.repoUrl),
commit: options.commit,
serviceId: options.serviceId,
dockerfile: options.dockerfile,
sourceHostPath: options.sourceHostPath,
},
artifact: plannedArtifact.imageRef,
artifactSummary: plannedArtifact,
boundary: "dry-run only; no D601 source export, no Tekton submission, no production mutation",
next: [
`bun scripts/cli.ts ci publish-user-service --service ${options.serviceId} --commit ${options.commit} --wait-ms 1200000`,
],
};
}
const source = await prepareUserServiceArtifactSource(config, options);
const name = await remoteCreatePipelineRun(userServiceArtifactPipelineRunManifest(options));
const wait = await waitForPipelineRun(name, options.waitMs);
const condition = wait === null ? null : await readPipelineRunCondition(name);
const waitSucceeded = pipelineRunWaitSucceeded(wait, condition);
const artifact = waitSucceeded && wait !== null
? parseArtifactSummaryFromOutput(await readPipelineRunLogText(name), options.serviceId, options.commit, options.repoUrl, options.dockerfile)
: plannedArtifact;
if (waitSucceeded && wait !== null) assertArtifactSummaryComplete(artifact, name);
return {
ok: waitSucceeded,
pipelineRun: name,
namespace: "unidesk-ci",
repoUrl: options.repoUrl,
commit: options.commit,
serviceId: options.serviceId,
source,
artifact: artifact.imageRef,
artifactSummary: artifact,
boundary: "CI publishes the user-service image to the D601 registry only; it must not deploy production or mutate the production namespace",
wait: wait === null ? null : {
ok: waitSucceeded,
dispatchOk: wait.ok,
dispatchStatus: wait.status,
dispatchExitCode: wait.exitCode,
stdoutTail: wait.stdout.slice(-6000),
stderrTail: wait.stderr.slice(-6000),
},
condition,
next: [
`bun scripts/cli.ts ci logs ${name}`,
"use artifactSummary.imageRef or artifactSummary.digestRef as later dev/prod deployment input",
],
};
}
async function runRemoteDevE2ELauncher(options: CiDevE2EOptions): Promise<DispatchResult> {
const scriptTimeoutMs = Math.max(options.scriptTimeoutMs, options.waitMs, 60_000);
const remoteTimeoutMs = 45_000;
@@ -995,12 +1301,13 @@ async function logs(name: string): Promise<Record<string, unknown>> {
export function ciHelp(): Record<string, unknown> {
return {
command: "ci install|status|run|publish-backend-core|run-dev-e2e|logs",
description: "Manage the D601 k3s Tekton CI gate. CI may publish backend-core image artifacts, but it intentionally does not deploy CD.",
command: "ci install|status|run|publish-backend-core|publish-user-service|run-dev-e2e|logs",
description: "Manage the D601 k3s Tekton CI gate. CI may publish commit-pinned image artifacts, but it intentionally does not deploy CD.",
examples: [
"bun scripts/cli.ts ci install",
"bun scripts/cli.ts ci run --revision <commit>",
"bun scripts/cli.ts ci publish-backend-core --commit <full-sha>",
"bun scripts/cli.ts ci publish-user-service --service decision-center --commit <full-sha>",
"bun scripts/cli.ts ci run-dev-e2e --wait-ms 600000",
"bun scripts/cli.ts ci logs <runId>",
],
@@ -1018,6 +1325,14 @@ export function ciHelp(): Record<string, unknown> {
registry: "127.0.0.1:5000/unidesk/backend-core:<commit>",
cdCommand: "bun scripts/cli.ts artifact-registry deploy-backend-core --commit <full-sha>",
},
userServiceArtifact: {
producer: "D601 CI",
command: "bun scripts/cli.ts ci publish-user-service --service <service-id> --commit <full-sha>",
initiallySupportedServices: ["decision-center"],
registry: "127.0.0.1:5000/unidesk/<service-id>:<commit>",
outputFields: ["imageRef", "tag", "digest", "sourceCommit", "serviceId"],
boundary: "artifact producer only; no prod deploy and no production namespace mutation",
},
runDevE2E: {
defaultTriggerMode: "commit-pinned-ssh-launcher",
desiredState: "origin/master:deploy.json#environments.dev",
@@ -1050,6 +1365,27 @@ export async function runCiCommand(config: UniDeskConfig, args: string[]): Promi
const waitMs = numberOption(args, "--wait-ms", 0);
return publishBackendCoreArtifact(config, { repoUrl, commit, waitMs, sourceHostPath: backendCoreArtifactSourceHostPath(commit) });
}
if (action === "publish-user-service") {
const serviceId = requireServiceId(stringOption(args, "--service") ?? stringOption(args, "--service-id"));
const service = requireSupportedUserService(config, serviceId);
const repoUrl = stringOption(args, "--repo") ?? stringOption(args, "--repo-url") ?? service.repository.url;
const commit = requireFullCommit(stringOption(args, "--commit") ?? stringOption(args, "--revision"));
const waitMs = numberOption(args, "--wait-ms", 0);
const dryRun = boolFlag(args, "--dry-run");
const dockerfile = requireRepoRelativePath(service.repository.dockerfile, `microservices.${serviceId}.repository.dockerfile`);
if (serviceId !== "decision-center") {
throw new Error("ci publish-user-service currently allows only decision-center until each user-service Dockerfile contract is reviewed");
}
return publishUserServiceArtifact(config, {
repoUrl,
commit,
waitMs,
serviceId,
dockerfile,
sourceHostPath: userServiceArtifactSourceHostPath(serviceId, commit),
dryRun,
});
}
if (action === "run-dev-e2e") {
const repoUrl = stringOption(args, "--repo") ?? stringOption(args, "--repo-url") ?? "https://github.com/pikasTech/unidesk";
const desiredRef = requireDesiredRef(stringOption(args, "--desired-ref") ?? stringOption(args, "--deploy-branch"));
@@ -1071,7 +1407,7 @@ export async function runCiCommand(config: UniDeskConfig, args: string[]): Promi
});
}
if (action === "logs") return logs(nameArg ?? "");
throw new Error("ci command must be one of: install, status, run, publish-backend-core, run-dev-e2e, logs");
throw new Error("ci command must be one of: install, status, run, publish-backend-core, publish-user-service, run-dev-e2e, logs");
}
export function startCiInstallJob(): Record<string, unknown> {
@@ -400,8 +400,10 @@ spec:
set -euo pipefail
commit="$(cat "$(workspaces.source.path)/backend-core-artifact-commit.txt")"
registry="$(params.registry)"
test "$registry" = "127.0.0.1:5000" || { echo "backend_core_artifact_registry_must_be_d601_loopback=$registry" >&2; exit 2; }
local_image="unidesk/backend-core:$commit"
registry_image="$registry/unidesk/backend-core:$commit"
repository="$registry/unidesk/backend-core"
command -v docker
docker version >/dev/null
docker run --rm --network host rancher/mirrored-library-busybox:1.36.1 wget -q -O- "http://$registry/v2/" >/dev/null
@@ -422,8 +424,11 @@ spec:
actual_commit="$(docker image inspect "$registry_image" --format '{{ index .Config.Labels "unidesk.ai/source-commit" }}')"
test "$actual_commit" = "$commit"
docker push "$registry_image"
docker pull "$registry_image" >/dev/null
repo_digests="$(docker image inspect "$registry_image" --format '{{json .RepoDigests}}')"
printf 'backend_core_artifact_image=%s\nbackend_core_artifact_repo_digests=%s\n' "$registry_image" "$repo_digests"
digest="$(docker image inspect "$registry_image" --format '{{range .RepoDigests}}{{println .}}{{end}}' | awk -F@ -v repo="$repository" '$1 == repo { print $2; exit }')"
test -n "$digest"
printf 'backend_core_artifact_service_id=backend-core\nbackend_core_artifact_image=%s\nbackend_core_artifact_repository=%s\nbackend_core_artifact_tag=%s\nbackend_core_artifact_digest=%s\nbackend_core_artifact_digest_ref=%s@%s\nbackend_core_artifact_source_commit=%s\nbackend_core_artifact_source_repo=%s\nbackend_core_artifact_dockerfile=src/components/backend-core/Dockerfile\nbackend_core_artifact_repo_digests=%s\n' "$registry_image" "$repository" "$commit" "$digest" "$repository" "$digest" "$commit" "$(params.repo-url)" "$repo_digests"
---
apiVersion: tekton.dev/v1
kind: Pipeline
@@ -472,6 +477,211 @@ spec:
---
apiVersion: tekton.dev/v1
kind: Task
metadata:
name: unidesk-user-service-artifact-publish
namespace: unidesk-ci
labels:
app.kubernetes.io/name: unidesk-ci
app.kubernetes.io/component: user-service-artifact
spec:
params:
- name: repo-url
type: string
- name: revision
type: string
- name: service-id
type: string
- name: dockerfile
type: string
- name: app-image
type: string
default: unidesk-code-queue:dev
- name: registry
type: string
default: 127.0.0.1:5000
- name: source-host-path
type: string
workspaces:
- name: source
volumes:
- name: docker-sock
hostPath:
path: /var/run/docker.sock
type: Socket
- name: prepared-source
hostPath:
path: /home/ubuntu/.unidesk/ci/user-service-artifacts
type: Directory
steps:
- name: prepare-source
image: alpine/git:2.45.2
volumeMounts:
- name: prepared-source
mountPath: /prepared-source
readOnly: true
script: |
#!/bin/sh
set -eu
service_id="$(params.service-id)"
dockerfile="$(params.dockerfile)"
source_host_path="$(params.source-host-path)"
case "$(params.revision)" in
[0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f]) ;;
*) echo "user_service_artifact_revision_must_be_full_sha=$(params.revision)" >&2; exit 2 ;;
esac
printf '%s\n' "$service_id" | grep -Eq '^[a-z0-9]([-a-z0-9]{0,62}[a-z0-9])?$' || { echo "user_service_artifact_service_id_invalid=$service_id" >&2; exit 2; }
case "$dockerfile" in
/*|*..*|""|*latest*) echo "user_service_artifact_dockerfile_invalid=$dockerfile" >&2; exit 2 ;;
esac
expected_prefix="/home/ubuntu/.unidesk/ci/user-service-artifacts/$service_id/"
case "$source_host_path" in
"$expected_prefix"*) ;;
*) echo "user_service_artifact_source_host_path_invalid=$source_host_path" >&2; exit 2 ;;
esac
mkdir -p "$(workspaces.source.path)/user-service-artifact-repo"
find "$(workspaces.source.path)/user-service-artifact-repo" -mindepth 1 -maxdepth 1 -exec rm -rf {} +
source_name="$(basename "$source_host_path")"
source_dir="/prepared-source/$service_id/$source_name"
test -f "$source_dir/.unidesk-source-commit"
test -f "$source_dir/.unidesk-service-id"
test -f "$source_dir/.unidesk-dockerfile"
prepared_commit="$(cat "$source_dir/.unidesk-source-commit")"
prepared_service_id="$(cat "$source_dir/.unidesk-service-id")"
prepared_dockerfile="$(cat "$source_dir/.unidesk-dockerfile")"
test "$prepared_commit" = "$(params.revision)"
test "$prepared_service_id" = "$service_id"
test "$prepared_dockerfile" = "$dockerfile"
cp -a "$source_dir/." "$(workspaces.source.path)/user-service-artifact-repo/"
cd "$(workspaces.source.path)/user-service-artifact-repo"
test -f "$dockerfile"
test -d "$(dirname "$dockerfile")/src"
printf '%s\n' "$prepared_commit" | tee "$(workspaces.source.path)/user-service-artifact-commit.txt"
printf '%s\n' "$service_id" | tee "$(workspaces.source.path)/user-service-artifact-service-id.txt"
printf '%s\n' "$dockerfile" | tee "$(workspaces.source.path)/user-service-artifact-dockerfile.txt"
- name: build-and-push
image: "$(params.app-image)"
imagePullPolicy: Never
workingDir: "$(workspaces.source.path)/user-service-artifact-repo"
env:
- name: DOCKER_HOST
value: unix:///var/run/docker.sock
- name: HTTP_PROXY
value: "http://d601-provider-egress-proxy.unidesk.svc.cluster.local:18789"
- name: HTTPS_PROXY
value: "http://d601-provider-egress-proxy.unidesk.svc.cluster.local:18789"
- name: ALL_PROXY
value: "http://d601-provider-egress-proxy.unidesk.svc.cluster.local:18789"
- name: NO_PROXY
value: "localhost,127.0.0.1,::1,host.docker.internal,d601-provider-egress-proxy,d601-provider-egress-proxy.unidesk,d601-provider-egress-proxy.unidesk.svc,d601-provider-egress-proxy.unidesk.svc.cluster.local"
- name: http_proxy
value: "http://d601-provider-egress-proxy.unidesk.svc.cluster.local:18789"
- name: https_proxy
value: "http://d601-provider-egress-proxy.unidesk.svc.cluster.local:18789"
- name: all_proxy
value: "http://d601-provider-egress-proxy.unidesk.svc.cluster.local:18789"
- name: no_proxy
value: "localhost,127.0.0.1,::1,host.docker.internal,d601-provider-egress-proxy,d601-provider-egress-proxy.unidesk,d601-provider-egress-proxy.unidesk.svc,d601-provider-egress-proxy.unidesk.svc.cluster.local"
volumeMounts:
- name: docker-sock
mountPath: /var/run/docker.sock
script: |
#!/usr/bin/env bash
set -euo pipefail
commit="$(cat "$(workspaces.source.path)/user-service-artifact-commit.txt")"
service_id="$(cat "$(workspaces.source.path)/user-service-artifact-service-id.txt")"
dockerfile="$(cat "$(workspaces.source.path)/user-service-artifact-dockerfile.txt")"
registry="$(params.registry)"
test "$registry" = "127.0.0.1:5000" || { echo "user_service_artifact_registry_must_be_d601_loopback=$registry" >&2; exit 2; }
case "$commit" in
[0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f]) ;;
*) echo "user_service_artifact_commit_invalid=$commit" >&2; exit 2 ;;
esac
local_image="unidesk/$service_id:$commit"
repository="$registry/unidesk/$service_id"
registry_image="$repository:$commit"
command -v docker
docker version >/dev/null
docker run --rm --network host rancher/mirrored-library-busybox:1.36.1 wget -q -O- "http://$registry/v2/" >/dev/null
DOCKER_BUILDKIT=0 docker build \
--network host \
--build-arg HTTP_PROXY=http://127.0.0.1:18789 \
--build-arg HTTPS_PROXY=http://127.0.0.1:18789 \
--build-arg ALL_PROXY=http://127.0.0.1:18789 \
--build-arg NO_PROXY=localhost,127.0.0.1,::1,host.docker.internal \
--label "unidesk.ai/service-id=$service_id" \
--label "unidesk.ai/source-repo=$(params.repo-url)" \
--label "unidesk.ai/source-commit=$commit" \
--label "unidesk.ai/dockerfile=$dockerfile" \
-t "$local_image" \
-t "$registry_image" \
-f "$dockerfile" \
.
actual_service="$(docker image inspect "$registry_image" --format '{{ index .Config.Labels "unidesk.ai/service-id" }}')"
actual_commit="$(docker image inspect "$registry_image" --format '{{ index .Config.Labels "unidesk.ai/source-commit" }}')"
test "$actual_service" = "$service_id"
test "$actual_commit" = "$commit"
docker push "$registry_image"
docker pull "$registry_image" >/dev/null
repo_digests="$(docker image inspect "$registry_image" --format '{{json .RepoDigests}}')"
digest="$(docker image inspect "$registry_image" --format '{{range .RepoDigests}}{{println .}}{{end}}' | awk -F@ -v repo="$repository" '$1 == repo { print $2; exit }')"
test -n "$digest"
printf 'user_service_artifact_service_id=%s\nuser_service_artifact_image=%s\nuser_service_artifact_repository=%s\nuser_service_artifact_tag=%s\nuser_service_artifact_digest=%s\nuser_service_artifact_digest_ref=%s@%s\nuser_service_artifact_source_commit=%s\nuser_service_artifact_source_repo=%s\nuser_service_artifact_dockerfile=%s\nuser_service_artifact_registry=%s\nuser_service_artifact_repo_digests=%s\n' "$service_id" "$registry_image" "$repository" "$commit" "$digest" "$repository" "$digest" "$commit" "$(params.repo-url)" "$dockerfile" "$registry" "$repo_digests"
---
apiVersion: tekton.dev/v1
kind: Pipeline
metadata:
name: unidesk-user-service-artifact-publish
namespace: unidesk-ci
labels:
app.kubernetes.io/name: unidesk-ci
app.kubernetes.io/component: user-service-artifact
app.kubernetes.io/part-of: unidesk
spec:
params:
- name: repo-url
type: string
default: https://github.com/pikasTech/unidesk
- name: revision
type: string
- name: service-id
type: string
- name: dockerfile
type: string
- name: app-image
type: string
default: unidesk-code-queue:dev
- name: registry
type: string
default: 127.0.0.1:5000
- name: source-host-path
type: string
workspaces:
- name: shared-workspace
tasks:
- name: publish-user-service-artifact
taskRef:
name: unidesk-user-service-artifact-publish
params:
- name: repo-url
value: "$(params.repo-url)"
- name: revision
value: "$(params.revision)"
- name: service-id
value: "$(params.service-id)"
- name: dockerfile
value: "$(params.dockerfile)"
- name: app-image
value: "$(params.app-image)"
- name: registry
value: "$(params.registry)"
- name: source-host-path
value: "$(params.source-host-path)"
workspaces:
- name: source
workspace: shared-workspace
---
apiVersion: tekton.dev/v1
kind: Task
metadata:
name: unidesk-code-queue-read-perf
namespace: unidesk-ci