From 292d0cee604c52ce11f68f93bc922c6aefa45e93 Mon Sep 17 00:00:00 2001 From: Codex Date: Tue, 19 May 2026 10:05:15 +0000 Subject: [PATCH] feat: standardize user service artifact producer --- docs/reference/ci.md | 31 +- docs/reference/user-service-delivery.md | 6 + scripts/src/ci.ts | 346 +++++++++++++++++- .../k3s/ci/unidesk-ci.pipeline.yaml | 212 ++++++++++- 4 files changed, 588 insertions(+), 7 deletions(-) diff --git a/docs/reference/ci.md b/docs/reference/ci.md index f49f0448..c0fd1411 100644 --- a/docs/reference/ci.md +++ b/docs/reference/ci.md @@ -8,7 +8,7 @@ UniDesk CI is hosted on the D601 native k3s cluster with Tekton Pipelines and Te - Tekton Triggers: `v0.34.0`. - UniDesk CI namespace: `unidesk-ci`. - Manifests: `src/components/microservices/k3sctl-adapter/k3s/ci/`. -- CLI entry: `bun scripts/cli.ts ci install|status|run|run-dev-e2e|logs`. +- CLI entry: `bun scripts/cli.ts ci install|status|run|publish-backend-core|publish-user-service|run-dev-e2e|logs`. - Dev namespace e2e runner: `bun scripts/cli.ts ci run-dev-e2e`; authoritative runner path, manifest contract and safety boundary are in `docs/reference/dev-ci-runner.md`. - Rust backend-core check/build boundary: CI may run `UNIDESK_D601_RUST_CHECK=1 bun scripts/cli.ts check --full --rust` on D601; the master server must not compile Rust for backend-core iteration. The authoritative dev environment rule is `docs/reference/dev-environment.md`. @@ -76,6 +76,27 @@ The CI artifact task must follow these rules: The artifact registry contract and CD consumption path are defined in `docs/reference/artifact-registry.md`. CI is the producer of the backend-core image artifact; CD is only the consumer. +## User-Service Artifact Publication + +User-service image creation uses the same CI producer boundary as backend-core, but the service identity and Dockerfile come from the registered `config.json.microservices[]` entry. The minimal supported service is `decision-center`. + +The CI user-service artifact task must follow these rules: + +- Inputs are a pushed full 40-character Git commit and a registered service id. Dirty worktrees, operator-uploaded source trees and local-only commits are not valid artifact sources. +- D601 prepares a commit-pinned source export under `/home/ubuntu/.unidesk/ci/user-service-artifacts//` using the existing GitHub SSH deploy identity and node-local provider-gateway WS egress proxy. Tekton consumes that export through a read-only hostPath. +- The image is tagged only with the source commit and pushed to the D601 registry as `127.0.0.1:5000/unidesk/:`. The producer must reject third-party registries and must not publish or consume a mutable `latest` tag. +- The image must carry `unidesk.ai/service-id`, `unidesk.ai/source-repo`, `unidesk.ai/source-commit` and `unidesk.ai/dockerfile` labels. +- The command output must include the image ref, tag, digest, source commit and service id. The digest ref is suitable as immutable input for later dev/prod deployment work. +- CI is an artifact producer only. It must not restart production services, call production `deploy apply`, mutate the production namespace, or change `deploy.json`. + +Publish a Decision Center artifact: + +```bash +bun scripts/cli.ts ci publish-user-service --service decision-center --commit --wait-ms 1200000 +``` + +This command creates the `unidesk-user-service-artifact-publish` Tekton PipelineRun and pushes `127.0.0.1:5000/unidesk/decision-center:`. + ## Dev Namespace E2E `ci run-dev-e2e` is the manual dev desired-state smoke flow. The single authoritative reference for its Git-controlled runner script, short launcher, result directory and no-CD boundary is `docs/reference/dev-ci-runner.md`. @@ -122,6 +143,14 @@ bun scripts/cli.ts ci publish-backend-core --commit --wait-ms 1200000 This command creates the `unidesk-backend-core-artifact-publish` Tekton PipelineRun. It is a CI producer action only: it may build and push `127.0.0.1:5000/unidesk/backend-core:`, but it must not recreate the master server container. Production deployment is triggered separately with `artifact-registry deploy-backend-core`. +Publish a user-service artifact: + +```bash +bun scripts/cli.ts ci publish-user-service --service decision-center --commit --wait-ms 1200000 +``` + +This command is a CI producer action only. For Decision Center, it builds and pushes `127.0.0.1:5000/unidesk/decision-center:` and reports the immutable digest without deploying production. + Run the dev namespace e2e harness manually: ```bash diff --git a/docs/reference/user-service-delivery.md b/docs/reference/user-service-delivery.md index d9cfc5bc..6a8b4f63 100644 --- a/docs/reference/user-service-delivery.md +++ b/docs/reference/user-service-delivery.md @@ -32,6 +32,8 @@ The default release flow for a user-service change is: - No user-service artifact may rely on a third-party registry as source of truth. - No production deploy may rebuild the source from a dirty worktree. - Commit-pinned image tags are the deployment truth; mutable `latest` tags are not. +- The standard CI artifact producer is `bun scripts/cli.ts ci publish-user-service --service --commit `. It accepts only a pushed Git commit and a registered service id, and reports image ref, tag, digest, source commit and service id. +- The CI artifact producer is not a deploy executor. It must not mutate the production namespace, restart production services, or update `deploy.json`. - Every production release must finish with a manual acceptance step after the automated checks pass. ## Frontend Pairing @@ -42,10 +44,14 @@ Many user services are surfaced through the UniDesk frontend rather than through Decision Center is the canonical example of a user service that doubles as a product workflow for requirements, decisions, and daily work diaries. +- The minimal standard artifact command is `bun scripts/cli.ts ci publish-user-service --service decision-center --commit --wait-ms 1200000`. +- The expected artifact is `127.0.0.1:5000/unidesk/decision-center:` plus its registry digest from the CI output. - Requirements and follow-up items should be represented with structured records such as `goal`, `decision`, `blocker`, `debt`, and `experiment`, with linked evidence and goal references. - The service should act as a demand-management surface for external goals that need to be decomposed into internal tasks, blockers, and decisions. - The work-diary surface should support creating today's diary entry automatically from the real current date. - Historical diary entries should be editable by date, with PostgreSQL remaining the source of truth. - The resulting product surface should stay structured and reviewable, not collapse into an untracked chat log. +- Before any production deploy, the dev gate must pass first with the focused Decision Center E2E set covering `microservice:decision-center-record-crud`, `microservice:decision-center-diary-lifecycle`, `frontend:decision-center-visible`, `frontend:decision-center-demand-management-visible`, and `frontend:decision-center-diary-visible`; this is an automated user-service validation gate, not a production deploy. +- Production acceptance is manual after the dev gate and must explicitly verify `health`, `records`, `diary editor`, the paired frontend page, no public business ports, and live commit / artifact information. Detailed service-level API and UI contracts remain in `docs/reference/microservices.md`. diff --git a/scripts/src/ci.ts b/scripts/src/ci.ts index ca7ddd14..b90cc421 100644 --- a/scripts/src/ci.ts +++ b/scripts/src/ci.ts @@ -1,7 +1,8 @@ import { randomUUID } from "node:crypto"; import { existsSync, readFileSync } from "node:fs"; +import { posix as posixPath } from "node:path"; import { runCommand } from "./command"; -import { type UniDeskConfig, repoRoot, rootPath } from "./config"; +import { type UniDeskConfig, type UniDeskMicroserviceConfig, repoRoot, rootPath } from "./config"; import { ensureGithubSshIdentityForProvider, gitSshHttpConnectProxySource } from "./deploy-ssh-identity"; import { startJob } from "./jobs"; import { coreInternalFetch } from "./microservices"; @@ -42,6 +43,16 @@ interface CiPublishBackendCoreOptions { sourceHostPath: string; } +interface CiPublishUserServiceArtifactOptions { + repoUrl: string; + commit: string; + waitMs: number; + sourceHostPath: string; + serviceId: string; + dockerfile: string; + dryRun: boolean; +} + interface CiDevE2EOptions { repoUrl: string; desiredRef: string; @@ -92,6 +103,19 @@ interface DeployDevManifestSummary { services: Array<{ id: string; commitId: string; repo: string }>; } +interface ArtifactSummary { + serviceId: string; + sourceCommit: string; + sourceRepo: string; + dockerfile: string; + registry: string; + repository: string; + tag: string; + imageRef: string; + digest: string | null; + digestRef: string | null; +} + function stringOption(args: string[], name: string): string | null { const index = args.indexOf(name); if (index === -1) return null; @@ -120,6 +144,14 @@ function requireFullCommit(value: string | null, option = "--commit"): string { return commit; } +function requireServiceId(value: string | null): string { + const serviceId = value ?? ""; + if (!/^[a-z0-9]([-a-z0-9]{0,62}[a-z0-9])?$/u.test(serviceId)) { + throw new Error("ci publish-user-service requires --service "); + } + return serviceId; +} + function requireDesiredRef(value: string | null): string { const ref = value ?? "master"; if (!/^[A-Za-z0-9._/-]{1,160}$/u.test(ref) || ref.startsWith("-") || ref.includes("..")) { @@ -149,6 +181,30 @@ function repoSshUrl(repoUrl: string): string { return repoUrl; } +function requireRepoRelativePath(path: string, label: string): string { + if (path.length === 0 || path.startsWith("/") || path.includes("\0") || path.split("/").includes("..")) { + throw new Error(`${label} must be a repo-relative path`); + } + const normalized = posixPath.normalize(path); + if (normalized === "." || normalized.startsWith("../")) throw new Error(`${label} must be a repo-relative path`); + return normalized; +} + +function requireSupportedUserService(config: UniDeskConfig, serviceId: string): UniDeskMicroserviceConfig { + if (serviceId === "backend-core") { + throw new Error("backend-core uses ci publish-backend-core; publish-user-service is for registered user services"); + } + const service = config.microservices.find((item) => item.id === serviceId); + if (service === undefined) throw new Error(`unknown user service: ${serviceId}`); + if (service.providerId !== d601ProviderId || service.development.providerId !== d601ProviderId) { + throw new Error(`ci publish-user-service currently supports only D601 user services; ${serviceId} is assigned to ${service.providerId}`); + } + if (service.deployment.mode !== "k3sctl-managed") { + throw new Error(`ci publish-user-service currently supports k3sctl-managed D601 user services; ${serviceId} uses ${service.deployment.mode}`); + } + return service; +} + function chunks(value: string, size: number): string[] { const result: string[] = []; for (let index = 0; index < value.length; index += size) { @@ -542,10 +598,49 @@ spec: `; } +function userServiceArtifactPipelineRunManifest(options: CiPublishUserServiceArtifactOptions): string { + const safeSuffix = new Date().toISOString().replace(/[-:.TZ]/g, "").slice(0, 14).toLowerCase(); + return `apiVersion: tekton.dev/v1 +kind: PipelineRun +metadata: + generateName: user-service-artifact-${options.serviceId}-${safeSuffix}- + namespace: unidesk-ci + labels: + app.kubernetes.io/name: unidesk-user-service-artifact-publish + app.kubernetes.io/part-of: unidesk + unidesk.ai/service-id: ${JSON.stringify(options.serviceId)} + unidesk.ai/revision: ${JSON.stringify(options.commit)} +spec: + pipelineRef: + name: unidesk-user-service-artifact-publish + taskRunTemplate: + serviceAccountName: unidesk-ci-runner + params: + - name: repo-url + value: ${JSON.stringify(options.repoUrl)} + - name: revision + value: ${JSON.stringify(options.commit)} + - name: service-id + value: ${JSON.stringify(options.serviceId)} + - name: dockerfile + value: ${JSON.stringify(options.dockerfile)} + - name: source-host-path + value: ${JSON.stringify(options.sourceHostPath)} + workspaces: + - name: shared-workspace + persistentVolumeClaim: + claimName: unidesk-ci-cache +`; +} + function backendCoreArtifactSourceHostPath(commit: string): string { return `/home/ubuntu/.unidesk/ci/backend-core-artifacts/${commit}`; } +function userServiceArtifactSourceHostPath(serviceId: string, commit: string): string { + return `/home/ubuntu/.unidesk/ci/user-service-artifacts/${serviceId}/${commit}`; +} + async function prepareBackendCoreArtifactSource(config: UniDeskConfig, options: CiPublishBackendCoreOptions): Promise> { const sshIdentity = await ensureGithubSshIdentityForProvider(config, d601ProviderId); if (!sshIdentity.ok) throw new Error(sshIdentity.detail); @@ -613,6 +708,82 @@ async function prepareBackendCoreArtifactSource(config: UniDeskConfig, options: }; } +async function prepareUserServiceArtifactSource(config: UniDeskConfig, options: CiPublishUserServiceArtifactOptions): Promise> { + const sshIdentity = await ensureGithubSshIdentityForProvider(config, d601ProviderId); + if (!sshIdentity.ok) throw new Error(sshIdentity.detail); + const proxyPython = gitSshHttpConnectProxySource(); + const sourceRoot = `/home/ubuntu/.unidesk/ci/user-service-artifacts/${options.serviceId}`; + const sourceHostPath = options.sourceHostPath; + const repoCache = "/home/ubuntu/.unidesk/ci/git/unidesk.git"; + const repoFetchUrl = repoSshUrl(options.repoUrl); + const dockerfileDir = posixPath.dirname(options.dockerfile); + const script = [ + "set -euo pipefail", + `service_id=${shellQuote(options.serviceId)}`, + `commit=${shellQuote(options.commit)}`, + `repo_url=${shellQuote(options.repoUrl)}`, + `repo_fetch_url=${shellQuote(repoFetchUrl)}`, + `dockerfile=${shellQuote(options.dockerfile)}`, + `dockerfile_dir=${shellQuote(dockerfileDir)}`, + `source_root=${shellQuote(sourceRoot)}`, + `source_dir=${shellQuote(sourceHostPath)}`, + `repo_cache=${shellQuote(repoCache)}`, + `proxy_url=${shellQuote(providerGatewayWsEgressProxyUrl)}`, + "mkdir -p \"$(dirname \"$repo_cache\")\" \"$source_root\"", + "export HTTP_PROXY=\"$proxy_url\" HTTPS_PROXY=\"$proxy_url\" ALL_PROXY=\"$proxy_url\"", + "export NO_PROXY=\"localhost,127.0.0.1,::1,host.docker.internal,.svc,.cluster.local,kubernetes.default.svc\"", + "curl -fsSI --max-time 20 -x \"$proxy_url\" https://github.com >/dev/null", + "git_ssh_proxy=/tmp/unidesk-git-ssh-http-connect.py", + "cat > \"$git_ssh_proxy\" <<'UNIDESK_GIT_SSH_PROXY'", + proxyPython, + "UNIDESK_GIT_SSH_PROXY", + "chmod 700 \"$git_ssh_proxy\"", + "export UNIDESK_GIT_SSH_HTTP_PROXY=\"$proxy_url\"", + "export GIT_SSH_COMMAND=\"ssh -o BatchMode=yes -o IdentitiesOnly=yes -o StrictHostKeyChecking=yes -o UserKnownHostsFile=$HOME/.ssh/known_hosts -i $HOME/.ssh/id_ed25519 -o 'ProxyCommand=$git_ssh_proxy %h %p'\"", + "echo user_service_artifact_source_proxy=provider-gateway-ws-egress:$proxy_url", + "echo user_service_artifact_repo_fetch_url=$repo_fetch_url", + "echo user_service_artifact_service_id=$service_id", + "if [ ! -d \"$repo_cache\" ]; then git clone --mirror \"$repo_fetch_url\" \"$repo_cache\"; fi", + "git -C \"$repo_cache\" remote set-url origin \"$repo_fetch_url\"", + "git -C \"$repo_cache\" fetch --no-tags origin \"$commit\" || git -C \"$repo_cache\" fetch --no-tags origin '+refs/heads/*:refs/remotes/origin/*'", + "resolved=$(git -C \"$repo_cache\" rev-parse --verify \"$commit^{commit}\")", + "test \"$resolved\" = \"$commit\" || { echo \"user_service_artifact_resolved_commit_mismatch=$resolved expected=$commit\" >&2; exit 1; }", + "git -C \"$repo_cache\" cat-file -e \"$commit:$dockerfile\"", + "git -C \"$repo_cache\" cat-file -e \"$commit:$dockerfile_dir/src\"", + "tmp_dir=\"$source_root/.tmp-$commit-$$\"", + "rm -rf \"$tmp_dir\"", + "mkdir -p \"$tmp_dir\"", + "git -C \"$repo_cache\" archive \"$commit\" | tar -x -C \"$tmp_dir\"", + "printf '%s\\n' \"$commit\" > \"$tmp_dir/.unidesk-source-commit\"", + "printf '%s\\n' \"$repo_url\" > \"$tmp_dir/.unidesk-source-repo\"", + "printf '%s\\n' \"$service_id\" > \"$tmp_dir/.unidesk-service-id\"", + "printf '%s\\n' \"$dockerfile\" > \"$tmp_dir/.unidesk-dockerfile\"", + "rm -rf \"$source_dir\"", + "mv \"$tmp_dir\" \"$source_dir\"", + "test -f \"$source_dir/$dockerfile\"", + "test -d \"$source_dir/$dockerfile_dir/src\"", + "echo user_service_artifact_source_host_path=$source_dir", + ].join("\n"); + const result = await runRemoteBackground(`prepare-${options.serviceId}-source`, script, 300_000); + if (!result.ok) throw new Error(`failed to prepare ${options.serviceId} source on D601: ${result.stderr || result.stdout || JSON.stringify(result.raw)}`); + return { + ok: true, + mode: "d601-host-github-ssh-export", + providerId: d601ProviderId, + repoUrl: options.repoUrl, + repoFetchUrl, + commit: options.commit, + serviceId: options.serviceId, + dockerfile: options.dockerfile, + sourceHostPath, + identity: { + fingerprint: sshIdentity.fingerprint, + seededFromLocal: sshIdentity.seededFromLocal, + }, + stdoutTail: result.stdout.slice(-4000), + }; +} + async function remoteCreatePipelineRun(manifest: string): Promise { const encoded = Buffer.from(manifest, "utf8").toString("base64"); const token = randomUUID().replace(/-/gu, "").slice(0, 12); @@ -689,6 +860,46 @@ function pipelineRunWaitSucceeded(wait: DispatchResult | null, condition: Pipeli return wait.ok || wait.exitCode === 0 || wait.stdout.includes("condition=True\tSucceeded\t"); } +function parseArtifactSummaryFromOutput(output: string, serviceId: string, commit: string, repoUrl: string, dockerfile: string): ArtifactSummary { + const fields = new Map(); + for (const line of output.split(/\r?\n/u)) { + const match = /^(user_service_artifact_[a-z_]+|backend_core_artifact_[a-z_]+)=(.*)$/u.exec(line.trim()); + if (match !== null) fields.set(match[1], match[2]); + } + const digest = fields.get("user_service_artifact_digest") ?? fields.get("backend_core_artifact_digest") ?? null; + const imageRef = fields.get("user_service_artifact_image") ?? fields.get("backend_core_artifact_image") ?? `127.0.0.1:5000/unidesk/${serviceId}:${commit}`; + const repository = fields.get("user_service_artifact_repository") ?? `127.0.0.1:5000/unidesk/${serviceId}`; + const digestRef = fields.get("user_service_artifact_digest_ref") ?? fields.get("backend_core_artifact_digest_ref") ?? (digest === null || digest.length === 0 ? null : `${repository}@${digest}`); + return { + serviceId: fields.get("user_service_artifact_service_id") ?? serviceId, + sourceCommit: fields.get("user_service_artifact_source_commit") ?? fields.get("backend_core_artifact_source_commit") ?? commit, + sourceRepo: fields.get("user_service_artifact_source_repo") ?? repoUrl, + dockerfile: fields.get("user_service_artifact_dockerfile") ?? dockerfile, + registry: fields.get("user_service_artifact_registry") ?? "127.0.0.1:5000", + repository, + tag: fields.get("user_service_artifact_tag") ?? commit, + imageRef, + digest, + digestRef, + }; +} + +function assertArtifactSummaryComplete(artifact: ArtifactSummary, pipelineRun: string): void { + if (artifact.serviceId.length === 0 || artifact.sourceCommit.length !== 40 || artifact.imageRef.length === 0 || artifact.tag.length === 0 || artifact.digest === null || artifact.digest.length === 0 || artifact.digestRef === null || artifact.digestRef.length === 0) { + throw new Error(`artifact summary for ${pipelineRun} is incomplete`); + } +} + +async function readPipelineRunLogText(name: string): Promise { + const result = await runRemoteKubectlRaw([ + "set -euo pipefail", + `kubectl get pipelinerun/${shellQuote(name)} -n unidesk-ci -o wide`, + `kubectl get taskrun -n unidesk-ci -l tekton.dev/pipelineRun=${shellQuote(name)} -o wide`, + `for pod in $(kubectl get pods -n unidesk-ci -l tekton.dev/pipelineRun=${shellQuote(name)} -o name); do echo "===== $pod"; kubectl logs -n unidesk-ci "$pod" --all-containers=true --tail=240 || true; done`, + ].join("\n"), 60_000, 45_000); + return `${result.stdout}\n${result.stderr}`.trim(); +} + async function run(options: CiOptions): Promise> { const name = await remoteCreatePipelineRun(pipelineRunManifest(options)); const wait = await waitForPipelineRun(name, options.waitMs); @@ -717,11 +928,27 @@ async function run(options: CiOptions): Promise> { } async function publishBackendCoreArtifact(config: UniDeskConfig, options: CiPublishBackendCoreOptions): Promise> { + const plannedArtifact: ArtifactSummary = { + serviceId: "backend-core", + sourceCommit: options.commit, + sourceRepo: options.repoUrl, + dockerfile: "src/components/backend-core/Dockerfile", + registry: "127.0.0.1:5000", + repository: "127.0.0.1:5000/unidesk/backend-core", + tag: options.commit, + imageRef: `127.0.0.1:5000/unidesk/backend-core:${options.commit}`, + digest: null, + digestRef: null, + }; const source = await prepareBackendCoreArtifactSource(config, options); const name = await remoteCreatePipelineRun(backendCoreArtifactPipelineRunManifest(options)); const wait = await waitForPipelineRun(name, options.waitMs); const condition = wait === null ? null : await readPipelineRunCondition(name); const waitSucceeded = pipelineRunWaitSucceeded(wait, condition); + const artifact = waitSucceeded && wait !== null + ? parseArtifactSummaryFromOutput(await readPipelineRunLogText(name), "backend-core", options.commit, options.repoUrl, "src/components/backend-core/Dockerfile") + : plannedArtifact; + if (waitSucceeded && wait !== null) assertArtifactSummaryComplete(artifact, name); return { ok: waitSucceeded, pipelineRun: name, @@ -729,7 +956,8 @@ async function publishBackendCoreArtifact(config: UniDeskConfig, options: CiPubl repoUrl: options.repoUrl, commit: options.commit, source, - artifact: `127.0.0.1:5000/unidesk/backend-core:${options.commit}`, + artifact: artifact.imageRef, + artifactSummary: artifact, boundary: "CI publishes the image to D601 registry; CD must pull it and must not build backend-core", wait: wait === null ? null : { ok: waitSucceeded, @@ -747,6 +975,84 @@ async function publishBackendCoreArtifact(config: UniDeskConfig, options: CiPubl }; } +async function publishUserServiceArtifact(config: UniDeskConfig, options: CiPublishUserServiceArtifactOptions): Promise> { + const plannedArtifact: ArtifactSummary = { + serviceId: options.serviceId, + sourceCommit: options.commit, + sourceRepo: options.repoUrl, + dockerfile: options.dockerfile, + registry: "127.0.0.1:5000", + repository: `127.0.0.1:5000/unidesk/${options.serviceId}`, + tag: options.commit, + imageRef: `127.0.0.1:5000/unidesk/${options.serviceId}:${options.commit}`, + digest: null, + digestRef: null, + } satisfies ArtifactSummary; + if (options.dryRun) { + return { + ok: true, + mode: "dry-run", + pipeline: "unidesk-user-service-artifact-publish", + namespace: "unidesk-ci", + repoUrl: options.repoUrl, + commit: options.commit, + serviceId: options.serviceId, + sourceHostPath: options.sourceHostPath, + source: { + ok: true, + mode: "planned-only", + providerId: d601ProviderId, + repoUrl: options.repoUrl, + repoFetchUrl: repoSshUrl(options.repoUrl), + commit: options.commit, + serviceId: options.serviceId, + dockerfile: options.dockerfile, + sourceHostPath: options.sourceHostPath, + }, + artifact: plannedArtifact.imageRef, + artifactSummary: plannedArtifact, + boundary: "dry-run only; no D601 source export, no Tekton submission, no production mutation", + next: [ + `bun scripts/cli.ts ci publish-user-service --service ${options.serviceId} --commit ${options.commit} --wait-ms 1200000`, + ], + }; + } + const source = await prepareUserServiceArtifactSource(config, options); + const name = await remoteCreatePipelineRun(userServiceArtifactPipelineRunManifest(options)); + const wait = await waitForPipelineRun(name, options.waitMs); + const condition = wait === null ? null : await readPipelineRunCondition(name); + const waitSucceeded = pipelineRunWaitSucceeded(wait, condition); + const artifact = waitSucceeded && wait !== null + ? parseArtifactSummaryFromOutput(await readPipelineRunLogText(name), options.serviceId, options.commit, options.repoUrl, options.dockerfile) + : plannedArtifact; + if (waitSucceeded && wait !== null) assertArtifactSummaryComplete(artifact, name); + return { + ok: waitSucceeded, + pipelineRun: name, + namespace: "unidesk-ci", + repoUrl: options.repoUrl, + commit: options.commit, + serviceId: options.serviceId, + source, + artifact: artifact.imageRef, + artifactSummary: artifact, + boundary: "CI publishes the user-service image to the D601 registry only; it must not deploy production or mutate the production namespace", + wait: wait === null ? null : { + ok: waitSucceeded, + dispatchOk: wait.ok, + dispatchStatus: wait.status, + dispatchExitCode: wait.exitCode, + stdoutTail: wait.stdout.slice(-6000), + stderrTail: wait.stderr.slice(-6000), + }, + condition, + next: [ + `bun scripts/cli.ts ci logs ${name}`, + "use artifactSummary.imageRef or artifactSummary.digestRef as later dev/prod deployment input", + ], + }; +} + async function runRemoteDevE2ELauncher(options: CiDevE2EOptions): Promise { const scriptTimeoutMs = Math.max(options.scriptTimeoutMs, options.waitMs, 60_000); const remoteTimeoutMs = 45_000; @@ -995,12 +1301,13 @@ async function logs(name: string): Promise> { export function ciHelp(): Record { return { - command: "ci install|status|run|publish-backend-core|run-dev-e2e|logs", - description: "Manage the D601 k3s Tekton CI gate. CI may publish backend-core image artifacts, but it intentionally does not deploy CD.", + command: "ci install|status|run|publish-backend-core|publish-user-service|run-dev-e2e|logs", + description: "Manage the D601 k3s Tekton CI gate. CI may publish commit-pinned image artifacts, but it intentionally does not deploy CD.", examples: [ "bun scripts/cli.ts ci install", "bun scripts/cli.ts ci run --revision ", "bun scripts/cli.ts ci publish-backend-core --commit ", + "bun scripts/cli.ts ci publish-user-service --service decision-center --commit ", "bun scripts/cli.ts ci run-dev-e2e --wait-ms 600000", "bun scripts/cli.ts ci logs ", ], @@ -1018,6 +1325,14 @@ export function ciHelp(): Record { registry: "127.0.0.1:5000/unidesk/backend-core:", cdCommand: "bun scripts/cli.ts artifact-registry deploy-backend-core --commit ", }, + userServiceArtifact: { + producer: "D601 CI", + command: "bun scripts/cli.ts ci publish-user-service --service --commit ", + initiallySupportedServices: ["decision-center"], + registry: "127.0.0.1:5000/unidesk/:", + outputFields: ["imageRef", "tag", "digest", "sourceCommit", "serviceId"], + boundary: "artifact producer only; no prod deploy and no production namespace mutation", + }, runDevE2E: { defaultTriggerMode: "commit-pinned-ssh-launcher", desiredState: "origin/master:deploy.json#environments.dev", @@ -1050,6 +1365,27 @@ export async function runCiCommand(config: UniDeskConfig, args: string[]): Promi const waitMs = numberOption(args, "--wait-ms", 0); return publishBackendCoreArtifact(config, { repoUrl, commit, waitMs, sourceHostPath: backendCoreArtifactSourceHostPath(commit) }); } + if (action === "publish-user-service") { + const serviceId = requireServiceId(stringOption(args, "--service") ?? stringOption(args, "--service-id")); + const service = requireSupportedUserService(config, serviceId); + const repoUrl = stringOption(args, "--repo") ?? stringOption(args, "--repo-url") ?? service.repository.url; + const commit = requireFullCommit(stringOption(args, "--commit") ?? stringOption(args, "--revision")); + const waitMs = numberOption(args, "--wait-ms", 0); + const dryRun = boolFlag(args, "--dry-run"); + const dockerfile = requireRepoRelativePath(service.repository.dockerfile, `microservices.${serviceId}.repository.dockerfile`); + if (serviceId !== "decision-center") { + throw new Error("ci publish-user-service currently allows only decision-center until each user-service Dockerfile contract is reviewed"); + } + return publishUserServiceArtifact(config, { + repoUrl, + commit, + waitMs, + serviceId, + dockerfile, + sourceHostPath: userServiceArtifactSourceHostPath(serviceId, commit), + dryRun, + }); + } if (action === "run-dev-e2e") { const repoUrl = stringOption(args, "--repo") ?? stringOption(args, "--repo-url") ?? "https://github.com/pikasTech/unidesk"; const desiredRef = requireDesiredRef(stringOption(args, "--desired-ref") ?? stringOption(args, "--deploy-branch")); @@ -1071,7 +1407,7 @@ export async function runCiCommand(config: UniDeskConfig, args: string[]): Promi }); } if (action === "logs") return logs(nameArg ?? ""); - throw new Error("ci command must be one of: install, status, run, publish-backend-core, run-dev-e2e, logs"); + throw new Error("ci command must be one of: install, status, run, publish-backend-core, publish-user-service, run-dev-e2e, logs"); } export function startCiInstallJob(): Record { diff --git a/src/components/microservices/k3sctl-adapter/k3s/ci/unidesk-ci.pipeline.yaml b/src/components/microservices/k3sctl-adapter/k3s/ci/unidesk-ci.pipeline.yaml index 1fcf0155..c7495ccf 100644 --- a/src/components/microservices/k3sctl-adapter/k3s/ci/unidesk-ci.pipeline.yaml +++ b/src/components/microservices/k3sctl-adapter/k3s/ci/unidesk-ci.pipeline.yaml @@ -400,8 +400,10 @@ spec: set -euo pipefail commit="$(cat "$(workspaces.source.path)/backend-core-artifact-commit.txt")" registry="$(params.registry)" + test "$registry" = "127.0.0.1:5000" || { echo "backend_core_artifact_registry_must_be_d601_loopback=$registry" >&2; exit 2; } local_image="unidesk/backend-core:$commit" registry_image="$registry/unidesk/backend-core:$commit" + repository="$registry/unidesk/backend-core" command -v docker docker version >/dev/null docker run --rm --network host rancher/mirrored-library-busybox:1.36.1 wget -q -O- "http://$registry/v2/" >/dev/null @@ -422,8 +424,11 @@ spec: actual_commit="$(docker image inspect "$registry_image" --format '{{ index .Config.Labels "unidesk.ai/source-commit" }}')" test "$actual_commit" = "$commit" docker push "$registry_image" + docker pull "$registry_image" >/dev/null repo_digests="$(docker image inspect "$registry_image" --format '{{json .RepoDigests}}')" - printf 'backend_core_artifact_image=%s\nbackend_core_artifact_repo_digests=%s\n' "$registry_image" "$repo_digests" + digest="$(docker image inspect "$registry_image" --format '{{range .RepoDigests}}{{println .}}{{end}}' | awk -F@ -v repo="$repository" '$1 == repo { print $2; exit }')" + test -n "$digest" + printf 'backend_core_artifact_service_id=backend-core\nbackend_core_artifact_image=%s\nbackend_core_artifact_repository=%s\nbackend_core_artifact_tag=%s\nbackend_core_artifact_digest=%s\nbackend_core_artifact_digest_ref=%s@%s\nbackend_core_artifact_source_commit=%s\nbackend_core_artifact_source_repo=%s\nbackend_core_artifact_dockerfile=src/components/backend-core/Dockerfile\nbackend_core_artifact_repo_digests=%s\n' "$registry_image" "$repository" "$commit" "$digest" "$repository" "$digest" "$commit" "$(params.repo-url)" "$repo_digests" --- apiVersion: tekton.dev/v1 kind: Pipeline @@ -472,6 +477,211 @@ spec: --- apiVersion: tekton.dev/v1 kind: Task +metadata: + name: unidesk-user-service-artifact-publish + namespace: unidesk-ci + labels: + app.kubernetes.io/name: unidesk-ci + app.kubernetes.io/component: user-service-artifact +spec: + params: + - name: repo-url + type: string + - name: revision + type: string + - name: service-id + type: string + - name: dockerfile + type: string + - name: app-image + type: string + default: unidesk-code-queue:dev + - name: registry + type: string + default: 127.0.0.1:5000 + - name: source-host-path + type: string + workspaces: + - name: source + volumes: + - name: docker-sock + hostPath: + path: /var/run/docker.sock + type: Socket + - name: prepared-source + hostPath: + path: /home/ubuntu/.unidesk/ci/user-service-artifacts + type: Directory + steps: + - name: prepare-source + image: alpine/git:2.45.2 + volumeMounts: + - name: prepared-source + mountPath: /prepared-source + readOnly: true + script: | + #!/bin/sh + set -eu + service_id="$(params.service-id)" + dockerfile="$(params.dockerfile)" + source_host_path="$(params.source-host-path)" + case "$(params.revision)" in + [0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f]) ;; + *) echo "user_service_artifact_revision_must_be_full_sha=$(params.revision)" >&2; exit 2 ;; + esac + printf '%s\n' "$service_id" | grep -Eq '^[a-z0-9]([-a-z0-9]{0,62}[a-z0-9])?$' || { echo "user_service_artifact_service_id_invalid=$service_id" >&2; exit 2; } + case "$dockerfile" in + /*|*..*|""|*latest*) echo "user_service_artifact_dockerfile_invalid=$dockerfile" >&2; exit 2 ;; + esac + expected_prefix="/home/ubuntu/.unidesk/ci/user-service-artifacts/$service_id/" + case "$source_host_path" in + "$expected_prefix"*) ;; + *) echo "user_service_artifact_source_host_path_invalid=$source_host_path" >&2; exit 2 ;; + esac + mkdir -p "$(workspaces.source.path)/user-service-artifact-repo" + find "$(workspaces.source.path)/user-service-artifact-repo" -mindepth 1 -maxdepth 1 -exec rm -rf {} + + source_name="$(basename "$source_host_path")" + source_dir="/prepared-source/$service_id/$source_name" + test -f "$source_dir/.unidesk-source-commit" + test -f "$source_dir/.unidesk-service-id" + test -f "$source_dir/.unidesk-dockerfile" + prepared_commit="$(cat "$source_dir/.unidesk-source-commit")" + prepared_service_id="$(cat "$source_dir/.unidesk-service-id")" + prepared_dockerfile="$(cat "$source_dir/.unidesk-dockerfile")" + test "$prepared_commit" = "$(params.revision)" + test "$prepared_service_id" = "$service_id" + test "$prepared_dockerfile" = "$dockerfile" + cp -a "$source_dir/." "$(workspaces.source.path)/user-service-artifact-repo/" + cd "$(workspaces.source.path)/user-service-artifact-repo" + test -f "$dockerfile" + test -d "$(dirname "$dockerfile")/src" + printf '%s\n' "$prepared_commit" | tee "$(workspaces.source.path)/user-service-artifact-commit.txt" + printf '%s\n' "$service_id" | tee "$(workspaces.source.path)/user-service-artifact-service-id.txt" + printf '%s\n' "$dockerfile" | tee "$(workspaces.source.path)/user-service-artifact-dockerfile.txt" + - name: build-and-push + image: "$(params.app-image)" + imagePullPolicy: Never + workingDir: "$(workspaces.source.path)/user-service-artifact-repo" + env: + - name: DOCKER_HOST + value: unix:///var/run/docker.sock + - name: HTTP_PROXY + value: "http://d601-provider-egress-proxy.unidesk.svc.cluster.local:18789" + - name: HTTPS_PROXY + value: "http://d601-provider-egress-proxy.unidesk.svc.cluster.local:18789" + - name: ALL_PROXY + value: "http://d601-provider-egress-proxy.unidesk.svc.cluster.local:18789" + - name: NO_PROXY + value: "localhost,127.0.0.1,::1,host.docker.internal,d601-provider-egress-proxy,d601-provider-egress-proxy.unidesk,d601-provider-egress-proxy.unidesk.svc,d601-provider-egress-proxy.unidesk.svc.cluster.local" + - name: http_proxy + value: "http://d601-provider-egress-proxy.unidesk.svc.cluster.local:18789" + - name: https_proxy + value: "http://d601-provider-egress-proxy.unidesk.svc.cluster.local:18789" + - name: all_proxy + value: "http://d601-provider-egress-proxy.unidesk.svc.cluster.local:18789" + - name: no_proxy + value: "localhost,127.0.0.1,::1,host.docker.internal,d601-provider-egress-proxy,d601-provider-egress-proxy.unidesk,d601-provider-egress-proxy.unidesk.svc,d601-provider-egress-proxy.unidesk.svc.cluster.local" + volumeMounts: + - name: docker-sock + mountPath: /var/run/docker.sock + script: | + #!/usr/bin/env bash + set -euo pipefail + commit="$(cat "$(workspaces.source.path)/user-service-artifact-commit.txt")" + service_id="$(cat "$(workspaces.source.path)/user-service-artifact-service-id.txt")" + dockerfile="$(cat "$(workspaces.source.path)/user-service-artifact-dockerfile.txt")" + registry="$(params.registry)" + test "$registry" = "127.0.0.1:5000" || { echo "user_service_artifact_registry_must_be_d601_loopback=$registry" >&2; exit 2; } + case "$commit" in + [0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f]) ;; + *) echo "user_service_artifact_commit_invalid=$commit" >&2; exit 2 ;; + esac + local_image="unidesk/$service_id:$commit" + repository="$registry/unidesk/$service_id" + registry_image="$repository:$commit" + command -v docker + docker version >/dev/null + docker run --rm --network host rancher/mirrored-library-busybox:1.36.1 wget -q -O- "http://$registry/v2/" >/dev/null + DOCKER_BUILDKIT=0 docker build \ + --network host \ + --build-arg HTTP_PROXY=http://127.0.0.1:18789 \ + --build-arg HTTPS_PROXY=http://127.0.0.1:18789 \ + --build-arg ALL_PROXY=http://127.0.0.1:18789 \ + --build-arg NO_PROXY=localhost,127.0.0.1,::1,host.docker.internal \ + --label "unidesk.ai/service-id=$service_id" \ + --label "unidesk.ai/source-repo=$(params.repo-url)" \ + --label "unidesk.ai/source-commit=$commit" \ + --label "unidesk.ai/dockerfile=$dockerfile" \ + -t "$local_image" \ + -t "$registry_image" \ + -f "$dockerfile" \ + . + actual_service="$(docker image inspect "$registry_image" --format '{{ index .Config.Labels "unidesk.ai/service-id" }}')" + actual_commit="$(docker image inspect "$registry_image" --format '{{ index .Config.Labels "unidesk.ai/source-commit" }}')" + test "$actual_service" = "$service_id" + test "$actual_commit" = "$commit" + docker push "$registry_image" + docker pull "$registry_image" >/dev/null + repo_digests="$(docker image inspect "$registry_image" --format '{{json .RepoDigests}}')" + digest="$(docker image inspect "$registry_image" --format '{{range .RepoDigests}}{{println .}}{{end}}' | awk -F@ -v repo="$repository" '$1 == repo { print $2; exit }')" + test -n "$digest" + printf 'user_service_artifact_service_id=%s\nuser_service_artifact_image=%s\nuser_service_artifact_repository=%s\nuser_service_artifact_tag=%s\nuser_service_artifact_digest=%s\nuser_service_artifact_digest_ref=%s@%s\nuser_service_artifact_source_commit=%s\nuser_service_artifact_source_repo=%s\nuser_service_artifact_dockerfile=%s\nuser_service_artifact_registry=%s\nuser_service_artifact_repo_digests=%s\n' "$service_id" "$registry_image" "$repository" "$commit" "$digest" "$repository" "$digest" "$commit" "$(params.repo-url)" "$dockerfile" "$registry" "$repo_digests" +--- +apiVersion: tekton.dev/v1 +kind: Pipeline +metadata: + name: unidesk-user-service-artifact-publish + namespace: unidesk-ci + labels: + app.kubernetes.io/name: unidesk-ci + app.kubernetes.io/component: user-service-artifact + app.kubernetes.io/part-of: unidesk +spec: + params: + - name: repo-url + type: string + default: https://github.com/pikasTech/unidesk + - name: revision + type: string + - name: service-id + type: string + - name: dockerfile + type: string + - name: app-image + type: string + default: unidesk-code-queue:dev + - name: registry + type: string + default: 127.0.0.1:5000 + - name: source-host-path + type: string + workspaces: + - name: shared-workspace + tasks: + - name: publish-user-service-artifact + taskRef: + name: unidesk-user-service-artifact-publish + params: + - name: repo-url + value: "$(params.repo-url)" + - name: revision + value: "$(params.revision)" + - name: service-id + value: "$(params.service-id)" + - name: dockerfile + value: "$(params.dockerfile)" + - name: app-image + value: "$(params.app-image)" + - name: registry + value: "$(params.registry)" + - name: source-host-path + value: "$(params.source-host-path)" + workspaces: + - name: source + workspace: shared-workspace +--- +apiVersion: tekton.dev/v1 +kind: Task metadata: name: unidesk-code-queue-read-perf namespace: unidesk-ci