fix(ci): reuse provider egress for backend-core artifacts

This commit is contained in:
Codex
2026-05-19 01:02:17 +00:00
parent 29dab8fab2
commit 49cf1e57bb
7 changed files with 408 additions and 256 deletions
+3 -3
View File
@@ -60,7 +60,7 @@ bun scripts/cli.ts artifact-registry deploy-backend-core --commit <full-sha>
真实 `install` 必须是幂等动作:创建远端目录,写入 CLI 渲染出的 config/compose/unit,执行 `systemctl daemon-reload`,启用并启动 `unidesk-artifact-registry.service`,然后运行与 `health` 相同的验收检查。若远端文件 hash 与期望不一致,install 可以覆盖由本 CLI 管理的 unit/config/compose,但不得删除 registry storage。
`deploy-backend-core` 是 production backend-core 的 CD 入口。它必须先确认 D601 registry 中已经存在 `unidesk/backend-core:<commit>`,随后只执行短生命周期 relay、`docker pull`、retag、Compose `--no-build` recreate 和 live commit 验证;如果镜像不存在,应失败并要求先运行 CI artifact publication。
`deploy-backend-core` 是 production backend-core 的 CD 入口。它必须先通过 CNCF Distribution HTTP API 确认 D601 registry 中已经存在 `unidesk/backend-core:<commit>`,随后通过 provider-gateway Host SSH 流式执行 `docker save | gzip`,在 master server 侧 `docker load`、retag、Compose `--no-build` recreate 和 live commit 验证;如果镜像不存在,应失败并要求先运行 CI artifact publication。
`status``health` 通过:
@@ -100,7 +100,7 @@ docker compose -p unidesk-artifact-registry -f /home/ubuntu/.unidesk/artifact-re
2. D601 CI 从 pushed Git checkout 构建 `unidesk/backend-core:<commit>`
3. CI 将镜像 push 到 `127.0.0.1:5000/unidesk/backend-core:<commit>`,并记录 image ref 与 digest。
4. CD 在 master server 上确认目标 commit/tag/digest 存在。
5. master server 通过受控、短生命周期的 localhost relay 从 D601 registry 取 commit-pinned 镜像。
5. master server 通过 provider-gateway Host SSH 从 D601 registry 流式读取 commit-pinned 镜像 tar,不开放 registry 端口,也不使用第三方镜像托管
6. master server retag 为 Compose 使用的 backend-core 镜像名,并执行 `docker compose up -d --no-build --no-deps --force-recreate backend-core`
7. 部署后通过 image label、runtime env、health payload 验证 live commit。
@@ -108,7 +108,7 @@ docker compose -p unidesk-artifact-registry -f /home/ubuntu/.unidesk/artifact-re
- source commit 来自 pushed Git,不来自 dirty worktree。
- 镜像 tag 必须 commit-pinned,不能用 mutable latest 作为部署真相。
- relay 是临时控制动作,不开放长期公网 registry。
- provider-gateway SSH image stream 是临时控制动作,不开放长期公网 registry。
- CI 可以有较多依赖;CD 只做拉取、retag、recreate 和 live commit 验证。
- CD 不执行 `cargo build``docker build``docker compose build backend-core` 或任何等价的 Rust 构建动作。
- `server rebuild backend-core` 仍不得作为 master server Rust 编译路径。
+2 -1
View File
@@ -26,7 +26,7 @@ Each commit CI run performs:
`ci install` also prewarms the D601 k3s containerd runtime with the Tekton entrypoint/workingdir helper images, `oven/bun:1-debian`, `alpine/git:2.45.2` and `unidesk-code-queue:dev`. Missing images are pulled through the node-local provider-gateway WS egress proxy and then imported into native k3s containerd with digests preserved, so PipelineRun pods do not hang on external registry pulls. Sustained pull throughput below 1 MB/s is treated as a provider/main-server network or proxy degradation first, not as a Dockerfile or application failure.
Git clone and dependency downloads inside the repo check task use `d601-provider-egress-proxy.unidesk.svc.cluster.local:18789`; the NO_PROXY list keeps the in-cluster read service, D601 TCP egress gateway and any in-cluster CI Git mirror on the cluster network.
Git clone and dependency downloads inside the repo check task use `d601-provider-egress-proxy.unidesk.svc.cluster.local:18789`; the NO_PROXY list keeps the in-cluster read service and D601 TCP egress gateway on the cluster network.
Private repository source authentication is part of the CI contract and follows `docs/reference/devops-hygiene.md`. If the repo-check task fails at `git clone` because credentials are unavailable, treat it as a CI infrastructure/auth gap, not as an application test result.
@@ -53,6 +53,7 @@ backend-core production image creation belongs to D601 CI, not to master server
The CI artifact task must follow these rules:
- Input revision comes from pushed Git and is resolved to a full 40-character commit. A dirty worktree or unpushed local tree must never be used as the image source.
- Source fetch for this artifact uses the existing D601 GitHub SSH deploy identity and the node-local provider-gateway WS egress proxy at `http://127.0.0.1:18789`. D601 prepares a commit-pinned source export under `/home/ubuntu/.unidesk/ci/backend-core-artifacts/<commit>` before creating the PipelineRun; Tekton consumes that prepared source through a read-only hostPath and must not clone GitHub itself, mount GitHub credentials, use an in-cluster Git mirror, or accept an operator-uploaded source tree.
- The source checkout, Rust build and Docker build run on D601 CI infrastructure. The master server must not run `cargo build`, `docker compose build backend-core` or `server rebuild backend-core` as part of production backend-core deployment.
- The image is tagged with the source commit, for example `unidesk/backend-core:<commit>`, and pushed to the D601 artifact registry as `127.0.0.1:5000/unidesk/backend-core:<commit>`.
- The image must carry at least `unidesk.ai/service-id=backend-core`, `unidesk.ai/source-repo`, `unidesk.ai/source-commit` and `unidesk.ai/dockerfile=src/components/backend-core/Dockerfile`.
+2
View File
@@ -44,6 +44,8 @@ Acceptable source access implementations are:
- an in-cluster Git mirror managed by UniDesk; or
- the same commit-pinned host-fetch boundary used by `ci run-dev-e2e`, where D601 fetches the manifest commit and passes only verified inputs into Tekton.
For backend-core artifact publication, the required implementation is the host-fetch boundary: D601 uses the existing GitHub SSH deploy identity plus the node-local provider-gateway WS egress proxy, exports the requested commit to `/home/ubuntu/.unidesk/ci/backend-core-artifacts/<commit>`, and passes only that verified source directory into Tekton. This path must not be replaced by an in-cluster Git mirror, a third-party source mirror, an operator local checkout, or Tekton-mounted GitHub credentials.
If a CI repo-check task fails at `git clone` because credentials are unavailable, classify it as a CI infrastructure/auth gap, not as an application test failure.
## Verification Priority
+143 -209
View File
@@ -1,6 +1,4 @@
import { createHash } from "node:crypto";
import { spawn, type ChildProcessWithoutNullStreams } from "node:child_process";
import { createServer, type AddressInfo, type Server, type Socket } from "node:net";
import { readFileSync, writeFileSync } from "node:fs";
import { runCommand, type CommandResult } from "./command";
import { readConfig, repoRoot, rootPath } from "./config";
@@ -24,7 +22,6 @@ interface ArtifactRegistryOptions {
dryRun: boolean;
runNow: boolean;
commit: string | null;
localPullPort: number | null;
targetImage: string;
}
@@ -61,7 +58,6 @@ const defaultOptions: ArtifactRegistryOptions = {
dryRun: false,
runNow: false,
commit: null,
localPullPort: null,
targetImage: "unidesk-backend-core",
};
@@ -81,12 +77,6 @@ function positiveInt(value: string, option: string): number {
return parsed;
}
function optionalPort(value: string, option: string): number {
const parsed = positiveInt(value, option);
if (parsed > 65535) throw new Error(`${option} must be <= 65535`);
return parsed;
}
function absolutePath(value: string, option: string): string {
if (!value.startsWith("/")) throw new Error(`${option} must be an absolute path`);
if (value.includes("\n") || value.includes("\0")) throw new Error(`${option} must not contain control characters`);
@@ -131,9 +121,6 @@ function parseOptions(args: string[]): ArtifactRegistryOptions {
} else if (arg === "--commit") {
options.commit = commitValue(requireValue(args, index, arg), arg);
index += 1;
} else if (arg === "--local-pull-port") {
options.localPullPort = optionalPort(requireValue(args, index, arg), arg);
index += 1;
} else if (arg === "--target-image") {
options.targetImage = requireValue(args, index, arg);
index += 1;
@@ -154,6 +141,18 @@ function shellQuote(value: string): string {
return `'${value.replace(/'/g, `'\\''`)}'`;
}
function rootExecPrelude(): string {
return [
"root_exec() {",
" if [ \"$(id -u)\" = \"0\" ]; then \"$@\"; return; fi",
" if sudo -n true >/dev/null 2>&1; then sudo -n \"$@\"; return; fi",
" if [ -x /mnt/c/Windows/System32/wsl.exe ]; then /mnt/c/Windows/System32/wsl.exe -u root -- \"$@\"; return; fi",
" echo 'artifact_registry_root_access=missing' >&2",
" return 1",
"}",
].join("\n");
}
function file(path: string, content: string, mode = "0644"): RenderedFile {
return { path, mode, sha256: sha256(content), content };
}
@@ -203,15 +202,16 @@ function systemdUnit(options: ArtifactRegistryOptions): string {
return `[Unit]
Description=UniDesk D601 Artifact Registry (CNCF Distribution)
Documentation=https://github.com/pikasTech/unidesk/blob/master/docs/reference/artifact-registry.md
Requires=docker.service
After=docker.service network-online.target
After=network-online.target
Wants=network-online.target
ConditionPathExists=/var/run/docker.sock
[Service]
Type=oneshot
RemainAfterExit=yes
WorkingDirectory=${options.baseDir}
ExecStartPre=/bin/mkdir -p ${options.baseDir} ${options.storageDir}
ExecStartPre=/bin/sh -lc 'docker info >/dev/null'
ExecStart=/bin/sh -lc 'docker compose -p ${options.composeProject} -f ${options.baseDir}/compose.yml up -d --remove-orphans'
ExecStop=/bin/sh -lc 'docker compose -p ${options.composeProject} -f ${options.baseDir}/compose.yml down'
TimeoutStartSec=120
@@ -490,17 +490,18 @@ function runReadonlyStatus(options: ArtifactRegistryOptions, healthMode: boolean
function remoteWriteFileCommand(item: RenderedFile): string {
const encoded = Buffer.from(item.content, "utf8").toString("base64");
const rootOwned = item.path.startsWith("/etc/");
return [
`target=${shellQuote(item.path)}`,
"tmp=$(mktemp /tmp/unidesk-artifact-registry.XXXXXX)",
"trap 'rm -f \"$tmp\"' EXIT",
`printf %s ${shellQuote(encoded)} | base64 -d > "$tmp"`,
"if [ \"$(id -u)\" = \"0\" ]; then",
`if [ ${rootOwned ? "1" : "0"} = 1 ]; then`,
" root_exec mkdir -p \"$(dirname \"$target\")\"",
` root_exec install -m ${shellQuote(item.mode)} "$tmp" "$target"`,
"else",
" mkdir -p \"$(dirname \"$target\")\"",
` install -m ${shellQuote(item.mode)} "$tmp" "$target"`,
"else",
" sudo mkdir -p \"$(dirname \"$target\")\"",
` sudo install -m ${shellQuote(item.mode)} "$tmp" "$target"`,
"fi",
`echo ${shellQuote(`artifact_registry_file_written path=${item.path} sha256=${item.sha256}`)}`,
].join("\n");
@@ -513,10 +514,11 @@ function install(options: ArtifactRegistryOptions): Record<string, unknown> {
"command -v docker >/dev/null",
"docker compose version >/dev/null",
"command -v systemctl >/dev/null",
rootExecPrelude(),
`mkdir -p ${shellQuote(bundle.paths.baseDir)} ${shellQuote(bundle.paths.storage)}`,
...bundle.files.map(remoteWriteFileCommand),
"if [ \"$(id -u)\" = \"0\" ]; then systemctl daemon-reload; else sudo systemctl daemon-reload; fi",
`if [ "$(id -u)" = "0" ]; then systemctl enable --now ${shellQuote(options.unitName)}; else sudo systemctl enable --now ${shellQuote(options.unitName)}; fi`,
"root_exec systemctl daemon-reload",
`root_exec systemctl enable --now ${shellQuote(options.unitName)}`,
"sleep 2",
`curl -fsS http://${options.host}:${options.port}/v2/ >/dev/null`,
].join("\n");
@@ -559,91 +561,6 @@ function installDryRun(options: ArtifactRegistryOptions): Record<string, unknown
};
}
async function startRegistryRelay(options: ArtifactRegistryOptions): Promise<{
server: Server;
port: number;
close: () => Promise<void>;
}> {
const children = new Set<ChildProcessWithoutNullStreams>();
const sockets = new Set<Socket>();
const remoteProxyScript = String.raw`
import select
import socket
import sys
target = socket.create_connection(("127.0.0.1", 5000), timeout=10)
target.setblocking(False)
sys.stdin.buffer.raw.fileno()
sys.stdout.buffer.raw.fileno()
stdin_fd = sys.stdin.fileno()
stdout_fd = sys.stdout.fileno()
target_fd = target.fileno()
while True:
readable, _, _ = select.select([stdin_fd, target_fd], [], [])
if stdin_fd in readable:
data = sys.stdin.buffer.read1(65536)
if not data:
try:
target.shutdown(socket.SHUT_WR)
except OSError:
pass
else:
target.sendall(data)
if target_fd in readable:
try:
data = target.recv(65536)
except BlockingIOError:
data = b""
if not data:
break
sys.stdout.buffer.write(data)
sys.stdout.buffer.flush()
`;
const server = createServer((socket) => {
sockets.add(socket);
socket.on("close", () => sockets.delete(socket));
const child = spawn(process.execPath, [
"scripts/cli.ts",
"ssh",
options.providerId,
"argv",
"python3",
"-u",
"-c",
remoteProxyScript,
], {
cwd: repoRoot,
stdio: ["pipe", "pipe", "pipe"],
});
children.add(child);
child.on("close", () => children.delete(child));
child.on("error", () => socket.destroy());
socket.pipe(child.stdin);
child.stdout.pipe(socket);
child.stderr.on("data", (chunk) => process.stderr.write(`[artifact-registry-relay] ${chunk.toString("utf8")}`));
socket.on("close", () => child.kill());
});
const listen = new Promise<number>((resolve, reject) => {
server.once("error", reject);
server.listen(options.localPullPort ?? 0, "127.0.0.1", () => {
server.off("error", reject);
const address = server.address() as AddressInfo;
resolve(address.port);
});
});
const port = await listen;
return {
server,
port,
close: async () => {
for (const socket of sockets) socket.destroy();
for (const child of children) child.kill();
await new Promise<void>((resolve) => server.close(() => resolve()));
},
};
}
function composeLockScript(innerScript: string): string {
const lockDir = rootPath(".state", "locks");
const lockPath = rootPath(".state", "locks", "server-compose.lock");
@@ -670,6 +587,31 @@ function upsertEnvFileValues(path: string, values: Record<string, string>): void
writeFileSync(path, `${lines.join("\n")}\n`, "utf8");
}
function pullBackendCoreArtifactFromD601(options: ArtifactRegistryOptions, sourceImage: string): CommandResult {
const remoteScript = [
"set -euo pipefail",
`image=${shellQuote(sourceImage)}`,
"docker pull -q \"$image\" >/dev/null",
"docker image inspect \"$image\" --format 'remote_source={{ index .Config.Labels \"unidesk.ai/source-commit\" }} remote_service={{ index .Config.Labels \"unidesk.ai/service-id\" }}' >&2",
"docker save \"$image\" | gzip -1",
].join("\n");
const sshCommand = [
process.execPath,
rootPath("scripts", "cli.ts"),
"ssh",
options.providerId,
"argv",
"bash",
"-lc",
remoteScript,
].map(shellQuote).join(" ");
const pipeline = [
"set -euo pipefail",
`${sshCommand} | gzip -dc | docker load`,
].join("\n");
return runCommand(["bash", "-lc", pipeline], repoRoot, { timeoutMs: Math.max(options.timeoutMs, 900_000) });
}
async function deployBackendCoreNow(options: ArtifactRegistryOptions): Promise<Record<string, unknown>> {
const commit = options.commit;
if (commit === null) throw new Error("artifact-registry deploy-backend-core requires --commit <full-sha>");
@@ -682,11 +624,13 @@ async function deployBackendCoreNow(options: ArtifactRegistryOptions): Promise<R
const commitImage = `${options.targetImage}:${commit}`;
const registryProbeScript = [
"set -euo pipefail",
`image=${shellQuote(`unidesk/backend-core:${commit}`)}`,
`registry_image=${shellQuote(sourceImage)}`,
"docker manifest inspect \"$registry_image\" >/tmp/unidesk-backend-core-manifest.json",
"manifest_digest=$(docker manifest inspect --verbose \"$registry_image\" 2>/dev/null | awk -F'\"' '/Digest/ {print $4; exit}' || true)",
"printf 'registry_image=%s\\nmanifest_digest=%s\\n' \"$registry_image\" \"$manifest_digest\"",
`manifest_url=${shellQuote(`http://127.0.0.1:${options.port}/v2/unidesk/backend-core/manifests/${commit}`)}`,
"headers=$(mktemp /tmp/unidesk-backend-core-manifest.XXXXXX.headers)",
"trap 'rm -f \"$headers\"' EXIT",
"curl -fsSI -H 'Accept: application/vnd.docker.distribution.manifest.v2+json' -D \"$headers\" -o /dev/null \"$manifest_url\"",
"manifest_digest=$(awk 'BEGIN{IGNORECASE=1} /^Docker-Content-Digest:/ {gsub(/\\r/, \"\", $2); print $2; exit}' \"$headers\")",
"printf 'registry_image=%s\\nmanifest_url=%s\\nmanifest_digest=%s\\n' \"$registry_image\" \"$manifest_url\" \"$manifest_digest\"",
].join("\n");
const registryProbe = runRemoteScript(options, registryProbeScript, Math.max(options.timeoutMs, 120_000));
if (registryProbe.exitCode !== 0 || registryProbe.timedOut) {
@@ -698,110 +642,100 @@ async function deployBackendCoreNow(options: ArtifactRegistryOptions): Promise<R
registryProbe: commandTail(registryProbe),
};
}
const relay = await startRegistryRelay(options);
const localSourceImage = `127.0.0.1:${relay.port}/unidesk/backend-core:${commit}`;
try {
const pull = runCommand(["docker", "pull", localSourceImage], repoRoot, { timeoutMs: Math.max(options.timeoutMs, 900_000) });
if (pull.exitCode !== 0 || pull.timedOut) {
return {
ok: false,
step: "docker-pull",
sourceImage,
localRelayImage: localSourceImage,
registryProbe: commandTail(registryProbe),
pull: commandTail(pull),
};
}
const inspectPulled = runCommand(["docker", "image", "inspect", localSourceImage, "--format", "{{json .Config.Labels}}"], repoRoot);
const labelCommit = runCommand(["docker", "image", "inspect", localSourceImage, "--format", "{{ index .Config.Labels \"unidesk.ai/source-commit\" }}"], repoRoot);
const labelService = runCommand(["docker", "image", "inspect", localSourceImage, "--format", "{{ index .Config.Labels \"unidesk.ai/service-id\" }}"], repoRoot);
if (labelCommit.stdout.trim() !== commit || labelService.stdout.trim() !== "backend-core") {
return {
ok: false,
step: "image-label-verify",
expected: { commit, serviceId: "backend-core" },
observed: { commit: labelCommit.stdout.trim(), serviceId: labelService.stdout.trim(), labels: inspectPulled.stdout.trim() },
registryProbe: commandTail(registryProbe),
};
}
const tag = runCommand(["docker", "tag", localSourceImage, composeImage], repoRoot);
if (tag.exitCode !== 0 || tag.timedOut) {
return { ok: false, step: "docker-tag", targetImage: composeImage, tag: commandTail(tag), registryProbe: commandTail(registryProbe) };
}
const tagCommit = runCommand(["docker", "tag", localSourceImage, commitImage], repoRoot);
if (tagCommit.exitCode !== 0 || tagCommit.timedOut) {
return { ok: false, step: "docker-tag-commit", targetImage: commitImage, tag: commandTail(tagCommit), registryProbe: commandTail(registryProbe) };
}
const config = readConfig();
const runtimeEnv = writeComposeEnv(config, false);
upsertEnvFileValues(runtimeEnv.envFile, {
UNIDESK_DEPLOY_REF: "deploy.json#environments.prod.services.backend-core",
UNIDESK_DEPLOY_SERVICE_ID: "backend-core",
UNIDESK_DEPLOY_REPO: "https://github.com/pikasTech/unidesk",
UNIDESK_DEPLOY_COMMIT: commit,
UNIDESK_DEPLOY_REQUESTED_COMMIT: commit,
});
const compose = resolveComposeCommand(config, runtimeEnv.envFile);
const upScript = [
"set -euo pipefail",
`echo ${shellQuote(`backend_core_artifact_cd source=${sourceImage} target=${composeImage}`)}`,
`${compose.map(shellQuote).join(" ")} up -d --no-build --no-deps --force-recreate backend-core`,
"ready=0",
"for attempt in $(seq 1 60); do",
" cid=$(docker ps -q --filter label=com.docker.compose.project=unidesk --filter label=com.docker.compose.service=backend-core --filter label=com.docker.compose.oneoff=False | head -1 || true)",
" if [ -n \"$cid\" ]; then",
" health=$(docker inspect -f '{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}' \"$cid\" 2>/dev/null || true)",
" echo \"backend_core_container_probe attempt=$attempt cid=$cid health=$health\"",
" if [ \"$health\" = \"healthy\" ] || [ \"$health\" = \"running\" ]; then ready=1; break; fi",
" else",
" echo \"backend_core_container_probe attempt=$attempt cid=missing\"",
" fi",
" sleep 1",
"done",
"test \"$ready\" = \"1\"",
"cid=$(docker ps -q --filter label=com.docker.compose.project=unidesk --filter label=com.docker.compose.service=backend-core --filter label=com.docker.compose.oneoff=False | head -1)",
"image_id=$(docker inspect -f '{{.Image}}' \"$cid\")",
"actual_commit=$(docker image inspect -f '{{ index .Config.Labels \"unidesk.ai/source-commit\" }}' \"$image_id\")",
`test "$actual_commit" = ${shellQuote(commit)}`,
"docker exec \"$cid\" bun -e \"fetch('http://127.0.0.1:8080/health').then(async r=>{const text=await r.text(); console.log(text); process.exit(r.ok?0:1)}).catch(e=>{console.error(e); process.exit(1)})\" >/tmp/unidesk-backend-core-health.json",
"cat /tmp/unidesk-backend-core-health.json",
].join("\n");
const deploy = runCommand(["bash", "-lc", composeLockScript(upScript)], repoRoot, { timeoutMs: Math.max(options.timeoutMs, 300_000) });
if (deploy.exitCode !== 0 || deploy.timedOut) {
return {
ok: false,
step: "compose-recreate",
sourceImage,
targetImage: composeImage,
registryProbe: commandTail(registryProbe),
deploy: commandTail(deploy),
};
}
const running = runCommand(["docker", "inspect", "unidesk-backend-core", "--format", "image={{.Config.Image}} imageId={{.Image}} status={{.State.Status}} health={{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}"], repoRoot);
const pull = pullBackendCoreArtifactFromD601(options, sourceImage);
if (pull.exitCode !== 0 || pull.timedOut) {
return {
ok: true,
commit,
providerId: options.providerId,
ok: false,
step: "docker-load",
sourceImage,
registryProbe: commandTail(registryProbe),
localRelayImage: localSourceImage,
targetImage: composeImage,
targetCommitImage: commitImage,
relay: {
bind: `127.0.0.1:${relay.port}`,
persistent: false,
},
pull: commandTail(pull),
deploy: commandTail(deploy),
running: running.stdout.trim(),
rollback: {
previousImageHint: "Use docker image ls and docker inspect logs for the previous backend-core image id; Compose named volumes were not changed.",
commandShape: `${compose.map(shellQuote).join(" ")} up -d --no-build --no-deps --force-recreate backend-core`,
},
};
} finally {
await relay.close();
}
const localLoadedImage = sourceImage;
const inspectPulled = runCommand(["docker", "image", "inspect", localLoadedImage, "--format", "{{json .Config.Labels}}"], repoRoot);
const labelCommit = runCommand(["docker", "image", "inspect", localLoadedImage, "--format", "{{ index .Config.Labels \"unidesk.ai/source-commit\" }}"], repoRoot);
const labelService = runCommand(["docker", "image", "inspect", localLoadedImage, "--format", "{{ index .Config.Labels \"unidesk.ai/service-id\" }}"], repoRoot);
if (labelCommit.stdout.trim() !== commit || labelService.stdout.trim() !== "backend-core") {
return {
ok: false,
step: "image-label-verify",
expected: { commit, serviceId: "backend-core" },
observed: { commit: labelCommit.stdout.trim(), serviceId: labelService.stdout.trim(), labels: inspectPulled.stdout.trim() },
registryProbe: commandTail(registryProbe),
};
}
const tag = runCommand(["docker", "tag", localLoadedImage, composeImage], repoRoot);
if (tag.exitCode !== 0 || tag.timedOut) {
return { ok: false, step: "docker-tag", targetImage: composeImage, tag: commandTail(tag), registryProbe: commandTail(registryProbe) };
}
const tagCommit = runCommand(["docker", "tag", localLoadedImage, commitImage], repoRoot);
if (tagCommit.exitCode !== 0 || tagCommit.timedOut) {
return { ok: false, step: "docker-tag-commit", targetImage: commitImage, tag: commandTail(tagCommit), registryProbe: commandTail(registryProbe) };
}
const config = readConfig();
const runtimeEnv = writeComposeEnv(config, false);
upsertEnvFileValues(runtimeEnv.envFile, {
UNIDESK_DEPLOY_REF: "deploy.json#environments.prod.services.backend-core",
UNIDESK_DEPLOY_SERVICE_ID: "backend-core",
UNIDESK_DEPLOY_REPO: "https://github.com/pikasTech/unidesk",
UNIDESK_DEPLOY_COMMIT: commit,
UNIDESK_DEPLOY_REQUESTED_COMMIT: commit,
});
const compose = resolveComposeCommand(config, runtimeEnv.envFile);
const upScript = [
"set -euo pipefail",
`echo ${shellQuote(`backend_core_artifact_cd source=${sourceImage} target=${composeImage}`)}`,
`${compose.map(shellQuote).join(" ")} up -d --no-build --no-deps --force-recreate backend-core`,
"ready=0",
"for attempt in $(seq 1 60); do",
" cid=$(docker ps -q --filter label=com.docker.compose.project=unidesk --filter label=com.docker.compose.service=backend-core --filter label=com.docker.compose.oneoff=False | head -1 || true)",
" if [ -n \"$cid\" ]; then",
" health=$(docker inspect -f '{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}' \"$cid\" 2>/dev/null || true)",
" echo \"backend_core_container_probe attempt=$attempt cid=$cid health=$health\"",
" if [ \"$health\" = \"healthy\" ] || [ \"$health\" = \"running\" ]; then ready=1; break; fi",
" else",
" echo \"backend_core_container_probe attempt=$attempt cid=missing\"",
" fi",
" sleep 1",
"done",
"test \"$ready\" = \"1\"",
"cid=$(docker ps -q --filter label=com.docker.compose.project=unidesk --filter label=com.docker.compose.service=backend-core --filter label=com.docker.compose.oneoff=False | head -1)",
"image_id=$(docker inspect -f '{{.Image}}' \"$cid\")",
"actual_commit=$(docker image inspect -f '{{ index .Config.Labels \"unidesk.ai/source-commit\" }}' \"$image_id\")",
`test "$actual_commit" = ${shellQuote(commit)}`,
"docker exec \"$cid\" bun -e \"fetch('http://127.0.0.1:8080/health').then(async r=>{const text=await r.text(); console.log(text); process.exit(r.ok?0:1)}).catch(e=>{console.error(e); process.exit(1)})\" >/tmp/unidesk-backend-core-health.json",
"cat /tmp/unidesk-backend-core-health.json",
].join("\n");
const deploy = runCommand(["bash", "-lc", composeLockScript(upScript)], repoRoot, { timeoutMs: Math.max(options.timeoutMs, 300_000) });
if (deploy.exitCode !== 0 || deploy.timedOut) {
return {
ok: false,
step: "compose-recreate",
sourceImage,
targetImage: composeImage,
registryProbe: commandTail(registryProbe),
deploy: commandTail(deploy),
};
}
const running = runCommand(["docker", "inspect", "unidesk-backend-core", "--format", "image={{.Config.Image}} imageId={{.Image}} status={{.State.Status}} health={{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}"], repoRoot);
return {
ok: true,
commit,
providerId: options.providerId,
sourceImage,
registryProbe: commandTail(registryProbe),
localLoadedImage,
targetImage: composeImage,
targetCommitImage: commitImage,
pull: commandTail(pull),
deploy: commandTail(deploy),
running: running.stdout.trim(),
rollback: {
previousImageHint: "Use docker image ls and docker inspect logs for the previous backend-core image id; Compose named volumes were not changed.",
commandShape: `${compose.map(shellQuote).join(" ")} up -d --no-build --no-deps --force-recreate backend-core`,
},
};
}
function deployBackendCoreJob(args: string[], options: ArtifactRegistryOptions): Record<string, unknown> {
@@ -815,7 +749,7 @@ function deployBackendCoreJob(args: string[], options: ArtifactRegistryOptions):
job,
statusCommand: `bun scripts/cli.ts job status ${job.id}`,
tailCommand: `bun scripts/cli.ts job status ${job.id} --tail-bytes 30000`,
note: "Backend-core CD continues in the background: D601 registry health, short-lived local relay, docker pull, retag, no-build recreate, live commit verification.",
note: "Backend-core CD continues in the background: D601 registry health, provider-gateway SSH image stream, docker load, retag, no-build recreate, live commit verification.",
};
}
+152 -9
View File
@@ -2,6 +2,7 @@ import { randomUUID } from "node:crypto";
import { existsSync, readFileSync } from "node:fs";
import { runCommand } from "./command";
import { type UniDeskConfig, repoRoot, rootPath } from "./config";
import { ensureGithubSshIdentityForProvider, gitSshHttpConnectProxySource } from "./deploy-ssh-identity";
import { startJob } from "./jobs";
import { coreInternalFetch } from "./microservices";
@@ -38,6 +39,7 @@ interface CiPublishBackendCoreOptions {
repoUrl: string;
commit: string;
waitMs: number;
sourceHostPath: string;
}
interface CiDevE2EOptions {
@@ -64,6 +66,20 @@ interface DispatchResult {
raw: unknown;
}
interface PipelineRunCondition {
ok: boolean | null;
status: string;
reason: string;
message: string;
query: {
ok: boolean;
status: string | null;
exitCode: number | null;
stdoutTail: string;
stderrTail: string;
};
}
interface DeployDevManifestSummary {
deployCommit: string;
desiredRef: string;
@@ -124,6 +140,15 @@ function shellQuote(value: string): string {
return `'${value.replace(/'/gu, "'\\''")}'`;
}
function repoSshUrl(repoUrl: string): string {
if (repoUrl.startsWith("git@")) return repoUrl;
if (repoUrl.startsWith("https://github.com/")) {
const repoPath = repoUrl.slice("https://github.com/".length).replace(/\.git$/u, "");
return `git@github.com:${repoPath}.git`;
}
return repoUrl;
}
function chunks(value: string, size: number): string[] {
const result: string[] = [];
for (let index = 0; index < value.length; index += size) {
@@ -508,6 +533,8 @@ spec:
value: ${JSON.stringify(options.repoUrl)}
- name: revision
value: ${JSON.stringify(options.commit)}
- name: source-host-path
value: ${JSON.stringify(options.sourceHostPath)}
workspaces:
- name: shared-workspace
persistentVolumeClaim:
@@ -515,6 +542,77 @@ spec:
`;
}
function backendCoreArtifactSourceHostPath(commit: string): string {
return `/home/ubuntu/.unidesk/ci/backend-core-artifacts/${commit}`;
}
async function prepareBackendCoreArtifactSource(config: UniDeskConfig, options: CiPublishBackendCoreOptions): Promise<Record<string, unknown>> {
const sshIdentity = await ensureGithubSshIdentityForProvider(config, d601ProviderId);
if (!sshIdentity.ok) throw new Error(sshIdentity.detail);
const proxyPython = gitSshHttpConnectProxySource();
const sourceRoot = "/home/ubuntu/.unidesk/ci/backend-core-artifacts";
const sourceHostPath = options.sourceHostPath;
const repoCache = "/home/ubuntu/.unidesk/ci/git/unidesk.git";
const repoFetchUrl = repoSshUrl(options.repoUrl);
const script = [
"set -euo pipefail",
`commit=${shellQuote(options.commit)}`,
`repo_url=${shellQuote(options.repoUrl)}`,
`repo_fetch_url=${shellQuote(repoFetchUrl)}`,
`source_root=${shellQuote(sourceRoot)}`,
`source_dir=${shellQuote(sourceHostPath)}`,
`repo_cache=${shellQuote(repoCache)}`,
`proxy_url=${shellQuote(providerGatewayWsEgressProxyUrl)}`,
"mkdir -p \"$(dirname \"$repo_cache\")\" \"$source_root\"",
"export HTTP_PROXY=\"$proxy_url\" HTTPS_PROXY=\"$proxy_url\" ALL_PROXY=\"$proxy_url\"",
"export NO_PROXY=\"localhost,127.0.0.1,::1,host.docker.internal,.svc,.cluster.local,kubernetes.default.svc\"",
"curl -fsSI --max-time 20 -x \"$proxy_url\" https://github.com >/dev/null",
"git_ssh_proxy=/tmp/unidesk-git-ssh-http-connect.py",
"cat > \"$git_ssh_proxy\" <<'UNIDESK_GIT_SSH_PROXY'",
proxyPython,
"UNIDESK_GIT_SSH_PROXY",
"chmod 700 \"$git_ssh_proxy\"",
"export UNIDESK_GIT_SSH_HTTP_PROXY=\"$proxy_url\"",
"export GIT_SSH_COMMAND=\"ssh -o BatchMode=yes -o IdentitiesOnly=yes -o StrictHostKeyChecking=yes -o UserKnownHostsFile=$HOME/.ssh/known_hosts -i $HOME/.ssh/id_ed25519 -o 'ProxyCommand=$git_ssh_proxy %h %p'\"",
"echo backend_core_artifact_source_proxy=provider-gateway-ws-egress:$proxy_url",
"echo backend_core_artifact_repo_fetch_url=$repo_fetch_url",
"if [ ! -d \"$repo_cache\" ]; then git clone --mirror \"$repo_fetch_url\" \"$repo_cache\"; fi",
"git -C \"$repo_cache\" remote set-url origin \"$repo_fetch_url\"",
"git -C \"$repo_cache\" fetch --no-tags origin \"$commit\" || git -C \"$repo_cache\" fetch --no-tags origin '+refs/heads/*:refs/remotes/origin/*'",
"resolved=$(git -C \"$repo_cache\" rev-parse --verify \"$commit^{commit}\")",
"test \"$resolved\" = \"$commit\" || { echo \"backend_core_artifact_resolved_commit_mismatch=$resolved expected=$commit\" >&2; exit 1; }",
"git -C \"$repo_cache\" cat-file -e \"$commit:src/components/backend-core/Dockerfile\"",
"git -C \"$repo_cache\" cat-file -e \"$commit:src/components/backend-core/src\"",
"tmp_dir=\"$source_root/.tmp-$commit-$$\"",
"rm -rf \"$tmp_dir\"",
"mkdir -p \"$tmp_dir\"",
"git -C \"$repo_cache\" archive \"$commit\" | tar -x -C \"$tmp_dir\"",
"printf '%s\\n' \"$commit\" > \"$tmp_dir/.unidesk-source-commit\"",
"printf '%s\\n' \"$repo_url\" > \"$tmp_dir/.unidesk-source-repo\"",
"rm -rf \"$source_dir\"",
"mv \"$tmp_dir\" \"$source_dir\"",
"test -f \"$source_dir/src/components/backend-core/Dockerfile\"",
"test -d \"$source_dir/src/components/backend-core/src\"",
"echo backend_core_artifact_source_host_path=$source_dir",
].join("\n");
const result = await runRemoteBackground("prepare-backend-core-source", script, 300_000);
if (!result.ok) throw new Error(`failed to prepare backend-core source on D601: ${result.stderr || result.stdout || JSON.stringify(result.raw)}`);
return {
ok: true,
mode: "d601-host-github-ssh-export",
providerId: d601ProviderId,
repoUrl: options.repoUrl,
repoFetchUrl,
commit: options.commit,
sourceHostPath,
identity: {
fingerprint: sshIdentity.fingerprint,
seededFromLocal: sshIdentity.seededFromLocal,
},
stdoutTail: result.stdout.slice(-4000),
};
}
async function remoteCreatePipelineRun(manifest: string): Promise<string> {
const encoded = Buffer.from(manifest, "utf8").toString("base64");
const token = randomUUID().replace(/-/gu, "").slice(0, 12);
@@ -536,18 +634,19 @@ async function waitForPipelineRun(name: string, waitMs: number): Promise<Dispatc
const command = [
"set -euo pipefail",
`export KUBECONFIG=${shellQuote(d601Kubeconfig)}`,
`printf 'waiting_pipelinerun=%s\\n' ${shellQuote(name)}`,
`deadline=$((SECONDS + ${Math.ceil(waitMs / 1000)}))`,
"while [ \"$SECONDS\" -lt \"$deadline\" ]; do",
` condition="$(kubectl get pipelinerun/${shellQuote(name)} -n unidesk-ci -o jsonpath='{range .status.conditions[?(@.type==\"Succeeded\")]}{.status}{\"\\t\"}{.reason}{\"\\t\"}{.message}{end}' 2>/dev/null || true)"`,
" case \"$condition\" in",
" True*)",
" echo \"$condition\"",
` kubectl get pipelinerun/${shellQuote(name)} -n unidesk-ci -o json`,
" printf 'condition=%s\\n' \"$condition\"",
` kubectl get taskrun -n unidesk-ci -l tekton.dev/pipelineRun=${shellQuote(name)} --no-headers 2>/dev/null || true`,
" exit 0",
" ;;",
" False*)",
" echo \"$condition\"",
` kubectl get pipelinerun/${shellQuote(name)} -n unidesk-ci -o json`,
" printf 'condition=%s\\n' \"$condition\"",
` kubectl get taskrun,pod -n unidesk-ci -l tekton.dev/pipelineRun=${shellQuote(name)} 2>/dev/null || true`,
" exit 1",
" ;;",
" esac",
@@ -560,10 +659,41 @@ async function waitForPipelineRun(name: string, waitMs: number): Promise<Dispatc
return dispatchSsh(command, waitMs + 30_000, waitMs + 20_000);
}
async function readPipelineRunCondition(name: string): Promise<PipelineRunCondition> {
const result = await runRemoteKubectlRaw([
"set -euo pipefail",
`condition="$(kubectl get pipelinerun/${shellQuote(name)} -n unidesk-ci -o jsonpath='{range .status.conditions[?(@.type==\"Succeeded\")]}{.status}{\"\\t\"}{.reason}{\"\\t\"}{.message}{end}' 2>/dev/null || true)"`,
"printf '%s\\n' \"$condition\"",
].join("\n"), 30_000, 15_000);
const [status = "", reason = "", ...messageParts] = result.stdout.trim().split("\t");
const message = messageParts.join("\t");
return {
ok: status === "True" ? true : status === "False" ? false : null,
status,
reason,
message,
query: {
ok: result.ok,
status: result.status,
exitCode: result.exitCode,
stdoutTail: result.stdout.slice(-1200),
stderrTail: result.stderr.slice(-1200),
},
};
}
function pipelineRunWaitSucceeded(wait: DispatchResult | null, condition: PipelineRunCondition | null): boolean {
if (wait === null) return true;
if (condition?.ok === true) return true;
if (condition?.ok === false) return false;
return wait.ok || wait.exitCode === 0 || wait.stdout.includes("condition=True\tSucceeded\t");
}
async function run(options: CiOptions): Promise<Record<string, unknown>> {
const name = await remoteCreatePipelineRun(pipelineRunManifest(options));
const wait = await waitForPipelineRun(name, options.waitMs);
const waitSucceeded = wait === null || wait.exitCode === 0 || wait.stdout.trimStart().startsWith("True\tSucceeded\t");
const condition = wait === null ? null : await readPipelineRunCondition(name);
const waitSucceeded = pipelineRunWaitSucceeded(wait, condition);
return {
ok: waitSucceeded,
pipelineRun: name,
@@ -571,9 +701,14 @@ async function run(options: CiOptions): Promise<Record<string, unknown>> {
repoUrl: options.repoUrl,
revision: options.revision,
wait: wait === null ? null : {
ok: waitSucceeded,
dispatchOk: wait.ok,
dispatchStatus: wait.status,
dispatchExitCode: wait.exitCode,
stdoutTail: wait.stdout.slice(-6000),
stderrTail: wait.stderr.slice(-6000),
},
condition,
next: [
`bun scripts/cli.ts ci logs ${name}`,
"bun scripts/cli.ts ci status",
@@ -581,22 +716,30 @@ async function run(options: CiOptions): Promise<Record<string, unknown>> {
};
}
async function publishBackendCoreArtifact(options: CiPublishBackendCoreOptions): Promise<Record<string, unknown>> {
async function publishBackendCoreArtifact(config: UniDeskConfig, options: CiPublishBackendCoreOptions): Promise<Record<string, unknown>> {
const source = await prepareBackendCoreArtifactSource(config, options);
const name = await remoteCreatePipelineRun(backendCoreArtifactPipelineRunManifest(options));
const wait = await waitForPipelineRun(name, options.waitMs);
const waitSucceeded = wait === null || wait.exitCode === 0 || wait.stdout.trimStart().startsWith("True\tSucceeded\t");
const condition = wait === null ? null : await readPipelineRunCondition(name);
const waitSucceeded = pipelineRunWaitSucceeded(wait, condition);
return {
ok: waitSucceeded,
pipelineRun: name,
namespace: "unidesk-ci",
repoUrl: options.repoUrl,
commit: options.commit,
source,
artifact: `127.0.0.1:5000/unidesk/backend-core:${options.commit}`,
boundary: "CI publishes the image to D601 registry; CD must pull it and must not build backend-core",
wait: wait === null ? null : {
ok: waitSucceeded,
dispatchOk: wait.ok,
dispatchStatus: wait.status,
dispatchExitCode: wait.exitCode,
stdoutTail: wait.stdout.slice(-6000),
stderrTail: wait.stderr.slice(-6000),
},
condition,
next: [
`bun scripts/cli.ts ci logs ${name}`,
`bun scripts/cli.ts artifact-registry deploy-backend-core --commit ${options.commit}`,
@@ -890,7 +1033,7 @@ function requireRunId(value: string): string {
return value;
}
export async function runCiCommand(_config: UniDeskConfig, args: string[]): Promise<Record<string, unknown>> {
export async function runCiCommand(config: UniDeskConfig, args: string[]): Promise<Record<string, unknown>> {
const [action = "status", nameArg] = args;
if (isHelpArg(action) || args.slice(1).some(isHelpArg)) return ciHelp();
if (action === "install") return install();
@@ -905,7 +1048,7 @@ export async function runCiCommand(_config: UniDeskConfig, args: string[]): Prom
const repoUrl = stringOption(args, "--repo") ?? stringOption(args, "--repo-url") ?? "https://github.com/pikasTech/unidesk";
const commit = requireFullCommit(stringOption(args, "--commit") ?? stringOption(args, "--revision"));
const waitMs = numberOption(args, "--wait-ms", 0);
return publishBackendCoreArtifact({ repoUrl, commit, waitMs });
return publishBackendCoreArtifact(config, { repoUrl, commit, waitMs, sourceHostPath: backendCoreArtifactSourceHostPath(commit) });
}
if (action === "run-dev-e2e") {
const repoUrl = stringOption(args, "--repo") ?? stringOption(args, "--repo-url") ?? "https://github.com/pikasTech/unidesk";
+74 -2
View File
@@ -21,6 +21,7 @@ interface GithubSshIdentity {
const identityId = "github.com";
const defaultPrivateKeyPath = "/root/.ssh/id_ed25519";
const defaultKnownHostsPath = "/root/.ssh/known_hosts";
const providerGatewayWsEgressProxyUrl = "http://127.0.0.1:18789";
function pgLiteral(value: string): string {
return `'${value.replace(/'/gu, "''")}'`;
@@ -255,6 +256,66 @@ function shellQuote(value: string): string {
return `'${value.replace(/'/gu, `'\\''`)}'`;
}
export function gitSshHttpConnectProxySource(): string {
return String.raw`#!/usr/bin/env python3
import os
import select
import socket
import sys
from urllib.parse import urlparse
if len(sys.argv) != 3:
raise SystemExit("usage: unidesk-git-ssh-http-connect.py host port")
target_host = sys.argv[1]
target_port = int(sys.argv[2])
proxy = urlparse(os.environ.get("UNIDESK_GIT_SSH_HTTP_PROXY", "http://127.0.0.1:18789"))
proxy_host = proxy.hostname or "127.0.0.1"
proxy_port = proxy.port or 80
sock = socket.create_connection((proxy_host, proxy_port), timeout=20)
sock.sendall(f"CONNECT {target_host}:{target_port} HTTP/1.1\r\nHost: {target_host}:{target_port}\r\n\r\n".encode("ascii"))
header = b""
while b"\r\n\r\n" not in header:
chunk = sock.recv(4096)
if not chunk:
raise SystemExit("proxy closed before CONNECT response")
header += chunk
head, rest = header.split(b"\r\n\r\n", 1)
if not (head.startswith(b"HTTP/1.1 200") or head.startswith(b"HTTP/1.0 200")):
sys.stderr.write(head.decode("latin1", "replace") + "\n")
raise SystemExit(1)
if rest:
os.write(1, rest)
stdin_open = True
sock.setblocking(False)
while True:
readers = [sock]
if stdin_open:
readers.append(sys.stdin.buffer)
ready, _, _ = select.select(readers, [], [])
if sock in ready:
try:
data = sock.recv(65536)
except BlockingIOError:
data = b""
if not data:
break
os.write(1, data)
if stdin_open and sys.stdin.buffer in ready:
data = os.read(0, 65536)
if data:
sock.sendall(data)
else:
stdin_open = False
try:
sock.shutdown(socket.SHUT_WR)
except OSError:
pass
`;
}
function distributeGithubIdentity(providerId: string, identity: GithubSshIdentity): { ok: boolean; detail: string; raw: unknown } {
const payload = JSON.stringify({
privateKey: identity.privateKey,
@@ -262,12 +323,23 @@ function distributeGithubIdentity(providerId: string, identity: GithubSshIdentit
knownHosts: identity.knownHosts,
});
const remotePython = remoteInstallPythonSource();
const proxyPython = gitSshHttpConnectProxySource();
const remoteCommand = [
"set -euo pipefail",
`python3 -c ${shellQuote(remotePython)}`,
"auth_output=$(ssh -o BatchMode=yes -o IdentitiesOnly=yes -o StrictHostKeyChecking=yes -T git@github.com 2>&1 || true)",
`proxy_url=${shellQuote(providerGatewayWsEgressProxyUrl)}`,
"curl -fsSI --max-time 20 -x \"$proxy_url\" https://github.com >/dev/null",
"git_ssh_proxy=/tmp/unidesk-git-ssh-http-connect.py",
"cat > \"$git_ssh_proxy\" <<'UNIDESK_GIT_SSH_PROXY'",
proxyPython,
"UNIDESK_GIT_SSH_PROXY",
"chmod 700 \"$git_ssh_proxy\"",
"export UNIDESK_GIT_SSH_HTTP_PROXY=\"$proxy_url\"",
"export GIT_SSH_COMMAND=\"ssh -o BatchMode=yes -o IdentitiesOnly=yes -o StrictHostKeyChecking=yes -o UserKnownHostsFile=$HOME/.ssh/known_hosts -i $HOME/.ssh/id_ed25519 -o 'ProxyCommand=$git_ssh_proxy %h %p'\"",
"auth_output=$(ssh -o BatchMode=yes -o IdentitiesOnly=yes -o StrictHostKeyChecking=yes -o UserKnownHostsFile=$HOME/.ssh/known_hosts -i $HOME/.ssh/id_ed25519 -o \"ProxyCommand=$git_ssh_proxy %h %p\" -T git@github.com 2>&1 || true)",
"printf '%s\\n' \"$auth_output\"",
"printf '%s\\n' \"$auth_output\" | grep -q 'successfully authenticated'",
].join(" && ");
].join("\n");
const result = spawnSync(process.execPath, [
rootPath("scripts", "cli.ts"),
"ssh",
@@ -276,6 +276,8 @@ spec:
- name: registry
type: string
default: 127.0.0.1:5000
- name: source-host-path
type: string
workspaces:
- name: source
volumes:
@@ -283,26 +285,17 @@ spec:
hostPath:
path: /var/run/docker.sock
type: Socket
- name: prepared-source
hostPath:
path: /home/ubuntu/.unidesk/ci/backend-core-artifacts
type: Directory
steps:
- name: clone
- name: prepare-source
image: alpine/git:2.45.2
env:
- name: HTTP_PROXY
value: "http://d601-provider-egress-proxy.unidesk.svc.cluster.local:18789"
- name: HTTPS_PROXY
value: "http://d601-provider-egress-proxy.unidesk.svc.cluster.local:18789"
- name: ALL_PROXY
value: "http://d601-provider-egress-proxy.unidesk.svc.cluster.local:18789"
- name: NO_PROXY
value: "localhost,127.0.0.1,::1,host.docker.internal,d601-provider-egress-proxy,d601-provider-egress-proxy.unidesk,d601-provider-egress-proxy.unidesk.svc,d601-provider-egress-proxy.unidesk.svc.cluster.local"
- name: http_proxy
value: "http://d601-provider-egress-proxy.unidesk.svc.cluster.local:18789"
- name: https_proxy
value: "http://d601-provider-egress-proxy.unidesk.svc.cluster.local:18789"
- name: all_proxy
value: "http://d601-provider-egress-proxy.unidesk.svc.cluster.local:18789"
- name: no_proxy
value: "localhost,127.0.0.1,::1,host.docker.internal,d601-provider-egress-proxy,d601-provider-egress-proxy.unidesk,d601-provider-egress-proxy.unidesk.svc,d601-provider-egress-proxy.unidesk.svc.cluster.local"
volumeMounts:
- name: prepared-source
mountPath: /prepared-source
readOnly: true
script: |
#!/bin/sh
set -eu
@@ -310,16 +303,22 @@ spec:
[0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f]) ;;
*) echo "backend_core_artifact_revision_must_be_full_sha=$(params.revision)" >&2; exit 2 ;;
esac
rm -rf "$(workspaces.source.path)/backend-core-artifact-repo"
git clone --filter=blob:none "$(params.repo-url)" "$(workspaces.source.path)/backend-core-artifact-repo"
mkdir -p "$(workspaces.source.path)/backend-core-artifact-repo"
find "$(workspaces.source.path)/backend-core-artifact-repo" -mindepth 1 -maxdepth 1 -exec rm -rf {} +
case "$(params.source-host-path)" in
/home/ubuntu/.unidesk/ci/backend-core-artifacts/*) ;;
*) echo "backend_core_artifact_source_host_path_invalid=$(params.source-host-path)" >&2; exit 2 ;;
esac
source_name="$(basename "$(params.source-host-path)")"
source_dir="/prepared-source/$source_name"
test -f "$source_dir/.unidesk-source-commit"
prepared_commit="$(cat "$source_dir/.unidesk-source-commit")"
test "$prepared_commit" = "$(params.revision)"
cp -a "$source_dir/." "$(workspaces.source.path)/backend-core-artifact-repo/"
cd "$(workspaces.source.path)/backend-core-artifact-repo"
git fetch --depth=1 origin "$(params.revision)"
git checkout --detach FETCH_HEAD
resolved="$(git rev-parse HEAD)"
test "$resolved" = "$(params.revision)"
git cat-file -e "$resolved:src/components/backend-core/Dockerfile"
git cat-file -e "$resolved:src/components/backend-core/src"
git rev-parse HEAD | tee "$(workspaces.source.path)/backend-core-artifact-commit.txt"
test -f src/components/backend-core/Dockerfile
test -d src/components/backend-core/src
printf '%s\n' "$prepared_commit" | tee "$(workspaces.source.path)/backend-core-artifact-commit.txt"
- name: build-and-push
image: "$(params.app-image)"
imagePullPolicy: Never
@@ -354,13 +353,10 @@ spec:
local_image="unidesk/backend-core:$commit"
registry_image="$registry/unidesk/backend-core:$commit"
command -v docker
docker buildx version >/dev/null 2>&1 || docker buildx inspect default >/dev/null 2>&1
docker version >/dev/null
docker run --rm --network host rancher/mirrored-library-busybox:1.36.1 wget -q -O- "http://$registry/v2/" >/dev/null
docker buildx build \
--builder default \
--load \
DOCKER_BUILDKIT=0 docker build \
--network host \
--progress=plain \
--build-arg HTTP_PROXY=http://127.0.0.1:18789 \
--build-arg HTTPS_PROXY=http://127.0.0.1:18789 \
--build-arg ALL_PROXY=http://127.0.0.1:18789 \
@@ -401,6 +397,8 @@ spec:
- name: registry
type: string
default: 127.0.0.1:5000
- name: source-host-path
type: string
workspaces:
- name: shared-workspace
tasks:
@@ -416,6 +414,8 @@ spec:
value: "$(params.app-image)"
- name: registry
value: "$(params.registry)"
- name: source-host-path
value: "$(params.source-host-path)"
workspaces:
- name: source
workspace: shared-workspace