feat(ci): add code queue dev smoke
This commit is contained in:
@@ -86,6 +86,11 @@
|
||||
"id": "frontend",
|
||||
"repo": "https://github.com/pikasTech/unidesk",
|
||||
"commitId": "c09beb09e8f7e72cfe8dc7c1379d39f7facbfb3a"
|
||||
},
|
||||
{
|
||||
"id": "code-queue",
|
||||
"repo": "https://github.com/pikasTech/unidesk",
|
||||
"commitId": "60f991f826b16e0784bb06e3d1af5406258a8c9e"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
@@ -21,6 +21,7 @@ Each commit CI run performs:
|
||||
- `UNIDESK_D601_RUST_CHECK=1 bun scripts/cli.ts check --full --rust`, so Rust backend-core is checked only inside the D601 CI execution boundary.
|
||||
- Temporary `code-queue-ci-read` Deployment and ClusterIP Service in `unidesk-ci`.
|
||||
- Code Queue read performance checks against the production PostgreSQL through `d601-tcp-egress-gateway`.
|
||||
- Manual dev desired-state smoke for Code Queue via `ci run-dev-e2e`, using the Git-pinned `code-queue` service commit from `origin/master:deploy.json#environments.dev`.
|
||||
|
||||
`ci install` also prewarms the D601 k3s containerd runtime with the Tekton entrypoint/workingdir helper images, `oven/bun:1-debian`, `alpine/git:2.45.2` and `unidesk-code-queue:dev`. Missing images are pulled through the node-local provider-gateway WS egress proxy and then imported into native k3s containerd with digests preserved, so PipelineRun pods do not hang on external registry pulls. Sustained pull throughput below 1 MB/s is treated as a provider/main-server network or proxy degradation first, not as a Dockerfile or application failure.
|
||||
|
||||
@@ -48,7 +49,7 @@ This means the CI service can read existing tasks, Trace summaries, Trace steps
|
||||
|
||||
`ci run-dev-e2e` is the manual dev desired-state smoke flow. The single authoritative reference for its Git-controlled runner script, short launcher, result directory and no-CD boundary is `docs/reference/dev-ci-runner.md`.
|
||||
|
||||
The current dev namespace e2e is a harness and smoke gate, not a full frontend/backend/code-queue stack rollout. Full-stack temporary namespace deployment can be added behind the same command only after image build/import and per-run database bootstrap are promoted into a controlled deployment design.
|
||||
The current dev namespace e2e is a harness and smoke gate, not a full frontend/backend stack rollout. It does include a controlled Code Queue slice: D601 builds or reuses the `environments.dev.services[].id=code-queue` commit, imports the image into native k3s containerd, starts temporary PostgreSQL plus Code Queue scheduler/read/write Services in `unidesk-ci-e2e-<runId>`, and verifies the HTTP API through the Kubernetes API service proxy. This remains CI-only and must not deploy persistent `unidesk-dev` or production resources.
|
||||
|
||||
## Performance Gate
|
||||
|
||||
|
||||
@@ -15,6 +15,7 @@ The runner exists to prove the dev desired state without interrupting production
|
||||
- D601 execution: Git fetch, Tekton PipelineRun creation, Kubernetes polling and e2e log collection happen on D601, not on the main master.
|
||||
- CLI observability: the submit command returns a `runId`, result directory and next commands; `ci logs <runId>` can recover status after the local CLI exits.
|
||||
- CI only: the flow may create CI-owned temporary resources, but it must not deploy backend-core, frontend, Code Queue, Decision Center, k3sctl-adapter or any other direct/managed service.
|
||||
- Code Queue reproducibility: the runner must use the `code-queue` commit from `environments.dev.services`, build or reuse a labeled image from that Git commit on D601, import it into native k3s containerd, and validate the HTTP API inside a temporary namespace.
|
||||
|
||||
## Design Boundary
|
||||
|
||||
@@ -54,6 +55,11 @@ Do not add a long-lived DevOps service, run broker, webhook listener or second d
|
||||
"id": "frontend",
|
||||
"repo": "https://github.com/pikasTech/unidesk",
|
||||
"commitId": "<pushed-commit>"
|
||||
},
|
||||
{
|
||||
"id": "code-queue",
|
||||
"repo": "https://github.com/pikasTech/unidesk",
|
||||
"commitId": "<pushed-commit>"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -61,7 +67,7 @@ Do not add a long-lived DevOps service, run broker, webhook listener or second d
|
||||
}
|
||||
```
|
||||
|
||||
`scriptPath` must be a repo-relative `scripts/ci/*.sh` path. Inline shell bodies, arbitrary script paths, local dirty scripts and separate `develop.json` or CI manifest files are forbidden. The script is fetched from the same full 40-character manifest commit that supplied `deploy.json`, so the runner logic is auditable and rollbackable with the desired state. Persistent dev rollout service scope is owned by `docs/reference/dev-environment.md`; this runner only consumes the dev service list for smoke verification and must not deploy it.
|
||||
`scriptPath` must be a repo-relative `scripts/ci/*.sh` path. Inline shell bodies, arbitrary script paths, local dirty scripts and separate `develop.json` or CI manifest files are forbidden. The script is fetched from the same full 40-character manifest commit that supplied `deploy.json`, so the runner logic is auditable and rollbackable with the desired state. Persistent dev rollout service scope is owned by `docs/reference/dev-environment.md`; this runner only consumes the dev service list for smoke verification and must not deploy it. `code-queue` is required in the dev service list for this smoke runner, but that does not enable `deploy apply --env dev --service code-queue`.
|
||||
|
||||
## Execution Path
|
||||
|
||||
@@ -73,7 +79,7 @@ The automatic path is intentionally single and narrow:
|
||||
4. D601 creates `/tmp/unidesk-ci/<runId>` and `/home/ubuntu/.unidesk/runs/<runId>`.
|
||||
5. D601 fetches the manifest commit from GitHub through the node-local provider-gateway WS egress proxy at `http://127.0.0.1:18789`.
|
||||
6. D601 extracts the runner with `git show <commit>:<scriptPath> > /tmp/unidesk-ci/<runId>/runner.sh` and the desired-state blob with `git show <commit>:deploy.json > /tmp/unidesk-ci/<runId>/deploy.json`.
|
||||
7. The runner parses the host-fetched `deploy.json`, creates the Tekton PipelineRun in `unidesk-ci`, passes the required dev service commits as PipelineRun params, waits for completion when requested, and writes `result.json`, `launcher.log`, `runner.log`, PipelineRun JSON and pod logs under `/home/ubuntu/.unidesk/runs/<runId>/`.
|
||||
7. The runner parses the host-fetched `deploy.json`, requires a full-SHA `code-queue` service commit, builds or reuses a D601 Docker image for that commit, imports the image and `postgres:16-alpine` into native k3s containerd, creates the Tekton PipelineRun in `unidesk-ci`, passes the required dev service commits and Code Queue image tag as PipelineRun params, waits for completion when requested, and writes `result.json`, `launcher.log`, `runner.log`, PipelineRun JSON and pod logs under `/home/ubuntu/.unidesk/runs/<runId>/`.
|
||||
|
||||
The CLI must not upload the runner script body. Tekton dev e2e must not clone the private UniDesk repo itself; repo access and desired-state extraction happen once in the D601 host launcher under the manifest commit. The submitted launcher may contain only repo, full commit, script path, run id, environment, timeout, keep-namespace and fixed workspace path settings plus the fixed fetch/execute wrapper. If k3s, Tekton or the provider egress proxy is unavailable, the run fails with visible logs; it must not fall back to an alternate deployment path.
|
||||
|
||||
@@ -96,6 +102,8 @@ scripts/ci/dev-e2e.sh \
|
||||
|
||||
The current script creates a Tekton `PipelineRun` for `pipeline/unidesk-dev-namespace-e2e`, stores the generated PipelineRun name in `pipelinerun.txt`, and writes a final `result.json` with `ok`, `status`, `runId`, `manifestCommit`, `pipelineRun`, `temporaryNamespace` and `finishedAt`.
|
||||
|
||||
The Tekton task creates a temporary namespace `unidesk-ci-e2e-<runId>` and may create only CI-owned smoke resources there: `postgres-dev`, `code-queue-scheduler-dev`, `code-queue-read-dev`, `code-queue-write-dev`, their ClusterIP Services and a per-run Secret/ConfigMap. It must not mutate `unidesk` or persistent `unidesk-dev`. Code Queue API validation must use ClusterIP Services and the Kubernetes API `services/.../proxy` subresource; NodePort, D601 host ports and direct public service exposure are forbidden. The smoke currently proves `/health`, `/live` and `/api/workdirs` GET/POST/DELETE on read/write/scheduler roles, giving follow-up Code Queue API fixes a reproducible CI target before production rollout.
|
||||
|
||||
## Commands
|
||||
|
||||
Start a run and return after dispatch:
|
||||
|
||||
@@ -30,14 +30,14 @@ The unrestricted public network entries are therefore production frontend, dev f
|
||||
|
||||
## Desired State
|
||||
|
||||
`deploy.json` remains the only version intent file. Dev entries live under `environments.dev` and are read from `origin/master:deploy.json`, never from a dirty local file, when using `--env dev`.
|
||||
`deploy.json` remains the only version intent file. Dev entries live under `environments.dev` and are read from `origin/master:deploy.json`, never from a dirty local file, when using `--env dev` or `ci run-dev-e2e`.
|
||||
|
||||
The persistent dev rollout currently supports only:
|
||||
|
||||
- `backend-core`
|
||||
- `frontend`
|
||||
|
||||
`code-queue`, Decision Center, k3sctl-adapter and other D601 services are not part of persistent dev apply yet. Their smoke validation stays under `ci run-dev-e2e` or service-specific future designs. The `environments.dev.ci` declaration and short launcher runner are owned by `docs/reference/dev-ci-runner.md`.
|
||||
`code-queue` is present in `environments.dev.services` only so `ci run-dev-e2e` can build a Git-pinned Code Queue image and run a temporary namespace smoke. It is not part of persistent dev apply: `deploy apply --env dev --service code-queue` must still be rejected. Decision Center, k3sctl-adapter and other D601 services are not part of persistent dev apply yet. Their smoke validation stays under `ci run-dev-e2e` or service-specific future designs. The `environments.dev.ci` declaration and short launcher runner are owned by `docs/reference/dev-ci-runner.md`.
|
||||
|
||||
## Rust Backend-Core Boundary
|
||||
|
||||
@@ -94,7 +94,7 @@ Use this sequence for backend-core Rust and frontend dev work:
|
||||
7. If the dev service catalog changes, deploy the pushed `k3sctl-adapter` commit through the controlled local manifest exception, then verify `/api/control-plane` lists `k3s/dev/unidesk-dev-core.k3s.json`.
|
||||
8. Rebuild or verify `dev-frontend-proxy` on the main server with `bun scripts/cli.ts server rebuild dev-frontend-proxy` when the proxy config or port changes.
|
||||
9. Manually test `http://74.48.78.17:18083/` and the dev health endpoints.
|
||||
10. Run D601 CI for the commit and the dev smoke runner: `bun scripts/cli.ts ci run --revision <commit> --wait-ms <ms>` and `bun scripts/cli.ts ci run-dev-e2e --wait-ms <ms>`.
|
||||
10. Run D601 CI for the commit and the dev smoke runner: `bun scripts/cli.ts ci run --revision <commit> --wait-ms <ms>` and `bun scripts/cli.ts ci run-dev-e2e --wait-ms <ms>`. When Code Queue behavior changes, update the `code-queue` entry in `environments.dev.services` to the pushed commit before running the dev smoke; do not use `deploy apply --env dev --service code-queue`.
|
||||
|
||||
## Validation Commands
|
||||
|
||||
@@ -105,6 +105,8 @@ bun scripts/cli.ts server status
|
||||
bun scripts/cli.ts deploy plan --env dev
|
||||
bun scripts/cli.ts deploy plan --env dev --service backend-core
|
||||
bun scripts/cli.ts dev-env validate --manifest src/components/microservices/k3sctl-adapter/k3s/dev/unidesk-dev-core.k8s.yaml
|
||||
bun scripts/cli.ts dev-env validate --manifest src/components/microservices/k3sctl-adapter/k3s/dev/unidesk-dev-code-queue.k8s.yaml
|
||||
bun scripts/cli.ts ci run-dev-e2e --wait-ms 600000
|
||||
bun scripts/cli.ts microservice proxy k3sctl-adapter /api/services/backend-core-dev/proxy/health --raw --full
|
||||
bun scripts/cli.ts microservice proxy k3sctl-adapter /api/services/frontend-dev/proxy/health --raw --full
|
||||
curl -fsS http://127.0.0.1:18083/health
|
||||
|
||||
+104
-2
@@ -116,12 +116,12 @@ write_result() {
|
||||
local ok="$1"
|
||||
local status="$2"
|
||||
local detail="$3"
|
||||
python3 - "$result_json" "$ok" "$status" "$detail" "$run_id" "$repo_url" "$desired_ref" "$manifest_commit" "$environment" "$pipeline_run" "$temporary_namespace" <<'PY'
|
||||
python3 - "$result_json" "$ok" "$status" "$detail" "$run_id" "$repo_url" "$desired_ref" "$manifest_commit" "$environment" "$pipeline_run" "$temporary_namespace" "$code_queue_image" <<'PY'
|
||||
import json
|
||||
import sys
|
||||
from datetime import datetime, timezone
|
||||
|
||||
path, ok, status, detail, run_id, repo, desired_ref, commit, environment, pipeline_run, temporary_namespace = sys.argv[1:]
|
||||
path, ok, status, detail, run_id, repo, desired_ref, commit, environment, pipeline_run, temporary_namespace, code_queue_image = sys.argv[1:]
|
||||
record = {
|
||||
"ok": ok == "true",
|
||||
"status": status,
|
||||
@@ -133,6 +133,7 @@ record = {
|
||||
"environment": environment,
|
||||
"pipelineRun": pipeline_run or None,
|
||||
"temporaryNamespace": temporary_namespace or None,
|
||||
"codeQueueImage": code_queue_image or None,
|
||||
"finishedAt": datetime.now(timezone.utc).isoformat(),
|
||||
}
|
||||
with open(path, "w", encoding="utf-8") as handle:
|
||||
@@ -144,6 +145,7 @@ PY
|
||||
|
||||
pipeline_run=""
|
||||
temporary_namespace="unidesk-ci-e2e-$run_id"
|
||||
code_queue_image=""
|
||||
trap 'code=$?; if [ "$code" -ne 0 ] && [ ! -f "$result_json" ]; then write_result false failed "runner exited with code $code" || true; fi' EXIT
|
||||
|
||||
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
|
||||
@@ -188,8 +190,106 @@ source "$service_env"
|
||||
backend_commit="${BACKEND_CORE_COMMIT:-unknown}"
|
||||
frontend_commit="${FRONTEND_COMMIT:-unknown}"
|
||||
code_queue_commit="${CODE_QUEUE_COMMIT:-unknown}"
|
||||
if ! [[ "$code_queue_commit" =~ ^[0-9a-f]{40}$ ]]; then
|
||||
echo "deploy.json environments.$environment.services must include code-queue with a full 40 character commitId; got: $code_queue_commit" >&2
|
||||
exit 2
|
||||
fi
|
||||
deploy_json_b64="$(base64 -w0 "$manifest_file")"
|
||||
|
||||
work_dir="$(dirname "$manifest_file")"
|
||||
repo_dir="$work_dir/repo"
|
||||
if [ ! -d "$repo_dir/.git" ]; then
|
||||
echo "launcher repo checkout is missing: $repo_dir" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
root_exec() {
|
||||
if [ "$(id -u)" = "0" ]; then
|
||||
"$@"
|
||||
return
|
||||
fi
|
||||
if sudo -n true >/dev/null 2>&1; then
|
||||
sudo -n "$@"
|
||||
return
|
||||
fi
|
||||
if [ -x /mnt/c/Windows/System32/wsl.exe ]; then
|
||||
/mnt/c/Windows/System32/wsl.exe -u root -- "$@"
|
||||
return
|
||||
fi
|
||||
echo "dev_e2e_native_k3s_root_access=missing" >&2
|
||||
return 1
|
||||
}
|
||||
|
||||
import_image_to_k3s() {
|
||||
local image="$1"
|
||||
local archive="/tmp/unidesk-ci-image-${run_id}-${image//[^A-Za-z0-9_.-]/-}.tar"
|
||||
rm -f "$archive"
|
||||
docker save "$image" -o "$archive"
|
||||
root_exec ctr --address /run/k3s/containerd/containerd.sock -n k8s.io images import --digests --all-platforms "$archive"
|
||||
rm -f "$archive"
|
||||
echo "dev_e2e_image_imported=$image"
|
||||
}
|
||||
|
||||
ensure_runtime_image() {
|
||||
local image="$1"
|
||||
if ! docker image inspect "$image" >/dev/null 2>&1; then
|
||||
echo "dev_e2e_runtime_image_pull=$image"
|
||||
docker pull --platform linux/amd64 "$image"
|
||||
else
|
||||
echo "dev_e2e_runtime_image_cached=$image"
|
||||
fi
|
||||
import_image_to_k3s "$image"
|
||||
}
|
||||
|
||||
build_code_queue_image() {
|
||||
local short="${code_queue_commit:0:12}"
|
||||
local commit_image="unidesk-code-queue:ci-$short"
|
||||
local run_image="unidesk-code-queue:ci-dev-e2e-$run_id"
|
||||
local source_dir="$work_dir/code-queue-src-$short"
|
||||
local resolved
|
||||
git -C "$repo_dir" fetch --no-tags origin "$code_queue_commit" || git -C "$repo_dir" fetch --no-tags origin '+refs/heads/*:refs/remotes/origin/*'
|
||||
resolved="$(git -C "$repo_dir" rev-parse --verify "$code_queue_commit^{commit}")"
|
||||
if [ "$resolved" != "$code_queue_commit" ]; then
|
||||
echo "code_queue_commit_mismatch resolved=$resolved expected=$code_queue_commit" >&2
|
||||
exit 1
|
||||
fi
|
||||
local existing_rev
|
||||
existing_rev="$(docker image inspect "$commit_image" --format '{{ index .Config.Labels "org.opencontainers.image.revision" }}' 2>/dev/null || true)"
|
||||
if [ "$existing_rev" != "$resolved" ]; then
|
||||
rm -rf "$source_dir"
|
||||
mkdir -p "$source_dir"
|
||||
git -C "$repo_dir" archive "$resolved" | tar -x -C "$source_dir"
|
||||
local base_args=()
|
||||
if docker image inspect unidesk-code-queue:d601-build-base >/dev/null 2>&1; then
|
||||
base_args=(--build-arg CODE_QUEUE_BASE_IMAGE=unidesk-code-queue:d601-build-base)
|
||||
elif docker image inspect unidesk-code-queue:d601 >/dev/null 2>&1; then
|
||||
base_args=(--build-arg CODE_QUEUE_BASE_IMAGE=unidesk-code-queue:d601)
|
||||
fi
|
||||
echo "dev_e2e_code_queue_image_build=$commit_image commit=$resolved"
|
||||
docker build \
|
||||
"${base_args[@]}" \
|
||||
--build-arg HTTP_PROXY="${HTTP_PROXY:-}" \
|
||||
--build-arg HTTPS_PROXY="${HTTPS_PROXY:-}" \
|
||||
--build-arg ALL_PROXY="${ALL_PROXY:-}" \
|
||||
--build-arg NO_PROXY="${NO_PROXY:-}" \
|
||||
--label "org.opencontainers.image.source=$repo_url" \
|
||||
--label "org.opencontainers.image.revision=$resolved" \
|
||||
--label "unidesk.ai/ci-run-id=$run_id" \
|
||||
--label "unidesk.ai/ci-kind=dev-e2e" \
|
||||
-t "$commit_image" \
|
||||
-f "$source_dir/src/components/microservices/code-queue/Dockerfile" \
|
||||
"$source_dir"
|
||||
else
|
||||
echo "dev_e2e_code_queue_image_cached=$commit_image commit=$resolved"
|
||||
fi
|
||||
docker tag "$commit_image" "$run_image"
|
||||
import_image_to_k3s "$run_image"
|
||||
code_queue_image="$run_image"
|
||||
}
|
||||
|
||||
ensure_runtime_image "postgres:16-alpine"
|
||||
build_code_queue_image
|
||||
|
||||
pipeline_manifest="$result_dir/pipelinerun.yaml"
|
||||
cat >"$pipeline_manifest" <<YAML
|
||||
apiVersion: tekton.dev/v1
|
||||
@@ -227,6 +327,8 @@ spec:
|
||||
value: "$frontend_commit"
|
||||
- name: code-queue-commit
|
||||
value: "$code_queue_commit"
|
||||
- name: app-image
|
||||
value: "$code_queue_image"
|
||||
- name: deploy-json-b64
|
||||
value: "$deploy_json_b64"
|
||||
workspaces:
|
||||
|
||||
@@ -683,6 +683,13 @@ function resolveDeployDevManifest(desiredRef: string): DeployDevManifestSummary
|
||||
};
|
||||
}).filter((service) => service.id.length > 0 && service.commitId.length > 0);
|
||||
if (services.length === 0) throw new Error(`origin/${desiredRef}:deploy.json has no environments.dev services with commitId`);
|
||||
const codeQueueService = services.find((service) => service.id === "code-queue");
|
||||
if (codeQueueService === undefined) {
|
||||
throw new Error(`origin/${desiredRef}:deploy.json environments.dev.services must include code-queue for ci run-dev-e2e`);
|
||||
}
|
||||
if (!/^[0-9a-f]{40}$/u.test(codeQueueService.commitId)) {
|
||||
throw new Error(`origin/${desiredRef}:deploy.json environments.dev.services code-queue commitId must be a full 40-character SHA`);
|
||||
}
|
||||
return {
|
||||
deployCommit: deployCommitResult.stdout.trim(),
|
||||
desiredRef,
|
||||
|
||||
@@ -88,7 +88,7 @@ rules:
|
||||
resources: ["namespaces"]
|
||||
verbs: ["get", "list", "watch", "create", "delete", "patch"]
|
||||
- apiGroups: [""]
|
||||
resources: ["configmaps", "services", "pods", "pods/log"]
|
||||
resources: ["configmaps", "secrets", "services", "services/proxy", "pods", "pods/log"]
|
||||
verbs: ["get", "list", "watch", "create", "delete", "patch"]
|
||||
- apiGroups: ["apps"]
|
||||
resources: ["deployments"]
|
||||
@@ -701,6 +701,8 @@ spec:
|
||||
backend_commit="$(params.backend-core-commit)"
|
||||
frontend_commit="$(params.frontend-commit)"
|
||||
code_queue_commit="$(params.code-queue-commit)"
|
||||
app_image="$(params.app-image)"
|
||||
database_url="postgres://unidesk_ci:unidesk_ci_password@postgres-dev.$ns.svc.cluster.local:5432/unidesk_ci"
|
||||
kube_api="https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT_HTTPS}"
|
||||
kube_token="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)"
|
||||
kube_ca="/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
|
||||
@@ -719,6 +721,210 @@ spec:
|
||||
cat /tmp/unidesk-dev-e2e-delete-response >&2
|
||||
return 1
|
||||
}
|
||||
proxy_get() {
|
||||
local service_name="$1"
|
||||
local path="$2"
|
||||
curl -fsS --cacert "$kube_ca" -H "Authorization: Bearer $kube_token" \
|
||||
"$kube_api/api/v1/namespaces/$ns/services/${service_name}:4222/proxy$path"
|
||||
}
|
||||
proxy_json() {
|
||||
local method="$1"
|
||||
local service_name="$2"
|
||||
local path="$3"
|
||||
local body="${4:-}"
|
||||
if [ -n "$body" ]; then
|
||||
curl -fsS --cacert "$kube_ca" -H "Authorization: Bearer $kube_token" -H "Content-Type: application/json" \
|
||||
-X "$method" --data "$body" \
|
||||
"$kube_api/api/v1/namespaces/$ns/services/${service_name}:4222/proxy$path"
|
||||
else
|
||||
curl -fsS --cacert "$kube_ca" -H "Authorization: Bearer $kube_token" \
|
||||
-X "$method" \
|
||||
"$kube_api/api/v1/namespaces/$ns/services/${service_name}:4222/proxy$path"
|
||||
fi
|
||||
}
|
||||
wait_deployment_available() {
|
||||
local deployment="$1"
|
||||
local timeout_seconds="$2"
|
||||
local deadline=$((SECONDS + timeout_seconds))
|
||||
while [ "$SECONDS" -lt "$deadline" ]; do
|
||||
status="$(kube GET "$kube_api/apis/apps/v1/namespaces/$ns/deployments/$deployment")"
|
||||
replicas="$(printf '%s' "$status" | jq -r '.spec.replicas // 1')"
|
||||
available="$(printf '%s' "$status" | jq -r '.status.availableReplicas // 0')"
|
||||
updated="$(printf '%s' "$status" | jq -r '.status.updatedReplicas // 0')"
|
||||
observed="$(printf '%s' "$status" | jq -r '.status.observedGeneration // 0')"
|
||||
generation="$(printf '%s' "$status" | jq -r '.metadata.generation // 0')"
|
||||
if [ "$available" -ge "$replicas" ] && [ "$updated" -ge "$replicas" ] && [ "$observed" -ge "$generation" ]; then
|
||||
echo "dev_e2e_rollout=available deployment=$deployment namespace=$ns replicas=$available generation=$generation"
|
||||
return 0
|
||||
fi
|
||||
sleep 2
|
||||
done
|
||||
echo "dev_e2e_rollout=timeout deployment=$deployment namespace=$ns" >&2
|
||||
kube GET "$kube_api/apis/apps/v1/namespaces/$ns/deployments/$deployment" >&2
|
||||
return 1
|
||||
}
|
||||
apply_code_queue_role() {
|
||||
local role="$1"
|
||||
local readiness_path="$2"
|
||||
local scheduler_enabled="$3"
|
||||
local name="code-queue-${role}-dev"
|
||||
cat >/tmp/dev-e2e-code-queue-$role.yaml <<YAML
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: $name
|
||||
namespace: $ns
|
||||
labels:
|
||||
app.kubernetes.io/name: code-queue
|
||||
app.kubernetes.io/component: $role
|
||||
app.kubernetes.io/part-of: unidesk
|
||||
unidesk.ai/ci-run-id: "$(params.run-id)"
|
||||
unidesk.ai/deploy-service-id: code-queue
|
||||
spec:
|
||||
replicas: 1
|
||||
strategy:
|
||||
type: Recreate
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: code-queue
|
||||
app.kubernetes.io/component: $role
|
||||
unidesk.ai/ci-run-id: "$(params.run-id)"
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: code-queue
|
||||
app.kubernetes.io/component: $role
|
||||
app.kubernetes.io/part-of: unidesk
|
||||
unidesk.ai/ci-run-id: "$(params.run-id)"
|
||||
unidesk.ai/node-id: D601
|
||||
unidesk.ai/deploy-service-id: code-queue
|
||||
spec:
|
||||
nodeSelector:
|
||||
unidesk.ai/node-id: D601
|
||||
terminationGracePeriodSeconds: 10
|
||||
containers:
|
||||
- name: code-queue
|
||||
image: "$app_image"
|
||||
imagePullPolicy: Never
|
||||
ports:
|
||||
- name: http
|
||||
containerPort: 4222
|
||||
envFrom:
|
||||
- secretRef:
|
||||
name: code-queue-dev-e2e-env
|
||||
env:
|
||||
- name: HOST
|
||||
value: "0.0.0.0"
|
||||
- name: PORT
|
||||
value: "4222"
|
||||
- name: UNIDESK_ENV
|
||||
value: dev
|
||||
- name: UNIDESK_NAMESPACE
|
||||
value: "$ns"
|
||||
- name: UNIDESK_DATABASE_NAME
|
||||
value: unidesk_ci
|
||||
- name: UNIDESK_DEPLOY_REF
|
||||
value: origin/master:deploy.json#environments.dev
|
||||
- name: UNIDESK_DEPLOY_SERVICE_ID
|
||||
value: code-queue
|
||||
- name: CODE_QUEUE_DEPLOY_COMMIT
|
||||
value: "$code_queue_commit"
|
||||
- name: CODE_QUEUE_DEPLOY_REQUESTED_COMMIT
|
||||
value: "$code_queue_commit"
|
||||
- name: CODE_QUEUE_INSTANCE_ID
|
||||
value: D601-dev-ci-$role
|
||||
- name: CODE_QUEUE_SERVICE_ROLE
|
||||
value: "$role"
|
||||
- name: CODE_QUEUE_SCHEDULER_ENABLED
|
||||
value: "$scheduler_enabled"
|
||||
- name: CODE_QUEUE_MAX_ACTIVE_QUEUES
|
||||
value: "0"
|
||||
- name: CODE_QUEUE_DATABASE_POOL_MAX
|
||||
value: "2"
|
||||
- name: CODE_QUEUE_MAIN_PROVIDER_ID
|
||||
value: D601-dev
|
||||
- name: CODE_QUEUE_EXECUTION_PROVIDER_IDS
|
||||
value: D601-dev
|
||||
- name: CODE_QUEUE_WORKDIR
|
||||
value: /workspace-dev
|
||||
- name: CODE_QUEUE_REMOTE_WORKDIR
|
||||
value: /home/ubuntu/unidesk-dev-workspace
|
||||
- name: CODE_QUEUE_DATA_DIR
|
||||
value: /var/lib/unidesk-dev-e2e/code-queue
|
||||
- name: CODE_QUEUE_CODEX_HOME
|
||||
value: /var/lib/unidesk-dev-e2e/code-queue/codex-home
|
||||
- name: CODE_QUEUE_OPENCODE_XDG_DIR
|
||||
value: /var/lib/unidesk-dev-e2e/code-queue/opencode-xdg
|
||||
- name: CODE_QUEUE_EGRESS_PROXY_ENABLED
|
||||
value: "false"
|
||||
- name: CODE_QUEUE_NOTIFY_CLAUDEQQ_ENABLED
|
||||
value: "false"
|
||||
- name: CODE_QUEUE_STARTUP_OA_BACKFILL_ENABLED
|
||||
value: "false"
|
||||
- name: CODE_QUEUE_CODEX_SQLITE_LOG_EXPORT_ENABLED
|
||||
value: "false"
|
||||
- name: OA_EVENT_FLOW_BASE_URL
|
||||
value: http://127.0.0.1:9
|
||||
- name: LOG_FILE
|
||||
value: /var/log/unidesk/code-queue-$role-dev-e2e.jsonl
|
||||
- name: NODE_OPTIONS
|
||||
value: --max-old-space-size=512
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: $readiness_path
|
||||
port: http
|
||||
periodSeconds: 5
|
||||
timeoutSeconds: 3
|
||||
failureThreshold: 24
|
||||
startupProbe:
|
||||
httpGet:
|
||||
path: /live
|
||||
port: http
|
||||
periodSeconds: 5
|
||||
timeoutSeconds: 3
|
||||
failureThreshold: 60
|
||||
resources:
|
||||
requests:
|
||||
cpu: 80m
|
||||
memory: 192Mi
|
||||
limits:
|
||||
cpu: 300m
|
||||
memory: 768Mi
|
||||
volumeMounts:
|
||||
- name: state
|
||||
mountPath: /var/lib/unidesk-dev-e2e/code-queue
|
||||
- name: logs
|
||||
mountPath: /var/log/unidesk
|
||||
volumes:
|
||||
- name: state
|
||||
emptyDir: {}
|
||||
- name: logs
|
||||
emptyDir: {}
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: $name
|
||||
namespace: $ns
|
||||
labels:
|
||||
app.kubernetes.io/name: code-queue
|
||||
app.kubernetes.io/component: $role
|
||||
app.kubernetes.io/part-of: unidesk
|
||||
spec:
|
||||
type: ClusterIP
|
||||
selector:
|
||||
app.kubernetes.io/name: code-queue
|
||||
app.kubernetes.io/component: $role
|
||||
unidesk.ai/ci-run-id: "$(params.run-id)"
|
||||
ports:
|
||||
- name: http
|
||||
port: 4222
|
||||
targetPort: http
|
||||
YAML
|
||||
csplit -s -f /tmp/dev-e2e-code-queue-$role- /tmp/dev-e2e-code-queue-$role.yaml '/^---$/' '{*}'
|
||||
kube PATCH -H "Content-Type: application/apply-patch+yaml" --data-binary @/tmp/dev-e2e-code-queue-$role-00 "$kube_api/apis/apps/v1/namespaces/$ns/deployments/$name?fieldManager=unidesk-ci&force=true" >/dev/null
|
||||
kube PATCH -H "Content-Type: application/apply-patch+yaml" --data-binary @/tmp/dev-e2e-code-queue-$role-01 "$kube_api/api/v1/namespaces/$ns/services/$name?fieldManager=unidesk-ci&force=true" >/dev/null
|
||||
}
|
||||
cleanup() {
|
||||
if [ "$keep" = "true" ]; then
|
||||
echo "dev_e2e_namespace_retained=$ns"
|
||||
@@ -767,161 +973,181 @@ spec:
|
||||
--data-binary @/tmp/dev-e2e-configmap.yaml \
|
||||
"$kube_api/api/v1/namespaces/$ns/configmaps/desired-manifest?fieldManager=unidesk-ci&force=true" >/dev/null
|
||||
|
||||
cat >/tmp/dev-e2e-target.yaml <<YAML
|
||||
database_url_b64="$(printf '%s' "$database_url" | base64 -w0)"
|
||||
cat >/tmp/dev-e2e-secret.yaml <<YAML
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: code-queue-dev-e2e-env
|
||||
namespace: $ns
|
||||
labels:
|
||||
app.kubernetes.io/name: code-queue
|
||||
app.kubernetes.io/component: ci-dev-e2e
|
||||
app.kubernetes.io/part-of: unidesk
|
||||
type: Opaque
|
||||
data:
|
||||
DATABASE_URL: "$database_url_b64"
|
||||
YAML
|
||||
kube PATCH \
|
||||
-H "Content-Type: application/apply-patch+yaml" \
|
||||
--data-binary @/tmp/dev-e2e-secret.yaml \
|
||||
"$kube_api/api/v1/namespaces/$ns/secrets/code-queue-dev-e2e-env?fieldManager=unidesk-ci&force=true" >/dev/null
|
||||
|
||||
cat >/tmp/dev-e2e-postgres.yaml <<YAML
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: dev-e2e-target
|
||||
name: postgres-dev
|
||||
namespace: $ns
|
||||
labels:
|
||||
app.kubernetes.io/name: unidesk-dev-namespace-e2e
|
||||
app.kubernetes.io/component: smoke-target
|
||||
app.kubernetes.io/name: postgres-dev
|
||||
app.kubernetes.io/component: database
|
||||
app.kubernetes.io/part-of: unidesk
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: unidesk-dev-namespace-e2e
|
||||
app.kubernetes.io/component: smoke-target
|
||||
app.kubernetes.io/name: postgres-dev
|
||||
app.kubernetes.io/component: database
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: unidesk-dev-namespace-e2e
|
||||
app.kubernetes.io/component: smoke-target
|
||||
app.kubernetes.io/name: postgres-dev
|
||||
app.kubernetes.io/component: database
|
||||
app.kubernetes.io/part-of: unidesk
|
||||
spec:
|
||||
nodeSelector:
|
||||
unidesk.ai/node-id: D601
|
||||
terminationGracePeriodSeconds: 5
|
||||
containers:
|
||||
- name: smoke-target
|
||||
image: "$(params.app-image)"
|
||||
- name: postgres
|
||||
image: postgres:16-alpine
|
||||
imagePullPolicy: IfNotPresent
|
||||
command:
|
||||
- bun
|
||||
- -e
|
||||
- |
|
||||
const port = Number(process.env.PORT || 8080);
|
||||
const payload = {
|
||||
ok: true,
|
||||
environment: "dev",
|
||||
namespace: process.env.CI_E2E_NAMESPACE,
|
||||
deployCommit: process.env.CI_E2E_DEPLOY_COMMIT,
|
||||
backendCoreCommit: process.env.BACKEND_CORE_COMMIT,
|
||||
frontendCommit: process.env.FRONTEND_COMMIT,
|
||||
codeQueueCommit: process.env.CODE_QUEUE_COMMIT
|
||||
};
|
||||
Bun.serve({
|
||||
hostname: "0.0.0.0",
|
||||
port,
|
||||
fetch(req) {
|
||||
const url = new URL(req.url);
|
||||
if (url.pathname === "/health" || url.pathname === "/") {
|
||||
return Response.json(payload);
|
||||
}
|
||||
return new Response("not found", { status: 404 });
|
||||
}
|
||||
});
|
||||
console.log(JSON.stringify({ listening: port, ...payload }));
|
||||
await new Promise(() => {});
|
||||
ports:
|
||||
- name: http
|
||||
containerPort: 8080
|
||||
- name: postgres
|
||||
containerPort: 5432
|
||||
env:
|
||||
- name: PORT
|
||||
value: "8080"
|
||||
- name: CI_E2E_NAMESPACE
|
||||
value: "$ns"
|
||||
- name: CI_E2E_DEPLOY_COMMIT
|
||||
value: "$(params.deploy-commit)"
|
||||
- name: BACKEND_CORE_COMMIT
|
||||
value: "$backend_commit"
|
||||
- name: FRONTEND_COMMIT
|
||||
value: "$frontend_commit"
|
||||
- name: CODE_QUEUE_COMMIT
|
||||
value: "$code_queue_commit"
|
||||
- name: POSTGRES_USER
|
||||
value: unidesk_ci
|
||||
- name: POSTGRES_PASSWORD
|
||||
value: unidesk_ci_password
|
||||
- name: POSTGRES_DB
|
||||
value: unidesk_ci
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /health
|
||||
port: http
|
||||
periodSeconds: 3
|
||||
timeoutSeconds: 2
|
||||
failureThreshold: 20
|
||||
exec:
|
||||
command:
|
||||
- pg_isready
|
||||
- -U
|
||||
- unidesk_ci
|
||||
- -d
|
||||
- unidesk_ci
|
||||
periodSeconds: 5
|
||||
timeoutSeconds: 3
|
||||
failureThreshold: 24
|
||||
resources:
|
||||
requests:
|
||||
cpu: 20m
|
||||
memory: 64Mi
|
||||
cpu: 50m
|
||||
memory: 128Mi
|
||||
limits:
|
||||
memory: 256Mi
|
||||
cpu: 250m
|
||||
memory: 512Mi
|
||||
volumeMounts:
|
||||
- name: data
|
||||
mountPath: /var/lib/postgresql/data
|
||||
volumes:
|
||||
- name: data
|
||||
emptyDir: {}
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: dev-e2e-target
|
||||
name: postgres-dev
|
||||
namespace: $ns
|
||||
labels:
|
||||
app.kubernetes.io/name: unidesk-dev-namespace-e2e
|
||||
app.kubernetes.io/component: smoke-target
|
||||
app.kubernetes.io/name: postgres-dev
|
||||
app.kubernetes.io/component: database
|
||||
app.kubernetes.io/part-of: unidesk
|
||||
spec:
|
||||
type: ClusterIP
|
||||
selector:
|
||||
app.kubernetes.io/name: unidesk-dev-namespace-e2e
|
||||
app.kubernetes.io/component: smoke-target
|
||||
app.kubernetes.io/name: postgres-dev
|
||||
app.kubernetes.io/component: database
|
||||
ports:
|
||||
- name: http
|
||||
port: 8080
|
||||
targetPort: http
|
||||
- name: postgres
|
||||
port: 5432
|
||||
targetPort: postgres
|
||||
YAML
|
||||
csplit -s -f /tmp/dev-e2e-target- /tmp/dev-e2e-target.yaml '/^---$/' '{*}'
|
||||
csplit -s -f /tmp/dev-e2e-postgres- /tmp/dev-e2e-postgres.yaml '/^---$/' '{*}'
|
||||
kube PATCH \
|
||||
-H "Content-Type: application/apply-patch+yaml" \
|
||||
--data-binary @/tmp/dev-e2e-target-00 \
|
||||
"$kube_api/apis/apps/v1/namespaces/$ns/deployments/dev-e2e-target?fieldManager=unidesk-ci&force=true" >/dev/null
|
||||
--data-binary @/tmp/dev-e2e-postgres-00 \
|
||||
"$kube_api/apis/apps/v1/namespaces/$ns/deployments/postgres-dev?fieldManager=unidesk-ci&force=true" >/dev/null
|
||||
kube PATCH \
|
||||
-H "Content-Type: application/apply-patch+yaml" \
|
||||
--data-binary @/tmp/dev-e2e-target-01 \
|
||||
"$kube_api/api/v1/namespaces/$ns/services/dev-e2e-target?fieldManager=unidesk-ci&force=true" >/dev/null
|
||||
--data-binary @/tmp/dev-e2e-postgres-01 \
|
||||
"$kube_api/api/v1/namespaces/$ns/services/postgres-dev?fieldManager=unidesk-ci&force=true" >/dev/null
|
||||
wait_deployment_available postgres-dev 180
|
||||
|
||||
deadline=$((SECONDS + 180))
|
||||
while [ "$SECONDS" -lt "$deadline" ]; do
|
||||
status="$(kube GET "$kube_api/apis/apps/v1/namespaces/$ns/deployments/dev-e2e-target")"
|
||||
available="$(printf '%s' "$status" | jq -r '.status.availableReplicas // 0')"
|
||||
observed="$(printf '%s' "$status" | jq -r '.status.observedGeneration // 0')"
|
||||
generation="$(printf '%s' "$status" | jq -r '.metadata.generation // 0')"
|
||||
if [ "$available" -ge 1 ] && [ "$observed" -ge "$generation" ]; then
|
||||
echo "dev_e2e_target_rollout=available namespace=$ns"
|
||||
break
|
||||
fi
|
||||
sleep 2
|
||||
done
|
||||
if [ "$SECONDS" -ge "$deadline" ]; then
|
||||
echo "dev_e2e_target_rollout=timeout namespace=$ns" >&2
|
||||
kube GET "$kube_api/apis/apps/v1/namespaces/$ns/deployments/dev-e2e-target" >&2
|
||||
exit 1
|
||||
fi
|
||||
apply_code_queue_role scheduler /health true
|
||||
apply_code_queue_role read /live false
|
||||
apply_code_queue_role write /health false
|
||||
wait_deployment_available code-queue-scheduler-dev 420
|
||||
wait_deployment_available code-queue-read-dev 420
|
||||
wait_deployment_available code-queue-write-dev 420
|
||||
|
||||
bun - "$ns" "$(params.deploy-commit)" "$backend_commit" "$frontend_commit" "$code_queue_commit" "$result_json" <<'BUN'
|
||||
const [ns, deployCommit, backendCommit, frontendCommit, codeQueueCommit, resultPath] = process.argv.slice(2);
|
||||
const url = `http://dev-e2e-target.${ns}.svc.cluster.local:8080/health`;
|
||||
const started = performance.now();
|
||||
const response = await fetch(url);
|
||||
const elapsedMs = Math.round(performance.now() - started);
|
||||
const body = await response.json();
|
||||
proxy_get code-queue-write-dev /health >"$run_dir/code-queue-write-health.json"
|
||||
proxy_get code-queue-scheduler-dev /health >"$run_dir/code-queue-scheduler-health.json"
|
||||
proxy_get code-queue-read-dev /live >"$run_dir/code-queue-read-live.json"
|
||||
proxy_get code-queue-read-dev /api/workdirs >"$run_dir/code-queue-read-workdirs.json"
|
||||
proxy_json POST code-queue-write-dev /api/workdirs '{"providerId":"D601-dev","executionMode":"default","path":"/home/ubuntu/unidesk-dev-workspace/ci-workdirs-smoke"}' >"$run_dir/code-queue-workdir-created.json"
|
||||
proxy_get code-queue-write-dev '/api/workdirs?providerId=D601-dev&executionMode=default' >"$run_dir/code-queue-write-workdirs.json"
|
||||
proxy_json DELETE code-queue-write-dev /api/workdirs/D601-dev/default/%2Fhome%2Fubuntu%2Funidesk-dev-workspace%2Fci-workdirs-smoke >"$run_dir/code-queue-workdir-deleted.json"
|
||||
|
||||
bun - "$ns" "$(params.deploy-commit)" "$backend_commit" "$frontend_commit" "$code_queue_commit" "$app_image" "$result_json" "$run_dir" <<'BUN'
|
||||
const [ns, deployCommit, backendCommit, frontendCommit, codeQueueCommit, appImage, resultPath, runDir] = process.argv.slice(2);
|
||||
const runBase = new URL(`file://${runDir.replace(/\/+$/u, "")}/`);
|
||||
async function readJson(name) {
|
||||
return await Bun.file(new URL(name, runBase)).json();
|
||||
}
|
||||
const health = await readJson("code-queue-write-health.json");
|
||||
const scheduler = await readJson("code-queue-scheduler-health.json");
|
||||
const readLive = await readJson("code-queue-read-live.json");
|
||||
const initialWorkdirs = await readJson("code-queue-read-workdirs.json");
|
||||
const created = await readJson("code-queue-workdir-created.json");
|
||||
const listed = await readJson("code-queue-write-workdirs.json");
|
||||
const deleted = await readJson("code-queue-workdir-deleted.json");
|
||||
const checks = [
|
||||
response.ok,
|
||||
body.ok === true,
|
||||
body.environment === "dev",
|
||||
body.namespace === ns,
|
||||
body.deployCommit === deployCommit,
|
||||
body.backendCoreCommit === backendCommit,
|
||||
body.frontendCommit === frontendCommit,
|
||||
body.codeQueueCommit === codeQueueCommit
|
||||
health?.ok === true && health?.role === "write" && health?.deploy?.commit === codeQueueCommit,
|
||||
scheduler?.ok === true && scheduler?.role === "scheduler" && scheduler?.schedulerEnabled === true,
|
||||
readLive?.ok === true && readLive?.role === "read",
|
||||
initialWorkdirs?.ok === true && Array.isArray(initialWorkdirs?.workdirs),
|
||||
created?.ok === true && created?.workdir?.providerId === "D601-dev" && created?.workdir?.path === "/home/ubuntu/unidesk-dev-workspace/ci-workdirs-smoke",
|
||||
Array.isArray(listed?.workdirs) && listed.workdirs.some((item) => item?.path === "/home/ubuntu/unidesk-dev-workspace/ci-workdirs-smoke"),
|
||||
deleted?.ok === true,
|
||||
];
|
||||
const result = { ok: checks.every(Boolean), elapsedMs, url, body };
|
||||
await Bun.write(resultPath, JSON.stringify(result, null, 2) + "\n");
|
||||
const result = {
|
||||
ok: checks.every(Boolean),
|
||||
namespace: ns,
|
||||
deployCommit,
|
||||
backendCoreCommit: backendCommit,
|
||||
frontendCommit,
|
||||
codeQueueCommit,
|
||||
codeQueueImage: appImage,
|
||||
accessPath: "kubernetes-api-service-proxy",
|
||||
services: ["code-queue-scheduler-dev", "code-queue-read-dev", "code-queue-write-dev"],
|
||||
health,
|
||||
scheduler,
|
||||
readLive,
|
||||
initialWorkdirs,
|
||||
created,
|
||||
listed,
|
||||
deleted,
|
||||
};
|
||||
await Bun.write(resultPath, `${JSON.stringify(result, null, 2)}\n`);
|
||||
console.log(JSON.stringify(result));
|
||||
if (!result.ok) process.exit(1);
|
||||
if (!result.ok) {
|
||||
process.exit(1);
|
||||
}
|
||||
BUN
|
||||
---
|
||||
apiVersion: tekton.dev/v1
|
||||
|
||||
Reference in New Issue
Block a user