diff --git a/docs/reference/cicd-standardization.md b/docs/reference/cicd-standardization.md index 39f62ee8..8647712c 100644 --- a/docs/reference/cicd-standardization.md +++ b/docs/reference/cicd-standardization.md @@ -16,6 +16,43 @@ The standard release shape is: `backend-core` and D601 `code-queue` may be validated only in dev in this phase. This document must not be used to introduce production deploy validation for either service. +## Phase-One Deploy.json Consolidation Contract + +Issue #60 moves CI/CD image deployment toward one release-intent source. The implemented `deploy.json` schema still carries only `id`, `repo` and `commitId`; this section defines the phase-one migration contract before changing runtime behavior. + +Current duplicated configuration surfaces are: + +- Service source and version: `deploy.json.environments.*.services[]`, `config.json.microservices[].repository`, `CI.json.artifacts[].source`, runtime `UNIDESK_DEPLOY_*` metadata and service health payloads. +- Image repository, tag and digest: `CI.json.artifacts[].image`, artifact-registry consumer specs, Compose/Kubernetes image fields, registry digest probes and docs tables. +- Runtime target metadata: `config.json` provider/deployment entries, Compose service/container names, k3s namespaces, Kubernetes Deployment/Service names, artifact-registry target specs and runtime manifests. +- Deployment parameters: runtime env prefixes, required secret keys, health deploy metadata requirements, resource requests/limits, rollout strategy and dev/prod support gates. +- Consumer scope: `deploy.json` dev/prod service lists, deploy/artifact-registry allowlists, dev/prod Kubernetes manifests and service-specific documentation. + +Phase one extends the desired-state model in this order: + +1. Keep existing `deploy.json` `id`, `repo` and `commitId` as the deployed source/version authority. All environment-ref deploy commands continue reading `origin/master:deploy.json`. +2. Add optional deploy-owned artifact identity fields after the static drift guard is green: `artifact.kind`, `artifact.repository`, `artifact.tag`, optional `artifact.digestRef`, and `artifact.upstreamDigestRef` for upstream-image services. For source-build services, `artifact.tag` must equal the selected commit unless a future schema explicitly records a digest-only release. +3. Add consumer intent fields: `consumer.kind`, `consumer.dev.enabled`, `consumer.prod.enabled`, `consumer.supportLevel`, `consumer.targetRef` and `consumer.noRuntimeSourceBuild`. These describe whether the service is `main-server-compose`, `d601-k3s-managed`, `d601-direct-compose`, `upstream-digest`, `dry-run-only` or `unsupported`; low-level object names still come from config/manifests until renderers exist. +4. Add deployment-time metadata that must be common across CI, dev CD and prod CD: deploy env prefix, health deploy-metadata requirement, required runtime secret key names, rollback policy, and resource profile identifiers. The planned first keys are `health.deployMetadataRequired` and `runtime.requiredSecretKeys`. Secret values, credentials, volumes and host paths do not move into `deploy.json`. +5. Teach CI producer dry-run, deploy plan and artifact-registry dry-run to render from `deploy.json` first and compare mirrored `CI.json`/`config.json`/manifest values as derived copies. A mismatch is drift, not an alternate source of truth. + +Fields that stay outside `deploy.json` during phase one: + +- `CI.json`: producer pipeline name, source root, Dockerfile path and success summary shape. Once artifact identity is in `deploy.json`, `CI.json.artifacts[].image.repository` becomes a compatibility mirror checked for drift. +- `config.json`: provider id, proxy route and port policy, backend health path, Compose file/service/container names, development SSH/worktree details, and secret source locations. +- Compose and Kubernetes manifests: concrete container specs, volumes, PVCs, security context, rollout strategy and raw resource requests/limits until a renderer owns those files. +- Artifact-registry executor code: low-level pull/import/retag/recreate commands, registry probes and platform-specific verification scripts. The executor must consume the normalized deploy plan instead of hardcoding release intent. + +The drift contract is: + +- `deploy.json` wins for release intent. `config.json.microservices[].repository.commitId` is compatibility data only and must not drive env-ref deploys. +- `deploy.json.repo` and `CI.json.artifacts[].source.repo` must match for every source-build service that appears in `deploy.json`. +- Image tags used by CI/CD must be commit-pinned or digest-pinned. Mutable tags such as `latest` are never valid release evidence. +- Runtime topology may be mirrored in dry-run output, but deployment commands must treat it as derived target metadata and keep `noRuntimeSourceBuild=true` for reviewed artifact consumers. +- Upstream-image services must stay `blocked` in CI source-build publishing until an upstream digest or mirror digest consumer exists. + +Lightweight evidence for this contract is `bun scripts/issue-60-cicd-drift-contract-test.ts`. The test parses local JSON/docs only; it does not publish images, deploy services, restart containers, run Playwright or run full e2e. + ## Artifact Catalog Root `CI.json` is the CI producer catalog. It is not a deployment manifest. @@ -127,6 +164,7 @@ Minimum evidence for this lane is: | k3s control bridge dry-run | `bun scripts/cli.ts deploy apply --env prod --service k3sctl-adapter --dry-run` | | CI producer preflight | `bun scripts/cli.ts ci publish-user-service --service --commit --dry-run` | | Decision Center desired/live no-build drift guard | `bun scripts/decision-center-desired-state-contract-test.ts` | +| Issue #60 phase-one deploy.json drift guard | `bun scripts/issue-60-cicd-drift-contract-test.ts` | ### Upstream Image Evidence diff --git a/scripts/issue-60-cicd-drift-contract-test.ts b/scripts/issue-60-cicd-drift-contract-test.ts new file mode 100644 index 00000000..a179c56b --- /dev/null +++ b/scripts/issue-60-cicd-drift-contract-test.ts @@ -0,0 +1,241 @@ +import { readFileSync } from "node:fs"; +import { rootPath } from "./src/config"; + +type JsonRecord = Record; +type Environment = "dev" | "prod"; + +const reviewedSourceBuildServices = new Set([ + "backend-core", + "frontend", + "baidu-netdisk", + "decision-center", + "project-manager", + "oa-event-flow", + "todo-note", + "code-queue-mgr", + "findjob", + "pipeline", + "met-nonlinear", + "k3sctl-adapter", + "mdtodo", + "claudeqq", + "code-queue", +]); + +const reviewedArtifactConsumers = new Set([ + "backend-core", + "frontend", + "baidu-netdisk", + "decision-center", + "project-manager", + "oa-event-flow", + "todo-note", + "code-queue-mgr", + "findjob", + "pipeline", + "met-nonlinear", + "k3sctl-adapter", + "mdtodo", + "claudeqq", + "code-queue", +]); + +const plannedDeployJsonFields = [ + "artifact.kind", + "artifact.repository", + "artifact.tag", + "artifact.digestRef", + "consumer.kind", + "consumer.dev.enabled", + "consumer.prod.enabled", + "consumer.supportLevel", + "consumer.noRuntimeSourceBuild", + "health.deployMetadataRequired", + "runtime.requiredSecretKeys", +]; + +function assertCondition(condition: unknown, message: string, detail: unknown = {}): void { + if (!condition) throw new Error(`${message}: ${JSON.stringify(detail)}`); +} + +function asRecord(value: unknown, label: string): JsonRecord { + assertCondition(typeof value === "object" && value !== null && !Array.isArray(value), `${label} must be an object`, value); + return value as JsonRecord; +} + +function asArray(value: unknown, label: string): unknown[] { + assertCondition(Array.isArray(value), `${label} must be an array`, value); + return value as unknown[]; +} + +function stringField(value: unknown, label: string): string { + assertCondition(typeof value === "string" && value.length > 0, `${label} must be a non-empty string`, value); + return value as string; +} + +function loadJson(path: string): JsonRecord { + return asRecord(JSON.parse(readFileSync(rootPath(path), "utf8")) as unknown, path); +} + +function isFullSha(value: string): boolean { + return /^[0-9a-f]{40}$/u.test(value); +} + +function deployServicesByEnvironment(deploy: JsonRecord, environment: Environment): JsonRecord[] { + const environments = asRecord(deploy.environments, "deploy.json.environments"); + const env = asRecord(environments[environment], `deploy.json.environments.${environment}`); + return asArray(env.services, `deploy.json.environments.${environment}.services`).map((item, index) => asRecord(item, `${environment}.services[${index}]`)); +} + +function serviceKey(environment: Environment, serviceId: string): string { + return `${environment}/${serviceId}`; +} + +function artifactByService(ciCatalog: JsonRecord): Map { + const artifacts = asArray(ciCatalog.artifacts, "CI.json.artifacts").map((item, index) => asRecord(item, `CI.json.artifacts[${index}]`)); + return new Map(artifacts.map((artifact) => [stringField(artifact.serviceId, "CI.json artifact serviceId"), artifact])); +} + +function configByService(config: JsonRecord): Map { + const microservices = asArray(config.microservices, "config.json.microservices").map((item, index) => asRecord(item, `config.json.microservices[${index}]`)); + return new Map(microservices.map((service) => [stringField(service.id, "config service id"), service])); +} + +function assertDeployJsonReleaseIntent(deploy: JsonRecord): Array<{ environment: Environment; serviceId: string; repo: string; commitId: string }> { + assertCondition(deploy.schemaVersion === 2, "deploy.json must remain schemaVersion=2", deploy); + const seen = new Set(); + const services: Array<{ environment: Environment; serviceId: string; repo: string; commitId: string }> = []; + for (const environment of ["dev", "prod"] as const) { + for (const service of deployServicesByEnvironment(deploy, environment)) { + const keys = Object.keys(service).sort(); + assertCondition(keys.join(",") === "commitId,id,repo", "phase-one deploy.json service entries must remain only id/repo/commitId until schema migration lands", { environment, service, keys }); + const serviceId = stringField(service.id, `${environment} service id`); + const repo = stringField(service.repo, `${environment}/${serviceId}.repo`); + const commitId = stringField(service.commitId, `${environment}/${serviceId}.commitId`).toLowerCase(); + assertCondition(isFullSha(commitId), "deploy.json phase-one commit pins must be full 40-character SHAs", { environment, serviceId, commitId }); + const key = serviceKey(environment, serviceId); + assertCondition(!seen.has(key), "deploy.json service must not be duplicated within an environment", { key }); + seen.add(key); + services.push({ environment, serviceId, repo, commitId }); + } + } + return services; +} + +function assertCiCatalogAlignment(services: Array<{ environment: Environment; serviceId: string; repo: string; commitId: string }>, ciCatalog: JsonRecord): void { + const artifacts = artifactByService(ciCatalog); + const defaults = asRecord(ciCatalog.defaults, "CI.json.defaults"); + assertCondition(defaults.mutableTagsAllowed === false, "CI.json must continue rejecting mutable tags", defaults); + assertCondition(defaults.tagTemplate === "{{sourceCommit}}", "CI.json tag template must remain commit-pinned", defaults); + + for (const service of services) { + const artifact = artifacts.get(service.serviceId); + assertCondition(artifact !== undefined, "every deploy.json service must have a CI catalog entry or a reviewed exception", service); + assertCondition(reviewedSourceBuildServices.has(service.serviceId), "deploy.json service must be in the reviewed source-build/drift set for phase one", service); + const kind = stringField(artifact?.kind, `${service.serviceId}.kind`); + const status = stringField(artifact?.status, `${service.serviceId}.status`); + assertCondition(kind === "source-build", "deploy.json source services must stay source-build in CI.json during phase one", { service, artifact }); + assertCondition(status === "supported", "deploy.json source services must stay supported in CI.json during phase one", { service, artifact }); + const source = asRecord(artifact?.source, `${service.serviceId}.source`); + const image = asRecord(artifact?.image, `${service.serviceId}.image`); + assertCondition(source.repo === service.repo, "deploy.json repo and CI.json source.repo must not drift", { service, source }); + assertCondition(typeof source.dockerfile === "string" && source.dockerfile.length > 0, "CI.json must keep the producer Dockerfile until deploy.json owns artifact identity", { service, source }); + assertCondition(image.repository === `unidesk/${service.serviceId}`, "CI image repository must remain service-id derived before deploy.json artifact.repository migration", { service, image }); + } +} + +function assertConfigCompatibility(services: Array<{ environment: Environment; serviceId: string; repo: string; commitId: string }>, config: JsonRecord): Array<{ environment: Environment; serviceId: string; deployCommitId: string; configCommitId: string }> { + const configs = configByService(config); + const commitMirrors: Array<{ environment: Environment; serviceId: string; deployCommitId: string; configCommitId: string }> = []; + for (const service of services) { + if (service.serviceId === "backend-core" || service.serviceId === "frontend") continue; + const configService = configs.get(service.serviceId); + assertCondition(configService !== undefined, "deploy.json service must exist in config.json until renderers replace config lookups", service); + const repository = asRecord(configService?.repository, `${service.serviceId}.repository`); + assertCondition(repository.url === service.repo, "deploy.json repo and config.json repository.url must not drift", { service, repository }); + assertCondition(typeof repository.dockerfile === "string" && repository.dockerfile.length > 0, "config.json keeps Dockerfile target metadata in phase one", { service, repository }); + assertCondition(typeof repository.composeService === "string" && repository.composeService.length > 0, "config.json keeps compose/k3s service name in phase one", { service, repository }); + assertCondition(typeof repository.containerName === "string" && repository.containerName.length > 0, "config.json keeps container name in phase one", { service, repository }); + const configCommitId = stringField(repository.commitId, `${service.serviceId}.repository.commitId`); + if (configCommitId === service.commitId) { + commitMirrors.push({ environment: service.environment, serviceId: service.serviceId, deployCommitId: service.commitId, configCommitId }); + } + } + return commitMirrors; +} + +function assertConsumerCoverage(services: Array<{ environment: Environment; serviceId: string; repo: string; commitId: string }>): void { + for (const service of services) { + assertCondition(reviewedArtifactConsumers.has(service.serviceId), "deploy.json service must be covered by the phase-one consumer drift set", service); + if (service.environment === "prod" && service.serviceId === "code-queue") { + continue; + } + if (service.serviceId === "k3sctl-adapter" || service.serviceId === "met-nonlinear") { + continue; + } + assertCondition( + service.serviceId !== "code-agent-sandbox", + "code-agent-sandbox must not become a deploy.json artifact consumer before the schema contract is reviewed", + service, + ); + } +} + +function assertUpstreamImageBoundary(ciCatalog: JsonRecord, config: JsonRecord): void { + const artifacts = artifactByService(ciCatalog); + const configs = configByService(config); + for (const serviceId of ["filebrowser", "filebrowser-d601"]) { + const artifact = asRecord(artifacts.get(serviceId), `CI.json ${serviceId}`); + const upstream = asRecord(artifact.upstream, `CI.json ${serviceId}.upstream`); + const configService = asRecord(configs.get(serviceId), `config.json ${serviceId}`); + const repository = asRecord(configService.repository, `config.json ${serviceId}.repository`); + const artifactSource = asRecord(repository.artifactSource, `config.json ${serviceId}.repository.artifactSource`); + assertCondition(artifact.kind === "upstream-image", "File Browser services must stay upstream-image catalog entries", artifact); + assertCondition(artifact.status === "blocked", "File Browser upstream-image services must stay blocked until mirror consumer exists", artifact); + assertCondition(upstream.imageRef === artifactSource.imageRef, "CI upstream imageRef and config artifactSource imageRef must not drift", { serviceId, upstream, artifactSource }); + assertCondition(artifactSource.ciDockerfileBuild === false, "File Browser must not be treated as a CI Dockerfile build", { serviceId, artifactSource }); + assertCondition(artifactSource.pullOnlyCd === true, "File Browser must remain pull-only CD", { serviceId, artifactSource }); + assertCondition(typeof upstream.digestRef === "string" && String(upstream.digestRef).includes("@sha256:"), "upstream image entry must keep a digest pin", { serviceId, upstream }); + } +} + +function assertDocsContract(): void { + const docs = readFileSync(rootPath("docs/reference/cicd-standardization.md"), "utf8"); + for (const phrase of [ + "Phase-One Deploy.json Consolidation Contract", + "Current duplicated configuration surfaces", + "Fields that stay outside `deploy.json` during phase one", + "The drift contract is:", + "bun scripts/issue-60-cicd-drift-contract-test.ts", + ]) { + assertCondition(docs.includes(phrase), "CI/CD standardization docs must include the issue #60 phase-one contract", { phrase }); + } + for (const field of plannedDeployJsonFields) { + assertCondition(docs.includes(field), "phase-one docs must name the planned deploy.json schema field", { field }); + } +} + +const deploy = loadJson("deploy.json"); +const ciCatalog = loadJson("CI.json"); +const config = loadJson("config.json"); +const services = assertDeployJsonReleaseIntent(deploy); +assertCiCatalogAlignment(services, ciCatalog); +const configCommitMirrors = assertConfigCompatibility(services, config); +assertConsumerCoverage(services); +assertUpstreamImageBoundary(ciCatalog, config); +assertDocsContract(); + +process.stdout.write(`${JSON.stringify({ + ok: true, + checks: [ + "deploy.json remains the phase-one release-intent source with full commit pins", + "deploy.json repo values match CI.json source-build producer repos", + "CI.json keeps commit-tag producer policy and service-id image repositories", + "config.json runtime topology remains compatibility metadata and does not own env-ref commits", + "File Browser upstream-image entries stay blocked from Dockerfile CI", + "docs/reference/cicd-standardization.md names the phase-one schema and drift contract", + ], + deployServicesChecked: services.length, + compatibilityCommitMirrors: configCommitMirrors, + plannedDeployJsonFields, +}, null, 2)}\n`);