diff --git a/docs/reference/cli.md b/docs/reference/cli.md index 31471a8b..d3f58889 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -34,7 +34,7 @@ CI/CD、GitOps、rollout、artifact 发布、PR 合并后的 DEV/PROD 滚动、P - `provider attach [--master-server URL] [--up] [--force]` 在新计算节点生成两项配置的 provider-gateway 挂载包：`.state/provider-.env` 默认只包含 `UNIDESK_MASTER_SERVER` 与 `PROVIDER_ID`，`provider-.yml` 固定 Docker socket、`pid: "host"`、`restart: always`、只读 `/workspace` 和 SSH 维护私钥挂载；`--up` 会立即执行生成的 `docker compose up -d --build`。`provider triage [--observed-error text] [--observed-scope scope] [--microservice id ...] [--full|--raw]` 是只读多信号健康裁决入口，会把单路径 `provider is not online`、SSH 超时、registry 失败和 service proxy 失败归类成 `runner-local-observation-gap`、`service-degraded`、`provider-degraded` 或 `global-blocker`。默认输出只返回裁决、scope、失败/降级/未知信号和有界 evidence 摘要，完整 evidence 必须显式加 `--full` 或 `--raw`；推荐交叉验证命令仍包含 `debug health`、`debug dispatch host.ssh --wait-ms 15000`、`trans argv true`、`artifact-registry health --provider-id `、`microservice health k3sctl-adapter`、`microservice health code-queue` 和 `codex tasks --view supervisor --limit 20`。 - `trans [operation args...]` / `tran [operation args...]` 通过 backend-core 内网 WebSocket broker 和 provider-gateway 的 Host SSH / WSL SSH 维护桥连接目标节点；`route` 基础形态是 provider id，例如 `D601` 或 `G14`，也可以扩展为纯定位路径 `provider:plane[:namespace:resource[:container]]`，例如 `D601:win`、`D601:win/c/test`、`G14:k3s`、`D601:k3s` 或 `G14:k3s::`。WSL provider 的 Windows plane 固定使用 `win`，不得使用 `win32`；Windows operation 必须显式区分：`ps` 执行 Windows PowerShell heredoc 或一行 PowerShell 命令，`cmd` 执行 cmd.exe/batch，`skills` 发现 Windows skill 目录。需要 Windows cwd 时用 `trans D601:win/c/test ps` 或 `trans D601:win/c/test cmd cd`，CLI 自动设置 UTF-8/Python 编码默认值；`cmd` 额外设置 `chcp 65001`。非交互远端命令优先使用 `trans argv ...`；需要 POSIX shell 脚本、管道、变量或循环时优先使用 quoted heredoc 单步传输，例如 `trans G14 script <<'SCRIPT'`、`trans G14:k3s script <<'SCRIPT'` 或 `trans G14:k3s:: script <<'SCRIPT'`，把脚本走 stdin。`script` 只表示 host/k3s POSIX shell，不表示 Windows PowerShell；Windows PowerShell 必须写 `trans :win ps <<'PS'`。`script -- '<单个字符串>'` 是无需 stdin 的远端 POSIX shell one-liner，例如 `trans G14:/root/hwlab script -- 'cd /root/hwlab && git status --short --branch'`；`script -- <多个 argv>` 才是 direct argv，适合 `trans D601:/path script -- sed -n '1,20p' file` 这类带短横线的单进程命令。顶层 remote option parser 必须保留命令已经开始后的 `--`，不得把它吞成全局选项结束符。需要远端改文本文件时默认优先使用 ` apply-patch < patch.diff`；需要可靠传输非文本或整文件时使用 ` upload ` 和 ` download `，CLI 会按字节数与 SHA-256 自动校验并在 provider-gateway stdin/argv 限制下切换客户端分块策略；需要旧 helper 时显式使用 `:k3s:: apply-patch-v1` 或 ` apply-patch-v1`。ssh-like 命令遇到 timeout/kex/255 类失败时，CLI 会在 stderr 追加一行 `UNIDESK_SSH_HINT` JSON，提示 stdin script/argv 重试和 provider triage 交叉验证。 - `trans apply-patch < patch.diff` 是默认推荐的远端 patch 入口：本地 TypeScript line-based engine 解析和计算新文件内容，远端 route 只负责读写文件；支持 host workspace、k3s pod workspace、Windows workspace route（例如 `D601:win/c/test`）和 frontend transport，并优先处理长中文/Unicode、低上下文插入、重复块 `@@` 定位等旧 helper 容易失败的场景。`apply-patch` 输出按 Codex 标准文本口径，不套 UniDesk JSON 限制：成功 stdout 为 `Success. Updated the following files:`，失败 stdout 为空、stderr 写失败原因；多文件补丁中途失败时，stderr 只列出第一个失败前已成功执行的 hunk 和失败 hunk，随后按 Codex 语义停止，不继续尝试后续 hunk。v2 兼容常见 MiniMax/MXCX 非标准补丁输入，例如重复 nested `*** Begin Patch` / `*** End Patch` envelope、unified-diff hunk header、Add/Delete 误加 `@@`、Update context 漏掉前导空格，并在 stderr 给出 canonical 写法 hint；parser 或上下文失败时仍坚持唯一 v2 引擎，只提示修正 patch 文本或 hunk context，不自动重试或切换到 `apply-patch-v1`；大块/函数替换因上下文过期失败时，正确动作是重新读取当前目标块、缩小或拆分 Update File hunk 后继续用 `apply-patch`，不得改走 `download`/`upload`、远端 Python/Perl/sed heredoc 或整文件重写。Windows route 复用同一套 v2 核心算法，只把底层读写替换成 PowerShell 文件系统接口；`trans apply-patch-v1 [tool args...] < patch.diff` 保留为显式 legacy 入口，直接调用远端注入的 `apply_patch` sh/perl helper；默认 `apply-patch` 不把 v1 当 fallback。 -- `apply-patch` v2 每次结束都会在 stderr 追加一行 `UNIDESK_APPLY_PATCH_TIMING {json}`，字段包含 `durationMs`、`patchBytes`、`fileCount`、`hunkCount`、`changedCount`、`remoteOperationCount`、`remoteOperationCounts`、`remoteElapsedMs`、`remoteFailureCount`、`providerId`、`route` 和 `transport`（可得时）。普通 POSIX host/k3s 远端的多文件 `Update File` patch 会优先合并成 bulk read/write，避免每个文件单独 stat/read/write 的 SSH 往返；Windows route 和 Add/Delete/Move 等复杂 patch 保持原有逐步语义。timing 摘要只用于定位慢在 patch 解析、远端 stat/read/write 或 bulk read/write、provider session 还是传输层，不能替代 Codex 标准 stdout/stderr 成功失败文本，也不是门禁或自动判断。 +- `apply-patch` v2 每次结束都会在 stderr 追加一行 `UNIDESK_APPLY_PATCH_TIMING {json}`，字段包含 `durationMs`、`patchBytes`、`fileCount`、`hunkCount`、`changedCount`、`remoteOperationCount`、`remoteOperationCounts`、`remoteElapsedMs`、`remoteFailureCount`、`providerId`、`route` 和 `transport`（可得时）。普通 POSIX host/k3s 和 Windows workspace 远端的多文件 `Update File` patch 会优先合并成 bulk read/write，避免每个文件单独 stat/read/write 的 SSH 往返；Add/Delete/Move 等复杂 patch 保持原有逐步语义。timing 摘要只用于定位慢在 patch 解析、远端 stat/read/write 或 bulk read/write、provider session 还是传输层，不能替代 Codex 标准 stdout/stderr 成功失败文本，也不是门禁或自动判断。 - `trans py [script-args...] < script.py` 把本地 stdin 落到远端临时 `.py` 文件后再以 `python3 -u` 执行并自动清理，避免再手写 `'python3 -'`、heredoc 或多层引号；`script-args` 会按 argv 安全透传给远端脚本。 - `trans skills [--scope all|wsl|windows] [--limit N]` 发现目标节点上的 WSL/Linux skill 根目录；当 provider 是 WSL 时同一次调用还会扫描 Windows 用户目录下的 `.agents/skills` 与 `.codex/skills`。 - `trans :k3s[:namespace:workload[:container]] ...` 是原生 k3s 结构化 route 入口，route 只定位控制面或 workload，`kubectl`、`logs`、`exec`、`script`、`apply-patch`、旧 `apply-patch-v1` fallback 和普通容器命令作为 operation 放在 route 之后；CLI 固定注入 `KUBECONFIG=/etc/rancher/k3s/k3s.yaml` 并把 kubectl、workload exec、logs 和 pod workspace 读写参数组装成 argv，避免在 Host SSH、bash、kubectl exec 和容器 shell 之间反复手写多层引号；D601 与 G14 都有 provider-specific guard，分别校验 `d601` 和 G14 k3s 节点身份。 diff --git a/scripts/src/apply-patch-v2.ts b/scripts/src/apply-patch-v2.ts index 4aa36c57..6bdc7e2f 100644 --- a/scripts/src/apply-patch-v2.ts +++ b/scripts/src/apply-patch-v2.ts @@ -63,6 +63,8 @@ export interface ApplyPatchV2FileSystem { readBlock(path: string, blockIndex: number, blockBytes: number): Promise; writeFile(path: string, content: Buffer): Promise; deleteFile(path: string): Promise; + readFiles?(paths: string[]): Promise>; + applyReplacementsBulk?(paths: Iterable, plans: Map): Promise; } export interface ApplyPatchV2FileStat { @@ -114,15 +116,17 @@ export class ApplyPatchV2Error extends Error { type PlannedFileState = { exists: true; content: string } | { exists: false; content: "" }; type PlannedOperation = { kind: "write"; path: string; content: string } | { kind: "delete"; path: string }; -type Replacement = [start: number, oldLength: number, newLines: string[]]; -interface BulkReplacementWritePlan { +export type ApplyPatchV2Replacement = [start: number, oldLength: number, newLines: string[]]; +type Replacement = ApplyPatchV2Replacement; +export interface ApplyPatchV2BulkReplacementWritePlan { path: string; originalBytes: number; originalSha256: string; finalBytes: number; finalSha256: string; - replacements: Replacement[]; + replacements: ApplyPatchV2Replacement[]; } +type BulkReplacementWritePlan = ApplyPatchV2BulkReplacementWritePlan; interface ApplyPatchV2Plan { changed: string[]; @@ -432,12 +436,19 @@ function instrumentApplyPatchV2Executor(executor: ApplyPatchV2Executor, metrics: } function instrumentApplyPatchV2FileSystem(fs: ApplyPatchV2FileSystem, metrics: ApplyPatchV2RemoteMetrics): ApplyPatchV2FileSystem { - return { + const instrumented: ApplyPatchV2FileSystem = { stat: (path) => recordApplyPatchV2FsOperation(metrics, "fs.stat", () => fs.stat(path)), readBlock: (path, blockIndex, blockBytes) => recordApplyPatchV2FsOperation(metrics, "fs.readBlock", () => fs.readBlock(path, blockIndex, blockBytes)), writeFile: (path, content) => recordApplyPatchV2FsOperation(metrics, "fs.writeFile", () => fs.writeFile(path, content)), deleteFile: (path) => recordApplyPatchV2FsOperation(metrics, "fs.deleteFile", () => fs.deleteFile(path)), }; + if (fs.readFiles !== undefined) { + instrumented.readFiles = (paths) => recordApplyPatchV2FsOperation(metrics, "fs.readFiles", () => fs.readFiles!(paths)); + } + if (fs.applyReplacementsBulk !== undefined) { + instrumented.applyReplacementsBulk = (paths, plans) => recordApplyPatchV2FsOperation(metrics, "fs.applyReplacementsBulk", () => fs.applyReplacementsBulk!(paths, plans)); + } + return instrumented; } async function recordApplyPatchV2FsOperation(metrics: ApplyPatchV2RemoteMetrics, operation: string, run: () => Promise): Promise { @@ -621,7 +632,11 @@ function isBulkReadFailure(error: unknown): boolean { } function shouldUseBulkUpdatePath(executor: ApplyPatchV2Executor, hunks: PatchHunk[]): boolean { - if (!executor.run || executor.fs !== undefined) return false; + if (executor.fs !== undefined) { + if (executor.fs.readFiles === undefined || executor.fs.applyReplacementsBulk === undefined) return false; + } else if (!executor.run) { + return false; + } const paths = new Set(); for (const hunk of hunks) { if (hunk.kind !== "update" || hunk.movePath !== null) return false; @@ -926,13 +941,20 @@ async function readRemoteText(executor: ApplyPatchV2Executor, target: string): P async function readRemoteTextsBulk(executor: ApplyPatchV2Executor, targets: string[]): Promise> { if (targets.length === 0) return new Map(); + if (executor.fs?.readFiles !== undefined && targets.length > 1) { + const files = await executor.fs.readFiles(targets); + for (const target of targets) { + if (!files.has(target)) throw new ApplyPatchV2Error("remote apply-patch v2 fs bulk read omitted target", { target }); + } + return files; + } if (executor.fs || targets.length === 1) { const files = new Map(); for (const target of targets) files.set(target, await readRemoteText(executor, target)); return files; } const result = await checkedRemoteV2(executor, "read-bulk-b64", targets); - return decodeRemoteBulkRead(result.stdout, targets); + return decodeApplyPatchV2BulkRead(result.stdout, targets); } function decodeRemoteReadBlock(stdout: string, target: string, blockIndex: number, expectedChunkBytes: number): Buffer { @@ -979,7 +1001,7 @@ function decodeRemoteReadBlock(stdout: string, target: string, blockIndex: numbe return decoded.length > expectedChunkBytes ? decoded.subarray(0, expectedChunkBytes) : decoded; } -function decodeRemoteBulkRead(stdout: string, targets: string[]): Map { +export function decodeApplyPatchV2BulkRead(stdout: string, targets: string[]): Map { const lines = stdout.split(/\r?\n/u); const markerIndex = lines.findIndex((line) => line.startsWith(`${remoteBulkReadMarker} `)); if (markerIndex < 0) { @@ -1050,8 +1072,17 @@ async function writeRemoteText(executor: ApplyPatchV2Executor, target: string, c } async function writeRemoteReplacementsBulk(executor: ApplyPatchV2Executor, paths: Iterable, plans: Map): Promise { - const targets = Array.from(paths); + const { targets, payload } = formatApplyPatchV2BulkReplacementPayload(paths, plans); if (targets.length === 0) return; + if (executor.fs?.applyReplacementsBulk !== undefined) { + await executor.fs.applyReplacementsBulk(targets, plans); + return; + } + await checkedRemoteV2(executor, "apply-replacements-bulk-stdin", [String(targets.length)], payload); +} + +export function formatApplyPatchV2BulkReplacementPayload(paths: Iterable, plans: Map): { targets: string[]; payload: string } { + const targets = Array.from(paths); const records: string[] = []; for (const target of targets) { const plan = plans.get(target); @@ -1073,7 +1104,7 @@ async function writeRemoteReplacementsBulk(executor: ApplyPatchV2Executor, paths replacementFields.join(";"), ].join(" ")); } - await checkedRemoteV2(executor, "apply-replacements-bulk-stdin", [String(targets.length)], `${records.join("\n")}\n`); + return { targets, payload: `${records.join("\n")}\n` }; } type RemoteV2Operation = diff --git a/scripts/src/help.ts b/scripts/src/help.ts index ef1e492e..6e4c45cb 100644 --- a/scripts/src/help.ts +++ b/scripts/src/help.ts @@ -202,7 +202,7 @@ export function sshHelp(): unknown { "When a one-line shell command is easier to type through the script path, `script -- ''` runs that single string through the remote shell without waiting for stdin. When `script --` is followed by multiple tokens, it stays a direct argv form for commands such as `trans D601:/work script -- sed -n '1,20p' file`.", "script and shell helper modes inject a tiny POSIX-compatible printf wrapper before user shell text, so portable printf headings such as `printf \"--- section ---\\n\"` work consistently under dash/sh and bash. Direct argv commands are unchanged.", "For arbitrary stdin streams into a workload command, use a workload route plus `exec --stdin -- ...`; this keeps the route as location-only and avoids heredoc/base64/tar shell wrapping.", - "`apply-patch` is the default remote text patch entry and uses the v2 local line-based patch engine with remote read/write operations, including Windows routes such as `D601:win/c/test`, so long Unicode/Chinese lines and pure insertion hunks avoid the legacy remote shell hunk parser. Plain multi-file Update File patches on POSIX host/k3s routes use bulk read/write operations to avoid per-file SSH round trips. Its stdout follows Codex apply_patch text output rather than UniDesk JSON output; stderr keeps Codex-style failure text and appends one `UNIDESK_APPLY_PATCH_TIMING` JSON summary with durationMs, patchBytes, fileCount, hunkCount, changedCount, remoteOperationCount, remoteOperationCounts and remoteElapsedMs so slow patch runs can be attributed without changing success stdout.", + "`apply-patch` is the default remote text patch entry and uses the v2 local line-based patch engine with remote read/write operations, including Windows routes such as `D601:win/c/test`, so long Unicode/Chinese lines and pure insertion hunks avoid the legacy remote shell hunk parser. Plain multi-file Update File patches on POSIX host/k3s and Windows workspace routes use bulk read/write operations to avoid per-file SSH round trips. Its stdout follows Codex apply_patch text output rather than UniDesk JSON output; stderr keeps Codex-style failure text and appends one `UNIDESK_APPLY_PATCH_TIMING` JSON summary with durationMs, patchBytes, fileCount, hunkCount, changedCount, remoteOperationCount, remoteOperationCounts and remoteElapsedMs so slow patch runs can be attributed without changing success stdout.", "`upload` and `download` are the default whole-file transfer entries for non-text and generated files. They write through remote temp files, verify byte count and SHA-256 on both sides, and return `verification.automatic=true`, `verification.verified=true`, and `verification.match.{bytes,sha256}=true`; this JSON is the transfer integrity proof, so callers do not need a separate manual `sha256sum` check. The client falls back from a single stdin payload to bounded chunks before treating provider-gateway limits as a server-side problem.", "`apply-patch-v1` is the only legacy fallback entry: it rejects low-context update hunks by default, reports the matched file:line for each hunk on stderr, and only accepts --allow-loose when the caller has manually reviewed an intentionally ambiguous insertion.", "script defaults to target /bin/sh and inherits provider proxy variables such as HTTP_PROXY/HTTPS_PROXY/ALL_PROXY/NO_PROXY; it is for host/k3s POSIX shell only. Use --shell bash only for bash syntax such as pipefail, arrays, or [[ ... ]], not as a proxy workaround.", diff --git a/scripts/src/ssh.ts b/scripts/src/ssh.ts index ea6e9d72..4ced2e3a 100644 --- a/scripts/src/ssh.ts +++ b/scripts/src/ssh.ts @@ -1,7 +1,15 @@ import { spawn } from "node:child_process"; import { createHash, randomBytes } from "node:crypto"; import { type UniDeskConfig, repoRoot } from "./config"; -import { isApplyPatchV2HelpArgs, runApplyPatchV2, type ApplyPatchV2Executor, type ApplyPatchV2FileSystem } from "./apply-patch-v2"; +import { + decodeApplyPatchV2BulkRead, + formatApplyPatchV2BulkReplacementPayload, + isApplyPatchV2HelpArgs, + runApplyPatchV2, + type ApplyPatchV2BulkReplacementWritePlan, + type ApplyPatchV2Executor, + type ApplyPatchV2FileSystem, +} from "./apply-patch-v2"; import { isSshFileTransferOperation, runSshFileTransferOperation, type SshRemoteCommandExecutor } from "./ssh-file-transfer"; export interface ParsedSshArgs { @@ -2409,6 +2417,8 @@ export function remoteCommandForRoute(route: ParsedSshRoute, command: string[], type WindowsApplyPatchFsOperation = | "stat" | "read-b64-block" + | "read-bulk-b64" + | "apply-replacements-bulk-stdin" | "write-b64-stdin" | "write-b64-begin" | "write-b64-append-stdin" @@ -2465,6 +2475,15 @@ function createWindowsApplyPatchFileSystem(config: UniDeskConfig, invocation: Pa async deleteFile(filePath) { await checked("delete", [filePath]); }, + async readFiles(filePaths) { + const result = await checked("read-bulk-b64", [String(filePaths.length)], `${filePaths.join("\n")}\n`); + return decodeApplyPatchV2BulkRead(result.stdout, filePaths); + }, + async applyReplacementsBulk(filePaths, plans: Map) { + const { targets, payload } = formatApplyPatchV2BulkReplacementPayload(filePaths, plans); + if (targets.length === 0) return; + await checked("apply-replacements-bulk-stdin", [String(targets.length)], payload); + }, }; } @@ -2489,6 +2508,9 @@ function windowsApplyPatchFsScript(basePath: string | null, operation: WindowsAp "function Resolve-UnideskPath([string]$Raw) { if ([string]::IsNullOrWhiteSpace($Raw)) { Fail 'empty apply-patch path' 2 }; if ([System.IO.Path]::IsPathRooted($Raw)) { return [System.IO.Path]::GetFullPath($Raw) }; if (-not [string]::IsNullOrWhiteSpace($basePath)) { return [System.IO.Path]::GetFullPath([System.IO.Path]::Combine($basePath, $Raw)) }; return [System.IO.Path]::GetFullPath($Raw) }", "function Ensure-Parent([string]$Target) { $parent = [System.IO.Path]::GetDirectoryName($Target); if (-not [string]::IsNullOrWhiteSpace($parent)) { [System.IO.Directory]::CreateDirectory($parent) | Out-Null } }", "function Get-Sha256([string]$Path) { return (Get-FileHash -LiteralPath $Path -Algorithm SHA256).Hash.ToLowerInvariant() }", + "function Get-Sha256Bytes([byte[]]$Bytes) { $sha = [System.Security.Cryptography.SHA256]::Create(); try { return ([System.BitConverter]::ToString($sha.ComputeHash($Bytes))).Replace('-', '').ToLowerInvariant() } finally { $sha.Dispose() } }", + "function Decode-Utf8([string]$Encoded) { return [System.Text.Encoding]::UTF8.GetString([System.Convert]::FromBase64String($Encoded)) }", + "function Encode-Utf8([string]$Value) { return [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($Value)) }", "function Set-TmpPaths([string]$Target, [string]$Token) { if ($Token -notmatch '^[A-Za-z0-9_.-]+$') { Fail 'invalid apply-patch temp token' 2 }; $dir = [System.IO.Path]::GetDirectoryName($Target); if ([string]::IsNullOrWhiteSpace($dir)) { $dir = (Get-Location).ProviderPath }; $base = [System.IO.Path]::GetFileName($Target); $script:tmp = [System.IO.Path]::Combine($dir, '.' + $base + '.unidesk-v2-' + $Token + '.tmp'); $script:tmpB64 = $script:tmp + '.b64' }", "function Verify-Temp([string]$Target, [string]$Tmp, [Int64]$ExpectedBytes, [string]$ExpectedSha256) { $actualBytes = ([System.IO.FileInfo]$Tmp).Length; if ($actualBytes -ne $ExpectedBytes) { Remove-Item -LiteralPath $Tmp -Force -ErrorAction SilentlyContinue; Fail ('apply-patch byte count mismatch for ' + $Target + ': expected=' + $ExpectedBytes + ' actual=' + $actualBytes) 23 }; $actualSha256 = Get-Sha256 $Tmp; if ($actualSha256 -ne $ExpectedSha256) { Remove-Item -LiteralPath $Tmp -Force -ErrorAction SilentlyContinue; Fail ('apply-patch sha256 mismatch for ' + $Target + ': expected=' + $ExpectedSha256 + ' actual=' + $actualSha256) 24 } }", "function Decode-ToTarget([string]$Target, [string]$Encoded, [Int64]$ExpectedBytes, [string]$ExpectedSha256) { Ensure-Parent $Target; Set-TmpPaths $Target ([guid]::NewGuid().ToString('N')); try { $bytes = [Convert]::FromBase64String(($Encoded -replace '\\s','')) } catch { Fail ('apply-patch base64 decode failed for ' + $Target + ': ' + $_.Exception.Message) 22 }; [System.IO.File]::WriteAllBytes($script:tmp, $bytes); Verify-Temp $Target $script:tmp $ExpectedBytes $ExpectedSha256; Move-Item -LiteralPath $script:tmp -Destination $Target -Force; $actualSha256 = Get-Sha256 $Target; if ($actualSha256 -ne $ExpectedSha256) { Fail ('apply-patch final sha256 mismatch for ' + $Target) 25 } }", @@ -2496,6 +2518,8 @@ function windowsApplyPatchFsScript(basePath: string | null, operation: WindowsAp "switch ($operation) {", " 'stat' { if (-not (Test-Path -LiteralPath $target -PathType Leaf)) { Fail ('file not found: ' + $target) 1 }; $bytes = ([System.IO.FileInfo]$target).Length; $digest = Get-Sha256 $target; [Console]::Out.WriteLine(([string]$bytes) + ' ' + $digest); break }", " 'read-b64-block' { $blockIndex = [Int64]$arg1; $blockSize = [Int32]$arg2; if ($blockIndex -lt 0 -or $blockSize -le 0) { Fail 'invalid read block args' 2 }; $fs = [System.IO.File]::Open($target, [System.IO.FileMode]::Open, [System.IO.FileAccess]::Read, [System.IO.FileShare]::ReadWrite); try { [void]$fs.Seek($blockIndex * $blockSize, [System.IO.SeekOrigin]::Begin); $buffer = New-Object byte[] $blockSize; $read = $fs.Read($buffer, 0, $blockSize); if ($read -gt 0) { [Console]::Out.Write([Convert]::ToBase64String($buffer, 0, $read)) } } finally { $fs.Dispose() }; break }", + " 'read-bulk-b64' { $expectedCount = [Int32]$targetArg; $items = @([Console]::In.ReadToEnd() -split \"`r?`n\" | Where-Object { -not [string]::IsNullOrWhiteSpace($_) }); if ($items.Count -ne $expectedCount) { Fail ('bulk read record count mismatch: expected=' + $expectedCount + ' actual=' + $items.Count) 23 }; [Console]::Out.WriteLine('UNIDESK_APPLY_PATCH_V2_BULK_READ ' + $items.Count); foreach ($item in $items) { $path = Resolve-UnideskPath $item; if (-not (Test-Path -LiteralPath $path -PathType Leaf)) { Fail ('file not found: ' + $path) 1 }; $bytes = [System.IO.File]::ReadAllBytes($path); $pathB64 = Encode-Utf8 $item; $digest = Get-Sha256Bytes $bytes; [Console]::Out.Write($pathB64 + ' ' + $bytes.Length + ' ' + $digest + ' '); [Console]::Out.Write([System.Convert]::ToBase64String($bytes)); [Console]::Out.WriteLine() }; break }", + " 'apply-replacements-bulk-stdin' { $expectedCount = [Int32]$targetArg; $records = @([Console]::In.ReadToEnd() -split \"`r?`n\" | Where-Object { -not [string]::IsNullOrWhiteSpace($_) }); if ($records.Count -ne $expectedCount) { Fail ('bulk replacement record count mismatch: expected=' + $expectedCount + ' actual=' + $records.Count) 23 }; $utf8 = [System.Text.UTF8Encoding]::new($false); $plans = New-Object System.Collections.Generic.List[object]; $index = 0; foreach ($record in $records) { $index += 1; $fields = $record -split '\\s+'; if ($fields.Count -ne 6) { Fail 'bulk replacement malformed record' 23 }; $targetRelative = Decode-Utf8 $fields[0]; $resolved = Resolve-UnideskPath $targetRelative; $originalBytes = [Int64]$fields[1]; $originalSha256 = $fields[2]; $finalBytes = [Int64]$fields[3]; $finalSha256 = $fields[4]; $replacementsText = $fields[5]; if (-not (Test-Path -LiteralPath $resolved -PathType Leaf)) { Fail ('file not found: ' + $resolved) 1 }; $source = [System.IO.File]::ReadAllBytes($resolved); if ($source.Length -ne $originalBytes -or (Get-Sha256Bytes $source) -ne $originalSha256) { Fail ('bulk replacement original integrity mismatch for ' + $resolved) 23 }; $lines = New-Object System.Collections.Generic.List[string]; $sourceText = [System.Text.Encoding]::UTF8.GetString($source); foreach ($line in ($sourceText -split \"`n\", -1)) { $lines.Add($line) | Out-Null }; if ($lines.Count -gt 0 -and $lines[$lines.Count - 1] -eq '') { $lines.RemoveAt($lines.Count - 1) }; $replacements = New-Object System.Collections.Generic.List[object]; if (-not [string]::IsNullOrEmpty($replacementsText)) { foreach ($item in ($replacementsText -split ';')) { if ([string]::IsNullOrWhiteSpace($item)) { continue }; $parts = $item -split ',', 3; if ($parts.Count -ne 3) { Fail 'bulk replacement malformed replacement' 23 }; $newText = Decode-Utf8 $parts[2]; $newLines = New-Object System.Collections.Generic.List[string]; foreach ($line in ($newText -split \"`n\", -1)) { $newLines.Add($line) | Out-Null }; if ($newLines.Count -gt 0 -and $newLines[$newLines.Count - 1] -eq '') { $newLines.RemoveAt($newLines.Count - 1) }; $replacements.Add([pscustomobject]@{ start = [Int32]$parts[0]; oldLength = [Int32]$parts[1]; newLines = [string[]]$newLines }) | Out-Null } }; foreach ($replacement in @($replacements | Sort-Object -Property start -Descending)) { if ($replacement.start -lt 0 -or $replacement.oldLength -lt 0 -or ($replacement.start + $replacement.oldLength) -gt $lines.Count) { Fail ('bulk replacement out of bounds for ' + $resolved) 23 }; $removeCount = $replacement.oldLength; if ($removeCount -gt 0) { $lines.RemoveRange($replacement.start, $removeCount) }; if ($replacement.newLines.Count -gt 0) { $lines.InsertRange($replacement.start, [string[]]$replacement.newLines) } }; $outputText = if ($lines.Count -eq 0) { '' } else { ([string]::Join(\"`n\", [string[]]$lines) + \"`n\") }; $outputBytes = [System.Text.Encoding]::UTF8.GetBytes($outputText); if ($outputBytes.Length -ne $finalBytes -or (Get-Sha256Bytes $outputBytes) -ne $finalSha256) { Fail ('bulk replacement final integrity mismatch for ' + $resolved) 23 }; $tmp = [System.IO.Path]::Combine([System.IO.Path]::GetDirectoryName($resolved), ('.' + [System.IO.Path]::GetFileName($resolved) + '.unidesk-v2-bulk-' + [guid]::NewGuid().ToString('N') + '.tmp')); [System.IO.File]::WriteAllText($tmp, $outputText, $utf8); $plans.Add([pscustomobject]@{ target = $resolved; tmp = $tmp; sha256 = $finalSha256 }) | Out-Null }; foreach ($plan in $plans) { Ensure-Parent $plan.target; Move-Item -LiteralPath $plan.tmp -Destination $plan.target -Force; if ((Get-Sha256 $plan.target) -ne $plan.sha256) { Fail ('bulk replacement final sha256 mismatch for ' + $plan.target) 25 } }; break }", " 'write-b64-stdin' { Decode-ToTarget $target ([Console]::In.ReadToEnd()) ([Int64]$arg1) $arg2; break }", " 'write-b64-begin' { Ensure-Parent $target; Set-TmpPaths $target $arg1; [System.IO.File]::WriteAllText($script:tmpB64, '', [System.Text.Encoding]::ASCII); break }", " 'write-b64-append-stdin' { Set-TmpPaths $target $arg1; $chunk = ([Console]::In.ReadToEnd()) -replace '\\s',''; [System.IO.File]::AppendAllText($script:tmpB64, $chunk, [System.Text.Encoding]::ASCII); break }", diff --git a/scripts/ssh-argv-guidance-contract-test.ts b/scripts/ssh-argv-guidance-contract-test.ts index 33e6c5da..f042486f 100644 --- a/scripts/ssh-argv-guidance-contract-test.ts +++ b/scripts/ssh-argv-guidance-contract-test.ts @@ -5,7 +5,7 @@ import { chmodSync, existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, wr import os from "node:os"; import path from "node:path"; import { sshHelp } from "./src/help"; -import { runApplyPatchV2, type ApplyPatchV2TimingSummary } from "./src/apply-patch-v2"; +import { runApplyPatchV2, type ApplyPatchV2TimingSummary, type ApplyPatchV2BulkReplacementWritePlan } from "./src/apply-patch-v2"; import { providerTriageRecommendedCrossChecks } from "./src/provider-triage"; import { extractRemoteCliOptions, remoteSshFrontendPlanForTest } from "./src/remote"; import { runSshFileTransferOperation, type SshFileTransferCommandBuilders, type SshRemoteCommandExecutor } from "./src/ssh-file-transfer"; @@ -341,6 +341,95 @@ async function applyPatchV2Fixture(patch: string, files: Record) return { stdout: result.stdout, files: result.files, commands: result.commands }; } +async function applyPatchV2FsBulkFixtureAttempt(patch: string, files: Record): Promise<{ stdout: string; stderr: string; exitCode: number | null; files: Record; operations: string[]; error: unknown | null }> { + const state = new Map(Object.entries(files)); + const operations: string[] = []; + const stdin = new PassThrough(); + stdin.end(patch); + let stdout = ""; + let stderr = ""; + const stdoutSink = new Writable({ + write(chunk, _encoding, callback) { + stdout += Buffer.isBuffer(chunk) ? chunk.toString("utf8") : String(chunk); + callback(); + }, + }); + const stderrSink = new Writable({ + write(chunk, _encoding, callback) { + stderr += Buffer.isBuffer(chunk) ? chunk.toString("utf8") : String(chunk); + callback(); + }, + }); + let error: unknown | null = null; + let exitCode: number | null = null; + try { + exitCode = await runApplyPatchV2({ + stdin, + stdout: stdoutSink, + stderr: stderrSink, + executor: { + fs: { + async stat(filePath) { + operations.push(`stat ${filePath}`); + const content = state.get(filePath); + if (content === undefined) throw new Error(`missing ${filePath}`); + const buffer = Buffer.from(content, "utf8"); + return { bytes: buffer.length, sha256: sha256BufferHex(buffer) }; + }, + async readBlock(filePath, blockIndex, blockBytes) { + operations.push(`readBlock ${filePath}`); + const content = state.get(filePath); + if (content === undefined) throw new Error(`missing ${filePath}`); + const buffer = Buffer.from(content, "utf8"); + return buffer.subarray(blockIndex * blockBytes, (blockIndex + 1) * blockBytes); + }, + async writeFile(filePath, content) { + operations.push(`writeFile ${filePath}`); + state.set(filePath, content.toString("utf8")); + }, + async deleteFile(filePath) { + operations.push(`deleteFile ${filePath}`); + state.delete(filePath); + }, + async readFiles(paths) { + operations.push(`readFiles ${paths.join(",")}`); + const result = new Map(); + for (const filePath of paths) { + const content = state.get(filePath); + if (content === undefined) throw new Error(`missing ${filePath}`); + result.set(filePath, content); + } + return result; + }, + async applyReplacementsBulk(paths: Iterable, plans: Map) { + const targets = Array.from(paths); + operations.push(`applyReplacementsBulk ${targets.join(",")}`); + for (const filePath of targets) { + const plan = plans.get(filePath); + const original = state.get(filePath); + if (plan === undefined || original === undefined) throw new Error(`missing replacement plan ${filePath}`); + const originalBuffer = Buffer.from(original, "utf8"); + assertCondition(originalBuffer.length === plan.originalBytes && sha256BufferHex(originalBuffer) === plan.originalSha256, "fs bulk fixture original integrity mismatch", { filePath, plan }); + const lines = original.split("\n"); + if (lines.at(-1) === "") lines.pop(); + for (const [start, oldLength, newLines] of [...plan.replacements].reverse()) { + lines.splice(start, oldLength, ...newLines); + } + const updated = lines.length === 0 ? "" : `${lines.join("\n")}\n`; + const updatedBuffer = Buffer.from(updated, "utf8"); + assertCondition(updatedBuffer.length === plan.finalBytes && sha256BufferHex(updatedBuffer) === plan.finalSha256, "fs bulk fixture final integrity mismatch", { filePath, plan }); + state.set(filePath, updated); + } + }, + }, + }, + }); + } catch (caught) { + error = caught; + } + return { stdout, stderr, exitCode, files: Object.fromEntries(state), operations, error }; +} + function fileTransferFixture(initial: Record = {}, options: { emptyReadOnce?: Record; shortReadOnce?: Record> } = {}): { state: Map; commands: Array<{ operation: string; stdin: boolean }>; @@ -865,6 +954,22 @@ export async function runSshArgvGuidanceContract(): Promise { const bulkTiming = applyPatchTimingFromStderr(bulkV2.stderr); assertCondition(bulkTiming.remoteOperationCount === 2 && bulkTiming.remoteOperationCounts["read-bulk-b64"] === 1 && bulkTiming.remoteOperationCounts["apply-replacements-bulk-stdin"] === 1, "v2 timing summary should classify bulk read and line-level apply operations", bulkTiming); + const fsBulkV2 = await applyPatchV2FsBulkFixtureAttempt(bulkPatchLines.join("\n"), bulkFiles); + assertCondition(fsBulkV2.exitCode === 0 && fsBulkV2.error === null, "v2 fs executor should support the same multi-file bulk update path", fsBulkV2); + assertCondition( + fsBulkV2.operations.length === 2 + && fsBulkV2.operations[0]?.startsWith("readFiles ") + && fsBulkV2.operations[1]?.startsWith("applyReplacementsBulk "), + "v2 fs multi-file update path should use fs bulk read and line-level apply operations", + fsBulkV2.operations, + ); + assertCondition(!fsBulkV2.operations.some((operation) => operation.startsWith("stat ") || operation.startsWith("readBlock ") || operation.startsWith("writeFile ")), "v2 fs bulk path should avoid per-file stat/read/write operations", fsBulkV2.operations); + for (let fileIndex = 0; fileIndex < 4; fileIndex += 1) { + assertCondition(fsBulkV2.files[`bulk-${fileIndex}.txt`]?.includes(`file=${fileIndex} line=40 value=beta`), "v2 fs bulk path should write all changed files", { fileIndex, files: fsBulkV2.files }); + } + const fsBulkTiming = applyPatchTimingFromStderr(fsBulkV2.stderr); + assertCondition(fsBulkTiming.remoteOperationCount === 2 && fsBulkTiming.remoteOperationCounts["fs.readFiles"] === 1 && fsBulkTiming.remoteOperationCounts["fs.applyReplacementsBulk"] === 1, "v2 timing summary should classify fs bulk read and line-level apply operations", fsBulkTiming); + const unprefixedUpdateContextV2 = await applyPatchV2FixtureAttempt([ "*** Begin Patch", "*** Update File: internal/cloud/access-control.ts",