mirrors/pikasTech-unidesk
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: add code queue services and baidu netdisk

Browse Source
This commit is contained in:
Codex
2026-05-13 00:59:36 +00:00
parent ae462ed9ef
commit 6a04144d3f
63 changed files with 10983 additions and 1101 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/issue/baidu-netdisk-env-setup.md
+127
View File
@@ -0,0 +1,127 @@
# Baidu Netdisk 环境变量配置说明
Date: 2026-05-11
## 我能补什么，不能补什么
我可以在本机生成并持久化这个本地密钥：
- `UNIDESK_BAIDU_NETDISK_TOKEN_KEY`：UniDesk 本地随机密钥，只用于把百度 `access_token` / `refresh_token` 加密后写入 PostgreSQL。
我不能代替你生成下面两个值，因为它们属于你的百度网盘开放平台应用：
- `UNIDESK_BAIDU_NETDISK_CLIENT_ID`：百度 OAuth `client_id`，在部分百度控制台页面也叫 AppKey/API Key。
- `UNIDESK_BAIDU_NETDISK_CLIENT_SECRET`：百度 OAuth `client_secret`，在部分百度控制台页面也叫 Secret Key。
截至 2026-05-11，本机已经在 `.state/docker-compose.env` 中生成并写入了 `UNIDESK_BAIDU_NETDISK_TOKEN_KEY`。百度应用的 client id 和 client secret 仍保持未配置状态，因为它们是账号归属的密钥，不能猜测，也不能提交到仓库。
## 获取百度应用凭证
1. 打开百度网盘开放平台控制台，为 UniDesk 创建一个应用，或复用你已有的应用。
2. 开启 OAuth 和网盘文件 API 所需权限；当前服务请求的 scope 是 `basic,netdisk`。
3. 复制应用的 OAuth client id/AppKey 和 client secret/Secret Key。
4. 妥善保管密钥，不要把它粘贴到 `config.json`、文档、源码、git commit、截图、issue 评论或前端代码里。
官方参考：
- 开放平台概览：https://pan.baidu.com/union/doc/nksg0sbfs
- 设备码模式：https://pan.baidu.com/union/doc/fl1x114ti
- 授权码模式：https://pan.baidu.com/union/doc/al0rwqzzl
## 配置这台主机
UniDesk CLI 会把 Docker 运行时变量写到 `.state/docker-compose.env`；该文件已被 git 忽略，适合放本机密钥。请把下面占位值替换为真实百度凭证：
```bash
cd /root/unidesk
python3 - <<'PY'
from pathlib import Path
updates = {
"UNIDESK_BAIDU_NETDISK_CLIENT_ID": "paste-baidu-client-id-here",
"UNIDESK_BAIDU_NETDISK_CLIENT_SECRET": "paste-baidu-client-secret-here",
# Optional. Keep the existing value unless you intentionally want another app-folder name.
"UNIDESK_BAIDU_NETDISK_APP_ROOT": "/apps/UniDeskBaiduNetdisk",
}
path = Path(".state/docker-compose.env")
lines = path.read_text("utf-8").splitlines() if path.exists() else []
seen = set()
for index, line in enumerate(lines):
if "=" not in line:
continue
key = line.split("=", 1)[0]
if key in updates:
lines[index] = f"{key}={updates[key]}"
seen.add(key)
for key, value in updates.items():
if key not in seen:
lines.append(f"{key}={value}")
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text("\n".join(lines) + "\n", "utf-8")
PY
```
也可以在当前 shell 里临时 export，然后运行会调用 `writeComposeEnv` 的 UniDesk CLI 命令，例如 `server rebuild baidu-netdisk`。CLI 会把这些 shell 变量持久化到 `.state/docker-compose.env`：
```bash
cd /root/unidesk
export UNIDESK_BAIDU_NETDISK_CLIENT_ID='paste-baidu-client-id-here'
export UNIDESK_BAIDU_NETDISK_CLIENT_SECRET='paste-baidu-client-secret-here'
export UNIDESK_BAIDU_NETDISK_APP_ROOT='/apps/UniDeskBaiduNetdisk'
bun scripts/cli.ts server rebuild baidu-netdisk
```
## 应用并验证
填好凭证后，只重建 Baidu Netdisk 后端，并等待异步 job 完成：
```bash
cd /root/unidesk
bun scripts/cli.ts server rebuild baidu-netdisk
bun scripts/cli.ts job status latest
```
然后验证服务并启动二维码/设备码登录流程：
```bash
bun scripts/cli.ts microservice health baidu-netdisk
bun scripts/cli.ts microservice proxy baidu-netdisk /api/auth/device/start --method POST --raw
```
手动扫码授权前的预期状态：
- `health.body.auth.configured` 为 `true`。
- `health.body.auth.clientIdConfigured` 为 `true`。
- `health.body.auth.clientSecretConfigured` 为 `true`。
- `health.body.auth.tokenKeyConfigured` 为 `true`。
- `/api/auth/device/start` 返回设备登录会话、二维码/用户码元数据，不返回原始百度 token。
扫码并授权后：
```bash
bun scripts/cli.ts microservice proxy baidu-netdisk '/api/account' --raw
bun scripts/cli.ts microservice proxy baidu-netdisk '/api/files?dir=/apps/UniDeskBaiduNetdisk&limit=20' --raw
bun scripts/cli.ts microservice proxy baidu-netdisk /api/self-test --method POST --raw
```
`/api/files` 首次访问空应用目录时应返回 `ok=true` 和文件数组；如果远端应用目录还不存在，后端会先创建 `UNIDESK_BAIDU_NETDISK_APP_ROOT` 指向的 `/apps/...` 目录。`/api/self-test` 会生成小文本、上传、列表确认、下载并比较 MD5，适合授权完成后的端到端验收。
## Token Key 轮换
只应在首次登录前，或明确退出登录并丢弃旧加密 token 记录后，才轮换 `UNIDESK_BAIDU_NETDISK_TOKEN_KEY`。这个 key 用于解密 `baidu_netdisk_tokens` 和 `baidu_netdisk_auth_sessions` 中的既有加密记录；如果已有加密记录后再改 key，旧记录会不可读，需要重新扫码登录。
如果测试登录后必须轮换，先正常退出登录：
```bash
bun scripts/cli.ts microservice proxy baidu-netdisk /api/auth/logout --method POST --raw
```
然后更新 `UNIDESK_BAIDU_NETDISK_TOKEN_KEY`，重建 `baidu-netdisk`，再重新发起二维码登录。
## 安全检查清单
- 百度凭证只放在 `.state/docker-compose.env`、root 管理的密钥系统或临时 shell 环境里。
- 不要提交 `.env`、`.state/docker-compose.env`、access token、refresh token、dlink 或包含密钥的截图。
- 不要把 `4244` 暴露到公网；Baidu Netdisk 必须继续走 UniDesk microservice proxy。
- 成功登录后不要随意轮换 token key，除非已经准备重新登录。
docs/issue/baidu-netdisk-user-service.md
+255
View File
@@ -0,0 +1,255 @@
# Baidu Netdisk User Service Research
Date: 2026-05-11
Implementation note: the first UniDesk-integrated version now lives in this repo as a main-server private user service, with backend source at `src/components/microservices/baidu-netdisk`, Compose service `baidu-netdisk`, config id `baidu-netdisk`, and frontend page `src/components/frontend/src/baidu-netdisk.tsx`. It keeps the recommended v1 boundary: JSON control API through UniDesk microservice proxy, OAuth Device Code login, PostgreSQL-backed encrypted token/task state, and staging-directory upload/download jobs instead of browser byte streaming.
Environment setup note: Baidu app credentials are account-owned secrets and must be supplied out of band. The local encryption key and exact host configuration steps are documented in `docs/issue/baidu-netdisk-env-setup.md`.
## Goal
Create a UniDesk user service that connects to Baidu Netdisk in a containerized way and exposes file storage operations such as login, browse, upload, download, move, rename and delete. The user-facing login should feel similar to the ClaudeQQ page: the UniDesk frontend shows a login card/QR, backend state, recent transfer jobs and explicit raw JSON buttons, while the business backend remains private behind the UniDesk microservice proxy.
## Recommended Approach
Build a small pure-backend user service named `baidu-netdisk` and integrate it as a UniDesk user service. Use Baidu Netdisk official OAuth/API directly in the backend for the first version; optionally add AList or CLI tools later as transfer workers, not as the primary auth or frontend.
Why this route:
- The official API supports OAuth scopes `basic,netdisk`, QR-like login flows, file listing, metadata/dlink retrieval, multipart upload and file management.
- A ClaudeQQ-like login UI maps well to Baidu's Device Code flow: backend requests a `device_code`, frontend displays `qrcode_url` and `user_code`, backend polls token status at the documented interval.
- The UniDesk proxy currently handles text JSON request/response bodies, with a 1 MiB incoming body limit and an 8 MiB response body limit. Large file bytes should therefore not be pushed through `/api/microservices/*/proxy` in v1.
- Keeping tokens and jobs in PostgreSQL gives restart recovery and avoids storing credentials in local JSON files.
## Important UniDesk Constraint
Current `microservice.http` is suitable for control-plane JSON, not bulk binary file transfer:
- backend-core reads non-GET bodies using `req.text()` and rejects bodies larger than 1 MiB.
- provider-gateway reads upstream responses using `response.text()` and caps returned body text at 8 MiB.
- The proxy only forwards `content-type`, not `range`, `content-disposition` or arbitrary binary headers.
So v1 should expose APIs such as `POST /api/transfers/upload-from-path` and `POST /api/transfers/download-to-path`, where the backend container reads/writes files from a mounted staging directory. If browser-to-local uploads/downloads are required, add a separate binary streaming capability to backend-core/provider-gateway in a future change set. That future gateway change would trigger the provider-gateway version-bump rule.
## Baidu Netdisk Access Model
### Login
Use Device Code as the default container login:
1. `POST /api/auth/device/start` calls `GET https://openapi.baidu.com/oauth/2.0/device/code?response_type=device_code&client_id=<appKey>&scope=basic,netdisk`.
2. Backend stores `device_code`, `user_code`, `verification_url`, `qrcode_url`, `expires_in` and `interval` in PostgreSQL.
3. UniDesk frontend displays the QR code and user code.
4. `GET /api/auth/device/status?sessionId=...` returns current login state; backend polls Baidu token endpoint no more frequently than the returned interval and at least 5 seconds.
5. On success, backend stores `access_token`, `refresh_token`, `expires_in`, `scope`, account metadata and refresh timestamps.
Authorization Code mode is also possible and can pass `qrcode=1`, but it requires a redirect URI and callback endpoint. Device Code is simpler for a private container behind UniDesk because the browser only needs to display a QR URL and poll backend state.
Token handling requirements:
- Access token lifetime is 30 days in the Baidu docs.
- Refresh token is long-lived but the Netdisk docs say it is single-use: after a refresh, store the new refresh token immediately and never retry with the old one in a loop.
- Use a PostgreSQL row lock or advisory lock around refresh to avoid two workers spending the same refresh token concurrently.
- Never log tokens or dlinks; health/status endpoints should only expose redacted auth state.
### Scope and App Folder
Use `scope=basic,netdisk`. Official docs describe third-party app data under `/apps/<productName>` and visible to users as `/我的应用数据/<productName>`. The service should default to a configured app root, for example `/apps/UniDeskBaiduNetdisk`, and reject paths outside that root unless the final approved app permission explicitly allows broader access.
### Browse and Metadata
Useful official endpoints:
- User info: `GET /rest/2.0/xpan/nas?method=uinfo`.
- Quota: `GET /api/quota`.
- List directory: `GET /rest/2.0/xpan/file?method=list` with `dir`, paging and sort parameters.
- File metadata and download URL: `GET /rest/2.0/xpan/multimedia?method=filemetas&fsids=[...]&dlink=1`.
### Upload
Official multipart upload sequence:
1. Compute full-file MD5 and per-part MD5 list. For normal users, part size is fixed at 4 MiB. Docs list higher part and total file limits for paid membership tiers.
2. `POST /rest/2.0/xpan/file?method=precreate` with `path`, `size`, `isdir=0`, `autoinit=1`, `rtype` and `block_list`.
3. `GET /rest/2.0/pcs/file?method=locateupload&appid=250528&uploadid=...&upload_version=2.0` and choose an HTTPS upload domain from `servers`.
4. `POST https://<upload-domain>/rest/2.0/pcs/superfile2?method=upload&type=tmpfile&path=...&uploadid=...&partseq=N` as multipart form `file=@chunk` for each required part.
5. `POST /rest/2.0/xpan/file?method=create` with the same `path`, `size`, `isdir`, `rtype`, `uploadid` and ordered `block_list`.
For small files, the official single-step upload endpoint can be a convenience path, but the multipart path is enough for all sizes and gives progress/resume semantics.
### Download
Official download sequence:
1. Get `fs_id` from list/search.
2. Request file metadata with `dlink=1`.
3. Fetch `dlink&access_token=<token>` using `User-Agent: pan.baidu.com`.
4. Respect `302` redirects, `Range` for resume, and the documented 8-hour dlink lifetime.
Because browsers cannot safely set the required `User-Agent` and current UniDesk proxy cannot stream large binary responses, the backend should download to staging storage in v1. Browser download can be offered later via a binary streaming proxy endpoint or a backend-owned short-lived internal file endpoint if the gateway/core are upgraded to stream bytes safely.
## Third-Party Technology Options
1. Official API in custom backend (recommended for v1)
- Best fit for UniDesk security and UI conventions.
- Precise control over token rotation, path sandboxing, transfer jobs and PostgreSQL persistence.
- Needs implementation of multipart upload/download resume, but the API flow is straightforward.
2. AList as a sidecar or reference implementation
- AList already has a Baidu Netdisk driver and supports storage mounting through its own server.
- Useful if we want WebDAV-like access or want to validate behavior quickly.
- Treat it as an internal sidecar behind the `baidu-netdisk` backend; do not expose AList WebUI as the UniDesk frontend.
- Watch license/upgrade/security posture before embedding it into production.
3. bypy as a Python worker
- Good for app-folder upload/download automation and quick scripts.
- Can run in a worker container for batch operations if we accept Python dependency and app-folder assumptions.
- Less ideal as the primary service API because UniDesk still needs its own auth state, job model and structured frontend.
4. BaiduPCS-Go as a worker
- Strong CLI for batch transfers and resume behavior.
- Could be invoked from the service for jobs after controlled login/config injection.
- Avoid making CLI config files the credential authority; PostgreSQL should remain authoritative.
5. Unofficial or cracked web APIs
- Avoid. They are unstable, hard to validate, and may violate Baidu terms or trigger account risk controls.
## Proposed User Service Contract
### Backend APIs
Expose a pure JSON control API first:
- `GET /health`: service, storage, auth, queue and Baidu API reachability summary.
- `GET /api/auth/status`: redacted configured/logged-in/auth-session summary.
- `POST /api/auth/device/start`: start QR/device login.
- `GET /api/auth/device/status?sessionId=...`: login state and QR metadata. OAuth `authorization_pending` and `slow_down` responses are normal pending states and must not be surfaced as frontend HTTP errors.
- `POST /api/auth/refresh`: force token refresh for diagnostics.
- `POST /api/auth/logout`: revoke local tokens and stop jobs.
- `GET /api/account`: user info and quota.
- `GET /api/files?dir=/apps/UniDeskBaiduNetdisk&start=0&limit=100`: directory listing.
- `GET /api/files/meta?fsids=...&dlink=0|1`: metadata, optionally dlink redacted by default.
- `POST /api/folders`: create folder through `method=create&isdir=1`.
- `POST /api/files/manage`: copy/move/rename/delete using `method=filemanager`.
- `POST /api/transfers/upload-from-path`: read a file inside the mounted staging directory and upload it to Baidu.
- `POST /api/transfers/download-to-path`: download a Baidu file to the staging directory.
- `POST /api/self-test`: create a tiny staging fixture, upload it, verify it appears in `/api/files`, download it back to staging and compare MD5.
- `GET /api/transfers`: list transfer jobs.
- `GET /api/transfers/{id}`: job detail, progress, retry and last error.
- `POST /api/transfers/{id}/cancel` and `POST /api/transfers/{id}/retry`.
- `GET /logs`: recent structured service logs with tokens/dlinks redacted.
If a future binary proxy is added, extend with:
- `POST /api/uploads/sessions` + chunk PUT/POST endpoints.
- `GET /api/downloads/{jobId}/stream` with Range support.
### PostgreSQL Tables
Minimum schema:
- `baidu_netdisk_accounts(id, baidu_uid, username, avatar_url, vip_type, root_path, created_at, updated_at)`.
- `baidu_netdisk_tokens(account_id, access_token_ciphertext, refresh_token_ciphertext, expires_at, scope, generation, last_refresh_at)`.
- `baidu_netdisk_auth_sessions(id, device_code_ciphertext, user_code, verification_url, qrcode_url, expires_at, poll_interval_seconds, status, error, created_at, updated_at)`.
- `baidu_netdisk_transfer_jobs(id, account_id, direction, status, local_path, remote_path, fs_id, size_bytes, bytes_done, part_size, block_list_json, uploadid, retry_count, error, created_at, updated_at)`.
- `baidu_netdisk_transfer_events(id, job_id, level, message, data_json, created_at)`.
Token encryption key should come from an environment variable such as `BAIDU_NETDISK_TOKEN_KEY`; no secrets should be committed.
### Container and Deployment
If deployed on D601, use the normal compute-node user-service boundary:
```json
{
"id": "baidu-netdisk",
"name": "Baidu Netdisk",
"providerId": "D601",
"description": "Containerized Baidu Netdisk storage gateway with QR/device login and transfer jobs.",
"repository": {
"url": "https://github.com/pikasTech/baidu-netdisk-unidesk",
"commitId": "<commit>",
"dockerfile": "Dockerfile",
"composeFile": "docker-compose.unidesk.yml",
"composeService": "baidu-netdisk",
"containerName": "baidu-netdisk-backend"
},
"backend": {
"nodeBaseUrl": "http://host.docker.internal:3295",
"nodeBindHost": "127.0.0.1",
"nodePort": 3295,
"proxyMode": "provider-gateway-http",
"frontendOnly": true,
"public": false,
"allowedMethods": ["GET", "HEAD", "POST", "DELETE"],
"allowedPathPrefixes": ["/health", "/logs", "/api/"],
"healthPath": "/health",
"timeoutMs": 30000
},
"development": {
"providerId": "D601",
"sshPassthrough": true,
"worktreePath": "/home/ubuntu/baidu-netdisk-unidesk"
},
"frontend": {
"route": "/apps/baidu-netdisk",
"integrated": true
}
}
```
If deployed on main server, use a Compose service name such as `http://baidu-netdisk:4244` and add a root `docker-compose.yml` service. Main-server deployment is justified only if UniDesk needs central storage on the main server; otherwise D601 or another compute node is cleaner.
### Frontend Page
Add a dedicated `src/components/frontend/src/baidu-netdisk.tsx` page and route tab:
- Login panel: QR image from `qrcode_url`, user code, expires timer, poll status, refresh QR and logout buttons.
- Account cards: username, UID, quota used/total, VIP state, app root path.
- File browser: breadcrumb rooted at `/apps/<appName>`, paginated table, folder creation, rename/delete/move controls.
- Transfer panel: upload-from-path form, download-to-path form, job rows, progress bars, speed, ETA, retry/cancel buttons.
- Safety text: private backend mapping, token storage redacted, no direct public Baidu token exposure.
- Raw JSON only behind explicit buttons, following existing user-service conventions.
## Acceptance Plan
Focused checks after implementation:
- `bun scripts/cli.ts microservice list` shows `baidu-netdisk`, private backend, target provider and container summary.
- `bun scripts/cli.ts microservice health baidu-netdisk` returns `ok=true`, `service=baidu-netdisk`, `storage=postgres` and redacted auth state.
- `bun scripts/cli.ts microservice proxy baidu-netdisk /api/auth/device/start --method POST` returns a login session with QR/user-code metadata but no token.
- After manual QR authorization, `/api/account` and `/api/files?dir=<root>` return user/quota/list data.
- Upload/download tests can use `bun scripts/cli.ts microservice proxy baidu-netdisk /api/self-test --method POST --raw`; the response must include a remote path in the app root, an `fsId`, succeeded upload/download jobs, and matching `expectedMd5`/`downloadedMd5`.
- Public port probes must fail for the service port; frontend access only through UniDesk.
- Playwright verifies `/app/baidu-netdisk/` renders shell, login card, account/quota, file browser, transfer panel and no naked JSON.
Full regression should later add `microservice:catalog-baidu-netdisk`, `microservice:baidu-netdisk-health`, login-state checks and `frontend:baidu-netdisk-integrated-visible` to `scripts/src/e2e.ts`.
## Open Risks
- Baidu app review/permissions may block upload/download until the app is approved and scoped correctly.
- Device Code QR expires quickly; frontend needs clear countdown and refresh behavior.
- Refresh token is single-use per Netdisk docs; a race can force re-login if token rotation is not serialized.
- Browser direct download is not a v1 fit because official dlink download requires `User-Agent: pan.baidu.com` and current UniDesk proxy cannot stream large binary safely.
- Large upload/download jobs need resumable local job records and cleanup of temporary chunks/staged files.
- Using AList/BaiduPCS-Go/byPy may introduce third-party license and maintenance risk; keep their configs/token caches derived from UniDesk PostgreSQL, not authoritative.
## Sources
- Baidu Netdisk Open Platform overview: https://pan.baidu.com/union/doc/nksg0sbfs
- Authorization Code mode: https://pan.baidu.com/union/doc/al0rwqzzl
- Device Code mode: https://pan.baidu.com/union/doc/fl1x114ti
- User info: https://pan.baidu.com/union/doc/pksg0s9ns
- Quota: https://pan.baidu.com/union/doc/Cksg0s9ic
- File list: https://pan.baidu.com/union/doc/nksg0sat9
- File metadata/dlink: https://pan.baidu.com/union/doc/Fksg0sbcm
- Precreate upload: https://pan.baidu.com/union/doc/3ksg0s9r7
- Multipart upload: https://pan.baidu.com/union/doc/nksg0s9vi
- Create file/folder: https://pan.baidu.com/union/doc/rksg0sa17
- Single-step upload: https://pan.baidu.com/union/doc/olkuuy5kz
- Locate upload domain: https://pan.baidu.com/union/doc/Mlvw5hfnr
- Download: https://pan.baidu.com/union/doc/pkuo3snyp
- File manager: https://pan.baidu.com/union/doc/mksg0s9l4
- AList Baidu Netdisk driver docs: https://alist-repo.github.io/docs/guide/drivers/baidu.html
- bypy project page: https://pypi.org/project/bypy/
- BaiduPCS-Go repository: https://github.com/qjfoidnh/BaiduPCS-Go