diff --git a/.gitignore b/.gitignore index 3e081cb..7414300 100644 --- a/.gitignore +++ b/.gitignore @@ -20,3 +20,6 @@ deployable/runtime/ deployable/backups/ deployable/wheelhouse/ deployable/images/ +**/credential-keyring.json +**/compose.broker.json +**/compose.rollback.json diff --git a/deployable/CREDENTIAL_MIGRATION.md b/deployable/CREDENTIAL_MIGRATION.md index 636b4ff..94b5429 100644 --- a/deployable/CREDENTIAL_MIGRATION.md +++ b/deployable/CREDENTIAL_MIGRATION.md @@ -12,7 +12,7 @@ | 2 | 集中凭据Broker及刷新 | 应用/用户锁、持锁重读、版本CAS、响应校验、刷新unknown;新授权不受旧刷新覆盖 | 本次完成本地开发测试 | | 3 | OAuth与认证领取 | 共享state事务、实际用户核验、绑定代次、只返回access的独立DTO及领取认证 | 本次完成本地开发测试,显式broker模式启用 | | 4 | ECS权威解绑与消息任务关联 | 停发Token、失效事务/任务、连接和旧码联动,多实例/异常恢复 | 本次完成本地开发测试,broker模式生效 | -| 5 | 配置部署、迁移与旧接口退出 | 密钥挂载、产品权限配置、客户端对接、停用旧刷新/claim及代理、ARM64和真实平台验收 | 待开发 | +| 5 | 配置部署、迁移与旧接口退出 | 密钥挂载、产品权限配置、客户端对接、停用旧刷新/claim及代理、ARM64和真实平台验收 | 已交付现有部署准备/检查/构建/切换/回退工具;现机及客户端验收待执行 | 产品经理决定功能和Scope清单。阶段1不预设具体权限,不生成真实凭据、不读取现有设备、不调用平台。 @@ -151,3 +151,22 @@ Windows / Python 3.12.10,平台桩、临时证书/密钥、SQLite及带Lua的F 最后完整回归:**213 passed,3 skipped,30 warnings**(45.87秒)。3个跳过项均需要真实PostgreSQL连接,warnings为既有uvicorn/websockets旧API弃用提示(新增真实回环连接增加提示次数);relay模块AST、文档和diff检查通过。 生产部署、真实Redis/PostgreSQL多实例、ARM64镜像、平台实际撤权/Scope/容量和客户端安装升级仍待阶段5及现场验收。平台已接收的请求不可靠解绑撤回,已发access仍可能有效到期。 + +## 阶段5工具交付:保留现有部署的迁移接线(2026-11-04) + +用户提供现场Docker/Compose、7份配置路径、不同组件镜像ID及已生成备份,数据库dump目录解析通过,并建立4个原镜像回滚标签。该信息仅是当次现场证据,不等同真实恢复或新架构上线验收。 + +新增标准库工具`scripts/broker_deploy.py`与[现有ECS操作指导](EXISTING_ECS_BROKER_DEPLOY.md): + +- `preflight`只读识别实际容器标签、Compose文件、项目与ARM64镜像,不写文件/密钥,不输出解析配置,不重启。 +- `prepare`创建运行目录外的新私有计划:保留全部原Compose、CA、数据库卷和异构镜像;生成独立keyring/准入Secret和仅影响connector/router的叠加,复制无凭据源代码快照。不修改现有.env或初始化数据库。 +- `check`拒绝配置/源码/密钥/镜像漂移,核对原挂载、网络、数据库卷及内部路径拒绝规则;密钥权限/UID要求明确。不打印展开后的Secret。 +- `build`复用确切现场ARM64运行镜像,以禁网/禁止拉取方式构建候选;不替换原镜像、不下载依赖。导入smoke与两角色配置/密钥读取检查通过后才标记候选就绪;配置通过stdin送入无网络临时容器,不放在命令行或日志。 +- `activate`需明确声明v2客户端就绪,重新检查候选及运行基线后仅up connector/router;缺标志时在发切换命令前拒绝。健康通过后仍需实际业务验收。 +- `rollback`有独立停用broker声明,按每组件原实际镜像回退connector/router,不恢复旧refresh或删除数据库。保持新密钥和凭据数据,避免覆盖平台已轮换状态。 + +默认部署配置不自动切换;本次工具通过本地模拟Docker编排测试,但开发机没有真实Docker/ARM64环境,不能宣称已构建或验证候选镜像。新客户端尚未适配,现机当前只应执行只读preflight或经产品权限确认的准备/构建,不能启用broker。 + +本地新增工具策略测试26项,覆盖7份配置保留、独立旧镜像、准备不启动服务、密钥不覆盖、路径/配置漂移、私有入口拒绝、产品权限格式、候选失败不标就绪、候选换标签拒绝、客户端就绪门槛、离线构建和仅重建两服务。最后完整回归:**239 passed,3 skipped,30 warnings**(46.94秒);跳过项需要真实PostgreSQL,warnings仍为旧依赖弃用提示。宿主脚本通过Python3.7语法兼容检查、CLI help及diff检查。 + +测试中Docker命令均为受控桩,不声称实际执行Compose合并、ARM64构建或密钥挂载。真实Compose/ARM64、PostgreSQL/Redis、多用户平台、客户端升级和密钥恢复仍待执行,因此阶段5总体尚未验收完成。 diff --git a/deployable/EXISTING_ECS_BROKER_DEPLOY.md b/deployable/EXISTING_ECS_BROKER_DEPLOY.md new file mode 100644 index 0000000..ff819c8 --- /dev/null +++ b/deployable/EXISTING_ECS_BROKER_DEPLOY.md @@ -0,0 +1,103 @@ +# 现有ARM64 ECS:集中凭据模式部署指导 + +日期:2026-11-04。适用已运行的`hainuo-relay`七服务项目。使用新脚本`deployable/scripts/broker_deploy.py`,不重新执行init,不复制新Compose替换旧部署,也不操作其他项目。 + +## 1. 已取得的现场基线 + +- 部署目录:`/home/cnooc/hainuo-offline-arm64/hainuo-relay`。 +- Docker Engine 29.9.0 / Linux ARM64,Compose v2.40.3;cnooc账号操作Docker需要sudo。 +- 当前运行Compose由容器标签记载的7份配置叠加。脚本动态读取并保留全套文件,不要求人工少选几份。 +- 备份:`/home/cnooc/hainuo-relay-backups/20261104-115742`,数据库dump已完成`pg_restore --list`解析;这不等于真实恢复演练。 +- 已保留`rollback-connector-20261104`、`rollback-router-20261104`、`rollback-longconn-20261104`、`rollback-signer-20261104`;connector实际镜像与router/longconn不同,回退必须分别保留。 + +这是用户提供的当次运行信息,后续有变化时以预检查为准。不要提交/上传备份、.env、密钥或实际Token。 + +## 2. 现在可以执行:下载代码并只读预检查 + +如果目标暂存目录尚不存在: + +```bash +cd /home/cnooc +git clone --branch master git@git.yssing.cloud:ex_tianzy/ecs----.git ecs-relay-service-staging +cd /home/cnooc/ecs-relay-service-staging +git log -1 --oneline +sudo python3 deployable/scripts/broker_deploy.py preflight \ + --deployment-root /home/cnooc/hainuo-offline-arm64/hainuo-relay +``` + +如果之前已经clone,先在该暂存仓库执行`git status --short --branch`确认没有个人改动,再使用`git pull --ff-only`更新;clone或pull失败时停止,不在运行目录内强制reset。仓库当前SSH地址沿用已核验的remote,页面显示名与历史路径不同不影响这一路由,若服务器提示不存在则回报错误,不在命令行粘贴访问Token。 + +脚本只依赖Python标准库(Python≥3.7)及现有Docker/Compose,不要求服务器安装本机开发venv、pytest或cryptography。`preflight`读取容器/镜像/Compose,仅输出项目、文件数量、实际镜像及模式,不打印解析后的配置、Secret或Token;不写密钥、不构建、不重启。 + +**当前客户端尚未适配v2,执行到这里即可。** 后续prepare/check/build同样不启用新模式,但需要产品权限集合;activate必须等新客户端和测试计划就绪,不能因为代码测试通过就切换现网。 + +## 3. 产品权限与准备阶段 + +客户端产品经理确认功能范围、读写边界和集中授权体验,开发映射API/Scope、管理员确认发布生效后,将批准的权限profile映射保存为本机JSON文件。格式为对象:产品权限版本名映射到Scope字符串数组。不要用本方案中的任意示例权限替代产品决定,不用每接口单独授权。 + +假定批准文件已放到`/home/cnooc/work-product-profiles.json`,选择一个**尚不存在、位于运行目录之外**的准备目录: + +```bash +cd /home/cnooc/ecs-relay-service-staging +sudo python3 deployable/scripts/broker_deploy.py prepare \ + --deployment-root /home/cnooc/hainuo-offline-arm64/hainuo-relay \ + --profiles /home/cnooc/work-product-profiles.json \ + --output /home/cnooc/hainuo-broker-prepared-20261104 +``` + +prepare生成独立的root私有目录、随机AES密钥keyring、独立router准入Secret、broker Compose叠加配置、精确镜像回滚叠加、基线和源码快照。密钥文件600且归属服务UID10001;目录700。**不改运行.env、CA、connector签名密钥、绑定、数据库卷,也不初始化数据库或服务。** 输出目录已存在时拒绝覆盖,不自动重新生成密钥。 + +这份目录包含真实密钥,不能上传聊天/Git,不能与普通源码归档一起公开分发。数据库备份和密钥需要分开保护;首次prepare产生的密钥还不在之前现场备份中,启用前必须落实单独密钥备份与恢复。丢失密钥会使集中凭据无法解密。 + +## 4. 检查与离线构建:仍不切换 + +```bash +cd /home/cnooc/ecs-relay-service-staging +sudo python3 deployable/scripts/broker_deploy.py check \ + --plan /home/cnooc/hainuo-broker-prepared-20261104 + +sudo python3 deployable/scripts/broker_deploy.py build \ + --plan /home/cnooc/hainuo-broker-prepared-20261104 +``` + +check确认旧.env/7份Compose/准备配置/密钥/源码快照无漂移,各服务实际镜像仍匹配,数据库卷/网络/原挂载保持,router不持有管理员/事件/App Secret,edge有已审阅的内部接口拒绝规则。只在内存解析Compose,不把展开后的Secret打印出来。未识别配置时停止,不自动“修复”现网。 + +build基于**当前connector的确切ARM64运行镜像**创建候选,复制新版relay代码;`--network none --pull=false`,不下载Python包或基础镜像,不覆盖原`hainuo-relay:0.1.0`。构建阶段验证新模块可导入,完成后记录候选镜像ID;不使用依赖不明的新基础镜像,不替换正在运行的应用。 + +这种方式复用现场已有运行依赖,因此必须在ECS执行并验证候选构建/导入。开发机的Docker命令桩只能证明命令编排,不能代替ARM64镜像构建、真实PostgreSQL、真实Redis或业务联调。 + +## 5. 启用前门槛与切换 + +必须已经具备:新PC的集中凭据领取/v2握手与任务上下文、产品Scope生效、准确HTTPS回调白名单、候选镜像真实构建通过、数据库及密钥备份、两用户测试资源、升级与回滚窗口。客户端access只内存缓存,refresh仅在ECS;平台已发Bearer是否即时撤销仍需实测。 + +在上述条件满足后,才允许: + +```bash +cd /home/cnooc/ecs-relay-service-staging +sudo python3 deployable/scripts/broker_deploy.py activate \ + --plan /home/cnooc/hainuo-broker-prepared-20261104 \ + --client-v2-ready +``` + +该标志是操作方对就绪状态的明确声明,不是脚本自动验收客户端。没有该标志,脚本在发任何切换Docker命令前拒绝执行。 + +只重建connector/router容器,保留原数据库、Redis、edge、longconn、签发服务和其他知识库项目;不创建第二个机器人长连接接收器。旧有longconn继续内部事件协议,不需要持有router准入密钥。broker切换将关闭旧refresh/claim/业务代理,旧PC会不兼容;切换前完成客户端安排,使用一次新授权建立集中凭据。 + +健康检查通过只代表容器就绪,随后必须验证:本人OAuth和实际user_info、access-only领取、集中刷新、PC直连业务、原消息回复、通知、解绑/旧码/旧回包和两用户ACL。没有结果不能宣称上线验收完成。 + +启用后继续使用该plan和叠加配置管理connector/router;直接运行原7份Compose的up可能撤销broker配置。不要删除准备目录或重跑init来恢复连接。长期维护入口统一化与运行配置纳管需在正式发布交接中明确。 + +## 6. 应急容器回退 + +若切换失败,先停止broker客户端继续使用,按记录的每服务镜像回退: + +```bash +cd /home/cnooc/ecs-relay-service-staging +sudo python3 deployable/scripts/broker_deploy.py rollback \ + --plan /home/cnooc/hainuo-broker-prepared-20261104 \ + --stop-broker-use +``` + +回退仅对connector/router执行up,不操作数据库卷或恢复陈旧refresh。脚本验证原配置和非目标服务未漂移后使用实际旧镜像;原longconn/签发镜像单独保留。**容器回退不等于用户授权无损回退**:服务端已轮换的refresh不能交回旧PC继续刷新,旧客户端需要协调模式/重新授权。保留集中凭据数据库及密钥,不用旧数据库备份覆盖新的平台轮换状态。 + +activate失败不会自动猜测执行回滚,停止后按此流程处置;如果配置、密钥或运行版本漂移导致check拒绝,应先审阅差异,不用强制选项绕过。 diff --git a/deployable/README.md b/deployable/README.md index a789b47..38230f3 100644 --- a/deployable/README.md +++ b/deployable/README.md @@ -4,6 +4,8 @@ 阶段4新增[权威解绑、连接准入和任务生命周期](BINDING_TASK_LIFECYCLE.md),broker模式要求router独立准入凭据及PC v2握手。默认legacy不变;部署接线和客户端适配未完成前不要直接切换生产模式。 +已有ECS升级请使用[保留现有部署的broker迁移指导](EXISTING_ECS_BROKER_DEPLOY.md),先运行只读preflight;工具可生成独立密钥和Compose叠加,构建候选但不自动启用。新客户端未就绪时不要使用activate,也不要重新init已有CA/数据库。 + 2026-10-28 新增[客户端主动消息与授权卡片](NOTIFICATIONS.md),本地代码已实现,现网部署和卡片验收待完成。 本目录用于新环境部署,源码对应 2026-10-27 ECS connector/router/longconn 导出版本。协议和业务源码保持原样。仓库其他快照目录用于追溯,不需要手工合并。 diff --git a/deployable/scripts/broker_deploy.py b/deployable/scripts/broker_deploy.py new file mode 100644 index 0000000..6fa9d6f --- /dev/null +++ b/deployable/scripts/broker_deploy.py @@ -0,0 +1,398 @@ +#!/usr/bin/env python3 +"""Existing-deployment broker migration: stdlib-only host tool, Python >=3.7. + +Never initialize identities, edit the live .env, or dump resolved Compose output. +prepare/check/build do not recreate services. Activation requires a v2 client. +""" +import argparse +import base64 +import hashlib +import json +import os +from pathlib import Path +import re +import secrets +import shutil +import subprocess +import sys + + +class DeployError(RuntimeError): + pass + + +def run(args, json_result=False, optional=False, input_text=None): + result = subprocess.run(args, input=input_text, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True) + if result.returncode: + if optional: + return None + raise DeployError("Docker/Git command failed; raw output withheld to protect configuration") + try: + return json.loads(result.stdout) if json_result else result.stdout.strip() + except ValueError: + raise DeployError("invalid command response") from None + + +def digest(path): + return hashlib.sha256(path.read_bytes()).hexdigest() + + +def write_json(path, value): + # All outputs belong to the newly created, private migration directory. + fd = os.open(str(path), os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600) + with os.fdopen(fd, "w", encoding="utf-8") as stream: + json.dump(value, stream, ensure_ascii=False, indent=2) + stream.write("\n") + + +def inspect(container): + template = ('{"id":{{json .Id}},"image":{{json .Image}},' + '"image_ref":{{json .Config.Image}},"labels":{{json .Config.Labels}}}') + return run(["docker", "inspect", container, "--format", template], True) + + +def service(project, name): + values = run(["docker", "ps", "-q", "--filter", "label=com.docker.compose.project="+project, + "--filter", "label=com.docker.compose.service="+name]).splitlines() + if len(values) != 1: + raise DeployError("expected exactly one running relay service: "+name) + return inspect(values[0]) + + +def compose(plan, override=None): + args = ["docker", "compose", "--project-directory", plan["deployment_root"], + "--env-file", str(Path(plan["deployment_root"])/".env"), "-p", plan["project"]] + for file in plan["compose_files"]: + args.extend(["-f", file]) + if override: + args.extend(["-f", str(override)]) + return args + + +def profiles_from(path): + try: + data = json.loads(path.read_text(encoding="utf-8")) + if not isinstance(data, dict) or not data or len(data) > 32: + raise ValueError() + for name, scopes in data.items(): + if (not re.fullmatch(r"[A-Za-z0-9_-]{1,64}", name) or not isinstance(scopes, list) + or not scopes or len(scopes) > 128 or any(not isinstance(s, str) + or not re.fullmatch(r"[A-Za-z0-9_:.-]{1,256}", s) for s in scopes)): + raise ValueError() + return {name: sorted(set(scopes)) for name, scopes in data.items()} + except (OSError, ValueError, TypeError): + raise DeployError("invalid product permission profiles; use the product-approved JSON object") from None + + +def validate_overlay(base, merged, candidate): + old, new = base["services"], merged["services"] + if set(old) != set(new) or set(old) != {"connector","router","longconn","enrollment-signer","postgres","redis","edge"}: + raise DeployError("unexpected Compose services; migration requires review") + for name in set(old)-{"connector","router"}: + if old[name] != new[name]: + raise DeployError("overlay changes an unrelated relay service: "+name) + for name in ("connector","router"): + for field in set(old[name]) | set(new[name]): + if field not in {"environment","image","volumes"} and old[name].get(field) != new[name].get(field): + raise DeployError("overlay changes non-migration service settings") + if new[name].get("image") != candidate: + raise DeployError("candidate image mismatch") + env = new[name]["environment"] + for key, value in old[name].get("environment", {}).items(): + if key not in {"HN_CREDENTIAL_MODE","HN_ROUTER_ADMISSION_SECRET","HN_CREDENTIAL_KEYRING_FILE", + "HN_CREDENTIAL_PERMISSION_PROFILES","HN_CONNECTOR_INTERNAL_URL","HN_FEISHU_APP_ID","HN_OAUTH_ENABLED"}: + if env.get(key) != value: + raise DeployError("existing environment changed: "+key) + old_volumes = {v["target"]:v for v in old[name].get("volumes", [])} + new_volumes = {v["target"]:v for v in new[name].get("volumes", [])} + if any(new_volumes.get(key) != value for key,value in old_volumes.items()): + raise DeployError("existing storage or trust mount changed") + expected_targets = set(old_volumes) | ({"/run/hainuo-credentials/keyring.json"} if name == "connector" else set()) + if set(new_volumes) != expected_targets or env.get("HN_CREDENTIAL_MODE") != "broker": + raise DeployError("unexpected migration mount or credential mode") + if merged.get("volumes") != base.get("volumes") or merged.get("networks") != base.get("networks"): + raise DeployError("database volume or network definitions changed") + env = new["connector"]["environment"] + if (env.get("HN_FEISHU_APP_ID") != old["connector"]["environment"].get("HN_FEISHU_APP_ID") + or new["router"]["environment"].get("HN_FEISHU_APP_ID") != env.get("HN_FEISHU_APP_ID") + or new["router"]["environment"].get("HN_ENVIRONMENT") != env.get("HN_ENVIRONMENT") + or env.get("HN_CREDENTIAL_KEYRING_FILE") != "/run/hainuo-credentials/keyring.json" + or new["router"]["environment"].get("HN_CONNECTOR_INTERNAL_URL") != "http://connector:8000"): + raise DeployError("application/environment or credential routing mismatch") + if (not env.get("HN_DATABASE_URL", "").startswith(("postgresql://","postgres://")) + or env.get("HN_ENVIRONMENT") != "production" + or str(env.get("HN_FEISHU_ENABLED", "")).lower() != "true" + or str(env.get("HN_OAUTH_ENABLED", "")).lower() != "true"): + raise DeployError("broker production platform/database configuration incomplete") + for key in ("HN_ADMIN_SECRET","HN_INTERNAL_SECRET","HN_FEISHU_APP_SECRET"): + if env["HN_ROUTER_ADMISSION_SECRET"] == env.get(key): + raise DeployError("router admission credential must be independent") + if any(new["router"]["environment"].get(k) for k in ("HN_ADMIN_SECRET","HN_INTERNAL_SECRET","HN_FEISHU_APP_SECRET")): + raise DeployError("router has unrelated privileged credentials") + mounts = [v for v in new["connector"].get("volumes", []) if v["target"] == "/run/hainuo-credentials/keyring.json"] + if len(mounts) != 1 or not mounts[0].get("read_only"): + raise DeployError("missing read-only credential keyring mount") + + +def preflight(args): + deployment = Path(args.deployment_root).resolve() + current = inspect(args.connector_container) + labels = current["labels"] + if Path(labels["com.docker.compose.project.working_dir"]).resolve() != deployment: + raise DeployError("deployment path differs from the running connector") + files = [Path(p) for p in labels["com.docker.compose.project.config_files"].split(",")] + if any(not p.is_file() or p.resolve().parent != deployment for p in files): + raise DeployError("missing or external Compose file; review deployment layout") + plan = {"deployment_root":str(deployment),"project":labels["com.docker.compose.project"], + "compose_files":[str(p) for p in files]} + base = run(compose(plan)+["config","--format","json"],True) + names = set(base["services"]) + if names != {"connector","router","longconn","enrollment-signer","postgres","redis","edge"}: + raise DeployError("unexpected Compose services") + architecture = run(["docker","image","inspect",current["image"],"--format","{{.Architecture}}"]) + if architecture != "arm64": + raise DeployError("expected current ARM64 runtime") + for name in ("connector","router","longconn","enrollment-signer"): + entry = service(plan["project"],name) + print(name+": "+entry["image"]) + env = base["services"]["connector"].get("environment",{}) + print("Project: "+plan["project"]) + print("Compose files: "+str(len(files))) + print("Current credential mode: "+env.get("HN_CREDENTIAL_MODE","legacy")) + print("Preflight complete; no files, keys or services changed. Resolved settings withheld.") + + +def prepare(args): + if os.name != "nt" and os.geteuid() != 0: + raise DeployError("run preparation with sudo for protected key ownership") + deployment = Path(args.deployment_root).resolve() + output = Path(args.output) + source = Path(args.source_root).resolve() + profiles = profiles_from(Path(args.profiles)) + if output.exists() or output.is_symlink() or not output.is_absolute(): + raise DeployError("output must be a new absolute migration directory") + output = output.resolve() + if any(c in str(output) for c in ("\n","\r","$")): + raise DeployError("migration path contains unsupported Compose interpolation characters") + if output == deployment or deployment in output.parents: + raise DeployError("migration output must be outside the running deployment") + current = inspect(args.connector_container) + labels = current["labels"] + if Path(labels["com.docker.compose.project.working_dir"]).resolve() != deployment: + raise DeployError("running deployment directory does not match the supplied root") + files = [Path(p) for p in labels["com.docker.compose.project.config_files"].split(",")] + if any(not p.is_absolute() or p.resolve().parent != deployment or not p.is_file() for p in files): + raise DeployError("Compose files must belong to the existing deployment root") + if not (deployment/".env").is_file() or not (source/"relay/credential_api.py").is_file(): + raise DeployError("missing live configuration or staged broker source") + project = labels["com.docker.compose.project"] + if not re.fullmatch(r"[a-z0-9][a-z0-9_-]*", project): + raise DeployError("invalid Compose project") + roles = {name:service(project,name) for name in ("connector","router","longconn","enrollment-signer","edge")} + if any(not re.fullmatch(r"sha256:[a-f0-9]{64}",r["image"]) for r in roles.values()): + raise DeployError("invalid running image identity") + if roles["connector"]["id"] != current["id"]: + raise DeployError("connector inventory changed during preparation") + architecture = run(["docker","image","inspect",current["image"],"--format","{{.Architecture}}"]) + if architecture != "arm64": + raise DeployError("this migration targets the verified ARM64 runtime only") + plan_id = secrets.token_hex(8) + candidate = "hainuo-relay:broker-candidate-"+plan_id + plan = {"format":1,"plan_id":plan_id,"deployment_root":str(deployment),"project":project, + "compose_files":[str(p) for p in files],"roles":roles,"candidate_image":candidate, + "base_image":current["image"],"architecture":architecture, + "baseline_hashes":{str(p):digest(p) for p in [deployment/".env",*files]}} + # Durable tags preserve each heterogeneous running application image. + for name in ("connector","router","longconn","enrollment-signer"): + run(["docker","tag",roles[name]["image"],"hainuo-relay:broker-rollback-"+plan_id+"-"+name]) + base = run(compose(plan)+["config","--format","json"],True) + if base["services"]["connector"].get("environment",{}).get("HN_CREDENTIAL_MODE","legacy") != "legacy": + raise DeployError("already broker: preserve the existing keyring and migration plan") + output.mkdir(mode=0o700,parents=False) + os.chmod(str(output),0o700) + keyring = {"active_key_id":"initial-v1","keys":{"initial-v1":base64.b64encode(os.urandom(32)).decode("ascii")}} + write_json(output/"credential-keyring.json",keyring) + if os.name != "nt": + os.chown(str(output/"credential-keyring.json"),10001,10001) + admission = secrets.token_hex(32) + overlay = {"services":{ + "connector":{"image":candidate,"environment":{"HN_CREDENTIAL_MODE":"broker", + "HN_CREDENTIAL_KEYRING_FILE":"/run/hainuo-credentials/keyring.json", + "HN_CREDENTIAL_PERMISSION_PROFILES":json.dumps(profiles,separators=(",", ":")), + "HN_ROUTER_ADMISSION_SECRET":admission,"HN_OAUTH_ENABLED":"true"}, + "volumes":[{"type":"bind","source":str(output/"credential-keyring.json"), + "target":"/run/hainuo-credentials/keyring.json","read_only":True}]}, + "router":{"image":candidate,"environment":{"HN_CREDENTIAL_MODE":"broker", + "HN_ROUTER_ADMISSION_SECRET":admission,"HN_CONNECTOR_INTERNAL_URL":"http://connector:8000", + "HN_FEISHU_APP_ID":"${HN_FEISHU_APP_ID:?Configure existing application}", + "HN_ENVIRONMENT":"${HN_ENVIRONMENT:?Configure existing environment}"}}}} + write_json(output/"compose.broker.json",overlay) + write_json(output/"compose.rollback.json",{"services":{name:{"image":roles[name]["image"]} + for name in ("connector","router","longconn","enrollment-signer")}}) + payload = output/"build" + payload.mkdir(mode=0o700) + for path in (source/"relay").rglob("*"): + if path.is_symlink(): + raise DeployError("symlink in staged source") + if path.is_file() and path.suffix in {".py",".js",".css",".html"} and "__pycache__" not in path.parts: + target = payload/path.relative_to(source) + target.parent.mkdir(parents=True,exist_ok=True) + shutil.copyfile(str(path),str(target)) + os.chmod(str(target),0o644) + dockerfile = ('ARG BASE_IMAGE\nFROM ${BASE_IMAGE}\nWORKDIR /app\nUSER 0:0\n' + 'COPY --chown=10001:10001 relay /app/relay\nUSER 10001:10001\n' + 'RUN python -c "import relay.credential_api, relay.binding_state, relay.router"\n' + 'ENTRYPOINT ["python", "-m", "relay.cli"]\n') + (payload/"Dockerfile").write_bytes(dockerfile.encode("utf-8")) + plan["payload_hashes"] = {str(p.relative_to(output)):digest(p) for p in payload.rglob("*") if p.is_file()} + plan["prepared_hashes"] = {p:digest(output/p) for p in ( + "credential-keyring.json","compose.broker.json","compose.rollback.json")} + write_json(output/"plan.json",plan) + merged = run(compose(plan,output/"compose.broker.json")+["config","--format","json"],True) + validate_overlay(base,merged,candidate) + print("PREPARED: "+str(output)+" (services unchanged; private files must not be shared)") + + +def load_plan(root): + plan = json.loads((root/"plan.json").read_text(encoding="utf-8")) + if plan.get("format") != 1 or not re.fullmatch(r"[a-f0-9]{16}",plan.get("plan_id","")): + raise DeployError("unsupported migration plan") + return plan + + +def check(root, plan, baseline=True): + if os.name != "nt": + if root.stat().st_mode & 0o077: + raise DeployError("migration directory must not be accessible to other users") + for name in ("credential-keyring.json","compose.broker.json","compose.rollback.json","plan.json"): + if (root/name).stat().st_mode & 0o077: + raise DeployError("migration private file permissions are too broad") + if (root/"credential-keyring.json").stat().st_uid != 10001: + raise DeployError("credential keyring owner must match service UID 10001") + for path, expected in plan["baseline_hashes"].items(): + if digest(Path(path)) != expected: + raise DeployError("live Compose/configuration changed; review before proceeding") + for path, expected in plan["payload_hashes"].items(): + if digest(root/path) != expected: + raise DeployError("staged source changed after preparation") + for path, expected in plan["prepared_hashes"].items(): + if digest(root/path) != expected: + raise DeployError("prepared keyring or overlay changed; do not regenerate credential keys") + for name, old in plan["roles"].items(): + current = service(plan["project"],name) + if (baseline or name not in {"connector","router"}) and current["image"] != old["image"]: + raise DeployError("running service image changed since preparation: "+name) + base = run(compose(plan)+["config","--format","json"],True) + merged = run(compose(plan,root/"compose.broker.json")+["config","--format","json"],True) + validate_overlay(base,merged,plan["candidate_image"]) + # Require the current edge's explicit private-path block, before catch-all handling. + mounts = base["services"]["edge"].get("volumes",[]) + caddy = next((v["source"] for v in mounts if v.get("target") == "/etc/caddy/Caddyfile" and v.get("type") == "bind"),None) + if not caddy: + raise DeployError("cannot verify current edge configuration") + text = Path(caddy).read_text(encoding="utf-8") + rule = re.search(r"@([A-Za-z0-9_-]+)\s+path\s+/admin/\*\s+/internal/\*\s+/ready",text) + deny = re.search(r"respond\s+@"+re.escape(rule.group(1))+r"\s+404",text) if rule else None + first_handle = re.search(r"\bhandle(?:_path)?\b",text) + if not deny or (first_handle and deny.start() > first_handle.start()): + raise DeployError("edge private-path block is not the reviewed configuration") + print("CHECKED: existing Compose/storage/images retained; no services restarted") + + +def build(root,plan): + check(root,plan) + if run(["docker","image","inspect",plan["candidate_image"]],optional=True) is not None: + raise DeployError("candidate image already exists; do not overwrite a built candidate") + base_tag = "hainuo-relay:broker-base-"+plan["plan_id"] + run(["docker","tag",plan["base_image"],base_tag]) + run(["docker","build","--pull=false","--network","none","--build-arg","BASE_IMAGE="+base_tag, + "-t",plan["candidate_image"],str(root/"build")]) + image = run(["docker","image","inspect",plan["candidate_image"],"--format","{{.Id}}"]) + run(["docker","run","--rm","--network","none","--read-only","--entrypoint","python", + plan["candidate_image"],"-c","import relay.credential_api, relay.binding_state, relay.device_lifecycle"]) + runtime_check(root,plan) + write_json(root/"candidate.json",{"image":image,"tag":plan["candidate_image"]}) + print("BUILT: "+plan["candidate_image"]+" (offline runtime reuse; not activated)") + + +def runtime_check(root,plan): + merged = run(compose(plan,root/"compose.broker.json")+["config","--format","json"],True) + # Actual secrets travel via stdin only: no command-line values, env dump, + # database connection or platform request. The container has no network. + code = ("import json,os,sys\nfrom pathlib import Path\nfrom relay.config import Settings\n" + "from relay.credential_store import CredentialKeyring\n" + "try:\n data=json.load(sys.stdin)\n os.environ.update(data['environment'])\n" + " s=Settings(role=data['role'])\n" + " if data['role']=='connector': CredentialKeyring.from_file(s.credential_keyring_file)\n" + "except Exception:\n sys.exit(1)\n") + for role in ("connector","router"): + env = {key:str(value) for key,value in merged["services"][role]["environment"].items() if value is not None} + run(["docker","run","--rm","-i","--network","none","--read-only","--entrypoint","python", + "-v",str(root/"credential-keyring.json")+":/run/hainuo-credentials/keyring.json:ro", + plan["candidate_image"],"-c",code],input_text=json.dumps({"role":role,"environment":env})) + + +def activate(root,plan,ready): + if not ready: + raise DeployError("v2 client not confirmed ready; only prepare/check/build are allowed now") + check(root,plan) + built = json.loads((root/"candidate.json").read_text(encoding="utf-8")) + image = run(["docker","image","inspect",plan["candidate_image"],"--format","{{.Id}}"]) + if built != {"image":image,"tag":plan["candidate_image"]}: + raise DeployError("candidate image identity changed") + runtime_check(root,plan) + run(compose(plan,root/"compose.broker.json")+["up","-d","--no-deps","--no-build","--pull","never", + "--wait","--wait-timeout","180","connector","router"]) + print("ACTIVATED: connector/router only. Verify real OAuth, v2 client, unbind and task routing.") + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + sub = parser.add_subparsers(dest="action",required=True) + flight = sub.add_parser("preflight") + flight.add_argument("--deployment-root",required=True) + flight.add_argument("--connector-container",default="hainuo-relay-connector-1") + prep = sub.add_parser("prepare") + prep.add_argument("--deployment-root",required=True) + prep.add_argument("--source-root",default=str(Path(__file__).resolve().parents[1])) + prep.add_argument("--profiles",required=True) + prep.add_argument("--output",required=True) + prep.add_argument("--connector-container",default="hainuo-relay-connector-1") + for name in ("check","build","activate","rollback"): + cmd = sub.add_parser(name) + cmd.add_argument("--plan",required=True) + if name == "activate": + cmd.add_argument("--client-v2-ready",action="store_true") + if name == "rollback": + cmd.add_argument("--stop-broker-use",action="store_true") + args = parser.parse_args() + try: + if args.action == "preflight": + preflight(args) + return + if args.action == "prepare": + prepare(args) + return + root = Path(args.plan).resolve() + plan = load_plan(root) + if args.action == "check": + check(root,plan) + elif args.action == "build": + build(root,plan) + elif args.action == "activate": + activate(root,plan,args.client_v2_ready) + else: + if not args.stop_broker_use: + raise DeployError("rollback requires stopping broker clients; old refresh credentials are not restored") + check(root,plan,baseline=False) + run(compose(plan,root/"compose.rollback.json")+["up","-d","--no-deps","--no-build","--pull","never", + "--wait","--wait-timeout","180","connector","router"]) + print("ROLLED BACK containers only; preserve keyring/database. Client mode and reauthorization must be coordinated.") + except (DeployError,OSError,ValueError,KeyError,TypeError): + error = sys.exc_info()[1] + print("STOP: "+(str(error) if isinstance(error,DeployError) else "invalid/missing migration input; secret details withheld"),file=sys.stderr) + raise SystemExit(1) + + +if __name__ == "__main__": + main() diff --git a/deployable/tests/test_broker_deploy.py b/deployable/tests/test_broker_deploy.py new file mode 100644 index 0000000..32fdc5a --- /dev/null +++ b/deployable/tests/test_broker_deploy.py @@ -0,0 +1,252 @@ +"""Migration tooling policy tests; no host Docker or real deployment is used.""" +import argparse +import copy +import importlib.util +import json +from pathlib import Path + +import pytest + +ROOT = Path(__file__).resolve().parents[1] +spec = importlib.util.spec_from_file_location("broker_deploy_test",ROOT/"scripts/broker_deploy.py") +deploy = importlib.util.module_from_spec(spec) +spec.loader.exec_module(deploy) + + +@pytest.fixture +def scenario(tmp_path,monkeypatch): + live = tmp_path/"live" + live.mkdir() + (live/".env").write_text("FAKE_SECRET=synthetic-do-not-print\n") + files = [] + for i in range(7): + p = live/("compose"+str(i)+".yaml") + p.write_text("services: {}\n") + files.append(str(p)) + caddy = live/"Caddyfile" + caddy.write_text("@private path /admin/* /internal/* /ready\nrespond @private 404\n") + labels = {"com.docker.compose.project":"hainuo-relay", + "com.docker.compose.project.working_dir":str(live), + "com.docker.compose.project.config_files":",".join(files)} + roles = {name:{"id":"container-"+name,"image":"sha256:"+str(i+1)*64,"image_ref":"old:tag","labels":labels} + for i,name in enumerate(("connector","router","longconn","enrollment-signer","edge"))} + env = {"HN_ENVIRONMENT":"production","HN_DATABASE_URL":"postgresql://fake@postgres/fake", + "HN_FEISHU_ENABLED":"true","HN_OAUTH_ENABLED":"true","HN_FEISHU_APP_ID":"cli_test", + "HN_FEISHU_APP_SECRET":"synthetic-app-secret","HN_ADMIN_SECRET":"a"*40,"HN_INTERNAL_SECRET":"b"*40} + base = {"services":{name:{"image":"old:tag","environment":{},"networks":{"backend":None},"volumes":[]} + for name in [*roles,"postgres","redis"]},"volumes":{"metadata":{}},"networks":{"backend":{}}} + base["services"]["connector"]["environment"] = env + base["services"]["connector"]["volumes"] = [{"type":"bind","source":str(live/"runtime"),"target":"/data"}] + base["services"]["edge"]["volumes"] = [{"type":"bind","source":str(caddy),"target":"/etc/caddy/Caddyfile"}] + profiles = tmp_path/"product.json" + profiles.write_text(json.dumps({"product-v1":["read"]})) + output = tmp_path/"migration" + calls = [] + + def fake_run(args,json_result=False,optional=False,input_text=None): + calls.append(args) + if args[:2] == ["docker","ps"]: + name = next(arg.rsplit('=',1)[1] for arg in args if arg.startswith('label=com.docker.compose.service=')) + return "container-"+name + if args[:2] == ["docker","inspect"]: + name = args[2].replace("container-", "").replace("hainuo-relay-", "").removesuffix("-1") + return roles[name] + if args[:3] == ["docker","image","inspect"]: + if optional: + return None + return "arm64" if "{{.Architecture}}" in args else "sha256:"+"a"*64 + if args[:2] == ["docker","compose"]: + if "config" not in args: + return "" + result = copy.deepcopy(base) + candidates = [Path(args[i+1]) for i,value in enumerate(args[:-1]) if value == "-f"] + if candidates[-1].name == "compose.broker.json": + overlay = json.loads(candidates[-1].read_text()) + for name,fields in overlay["services"].items(): + result["services"][name]["image"] = fields["image"] + for key,value in fields["environment"].items(): + if value.startswith("${"): + value = env[value[2:].split(":")[0]] + result["services"][name]["environment"][key] = value + if "volumes" in fields: + result["services"][name]["volumes"].extend(fields["volumes"]) + return result + return "" + + monkeypatch.setattr(deploy,"run",fake_run) + args = argparse.Namespace(deployment_root=str(live),source_root=str(ROOT),profiles=str(profiles), + output=str(output),connector_container="hainuo-relay-connector-1") + return dict(args=args,live=live,output=output,calls=calls,roles=roles,base=base,profiles=profiles,caddy=caddy) + + +def prepared(scenario): + deploy.prepare(scenario["args"]) + return deploy.load_plan(scenario["output"]) + + +def test_preflight_is_read_only_and_redacted(scenario,capsys): + deploy.preflight(scenario["args"]) + assert not scenario["output"].exists() + assert "synthetic" not in capsys.readouterr().out + assert not any("up" in c or "build" in c or "tag" in c for c in scenario["calls"]) + + +def test_prepare_preserves_all_seven_files_runtime_identity_and_no_service_changes(scenario,capsys): + original = (scenario["live"]/".env").read_bytes() + plan = prepared(scenario) + assert len(plan["compose_files"]) == 7 + assert (scenario["live"]/".env").read_bytes() == original + assert not (scenario["live"]/"runtime").exists() + overlay = json.loads((scenario["output"]/"compose.broker.json").read_text()) + assert set(overlay["services"]) == {"connector","router"} + assert overlay["services"]["connector"]["environment"]["HN_CREDENTIAL_MODE"] == "broker" + assert len(json.loads((scenario["output"]/"credential-keyring.json").read_text())["keys"]) == 1 + rollback = json.loads((scenario["output"]/"compose.rollback.json").read_text()) + assert rollback["services"]["connector"]["image"] != rollback["services"]["router"]["image"] + assert not any("up" in c or "build" in c for c in scenario["calls"]) + assert "synthetic" not in capsys.readouterr().out + + +def test_prepare_refuses_existing_output_and_preserves_key(scenario): + prepared(scenario) + key = (scenario["output"]/"credential-keyring.json").read_bytes() + with pytest.raises(deploy.DeployError): + deploy.prepare(scenario["args"]) + assert (scenario["output"]/"credential-keyring.json").read_bytes() == key + + +def test_prepare_refuses_live_subdirectory(scenario): + scenario["args"].output = str(scenario["live"]/"broker") + with pytest.raises(deploy.DeployError,match="outside"): + deploy.prepare(scenario["args"]) + + +@pytest.mark.parametrize("value", [{}, {"v1":[]}, {"v1":["bad scope"]}, {"bad profile":["read"]},[]]) +def test_invalid_product_profiles_never_generate_keys(scenario,value): + scenario["profiles"].write_text(json.dumps(value)) + with pytest.raises(deploy.DeployError): + deploy.prepare(scenario["args"]) + assert not scenario["output"].exists() + + +@pytest.mark.parametrize("target", [".env","compose0.yaml"]) +def test_check_refuses_changed_live_configuration(scenario,target): + plan = prepared(scenario) + with (scenario["live"]/target).open('a') as f: + f.write("# drift\n") + with pytest.raises(deploy.DeployError,match="changed"): + deploy.check(scenario["output"],plan) + + +@pytest.mark.parametrize("target", ["credential-keyring.json","compose.broker.json","compose.rollback.json","build/relay/credential_api.py"]) +def test_check_refuses_changed_prepared_keys_or_payload(scenario,target): + plan = prepared(scenario) + with (scenario["output"]/target).open('a') as f: + f.write("\n") + with pytest.raises(deploy.DeployError,match="changed"): + deploy.check(scenario["output"],plan) + + +def test_check_refuses_unreviewed_edge_private_rules(scenario): + plan = prepared(scenario) + scenario["caddy"].write_text("reverse_proxy connector:8000\n") + with pytest.raises(deploy.DeployError,match="private"): + deploy.check(scenario["output"],plan) + + +def test_check_refuses_other_service_image_drift(scenario): + plan = prepared(scenario) + scenario["roles"]["longconn"]["image"] = "sha256:"+"f"*64 + with pytest.raises(deploy.DeployError,match="longconn"): + deploy.check(scenario["output"],plan) + + +def test_activate_requires_client_readiness_before_any_docker_command(scenario): + plan = prepared(scenario) + scenario["calls"].clear() + with pytest.raises(deploy.DeployError,match="v2 client"): + deploy.activate(scenario["output"],plan,False) + assert not scenario["calls"] + + +def test_build_reuses_exact_runtime_without_download_or_service_restart(scenario): + plan = prepared(scenario) + scenario["calls"].clear() + deploy.build(scenario["output"],plan) + build = next(c for c in scenario["calls"] if c[:2] == ["docker","build"]) + assert "--pull=false" in build and build[build.index('--network')+1] == 'none' + assert not any("up" in c for c in scenario["calls"]) + dockerfile = (scenario["output"]/"build/Dockerfile").read_text() + assert "pip install" not in dockerfile and "apt" not in dockerfile + assert (scenario["output"]/"candidate.json").exists() + probes = [c for c in scenario["calls"] if c[:2] == ["docker","run"]] + assert len(probes) == 3 + assert all("synthetic-app-secret" not in " ".join(c) for c in probes) + + +def test_failed_runtime_smoke_does_not_mark_candidate_ready(scenario,monkeypatch): + plan = prepared(scenario) + original = deploy.run + def fail(args,*a,**kw): + if args[:2] == ["docker","run"]: + raise deploy.DeployError("synthetic image smoke failure") + return original(args,*a,**kw) + monkeypatch.setattr(deploy,"run",fail) + with pytest.raises(deploy.DeployError,match="smoke"): + deploy.build(scenario["output"],plan) + assert not (scenario["output"]/"candidate.json").exists() + + +def test_activate_targets_only_connector_router_and_preserves_other_services(scenario): + plan = prepared(scenario) + deploy.build(scenario["output"],plan) + scenario["calls"].clear() + deploy.activate(scenario["output"],plan,True) + command = next(c for c in scenario["calls"] if "up" in c) + assert command[-2:] == ["connector","router"] and "--no-deps" in command + assert len([c for c in scenario["calls"] if "up" in c]) == 1 + + +def test_activate_rejects_retagged_candidate(scenario,monkeypatch): + plan = prepared(scenario) + deploy.build(scenario["output"],plan) + original = deploy.run + def altered(args,*a,**kw): + if args[:3] == ["docker","image","inspect"] and args[3] == plan["candidate_image"]: + return "sha256:"+"f"*64 + return original(args,*a,**kw) + monkeypatch.setattr(deploy,"run",altered) + scenario["calls"].clear() + with pytest.raises(deploy.DeployError,match="identity changed"): + deploy.activate(scenario["output"],plan,True) + assert not any("up" in c for c in scenario["calls"]) + + +def test_activation_validation_failure_never_recreates_services(scenario,monkeypatch): + plan = prepared(scenario) + deploy.build(scenario["output"],plan) + def fail(*args,**kwargs): + raise deploy.DeployError("runtime settings rejected") + monkeypatch.setattr(deploy,"runtime_check",fail) + scenario["calls"].clear() + with pytest.raises(deploy.DeployError,match="settings"): + deploy.activate(scenario["output"],plan,True) + assert not any("up" in c for c in scenario["calls"]) + + +def test_catchall_before_private_block_requires_manual_review(scenario): + plan = prepared(scenario) + original = scenario["caddy"].read_text() + scenario["caddy"].write_text("handle { reverse_proxy connector:8000 }\n"+original) + with pytest.raises(deploy.DeployError,match="private"): + deploy.check(scenario["output"],plan) + + +@pytest.mark.parametrize("field", ["volumes","networks"]) +def test_overlay_validator_rejects_database_volume_or_network_change(scenario,field): + plan = prepared(scenario) + base = scenario["base"] + merged = deploy.run(deploy.compose(plan,scenario["output"]/"compose.broker.json")+["config"],True) + merged[field] = {"unexpected":{}} + with pytest.raises(deploy.DeployError): + deploy.validate_overlay(base,merged,plan["candidate_image"])