fix(deploy): repair missing router public-key path after partial activation
Live evidence showed a healthy candidate connector and a restarting router: legacy router environment omitted HN_CONNECTOR_PUBLIC_KEY, so the new routing verifier tried runtime/pki/connector.pub instead of the existing /trust mount. Resolve the public key from the existing read-only trust bind, set its path in new overlays, and actually load it during isolated runtime checks. Add repair-router for a known healthy candidate connector: preserve credential keys and live configuration, back up overlay/plan, add only the router path, update integrity metadata, probe, and recreate router alone with --no-deps. Retries do not regenerate keys or overwrite repair backups. Add read-only state including stopped/restarting containers and fixed startup error classes, including stderr, without printing raw logs or credentials. Identify command stage/exit code and activation progress. Permit rollback of known stopped candidate targets and old overlays without requiring the new router key; unknown images still fail closed. Validation: full suite 262 passed, 3 skipped (real PostgreSQL unavailable), 30 existing warnings; final tooling suite 50 passed after the added CLI rollback regression. Python 3.7 syntax and staged diff checks pass. No live repair was executed by this agent; operator must run repair-router and verify health and the real client handshake on ECS.
This commit is contained in:
@@ -188,3 +188,13 @@ Windows / Python 3.12.10,平台桩、临时证书/密钥、SQLite及带Lua的F
|
|||||||
新增11项工具回归,覆盖合法环境保持、HTTPS/WSS/数据库失败在写密钥前拒绝、只读diagnose保留全部准备文件、错误不泄露数据库凭据;工具专项40项通过。实际现场哪项原始条件不符仍以更新后的diagnose输出为准,不将历史默认值当作当前配置证明。
|
新增11项工具回归,覆盖合法环境保持、HTTPS/WSS/数据库失败在写密钥前拒绝、只读diagnose保留全部准备文件、错误不泄露数据库凭据;工具专项40项通过。实际现场哪项原始条件不符仍以更新后的diagnose输出为准,不将历史默认值当作当前配置证明。
|
||||||
|
|
||||||
修订后完整本地回归:**253 passed,3 skipped,30 warnings**(49.46秒);真实PostgreSQL仍未提供,warnings类型同上,Python3.7语法与diff检查通过。
|
修订后完整本地回归:**253 passed,3 skipped,30 warnings**(49.46秒);真实PostgreSQL仍未提供,warnings类型同上,Python3.7语法与diff检查通过。
|
||||||
|
|
||||||
|
### 部分切换故障与router公钥路径修复
|
||||||
|
|
||||||
|
现场activate只输出通用Docker失败,用户随后提供状态:connector已是broker候选并healthy,router同候选持续Restarting(1)。进一步只读确认router未配置HN_CONNECTOR_PUBLIC_KEY,实际日志FileNotFoundError指向默认runtime/pki/connector.pub。这是旧router此前不消费connector公钥,而新模式开始验证路由签名后,迁移叠加漏传原信任目录公钥路径;首次import/Settings探测未实际打开公钥,没提前发现。
|
||||||
|
|
||||||
|
修订新prepare识别现有只读信任挂载中的公钥并显式传HN_CONNECTOR_PUBLIC_KEY,runtime_check挂载并实际load_public。新增repair-router面向已知健康候选connector的部分切换,备份叠加/plan、保留keyring/原.env/其他镜像,只补router路径并重建router。原密钥或运行配置漂移时拒绝,不自动删除/重准备。
|
||||||
|
|
||||||
|
新增state只读查看停止/重启容器并对本地日志只提取固定错误类(包含Docker logs的stderr);run错误报告阶段/退出码而非原始配置。activate输出阶段,明确up失败可能部分切换。rollback的目标服务可为已停止的已知原/候选容器,不再因只查询运行容器而无法识别;未知镜像仍拒绝。
|
||||||
|
|
||||||
|
工具专项50项通过(新增state/部分回滚/错误脱敏4项、公钥修复5项、旧部分计划CLI回滚1项)。完整本地回归262 passed、3 skipped、30 warnings(49.76秒),之后追加的回滚入口检查再次通过工具50项;Python3.7语法与diff检查通过。真实PostgreSQL仍未提供。实际修复结果仍需现场执行并核验,不能把本地Docker桩通过当作router已恢复。
|
||||||
|
|||||||
@@ -102,6 +102,30 @@ sudo python3 deployable/scripts/broker_deploy.py activate \
|
|||||||
|
|
||||||
## 6. 应急容器回退
|
## 6. 应急容器回退
|
||||||
|
|
||||||
|
### 已部分切换且router缺少公钥参数
|
||||||
|
|
||||||
|
现场已确认一种旧配置兼容遗漏:connector候选健康,router不断重启,日志报`FileNotFoundError: runtime/pki/connector.pub`;router原环境仅有`HN_CA_CERT=/trust/ca.pem`,没有`HN_CONNECTOR_PUBLIC_KEY`。不要修改公钥内容、重新生成CA或keyring,不要重复activate。
|
||||||
|
|
||||||
|
更新暂存仓库后可先只读查看包含停止/重启中容器的状态和固定错误类型:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo python3 deployable/scripts/broker_deploy.py state \
|
||||||
|
--plan /home/cnooc/hainuo-broker-prepared-20261104
|
||||||
|
```
|
||||||
|
|
||||||
|
仅在已知候选connector运行且healthy时执行:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo python3 deployable/scripts/broker_deploy.py repair-router \
|
||||||
|
--plan /home/cnooc/hainuo-broker-prepared-20261104
|
||||||
|
```
|
||||||
|
|
||||||
|
该操作先验证原配置/密钥/准备目录完整性及已知候选镜像,从router已有只读信任挂载识别实际公钥位置(该现场为`/trust/connector.pub`),备份原叠加和plan后只增加router公钥参数并同步计划校验。不会重生成任何密钥、修改原.env、重建connector或其他服务。重跑隔离配置/公钥实际读取检查后,只up router;重复执行不会覆盖备份或改密钥。
|
||||||
|
|
||||||
|
新prepare已包含此参数;build/activate的隔离容器现在会实际加载挂载的router公钥,不能只验证Settings字段就标记候选就绪。`state`不打印原始日志或环境,只报告原/候选镜像类别、容器状态/健康/退出码和固定错误类。已部分切换时rollback可识别停止或重启中的已知候选容器,但拒绝未知镜像;原有回滚限制仍适用。
|
||||||
|
|
||||||
|
### 完整回退
|
||||||
|
|
||||||
若切换失败,先停止broker客户端继续使用,按记录的每服务镜像回退:
|
若切换失败,先停止broker客户端继续使用,按记录的每服务镜像回退:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
|
|||||||
@@ -9,12 +9,13 @@ import base64
|
|||||||
import hashlib
|
import hashlib
|
||||||
import json
|
import json
|
||||||
import os
|
import os
|
||||||
from pathlib import Path
|
from pathlib import Path, PurePosixPath
|
||||||
import re
|
import re
|
||||||
import secrets
|
import secrets
|
||||||
import shutil
|
import shutil
|
||||||
import subprocess
|
import subprocess
|
||||||
import sys
|
import sys
|
||||||
|
import tempfile
|
||||||
from urllib.parse import urlsplit
|
from urllib.parse import urlsplit
|
||||||
|
|
||||||
|
|
||||||
@@ -27,7 +28,17 @@ def run(args, json_result=False, optional=False, input_text=None):
|
|||||||
if result.returncode:
|
if result.returncode:
|
||||||
if optional:
|
if optional:
|
||||||
return None
|
return None
|
||||||
raise DeployError("Docker/Git command failed; raw output withheld to protect configuration")
|
if args[:2] == ["docker","compose"]:
|
||||||
|
stage = "compose-up" if "up" in args else "compose-config"
|
||||||
|
elif args[:2] == ["docker","run"]:
|
||||||
|
stage = "candidate-runtime-check"
|
||||||
|
elif args[:3] == ["docker","image","inspect"]:
|
||||||
|
stage = "image-inspect"
|
||||||
|
else:
|
||||||
|
stage = "docker-command"
|
||||||
|
raise DeployError(stage+" failed (exit "+str(result.returncode)+"); raw output withheld; run state to inspect current services")
|
||||||
|
if args[:2] == ["docker","logs"]:
|
||||||
|
return (result.stdout+"\n"+result.stderr).strip()
|
||||||
try:
|
try:
|
||||||
return json.loads(result.stdout) if json_result else result.stdout.strip()
|
return json.loads(result.stdout) if json_result else result.stdout.strip()
|
||||||
except ValueError:
|
except ValueError:
|
||||||
@@ -46,17 +57,32 @@ def write_json(path, value):
|
|||||||
stream.write("\n")
|
stream.write("\n")
|
||||||
|
|
||||||
|
|
||||||
|
def replace_json(path,value):
|
||||||
|
name = None
|
||||||
|
try:
|
||||||
|
fd,name = tempfile.mkstemp(prefix='.'+path.name+'.',dir=str(path.parent))
|
||||||
|
with os.fdopen(fd,'w',encoding='utf-8') as stream:
|
||||||
|
json.dump(value,stream,ensure_ascii=False,indent=2)
|
||||||
|
stream.write('\n');stream.flush();os.fsync(stream.fileno())
|
||||||
|
os.replace(name,str(path))
|
||||||
|
finally:
|
||||||
|
if name and os.path.exists(name):os.unlink(name)
|
||||||
|
|
||||||
|
|
||||||
def inspect(container):
|
def inspect(container):
|
||||||
template = ('{"id":{{json .Id}},"image":{{json .Image}},'
|
template = ('{"id":{{json .Id}},"image":{{json .Image}},'
|
||||||
'"image_ref":{{json .Config.Image}},"labels":{{json .Config.Labels}}}')
|
'"image_ref":{{json .Config.Image}},"labels":{{json .Config.Labels}},'
|
||||||
|
'"state":{"status":{{json .State.Status}},"health":'
|
||||||
|
'{{if .State.Health}}{{json .State.Health.Status}}{{else}}"none"{{end}},'
|
||||||
|
'"exit_code":{{.State.ExitCode}},"restart_count":{{.RestartCount}}}}')
|
||||||
return run(["docker", "inspect", container, "--format", template], True)
|
return run(["docker", "inspect", container, "--format", template], True)
|
||||||
|
|
||||||
|
|
||||||
def service(project, name):
|
def service(project, name, include_stopped=False):
|
||||||
values = run(["docker", "ps", "-q", "--filter", "label=com.docker.compose.project="+project,
|
values = run(["docker", "ps", "-aq" if include_stopped else "-q", "--filter", "label=com.docker.compose.project="+project,
|
||||||
"--filter", "label=com.docker.compose.service="+name]).splitlines()
|
"--filter", "label=com.docker.compose.service="+name]).splitlines()
|
||||||
if len(values) != 1:
|
if len(values) != 1:
|
||||||
raise DeployError("expected exactly one running relay service: "+name)
|
raise DeployError("expected exactly one "+("existing" if include_stopped else "running")+" relay service: "+name)
|
||||||
return inspect(values[0])
|
return inspect(values[0])
|
||||||
|
|
||||||
|
|
||||||
@@ -125,7 +151,30 @@ def require_platform(env):
|
|||||||
raise DeployError("broker platform configuration incomplete; "+"; ".join(problems))
|
raise DeployError("broker platform configuration incomplete; "+"; ".join(problems))
|
||||||
|
|
||||||
|
|
||||||
def validate_overlay(base, merged, candidate):
|
def router_public_key(config):
|
||||||
|
router=config['services']['router']
|
||||||
|
env=router.get('environment',{})
|
||||||
|
value=env.get('HN_CONNECTOR_PUBLIC_KEY') or str(PurePosixPath(env.get('HN_CA_CERT','/trust/ca.pem')).parent/'connector.pub')
|
||||||
|
target=PurePosixPath(value)
|
||||||
|
if not target.is_absolute() or '..' in target.parts:
|
||||||
|
raise DeployError('invalid router connector public-key path')
|
||||||
|
for mount in router.get('volumes',[]):
|
||||||
|
if mount.get('type')!='bind' or not mount.get('read_only'):continue
|
||||||
|
base=PurePosixPath(mount['target'])
|
||||||
|
try:
|
||||||
|
relative=target.relative_to(base)
|
||||||
|
except ValueError:
|
||||||
|
continue
|
||||||
|
source=Path(mount['source'])
|
||||||
|
source=source.joinpath(*relative.parts) if relative.parts else source
|
||||||
|
if source.is_file() and not source.is_symlink():
|
||||||
|
data=source.read_bytes()
|
||||||
|
if b'-----BEGIN PUBLIC KEY-----' in data and b'PRIVATE KEY' not in data:
|
||||||
|
return str(target),source
|
||||||
|
raise DeployError('router connector public key is not a verified read-only trust mount')
|
||||||
|
|
||||||
|
|
||||||
|
def validate_overlay(base, merged, candidate, router_key_required=True):
|
||||||
old, new = base["services"], merged["services"]
|
old, new = base["services"], merged["services"]
|
||||||
require_services(old)
|
require_services(old)
|
||||||
require_services(new)
|
require_services(new)
|
||||||
@@ -172,6 +221,10 @@ def validate_overlay(base, merged, candidate):
|
|||||||
mounts = [v for v in new["connector"].get("volumes", []) if v["target"] == "/run/hainuo-credentials/keyring.json"]
|
mounts = [v for v in new["connector"].get("volumes", []) if v["target"] == "/run/hainuo-credentials/keyring.json"]
|
||||||
if len(mounts) != 1 or not mounts[0].get("read_only"):
|
if len(mounts) != 1 or not mounts[0].get("read_only"):
|
||||||
raise DeployError("missing read-only credential keyring mount")
|
raise DeployError("missing read-only credential keyring mount")
|
||||||
|
if router_key_required:
|
||||||
|
target,_=router_public_key(merged)
|
||||||
|
if new['router']['environment'].get('HN_CONNECTOR_PUBLIC_KEY')!=target:
|
||||||
|
raise DeployError('router HN_CONNECTOR_PUBLIC_KEY missing; use repair-router for an already switched plan')
|
||||||
|
|
||||||
|
|
||||||
def preflight(args):
|
def preflight(args):
|
||||||
@@ -249,6 +302,7 @@ def prepare(args):
|
|||||||
if base["services"]["connector"].get("environment",{}).get("HN_CREDENTIAL_MODE","legacy") != "legacy":
|
if base["services"]["connector"].get("environment",{}).get("HN_CREDENTIAL_MODE","legacy") != "legacy":
|
||||||
raise DeployError("already broker: preserve the existing keyring and migration plan")
|
raise DeployError("already broker: preserve the existing keyring and migration plan")
|
||||||
require_platform({**base["services"]["connector"].get("environment",{}),"HN_OAUTH_ENABLED":"true"})
|
require_platform({**base["services"]["connector"].get("environment",{}),"HN_OAUTH_ENABLED":"true"})
|
||||||
|
public_key,_=router_public_key(base)
|
||||||
output.mkdir(mode=0o700,parents=False)
|
output.mkdir(mode=0o700,parents=False)
|
||||||
os.chmod(str(output),0o700)
|
os.chmod(str(output),0o700)
|
||||||
keyring = {"active_key_id":"initial-v1","keys":{"initial-v1":base64.b64encode(os.urandom(32)).decode("ascii")}}
|
keyring = {"active_key_id":"initial-v1","keys":{"initial-v1":base64.b64encode(os.urandom(32)).decode("ascii")}}
|
||||||
@@ -264,6 +318,7 @@ def prepare(args):
|
|||||||
"volumes":[{"type":"bind","source":str(output/"credential-keyring.json"),
|
"volumes":[{"type":"bind","source":str(output/"credential-keyring.json"),
|
||||||
"target":"/run/hainuo-credentials/keyring.json","read_only":True}]},
|
"target":"/run/hainuo-credentials/keyring.json","read_only":True}]},
|
||||||
"router":{"image":candidate,"environment":{"HN_CREDENTIAL_MODE":"broker",
|
"router":{"image":candidate,"environment":{"HN_CREDENTIAL_MODE":"broker",
|
||||||
|
"HN_CONNECTOR_PUBLIC_KEY":public_key,
|
||||||
"HN_ROUTER_ADMISSION_SECRET":admission,"HN_CONNECTOR_INTERNAL_URL":"http://connector:8000",
|
"HN_ROUTER_ADMISSION_SECRET":admission,"HN_CONNECTOR_INTERNAL_URL":"http://connector:8000",
|
||||||
"HN_FEISHU_APP_ID":"${HN_FEISHU_APP_ID:?Configure existing application}",
|
"HN_FEISHU_APP_ID":"${HN_FEISHU_APP_ID:?Configure existing application}",
|
||||||
"HN_ENVIRONMENT":"${HN_ENVIRONMENT:?Configure existing environment}"}}}}
|
"HN_ENVIRONMENT":"${HN_ENVIRONMENT:?Configure existing environment}"}}}}
|
||||||
@@ -317,7 +372,45 @@ def diagnose(root,plan):
|
|||||||
raise DeployError("review the named fields above; do not paste configuration or regenerate keys")
|
raise DeployError("review the named fields above; do not paste configuration or regenerate keys")
|
||||||
|
|
||||||
|
|
||||||
def check(root, plan, baseline=True):
|
def log_signals(text):
|
||||||
|
"""Classify locally; never return arbitrary log lines, SQL, URLs or tokens."""
|
||||||
|
known = ('ModuleNotFoundError','ImportError','FileNotFoundError','PermissionError','ValidationError',
|
||||||
|
'CredentialError','OperationalError','ProgrammingError','UndefinedColumn','UndefinedTable',
|
||||||
|
'InsufficientPrivilege','ReadOnlySqlTransaction','SyntaxError','IndentationError',
|
||||||
|
'ConnectionRefusedError','AuthenticationError','ConnectionError','TimeoutError','MemoryError')
|
||||||
|
signals = [name for name in known if re.search(r'\b'+name+r'\b',text)]
|
||||||
|
if 'read-only file system' in text.lower():
|
||||||
|
signals.append('read-only-filesystem')
|
||||||
|
if 'address already in use' in text.lower():
|
||||||
|
signals.append('port-in-use')
|
||||||
|
return signals
|
||||||
|
|
||||||
|
|
||||||
|
def deployment_state(root,plan):
|
||||||
|
expected = None
|
||||||
|
if (root/"candidate.json").is_file():
|
||||||
|
expected = json.loads((root/"candidate.json").read_text(encoding="utf-8")).get("image")
|
||||||
|
for name in ("connector","router","longconn","enrollment-signer","edge"):
|
||||||
|
try:
|
||||||
|
entry = service(plan["project"],name,include_stopped=True)
|
||||||
|
except DeployError:
|
||||||
|
print(name+": no unique existing container; inspect docker ps -a for this project")
|
||||||
|
continue
|
||||||
|
state = entry.get("state",{})
|
||||||
|
status = state.get("status") if state.get("status") in {'created','running','paused','restarting','removing','exited','dead'} else 'unknown'
|
||||||
|
health = state.get("health") if state.get("health") in {'none','starting','healthy','unhealthy'} else 'unknown'
|
||||||
|
image = 'candidate' if expected and entry['image']==expected else (
|
||||||
|
'original' if entry['image']==plan['roles'][name]['image'] else 'different')
|
||||||
|
numbers = {k:v for k,v in state.items() if k in {'exit_code','restart_count'} and type(v) is int}
|
||||||
|
print(name+': image='+image+' status='+status+' health='+health+' '+json.dumps(numbers,sort_keys=True))
|
||||||
|
if name in {'connector','router'}:
|
||||||
|
raw = run(['docker','logs','--tail','100',entry['id']],optional=True)
|
||||||
|
if raw is not None:
|
||||||
|
print(' startup signals: '+(', '.join(log_signals(raw)) or 'no recognized error class; raw logs withheld'))
|
||||||
|
print('STATE inspection only; no files, keys or containers changed. Do not retry activation before reviewing partial switch state.')
|
||||||
|
|
||||||
|
|
||||||
|
def check(root, plan, baseline=True, router_key_required=True):
|
||||||
if os.name != "nt":
|
if os.name != "nt":
|
||||||
if root.stat().st_mode & 0o077:
|
if root.stat().st_mode & 0o077:
|
||||||
raise DeployError("migration directory must not be accessible to other users")
|
raise DeployError("migration directory must not be accessible to other users")
|
||||||
@@ -336,12 +429,18 @@ def check(root, plan, baseline=True):
|
|||||||
if digest(root/path) != expected:
|
if digest(root/path) != expected:
|
||||||
raise DeployError("prepared keyring or overlay changed; do not regenerate credential keys")
|
raise DeployError("prepared keyring or overlay changed; do not regenerate credential keys")
|
||||||
for name, old in plan["roles"].items():
|
for name, old in plan["roles"].items():
|
||||||
current = service(plan["project"],name)
|
current = service(plan["project"],name,include_stopped=not baseline and name in {"connector","router"})
|
||||||
if (baseline or name not in {"connector","router"}) and current["image"] != old["image"]:
|
if (baseline or name not in {"connector","router"}) and current["image"] != old["image"]:
|
||||||
raise DeployError("running service image changed since preparation: "+name)
|
raise DeployError("running service image changed since preparation: "+name)
|
||||||
|
if not baseline and name in {"connector","router"}:
|
||||||
|
allowed = {old['image']}
|
||||||
|
if (root/'candidate.json').is_file():
|
||||||
|
allowed.add(json.loads((root/'candidate.json').read_text(encoding='utf-8'))['image'])
|
||||||
|
if current['image'] not in allowed:
|
||||||
|
raise DeployError('unreviewed target image; refusing rollback: '+name)
|
||||||
base = run(compose(plan)+["config","--format","json"],True)
|
base = run(compose(plan)+["config","--format","json"],True)
|
||||||
merged = run(compose(plan,root/"compose.broker.json")+["config","--format","json"],True)
|
merged = run(compose(plan,root/"compose.broker.json")+["config","--format","json"],True)
|
||||||
validate_overlay(base,merged,plan["candidate_image"])
|
validate_overlay(base,merged,plan["candidate_image"],router_key_required=router_key_required)
|
||||||
# Require the current edge's explicit private-path block, before catch-all handling.
|
# Require the current edge's explicit private-path block, before catch-all handling.
|
||||||
mounts = base["services"]["edge"].get("volumes",[])
|
mounts = base["services"]["edge"].get("volumes",[])
|
||||||
caddy = next((v["source"] for v in mounts if v.get("target") == "/etc/caddy/Caddyfile" and v.get("type") == "bind"),None)
|
caddy = next((v["source"] for v in mounts if v.get("target") == "/etc/caddy/Caddyfile" and v.get("type") == "bind"),None)
|
||||||
@@ -353,7 +452,7 @@ def check(root, plan, baseline=True):
|
|||||||
first_handle = re.search(r"\bhandle(?:_path)?\b",text)
|
first_handle = re.search(r"\bhandle(?:_path)?\b",text)
|
||||||
if not deny or (first_handle and deny.start() > first_handle.start()):
|
if not deny or (first_handle and deny.start() > first_handle.start()):
|
||||||
raise DeployError("edge private-path block is not the reviewed configuration")
|
raise DeployError("edge private-path block is not the reviewed configuration")
|
||||||
print("CHECKED: existing Compose/storage/images retained; no services restarted")
|
print("CHECKED: Compose/storage and permitted service images verified; no services restarted")
|
||||||
|
|
||||||
|
|
||||||
def build(root,plan):
|
def build(root,plan):
|
||||||
@@ -374,30 +473,75 @@ def build(root,plan):
|
|||||||
|
|
||||||
def runtime_check(root,plan):
|
def runtime_check(root,plan):
|
||||||
merged = run(compose(plan,root/"compose.broker.json")+["config","--format","json"],True)
|
merged = run(compose(plan,root/"compose.broker.json")+["config","--format","json"],True)
|
||||||
|
public_key,public_source=router_public_key(merged)
|
||||||
# Actual secrets travel via stdin only: no command-line values, env dump,
|
# Actual secrets travel via stdin only: no command-line values, env dump,
|
||||||
# database connection or platform request. The container has no network.
|
# database connection or platform request. The container has no network.
|
||||||
code = ("import json,os,sys\nfrom pathlib import Path\nfrom relay.config import Settings\n"
|
code = ("import json,os,sys\nfrom pathlib import Path\nfrom relay.config import Settings\n"
|
||||||
"from relay.credential_store import CredentialKeyring\n"
|
"from relay.credential_store import CredentialKeyring\n"
|
||||||
|
"from relay.crypto import load_public\n"
|
||||||
"try:\n data=json.load(sys.stdin)\n os.environ.update(data['environment'])\n"
|
"try:\n data=json.load(sys.stdin)\n os.environ.update(data['environment'])\n"
|
||||||
" s=Settings(role=data['role'])\n"
|
" s=Settings(role=data['role'])\n"
|
||||||
" if data['role']=='connector': CredentialKeyring.from_file(s.credential_keyring_file)\n"
|
" if data['role']=='connector': CredentialKeyring.from_file(s.credential_keyring_file)\n"
|
||||||
|
" if data['role']=='router': load_public(s.connector_public_key)\n"
|
||||||
"except Exception:\n sys.exit(1)\n")
|
"except Exception:\n sys.exit(1)\n")
|
||||||
for role in ("connector","router"):
|
for role in ("connector","router"):
|
||||||
env = {key:str(value) for key,value in merged["services"][role]["environment"].items() if value is not None}
|
env = {key:str(value) for key,value in merged["services"][role]["environment"].items() if value is not None}
|
||||||
run(["docker","run","--rm","-i","--network","none","--read-only","--entrypoint","python",
|
command=["docker","run","--rm","-i","--network","none","--read-only","--entrypoint","python",
|
||||||
"-v",str(root/"credential-keyring.json")+":/run/hainuo-credentials/keyring.json:ro",
|
"-v",str(root/"credential-keyring.json")+":/run/hainuo-credentials/keyring.json:ro",
|
||||||
plan["candidate_image"],"-c",code],input_text=json.dumps({"role":role,"environment":env}))
|
"-v",str(public_source)+":"+public_key+":ro",plan["candidate_image"],"-c",code]
|
||||||
|
print('Runtime check: '+role+' configuration and required key files',flush=True)
|
||||||
|
run(command,input_text=json.dumps({"role":role,"environment":env}))
|
||||||
|
|
||||||
|
|
||||||
|
def repair_router(root,plan):
|
||||||
|
# Read all known partial-switch states, but do not weaken general activate checks.
|
||||||
|
check(root,plan,baseline=False,router_key_required=False)
|
||||||
|
built=json.loads((root/'candidate.json').read_text(encoding='utf-8'))
|
||||||
|
connector=service(plan['project'],'connector',include_stopped=True)
|
||||||
|
if (connector['image']!=built['image'] or connector.get('state',{}).get('status')!='running'
|
||||||
|
or connector.get('state',{}).get('health')!='healthy'):
|
||||||
|
raise DeployError('repair-router requires the known candidate connector to be running and healthy; inspect state or rollback')
|
||||||
|
base=run(compose(plan)+['config','--format','json'],True)
|
||||||
|
target,_=router_public_key(base)
|
||||||
|
path=root/'compose.broker.json'
|
||||||
|
overlay=json.loads(path.read_text(encoding='utf-8'))
|
||||||
|
old=overlay['services']['router']['environment'].get('HN_CONNECTOR_PUBLIC_KEY')
|
||||||
|
if old not in (None,'',target):
|
||||||
|
raise DeployError('unexpected prepared router public-key path; manual review required')
|
||||||
|
if old!=target:
|
||||||
|
backup=root/'compose.broker.before-router-key.json'
|
||||||
|
plan_backup=root/'plan.before-router-key.json'
|
||||||
|
if backup.exists() or plan_backup.exists():
|
||||||
|
raise DeployError('repair backup already exists; preserve keys and inspect the interrupted repair')
|
||||||
|
write_json(backup,overlay);write_json(plan_backup,plan)
|
||||||
|
overlay['services']['router']['environment']['HN_CONNECTOR_PUBLIC_KEY']=target
|
||||||
|
replace_json(path,overlay)
|
||||||
|
plan['prepared_hashes']['compose.broker.json']=digest(path)
|
||||||
|
replace_json(root/'plan.json',plan)
|
||||||
|
print('REPAIR: existing mounted connector public key selected; credential keys and connector preserved',flush=True)
|
||||||
|
check(root,plan,baseline=False)
|
||||||
|
runtime_check(root,plan)
|
||||||
|
image=run(['docker','image','inspect',plan['candidate_image'],'--format','{{.Id}}'])
|
||||||
|
if image!=built['image']:
|
||||||
|
raise DeployError('candidate image identity changed')
|
||||||
|
print('REPAIR: recreate router only',flush=True)
|
||||||
|
run(compose(plan,root/'compose.broker.json')+['up','-d','--no-deps','--no-build','--pull','never',
|
||||||
|
'--wait','--wait-timeout','180','router'])
|
||||||
|
print('REPAIRED: router only; verify state and real client handshake before acceptance')
|
||||||
|
|
||||||
|
|
||||||
def activate(root,plan,ready):
|
def activate(root,plan,ready):
|
||||||
if not ready:
|
if not ready:
|
||||||
raise DeployError("v2 client not confirmed ready; only prepare/check/build are allowed now")
|
raise DeployError("v2 client not confirmed ready; only prepare/check/build are allowed now")
|
||||||
check(root,plan)
|
check(root,plan)
|
||||||
|
print('ACTIVATION step: verify candidate image identity',flush=True)
|
||||||
built = json.loads((root/"candidate.json").read_text(encoding="utf-8"))
|
built = json.loads((root/"candidate.json").read_text(encoding="utf-8"))
|
||||||
image = run(["docker","image","inspect",plan["candidate_image"],"--format","{{.Id}}"])
|
image = run(["docker","image","inspect",plan["candidate_image"],"--format","{{.Id}}"])
|
||||||
if built != {"image":image,"tag":plan["candidate_image"]}:
|
if built != {"image":image,"tag":plan["candidate_image"]}:
|
||||||
raise DeployError("candidate image identity changed")
|
raise DeployError("candidate image identity changed")
|
||||||
|
print('ACTIVATION step: isolated candidate configuration/keyring check (no service switch)',flush=True)
|
||||||
runtime_check(root,plan)
|
runtime_check(root,plan)
|
||||||
|
print('ACTIVATION step: recreate connector/router; failures may leave a partial switch, inspect state',flush=True)
|
||||||
run(compose(plan,root/"compose.broker.json")+["up","-d","--no-deps","--no-build","--pull","never",
|
run(compose(plan,root/"compose.broker.json")+["up","-d","--no-deps","--no-build","--pull","never",
|
||||||
"--wait","--wait-timeout","180","connector","router"])
|
"--wait","--wait-timeout","180","connector","router"])
|
||||||
print("ACTIVATED: connector/router only. Verify real OAuth, v2 client, unbind and task routing.")
|
print("ACTIVATED: connector/router only. Verify real OAuth, v2 client, unbind and task routing.")
|
||||||
@@ -415,7 +559,7 @@ def main():
|
|||||||
prep.add_argument("--profiles",required=True)
|
prep.add_argument("--profiles",required=True)
|
||||||
prep.add_argument("--output",required=True)
|
prep.add_argument("--output",required=True)
|
||||||
prep.add_argument("--connector-container",default="hainuo-relay-connector-1")
|
prep.add_argument("--connector-container",default="hainuo-relay-connector-1")
|
||||||
for name in ("diagnose","check","build","activate","rollback"):
|
for name in ("repair-router","state","diagnose","check","build","activate","rollback"):
|
||||||
cmd = sub.add_parser(name)
|
cmd = sub.add_parser(name)
|
||||||
cmd.add_argument("--plan",required=True)
|
cmd.add_argument("--plan",required=True)
|
||||||
if name == "activate":
|
if name == "activate":
|
||||||
@@ -432,7 +576,11 @@ def main():
|
|||||||
return
|
return
|
||||||
root = Path(args.plan).resolve()
|
root = Path(args.plan).resolve()
|
||||||
plan = load_plan(root)
|
plan = load_plan(root)
|
||||||
if args.action == "diagnose":
|
if args.action == "repair-router":
|
||||||
|
repair_router(root,plan)
|
||||||
|
elif args.action == "state":
|
||||||
|
deployment_state(root,plan)
|
||||||
|
elif args.action == "diagnose":
|
||||||
diagnose(root,plan)
|
diagnose(root,plan)
|
||||||
elif args.action == "check":
|
elif args.action == "check":
|
||||||
check(root,plan)
|
check(root,plan)
|
||||||
@@ -443,7 +591,7 @@ def main():
|
|||||||
else:
|
else:
|
||||||
if not args.stop_broker_use:
|
if not args.stop_broker_use:
|
||||||
raise DeployError("rollback requires stopping broker clients; old refresh credentials are not restored")
|
raise DeployError("rollback requires stopping broker clients; old refresh credentials are not restored")
|
||||||
check(root,plan,baseline=False)
|
check(root,plan,baseline=False,router_key_required=False)
|
||||||
run(compose(plan,root/"compose.rollback.json")+["up","-d","--no-deps","--no-build","--pull","never",
|
run(compose(plan,root/"compose.rollback.json")+["up","-d","--no-deps","--no-build","--pull","never",
|
||||||
"--wait","--wait-timeout","180","connector","router"])
|
"--wait","--wait-timeout","180","connector","router"])
|
||||||
print("ROLLED BACK containers only; preserve keyring/database. Client mode and reauthorization must be coordinated.")
|
print("ROLLED BACK containers only; preserve keyring/database. Client mode and reauthorization must be coordinated.")
|
||||||
|
|||||||
@@ -38,6 +38,10 @@ def scenario(tmp_path,monkeypatch):
|
|||||||
base = {"services":{name:{"image":"old:tag","environment":{},"networks":{"backend":None},"volumes":[]}
|
base = {"services":{name:{"image":"old:tag","environment":{},"networks":{"backend":None},"volumes":[]}
|
||||||
for name in [*roles,"postgres","redis"]},"volumes":{"metadata":{}},"networks":{"backend":{}}}
|
for name in [*roles,"postgres","redis"]},"volumes":{"metadata":{}},"networks":{"backend":{}}}
|
||||||
base["services"]["connector"]["environment"] = env
|
base["services"]["connector"]["environment"] = env
|
||||||
|
trust=live/'runtime/trust';trust.mkdir(parents=True)
|
||||||
|
(trust/'connector.pub').write_text('-----BEGIN PUBLIC KEY-----\nsynthetic-public-material\n-----END PUBLIC KEY-----\n')
|
||||||
|
base['services']['router']['environment']['HN_CA_CERT']='/trust/ca.pem'
|
||||||
|
base['services']['router']['volumes']=[{'type':'bind','source':str(trust),'target':'/trust','read_only':True}]
|
||||||
base["services"]["connector"]["volumes"] = [{"type":"bind","source":str(live/"runtime"),"target":"/data"}]
|
base["services"]["connector"]["volumes"] = [{"type":"bind","source":str(live/"runtime"),"target":"/data"}]
|
||||||
base["services"]["edge"]["volumes"] = [{"type":"bind","source":str(caddy),"target":"/etc/caddy/Caddyfile"}]
|
base["services"]["edge"]["volumes"] = [{"type":"bind","source":str(caddy),"target":"/etc/caddy/Caddyfile"}]
|
||||||
profiles = tmp_path/"product.json"
|
profiles = tmp_path/"product.json"
|
||||||
@@ -135,10 +139,13 @@ def test_prepare_preserves_all_seven_files_runtime_identity_and_no_service_chang
|
|||||||
plan = prepared(scenario)
|
plan = prepared(scenario)
|
||||||
assert len(plan["compose_files"]) == 7
|
assert len(plan["compose_files"]) == 7
|
||||||
assert (scenario["live"]/".env").read_bytes() == original
|
assert (scenario["live"]/".env").read_bytes() == original
|
||||||
assert not (scenario["live"]/"runtime").exists()
|
assert sorted(p.relative_to(scenario['live']/'runtime').as_posix()
|
||||||
|
for p in (scenario['live']/'runtime').rglob('*') if p.is_file())==['trust/connector.pub']
|
||||||
|
assert not (scenario['live']/'runtime/issuer').exists()
|
||||||
overlay = json.loads((scenario["output"]/"compose.broker.json").read_text())
|
overlay = json.loads((scenario["output"]/"compose.broker.json").read_text())
|
||||||
assert set(overlay["services"]) == {"connector","router"}
|
assert set(overlay["services"]) == {"connector","router"}
|
||||||
assert overlay["services"]["connector"]["environment"]["HN_CREDENTIAL_MODE"] == "broker"
|
assert overlay["services"]["connector"]["environment"]["HN_CREDENTIAL_MODE"] == "broker"
|
||||||
|
assert overlay['services']['router']['environment']['HN_CONNECTOR_PUBLIC_KEY']=='/trust/connector.pub'
|
||||||
assert len(json.loads((scenario["output"]/"credential-keyring.json").read_text())["keys"]) == 1
|
assert len(json.loads((scenario["output"]/"credential-keyring.json").read_text())["keys"]) == 1
|
||||||
rollback = json.loads((scenario["output"]/"compose.rollback.json").read_text())
|
rollback = json.loads((scenario["output"]/"compose.rollback.json").read_text())
|
||||||
assert rollback["services"]["connector"]["image"] != rollback["services"]["router"]["image"]
|
assert rollback["services"]["connector"]["image"] != rollback["services"]["router"]["image"]
|
||||||
@@ -317,6 +324,132 @@ def test_activation_validation_failure_never_recreates_services(scenario,monkeyp
|
|||||||
assert not any("up" in c for c in scenario["calls"])
|
assert not any("up" in c for c in scenario["calls"])
|
||||||
|
|
||||||
|
|
||||||
|
def test_state_sees_exited_candidate_without_showing_logs_or_secrets(scenario,monkeypatch,capsys):
|
||||||
|
plan = prepared(scenario)
|
||||||
|
deploy.write_json(scenario['output']/'candidate.json',{'image':scenario['roles']['connector']['image'],'tag':plan['candidate_image']})
|
||||||
|
scenario['roles']['connector']['state']={'status':'exited','health':'unhealthy','exit_code':1,'restart_count':7}
|
||||||
|
original=deploy.run
|
||||||
|
def logs(args,*a,**kw):
|
||||||
|
if args[:2]==['docker','logs']:
|
||||||
|
scenario['calls'].append(args)
|
||||||
|
return 'Traceback PermissionError synthetic-private-password https://secret.invalid/private'
|
||||||
|
return original(args,*a,**kw)
|
||||||
|
monkeypatch.setattr(deploy,'run',logs)
|
||||||
|
scenario['calls'].clear()
|
||||||
|
deploy.deployment_state(scenario['output'],plan)
|
||||||
|
output=capsys.readouterr().out
|
||||||
|
assert 'image=candidate status=exited health=unhealthy' in output
|
||||||
|
assert 'PermissionError' in output and 'synthetic-private-password' not in output and 'secret.invalid' not in output
|
||||||
|
assert all('-aq' in c for c in scenario['calls'] if c[:2]==['docker','ps'])
|
||||||
|
assert not any('up' in c or 'tag' in c for c in scenario['calls'])
|
||||||
|
|
||||||
|
|
||||||
|
def test_rollback_checks_allow_stopped_known_candidate_but_reject_other_image(scenario):
|
||||||
|
plan=prepared(scenario)
|
||||||
|
scenario['roles']['connector']['image']='sha256:'+'a'*64
|
||||||
|
deploy.write_json(scenario['output']/'candidate.json',{'image':'sha256:'+'a'*64,'tag':plan['candidate_image']})
|
||||||
|
scenario['calls'].clear()
|
||||||
|
deploy.check(scenario['output'],plan,baseline=False)
|
||||||
|
assert any(c[:2]==['docker','ps'] and '-aq' in c for c in scenario['calls'])
|
||||||
|
scenario['roles']['router']['image']='sha256:'+'f'*64
|
||||||
|
with pytest.raises(deploy.DeployError,match='unreviewed target image'):
|
||||||
|
deploy.check(scenario['output'],plan,baseline=False)
|
||||||
|
|
||||||
|
|
||||||
|
def partial_old_plan(scenario):
|
||||||
|
plan=prepared(scenario)
|
||||||
|
overlay_path=scenario['output']/'compose.broker.json'
|
||||||
|
overlay=json.loads(overlay_path.read_text())
|
||||||
|
overlay['services']['router']['environment'].pop('HN_CONNECTOR_PUBLIC_KEY')
|
||||||
|
deploy.replace_json(overlay_path,overlay)
|
||||||
|
plan['prepared_hashes']['compose.broker.json']=deploy.digest(overlay_path)
|
||||||
|
deploy.replace_json(scenario['output']/'plan.json',plan)
|
||||||
|
image='sha256:'+'a'*64
|
||||||
|
deploy.write_json(scenario['output']/'candidate.json',{'image':image,'tag':plan['candidate_image']})
|
||||||
|
scenario['roles']['connector']['image']=image
|
||||||
|
scenario['roles']['connector']['state']={'status':'running','health':'healthy'}
|
||||||
|
scenario['roles']['router']['image']=image
|
||||||
|
scenario['roles']['router']['state']={'status':'restarting','health':'unhealthy'}
|
||||||
|
return plan
|
||||||
|
|
||||||
|
|
||||||
|
def test_repair_old_partial_switch_only_restarts_router_preserving_keys(scenario):
|
||||||
|
plan=partial_old_plan(scenario)
|
||||||
|
key=(scenario['output']/'credential-keyring.json').read_bytes()
|
||||||
|
original_env=(scenario['live']/'.env').read_bytes()
|
||||||
|
scenario['calls'].clear()
|
||||||
|
deploy.repair_router(scenario['output'],plan)
|
||||||
|
assert (scenario['output']/'credential-keyring.json').read_bytes()==key
|
||||||
|
assert (scenario['live']/'.env').read_bytes()==original_env
|
||||||
|
fixed=json.loads((scenario['output']/'compose.broker.json').read_text())
|
||||||
|
assert fixed['services']['router']['environment']['HN_CONNECTOR_PUBLIC_KEY']=='/trust/connector.pub'
|
||||||
|
saved=deploy.load_plan(scenario['output'])
|
||||||
|
assert saved['prepared_hashes']['compose.broker.json']==deploy.digest(scenario['output']/'compose.broker.json')
|
||||||
|
commands=[c for c in scenario['calls'] if 'up' in c]
|
||||||
|
assert len(commands)==1 and commands[0][-1]=='router' and commands[0][-2]=='180'
|
||||||
|
assert '--no-deps' in commands[0] and commands[0][-2:]!=['connector','router']
|
||||||
|
assert (scenario['output']/'plan.before-router-key.json').is_file()
|
||||||
|
assert (scenario['output']/'compose.broker.before-router-key.json').is_file()
|
||||||
|
|
||||||
|
|
||||||
|
def test_repair_retry_does_not_regenerate_key_or_overwrite_backup(scenario):
|
||||||
|
plan=partial_old_plan(scenario)
|
||||||
|
deploy.repair_router(scenario['output'],plan)
|
||||||
|
files={p.name:p.read_bytes() for p in scenario['output'].iterdir() if p.is_file()}
|
||||||
|
deploy.repair_router(scenario['output'],deploy.load_plan(scenario['output']))
|
||||||
|
assert files=={p.name:p.read_bytes() for p in scenario['output'].iterdir() if p.is_file()}
|
||||||
|
|
||||||
|
|
||||||
|
def test_repair_requires_healthy_known_connector(scenario):
|
||||||
|
plan=partial_old_plan(scenario)
|
||||||
|
scenario['roles']['connector']['state']['health']='unhealthy'
|
||||||
|
before=(scenario['output']/'compose.broker.json').read_bytes()
|
||||||
|
with pytest.raises(deploy.DeployError,match='healthy'):
|
||||||
|
deploy.repair_router(scenario['output'],plan)
|
||||||
|
assert (scenario['output']/'compose.broker.json').read_bytes()==before
|
||||||
|
assert not any('up' in c for c in scenario['calls'])
|
||||||
|
|
||||||
|
|
||||||
|
def test_old_plan_fails_normal_check_until_explicit_repair(scenario):
|
||||||
|
plan=partial_old_plan(scenario)
|
||||||
|
with pytest.raises(deploy.DeployError,match='HN_CONNECTOR_PUBLIC_KEY'):
|
||||||
|
deploy.check(scenario['output'],plan,baseline=False)
|
||||||
|
|
||||||
|
|
||||||
|
def test_cli_rollback_of_old_partial_plan_does_not_require_new_router_key(scenario,monkeypatch):
|
||||||
|
partial_old_plan(scenario)
|
||||||
|
monkeypatch.setattr(deploy.sys,'argv',['broker_deploy.py','rollback','--plan',str(scenario['output']),'--stop-broker-use'])
|
||||||
|
scenario['calls'].clear()
|
||||||
|
deploy.main()
|
||||||
|
commands=[c for c in scenario['calls'] if 'up' in c]
|
||||||
|
assert len(commands)==1 and commands[0][-2:]==['connector','router']
|
||||||
|
assert str(scenario['output']/'compose.rollback.json') in commands[0]
|
||||||
|
|
||||||
|
|
||||||
|
def test_missing_trust_public_key_refused_before_any_new_credentials(scenario):
|
||||||
|
trust=Path(scenario['base']['services']['router']['volumes'][0]['source'])
|
||||||
|
(trust/'connector.pub').unlink()
|
||||||
|
with pytest.raises(deploy.DeployError,match='public key'):
|
||||||
|
deploy.prepare(scenario['args'])
|
||||||
|
assert not scenario['output'].exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_run_failure_identifies_stage_without_echoing_command_secrets(monkeypatch):
|
||||||
|
import types
|
||||||
|
def fail(*a,**kw):
|
||||||
|
return types.SimpleNamespace(returncode=17,stdout='synthetic-access',stderr='synthetic-secret')
|
||||||
|
monkeypatch.setattr(deploy.subprocess,'run',fail)
|
||||||
|
with pytest.raises(deploy.DeployError) as error:
|
||||||
|
deploy.run(['docker','compose','--env-file','sensitive-path','up','connector'])
|
||||||
|
assert 'compose-up' in str(error.value) and '17' in str(error.value)
|
||||||
|
assert 'synthetic' not in str(error.value) and 'sensitive-path' not in str(error.value)
|
||||||
|
|
||||||
|
|
||||||
|
def test_log_signals_only_emit_fixed_error_classes():
|
||||||
|
assert deploy.log_signals('PermissionError: synthetic-secret\nread-only file system')==['PermissionError','read-only-filesystem']
|
||||||
|
assert deploy.log_signals('fake-secret https://secret.invalid')==[]
|
||||||
|
|
||||||
|
|
||||||
def test_catchall_before_private_block_requires_manual_review(scenario):
|
def test_catchall_before_private_block_requires_manual_review(scenario):
|
||||||
plan = prepared(scenario)
|
plan = prepared(scenario)
|
||||||
original = scenario["caddy"].read_text()
|
original = scenario["caddy"].read_text()
|
||||||
|
|||||||
Reference in New Issue
Block a user