fix(deploy): preserve legacy environment and diagnose prepared plans safely

The old initializer defaults HN_ENVIRONMENT to development, while migration
validation required the literal production and combined database/platform
prerequisites into one opaque error. Preserve supported environment identity
namespaces without editing the live .env; still require PostgreSQL and fixed
HTTPS/WSS endpoints for every existing-server migration. Match supported
boolean spellings and report failing field names without credential values.

Add read-only diagnose for an existing prepared plan so an earlier failed
prepare can be reviewed without deleting or regenerating its keyring. Move
future prerequisite validation before private directory/key generation.

Validation: 253 passed, 3 skipped (real PostgreSQL unavailable), 30 existing
warnings; 40 deployment-tool tests including 11 new compatibility/redaction
cases. Python 3.7 syntax and staged diff checks pass. Actual ECS diagnosis
must still confirm the current failing field; no live config or key changed.
This commit is contained in:
ex_tianzy
2026-11-04 13:59:22 +08:00
parent d3ce0099b7
commit 2657146334
4 changed files with 124 additions and 7 deletions
+10
View File
@@ -178,3 +178,13 @@ Windows / Python 3.12.10,平台桩、临时证书/密钥、SQLite及带Lua的F
修订Compose命令统一显式指定`--profile longconn`,不启用未知profile;保留服务集合严格校验并在不符时仅输出missing/extra服务名。后续up仍限connector/router且带--no-deps。新增3项回归模拟被profile隐藏的服务及缺失/多出服务的安全诊断,本地工具测试29项通过;现机实际解析结果仍需更新后重跑确认,不将该推断当作已经现场复现的唯一原因。 修订Compose命令统一显式指定`--profile longconn`,不启用未知profile;保留服务集合严格校验并在不符时仅输出missing/extra服务名。后续up仍限connector/router且带--no-deps。新增3项回归模拟被profile隐藏的服务及缺失/多出服务的安全诊断,本地工具测试29项通过;现机实际解析结果仍需更新后重跑确认,不将该推断当作已经现场复现的唯一原因。
修订后完整本地回归:**242 passed,3 skipped,30 warnings**(48.50秒)。跳过与warnings类型同上,宿主脚本Python3.7语法及diff检查通过。 修订后完整本地回归:**242 passed,3 skipped,30 warnings**(48.50秒)。跳过与warnings类型同上,宿主脚本Python3.7语法及diff检查通过。
### 准备阶段配置诊断兼容修订
现场prepare在platform/database配置检查处停止,未切换容器;原流程在最后验证前已生成私有plan/keyring,不得删除重建来绕过。核对旧初始化器默认environment=development,首版工具错误地要求字面production,可能误拒绝现有测试部署;同时数据库/开关条件合并导致现场无法辨别具体字段。
新版按现有Settings接受production/development/test,保持原身份命名空间和原.env,所有部署环境仍强制PostgreSQL、固定HTTPS平台/公开入口和WSS,不降低网络校验。布尔开关对齐Settings接受true/1/yes/on。错误只列字段,不回显数据库URL等敏感值;新增diagnose只读已有plan,无需重生成keyring。未来prepare在生成私有目录前先验证必要条件。
新增11项工具回归,覆盖合法环境保持、HTTPS/WSS/数据库失败在写密钥前拒绝、只读diagnose保留全部准备文件、错误不泄露数据库凭据;工具专项40项通过。实际现场哪项原始条件不符仍以更新后的diagnose输出为准,不将历史默认值当作当前配置证明。
修订后完整本地回归:**253 passed,3 skipped,30 warnings**(49.46秒);真实PostgreSQL仍未提供,warnings类型同上,Python3.7语法与diff检查通过。
+11
View File
@@ -53,6 +53,17 @@ prepare生成独立的root私有目录、随机AES密钥keyring、独立router
## 4. 检查与离线构建:仍不切换 ## 4. 检查与离线构建:仍不切换
如果旧版prepare返回`broker production platform/database configuration incomplete`,不要删除目录或重跑prepare。旧初始化器曾使用`HN_ENVIRONMENT=development`,首版工具把production写死,且把多个条件合并成同一错误。新版保留经过校验的原environment命名空间(production/development/test),所有模式仍要求PostgreSQL及HTTPS/WSS;不会为通过检查修改运行.env或将凭据换一个身份空间。
先更新暂存仓库,再对已经生成的计划执行只读诊断:
```bash
sudo python3 deployable/scripts/broker_deploy.py diagnose \
--plan /home/cnooc/hainuo-broker-prepared-20261104
```
只显示environment合法枚举、数据库类型及开关状态,不显示URL、账号密码、Secret或Token。通过后原地继续check,沿用原keyring和配置;若仍有错误按字段处理,不关闭HTTPS校验、不改用SQLite。新prepare在生成密钥前检查这些条件;旧目录因原检查失败留下的密钥不会被自动覆盖。
```bash ```bash
cd /home/cnooc/ecs-relay-service-staging cd /home/cnooc/ecs-relay-service-staging
sudo python3 deployable/scripts/broker_deploy.py check \ sudo python3 deployable/scripts/broker_deploy.py check \
+57 -7
View File
@@ -15,6 +15,7 @@ import secrets
import shutil import shutil
import subprocess import subprocess
import sys import sys
from urllib.parse import urlsplit
class DeployError(RuntimeError): class DeployError(RuntimeError):
@@ -93,6 +94,37 @@ def profiles_from(path):
raise DeployError("invalid product permission profiles; use the product-approved JSON object") from None raise DeployError("invalid product permission profiles; use the product-approved JSON object") from None
def enabled(value):
return str(value).strip().lower() in {"true","1","yes","on"}
def platform_problems(env):
problems = []
if not str(env.get("HN_DATABASE_URL", "")).startswith(("postgresql://","postgres://")):
problems.append("HN_DATABASE_URL: PostgreSQL connection required (value withheld)")
if env.get("HN_ENVIRONMENT") not in {"production","development","test"}:
problems.append("HN_ENVIRONMENT: expected production/development/test; preserve the reviewed environment")
for name in ("HN_FEISHU_ENABLED","HN_OAUTH_ENABLED"):
if not enabled(env.get(name)):
problems.append(name+": must be enabled")
for name,scheme in (("HN_PUBLIC_URL","https"),("HN_ROUTER_URL","wss"),
("HN_FEISHU_API_BASE","https"),("HN_FEISHU_AUTHORIZE_URL","https")):
try:
value = urlsplit(str(env.get(name,"")))
valid = value.scheme == scheme and value.hostname and not value.username and not value.password
except ValueError:
valid = False
if not valid:
problems.append(name+": fixed "+scheme.upper()+" endpoint required (value withheld)")
return problems
def require_platform(env):
problems = platform_problems(env)
if problems:
raise DeployError("broker platform configuration incomplete; "+"; ".join(problems))
def validate_overlay(base, merged, candidate): def validate_overlay(base, merged, candidate):
old, new = base["services"], merged["services"] old, new = base["services"], merged["services"]
require_services(old) require_services(old)
@@ -128,11 +160,10 @@ def validate_overlay(base, merged, candidate):
or env.get("HN_CREDENTIAL_KEYRING_FILE") != "/run/hainuo-credentials/keyring.json" or env.get("HN_CREDENTIAL_KEYRING_FILE") != "/run/hainuo-credentials/keyring.json"
or new["router"]["environment"].get("HN_CONNECTOR_INTERNAL_URL") != "http://connector:8000"): or new["router"]["environment"].get("HN_CONNECTOR_INTERNAL_URL") != "http://connector:8000"):
raise DeployError("application/environment or credential routing mismatch") raise DeployError("application/environment or credential routing mismatch")
if (not env.get("HN_DATABASE_URL", "").startswith(("postgresql://","postgres://")) # Old installer initializes HN_ENVIRONMENT=development. Preserve this identity
or env.get("HN_ENVIRONMENT") != "production" # namespace rather than silently converting it to production. All modes still
or str(env.get("HN_FEISHU_ENABLED", "")).lower() != "true" # require PostgreSQL and HTTPS/WSS for this existing-server migration.
or str(env.get("HN_OAUTH_ENABLED", "")).lower() != "true"): require_platform(env)
raise DeployError("broker production platform/database configuration incomplete")
for key in ("HN_ADMIN_SECRET","HN_INTERNAL_SECRET","HN_FEISHU_APP_SECRET"): for key in ("HN_ADMIN_SECRET","HN_INTERNAL_SECRET","HN_FEISHU_APP_SECRET"):
if env["HN_ROUTER_ADMISSION_SECRET"] == env.get(key): if env["HN_ROUTER_ADMISSION_SECRET"] == env.get(key):
raise DeployError("router admission credential must be independent") raise DeployError("router admission credential must be independent")
@@ -217,6 +248,7 @@ def prepare(args):
base = run(compose(plan)+["config","--format","json"],True) base = run(compose(plan)+["config","--format","json"],True)
if base["services"]["connector"].get("environment",{}).get("HN_CREDENTIAL_MODE","legacy") != "legacy": if base["services"]["connector"].get("environment",{}).get("HN_CREDENTIAL_MODE","legacy") != "legacy":
raise DeployError("already broker: preserve the existing keyring and migration plan") raise DeployError("already broker: preserve the existing keyring and migration plan")
require_platform({**base["services"]["connector"].get("environment",{}),"HN_OAUTH_ENABLED":"true"})
output.mkdir(mode=0o700,parents=False) output.mkdir(mode=0o700,parents=False)
os.chmod(str(output),0o700) os.chmod(str(output),0o700)
keyring = {"active_key_id":"initial-v1","keys":{"initial-v1":base64.b64encode(os.urandom(32)).decode("ascii")}} keyring = {"active_key_id":"initial-v1","keys":{"initial-v1":base64.b64encode(os.urandom(32)).decode("ascii")}}
@@ -269,6 +301,22 @@ def load_plan(root):
return plan return plan
def diagnose(root,plan):
config = run(compose(plan,root/"compose.broker.json")+["config","--format","json"],True)
env = config["services"]["connector"].get("environment",{})
value = env.get("HN_ENVIRONMENT")
print("HN_ENVIRONMENT: "+(value if value in {"production","development","test"} else "missing or invalid"))
print("HN_DATABASE_URL: "+("PostgreSQL configured (value withheld)" if str(env.get("HN_DATABASE_URL","")).startswith(("postgresql://","postgres://")) else "missing or unsupported (value withheld)"))
for name in ("HN_FEISHU_ENABLED","HN_OAUTH_ENABLED"):
print(name+": "+("enabled" if enabled(env.get(name)) else "disabled or missing"))
problems = platform_problems(env)
for problem in problems:
print("CHECK: "+problem)
print("Read-only diagnosis; existing preparation and keys preserved; no container restart.")
if problems:
raise DeployError("review the named fields above; do not paste configuration or regenerate keys")
def check(root, plan, baseline=True): def check(root, plan, baseline=True):
if os.name != "nt": if os.name != "nt":
if root.stat().st_mode & 0o077: if root.stat().st_mode & 0o077:
@@ -367,7 +415,7 @@ def main():
prep.add_argument("--profiles",required=True) prep.add_argument("--profiles",required=True)
prep.add_argument("--output",required=True) prep.add_argument("--output",required=True)
prep.add_argument("--connector-container",default="hainuo-relay-connector-1") prep.add_argument("--connector-container",default="hainuo-relay-connector-1")
for name in ("check","build","activate","rollback"): for name in ("diagnose","check","build","activate","rollback"):
cmd = sub.add_parser(name) cmd = sub.add_parser(name)
cmd.add_argument("--plan",required=True) cmd.add_argument("--plan",required=True)
if name == "activate": if name == "activate":
@@ -384,7 +432,9 @@ def main():
return return
root = Path(args.plan).resolve() root = Path(args.plan).resolve()
plan = load_plan(root) plan = load_plan(root)
if args.action == "check": if args.action == "diagnose":
diagnose(root,plan)
elif args.action == "check":
check(root,plan) check(root,plan)
elif args.action == "build": elif args.action == "build":
build(root,plan) build(root,plan)
+46
View File
@@ -33,6 +33,8 @@ def scenario(tmp_path,monkeypatch):
env = {"HN_ENVIRONMENT":"production","HN_DATABASE_URL":"postgresql://fake@postgres/fake", env = {"HN_ENVIRONMENT":"production","HN_DATABASE_URL":"postgresql://fake@postgres/fake",
"HN_FEISHU_ENABLED":"true","HN_OAUTH_ENABLED":"true","HN_FEISHU_APP_ID":"cli_test", "HN_FEISHU_ENABLED":"true","HN_OAUTH_ENABLED":"true","HN_FEISHU_APP_ID":"cli_test",
"HN_FEISHU_APP_SECRET":"synthetic-app-secret","HN_ADMIN_SECRET":"a"*40,"HN_INTERNAL_SECRET":"b"*40} "HN_FEISHU_APP_SECRET":"synthetic-app-secret","HN_ADMIN_SECRET":"a"*40,"HN_INTERNAL_SECRET":"b"*40}
env.update(HN_PUBLIC_URL="https://relay.example.test",HN_ROUTER_URL="wss://relay.example.test/device/ws",
HN_FEISHU_API_BASE="https://work.example.test",HN_FEISHU_AUTHORIZE_URL="https://work.example.test/open-apis/authen/v1/index")
base = {"services":{name:{"image":"old:tag","environment":{},"networks":{"backend":None},"volumes":[]} base = {"services":{name:{"image":"old:tag","environment":{},"networks":{"backend":None},"volumes":[]}
for name in [*roles,"postgres","redis"]},"volumes":{"metadata":{}},"networks":{"backend":{}}} for name in [*roles,"postgres","redis"]},"volumes":{"metadata":{}},"networks":{"backend":{}}}
base["services"]["connector"]["environment"] = env base["services"]["connector"]["environment"] = env
@@ -152,6 +154,50 @@ def test_prepare_refuses_existing_output_and_preserves_key(scenario):
assert (scenario["output"]/"credential-keyring.json").read_bytes() == key assert (scenario["output"]/"credential-keyring.json").read_bytes() == key
@pytest.mark.parametrize("environment", ["development","test","production"])
def test_preserves_existing_environment_namespace_with_tls(scenario,environment):
scenario["base"]["services"]["connector"]["environment"]["HN_ENVIRONMENT"] = environment
plan = prepared(scenario)
deploy.check(scenario["output"],plan)
merged = deploy.run(deploy.compose(plan,scenario["output"]/"compose.broker.json")+["config"],True)
assert merged["services"]["connector"]["environment"]["HN_ENVIRONMENT"] == environment
assert merged["services"]["router"]["environment"]["HN_ENVIRONMENT"] == environment
@pytest.mark.parametrize("name,value", [("HN_DATABASE_URL","sqlite://synthetic-secret"),
("HN_ENVIRONMENT","invalid-private-value"),("HN_FEISHU_ENABLED","false"),
("HN_PUBLIC_URL","http://synthetic-secret"),("HN_ROUTER_URL","ws://synthetic-secret"),
("HN_FEISHU_API_BASE","http://synthetic-secret")])
def test_prerequisite_failure_reports_field_before_generating_keys(scenario,name,value):
scenario["base"]["services"]["connector"]["environment"][name] = value
with pytest.raises(deploy.DeployError) as error:
deploy.prepare(scenario["args"])
assert name in str(error.value) and "synthetic-secret" not in str(error.value)
assert "invalid-private-value" not in str(error.value)
assert not scenario["output"].exists()
def test_diagnose_preserves_previously_generated_key_and_never_restarts(scenario,capsys):
plan = prepared(scenario)
before = {p.name:p.read_bytes() for p in scenario["output"].iterdir() if p.is_file()}
scenario["base"]["services"]["connector"]["environment"]["HN_ENVIRONMENT"] = "development"
scenario["calls"].clear()
deploy.diagnose(scenario["output"],plan)
assert "development" in capsys.readouterr().out
assert before == {p.name:p.read_bytes() for p in scenario["output"].iterdir() if p.is_file()}
assert not any("up" in c or "tag" in c or "build" in c for c in scenario["calls"])
def test_diagnose_never_echoes_database_credentials(scenario,capsys):
plan = prepared(scenario)
env = scenario["base"]["services"]["connector"]["environment"]
env["HN_DATABASE_URL"] = "unsupported://synthetic-db-secret@host"
with pytest.raises(deploy.DeployError):
deploy.diagnose(scenario["output"],plan)
output = capsys.readouterr().out
assert "HN_DATABASE_URL" in output and "synthetic-db-secret" not in output
def test_prepare_refuses_live_subdirectory(scenario): def test_prepare_refuses_live_subdirectory(scenario):
scenario["args"].output = str(scenario["live"]/"broker") scenario["args"].output = str(scenario["live"]/"broker")
with pytest.raises(deploy.DeployError,match="outside"): with pytest.raises(deploy.DeployError,match="outside"):