diff --git a/deployable/BINDING_TASK_LIFECYCLE.md b/deployable/BINDING_TASK_LIFECYCLE.md new file mode 100644 index 0000000..9698e51 --- /dev/null +++ b/deployable/BINDING_TASK_LIFECYCLE.md @@ -0,0 +1,69 @@ +# 阶段4:解绑、连接准入与任务关联指导 + +日期:2026-11-04;适用模式:显式`credential_mode=broker`。本文描述服务端开发结果和后续部署/客户端适配要求,不表示现机已切换。 + +## 1. ECS权威解绑 + +机器人`/解绑`、管理员`/admin/devices/{host}/unbind`、新设备接口`POST /work/v2/device/unbind`和吊销路径,共用`BindingState`事务逻辑。设备接口沿用v2设备签名和基础字段,不允许自行指定其他设备/用户;校验的owner/generation作为解绑条件,不能在并发重绑后清除新的关系。 + +数据库事务完成以下操作:清空owner或标记revoked,推进binding_generation,作废原设备未完成授权事务,终止该绑定版本在途/待回消息任务,推进用户配对码epoch,记录审计与失效outbox。事务中任意一步失败整体回滚,不返回假“已解绑”。解绑不删除CA/设备私钥,也不撤销用户平台OAuth。 + +解绑命令即使当前已无设备,也推进owner epoch,使其此前未消费的配对码失效。broker模式配对码以摘要存数据库,绑定时在事务中核验epoch/有效期并一次性消费,和实际设备绑定一起提交;无效设备签名不消费配对码。旧legacy Redis配对码在broker路径不被接受。 + +原设备后续Token领取、授权回调/状态恢复、通知及内部连接准入都核验当前绑定/代次;正常业务入口不依赖客户端自行清理才算解绑。Broker没有独立后台刷新调度,已无有效绑定时没有认证调用可触发正常领取刷新;后续后台保活仍须遵守同一绑定策略。 + +## 2. 失效事件补偿 + +`work_binding_events_v2`记录待执行失效事件。提交数据库后尝试删除旧在线presence、写吊销标记、发布带generation的断连通知;失败保留记录,当前或其他connector可补偿。维护周期5秒,每轮处理最多100条,未处理完继续后续轮次。 + +解绑接口返回`status=unbound`、`propagation=published/pending`;published只说明Redis失效动作已被接受,不表示PC已确认或所有在途平台调用已撤回。数据库始终是凭据和任务判定依据。 + +旧事件不允许删除较新generation的在线presence;使用比较删除避免竞争。新router也忽略不比自身generation新的断连事件,防止重试的旧通知误断新连接。事件发布成功但数据库确认失败时可以重复发布,不能据此重复业务任务。 + +## 3. Router独立内部准入 + +新配置`HN_ROUTER_ADMISSION_SECRET`只用于router查询绑定/任务准入,至少32字符,不能复用管理员、事件内部Secret或App Secret。connector和router各自获得该凭据,router不需要获得机器人事件接入Secret或应用Secret。新配置及挂载的正式Compose接线在阶段5完成。 + +router还需要公开`HN_FEISHU_APP_ID`、同环境标识、`HN_CONNECTOR_INTERNAL_URL`(默认`http://connector:8000`)及已有connector签名公钥。内部URL为固定运维配置,禁止用户名密码/query/fragment和自动重定向;请求禁用环境代理,超时3秒。 + +内部接口:`POST /internal/devices/{host}/admission`,头为`X-Router-Admission-Secret`,正文是随机nonce及可选turn_id/session_id。它不返回用户Token,只返回connector签名的环境/应用、设备/owner、当前generation/证书指纹、nonce/任务/session和5秒期限。router验证签名、nonce、用途上下文与期限;不能拿事件Secret调用该接口,也不能拿准入Secret注入`/internal/events`。 + +准入在握手、订阅后、每次指令下发、结果上行以及8秒周期检查时重新核对ECS状态。任务校验还要求当前generation、原session、inflight状态、deadline及所属worker仍存活。控制服务不可用时关闭或拒绝连接/转发,不继续用陈旧缓存接受任务;这牺牲部分可用性换取绑定控制。 + +即时断连优先走outbox通知;通知丢失时周期检查可在下一个检查周期发现无效绑定,实际时延还包括内部请求和调度。不能承诺数据库提交瞬间物理连接同步消失;已提交到平台的写操作也不能撤回。 + +## 4. 新设备WSS握手与任务信封 + +broker模式设备proof保留host_id、challenge、session_id、kx、ts,并增加严格整数`protocol_version=2`、environment、app_id、binding_owner、binding_generation;全部由设备签名。旧握手在broker模式拒绝,legacy模式保持旧协议。 + +connector下行消息新增routing签名,绑定turn_id、host、session、环境/应用、generation和完整密文envelope摘要。router验证后向内部接口查询该任务的当前准入,再下发原密文。密文内task_context包含task_id、environment、app_id、owner、host、generation、conversation_id、chat_id、source_message_id及Work通道。 + +**客户端尚未在本阶段修改。** 后续PC桥接必须发送新握手,将可信task_context传入原生API、MCP/工具运行时,并保留Profile与工具禁用限制。本机旧适配器的稳定单聊session键不能替代这个上下文,也不能把服务端路由测试说成已完成客户端Agent记忆/并发隔离。 + +当前仍只接受p2p文本事件;新模式要求有效chat_id,不截断后混用会话键。conversation_id按环境、应用、用户、chat_id和Work单聊通道确定。群聊、话题和多Profile产品行为未擅自扩展。 + +## 5. 持久任务状态与原路回复 + +`work_tasks_v2`保存原消息/会话、owner、设备、generation、session、worker、deadline和状态,**不保存明文任务或回答正文**。同一turn_id不重复建立/执行。密文结果必须通过设备签名/解密、原turn、原session及数据库当前状态验证,才能完成本进程pending future。 + +典型状态为`inflight → completed → replying → replied`。completed仅表示已收到有效结果,replying表示单次平台回复已认领;回复出错为reply_unknown,不因消息重复投递自动重发。诊断任务无原消息时可标记finished。解绑/吊销变为unbound/revoked,超时或任务所有者失联为unknown,迟到结果不能改变终态。 + +机器人回复位置从数据库原消息映射取得并进行单次认领,不接受客户端/模型改写chat_id/message_id。不同用户或同用户不同任务逆序返回仍回原消息;取消/解绑后不把旧结果投到新消息。解绑命令自身的确认回复不等于继续发送被取消任务的答案。 + +worker采用随机进程ID与30秒Redis租约,每5秒续期。维护扫描轮转检查活跃任务;进程退出或租约消失后遗留任务在后续维护/准入检查中拒绝。进程恢复不重放旧任务,不声称恢复已丢失的内存结果或提供持久消息队列。已发到平台但回复结果未知时保留reply_unknown,不承诺exactly-once。 + +连接/请求退出时先释放内存连接槽和pending记录;数据库状态更新或Redis清理失败不得阻止本地释放,遗留在线键由TTL清理、任务由持久状态维护处理,不凭遗留presence继续放行旧绑定。 + +## 6. 通知与凭据的解绑边界 + +broker模式`/device/notifications`新增必填binding_generation并纳入原work-notify-v1签名payload;缺失/陈旧代次拒绝。取得应用Token后发送前再查owner/generation,防止等待期间同用户重绑导致旧请求继续发消息。legacy可不传该字段,原签名格式保持不变。 + +授权卡仍引用server-owned事务,发送前再查事务状态和代次。设备解绑后停止再次发Token和通知,但**已发出的用户Bearer access token仍可能有效到期**;是否即时撤销取决于平台能力,不能把DB解绑称为平台授权撤销。正常客户端停止重连、清access缓存的交互需要后续客户端适配。 + +## 7. 配置、测试与待办 + +默认legacy及现有客户端不切换。启用阶段4 broker之前,必须同时准备router独立准入配置、内部网络、公钥、产品权限和新PC协议;新旧broker握手不混跑。不能为了旧客户端放开无绑定准入或重新下发refresh。 + +本地使用平台桩、加密合成凭据、SQLite、FakeRedis和真实回环WebSocket,验证协议和并发。PostgreSQL用例扩充配对epoch/outbox检查,但没有真实连接时明确skipped;FakeRedis不等于真实Redis多节点/故障演练。 + +阶段5待做:Compose与初始化密钥/权限参数接线、内部网络/证书检查、候选ARM64构建、客户端握手与上下文适配交接、数据留存清理、outbox/worker/任务监控、旧模式迁移和真实平台/双用户/规模验收。8秒连接复核会增加内部控制请求,必须实测容量后再扩大设备规模,不外推十万人能力。 diff --git a/deployable/CREDENTIAL_MIGRATION.md b/deployable/CREDENTIAL_MIGRATION.md index fc16dfb..636b4ff 100644 --- a/deployable/CREDENTIAL_MIGRATION.md +++ b/deployable/CREDENTIAL_MIGRATION.md @@ -11,7 +11,7 @@ | 1 | 服务端加密凭据存储基础 | 密文持久化、应用/用户/环境隔离、版本冲突、密钥轮换与篡改检测测试;不接入旧路由 | 已完成本地开发测试,提交71b9678 | | 2 | 集中凭据Broker及刷新 | 应用/用户锁、持锁重读、版本CAS、响应校验、刷新unknown;新授权不受旧刷新覆盖 | 本次完成本地开发测试 | | 3 | OAuth与认证领取 | 共享state事务、实际用户核验、绑定代次、只返回access的独立DTO及领取认证 | 本次完成本地开发测试,显式broker模式启用 | -| 4 | ECS权威解绑与消息任务关联 | 停发Token、失效事务/任务、连接和旧码联动,多实例/异常恢复 | 待开发 | +| 4 | ECS权威解绑与消息任务关联 | 停发Token、失效事务/任务、连接和旧码联动,多实例/异常恢复 | 本次完成本地开发测试,broker模式生效 | | 5 | 配置部署、迁移与旧接口退出 | 密钥挂载、产品权限配置、客户端对接、停用旧刷新/claim及代理、ARM64和真实平台验收 | 待开发 | 产品经理决定功能和Scope清单。阶段1不预设具体权限,不生成真实凭据、不读取现有设备、不调用平台。 @@ -126,3 +126,28 @@ Windows / Python 3.12.10,平台桩、临时证书/密钥、SQLite及带Lua的F 最后完整回归:**185 passed,3 skipped,15 warnings**(49.77秒)。3个跳过项均需要真实PostgreSQL测试连接;warnings仍为原有uvicorn/websockets弃用提示。文档表格/围栏、relay模块AST及`git diff --check`通过。 真实PostgreSQL、真实Redis多进程、ARM64镜像、私有化平台及现机业务验收尚未执行;本地ASGI/FakeRedis结果不能作为线上多节点或生产部署结论。 + +## 阶段4:ECS权威解绑、连接准入与任务关联(2026-11-04) + +细节见[阶段4协议和边界](BINDING_TASK_LIFECYCLE.md)。本次仅修改ECS项目,未修改或发布PC安装包,未部署现机。 + +### 改造内容 + +- 新增`BindingState`及Store生命周期挂钩,在broker模式统一bind/unbind/revoke,数据库事务内更改绑定代次、作废未完成授权/任务/配对码epoch、记录审计和outbox。legacy路径保持原行为。 +- 新增认证设备解绑入口;机器人命令、管理员和设备解绑共用同一权威状态。旧配对码按owner epoch拒绝;新broker配对码只存摘要,和绑定一起一次性消费,不接受原legacy Redis码。 +- Redis失效动作失败时保留outbox供本节点/其他connector重试;删除在线presence有比较保护,断连消息包含代次,旧事件不误断新连接。接口published/pending不等于PC确认。 +- router新增独立内部准入Secret,和事件接入Secret分权;请求带nonce并验证connector签名的短期准入结果,不取得Token。握手/订阅后/任务上下行/周期均复核数据库绑定及任务;控制服务失败时拒绝转发。 +- 新v2设备握手绑定应用、环境、owner和generation;任务下行带签名routing及密文摘要。新模式旧PC握手会拒绝,不能宣称对现有安装透明兼容。 +- 持久任务记录关联原message/chat/conversation、设备、generation、session、worker及deadline;结果逆序仍按原任务交付,平台回复单次认领;迟到/解绑/死亡worker/超时不改变终态、不重放写任务。 +- worker使用30秒Redis存活租约和5秒维护;任务扫描轮转避免后续任务饥饿;无结果正文持久化,不承诺重启后恢复丢失的内存结果。 +- 通知增加broker必填generation并在等待平台凭据后复查,防止同用户重绑后旧通知继续发送;legacy原签名兼容。应用及用户refresh均不下发。 + +### 本地验证 + +新增阶段4专项28项:真实回环WebSocket而非纯函数桩,覆盖两用户/同用户多任务逆序、机器人解绑断连与旧回包拒绝、数据库提交/Redis失败补偿、事务回滚、另一connector重开补偿、旧码拒绝与一次消费、迟到kill不影响新连接、通知期间重绑不发、吊销与旧握手拒绝、控制面中断拒绝转发、周期查库发现解绑、死亡worker和reply_unknown不重放,以及DB/Redis清理失败仍释放内存连接/等待记录。扩展可选PostgreSQL用例覆盖owner epoch和outbox,但无连接时仍跳过。 + +专项测试不代表真实Hermes客户端的记忆隔离已经完成;本次测试用合成PC设备/平台桩和本地SQLite/FakeRedis,客户端可信TaskContext传入Agent/MCP仍需后续适配。默认旧模式、服务器配置和真实凭据均未切换。 + +最后完整回归:**213 passed,3 skipped,30 warnings**(45.87秒)。3个跳过项均需要真实PostgreSQL连接,warnings为既有uvicorn/websockets旧API弃用提示(新增真实回环连接增加提示次数);relay模块AST、文档和diff检查通过。 + +生产部署、真实Redis/PostgreSQL多实例、ARM64镜像、平台实际撤权/Scope/容量和客户端安装升级仍待阶段5及现场验收。平台已接收的请求不可靠解绑撤回,已发access仍可能有效到期。 diff --git a/deployable/CREDENTIAL_PROTOCOL_V2.md b/deployable/CREDENTIAL_PROTOCOL_V2.md index 0de2be2..3a36ded 100644 --- a/deployable/CREDENTIAL_PROTOCOL_V2.md +++ b/deployable/CREDENTIAL_PROTOCOL_V2.md @@ -1,6 +1,6 @@ # 集中凭据协议 v2:阶段3接口指导 -日期:2026-11-04。本文是已经完成本地开发测试的服务端接口契约,不表示现机已启用或新客户端已交付。产品经理决定权限集合;本文件不替产品选择具体业务或Scope。 +日期:2026-11-04。本文是已经完成本地开发测试的服务端接口契约,不表示现机已启用或新客户端已交付。产品经理决定权限集合;本文件不替产品选择具体业务或Scope。阶段4的解绑、内部准入和WSS扩展见[连接与任务生命周期](BINDING_TASK_LIFECYCLE.md)。 ## 1. 启用与模式边界 @@ -12,6 +12,7 @@ Settings新增以下配置,环境变量均以`HN_`开头: | `CREDENTIAL_KEYRING_FILE` | broker必须提供受控服务端密钥文件,格式见阶段1记录;无有效密钥启动失败,不回退 | | `CREDENTIAL_PERMISSION_PROFILES` | JSON对象:产品权限版本名映射Scope字符串数组;broker不可为空,不允许客户端传任意Scope | | `DATABASE_URL` | 生产broker必须使用PostgreSQL;SQLite仅用于本地开发和测试 | +| `ROUTER_ADMISSION_SECRET` | 阶段4新增:connector与router共用的独立内部准入凭据,不能复用事件/管理/应用Secret;需受控挂载 | | 平台与OAuth | broker要求`FEISHU_ENABLED`和`OAUTH_ENABLED`为true,应用凭据仍只在ECS | | 授权入口与公开URL | 固定HTTPS授权端点,不能含预设query/fragment或用户名密码;公开HTTPS回调精确登记 | @@ -84,9 +85,9 @@ Broker换码后调用user_info核验实际授权人与原绑定一致,回写 ### POST /device/notifications(原接口的新授权卡兼容) -文本/通知卡协议不变。broker模式的`kind=authorization`将原`oauth_state`字段用作**transaction_id引用**,不是原始state或任意URL;ECS验证事务归属、当前绑定版本和waiting状态后从服务器生成链接。发送前再校验,不能把其他设备事务发给本人或用旧代次链接继续授权。 +legacy文本/通知卡协议不变;broker模式从阶段4起必须传当前`binding_generation`并纳入work-notify-v1签名payload。broker模式的`kind=authorization`将原`oauth_state`字段用作**transaction_id引用**,不是原始state或任意URL;ECS验证事务归属、当前绑定版本和waiting状态后从服务器生成链接。发送前再校验,不能把其他设备事务发给本人或用旧代次链接继续授权。 -本人通知收件人与路由继续由ECS确定,应用Token不下发。完整通知、路由和旧绑定码失效补偿在阶段4继续完善。 +本人通知收件人与路由继续由ECS确定,应用Token不下发。阶段4已补当前绑定版本与旧绑定码失效、路由准入及持久补偿;客户端侧对应升级和真实部署仍需后续验证。 ## 4. 错误与审计 @@ -99,4 +100,4 @@ Broker换码后调用user_info核验实际授权人与原绑定一致,回写 ## 5. 本阶段尚未完成 -无实际服务器部署、真实PostgreSQL/多节点Redis验收、真实Work OAuth复测或客户端发布。完整解绑连接和任务补偿、旧绑定码作废、凭据拒绝自动失效上报、集中刷新后台调度、运维清理和正式Compose参数挂载为后续阶段。不要绕过新领取认证临时开放旧refresh接口。 +无实际服务器部署、真实PostgreSQL/多节点Redis验收、真实Work OAuth复测或客户端发布。阶段4新增的解绑/任务/旧绑定码补偿以对应文档和本地测试为准;凭据拒绝自动失效上报、集中刷新后台调度、运维清理和正式Compose参数挂载仍为后续工作。不要绕过新领取认证临时开放旧refresh接口。 diff --git a/deployable/README.md b/deployable/README.md index adbb431..a789b47 100644 --- a/deployable/README.md +++ b/deployable/README.md @@ -2,6 +2,8 @@ 集中凭据改造按阶段交付,见 [改造与阶段记录](CREDENTIAL_MIGRATION.md)。阶段1/2已提供存储与刷新,阶段3新增显式broker模式的OAuth及认证领取接口,默认仍为legacy;[v2接口契约](CREDENTIAL_PROTOCOL_V2.md)包含签名、配置和兼容边界。尚未部署或发布新客户端,不能据此判断现网已切换。 +阶段4新增[权威解绑、连接准入和任务生命周期](BINDING_TASK_LIFECYCLE.md),broker模式要求router独立准入凭据及PC v2握手。默认legacy不变;部署接线和客户端适配未完成前不要直接切换生产模式。 + 2026-10-28 新增[客户端主动消息与授权卡片](NOTIFICATIONS.md),本地代码已实现,现网部署和卡片验收待完成。 本目录用于新环境部署,源码对应 2026-10-27 ECS connector/router/longconn 导出版本。协议和业务源码保持原样。仓库其他快照目录用于追溯,不需要手工合并。 diff --git a/deployable/relay/binding_state.py b/deployable/relay/binding_state.py new file mode 100644 index 0000000..7cb20aa --- /dev/null +++ b/deployable/relay/binding_state.py @@ -0,0 +1,182 @@ +"""Transactional broker-mode binding, pairing, invalidation and task metadata. + +No message bodies or credential values are stored here. All mutating binding +paths in a broker connector use this object through Store's lifecycle hook. +""" +import hashlib +import time +import uuid + + +class BindingState: + def __init__(self, store, environment, app_id): + self.store, self.db, self.environment, self.app_id = store, store.db, environment, app_id + with self.db: + self.db.execute("""CREATE TABLE IF NOT EXISTS work_owner_epochs_v2( + environment TEXT NOT NULL,app_id TEXT NOT NULL,owner TEXT NOT NULL,epoch BIGINT NOT NULL, + PRIMARY KEY(environment,app_id,owner))""") + self.db.execute("""CREATE TABLE IF NOT EXISTS work_binding_codes_v2( + code_hash TEXT PRIMARY KEY,environment TEXT NOT NULL,app_id TEXT NOT NULL, + owner TEXT NOT NULL,owner_epoch BIGINT NOT NULL,expires_at DOUBLE PRECISION NOT NULL, + consumed INTEGER NOT NULL DEFAULT 0)""") + self.db.execute("""CREATE TABLE IF NOT EXISTS work_binding_events_v2( + event_id TEXT PRIMARY KEY,environment TEXT NOT NULL,app_id TEXT NOT NULL,host_id TEXT NOT NULL, + owner TEXT NOT NULL,binding_generation BIGINT NOT NULL,reason TEXT NOT NULL, + created_at DOUBLE PRECISION NOT NULL,delivered INTEGER NOT NULL DEFAULT 0)""") + self.db.execute("""CREATE TABLE IF NOT EXISTS work_tasks_v2( + turn_id TEXT PRIMARY KEY,environment TEXT NOT NULL,app_id TEXT NOT NULL,owner TEXT NOT NULL, + host_id TEXT NOT NULL,binding_generation BIGINT NOT NULL,session_id TEXT NOT NULL, + worker_id TEXT NOT NULL,conversation_id TEXT NOT NULL,chat_id TEXT NOT NULL, + source_message_id TEXT NOT NULL,status TEXT NOT NULL,deadline DOUBLE PRECISION NOT NULL, + created_at DOUBLE PRECISION NOT NULL,updated_at DOUBLE PRECISION NOT NULL)""") + self.db.execute("""CREATE INDEX IF NOT EXISTS work_binding_events_pending_v2 + ON work_binding_events_v2(environment,app_id,delivered,created_at)""") + self.db.execute("""CREATE INDEX IF NOT EXISTS work_tasks_active_v2 + ON work_tasks_v2(environment,app_id,status,updated_at)""") + self.db.execute("""CREATE INDEX IF NOT EXISTS work_tasks_binding_v2 + ON work_tasks_v2(environment,app_id,host_id,binding_generation)""") + + def _owner_lock(self, owner, bump=False): + identity = (self.environment, self.app_id, owner) + self.db.execute("INSERT INTO work_owner_epochs_v2 VALUES(?,?,?,0) ON CONFLICT DO NOTHING", identity) + self.db.execute("UPDATE work_owner_epochs_v2 SET epoch=epoch+? WHERE environment=? AND app_id=? AND owner=?", + (1 if bump else 0, *identity)) + return self.db.execute("SELECT epoch FROM work_owner_epochs_v2 WHERE environment=? AND app_id=? AND owner=?", + identity).fetchone()["epoch"] + + def _bind(self, host, owner): + count = self.db.execute("""UPDATE devices SET owner=?,binding_generation=binding_generation+1 + WHERE host_id=? AND owner IS NULL AND revoked=0""", (owner, host)).rowcount + if count != 1: + raise ValueError("device unavailable or already bound") + + def bind(self, host, owner): + with self.db: + self._owner_lock(owner) + self._bind(host, owner) + + def issue_code(self, owner, code, ttl=300): + with self.db: + epoch = self._owner_lock(owner) + if self.db.execute("SELECT host_id FROM devices WHERE owner=?", (owner,)).fetchone(): + raise ValueError("owner already bound") + self.db.execute("INSERT INTO work_binding_codes_v2 VALUES(?,?,?,?,?,?,0)", + (hashlib.sha256(code.encode()).hexdigest(), self.environment, self.app_id, owner, epoch, time.time()+ttl)) + + def consume_code(self, host, code): + digest = hashlib.sha256(code.encode()).hexdigest() + row = self.db.execute("SELECT * FROM work_binding_codes_v2 WHERE code_hash=? AND environment=? AND app_id=?", + (digest, self.environment, self.app_id)).fetchone() + if not row: + raise LookupError("binding code unavailable") + with self.db: + epoch = self._owner_lock(row["owner"]) + count = self.db.execute("""UPDATE work_binding_codes_v2 SET consumed=1 + WHERE code_hash=? AND owner_epoch=? AND consumed=0 AND expires_at>?""", + (digest, epoch, time.time())).rowcount + if count != 1: + raise LookupError("binding code unavailable") + self._bind(host, row["owner"]) + return row["owner"] + + def _cancel(self, device, reason): + owner = device["owner"] or "" + generation = device["binding_generation"] + self.db.execute("""UPDATE work_authorizations_v2 SET status='invalidated' + WHERE environment=? AND app_id=? AND host_id=? AND binding_generation<=? + AND status IN ('waiting','exchanging')""", + (self.environment, self.app_id, device["host_id"], generation)) + self.db.execute("""UPDATE work_tasks_v2 SET status=?,updated_at=? + WHERE environment=? AND app_id=? AND host_id=? AND binding_generation<=? + AND status IN ('inflight','completed','replying')""", + ("revoked" if reason == "revoke" else "unbound", time.time(), self.environment, + self.app_id, device["host_id"], generation)) + event = uuid.uuid4().hex + self.db.execute("INSERT INTO work_binding_events_v2 VALUES(?,?,?,?,?,?,?,?,0)", + (event, self.environment, self.app_id, device["host_id"], owner, generation+1, reason, time.time())) + self.db.execute("INSERT INTO audit(ts,actor,action,resource,result) VALUES(?,?,?,?,?)", + (time.time(), owner or "admin", "binding_"+reason, device["host_id"], "committed")) + + def unbind(self, owner): + with self.db: + self._owner_lock(owner, bump=True) # Invalidates pre-unbind codes even if already unbound. + device = self.db.execute("SELECT * FROM devices WHERE owner=?", (owner,)).fetchone() + if device: + self.db.execute("UPDATE devices SET owner=NULL,binding_generation=binding_generation+1 WHERE host_id=?", + (device["host_id"],)) + self._cancel(dict(device), "unbind") + + def invalidate_host(self, host, reason, *, expected_owner=None, expected_generation=None): + if reason not in {"unbind", "revoke"}: + raise ValueError("invalid binding change") + device = self.store.device(host) + if not device: + raise LookupError("device not found") + if ((expected_owner is not None and device["owner"] != expected_owner) + or (expected_generation is not None and device["binding_generation"] != expected_generation)): + raise ValueError("binding changed; retry") + if reason == "unbind" and not device["owner"]: + return + if reason == "revoke" and device["revoked"]: + return + with self.db: + if device["owner"]: + self._owner_lock(device["owner"], bump=True) + mutation = "owner=NULL" if reason == "unbind" else "revoked=1" + count = self.db.execute(f"""UPDATE devices SET {mutation},binding_generation=binding_generation+1 + WHERE host_id=? AND binding_generation=? AND COALESCE(owner,'')=? AND revoked=?""", + (host, device["binding_generation"], device["owner"] or "", device["revoked"])).rowcount + if count != 1: + raise ValueError("binding changed; retry") + self._cancel(device, reason) + + def task(self, turn): + row = self.db.execute("SELECT * FROM work_tasks_v2 WHERE turn_id=? AND environment=? AND app_id=?", + (turn, self.environment, self.app_id)).fetchone() + return dict(row) if row else None + + def task_current(self, task): + device = self.store.device(task["host_id"]) if task else None + return bool(device and not device["revoked"] and device["owner"] == task["owner"] + and device["binding_generation"] == task["binding_generation"] + and task["deadline"] > time.time()) + + def create_task(self, turn, owner, host, generation, session, worker, deadline, chat_id="", message_id=""): + conversation = hashlib.sha256((self.environment+"\0"+self.app_id+"\0"+owner+"\0"+chat_id+"\0work-dm").encode()).hexdigest() + with self.db: + self._owner_lock(owner) + device = self.store.device(host) + if not device or device["revoked"] or device["owner"] != owner or device["binding_generation"] != generation: + raise ValueError("binding changed") + count = self.db.execute("""INSERT INTO work_tasks_v2 VALUES(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?) + ON CONFLICT DO NOTHING""", (turn,self.environment,self.app_id,owner,host,generation,session, + worker,conversation,chat_id,message_id,"inflight",deadline,time.time(),time.time())).rowcount + if count != 1: + raise ValueError("task already recorded") + return self.task(turn) + + def transition(self, turn, previous, status): + with self.db: + return self.db.execute("""UPDATE work_tasks_v2 SET status=?,updated_at=? + WHERE turn_id=? AND environment=? AND app_id=? AND status=?""", + (status,time.time(),turn,self.environment,self.app_id,previous)).rowcount == 1 + + def complete(self, turn, host, session, worker): + with self.db: + task = self.task(turn) + if not task: + return False + self._owner_lock(task["owner"]) + if not self.task_current(task) or (task["host_id"],task["session_id"],task["worker_id"],task["status"]) != (host,session,worker,"inflight"): + return False + return self.db.execute("UPDATE work_tasks_v2 SET status='completed',updated_at=? WHERE turn_id=? AND status='inflight'", + (time.time(),turn)).rowcount == 1 + + def claim_reply(self, turn, owner, message_id): + with self.db: + self._owner_lock(owner) + task = self.task(turn) + if not self.task_current(task) or (task["owner"],task["source_message_id"],task["status"]) != (owner,message_id,"completed"): + return False + return self.db.execute("UPDATE work_tasks_v2 SET status='replying',updated_at=? WHERE turn_id=? AND status='completed'", + (time.time(),turn)).rowcount == 1 diff --git a/deployable/relay/config.py b/deployable/relay/config.py index bc2b7af..a46b485 100644 --- a/deployable/relay/config.py +++ b/deployable/relay/config.py @@ -44,11 +44,17 @@ class Settings(BaseSettings): credential_mode: str = "legacy" credential_keyring_file: Path | None = None credential_permission_profiles: dict[str, list[str]] = {} + connector_internal_url: str = "http://connector:8000" + router_admission_secret: str = "" @model_validator(mode="after") def validate_runtime(self): if self.credential_mode not in {"legacy", "broker"}: raise ValueError("invalid credential mode") + if self.credential_mode == "broker" and self.role in {"connector","router"}: + if (len(self.router_admission_secret) < 32 or self.router_admission_secret in { + self.internal_secret,self.admin_secret,self.feishu_app_secret}): + raise ValueError("broker mode requires an independent router admission credential") for profile, scopes in self.credential_permission_profiles.items(): if (not re.fullmatch(r"[A-Za-z0-9_-]{1,64}", profile) or not scopes or len(scopes) > 128 or any(not re.fullmatch(r"[A-Za-z0-9_:.-]{1,256}", scope) for scope in scopes)): @@ -63,6 +69,12 @@ class Settings(BaseSettings): if (authorize.scheme != "https" or not authorize.hostname or authorize.username or authorize.password or authorize.query or authorize.fragment): raise ValueError("broker mode requires a fixed HTTPS authorization endpoint") + if self.credential_mode == "broker" and self.role == "router": + target = urlsplit(self.connector_internal_url) + if (not self.feishu_app_id or target.scheme not in {"http","https"} + or not target.hostname or target.username or target.password or target.query or target.fragment + or target.path not in {"","/"}): + raise ValueError("broker router requires authenticated fixed internal connector") if self.environment not in {"production", "test", "development"}: raise ValueError("invalid environment") if self.role == "connector" and min(len(self.admin_secret), len(self.internal_secret)) < 32: diff --git a/deployable/relay/connector.py b/deployable/relay/connector.py index 81d4e00..5dc04f2 100644 --- a/deployable/relay/connector.py +++ b/deployable/relay/connector.py @@ -24,7 +24,7 @@ from redis.asyncio import Redis from .common import BodyLimitMiddleware, no_cache, peer, rate_limit, require_secret from .config import Settings -from .crypto import (fingerprint, load_private, open_envelope, raw_public, seal, +from .crypto import (canonical, fingerprint, load_private, open_envelope, raw_public, seal, sign, verified, verify_device) from .oauth import OAuth from .platform import Feishu, PlatformError @@ -62,6 +62,8 @@ class Broker: self.s, self.r, self.store = settings, redis, store self.signer = load_private(settings.connector_key) self.pending: dict[str, Pending] = {} + self.worker_id = uuid.uuid4().hex + self.lifecycle = None async def replies(self, pub): async for message in pub.listen(): @@ -79,12 +81,14 @@ class Broker: result = plain.get("result", "") if not isinstance(result, str) or len(result) > self.s.max_result: continue + if self.lifecycle and not self.lifecycle.complete(data["turn_id"],item.host,item.session,self.worker_id): + continue item.future.set_result({"status": "completed", "turn_id": data["turn_id"], "result": result}) except Exception: # Invalid packets never consume a pending result; do not log packet contents. continue - async def dispatch(self, owner, text, turn): + async def dispatch(self, owner, text, turn, *, chat_id="", source_message_id=""): device = self.store.by_owner(owner) if not device: return {"status": "unavailable", "turn_id": turn, "result": "未绑定设备或设备不可用。"} @@ -104,36 +108,75 @@ class Broker: if proof["host_id"] != host or proof["challenge"] != presence["challenge"]: raise ValueError("invalid device session") session = proof["session_id"] + if self.lifecycle and (proof.get("binding_owner") != owner + or proof.get("binding_generation") != device["binding_generation"]): + raise ValueError("stale binding session") except Exception: self.store.audit(owner, "dispatch", host, "identity_rejected") raise HTTPException(409, "device identity rejected") from None if not self.store.begin_turn(turn, owner, host): return {"status": "duplicate", "turn_id": turn, "result": "该请求已接收,不会重复执行。"} + task = None + if self.lifecycle: + try: + task = self.lifecycle.create_task(turn,owner,host,device["binding_generation"],session, + self.worker_id,time.time()+self.s.request_timeout,chat_id,source_message_id) + except ValueError: + self.store.finish(turn,"unbound") + return {"status":"unbound","turn_id":turn,"result":"绑定已变化,任务未下发。"} private = x25519.X25519PrivateKey.generate() future = asyncio.get_running_loop().create_future() self.pending[turn] = Pending(future, private, public, host, session) - envelope = seal({"type": "instruction", "turn_id": turn, "owner": owner, + payload = {"type": "instruction", "turn_id": turn, "owner": owner, "text": text, "read_only": True, "channel": "im", - "reply_key": raw_public(private.public_key())}, proof["kx"], self.signer, - host_id=host, session_id=session) + "reply_key": raw_public(private.public_key())} + if task: + payload["task_context"] = {key:task[key] for key in ("environment","app_id","owner","host_id","binding_generation", + "conversation_id","chat_id","source_message_id")} + payload["task_context"].update(task_id=turn,channel="work-dm") + try: + envelope = seal(payload, proof["kx"], self.signer,host_id=host,session_id=session) + except Exception: + self.pending.pop(turn,None) + self.store.finish(turn,"unknown") + if task: + self.lifecycle.transition(turn,"inflight","unknown") + raise HTTPException(409,"invalid_device_crypto_session") from None key = f"hn:inflight:{host}:{turn}" try: await self.r.set(key, session, ex=self.s.request_timeout + 5) - count = await self.r.publish("hn:down:" + host, json.dumps({"type": "instruction", "envelope": envelope})) + packet = {"type":"instruction","envelope":envelope} + if task: + packet["routing"] = sign({"turn_id":turn,"host_id":host,"session_id":session, + "environment":self.s.environment,"app_id":self.s.feishu_app_id, + "binding_generation":task["binding_generation"], + "envelope_hash":hashlib.sha256(canonical(envelope)).hexdigest()}, self.signer) + count = await self.r.publish("hn:down:" + host, json.dumps(packet)) if not count: self.store.finish(turn, "offline") + if task: + self.lifecycle.transition(turn,"inflight","offline") return {"status": "offline", "turn_id": turn, "result": "你的电脑当前离线。"} result = await asyncio.wait_for(future, timeout=self.s.request_timeout) - self.store.finish(turn, "completed") - self.store.audit(owner, "instruction", host, "completed") + self.store.finish(turn, result.get("status","unknown") if task else "completed") + if task and not source_message_id: + self.lifecycle.transition(turn,"completed","finished") + self.store.audit(owner, "instruction", host, result.get("status","unknown") if task else "completed") return result except asyncio.TimeoutError: self.store.finish(turn, "unknown") + if task: + self.lifecycle.transition(turn,"inflight","unknown") return {"status": "unknown", "turn_id": turn, "result": "等待结果超时,执行状态未知;不要直接重发有副作用的任务。"} finally: self.pending.pop(turn, None) - await self.r.delete(key) + if task: + # Cancellation, publish failures and connector shutdown never replay work. + with contextlib.suppress(Exception): + self.lifecycle.transition(turn,"inflight","unknown") + with contextlib.suppress(Exception): + await self.r.delete(key) def create_app(settings=None, redis=None, platform=None): @@ -149,9 +192,15 @@ def create_app(settings=None, redis=None, platform=None): credential_store = CredentialStore(store, CredentialKeyring.from_file(s.credential_keyring_file)) credential_broker = CredentialBroker(credential_store, r, f, environment=s.environment, app_id=s.feishu_app_id) oauth = CredentialAPI(s, r, store, credential_broker) + from .binding_state import BindingState + from .device_lifecycle import DeviceLifecycle + store.lifecycle = BindingState(store,s.environment,s.feishu_app_id) + broker.lifecycle = store.lifecycle + lifecycle = DeviceLifecycle(store.lifecycle,r,broker) else: oauth = OAuth(s, r, f) oauth.store = store + lifecycle = None from .notifications import router as notification_router background = set() @@ -165,6 +214,8 @@ def create_app(settings=None, redis=None, platform=None): @asynccontextmanager async def lifespan(app): await r.ping() + if lifecycle: + await lifecycle.maintain() for row in store.db.execute("SELECT host_id,fingerprint,revoked FROM devices").fetchall(): await r.set("hn:device:" + row["host_id"], row["fingerprint"]) if row["revoked"]: @@ -180,12 +231,25 @@ def create_app(settings=None, redis=None, platform=None): async with oauth.lock: oauth.prune() sweep_task = asyncio.create_task(sweep()) + async def maintenance(): + while True: + await asyncio.sleep(5) + try: + await lifecycle.maintain() + except Exception: + # No credentials or raw exception text in background logs. + pass + lifecycle_task = asyncio.create_task(maintenance()) if lifecycle else None try: yield finally: - for task in [reply_task, sweep_task, *background]: + work = [reply_task,sweep_task,*background] + ([lifecycle_task] if lifecycle_task else []) + for task in work: task.cancel() - await asyncio.gather(reply_task, sweep_task, *background, return_exceptions=True) + await asyncio.gather(*work, return_exceptions=True) + if lifecycle: + with contextlib.suppress(Exception): + await r.delete("hn:task-worker:"+broker.worker_id) await pub.aclose() await f.close() store.close() @@ -201,6 +265,28 @@ def create_app(settings=None, redis=None, platform=None): app.include_router(web_sessions.router) app.state.store, app.state.broker, app.state.oauth = store, broker, oauth app.state.platform, app.state.redis = f, r + app.state.lifecycle = lifecycle + + if lifecycle: + from .device_lifecycle import AdmissionRequest + from .credential_api import DeviceRequest + @app.post("/work/v2/device/unbind") + async def device_unbind(body: DeviceRequest, request: Request): + await oauth.authenticate(body,request) + try: + store.lifecycle.invalidate_host(body.host_id,"unbind",expected_owner=body.binding_owner, + expected_generation=body.binding_generation) + except ValueError: + raise HTTPException(409,"binding_changed_retry") from None + lifecycle.cancel_waiters() + propagated = await lifecycle.flush() + return {"status":"unbound","binding_generation":store.device(body.host_id)["binding_generation"], + "propagation":"published" if propagated else "pending"} + + @app.post("/internal/devices/{host}/admission") + async def admission(host: str, body: AdmissionRequest, request: Request): + require_secret(request.headers.get("X-Router-Admission-Secret"),s.router_admission_secret) + return sign(await lifecycle.admission(host,body),broker.signer) @app.exception_handler(RequestValidationError) async def validation_error(request, exc): @@ -261,6 +347,14 @@ def create_app(settings=None, redis=None, platform=None): admin(request) if not store.device(host): raise HTTPException(404, "device not found") + if lifecycle: + try: + store.revoke(host) + except ValueError: + raise HTTPException(409,"binding_changed_retry") from None + lifecycle.cancel_waiters() + propagated = await lifecycle.flush() + return {"status":"revoked","propagation":"published" if propagated else "pending"} # Persistent source of truth first; router block + disconnect follows. store.revoke(host) await r.incr("hn:web-generation:" + host) @@ -277,6 +371,8 @@ def create_app(settings=None, redis=None, platform=None): async def reconcile(request: Request): """Rebuild router admission metadata after Redis recovery; no dialogue data.""" admin(request) + if lifecycle: + await lifecycle.maintain() rows = store.db.execute("SELECT host_id,fingerprint,revoked FROM devices").fetchall() for row in rows: await r.set("hn:device:" + row["host_id"], row["fingerprint"]) @@ -290,6 +386,15 @@ def create_app(settings=None, redis=None, platform=None): device = store.device(host) if not device: raise HTTPException(404, "device not found") + if lifecycle: + try: + store.lifecycle.invalidate_host(host,"unbind") + except ValueError: + raise HTTPException(409,"binding_changed_retry") from None + lifecycle.cancel_waiters() + propagated = await lifecycle.flush() + return {"status":"unbound","host_id":host,"revoked":bool(device["revoked"]), + "propagation":"published" if propagated else "pending"} if device["owner"]: store.unbind(device["owner"]) await r.incr("hn:web-generation:" + host) @@ -306,7 +411,13 @@ def create_app(settings=None, redis=None, platform=None): return "你已绑定电脑,请先发送 /解绑。" await rate_limit(r, "binding:" + owner, 5) code = secrets.token_urlsafe(12) - await r.set("hn:bind:" + hashlib.sha256(code.encode()).hexdigest(), owner, ex=300) + if lifecycle: + try: + store.lifecycle.issue_code(owner,code) + except ValueError: + return "你已绑定电脑,请先发送 /解绑。" + else: + await r.set("hn:bind:" + hashlib.sha256(code.encode()).hexdigest(), owner, ex=300) return "请在本人电脑的桥接客户端输入绑定码(5分钟有效):" + code @app.post("/device/bind") @@ -325,6 +436,16 @@ def create_app(settings=None, redis=None, platform=None): raise ValueError("invalid code") except Exception: raise HTTPException(403, "invalid device proof") from None + if lifecycle: + try: + owner = store.lifecycle.consume_code(body.host_id,code) + except LookupError: + raise HTTPException(404,"binding code unavailable") from None + except (*INTEGRITY_ERRORS,ValueError): + raise HTTPException(409,"owner/device already bound; request new code") from None + return sign({"host_id":body.host_id,"owner":owner, + "binding_generation":store.device(body.host_id)["binding_generation"], + "ts":int(time.time())},broker.signer) key = "hn:bind:" + hashlib.sha256(code.encode()).hexdigest() owner = await r.get(key) if not owner: @@ -365,11 +486,15 @@ def create_app(settings=None, redis=None, platform=None): owner = sender.get("sender_id", {}).get("open_id") message_id = message.get("message_id", "") if (message.get("chat_type") != "p2p" or message.get("message_type") != "text" - or sender.get("sender_type") != "user" or not owner + or sender.get("sender_type") != "user" or not isinstance(owner,str) + or not 0 < len(owner) <= 128 or any(ord(c) < 32 for c in owner) or not re.fullmatch(r"[A-Za-z0-9_-]{1,128}", message_id)): return if header.get("app_id") != s.feishu_app_id: return + if lifecycle and (not isinstance(message.get("chat_id"),str) or not 0 < len(message["chat_id"]) <= 128 + or any(ord(c) < 32 for c in message["chat_id"])): + return try: text = json.loads(message["content"])["text"].strip() if not text or len(text) > 8000: @@ -383,7 +508,10 @@ def create_app(settings=None, redis=None, platform=None): elif text == "/解绑": old = store.by_owner(owner) store.unbind(owner) - if old: + if lifecycle: + lifecycle.cancel_waiters() + await lifecycle.flush() + elif old: await r.incr("hn:web-generation:" + old["host_id"]) await r.publish("hn:kill:" + old["host_id"], "unbind") for item in broker.pending.values(): @@ -391,11 +519,25 @@ def create_app(settings=None, redis=None, platform=None): item.future.set_result({"status": "unbound", "result": "设备已解绑,结果已丢弃。"}) result = "已解绑;重新绑定后请更新本机绑定信息。" elif text == "/主机名" or text.startswith("/回显 "): - result = (await broker.dispatch(owner, text, "msg_" + hashlib.sha256(message_id.encode()).hexdigest()))["result"] + result = (await broker.dispatch(owner, text, "msg_" + hashlib.sha256(message_id.encode()).hexdigest(), + **({"chat_id":message["chat_id"],"source_message_id":message_id} if lifecycle else {})))["result"] else: instruction = text if text.startswith("/助手 ") else "/助手 " + text - result = (await broker.dispatch(owner, instruction, "msg_" + hashlib.sha256(message_id.encode()).hexdigest()))["result"] - await f.reply(message_id, result[:s.max_result]) + result = (await broker.dispatch(owner, instruction, "msg_" + hashlib.sha256(message_id.encode()).hexdigest(), + **({"chat_id":message["chat_id"],"source_message_id":message_id} if lifecycle else {})))["result"] + turn = "msg_"+hashlib.sha256(message_id.encode()).hexdigest() + task = store.lifecycle.task(turn) if lifecycle else None + if task and not store.lifecycle.claim_reply(turn,owner,message_id): + store.finish(event_turn,"result_discarded") + return + try: + await f.reply(message_id, result[:s.max_result]) + except Exception: + if task: + store.lifecycle.transition(turn,"replying","reply_unknown") + raise + if task: + store.lifecycle.transition(turn,"replying","replied") store.finish(event_turn, "replied") except Exception: store.audit(owner, "event", message_id, "failed_or_unknown") diff --git a/deployable/relay/device_lifecycle.py b/deployable/relay/device_lifecycle.py new file mode 100644 index 0000000..fdf627d --- /dev/null +++ b/deployable/relay/device_lifecycle.py @@ -0,0 +1,104 @@ +"""Broker-mode invalidation outbox, worker liveness and internal admission.""" +import json +import time + +from fastapi import HTTPException +from pydantic import BaseModel, ConfigDict, Field + +from .router import compare_delete + + +class AdmissionRequest(BaseModel): + model_config = ConfigDict(extra="forbid") + nonce: str = Field(pattern=r"^[a-f0-9]{32}$") + turn_id: str | None = Field(default=None, pattern=r"^[A-Za-z0-9_-]{1,128}$") + session_id: str | None = Field(default=None, min_length=1, max_length=64) + + +class DeviceLifecycle: + def __init__(self, state, redis, broker): + self.state, self.r, self.broker = state, redis, broker + + async def flush(self): + rows = self.state.db.execute("""SELECT * FROM work_binding_events_v2 + WHERE environment=? AND app_id=? AND delivered=0 ORDER BY created_at LIMIT 100""", + (self.state.environment,self.state.app_id)).fetchall() + for event in rows: + try: + device = self.state.store.device(event["host_id"]) + if device and device["revoked"]: + await self.r.set("hn:revoked:"+event["host_id"], "1") + key = "hn:online:"+event["host_id"] + presence = await self.r.get(key) + if presence: + try: + generation = json.loads(presence)["proof"]["binding_generation"] + if type(generation) is not int: + generation = -1 + except (ValueError, KeyError, TypeError): + generation = -1 + if generation < event["binding_generation"]: + await compare_delete(self.r, key, presence) + # Newer sessions ignore a delayed invalidation for older generations. + await self.r.publish("hn:kill:"+event["host_id"], json.dumps({ + "reason": event["reason"], "binding_generation": event["binding_generation"]})) + with self.state.db: + self.state.db.execute("UPDATE work_binding_events_v2 SET delivered=1 WHERE event_id=?", + (event["event_id"],)) + except Exception: + # DB remains authoritative; leave durable work for another attempt. + return False + self.cancel_waiters() + return not self.state.db.execute("""SELECT event_id FROM work_binding_events_v2 + WHERE environment=? AND app_id=? AND delivered=0 LIMIT 1""", + (self.state.environment,self.state.app_id)).fetchone() + + def cancel_waiters(self): + for turn, pending in list(self.broker.pending.items()): + task = self.state.task(turn) + if task and task["status"] not in {"inflight", "completed", "replying"} and not pending.future.done(): + pending.future.set_result({"status":task["status"], "turn_id":turn, + "result":"任务已结束,旧结果不会继续投递。"}) + + async def maintain(self): + await self.r.set("hn:task-worker:"+self.broker.worker_id,"alive",ex=30) + await self.flush() + rows = self.state.db.execute("""SELECT * FROM work_tasks_v2 + WHERE environment=? AND app_id=? AND status IN ('inflight','completed','replying') + ORDER BY updated_at LIMIT 100""", (self.state.environment,self.state.app_id)).fetchall() + for task in rows: + if task["deadline"] <= time.time() or not await self.r.exists("hn:task-worker:"+task["worker_id"]): + self.state.transition(task["turn_id"],task["status"], + "reply_unknown" if task["status"] == "replying" else "unknown") + else: + # Rotate bounded scanning instead of starving later tasks behind + # the same first 100 healthy workers on every maintenance pass. + with self.state.db: + self.state.db.execute("UPDATE work_tasks_v2 SET updated_at=? WHERE turn_id=? AND status=?", + (time.time(),task["turn_id"],task["status"])) + self.cancel_waiters() + + async def admission(self, host, body): + device = self.state.store.device(host) + if not device or device["revoked"] or not device["owner"]: + raise HTTPException(403,"device_unbound_or_revoked") + task = None + if body.turn_id: + task = self.state.task(body.turn_id) + if (not self.state.task_current(task) or task["host_id"] != host + or task["status"] != "inflight" or task["session_id"] != body.session_id + or not await self.r.exists("hn:task-worker:"+task["worker_id"])): + raise HTTPException(403,"task_not_active") + # Re-read after Redis await; never rely on router's cached admission. + latest = self.state.store.device(host) + if (not latest or latest["revoked"] or latest["owner"] != device["owner"] + or latest["binding_generation"] != device["binding_generation"]): + raise HTTPException(403,"stale_binding") + if task: + current = self.state.task(body.turn_id) + if not self.state.task_current(current) or current["status"] != "inflight": + raise HTTPException(403,"task_not_active") + return {"protocol_version":2,"environment":self.state.environment,"app_id":self.state.app_id, + "host_id":host,"owner":latest["owner"],"binding_generation":latest["binding_generation"], + "fingerprint":latest["fingerprint"],"nonce":body.nonce,"turn_id":body.turn_id, + "session_id":body.session_id,"expires_at":time.time()+5} diff --git a/deployable/relay/notifications.py b/deployable/relay/notifications.py index 3b1821c..0609fc3 100644 --- a/deployable/relay/notifications.py +++ b/deployable/relay/notifications.py @@ -21,6 +21,7 @@ class Notification(BaseModel): proof: dict request_id: str = Field(pattern=r'^[A-Za-z0-9_-]{8,100}$') binding_owner: str = Field(min_length=1, max_length=128) + binding_generation: int | None = Field(default=None,strict=True,ge=0) kind: str = Field(pattern=r'^(text|notice|authorization)$') title: str = Field(default='', max_length=80) text: str = Field(default='', max_length=2000) @@ -41,10 +42,16 @@ def router(settings, redis, store, platform, oauth): if not settings.feishu_enabled: raise HTTPException(503, 'platform disabled') payload = body.model_dump(exclude={'host_id', 'proof'}) + if body.binding_generation is None: + payload.pop('binding_generation') # Preserve exact legacy proof contract. owner = await device_owner(settings, redis, store, body.host_id, body.proof, 'work-notify-v1', payload) if body.binding_owner != owner: raise HTTPException(403, 'local binding is stale; pair the client again') + if settings.credential_mode == 'broker': + current = store.device(body.host_id) + if body.binding_generation is None or current['binding_generation'] != body.binding_generation: + raise HTTPException(403,'stale_binding') digest = hashlib.sha256(canonical(payload)).hexdigest() previous = store.db.execute('SELECT * FROM notifications WHERE host_id=? AND request_id=?', (body.host_id, body.request_id)).fetchone() @@ -106,7 +113,8 @@ def router(settings, redis, store, platform, oauth): # Recheck the transaction and generation after waiting on app credentials. oauth.authorization_url_for_notification(owner, body.host_id, body.oauth_state) device = store.device(body.host_id) - if not device or device['revoked'] or device['owner'] != owner: + if (not device or device['revoked'] or device['owner'] != owner + or (settings.credential_mode == 'broker' and device['binding_generation'] != body.binding_generation)): status = 'binding_changed' else: delivery_id = uuid.uuid5(uuid.NAMESPACE_URL, body.host_id + ':' + body.request_id).hex diff --git a/deployable/relay/router.py b/deployable/relay/router.py index e8d91e1..e1756a6 100644 --- a/deployable/relay/router.py +++ b/deployable/relay/router.py @@ -3,6 +3,8 @@ import contextlib import json import secrets import time +import hashlib +import httpx from contextlib import asynccontextmanager from fastapi import FastAPI, WebSocket, WebSocketDisconnect @@ -10,7 +12,7 @@ from redis.asyncio import Redis from redis.exceptions import WatchError from .config import Settings -from .crypto import fingerprint, raw_public, verified, verify_device +from .crypto import canonical, fingerprint, load_public, raw_public, verified, verify_device async def compare_delete(redis, key, expected): @@ -41,16 +43,38 @@ async def compare_expire(redis, key, expected, ttl): return False -def create_app(settings=None, redis=None): +def create_app(settings=None, redis=None, admission_transport=None): s = settings or Settings(role="router") r = redis or Redis.from_url(s.redis_url, decode_responses=True) ca = s.ca_cert.read_bytes() active = set() + internal = (httpx.AsyncClient(base_url=s.connector_internal_url.rstrip('/'),timeout=3, + trust_env=False,follow_redirects=False,transport=admission_transport) if s.credential_mode == "broker" else None) + connector_public = load_public(s.connector_public_key) if internal else None + + async def admission(host, owner, generation, *, turn=None, session=None, expected_fp=None): + nonce = secrets.token_hex(16) + response = await internal.post("/internal/devices/"+host+"/admission", + headers={"X-Router-Admission-Secret":s.router_admission_secret}, + json={"nonce":nonce,"turn_id":turn,"session_id":session}) + if response.status_code != 200: + raise ValueError("device admission rejected") + data = verified(response.json(),connector_public) + if (data["nonce"] != nonce or data["host_id"] != host or data["owner"] != owner + or data["binding_generation"] != generation or data["environment"] != s.environment + or data["app_id"] != s.feishu_app_id or data["protocol_version"] != 2 + or data["turn_id"] != turn or data["session_id"] != session + or not time.time() <= data["expires_at"] <= time.time()+10 + or data["fingerprint"] != expected_fp): + raise ValueError("stale device admission") + return data @asynccontextmanager async def lifespan(app): await r.ping() yield + if internal: + await internal.aclose() if redis is None: await r.aclose() @@ -101,6 +125,14 @@ def create_app(settings=None, redis=None): fp = fingerprint(hello["cert"]) if await r.get("hn:device:" + host) != fp or await r.exists("hn:revoked:" + host): raise ValueError("unregistered/revoked") + generation = owner = None + if internal: + generation, owner = proof["binding_generation"], proof["binding_owner"] + if (type(generation) is not int or generation < 0 or type(proof.get("protocol_version")) is not int + or proof.get("protocol_version") != 2 + or proof.get("environment") != s.environment or proof.get("app_id") != s.feishu_app_id): + raise ValueError("invalid binding handshake") + await admission(host,owner,generation,expected_fp=fp) session = proof["session_id"] presence = json.dumps({"cert": hello["cert"], "proof": hello["proof"], "challenge": challenge}) if not await r.set("hn:online:" + host, presence, nx=True, ex=30): @@ -108,6 +140,8 @@ def create_app(settings=None, redis=None): active.add(session) pub = r.pubsub() await pub.subscribe("hn:down:" + host, "hn:kill:" + host) + if internal: + await admission(host,owner,generation,expected_fp=fp) await ws.send_json({"type": "connected", "session_id": session}) async def downstream(): @@ -115,13 +149,28 @@ def create_app(settings=None, redis=None): if msg["type"] != "message": continue if msg["channel"] == "hn:kill:" + host: + if internal: + event = json.loads(msg["data"]) + if generation >= event["binding_generation"]: + continue await ws.close(1008) return + if internal: + packet = json.loads(msg["data"]) + routing = verified(packet["routing"],connector_public) + if (routing["host_id"] != host or routing["session_id"] != session + or routing["environment"] != s.environment or routing["app_id"] != s.feishu_app_id + or routing["binding_generation"] != generation + or routing["envelope_hash"] != hashlib.sha256(canonical(packet["envelope"])).hexdigest()): + raise ValueError("invalid instruction routing") + await admission(host,owner,generation,turn=routing["turn_id"],session=session,expected_fp=fp) await ws.send_text(msg["data"]) async def heartbeat(): while True: await asyncio.sleep(8) + if internal: + await admission(host,owner,generation,expected_fp=fp) if await r.exists("hn:revoked:" + host) or not await compare_expire(r, "hn:online:" + host, presence, 30): await ws.close(1008) return @@ -142,6 +191,8 @@ def create_app(settings=None, redis=None): verified(envelope, public) if envelope["host_id"] != host or envelope["session_id"] != session: raise ValueError("wrong reply destination") + if internal: + await admission(host,owner,generation,turn=turn,session=session,expected_fp=fp) await r.publish("hn:replies", json.dumps(data)) tasks = [asyncio.create_task(x()) for x in (downstream, heartbeat, upstream)] @@ -155,13 +206,15 @@ def create_app(settings=None, redis=None): with contextlib.suppress(Exception): await ws.close(1011) finally: + active.discard(session) for task in tasks: task.cancel() await asyncio.gather(*tasks, return_exceptions=True) if pub: - await pub.aclose() + with contextlib.suppress(Exception): + await pub.aclose() if presence: - await compare_delete(r, "hn:online:" + host, presence) - active.discard(session) + with contextlib.suppress(Exception): + await compare_delete(r, "hn:online:" + host, presence) return app diff --git a/deployable/relay/store.py b/deployable/relay/store.py index ce55fce..9440977 100644 --- a/deployable/relay/store.py +++ b/deployable/relay/store.py @@ -32,6 +32,7 @@ class Postgres: class Store: def __init__(self, path, database_url=""): + self.lifecycle = None path.parent.mkdir(parents=True, exist_ok=True) schema = """ CREATE TABLE IF NOT EXISTS devices( @@ -71,6 +72,8 @@ class Store: self.db.execute("INSERT INTO devices(host_id,cert,fingerprint) VALUES(?,?,?)", (host, cert, fp)) def bind(self, host, owner): + if self.lifecycle: + return self.lifecycle.bind(host, owner) with self.db: # Conditional update also protects against two connectors racing for the same device. count = self.db.execute("UPDATE devices SET owner=?,binding_generation=binding_generation+1 WHERE host_id=? AND owner IS NULL AND revoked=0", (owner, host)).rowcount @@ -78,10 +81,14 @@ class Store: raise ValueError("device unavailable or already bound") def unbind(self, owner): + if self.lifecycle: + return self.lifecycle.unbind(owner) with self.db: self.db.execute("UPDATE devices SET owner=NULL,binding_generation=binding_generation+1 WHERE owner=?", (owner,)) def revoke(self, host): + if self.lifecycle: + return self.lifecycle.invalidate_host(host, "revoke") with self.db: self.db.execute("UPDATE devices SET revoked=1,binding_generation=binding_generation+1 WHERE host_id=? AND revoked=0", (host,)) diff --git a/deployable/tests/test_credential_api.py b/deployable/tests/test_credential_api.py index 49868ac..71aabf3 100644 --- a/deployable/tests/test_credential_api.py +++ b/deployable/tests/test_credential_api.py @@ -35,6 +35,7 @@ class Platform: self.resume = None self.scopes = "read" self.sent = [] + self.replies = [] self.user_info_failure = False async def credential_exchange(self, code): @@ -61,6 +62,9 @@ class Platform: async def tenant_token(self): return "fake-tenant" + async def reply(self, message_id, text): + self.replies.append((message_id,text)) + async def send(self, owner, kind, content, delivery_id, *, token): self.sent.append((owner, content)) return {"message_id": "om_fake"} @@ -80,6 +84,7 @@ async def env(tmp_path): keyring.write_text(json.dumps({"active_key_id": "test-key", "keys": { "test-key": base64.b64encode(os.urandom(32)).decode()}})) settings = Settings(environment="test", admin_secret="a"*40, internal_secret="b"*40, + router_admission_secret="c"*40, ca_cert=ca_path, connector_key=connector_key, data_dir=tmp_path, feishu_enabled=True, feishu_app_id="cli_test", feishu_app_secret="fake-app-secret", credential_mode="broker", credential_keyring_file=keyring, credential_permission_profiles={"product-v1": ["read"], "product-v2": ["read", "write"]}, @@ -317,7 +322,8 @@ async def test_authorization_notification_uses_server_owned_transaction(env): item = await start(env) host = "a"*32 fields = dict(request_id="notify-test-001", kind="authorization", text="", title="", - oauth_state=item["transaction_id"], binding_owner="ou_a") + oauth_state=item["transaction_id"], binding_owner="ou_a", + binding_generation=env["store"].device(host)["binding_generation"]) proof = sign(dict(host_id=host, purpose="work-notify-v1", ts=int(time.time()), nonce=secrets.token_hex(16), payload_hash=hashlib.sha256(canonical(fields)).hexdigest()), env["keys"][host]) response = await env["client"].post("/device/notifications", json={**fields, "host_id": host, "proof": proof}) @@ -466,6 +472,14 @@ async def test_postgres_authorization_and_binding_generation(env, tmp_path): assert result.status_code == 200 and "fake-refresh" not in result.text app.state.store.unbind("ou_a") assert app.state.store.device("a"*32)["binding_generation"] == 2 + lifecycle = app.state.store.lifecycle + lifecycle.issue_code("ou_a","fake-pg-old-code") + app.state.store.unbind("ou_a") + with pytest.raises(LookupError): + lifecycle.consume_code("a"*32,"fake-pg-old-code") + lifecycle.issue_code("ou_a","fake-pg-new-code") + assert lifecycle.consume_code("a"*32,"fake-pg-new-code") == "ou_a" + assert app.state.store.db.execute("SELECT COUNT(*) AS n FROM work_binding_events_v2").fetchone()["n"] == 1 finally: if app: app.state.store.close() @@ -475,6 +489,7 @@ async def test_postgres_authorization_and_binding_generation(env, tmp_path): def test_settings_require_explicit_key_and_product_profiles(tmp_path): common = dict(environment="test", admin_secret="a"*40, internal_secret="b"*40, credential_mode="broker", + router_admission_secret="c"*40, feishu_enabled=True, feishu_app_id="test-app", feishu_app_secret="fake-secret") with pytest.raises(ValueError): Settings(**common) diff --git a/deployable/tests/test_device_lifecycle.py b/deployable/tests/test_device_lifecycle.py new file mode 100644 index 0000000..cf636ca --- /dev/null +++ b/deployable/tests/test_device_lifecycle.py @@ -0,0 +1,479 @@ +"""Broker-mode invalidation and real loopback WS task routing (no real Work).""" +import asyncio +import hashlib +import json +import secrets +import socket +import time + +import httpx +import pytest +import pytest_asyncio +import uvicorn +from cryptography.hazmat.primitives import serialization +from cryptography.hazmat.primitives.asymmetric import x25519 +from websockets.asyncio.client import connect +from websockets.exceptions import ConnectionClosed + +from test_credential_api import env, wire, start, authorize +from relay.crypto import canonical, load_private, load_public, open_envelope, raw_public, seal, sign +from relay.device_lifecycle import AdmissionRequest +from relay.router import create_app as router_app + + +@pytest_asyncio.fixture +async def routed(env, tmp_path): + public_path = tmp_path / "connector.pub" + public_path.write_bytes(load_private(env["settings"].connector_key).public_key().public_bytes( + serialization.Encoding.PEM,serialization.PublicFormat.SubjectPublicKeyInfo)) + settings = env["settings"].model_copy(update={"role":"router","connector_public_key":public_path, + "connector_internal_url":"http://connector:8000"}) + app = router_app(settings,env["redis"],admission_transport=httpx.ASGITransport(env["app"])) + sock = socket.socket() + sock.bind(("127.0.0.1",0)) + sock.listen(128) + port = sock.getsockname()[1] + server = uvicorn.Server(uvicorn.Config(app,log_level="critical",access_log=False,lifespan="on")) + server_task = asyncio.create_task(server.serve(sockets=[sock])) + try: + for _ in range(200): + if server.started: + break + if server_task.done(): + server_task.result() + await asyncio.sleep(0.01) + assert server.started + value = {**env,"url":f"ws://127.0.0.1:{port}/device/ws","public":load_public(public_path),"sessions":[],"router":app} + yield value + finally: + for ws in locals().get("value",{}).get("sessions",[]): + await ws.close() + server.should_exit = True + await asyncio.wait_for(server_task,5) + + +async def device(env, host="a"*32, *, generation=None, legacy=False): + ws = await connect(env["url"],proxy=None) + env["sessions"].append(ws) + challenge = json.loads(await ws.recv())["challenge"] + row = env["store"].device(host) + ephemeral = x25519.X25519PrivateKey.generate() + session = secrets.token_hex(16) + plain = {"host_id":host,"challenge":challenge,"session_id":session, + "kx":raw_public(ephemeral.public_key()),"ts":int(time.time())} + if not legacy: + plain.update(protocol_version=2,environment=env["settings"].environment, + app_id=env["settings"].feishu_app_id,binding_owner=row["owner"] or "ou_a", + binding_generation=row["binding_generation"] if generation is None else generation) + await ws.send(json.dumps({"host_id":host,"cert":row["cert"],"proof":sign(plain,env["keys"][host])})) + accepted = json.loads(await asyncio.wait_for(ws.recv(),3)) + assert accepted["type"] == "connected" + return ws,session,ephemeral + + +async def instruction(env, connection, host="a"*32): + ws,session,ephemeral = connection + packet = json.loads(await asyncio.wait_for(ws.recv(),3)) + payload = open_envelope(packet["envelope"],ephemeral,env["public"],host_id=host,session_id=session) + return payload + + +async def reply(env, connection, payload, text, host="a"*32): + ws,session,_ = connection + envelope = seal({"type":"result","turn_id":payload["turn_id"],"result":text}, + payload["reply_key"],env["keys"][host],host_id=host,session_id=session) + await ws.send(json.dumps({"type":"reply","turn_id":payload["turn_id"],"envelope":envelope})) + + +async def wait_for_reply(platform, count): + for _ in range(200): + if len(platform.replies) >= count: + return + await asyncio.sleep(0.01) + assert len(platform.replies) >= count + + +def event(env, message_id, text, chat="chat-a", owner="ou_a"): + return {"header":{"event_type":"im.message.receive_v1","app_id":env["settings"].feishu_app_id}, + "event":{"sender":{"sender_type":"user","sender_id":{"open_id":owner}}, + "message":{"message_id":message_id,"chat_type":"p2p","chat_id":chat,"message_type":"text", + "content":json.dumps({"text":text})}}} + + +async def send_event(env, data): + response = await env["client"].post("/internal/events",json=data, + headers={"X-Internal-Secret":env["settings"].internal_secret}) + assert response.status_code == 200 + + +async def test_unbind_commits_and_defers_redis_side_effects(env,monkeypatch): + await authorize(env) + pending = await start(env) + with monkeypatch.context() as patch: + async def fail(*args,**kwargs): + raise ConnectionError("synthetic redis outage") + patch.setattr(env["redis"],"publish",fail) + response = await env["client"].post("/admin/devices/"+"a"*32+"/unbind", + headers={"X-Admin-Secret":env["settings"].admin_secret}) + assert response.status_code == 200 and response.json()["propagation"] == "pending" + assert env["store"].device("a"*32)["owner"] is None + assert env["api"]._row(pending["transaction_id"])["status"] == "invalidated" + path = "/work/v2/credentials/access-token" + assert (await env["client"].post(path,json=wire(env,path,permission_profile="product-v1"))).status_code == 403 + assert await env["app"].state.lifecycle.flush() + assert env["store"].db.execute("SELECT COUNT(*) AS n FROM work_binding_events_v2 WHERE delivered=0").fetchone()["n"] == 0 + + +async def test_codes_are_one_time_and_pre_unbind_codes_stay_invalid_after_rebind(env): + state = env["store"].lifecycle + env["store"].unbind("ou_a") + state.issue_code("ou_a","fake-code-old") + env["store"].unbind("ou_a") + with pytest.raises(LookupError): + state.consume_code("a"*32,"fake-code-old") + state.issue_code("ou_a","fake-code-new") + assert state.consume_code("a"*32,"fake-code-new") == "ou_a" + with pytest.raises(LookupError): + state.consume_code("a"*32,"fake-code-new") + dump = "\n".join(env["store"].db.iterdump()) + assert "fake-code-old" not in dump and "fake-code-new" not in dump + + +async def test_failed_database_invalidation_rolls_back_binding_and_outbox(env,monkeypatch): + state = env["store"].lifecycle + original = state._cancel + def fail(device,reason): + original(device,reason) + raise RuntimeError("synthetic transaction abort") + monkeypatch.setattr(state,"_cancel",fail) + with pytest.raises(RuntimeError): + env["store"].unbind("ou_a") + current = env["store"].device("a"*32) + assert current["owner"] == "ou_a" and current["binding_generation"] == 1 + assert state.db.execute("SELECT COUNT(*) AS n FROM work_binding_events_v2").fetchone()["n"] == 0 + + +async def test_unbind_outbox_is_persistent_and_reprocessed_by_second_connector(env): + from relay.connector import create_app + env["store"].unbind("ou_a") + second = create_app(env["settings"],env["redis"],env["platform"]) + try: + assert second.state.store.device("a"*32)["owner"] is None + assert await second.state.lifecycle.flush() + assert second.state.store.db.execute("SELECT COUNT(*) AS n FROM work_binding_events_v2 WHERE delivered=0").fetchone()["n"] == 0 + finally: + second.state.store.close() + + +async def test_new_online_generation_not_removed_by_old_outbox(env): + env["store"].unbind("ou_a") + env["store"].bind("a"*32,"ou_a") + generation = env["store"].device("a"*32)["binding_generation"] + presence = json.dumps({"proof":{"binding_generation":generation}}) + await env["redis"].set("hn:online:"+"a"*32,presence) + assert await env["app"].state.lifecycle.flush() + assert await env["redis"].get("hn:online:"+"a"*32) == presence + + +def notification(env, *, generation=None, include_generation=True): + host = "a"*32 + payload = {"request_id":secrets.token_hex(8),"kind":"text","text":"synthetic notice","title":"", + "oauth_state":"","binding_owner":"ou_a"} + if include_generation: + payload["binding_generation"] = env["store"].device(host)["binding_generation"] if generation is None else generation + proof = sign({"host_id":host,"purpose":"work-notify-v1","ts":int(time.time()),"nonce":secrets.token_hex(16), + "payload_hash":hashlib.sha256(canonical(payload)).hexdigest()},env["keys"][host]) + return {**payload,"host_id":host,"proof":proof} + + +async def test_notifications_require_current_generation_in_broker_mode(env): + stale = notification(env) + env["store"].unbind("ou_a") + env["store"].bind("a"*32,"ou_a") + assert (await env["client"].post("/device/notifications",json=stale)).status_code == 403 + assert (await env["client"].post("/device/notifications",json=notification(env,include_generation=False))).status_code == 403 + result = await env["client"].post("/device/notifications",json=notification(env)) + assert result.json()["status"] == "sent" and len(env["platform"].sent) == 1 + + +async def test_notification_rebind_during_app_token_wait_does_not_send(env,monkeypatch): + async def rebind(): + env["store"].unbind("ou_a") + env["store"].bind("a"*32,"ou_a") + return "fake-app-token" + monkeypatch.setattr(env["platform"],"tenant_token",rebind) + result = await env["client"].post("/device/notifications",json=notification(env)) + assert result.json()["status"] == "binding_changed" and not env["platform"].sent + + +async def test_internal_admission_requires_secret_and_current_binding(env): + path = "/internal/devices/"+"a"*32+"/admission" + body = {"nonce":secrets.token_hex(16)} + assert (await env["client"].post(path,json=body)).status_code == 401 + assert (await env["client"].post(path,json=body, + headers={"X-Internal-Secret":env["settings"].internal_secret})).status_code == 401 + headers = {"X-Router-Admission-Secret":env["settings"].router_admission_secret} + assert (await env["client"].post(path,json=body,headers=headers)).status_code == 200 + env["store"].unbind("ou_a") + assert (await env["client"].post(path,json=body,headers=headers)).status_code == 403 + + +async def test_router_credential_cannot_inject_platform_events(env): + result = await env["client"].post("/internal/events",json=event(env,"om-injected","hello"), + headers={"X-Internal-Secret":env["settings"].router_admission_secret}) + assert result.status_code == 401 + + +async def test_device_can_only_unbind_its_current_authenticated_binding(env): + path = "/work/v2/device/unbind" + invalid = wire(env,path) + invalid["proof"] = {} + assert (await env["client"].post(path,json=invalid)).status_code == 403 + result = await env["client"].post(path,json=wire(env,path)) + assert result.status_code == 200 and result.json()["status"] == "unbound" + assert env["store"].device("a"*32)["owner"] is None + assert env["store"].device("b"*32)["owner"] == "ou_b" + + +async def test_authenticated_unbind_snapshot_cannot_clear_a_new_binding(env): + state = env["store"].lifecycle + old = env["store"].device("a"*32) + env["store"].unbind("ou_a") + env["store"].bind("a"*32,"ou_a") + with pytest.raises(ValueError): + state.invalidate_host("a"*32,"unbind",expected_owner="ou_a",expected_generation=old["binding_generation"]) + assert env["store"].device("a"*32)["owner"] == "ou_a" + + +async def test_v2_router_rejects_old_handshake_and_unbound_reconnect(routed): + with pytest.raises(ConnectionClosed): + await device(routed,legacy=True) + routed["store"].unbind("ou_a") + with pytest.raises(ConnectionClosed): + await device(routed) + + +async def test_two_devices_reverse_results_match_own_tasks(routed): + a = await device(routed) + b = await device(routed,"b"*32) + broker = routed["app"].state.broker + ta = asyncio.create_task(broker.dispatch("ou_a","a","turn-a",chat_id="chat-a",source_message_id="om-a")) + tb = asyncio.create_task(broker.dispatch("ou_b","b","turn-b",chat_id="chat-b",source_message_id="om-b")) + pa,pb = await asyncio.gather(instruction(routed,a),instruction(routed,b,"b"*32)) + assert pa["task_context"]["source_message_id"] == "om-a" + assert pb["task_context"]["owner"] == "ou_b" + assert pa["task_context"]["conversation_id"] != pb["task_context"]["conversation_id"] + await reply(routed,b,pb,"answer-b","b"*32) + await reply(routed,a,pa,"answer-a") + ra,rb = await asyncio.gather(ta,tb) + assert (ra["result"],rb["result"]) == ("answer-a","answer-b") + state = routed["store"].lifecycle + assert not state.claim_reply("turn-a","ou_a","om-b") + assert state.claim_reply("turn-a","ou_a","om-a") + assert not state.claim_reply("turn-a","ou_a","om-a") + + +async def test_same_owner_two_conversations_reply_to_original_message(routed): + connection = await device(routed) + await send_event(routed,event(routed,"om-first","first","chat-first")) + first = await instruction(routed,connection) + await send_event(routed,event(routed,"om-second","second","chat-second")) + second = await instruction(routed,connection) + assert first["task_context"]["conversation_id"] != second["task_context"]["conversation_id"] + await reply(routed,connection,second,"second-answer") + await reply(routed,connection,first,"first-answer") + await wait_for_reply(routed["platform"],2) + assert routed["platform"].replies == [("om-second","second-answer"),("om-first","first-answer")] + await send_event(routed,event(routed,"om-first","first","chat-first")) + await asyncio.sleep(0.05) + assert len(routed["platform"].replies) == 2 + + +async def test_robot_unbind_discards_previous_result_and_closes_connection(routed): + connection = await device(routed) + await send_event(routed,event(routed,"om-old-task","slow")) + payload = await instruction(routed,connection) + await send_event(routed,event(routed,"om-unbind","/解绑")) + await wait_for_reply(routed["platform"],1) + assert routed["platform"].replies[0][0] == "om-unbind" + assert routed["store"].device("a"*32)["owner"] is None + assert routed["store"].lifecycle.task(payload["turn_id"])["status"] == "unbound" + with pytest.raises(ConnectionClosed): + await asyncio.wait_for(connection[0].recv(),2) + await asyncio.sleep(0.05) + assert all(message != "om-old-task" for message,_ in routed["platform"].replies) + + +async def test_revoke_has_durable_state_and_blocks_connection(routed): + connection = await device(routed) + response = await routed["client"].post("/admin/devices/"+"a"*32+"/revoke", + headers={"X-Admin-Secret":routed["settings"].admin_secret}) + assert response.status_code == 200 + assert routed["store"].device("a"*32)["revoked"] == 1 + with pytest.raises(ConnectionClosed): + await asyncio.wait_for(connection[0].recv(),2) + + +async def test_delayed_kill_for_old_generation_does_not_disconnect_new_session(routed): + old = await device(routed) + routed["store"].unbind("ou_a") + await routed["app"].state.lifecycle.flush() + with pytest.raises(ConnectionClosed): + await asyncio.wait_for(old[0].recv(),2) + routed["store"].bind("a"*32,"ou_a") + fresh = await device(routed) + with routed["store"].db: + routed["store"].db.execute("UPDATE work_binding_events_v2 SET delivered=0") + await routed["app"].state.lifecycle.flush() + task = asyncio.create_task(routed["app"].state.broker.dispatch("ou_a","hello","fresh-task")) + packet = await instruction(routed,fresh) + await reply(routed,fresh,packet,"fresh-answer") + assert (await task)["result"] == "fresh-answer" + + +async def test_lost_disconnect_notification_still_rejects_old_reply(routed): + connection = await device(routed) + task = asyncio.create_task(routed["app"].state.broker.dispatch("ou_a","slow","lost-disconnect")) + packet = await instruction(routed,connection) + routed["store"].unbind("ou_a") # Deliberately do not flush PubSub notification. + await reply(routed,connection,packet,"must-not-arrive") + with pytest.raises(ConnectionClosed): + await asyncio.wait_for(connection[0].recv(),2) + routed["app"].state.lifecycle.cancel_waiters() + assert (await task)["status"] == "unbound" + + +async def test_idle_connection_rechecks_database_even_when_pubsub_flush_fails(routed,monkeypatch): + connection = await device(routed) + async def unavailable(): + return False + monkeypatch.setattr(routed["app"].state.lifecycle,"flush",unavailable) + routed["store"].unbind("ou_a") + # No client cooperation or kill publication: router's 8-second control check + # must close the old session from authoritative database state. + with pytest.raises(ConnectionClosed): + await asyncio.wait_for(connection[0].recv(),12) + + +async def test_control_plane_outage_does_not_forward_new_task(routed,monkeypatch): + from fastapi import HTTPException + connection = await device(routed) + async def unavailable(*args,**kwargs): + raise HTTPException(503,"synthetic_control_outage") + monkeypatch.setattr(routed["app"].state.lifecycle,"admission",unavailable) + task = asyncio.create_task(routed["app"].state.broker.dispatch("ou_a","do not deliver","control-outage")) + try: + with pytest.raises(ConnectionClosed): + await asyncio.wait_for(connection[0].recv(),3) + finally: + task.cancel() + with pytest.raises(asyncio.CancelledError): + await task + assert routed["store"].lifecycle.task("control-outage")["status"] == "unknown" + + +async def test_dispatch_cleanup_releases_waiter_despite_database_failure(routed,monkeypatch): + connection = await device(routed) + broker = routed["app"].state.broker + task = asyncio.create_task(broker.dispatch("ou_a","pending","cleanup-task")) + await instruction(routed,connection) + def fail(*args,**kwargs): + raise OSError("synthetic cleanup database failure") + monkeypatch.setattr(routed["store"].lifecycle,"transition",fail) + task.cancel() + with pytest.raises(asyncio.CancelledError): + await task + assert "cleanup-task" not in broker.pending + assert not await routed["redis"].exists("hn:inflight:"+"a"*32+":cleanup-task") + + +async def test_router_releases_connection_slot_despite_redis_cleanup_failure(routed,monkeypatch): + import relay.router as module + connection = await device(routed) + async def fail(*args,**kwargs): + raise ConnectionError("synthetic cleanup Redis failure") + monkeypatch.setattr(module,"compare_delete",fail) + await connection[0].close() + async with httpx.AsyncClient(transport=httpx.ASGITransport(routed["router"]),base_url="http://router") as client: + for _ in range(100): + result = await client.get("/ready") + if result.json()["connections"] == 0: + break + await asyncio.sleep(0.01) + assert result.json()["connections"] == 0 + + +async def test_platform_reply_unknown_not_replayed_on_duplicate_event(routed,monkeypatch): + attempts = [] + async def ambiguous(message,text): + attempts.append(message) + raise httpx.ReadTimeout("synthetic platform timeout") + monkeypatch.setattr(routed["platform"],"reply",ambiguous) + connection = await device(routed) + data = event(routed,"om-reply-unknown","hello") + await send_event(routed,data) + packet = await instruction(routed,connection) + await reply(routed,connection,packet,"answer") + for _ in range(100): + if routed["store"].lifecycle.task(packet["turn_id"])["status"] == "reply_unknown": + break + await asyncio.sleep(0.01) + assert routed["store"].lifecycle.task(packet["turn_id"])["status"] == "reply_unknown" + await send_event(routed,data) + await asyncio.sleep(0.05) + assert attempts == ["om-reply-unknown"] + + +async def test_binding_api_validates_signature_before_consuming_code(env): + state = env["store"].lifecycle + env["store"].unbind("ou_a") + state.issue_code("ou_a","fake-binding-once") + host = "a"*32 + proof = sign({"host_id":host,"code":"fake-binding-once","ts":int(time.time())},env["keys"][host]) + invalid = {**proof,"signature":"bad"} + assert (await env["client"].post("/device/bind",json={"host_id":host,"proof":invalid})).status_code == 403 + response = await env["client"].post("/device/bind",json={"host_id":host,"proof":proof}) + assert response.status_code == 200 and response.json()["binding_generation"] == 3 + assert (await env["client"].post("/device/bind",json={"host_id":host,"proof":proof})).status_code == 404 + + +async def test_dead_worker_tasks_unknown_and_never_replayed(env): + state = env["store"].lifecycle + row = env["store"].device("a"*32) + state.create_task("abandoned-task","ou_a","a"*32,row["binding_generation"],"session","dead-worker",time.time()+300,"chat","om-abandoned") + await env["app"].state.lifecycle.maintain() + assert state.task("abandoned-task")["status"] == "unknown" + assert not state.complete("abandoned-task","a"*32,"session","dead-worker") + assert not state.claim_reply("abandoned-task","ou_a","om-abandoned") + with pytest.raises(ValueError): + state.create_task("abandoned-task","ou_a","a"*32,row["binding_generation"],"new","new-worker",time.time()+300) + + +async def test_task_unknown_after_deadline_and_replying_after_crash(env): + state = env["store"].lifecycle + gen = env["store"].device("a"*32)["binding_generation"] + state.create_task("expired-task","ou_a","a"*32,gen,"s","missing",time.time()-1) + state.create_task("reply-crash","ou_a","a"*32,gen,"s","missing",time.time()+300) + state.transition("reply-crash","inflight","replying") + await env["app"].state.lifecycle.maintain() + assert state.task("expired-task")["status"] == "unknown" + assert state.task("reply-crash")["status"] == "reply_unknown" + + +async def test_wrong_session_and_old_generation_cannot_complete_task(env): + state = env["store"].lifecycle + gen = env["store"].device("a"*32)["binding_generation"] + state.create_task("old-generation","ou_a","a"*32,gen,"session","worker",time.time()+300) + assert not state.complete("old-generation","a"*32,"other-session","worker") + env["store"].unbind("ou_a") + env["store"].bind("a"*32,"ou_a") + assert not state.complete("old-generation","a"*32,"session","worker") + + +async def test_admission_denies_dead_tasks_without_exposing_credentials(env): + gen = env["store"].device("a"*32)["binding_generation"] + env["store"].lifecycle.create_task("task-check","ou_a","a"*32,gen,"s","dead",time.time()+300) + response = await env["client"].post("/internal/devices/"+"a"*32+"/admission", + headers={"X-Router-Admission-Secret":env["settings"].router_admission_secret}, + json={"nonce":secrets.token_hex(16),"turn_id":"task-check","session_id":"s"}) + assert response.status_code == 403 and "token" not in response.text.lower()