feat(lifecycle): add stage 4 authoritative unbind and durable task routing

Unify broker-mode binding changes in database transactions: advance device
generation, invalidate pending authorizations/tasks and prior pairing-code
epochs, and persist audit plus invalidation outbox. Add authenticated device
unbind alongside robot/admin paths. Pairing codes are hashed, generation-
fenced and consumed atomically with binding.

Replay Redis invalidation effects without undoing committed unbind. Guard
online cleanup and delayed kill events by generation. Introduce a separate
router admission credential with no platform-event injection authority.
Router verifies signed nonce-bound internal admission on handshake, task
traffic and periodic checks; reject stale sessions and control-plane outages.

Persist task owner/device/generation/session/worker and original message
mapping. Sign routing metadata, verify late results, claim replies once and
mark ambiguous/crashed tasks unknown instead of replaying writes. Fence
notifications across same-user rebind. Release in-memory slots/waiters even
when database or Redis cleanup fails.

Validation: 213 passed, 3 skipped (real PostgreSQL not configured), 30 existing
uvicorn/websockets deprecation warnings. New lifecycle coverage: 28 tests,
including real loopback WebSocket isolation, reverse completion, missed kill,
periodic unbind checks, outbox recovery, transaction rollback and cleanup
failures. Syntax, docs and staged diff checks pass.

Default legacy mode remains unchanged. Broker mode now requires coordinated
router admission configuration and a v2 client handshake. No production ECS
configuration, real credentials or client installer was changed. PostgreSQL,
real Redis multi-node, ARM64 and actual Work/client rollout remain stage 5.
This commit is contained in:
ex_tianzy
2026-11-04 11:43:53 +08:00
parent 93bfbca83a
commit d1ab834ecb
13 changed files with 1128 additions and 29 deletions
+69
View File
@@ -0,0 +1,69 @@
# 阶段4:解绑、连接准入与任务关联指导
日期:2026-11-04;适用模式:显式`credential_mode=broker`。本文描述服务端开发结果和后续部署/客户端适配要求,不表示现机已切换。
## 1. ECS权威解绑
机器人`/解绑`、管理员`/admin/devices/{host}/unbind`、新设备接口`POST /work/v2/device/unbind`和吊销路径,共用`BindingState`事务逻辑。设备接口沿用v2设备签名和基础字段,不允许自行指定其他设备/用户;校验的owner/generation作为解绑条件,不能在并发重绑后清除新的关系。
数据库事务完成以下操作:清空owner或标记revoked,推进binding_generation,作废原设备未完成授权事务,终止该绑定版本在途/待回消息任务,推进用户配对码epoch,记录审计与失效outbox。事务中任意一步失败整体回滚,不返回假“已解绑”。解绑不删除CA/设备私钥,也不撤销用户平台OAuth。
解绑命令即使当前已无设备,也推进owner epoch,使其此前未消费的配对码失效。broker模式配对码以摘要存数据库,绑定时在事务中核验epoch/有效期并一次性消费,和实际设备绑定一起提交;无效设备签名不消费配对码。旧legacy Redis配对码在broker路径不被接受。
原设备后续Token领取、授权回调/状态恢复、通知及内部连接准入都核验当前绑定/代次;正常业务入口不依赖客户端自行清理才算解绑。Broker没有独立后台刷新调度,已无有效绑定时没有认证调用可触发正常领取刷新;后续后台保活仍须遵守同一绑定策略。
## 2. 失效事件补偿
`work_binding_events_v2`记录待执行失效事件。提交数据库后尝试删除旧在线presence、写吊销标记、发布带generation的断连通知;失败保留记录,当前或其他connector可补偿。维护周期5秒,每轮处理最多100条,未处理完继续后续轮次。
解绑接口返回`status=unbound`、`propagation=published/pending`;published只说明Redis失效动作已被接受,不表示PC已确认或所有在途平台调用已撤回。数据库始终是凭据和任务判定依据。
旧事件不允许删除较新generation的在线presence;使用比较删除避免竞争。新router也忽略不比自身generation新的断连事件,防止重试的旧通知误断新连接。事件发布成功但数据库确认失败时可以重复发布,不能据此重复业务任务。
## 3. Router独立内部准入
新配置`HN_ROUTER_ADMISSION_SECRET`只用于router查询绑定/任务准入,至少32字符,不能复用管理员、事件内部Secret或App Secret。connector和router各自获得该凭据,router不需要获得机器人事件接入Secret或应用Secret。新配置及挂载的正式Compose接线在阶段5完成。
router还需要公开`HN_FEISHU_APP_ID`、同环境标识、`HN_CONNECTOR_INTERNAL_URL`(默认`http://connector:8000`)及已有connector签名公钥。内部URL为固定运维配置,禁止用户名密码/query/fragment和自动重定向;请求禁用环境代理,超时3秒。
内部接口:`POST /internal/devices/{host}/admission`,头为`X-Router-Admission-Secret`,正文是随机nonce及可选turn_id/session_id。它不返回用户Token,只返回connector签名的环境/应用、设备/owner、当前generation/证书指纹、nonce/任务/session和5秒期限。router验证签名、nonce、用途上下文与期限;不能拿事件Secret调用该接口,也不能拿准入Secret注入`/internal/events`。
准入在握手、订阅后、每次指令下发、结果上行以及8秒周期检查时重新核对ECS状态。任务校验还要求当前generation、原session、inflight状态、deadline及所属worker仍存活。控制服务不可用时关闭或拒绝连接/转发,不继续用陈旧缓存接受任务;这牺牲部分可用性换取绑定控制。
即时断连优先走outbox通知;通知丢失时周期检查可在下一个检查周期发现无效绑定,实际时延还包括内部请求和调度。不能承诺数据库提交瞬间物理连接同步消失;已提交到平台的写操作也不能撤回。
## 4. 新设备WSS握手与任务信封
broker模式设备proof保留host_id、challenge、session_id、kx、ts,并增加严格整数`protocol_version=2`、environment、app_id、binding_owner、binding_generation;全部由设备签名。旧握手在broker模式拒绝,legacy模式保持旧协议。
connector下行消息新增routing签名,绑定turn_id、host、session、环境/应用、generation和完整密文envelope摘要。router验证后向内部接口查询该任务的当前准入,再下发原密文。密文内task_context包含task_id、environment、app_id、owner、host、generation、conversation_id、chat_id、source_message_id及Work通道。
**客户端尚未在本阶段修改。** 后续PC桥接必须发送新握手,将可信task_context传入原生API、MCP/工具运行时,并保留Profile与工具禁用限制。本机旧适配器的稳定单聊session键不能替代这个上下文,也不能把服务端路由测试说成已完成客户端Agent记忆/并发隔离。
当前仍只接受p2p文本事件;新模式要求有效chat_id,不截断后混用会话键。conversation_id按环境、应用、用户、chat_id和Work单聊通道确定。群聊、话题和多Profile产品行为未擅自扩展。
## 5. 持久任务状态与原路回复
`work_tasks_v2`保存原消息/会话、owner、设备、generation、session、worker、deadline和状态,**不保存明文任务或回答正文**。同一turn_id不重复建立/执行。密文结果必须通过设备签名/解密、原turn、原session及数据库当前状态验证,才能完成本进程pending future。
典型状态为`inflight → completed → replying → replied`。completed仅表示已收到有效结果,replying表示单次平台回复已认领;回复出错为reply_unknown,不因消息重复投递自动重发。诊断任务无原消息时可标记finished。解绑/吊销变为unbound/revoked,超时或任务所有者失联为unknown,迟到结果不能改变终态。
机器人回复位置从数据库原消息映射取得并进行单次认领,不接受客户端/模型改写chat_id/message_id。不同用户或同用户不同任务逆序返回仍回原消息;取消/解绑后不把旧结果投到新消息。解绑命令自身的确认回复不等于继续发送被取消任务的答案。
worker采用随机进程ID与30秒Redis租约,每5秒续期。维护扫描轮转检查活跃任务;进程退出或租约消失后遗留任务在后续维护/准入检查中拒绝。进程恢复不重放旧任务,不声称恢复已丢失的内存结果或提供持久消息队列。已发到平台但回复结果未知时保留reply_unknown,不承诺exactly-once。
连接/请求退出时先释放内存连接槽和pending记录;数据库状态更新或Redis清理失败不得阻止本地释放,遗留在线键由TTL清理、任务由持久状态维护处理,不凭遗留presence继续放行旧绑定。
## 6. 通知与凭据的解绑边界
broker模式`/device/notifications`新增必填binding_generation并纳入原work-notify-v1签名payload;缺失/陈旧代次拒绝。取得应用Token后发送前再查owner/generation,防止等待期间同用户重绑导致旧请求继续发消息。legacy可不传该字段,原签名格式保持不变。
授权卡仍引用server-owned事务,发送前再查事务状态和代次。设备解绑后停止再次发Token和通知,但**已发出的用户Bearer access token仍可能有效到期**;是否即时撤销取决于平台能力,不能把DB解绑称为平台授权撤销。正常客户端停止重连、清access缓存的交互需要后续客户端适配。
## 7. 配置、测试与待办
默认legacy及现有客户端不切换。启用阶段4 broker之前,必须同时准备router独立准入配置、内部网络、公钥、产品权限和新PC协议;新旧broker握手不混跑。不能为了旧客户端放开无绑定准入或重新下发refresh。
本地使用平台桩、加密合成凭据、SQLite、FakeRedis和真实回环WebSocket,验证协议和并发。PostgreSQL用例扩充配对epoch/outbox检查,但没有真实连接时明确skipped;FakeRedis不等于真实Redis多节点/故障演练。
阶段5待做:Compose与初始化密钥/权限参数接线、内部网络/证书检查、候选ARM64构建、客户端握手与上下文适配交接、数据留存清理、outbox/worker/任务监控、旧模式迁移和真实平台/双用户/规模验收。8秒连接复核会增加内部控制请求,必须实测容量后再扩大设备规模,不外推十万人能力。
+26 -1
View File
@@ -11,7 +11,7 @@
| 1 | 服务端加密凭据存储基础 | 密文持久化、应用/用户/环境隔离、版本冲突、密钥轮换与篡改检测测试;不接入旧路由 | 已完成本地开发测试,提交71b9678 | | 1 | 服务端加密凭据存储基础 | 密文持久化、应用/用户/环境隔离、版本冲突、密钥轮换与篡改检测测试;不接入旧路由 | 已完成本地开发测试,提交71b9678 |
| 2 | 集中凭据Broker及刷新 | 应用/用户锁、持锁重读、版本CAS、响应校验、刷新unknown;新授权不受旧刷新覆盖 | 本次完成本地开发测试 | | 2 | 集中凭据Broker及刷新 | 应用/用户锁、持锁重读、版本CAS、响应校验、刷新unknown;新授权不受旧刷新覆盖 | 本次完成本地开发测试 |
| 3 | OAuth与认证领取 | 共享state事务、实际用户核验、绑定代次、只返回access的独立DTO及领取认证 | 本次完成本地开发测试,显式broker模式启用 | | 3 | OAuth与认证领取 | 共享state事务、实际用户核验、绑定代次、只返回access的独立DTO及领取认证 | 本次完成本地开发测试,显式broker模式启用 |
| 4 | ECS权威解绑与消息任务关联 | 停发Token、失效事务/任务、连接和旧码联动,多实例/异常恢复 | 待开发 | | 4 | ECS权威解绑与消息任务关联 | 停发Token、失效事务/任务、连接和旧码联动,多实例/异常恢复 | 本次完成本地开发测试,broker模式生效 |
| 5 | 配置部署、迁移与旧接口退出 | 密钥挂载、产品权限配置、客户端对接、停用旧刷新/claim及代理、ARM64和真实平台验收 | 待开发 | | 5 | 配置部署、迁移与旧接口退出 | 密钥挂载、产品权限配置、客户端对接、停用旧刷新/claim及代理、ARM64和真实平台验收 | 待开发 |
产品经理决定功能和Scope清单。阶段1不预设具体权限,不生成真实凭据、不读取现有设备、不调用平台。 产品经理决定功能和Scope清单。阶段1不预设具体权限,不生成真实凭据、不读取现有设备、不调用平台。
@@ -126,3 +126,28 @@ Windows / Python 3.12.10,平台桩、临时证书/密钥、SQLite及带Lua的F
最后完整回归:**185 passed,3 skipped,15 warnings**(49.77秒)。3个跳过项均需要真实PostgreSQL测试连接;warnings仍为原有uvicorn/websockets弃用提示。文档表格/围栏、relay模块AST及`git diff --check`通过。 最后完整回归:**185 passed,3 skipped,15 warnings**(49.77秒)。3个跳过项均需要真实PostgreSQL测试连接;warnings仍为原有uvicorn/websockets弃用提示。文档表格/围栏、relay模块AST及`git diff --check`通过。
真实PostgreSQL、真实Redis多进程、ARM64镜像、私有化平台及现机业务验收尚未执行;本地ASGI/FakeRedis结果不能作为线上多节点或生产部署结论。 真实PostgreSQL、真实Redis多进程、ARM64镜像、私有化平台及现机业务验收尚未执行;本地ASGI/FakeRedis结果不能作为线上多节点或生产部署结论。
## 阶段4:ECS权威解绑、连接准入与任务关联(2026-11-04)
细节见[阶段4协议和边界](BINDING_TASK_LIFECYCLE.md)。本次仅修改ECS项目,未修改或发布PC安装包,未部署现机。
### 改造内容
- 新增`BindingState`及Store生命周期挂钩,在broker模式统一bind/unbind/revoke,数据库事务内更改绑定代次、作废未完成授权/任务/配对码epoch、记录审计和outbox。legacy路径保持原行为。
- 新增认证设备解绑入口;机器人命令、管理员和设备解绑共用同一权威状态。旧配对码按owner epoch拒绝;新broker配对码只存摘要,和绑定一起一次性消费,不接受原legacy Redis码。
- Redis失效动作失败时保留outbox供本节点/其他connector重试;删除在线presence有比较保护,断连消息包含代次,旧事件不误断新连接。接口published/pending不等于PC确认。
- router新增独立内部准入Secret,和事件接入Secret分权;请求带nonce并验证connector签名的短期准入结果,不取得Token。握手/订阅后/任务上下行/周期均复核数据库绑定及任务;控制服务失败时拒绝转发。
- 新v2设备握手绑定应用、环境、owner和generation;任务下行带签名routing及密文摘要。新模式旧PC握手会拒绝,不能宣称对现有安装透明兼容。
- 持久任务记录关联原message/chat/conversation、设备、generation、session、worker及deadline;结果逆序仍按原任务交付,平台回复单次认领;迟到/解绑/死亡worker/超时不改变终态、不重放写任务。
- worker使用30秒Redis存活租约和5秒维护;任务扫描轮转避免后续任务饥饿;无结果正文持久化,不承诺重启后恢复丢失的内存结果。
- 通知增加broker必填generation并在等待平台凭据后复查,防止同用户重绑后旧通知继续发送;legacy原签名兼容。应用及用户refresh均不下发。
### 本地验证
新增阶段4专项28项:真实回环WebSocket而非纯函数桩,覆盖两用户/同用户多任务逆序、机器人解绑断连与旧回包拒绝、数据库提交/Redis失败补偿、事务回滚、另一connector重开补偿、旧码拒绝与一次消费、迟到kill不影响新连接、通知期间重绑不发、吊销与旧握手拒绝、控制面中断拒绝转发、周期查库发现解绑、死亡worker和reply_unknown不重放,以及DB/Redis清理失败仍释放内存连接/等待记录。扩展可选PostgreSQL用例覆盖owner epoch和outbox,但无连接时仍跳过。
专项测试不代表真实Hermes客户端的记忆隔离已经完成;本次测试用合成PC设备/平台桩和本地SQLite/FakeRedis,客户端可信TaskContext传入Agent/MCP仍需后续适配。默认旧模式、服务器配置和真实凭据均未切换。
最后完整回归:**213 passed,3 skipped,30 warnings**(45.87秒)。3个跳过项均需要真实PostgreSQL连接,warnings为既有uvicorn/websockets旧API弃用提示(新增真实回环连接增加提示次数);relay模块AST、文档和diff检查通过。
生产部署、真实Redis/PostgreSQL多实例、ARM64镜像、平台实际撤权/Scope/容量和客户端安装升级仍待阶段5及现场验收。平台已接收的请求不可靠解绑撤回,已发access仍可能有效到期。
+5 -4
View File
@@ -1,6 +1,6 @@
# 集中凭据协议 v2:阶段3接口指导 # 集中凭据协议 v2:阶段3接口指导
日期:2026-11-04。本文是已经完成本地开发测试的服务端接口契约,不表示现机已启用或新客户端已交付。产品经理决定权限集合;本文件不替产品选择具体业务或Scope。 日期:2026-11-04。本文是已经完成本地开发测试的服务端接口契约,不表示现机已启用或新客户端已交付。产品经理决定权限集合;本文件不替产品选择具体业务或Scope。阶段4的解绑、内部准入和WSS扩展见[连接与任务生命周期](BINDING_TASK_LIFECYCLE.md)。
## 1. 启用与模式边界 ## 1. 启用与模式边界
@@ -12,6 +12,7 @@ Settings新增以下配置,环境变量均以`HN_`开头:
| `CREDENTIAL_KEYRING_FILE` | broker必须提供受控服务端密钥文件,格式见阶段1记录;无有效密钥启动失败,不回退 | | `CREDENTIAL_KEYRING_FILE` | broker必须提供受控服务端密钥文件,格式见阶段1记录;无有效密钥启动失败,不回退 |
| `CREDENTIAL_PERMISSION_PROFILES` | JSON对象:产品权限版本名映射Scope字符串数组;broker不可为空,不允许客户端传任意Scope | | `CREDENTIAL_PERMISSION_PROFILES` | JSON对象:产品权限版本名映射Scope字符串数组;broker不可为空,不允许客户端传任意Scope |
| `DATABASE_URL` | 生产broker必须使用PostgreSQL;SQLite仅用于本地开发和测试 | | `DATABASE_URL` | 生产broker必须使用PostgreSQL;SQLite仅用于本地开发和测试 |
| `ROUTER_ADMISSION_SECRET` | 阶段4新增:connector与router共用的独立内部准入凭据,不能复用事件/管理/应用Secret;需受控挂载 |
| 平台与OAuth | broker要求`FEISHU_ENABLED`和`OAUTH_ENABLED`为true,应用凭据仍只在ECS | | 平台与OAuth | broker要求`FEISHU_ENABLED`和`OAUTH_ENABLED`为true,应用凭据仍只在ECS |
| 授权入口与公开URL | 固定HTTPS授权端点,不能含预设query/fragment或用户名密码;公开HTTPS回调精确登记 | | 授权入口与公开URL | 固定HTTPS授权端点,不能含预设query/fragment或用户名密码;公开HTTPS回调精确登记 |
@@ -84,9 +85,9 @@ Broker换码后调用user_info核验实际授权人与原绑定一致,回写
### POST /device/notifications(原接口的新授权卡兼容) ### POST /device/notifications(原接口的新授权卡兼容)
文本/通知卡协议不变。broker模式的`kind=authorization`将原`oauth_state`字段用作**transaction_id引用**,不是原始state或任意URL;ECS验证事务归属、当前绑定版本和waiting状态后从服务器生成链接。发送前再校验,不能把其他设备事务发给本人或用旧代次链接继续授权。 legacy文本/通知卡协议不变;broker模式从阶段4起必须传当前`binding_generation`并纳入work-notify-v1签名payload。broker模式的`kind=authorization`将原`oauth_state`字段用作**transaction_id引用**,不是原始state或任意URL;ECS验证事务归属、当前绑定版本和waiting状态后从服务器生成链接。发送前再校验,不能把其他设备事务发给本人或用旧代次链接继续授权。
本人通知收件人与路由继续由ECS确定,应用Token不下发。完整通知、路由和旧绑定码失效补偿在阶段4继续完善。 本人通知收件人与路由继续由ECS确定,应用Token不下发。阶段4已补当前绑定版本与旧绑定码失效、路由准入及持久补偿;客户端侧对应升级和真实部署仍需后续验证。
## 4. 错误与审计 ## 4. 错误与审计
@@ -99,4 +100,4 @@ Broker换码后调用user_info核验实际授权人与原绑定一致,回写
## 5. 本阶段尚未完成 ## 5. 本阶段尚未完成
无实际服务器部署、真实PostgreSQL/多节点Redis验收、真实Work OAuth复测或客户端发布。完整解绑连接和任务补偿、旧绑定码作废、凭据拒绝自动失效上报、集中刷新后台调度、运维清理和正式Compose参数挂载为后续阶段。不要绕过新领取认证临时开放旧refresh接口。 无实际服务器部署、真实PostgreSQL/多节点Redis验收、真实Work OAuth复测或客户端发布。阶段4新增的解绑/任务/旧绑定码补偿以对应文档和本地测试为准;凭据拒绝自动失效上报、集中刷新后台调度、运维清理和正式Compose参数挂载仍为后续工作。不要绕过新领取认证临时开放旧refresh接口。
+2
View File
@@ -2,6 +2,8 @@
集中凭据改造按阶段交付,见 [改造与阶段记录](CREDENTIAL_MIGRATION.md)。阶段1/2已提供存储与刷新,阶段3新增显式broker模式的OAuth及认证领取接口,默认仍为legacy;[v2接口契约](CREDENTIAL_PROTOCOL_V2.md)包含签名、配置和兼容边界。尚未部署或发布新客户端,不能据此判断现网已切换。 集中凭据改造按阶段交付,见 [改造与阶段记录](CREDENTIAL_MIGRATION.md)。阶段1/2已提供存储与刷新,阶段3新增显式broker模式的OAuth及认证领取接口,默认仍为legacy;[v2接口契约](CREDENTIAL_PROTOCOL_V2.md)包含签名、配置和兼容边界。尚未部署或发布新客户端,不能据此判断现网已切换。
阶段4新增[权威解绑、连接准入和任务生命周期](BINDING_TASK_LIFECYCLE.md),broker模式要求router独立准入凭据及PC v2握手。默认legacy不变;部署接线和客户端适配未完成前不要直接切换生产模式。
2026-10-28 新增[客户端主动消息与授权卡片](NOTIFICATIONS.md),本地代码已实现,现网部署和卡片验收待完成。 2026-10-28 新增[客户端主动消息与授权卡片](NOTIFICATIONS.md),本地代码已实现,现网部署和卡片验收待完成。
本目录用于新环境部署,源码对应 2026-10-27 ECS connector/router/longconn 导出版本。协议和业务源码保持原样。仓库其他快照目录用于追溯,不需要手工合并。 本目录用于新环境部署,源码对应 2026-10-27 ECS connector/router/longconn 导出版本。协议和业务源码保持原样。仓库其他快照目录用于追溯,不需要手工合并。
+182
View File
@@ -0,0 +1,182 @@
"""Transactional broker-mode binding, pairing, invalidation and task metadata.
No message bodies or credential values are stored here. All mutating binding
paths in a broker connector use this object through Store's lifecycle hook.
"""
import hashlib
import time
import uuid
class BindingState:
def __init__(self, store, environment, app_id):
self.store, self.db, self.environment, self.app_id = store, store.db, environment, app_id
with self.db:
self.db.execute("""CREATE TABLE IF NOT EXISTS work_owner_epochs_v2(
environment TEXT NOT NULL,app_id TEXT NOT NULL,owner TEXT NOT NULL,epoch BIGINT NOT NULL,
PRIMARY KEY(environment,app_id,owner))""")
self.db.execute("""CREATE TABLE IF NOT EXISTS work_binding_codes_v2(
code_hash TEXT PRIMARY KEY,environment TEXT NOT NULL,app_id TEXT NOT NULL,
owner TEXT NOT NULL,owner_epoch BIGINT NOT NULL,expires_at DOUBLE PRECISION NOT NULL,
consumed INTEGER NOT NULL DEFAULT 0)""")
self.db.execute("""CREATE TABLE IF NOT EXISTS work_binding_events_v2(
event_id TEXT PRIMARY KEY,environment TEXT NOT NULL,app_id TEXT NOT NULL,host_id TEXT NOT NULL,
owner TEXT NOT NULL,binding_generation BIGINT NOT NULL,reason TEXT NOT NULL,
created_at DOUBLE PRECISION NOT NULL,delivered INTEGER NOT NULL DEFAULT 0)""")
self.db.execute("""CREATE TABLE IF NOT EXISTS work_tasks_v2(
turn_id TEXT PRIMARY KEY,environment TEXT NOT NULL,app_id TEXT NOT NULL,owner TEXT NOT NULL,
host_id TEXT NOT NULL,binding_generation BIGINT NOT NULL,session_id TEXT NOT NULL,
worker_id TEXT NOT NULL,conversation_id TEXT NOT NULL,chat_id TEXT NOT NULL,
source_message_id TEXT NOT NULL,status TEXT NOT NULL,deadline DOUBLE PRECISION NOT NULL,
created_at DOUBLE PRECISION NOT NULL,updated_at DOUBLE PRECISION NOT NULL)""")
self.db.execute("""CREATE INDEX IF NOT EXISTS work_binding_events_pending_v2
ON work_binding_events_v2(environment,app_id,delivered,created_at)""")
self.db.execute("""CREATE INDEX IF NOT EXISTS work_tasks_active_v2
ON work_tasks_v2(environment,app_id,status,updated_at)""")
self.db.execute("""CREATE INDEX IF NOT EXISTS work_tasks_binding_v2
ON work_tasks_v2(environment,app_id,host_id,binding_generation)""")
def _owner_lock(self, owner, bump=False):
identity = (self.environment, self.app_id, owner)
self.db.execute("INSERT INTO work_owner_epochs_v2 VALUES(?,?,?,0) ON CONFLICT DO NOTHING", identity)
self.db.execute("UPDATE work_owner_epochs_v2 SET epoch=epoch+? WHERE environment=? AND app_id=? AND owner=?",
(1 if bump else 0, *identity))
return self.db.execute("SELECT epoch FROM work_owner_epochs_v2 WHERE environment=? AND app_id=? AND owner=?",
identity).fetchone()["epoch"]
def _bind(self, host, owner):
count = self.db.execute("""UPDATE devices SET owner=?,binding_generation=binding_generation+1
WHERE host_id=? AND owner IS NULL AND revoked=0""", (owner, host)).rowcount
if count != 1:
raise ValueError("device unavailable or already bound")
def bind(self, host, owner):
with self.db:
self._owner_lock(owner)
self._bind(host, owner)
def issue_code(self, owner, code, ttl=300):
with self.db:
epoch = self._owner_lock(owner)
if self.db.execute("SELECT host_id FROM devices WHERE owner=?", (owner,)).fetchone():
raise ValueError("owner already bound")
self.db.execute("INSERT INTO work_binding_codes_v2 VALUES(?,?,?,?,?,?,0)",
(hashlib.sha256(code.encode()).hexdigest(), self.environment, self.app_id, owner, epoch, time.time()+ttl))
def consume_code(self, host, code):
digest = hashlib.sha256(code.encode()).hexdigest()
row = self.db.execute("SELECT * FROM work_binding_codes_v2 WHERE code_hash=? AND environment=? AND app_id=?",
(digest, self.environment, self.app_id)).fetchone()
if not row:
raise LookupError("binding code unavailable")
with self.db:
epoch = self._owner_lock(row["owner"])
count = self.db.execute("""UPDATE work_binding_codes_v2 SET consumed=1
WHERE code_hash=? AND owner_epoch=? AND consumed=0 AND expires_at>?""",
(digest, epoch, time.time())).rowcount
if count != 1:
raise LookupError("binding code unavailable")
self._bind(host, row["owner"])
return row["owner"]
def _cancel(self, device, reason):
owner = device["owner"] or ""
generation = device["binding_generation"]
self.db.execute("""UPDATE work_authorizations_v2 SET status='invalidated'
WHERE environment=? AND app_id=? AND host_id=? AND binding_generation<=?
AND status IN ('waiting','exchanging')""",
(self.environment, self.app_id, device["host_id"], generation))
self.db.execute("""UPDATE work_tasks_v2 SET status=?,updated_at=?
WHERE environment=? AND app_id=? AND host_id=? AND binding_generation<=?
AND status IN ('inflight','completed','replying')""",
("revoked" if reason == "revoke" else "unbound", time.time(), self.environment,
self.app_id, device["host_id"], generation))
event = uuid.uuid4().hex
self.db.execute("INSERT INTO work_binding_events_v2 VALUES(?,?,?,?,?,?,?,?,0)",
(event, self.environment, self.app_id, device["host_id"], owner, generation+1, reason, time.time()))
self.db.execute("INSERT INTO audit(ts,actor,action,resource,result) VALUES(?,?,?,?,?)",
(time.time(), owner or "admin", "binding_"+reason, device["host_id"], "committed"))
def unbind(self, owner):
with self.db:
self._owner_lock(owner, bump=True) # Invalidates pre-unbind codes even if already unbound.
device = self.db.execute("SELECT * FROM devices WHERE owner=?", (owner,)).fetchone()
if device:
self.db.execute("UPDATE devices SET owner=NULL,binding_generation=binding_generation+1 WHERE host_id=?",
(device["host_id"],))
self._cancel(dict(device), "unbind")
def invalidate_host(self, host, reason, *, expected_owner=None, expected_generation=None):
if reason not in {"unbind", "revoke"}:
raise ValueError("invalid binding change")
device = self.store.device(host)
if not device:
raise LookupError("device not found")
if ((expected_owner is not None and device["owner"] != expected_owner)
or (expected_generation is not None and device["binding_generation"] != expected_generation)):
raise ValueError("binding changed; retry")
if reason == "unbind" and not device["owner"]:
return
if reason == "revoke" and device["revoked"]:
return
with self.db:
if device["owner"]:
self._owner_lock(device["owner"], bump=True)
mutation = "owner=NULL" if reason == "unbind" else "revoked=1"
count = self.db.execute(f"""UPDATE devices SET {mutation},binding_generation=binding_generation+1
WHERE host_id=? AND binding_generation=? AND COALESCE(owner,'')=? AND revoked=?""",
(host, device["binding_generation"], device["owner"] or "", device["revoked"])).rowcount
if count != 1:
raise ValueError("binding changed; retry")
self._cancel(device, reason)
def task(self, turn):
row = self.db.execute("SELECT * FROM work_tasks_v2 WHERE turn_id=? AND environment=? AND app_id=?",
(turn, self.environment, self.app_id)).fetchone()
return dict(row) if row else None
def task_current(self, task):
device = self.store.device(task["host_id"]) if task else None
return bool(device and not device["revoked"] and device["owner"] == task["owner"]
and device["binding_generation"] == task["binding_generation"]
and task["deadline"] > time.time())
def create_task(self, turn, owner, host, generation, session, worker, deadline, chat_id="", message_id=""):
conversation = hashlib.sha256((self.environment+"\0"+self.app_id+"\0"+owner+"\0"+chat_id+"\0work-dm").encode()).hexdigest()
with self.db:
self._owner_lock(owner)
device = self.store.device(host)
if not device or device["revoked"] or device["owner"] != owner or device["binding_generation"] != generation:
raise ValueError("binding changed")
count = self.db.execute("""INSERT INTO work_tasks_v2 VALUES(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)
ON CONFLICT DO NOTHING""", (turn,self.environment,self.app_id,owner,host,generation,session,
worker,conversation,chat_id,message_id,"inflight",deadline,time.time(),time.time())).rowcount
if count != 1:
raise ValueError("task already recorded")
return self.task(turn)
def transition(self, turn, previous, status):
with self.db:
return self.db.execute("""UPDATE work_tasks_v2 SET status=?,updated_at=?
WHERE turn_id=? AND environment=? AND app_id=? AND status=?""",
(status,time.time(),turn,self.environment,self.app_id,previous)).rowcount == 1
def complete(self, turn, host, session, worker):
with self.db:
task = self.task(turn)
if not task:
return False
self._owner_lock(task["owner"])
if not self.task_current(task) or (task["host_id"],task["session_id"],task["worker_id"],task["status"]) != (host,session,worker,"inflight"):
return False
return self.db.execute("UPDATE work_tasks_v2 SET status='completed',updated_at=? WHERE turn_id=? AND status='inflight'",
(time.time(),turn)).rowcount == 1
def claim_reply(self, turn, owner, message_id):
with self.db:
self._owner_lock(owner)
task = self.task(turn)
if not self.task_current(task) or (task["owner"],task["source_message_id"],task["status"]) != (owner,message_id,"completed"):
return False
return self.db.execute("UPDATE work_tasks_v2 SET status='replying',updated_at=? WHERE turn_id=? AND status='completed'",
(time.time(),turn)).rowcount == 1
+12
View File
@@ -44,11 +44,17 @@ class Settings(BaseSettings):
credential_mode: str = "legacy" credential_mode: str = "legacy"
credential_keyring_file: Path | None = None credential_keyring_file: Path | None = None
credential_permission_profiles: dict[str, list[str]] = {} credential_permission_profiles: dict[str, list[str]] = {}
connector_internal_url: str = "http://connector:8000"
router_admission_secret: str = ""
@model_validator(mode="after") @model_validator(mode="after")
def validate_runtime(self): def validate_runtime(self):
if self.credential_mode not in {"legacy", "broker"}: if self.credential_mode not in {"legacy", "broker"}:
raise ValueError("invalid credential mode") raise ValueError("invalid credential mode")
if self.credential_mode == "broker" and self.role in {"connector","router"}:
if (len(self.router_admission_secret) < 32 or self.router_admission_secret in {
self.internal_secret,self.admin_secret,self.feishu_app_secret}):
raise ValueError("broker mode requires an independent router admission credential")
for profile, scopes in self.credential_permission_profiles.items(): for profile, scopes in self.credential_permission_profiles.items():
if (not re.fullmatch(r"[A-Za-z0-9_-]{1,64}", profile) or not scopes or len(scopes) > 128 if (not re.fullmatch(r"[A-Za-z0-9_-]{1,64}", profile) or not scopes or len(scopes) > 128
or any(not re.fullmatch(r"[A-Za-z0-9_:.-]{1,256}", scope) for scope in scopes)): or any(not re.fullmatch(r"[A-Za-z0-9_:.-]{1,256}", scope) for scope in scopes)):
@@ -63,6 +69,12 @@ class Settings(BaseSettings):
if (authorize.scheme != "https" or not authorize.hostname or authorize.username or authorize.password if (authorize.scheme != "https" or not authorize.hostname or authorize.username or authorize.password
or authorize.query or authorize.fragment): or authorize.query or authorize.fragment):
raise ValueError("broker mode requires a fixed HTTPS authorization endpoint") raise ValueError("broker mode requires a fixed HTTPS authorization endpoint")
if self.credential_mode == "broker" and self.role == "router":
target = urlsplit(self.connector_internal_url)
if (not self.feishu_app_id or target.scheme not in {"http","https"}
or not target.hostname or target.username or target.password or target.query or target.fragment
or target.path not in {"","/"}):
raise ValueError("broker router requires authenticated fixed internal connector")
if self.environment not in {"production", "test", "development"}: if self.environment not in {"production", "test", "development"}:
raise ValueError("invalid environment") raise ValueError("invalid environment")
if self.role == "connector" and min(len(self.admin_secret), len(self.internal_secret)) < 32: if self.role == "connector" and min(len(self.admin_secret), len(self.internal_secret)) < 32:
+159 -17
View File
@@ -24,7 +24,7 @@ from redis.asyncio import Redis
from .common import BodyLimitMiddleware, no_cache, peer, rate_limit, require_secret from .common import BodyLimitMiddleware, no_cache, peer, rate_limit, require_secret
from .config import Settings from .config import Settings
from .crypto import (fingerprint, load_private, open_envelope, raw_public, seal, from .crypto import (canonical, fingerprint, load_private, open_envelope, raw_public, seal,
sign, verified, verify_device) sign, verified, verify_device)
from .oauth import OAuth from .oauth import OAuth
from .platform import Feishu, PlatformError from .platform import Feishu, PlatformError
@@ -62,6 +62,8 @@ class Broker:
self.s, self.r, self.store = settings, redis, store self.s, self.r, self.store = settings, redis, store
self.signer = load_private(settings.connector_key) self.signer = load_private(settings.connector_key)
self.pending: dict[str, Pending] = {} self.pending: dict[str, Pending] = {}
self.worker_id = uuid.uuid4().hex
self.lifecycle = None
async def replies(self, pub): async def replies(self, pub):
async for message in pub.listen(): async for message in pub.listen():
@@ -79,12 +81,14 @@ class Broker:
result = plain.get("result", "") result = plain.get("result", "")
if not isinstance(result, str) or len(result) > self.s.max_result: if not isinstance(result, str) or len(result) > self.s.max_result:
continue continue
if self.lifecycle and not self.lifecycle.complete(data["turn_id"],item.host,item.session,self.worker_id):
continue
item.future.set_result({"status": "completed", "turn_id": data["turn_id"], "result": result}) item.future.set_result({"status": "completed", "turn_id": data["turn_id"], "result": result})
except Exception: except Exception:
# Invalid packets never consume a pending result; do not log packet contents. # Invalid packets never consume a pending result; do not log packet contents.
continue continue
async def dispatch(self, owner, text, turn): async def dispatch(self, owner, text, turn, *, chat_id="", source_message_id=""):
device = self.store.by_owner(owner) device = self.store.by_owner(owner)
if not device: if not device:
return {"status": "unavailable", "turn_id": turn, "result": "未绑定设备或设备不可用。"} return {"status": "unavailable", "turn_id": turn, "result": "未绑定设备或设备不可用。"}
@@ -104,36 +108,75 @@ class Broker:
if proof["host_id"] != host or proof["challenge"] != presence["challenge"]: if proof["host_id"] != host or proof["challenge"] != presence["challenge"]:
raise ValueError("invalid device session") raise ValueError("invalid device session")
session = proof["session_id"] session = proof["session_id"]
if self.lifecycle and (proof.get("binding_owner") != owner
or proof.get("binding_generation") != device["binding_generation"]):
raise ValueError("stale binding session")
except Exception: except Exception:
self.store.audit(owner, "dispatch", host, "identity_rejected") self.store.audit(owner, "dispatch", host, "identity_rejected")
raise HTTPException(409, "device identity rejected") from None raise HTTPException(409, "device identity rejected") from None
if not self.store.begin_turn(turn, owner, host): if not self.store.begin_turn(turn, owner, host):
return {"status": "duplicate", "turn_id": turn, "result": "该请求已接收,不会重复执行。"} return {"status": "duplicate", "turn_id": turn, "result": "该请求已接收,不会重复执行。"}
task = None
if self.lifecycle:
try:
task = self.lifecycle.create_task(turn,owner,host,device["binding_generation"],session,
self.worker_id,time.time()+self.s.request_timeout,chat_id,source_message_id)
except ValueError:
self.store.finish(turn,"unbound")
return {"status":"unbound","turn_id":turn,"result":"绑定已变化,任务未下发。"}
private = x25519.X25519PrivateKey.generate() private = x25519.X25519PrivateKey.generate()
future = asyncio.get_running_loop().create_future() future = asyncio.get_running_loop().create_future()
self.pending[turn] = Pending(future, private, public, host, session) self.pending[turn] = Pending(future, private, public, host, session)
envelope = seal({"type": "instruction", "turn_id": turn, "owner": owner, payload = {"type": "instruction", "turn_id": turn, "owner": owner,
"text": text, "read_only": True, "channel": "im", "text": text, "read_only": True, "channel": "im",
"reply_key": raw_public(private.public_key())}, proof["kx"], self.signer, "reply_key": raw_public(private.public_key())}
host_id=host, session_id=session) if task:
payload["task_context"] = {key:task[key] for key in ("environment","app_id","owner","host_id","binding_generation",
"conversation_id","chat_id","source_message_id")}
payload["task_context"].update(task_id=turn,channel="work-dm")
try:
envelope = seal(payload, proof["kx"], self.signer,host_id=host,session_id=session)
except Exception:
self.pending.pop(turn,None)
self.store.finish(turn,"unknown")
if task:
self.lifecycle.transition(turn,"inflight","unknown")
raise HTTPException(409,"invalid_device_crypto_session") from None
key = f"hn:inflight:{host}:{turn}" key = f"hn:inflight:{host}:{turn}"
try: try:
await self.r.set(key, session, ex=self.s.request_timeout + 5) await self.r.set(key, session, ex=self.s.request_timeout + 5)
count = await self.r.publish("hn:down:" + host, json.dumps({"type": "instruction", "envelope": envelope})) packet = {"type":"instruction","envelope":envelope}
if task:
packet["routing"] = sign({"turn_id":turn,"host_id":host,"session_id":session,
"environment":self.s.environment,"app_id":self.s.feishu_app_id,
"binding_generation":task["binding_generation"],
"envelope_hash":hashlib.sha256(canonical(envelope)).hexdigest()}, self.signer)
count = await self.r.publish("hn:down:" + host, json.dumps(packet))
if not count: if not count:
self.store.finish(turn, "offline") self.store.finish(turn, "offline")
if task:
self.lifecycle.transition(turn,"inflight","offline")
return {"status": "offline", "turn_id": turn, "result": "你的电脑当前离线。"} return {"status": "offline", "turn_id": turn, "result": "你的电脑当前离线。"}
result = await asyncio.wait_for(future, timeout=self.s.request_timeout) result = await asyncio.wait_for(future, timeout=self.s.request_timeout)
self.store.finish(turn, "completed") self.store.finish(turn, result.get("status","unknown") if task else "completed")
self.store.audit(owner, "instruction", host, "completed") if task and not source_message_id:
self.lifecycle.transition(turn,"completed","finished")
self.store.audit(owner, "instruction", host, result.get("status","unknown") if task else "completed")
return result return result
except asyncio.TimeoutError: except asyncio.TimeoutError:
self.store.finish(turn, "unknown") self.store.finish(turn, "unknown")
if task:
self.lifecycle.transition(turn,"inflight","unknown")
return {"status": "unknown", "turn_id": turn, return {"status": "unknown", "turn_id": turn,
"result": "等待结果超时,执行状态未知;不要直接重发有副作用的任务。"} "result": "等待结果超时,执行状态未知;不要直接重发有副作用的任务。"}
finally: finally:
self.pending.pop(turn, None) self.pending.pop(turn, None)
await self.r.delete(key) if task:
# Cancellation, publish failures and connector shutdown never replay work.
with contextlib.suppress(Exception):
self.lifecycle.transition(turn,"inflight","unknown")
with contextlib.suppress(Exception):
await self.r.delete(key)
def create_app(settings=None, redis=None, platform=None): def create_app(settings=None, redis=None, platform=None):
@@ -149,9 +192,15 @@ def create_app(settings=None, redis=None, platform=None):
credential_store = CredentialStore(store, CredentialKeyring.from_file(s.credential_keyring_file)) credential_store = CredentialStore(store, CredentialKeyring.from_file(s.credential_keyring_file))
credential_broker = CredentialBroker(credential_store, r, f, environment=s.environment, app_id=s.feishu_app_id) credential_broker = CredentialBroker(credential_store, r, f, environment=s.environment, app_id=s.feishu_app_id)
oauth = CredentialAPI(s, r, store, credential_broker) oauth = CredentialAPI(s, r, store, credential_broker)
from .binding_state import BindingState
from .device_lifecycle import DeviceLifecycle
store.lifecycle = BindingState(store,s.environment,s.feishu_app_id)
broker.lifecycle = store.lifecycle
lifecycle = DeviceLifecycle(store.lifecycle,r,broker)
else: else:
oauth = OAuth(s, r, f) oauth = OAuth(s, r, f)
oauth.store = store oauth.store = store
lifecycle = None
from .notifications import router as notification_router from .notifications import router as notification_router
background = set() background = set()
@@ -165,6 +214,8 @@ def create_app(settings=None, redis=None, platform=None):
@asynccontextmanager @asynccontextmanager
async def lifespan(app): async def lifespan(app):
await r.ping() await r.ping()
if lifecycle:
await lifecycle.maintain()
for row in store.db.execute("SELECT host_id,fingerprint,revoked FROM devices").fetchall(): for row in store.db.execute("SELECT host_id,fingerprint,revoked FROM devices").fetchall():
await r.set("hn:device:" + row["host_id"], row["fingerprint"]) await r.set("hn:device:" + row["host_id"], row["fingerprint"])
if row["revoked"]: if row["revoked"]:
@@ -180,12 +231,25 @@ def create_app(settings=None, redis=None, platform=None):
async with oauth.lock: async with oauth.lock:
oauth.prune() oauth.prune()
sweep_task = asyncio.create_task(sweep()) sweep_task = asyncio.create_task(sweep())
async def maintenance():
while True:
await asyncio.sleep(5)
try:
await lifecycle.maintain()
except Exception:
# No credentials or raw exception text in background logs.
pass
lifecycle_task = asyncio.create_task(maintenance()) if lifecycle else None
try: try:
yield yield
finally: finally:
for task in [reply_task, sweep_task, *background]: work = [reply_task,sweep_task,*background] + ([lifecycle_task] if lifecycle_task else [])
for task in work:
task.cancel() task.cancel()
await asyncio.gather(reply_task, sweep_task, *background, return_exceptions=True) await asyncio.gather(*work, return_exceptions=True)
if lifecycle:
with contextlib.suppress(Exception):
await r.delete("hn:task-worker:"+broker.worker_id)
await pub.aclose() await pub.aclose()
await f.close() await f.close()
store.close() store.close()
@@ -201,6 +265,28 @@ def create_app(settings=None, redis=None, platform=None):
app.include_router(web_sessions.router) app.include_router(web_sessions.router)
app.state.store, app.state.broker, app.state.oauth = store, broker, oauth app.state.store, app.state.broker, app.state.oauth = store, broker, oauth
app.state.platform, app.state.redis = f, r app.state.platform, app.state.redis = f, r
app.state.lifecycle = lifecycle
if lifecycle:
from .device_lifecycle import AdmissionRequest
from .credential_api import DeviceRequest
@app.post("/work/v2/device/unbind")
async def device_unbind(body: DeviceRequest, request: Request):
await oauth.authenticate(body,request)
try:
store.lifecycle.invalidate_host(body.host_id,"unbind",expected_owner=body.binding_owner,
expected_generation=body.binding_generation)
except ValueError:
raise HTTPException(409,"binding_changed_retry") from None
lifecycle.cancel_waiters()
propagated = await lifecycle.flush()
return {"status":"unbound","binding_generation":store.device(body.host_id)["binding_generation"],
"propagation":"published" if propagated else "pending"}
@app.post("/internal/devices/{host}/admission")
async def admission(host: str, body: AdmissionRequest, request: Request):
require_secret(request.headers.get("X-Router-Admission-Secret"),s.router_admission_secret)
return sign(await lifecycle.admission(host,body),broker.signer)
@app.exception_handler(RequestValidationError) @app.exception_handler(RequestValidationError)
async def validation_error(request, exc): async def validation_error(request, exc):
@@ -261,6 +347,14 @@ def create_app(settings=None, redis=None, platform=None):
admin(request) admin(request)
if not store.device(host): if not store.device(host):
raise HTTPException(404, "device not found") raise HTTPException(404, "device not found")
if lifecycle:
try:
store.revoke(host)
except ValueError:
raise HTTPException(409,"binding_changed_retry") from None
lifecycle.cancel_waiters()
propagated = await lifecycle.flush()
return {"status":"revoked","propagation":"published" if propagated else "pending"}
# Persistent source of truth first; router block + disconnect follows. # Persistent source of truth first; router block + disconnect follows.
store.revoke(host) store.revoke(host)
await r.incr("hn:web-generation:" + host) await r.incr("hn:web-generation:" + host)
@@ -277,6 +371,8 @@ def create_app(settings=None, redis=None, platform=None):
async def reconcile(request: Request): async def reconcile(request: Request):
"""Rebuild router admission metadata after Redis recovery; no dialogue data.""" """Rebuild router admission metadata after Redis recovery; no dialogue data."""
admin(request) admin(request)
if lifecycle:
await lifecycle.maintain()
rows = store.db.execute("SELECT host_id,fingerprint,revoked FROM devices").fetchall() rows = store.db.execute("SELECT host_id,fingerprint,revoked FROM devices").fetchall()
for row in rows: for row in rows:
await r.set("hn:device:" + row["host_id"], row["fingerprint"]) await r.set("hn:device:" + row["host_id"], row["fingerprint"])
@@ -290,6 +386,15 @@ def create_app(settings=None, redis=None, platform=None):
device = store.device(host) device = store.device(host)
if not device: if not device:
raise HTTPException(404, "device not found") raise HTTPException(404, "device not found")
if lifecycle:
try:
store.lifecycle.invalidate_host(host,"unbind")
except ValueError:
raise HTTPException(409,"binding_changed_retry") from None
lifecycle.cancel_waiters()
propagated = await lifecycle.flush()
return {"status":"unbound","host_id":host,"revoked":bool(device["revoked"]),
"propagation":"published" if propagated else "pending"}
if device["owner"]: if device["owner"]:
store.unbind(device["owner"]) store.unbind(device["owner"])
await r.incr("hn:web-generation:" + host) await r.incr("hn:web-generation:" + host)
@@ -306,7 +411,13 @@ def create_app(settings=None, redis=None, platform=None):
return "你已绑定电脑,请先发送 /解绑。" return "你已绑定电脑,请先发送 /解绑。"
await rate_limit(r, "binding:" + owner, 5) await rate_limit(r, "binding:" + owner, 5)
code = secrets.token_urlsafe(12) code = secrets.token_urlsafe(12)
await r.set("hn:bind:" + hashlib.sha256(code.encode()).hexdigest(), owner, ex=300) if lifecycle:
try:
store.lifecycle.issue_code(owner,code)
except ValueError:
return "你已绑定电脑,请先发送 /解绑。"
else:
await r.set("hn:bind:" + hashlib.sha256(code.encode()).hexdigest(), owner, ex=300)
return "请在本人电脑的桥接客户端输入绑定码(5分钟有效):" + code return "请在本人电脑的桥接客户端输入绑定码(5分钟有效):" + code
@app.post("/device/bind") @app.post("/device/bind")
@@ -325,6 +436,16 @@ def create_app(settings=None, redis=None, platform=None):
raise ValueError("invalid code") raise ValueError("invalid code")
except Exception: except Exception:
raise HTTPException(403, "invalid device proof") from None raise HTTPException(403, "invalid device proof") from None
if lifecycle:
try:
owner = store.lifecycle.consume_code(body.host_id,code)
except LookupError:
raise HTTPException(404,"binding code unavailable") from None
except (*INTEGRITY_ERRORS,ValueError):
raise HTTPException(409,"owner/device already bound; request new code") from None
return sign({"host_id":body.host_id,"owner":owner,
"binding_generation":store.device(body.host_id)["binding_generation"],
"ts":int(time.time())},broker.signer)
key = "hn:bind:" + hashlib.sha256(code.encode()).hexdigest() key = "hn:bind:" + hashlib.sha256(code.encode()).hexdigest()
owner = await r.get(key) owner = await r.get(key)
if not owner: if not owner:
@@ -365,11 +486,15 @@ def create_app(settings=None, redis=None, platform=None):
owner = sender.get("sender_id", {}).get("open_id") owner = sender.get("sender_id", {}).get("open_id")
message_id = message.get("message_id", "") message_id = message.get("message_id", "")
if (message.get("chat_type") != "p2p" or message.get("message_type") != "text" if (message.get("chat_type") != "p2p" or message.get("message_type") != "text"
or sender.get("sender_type") != "user" or not owner or sender.get("sender_type") != "user" or not isinstance(owner,str)
or not 0 < len(owner) <= 128 or any(ord(c) < 32 for c in owner)
or not re.fullmatch(r"[A-Za-z0-9_-]{1,128}", message_id)): or not re.fullmatch(r"[A-Za-z0-9_-]{1,128}", message_id)):
return return
if header.get("app_id") != s.feishu_app_id: if header.get("app_id") != s.feishu_app_id:
return return
if lifecycle and (not isinstance(message.get("chat_id"),str) or not 0 < len(message["chat_id"]) <= 128
or any(ord(c) < 32 for c in message["chat_id"])):
return
try: try:
text = json.loads(message["content"])["text"].strip() text = json.loads(message["content"])["text"].strip()
if not text or len(text) > 8000: if not text or len(text) > 8000:
@@ -383,7 +508,10 @@ def create_app(settings=None, redis=None, platform=None):
elif text == "/解绑": elif text == "/解绑":
old = store.by_owner(owner) old = store.by_owner(owner)
store.unbind(owner) store.unbind(owner)
if old: if lifecycle:
lifecycle.cancel_waiters()
await lifecycle.flush()
elif old:
await r.incr("hn:web-generation:" + old["host_id"]) await r.incr("hn:web-generation:" + old["host_id"])
await r.publish("hn:kill:" + old["host_id"], "unbind") await r.publish("hn:kill:" + old["host_id"], "unbind")
for item in broker.pending.values(): for item in broker.pending.values():
@@ -391,11 +519,25 @@ def create_app(settings=None, redis=None, platform=None):
item.future.set_result({"status": "unbound", "result": "设备已解绑,结果已丢弃。"}) item.future.set_result({"status": "unbound", "result": "设备已解绑,结果已丢弃。"})
result = "已解绑;重新绑定后请更新本机绑定信息。" result = "已解绑;重新绑定后请更新本机绑定信息。"
elif text == "/主机名" or text.startswith("/回显 "): elif text == "/主机名" or text.startswith("/回显 "):
result = (await broker.dispatch(owner, text, "msg_" + hashlib.sha256(message_id.encode()).hexdigest()))["result"] result = (await broker.dispatch(owner, text, "msg_" + hashlib.sha256(message_id.encode()).hexdigest(),
**({"chat_id":message["chat_id"],"source_message_id":message_id} if lifecycle else {})))["result"]
else: else:
instruction = text if text.startswith("/助手 ") else "/助手 " + text instruction = text if text.startswith("/助手 ") else "/助手 " + text
result = (await broker.dispatch(owner, instruction, "msg_" + hashlib.sha256(message_id.encode()).hexdigest()))["result"] result = (await broker.dispatch(owner, instruction, "msg_" + hashlib.sha256(message_id.encode()).hexdigest(),
await f.reply(message_id, result[:s.max_result]) **({"chat_id":message["chat_id"],"source_message_id":message_id} if lifecycle else {})))["result"]
turn = "msg_"+hashlib.sha256(message_id.encode()).hexdigest()
task = store.lifecycle.task(turn) if lifecycle else None
if task and not store.lifecycle.claim_reply(turn,owner,message_id):
store.finish(event_turn,"result_discarded")
return
try:
await f.reply(message_id, result[:s.max_result])
except Exception:
if task:
store.lifecycle.transition(turn,"replying","reply_unknown")
raise
if task:
store.lifecycle.transition(turn,"replying","replied")
store.finish(event_turn, "replied") store.finish(event_turn, "replied")
except Exception: except Exception:
store.audit(owner, "event", message_id, "failed_or_unknown") store.audit(owner, "event", message_id, "failed_or_unknown")
+104
View File
@@ -0,0 +1,104 @@
"""Broker-mode invalidation outbox, worker liveness and internal admission."""
import json
import time
from fastapi import HTTPException
from pydantic import BaseModel, ConfigDict, Field
from .router import compare_delete
class AdmissionRequest(BaseModel):
model_config = ConfigDict(extra="forbid")
nonce: str = Field(pattern=r"^[a-f0-9]{32}$")
turn_id: str | None = Field(default=None, pattern=r"^[A-Za-z0-9_-]{1,128}$")
session_id: str | None = Field(default=None, min_length=1, max_length=64)
class DeviceLifecycle:
def __init__(self, state, redis, broker):
self.state, self.r, self.broker = state, redis, broker
async def flush(self):
rows = self.state.db.execute("""SELECT * FROM work_binding_events_v2
WHERE environment=? AND app_id=? AND delivered=0 ORDER BY created_at LIMIT 100""",
(self.state.environment,self.state.app_id)).fetchall()
for event in rows:
try:
device = self.state.store.device(event["host_id"])
if device and device["revoked"]:
await self.r.set("hn:revoked:"+event["host_id"], "1")
key = "hn:online:"+event["host_id"]
presence = await self.r.get(key)
if presence:
try:
generation = json.loads(presence)["proof"]["binding_generation"]
if type(generation) is not int:
generation = -1
except (ValueError, KeyError, TypeError):
generation = -1
if generation < event["binding_generation"]:
await compare_delete(self.r, key, presence)
# Newer sessions ignore a delayed invalidation for older generations.
await self.r.publish("hn:kill:"+event["host_id"], json.dumps({
"reason": event["reason"], "binding_generation": event["binding_generation"]}))
with self.state.db:
self.state.db.execute("UPDATE work_binding_events_v2 SET delivered=1 WHERE event_id=?",
(event["event_id"],))
except Exception:
# DB remains authoritative; leave durable work for another attempt.
return False
self.cancel_waiters()
return not self.state.db.execute("""SELECT event_id FROM work_binding_events_v2
WHERE environment=? AND app_id=? AND delivered=0 LIMIT 1""",
(self.state.environment,self.state.app_id)).fetchone()
def cancel_waiters(self):
for turn, pending in list(self.broker.pending.items()):
task = self.state.task(turn)
if task and task["status"] not in {"inflight", "completed", "replying"} and not pending.future.done():
pending.future.set_result({"status":task["status"], "turn_id":turn,
"result":"任务已结束,旧结果不会继续投递。"})
async def maintain(self):
await self.r.set("hn:task-worker:"+self.broker.worker_id,"alive",ex=30)
await self.flush()
rows = self.state.db.execute("""SELECT * FROM work_tasks_v2
WHERE environment=? AND app_id=? AND status IN ('inflight','completed','replying')
ORDER BY updated_at LIMIT 100""", (self.state.environment,self.state.app_id)).fetchall()
for task in rows:
if task["deadline"] <= time.time() or not await self.r.exists("hn:task-worker:"+task["worker_id"]):
self.state.transition(task["turn_id"],task["status"],
"reply_unknown" if task["status"] == "replying" else "unknown")
else:
# Rotate bounded scanning instead of starving later tasks behind
# the same first 100 healthy workers on every maintenance pass.
with self.state.db:
self.state.db.execute("UPDATE work_tasks_v2 SET updated_at=? WHERE turn_id=? AND status=?",
(time.time(),task["turn_id"],task["status"]))
self.cancel_waiters()
async def admission(self, host, body):
device = self.state.store.device(host)
if not device or device["revoked"] or not device["owner"]:
raise HTTPException(403,"device_unbound_or_revoked")
task = None
if body.turn_id:
task = self.state.task(body.turn_id)
if (not self.state.task_current(task) or task["host_id"] != host
or task["status"] != "inflight" or task["session_id"] != body.session_id
or not await self.r.exists("hn:task-worker:"+task["worker_id"])):
raise HTTPException(403,"task_not_active")
# Re-read after Redis await; never rely on router's cached admission.
latest = self.state.store.device(host)
if (not latest or latest["revoked"] or latest["owner"] != device["owner"]
or latest["binding_generation"] != device["binding_generation"]):
raise HTTPException(403,"stale_binding")
if task:
current = self.state.task(body.turn_id)
if not self.state.task_current(current) or current["status"] != "inflight":
raise HTTPException(403,"task_not_active")
return {"protocol_version":2,"environment":self.state.environment,"app_id":self.state.app_id,
"host_id":host,"owner":latest["owner"],"binding_generation":latest["binding_generation"],
"fingerprint":latest["fingerprint"],"nonce":body.nonce,"turn_id":body.turn_id,
"session_id":body.session_id,"expires_at":time.time()+5}
+9 -1
View File
@@ -21,6 +21,7 @@ class Notification(BaseModel):
proof: dict proof: dict
request_id: str = Field(pattern=r'^[A-Za-z0-9_-]{8,100}$') request_id: str = Field(pattern=r'^[A-Za-z0-9_-]{8,100}$')
binding_owner: str = Field(min_length=1, max_length=128) binding_owner: str = Field(min_length=1, max_length=128)
binding_generation: int | None = Field(default=None,strict=True,ge=0)
kind: str = Field(pattern=r'^(text|notice|authorization)$') kind: str = Field(pattern=r'^(text|notice|authorization)$')
title: str = Field(default='', max_length=80) title: str = Field(default='', max_length=80)
text: str = Field(default='', max_length=2000) text: str = Field(default='', max_length=2000)
@@ -41,10 +42,16 @@ def router(settings, redis, store, platform, oauth):
if not settings.feishu_enabled: if not settings.feishu_enabled:
raise HTTPException(503, 'platform disabled') raise HTTPException(503, 'platform disabled')
payload = body.model_dump(exclude={'host_id', 'proof'}) payload = body.model_dump(exclude={'host_id', 'proof'})
if body.binding_generation is None:
payload.pop('binding_generation') # Preserve exact legacy proof contract.
owner = await device_owner(settings, redis, store, body.host_id, body.proof, owner = await device_owner(settings, redis, store, body.host_id, body.proof,
'work-notify-v1', payload) 'work-notify-v1', payload)
if body.binding_owner != owner: if body.binding_owner != owner:
raise HTTPException(403, 'local binding is stale; pair the client again') raise HTTPException(403, 'local binding is stale; pair the client again')
if settings.credential_mode == 'broker':
current = store.device(body.host_id)
if body.binding_generation is None or current['binding_generation'] != body.binding_generation:
raise HTTPException(403,'stale_binding')
digest = hashlib.sha256(canonical(payload)).hexdigest() digest = hashlib.sha256(canonical(payload)).hexdigest()
previous = store.db.execute('SELECT * FROM notifications WHERE host_id=? AND request_id=?', previous = store.db.execute('SELECT * FROM notifications WHERE host_id=? AND request_id=?',
(body.host_id, body.request_id)).fetchone() (body.host_id, body.request_id)).fetchone()
@@ -106,7 +113,8 @@ def router(settings, redis, store, platform, oauth):
# Recheck the transaction and generation after waiting on app credentials. # Recheck the transaction and generation after waiting on app credentials.
oauth.authorization_url_for_notification(owner, body.host_id, body.oauth_state) oauth.authorization_url_for_notification(owner, body.host_id, body.oauth_state)
device = store.device(body.host_id) device = store.device(body.host_id)
if not device or device['revoked'] or device['owner'] != owner: if (not device or device['revoked'] or device['owner'] != owner
or (settings.credential_mode == 'broker' and device['binding_generation'] != body.binding_generation)):
status = 'binding_changed' status = 'binding_changed'
else: else:
delivery_id = uuid.uuid5(uuid.NAMESPACE_URL, body.host_id + ':' + body.request_id).hex delivery_id = uuid.uuid5(uuid.NAMESPACE_URL, body.host_id + ':' + body.request_id).hex
+58 -5
View File
@@ -3,6 +3,8 @@ import contextlib
import json import json
import secrets import secrets
import time import time
import hashlib
import httpx
from contextlib import asynccontextmanager from contextlib import asynccontextmanager
from fastapi import FastAPI, WebSocket, WebSocketDisconnect from fastapi import FastAPI, WebSocket, WebSocketDisconnect
@@ -10,7 +12,7 @@ from redis.asyncio import Redis
from redis.exceptions import WatchError from redis.exceptions import WatchError
from .config import Settings from .config import Settings
from .crypto import fingerprint, raw_public, verified, verify_device from .crypto import canonical, fingerprint, load_public, raw_public, verified, verify_device
async def compare_delete(redis, key, expected): async def compare_delete(redis, key, expected):
@@ -41,16 +43,38 @@ async def compare_expire(redis, key, expected, ttl):
return False return False
def create_app(settings=None, redis=None): def create_app(settings=None, redis=None, admission_transport=None):
s = settings or Settings(role="router") s = settings or Settings(role="router")
r = redis or Redis.from_url(s.redis_url, decode_responses=True) r = redis or Redis.from_url(s.redis_url, decode_responses=True)
ca = s.ca_cert.read_bytes() ca = s.ca_cert.read_bytes()
active = set() active = set()
internal = (httpx.AsyncClient(base_url=s.connector_internal_url.rstrip('/'),timeout=3,
trust_env=False,follow_redirects=False,transport=admission_transport) if s.credential_mode == "broker" else None)
connector_public = load_public(s.connector_public_key) if internal else None
async def admission(host, owner, generation, *, turn=None, session=None, expected_fp=None):
nonce = secrets.token_hex(16)
response = await internal.post("/internal/devices/"+host+"/admission",
headers={"X-Router-Admission-Secret":s.router_admission_secret},
json={"nonce":nonce,"turn_id":turn,"session_id":session})
if response.status_code != 200:
raise ValueError("device admission rejected")
data = verified(response.json(),connector_public)
if (data["nonce"] != nonce or data["host_id"] != host or data["owner"] != owner
or data["binding_generation"] != generation or data["environment"] != s.environment
or data["app_id"] != s.feishu_app_id or data["protocol_version"] != 2
or data["turn_id"] != turn or data["session_id"] != session
or not time.time() <= data["expires_at"] <= time.time()+10
or data["fingerprint"] != expected_fp):
raise ValueError("stale device admission")
return data
@asynccontextmanager @asynccontextmanager
async def lifespan(app): async def lifespan(app):
await r.ping() await r.ping()
yield yield
if internal:
await internal.aclose()
if redis is None: if redis is None:
await r.aclose() await r.aclose()
@@ -101,6 +125,14 @@ def create_app(settings=None, redis=None):
fp = fingerprint(hello["cert"]) fp = fingerprint(hello["cert"])
if await r.get("hn:device:" + host) != fp or await r.exists("hn:revoked:" + host): if await r.get("hn:device:" + host) != fp or await r.exists("hn:revoked:" + host):
raise ValueError("unregistered/revoked") raise ValueError("unregistered/revoked")
generation = owner = None
if internal:
generation, owner = proof["binding_generation"], proof["binding_owner"]
if (type(generation) is not int or generation < 0 or type(proof.get("protocol_version")) is not int
or proof.get("protocol_version") != 2
or proof.get("environment") != s.environment or proof.get("app_id") != s.feishu_app_id):
raise ValueError("invalid binding handshake")
await admission(host,owner,generation,expected_fp=fp)
session = proof["session_id"] session = proof["session_id"]
presence = json.dumps({"cert": hello["cert"], "proof": hello["proof"], "challenge": challenge}) presence = json.dumps({"cert": hello["cert"], "proof": hello["proof"], "challenge": challenge})
if not await r.set("hn:online:" + host, presence, nx=True, ex=30): if not await r.set("hn:online:" + host, presence, nx=True, ex=30):
@@ -108,6 +140,8 @@ def create_app(settings=None, redis=None):
active.add(session) active.add(session)
pub = r.pubsub() pub = r.pubsub()
await pub.subscribe("hn:down:" + host, "hn:kill:" + host) await pub.subscribe("hn:down:" + host, "hn:kill:" + host)
if internal:
await admission(host,owner,generation,expected_fp=fp)
await ws.send_json({"type": "connected", "session_id": session}) await ws.send_json({"type": "connected", "session_id": session})
async def downstream(): async def downstream():
@@ -115,13 +149,28 @@ def create_app(settings=None, redis=None):
if msg["type"] != "message": if msg["type"] != "message":
continue continue
if msg["channel"] == "hn:kill:" + host: if msg["channel"] == "hn:kill:" + host:
if internal:
event = json.loads(msg["data"])
if generation >= event["binding_generation"]:
continue
await ws.close(1008) await ws.close(1008)
return return
if internal:
packet = json.loads(msg["data"])
routing = verified(packet["routing"],connector_public)
if (routing["host_id"] != host or routing["session_id"] != session
or routing["environment"] != s.environment or routing["app_id"] != s.feishu_app_id
or routing["binding_generation"] != generation
or routing["envelope_hash"] != hashlib.sha256(canonical(packet["envelope"])).hexdigest()):
raise ValueError("invalid instruction routing")
await admission(host,owner,generation,turn=routing["turn_id"],session=session,expected_fp=fp)
await ws.send_text(msg["data"]) await ws.send_text(msg["data"])
async def heartbeat(): async def heartbeat():
while True: while True:
await asyncio.sleep(8) await asyncio.sleep(8)
if internal:
await admission(host,owner,generation,expected_fp=fp)
if await r.exists("hn:revoked:" + host) or not await compare_expire(r, "hn:online:" + host, presence, 30): if await r.exists("hn:revoked:" + host) or not await compare_expire(r, "hn:online:" + host, presence, 30):
await ws.close(1008) await ws.close(1008)
return return
@@ -142,6 +191,8 @@ def create_app(settings=None, redis=None):
verified(envelope, public) verified(envelope, public)
if envelope["host_id"] != host or envelope["session_id"] != session: if envelope["host_id"] != host or envelope["session_id"] != session:
raise ValueError("wrong reply destination") raise ValueError("wrong reply destination")
if internal:
await admission(host,owner,generation,turn=turn,session=session,expected_fp=fp)
await r.publish("hn:replies", json.dumps(data)) await r.publish("hn:replies", json.dumps(data))
tasks = [asyncio.create_task(x()) for x in (downstream, heartbeat, upstream)] tasks = [asyncio.create_task(x()) for x in (downstream, heartbeat, upstream)]
@@ -155,13 +206,15 @@ def create_app(settings=None, redis=None):
with contextlib.suppress(Exception): with contextlib.suppress(Exception):
await ws.close(1011) await ws.close(1011)
finally: finally:
active.discard(session)
for task in tasks: for task in tasks:
task.cancel() task.cancel()
await asyncio.gather(*tasks, return_exceptions=True) await asyncio.gather(*tasks, return_exceptions=True)
if pub: if pub:
await pub.aclose() with contextlib.suppress(Exception):
await pub.aclose()
if presence: if presence:
await compare_delete(r, "hn:online:" + host, presence) with contextlib.suppress(Exception):
active.discard(session) await compare_delete(r, "hn:online:" + host, presence)
return app return app
+7
View File
@@ -32,6 +32,7 @@ class Postgres:
class Store: class Store:
def __init__(self, path, database_url=""): def __init__(self, path, database_url=""):
self.lifecycle = None
path.parent.mkdir(parents=True, exist_ok=True) path.parent.mkdir(parents=True, exist_ok=True)
schema = """ schema = """
CREATE TABLE IF NOT EXISTS devices( CREATE TABLE IF NOT EXISTS devices(
@@ -71,6 +72,8 @@ class Store:
self.db.execute("INSERT INTO devices(host_id,cert,fingerprint) VALUES(?,?,?)", (host, cert, fp)) self.db.execute("INSERT INTO devices(host_id,cert,fingerprint) VALUES(?,?,?)", (host, cert, fp))
def bind(self, host, owner): def bind(self, host, owner):
if self.lifecycle:
return self.lifecycle.bind(host, owner)
with self.db: with self.db:
# Conditional update also protects against two connectors racing for the same device. # Conditional update also protects against two connectors racing for the same device.
count = self.db.execute("UPDATE devices SET owner=?,binding_generation=binding_generation+1 WHERE host_id=? AND owner IS NULL AND revoked=0", (owner, host)).rowcount count = self.db.execute("UPDATE devices SET owner=?,binding_generation=binding_generation+1 WHERE host_id=? AND owner IS NULL AND revoked=0", (owner, host)).rowcount
@@ -78,10 +81,14 @@ class Store:
raise ValueError("device unavailable or already bound") raise ValueError("device unavailable or already bound")
def unbind(self, owner): def unbind(self, owner):
if self.lifecycle:
return self.lifecycle.unbind(owner)
with self.db: with self.db:
self.db.execute("UPDATE devices SET owner=NULL,binding_generation=binding_generation+1 WHERE owner=?", (owner,)) self.db.execute("UPDATE devices SET owner=NULL,binding_generation=binding_generation+1 WHERE owner=?", (owner,))
def revoke(self, host): def revoke(self, host):
if self.lifecycle:
return self.lifecycle.invalidate_host(host, "revoke")
with self.db: with self.db:
self.db.execute("UPDATE devices SET revoked=1,binding_generation=binding_generation+1 WHERE host_id=? AND revoked=0", (host,)) self.db.execute("UPDATE devices SET revoked=1,binding_generation=binding_generation+1 WHERE host_id=? AND revoked=0", (host,))
+16 -1
View File
@@ -35,6 +35,7 @@ class Platform:
self.resume = None self.resume = None
self.scopes = "read" self.scopes = "read"
self.sent = [] self.sent = []
self.replies = []
self.user_info_failure = False self.user_info_failure = False
async def credential_exchange(self, code): async def credential_exchange(self, code):
@@ -61,6 +62,9 @@ class Platform:
async def tenant_token(self): async def tenant_token(self):
return "fake-tenant" return "fake-tenant"
async def reply(self, message_id, text):
self.replies.append((message_id,text))
async def send(self, owner, kind, content, delivery_id, *, token): async def send(self, owner, kind, content, delivery_id, *, token):
self.sent.append((owner, content)) self.sent.append((owner, content))
return {"message_id": "om_fake"} return {"message_id": "om_fake"}
@@ -80,6 +84,7 @@ async def env(tmp_path):
keyring.write_text(json.dumps({"active_key_id": "test-key", "keys": { keyring.write_text(json.dumps({"active_key_id": "test-key", "keys": {
"test-key": base64.b64encode(os.urandom(32)).decode()}})) "test-key": base64.b64encode(os.urandom(32)).decode()}}))
settings = Settings(environment="test", admin_secret="a"*40, internal_secret="b"*40, settings = Settings(environment="test", admin_secret="a"*40, internal_secret="b"*40,
router_admission_secret="c"*40,
ca_cert=ca_path, connector_key=connector_key, data_dir=tmp_path, feishu_enabled=True, ca_cert=ca_path, connector_key=connector_key, data_dir=tmp_path, feishu_enabled=True,
feishu_app_id="cli_test", feishu_app_secret="fake-app-secret", credential_mode="broker", feishu_app_id="cli_test", feishu_app_secret="fake-app-secret", credential_mode="broker",
credential_keyring_file=keyring, credential_permission_profiles={"product-v1": ["read"], "product-v2": ["read", "write"]}, credential_keyring_file=keyring, credential_permission_profiles={"product-v1": ["read"], "product-v2": ["read", "write"]},
@@ -317,7 +322,8 @@ async def test_authorization_notification_uses_server_owned_transaction(env):
item = await start(env) item = await start(env)
host = "a"*32 host = "a"*32
fields = dict(request_id="notify-test-001", kind="authorization", text="", title="", fields = dict(request_id="notify-test-001", kind="authorization", text="", title="",
oauth_state=item["transaction_id"], binding_owner="ou_a") oauth_state=item["transaction_id"], binding_owner="ou_a",
binding_generation=env["store"].device(host)["binding_generation"])
proof = sign(dict(host_id=host, purpose="work-notify-v1", ts=int(time.time()), nonce=secrets.token_hex(16), proof = sign(dict(host_id=host, purpose="work-notify-v1", ts=int(time.time()), nonce=secrets.token_hex(16),
payload_hash=hashlib.sha256(canonical(fields)).hexdigest()), env["keys"][host]) payload_hash=hashlib.sha256(canonical(fields)).hexdigest()), env["keys"][host])
response = await env["client"].post("/device/notifications", json={**fields, "host_id": host, "proof": proof}) response = await env["client"].post("/device/notifications", json={**fields, "host_id": host, "proof": proof})
@@ -466,6 +472,14 @@ async def test_postgres_authorization_and_binding_generation(env, tmp_path):
assert result.status_code == 200 and "fake-refresh" not in result.text assert result.status_code == 200 and "fake-refresh" not in result.text
app.state.store.unbind("ou_a") app.state.store.unbind("ou_a")
assert app.state.store.device("a"*32)["binding_generation"] == 2 assert app.state.store.device("a"*32)["binding_generation"] == 2
lifecycle = app.state.store.lifecycle
lifecycle.issue_code("ou_a","fake-pg-old-code")
app.state.store.unbind("ou_a")
with pytest.raises(LookupError):
lifecycle.consume_code("a"*32,"fake-pg-old-code")
lifecycle.issue_code("ou_a","fake-pg-new-code")
assert lifecycle.consume_code("a"*32,"fake-pg-new-code") == "ou_a"
assert app.state.store.db.execute("SELECT COUNT(*) AS n FROM work_binding_events_v2").fetchone()["n"] == 1
finally: finally:
if app: if app:
app.state.store.close() app.state.store.close()
@@ -475,6 +489,7 @@ async def test_postgres_authorization_and_binding_generation(env, tmp_path):
def test_settings_require_explicit_key_and_product_profiles(tmp_path): def test_settings_require_explicit_key_and_product_profiles(tmp_path):
common = dict(environment="test", admin_secret="a"*40, internal_secret="b"*40, credential_mode="broker", common = dict(environment="test", admin_secret="a"*40, internal_secret="b"*40, credential_mode="broker",
router_admission_secret="c"*40,
feishu_enabled=True, feishu_app_id="test-app", feishu_app_secret="fake-secret") feishu_enabled=True, feishu_app_id="test-app", feishu_app_secret="fake-secret")
with pytest.raises(ValueError): with pytest.raises(ValueError):
Settings(**common) Settings(**common)
+479
View File
@@ -0,0 +1,479 @@
"""Broker-mode invalidation and real loopback WS task routing (no real Work)."""
import asyncio
import hashlib
import json
import secrets
import socket
import time
import httpx
import pytest
import pytest_asyncio
import uvicorn
from cryptography.hazmat.primitives import serialization
from cryptography.hazmat.primitives.asymmetric import x25519
from websockets.asyncio.client import connect
from websockets.exceptions import ConnectionClosed
from test_credential_api import env, wire, start, authorize
from relay.crypto import canonical, load_private, load_public, open_envelope, raw_public, seal, sign
from relay.device_lifecycle import AdmissionRequest
from relay.router import create_app as router_app
@pytest_asyncio.fixture
async def routed(env, tmp_path):
public_path = tmp_path / "connector.pub"
public_path.write_bytes(load_private(env["settings"].connector_key).public_key().public_bytes(
serialization.Encoding.PEM,serialization.PublicFormat.SubjectPublicKeyInfo))
settings = env["settings"].model_copy(update={"role":"router","connector_public_key":public_path,
"connector_internal_url":"http://connector:8000"})
app = router_app(settings,env["redis"],admission_transport=httpx.ASGITransport(env["app"]))
sock = socket.socket()
sock.bind(("127.0.0.1",0))
sock.listen(128)
port = sock.getsockname()[1]
server = uvicorn.Server(uvicorn.Config(app,log_level="critical",access_log=False,lifespan="on"))
server_task = asyncio.create_task(server.serve(sockets=[sock]))
try:
for _ in range(200):
if server.started:
break
if server_task.done():
server_task.result()
await asyncio.sleep(0.01)
assert server.started
value = {**env,"url":f"ws://127.0.0.1:{port}/device/ws","public":load_public(public_path),"sessions":[],"router":app}
yield value
finally:
for ws in locals().get("value",{}).get("sessions",[]):
await ws.close()
server.should_exit = True
await asyncio.wait_for(server_task,5)
async def device(env, host="a"*32, *, generation=None, legacy=False):
ws = await connect(env["url"],proxy=None)
env["sessions"].append(ws)
challenge = json.loads(await ws.recv())["challenge"]
row = env["store"].device(host)
ephemeral = x25519.X25519PrivateKey.generate()
session = secrets.token_hex(16)
plain = {"host_id":host,"challenge":challenge,"session_id":session,
"kx":raw_public(ephemeral.public_key()),"ts":int(time.time())}
if not legacy:
plain.update(protocol_version=2,environment=env["settings"].environment,
app_id=env["settings"].feishu_app_id,binding_owner=row["owner"] or "ou_a",
binding_generation=row["binding_generation"] if generation is None else generation)
await ws.send(json.dumps({"host_id":host,"cert":row["cert"],"proof":sign(plain,env["keys"][host])}))
accepted = json.loads(await asyncio.wait_for(ws.recv(),3))
assert accepted["type"] == "connected"
return ws,session,ephemeral
async def instruction(env, connection, host="a"*32):
ws,session,ephemeral = connection
packet = json.loads(await asyncio.wait_for(ws.recv(),3))
payload = open_envelope(packet["envelope"],ephemeral,env["public"],host_id=host,session_id=session)
return payload
async def reply(env, connection, payload, text, host="a"*32):
ws,session,_ = connection
envelope = seal({"type":"result","turn_id":payload["turn_id"],"result":text},
payload["reply_key"],env["keys"][host],host_id=host,session_id=session)
await ws.send(json.dumps({"type":"reply","turn_id":payload["turn_id"],"envelope":envelope}))
async def wait_for_reply(platform, count):
for _ in range(200):
if len(platform.replies) >= count:
return
await asyncio.sleep(0.01)
assert len(platform.replies) >= count
def event(env, message_id, text, chat="chat-a", owner="ou_a"):
return {"header":{"event_type":"im.message.receive_v1","app_id":env["settings"].feishu_app_id},
"event":{"sender":{"sender_type":"user","sender_id":{"open_id":owner}},
"message":{"message_id":message_id,"chat_type":"p2p","chat_id":chat,"message_type":"text",
"content":json.dumps({"text":text})}}}
async def send_event(env, data):
response = await env["client"].post("/internal/events",json=data,
headers={"X-Internal-Secret":env["settings"].internal_secret})
assert response.status_code == 200
async def test_unbind_commits_and_defers_redis_side_effects(env,monkeypatch):
await authorize(env)
pending = await start(env)
with monkeypatch.context() as patch:
async def fail(*args,**kwargs):
raise ConnectionError("synthetic redis outage")
patch.setattr(env["redis"],"publish",fail)
response = await env["client"].post("/admin/devices/"+"a"*32+"/unbind",
headers={"X-Admin-Secret":env["settings"].admin_secret})
assert response.status_code == 200 and response.json()["propagation"] == "pending"
assert env["store"].device("a"*32)["owner"] is None
assert env["api"]._row(pending["transaction_id"])["status"] == "invalidated"
path = "/work/v2/credentials/access-token"
assert (await env["client"].post(path,json=wire(env,path,permission_profile="product-v1"))).status_code == 403
assert await env["app"].state.lifecycle.flush()
assert env["store"].db.execute("SELECT COUNT(*) AS n FROM work_binding_events_v2 WHERE delivered=0").fetchone()["n"] == 0
async def test_codes_are_one_time_and_pre_unbind_codes_stay_invalid_after_rebind(env):
state = env["store"].lifecycle
env["store"].unbind("ou_a")
state.issue_code("ou_a","fake-code-old")
env["store"].unbind("ou_a")
with pytest.raises(LookupError):
state.consume_code("a"*32,"fake-code-old")
state.issue_code("ou_a","fake-code-new")
assert state.consume_code("a"*32,"fake-code-new") == "ou_a"
with pytest.raises(LookupError):
state.consume_code("a"*32,"fake-code-new")
dump = "\n".join(env["store"].db.iterdump())
assert "fake-code-old" not in dump and "fake-code-new" not in dump
async def test_failed_database_invalidation_rolls_back_binding_and_outbox(env,monkeypatch):
state = env["store"].lifecycle
original = state._cancel
def fail(device,reason):
original(device,reason)
raise RuntimeError("synthetic transaction abort")
monkeypatch.setattr(state,"_cancel",fail)
with pytest.raises(RuntimeError):
env["store"].unbind("ou_a")
current = env["store"].device("a"*32)
assert current["owner"] == "ou_a" and current["binding_generation"] == 1
assert state.db.execute("SELECT COUNT(*) AS n FROM work_binding_events_v2").fetchone()["n"] == 0
async def test_unbind_outbox_is_persistent_and_reprocessed_by_second_connector(env):
from relay.connector import create_app
env["store"].unbind("ou_a")
second = create_app(env["settings"],env["redis"],env["platform"])
try:
assert second.state.store.device("a"*32)["owner"] is None
assert await second.state.lifecycle.flush()
assert second.state.store.db.execute("SELECT COUNT(*) AS n FROM work_binding_events_v2 WHERE delivered=0").fetchone()["n"] == 0
finally:
second.state.store.close()
async def test_new_online_generation_not_removed_by_old_outbox(env):
env["store"].unbind("ou_a")
env["store"].bind("a"*32,"ou_a")
generation = env["store"].device("a"*32)["binding_generation"]
presence = json.dumps({"proof":{"binding_generation":generation}})
await env["redis"].set("hn:online:"+"a"*32,presence)
assert await env["app"].state.lifecycle.flush()
assert await env["redis"].get("hn:online:"+"a"*32) == presence
def notification(env, *, generation=None, include_generation=True):
host = "a"*32
payload = {"request_id":secrets.token_hex(8),"kind":"text","text":"synthetic notice","title":"",
"oauth_state":"","binding_owner":"ou_a"}
if include_generation:
payload["binding_generation"] = env["store"].device(host)["binding_generation"] if generation is None else generation
proof = sign({"host_id":host,"purpose":"work-notify-v1","ts":int(time.time()),"nonce":secrets.token_hex(16),
"payload_hash":hashlib.sha256(canonical(payload)).hexdigest()},env["keys"][host])
return {**payload,"host_id":host,"proof":proof}
async def test_notifications_require_current_generation_in_broker_mode(env):
stale = notification(env)
env["store"].unbind("ou_a")
env["store"].bind("a"*32,"ou_a")
assert (await env["client"].post("/device/notifications",json=stale)).status_code == 403
assert (await env["client"].post("/device/notifications",json=notification(env,include_generation=False))).status_code == 403
result = await env["client"].post("/device/notifications",json=notification(env))
assert result.json()["status"] == "sent" and len(env["platform"].sent) == 1
async def test_notification_rebind_during_app_token_wait_does_not_send(env,monkeypatch):
async def rebind():
env["store"].unbind("ou_a")
env["store"].bind("a"*32,"ou_a")
return "fake-app-token"
monkeypatch.setattr(env["platform"],"tenant_token",rebind)
result = await env["client"].post("/device/notifications",json=notification(env))
assert result.json()["status"] == "binding_changed" and not env["platform"].sent
async def test_internal_admission_requires_secret_and_current_binding(env):
path = "/internal/devices/"+"a"*32+"/admission"
body = {"nonce":secrets.token_hex(16)}
assert (await env["client"].post(path,json=body)).status_code == 401
assert (await env["client"].post(path,json=body,
headers={"X-Internal-Secret":env["settings"].internal_secret})).status_code == 401
headers = {"X-Router-Admission-Secret":env["settings"].router_admission_secret}
assert (await env["client"].post(path,json=body,headers=headers)).status_code == 200
env["store"].unbind("ou_a")
assert (await env["client"].post(path,json=body,headers=headers)).status_code == 403
async def test_router_credential_cannot_inject_platform_events(env):
result = await env["client"].post("/internal/events",json=event(env,"om-injected","hello"),
headers={"X-Internal-Secret":env["settings"].router_admission_secret})
assert result.status_code == 401
async def test_device_can_only_unbind_its_current_authenticated_binding(env):
path = "/work/v2/device/unbind"
invalid = wire(env,path)
invalid["proof"] = {}
assert (await env["client"].post(path,json=invalid)).status_code == 403
result = await env["client"].post(path,json=wire(env,path))
assert result.status_code == 200 and result.json()["status"] == "unbound"
assert env["store"].device("a"*32)["owner"] is None
assert env["store"].device("b"*32)["owner"] == "ou_b"
async def test_authenticated_unbind_snapshot_cannot_clear_a_new_binding(env):
state = env["store"].lifecycle
old = env["store"].device("a"*32)
env["store"].unbind("ou_a")
env["store"].bind("a"*32,"ou_a")
with pytest.raises(ValueError):
state.invalidate_host("a"*32,"unbind",expected_owner="ou_a",expected_generation=old["binding_generation"])
assert env["store"].device("a"*32)["owner"] == "ou_a"
async def test_v2_router_rejects_old_handshake_and_unbound_reconnect(routed):
with pytest.raises(ConnectionClosed):
await device(routed,legacy=True)
routed["store"].unbind("ou_a")
with pytest.raises(ConnectionClosed):
await device(routed)
async def test_two_devices_reverse_results_match_own_tasks(routed):
a = await device(routed)
b = await device(routed,"b"*32)
broker = routed["app"].state.broker
ta = asyncio.create_task(broker.dispatch("ou_a","a","turn-a",chat_id="chat-a",source_message_id="om-a"))
tb = asyncio.create_task(broker.dispatch("ou_b","b","turn-b",chat_id="chat-b",source_message_id="om-b"))
pa,pb = await asyncio.gather(instruction(routed,a),instruction(routed,b,"b"*32))
assert pa["task_context"]["source_message_id"] == "om-a"
assert pb["task_context"]["owner"] == "ou_b"
assert pa["task_context"]["conversation_id"] != pb["task_context"]["conversation_id"]
await reply(routed,b,pb,"answer-b","b"*32)
await reply(routed,a,pa,"answer-a")
ra,rb = await asyncio.gather(ta,tb)
assert (ra["result"],rb["result"]) == ("answer-a","answer-b")
state = routed["store"].lifecycle
assert not state.claim_reply("turn-a","ou_a","om-b")
assert state.claim_reply("turn-a","ou_a","om-a")
assert not state.claim_reply("turn-a","ou_a","om-a")
async def test_same_owner_two_conversations_reply_to_original_message(routed):
connection = await device(routed)
await send_event(routed,event(routed,"om-first","first","chat-first"))
first = await instruction(routed,connection)
await send_event(routed,event(routed,"om-second","second","chat-second"))
second = await instruction(routed,connection)
assert first["task_context"]["conversation_id"] != second["task_context"]["conversation_id"]
await reply(routed,connection,second,"second-answer")
await reply(routed,connection,first,"first-answer")
await wait_for_reply(routed["platform"],2)
assert routed["platform"].replies == [("om-second","second-answer"),("om-first","first-answer")]
await send_event(routed,event(routed,"om-first","first","chat-first"))
await asyncio.sleep(0.05)
assert len(routed["platform"].replies) == 2
async def test_robot_unbind_discards_previous_result_and_closes_connection(routed):
connection = await device(routed)
await send_event(routed,event(routed,"om-old-task","slow"))
payload = await instruction(routed,connection)
await send_event(routed,event(routed,"om-unbind","/解绑"))
await wait_for_reply(routed["platform"],1)
assert routed["platform"].replies[0][0] == "om-unbind"
assert routed["store"].device("a"*32)["owner"] is None
assert routed["store"].lifecycle.task(payload["turn_id"])["status"] == "unbound"
with pytest.raises(ConnectionClosed):
await asyncio.wait_for(connection[0].recv(),2)
await asyncio.sleep(0.05)
assert all(message != "om-old-task" for message,_ in routed["platform"].replies)
async def test_revoke_has_durable_state_and_blocks_connection(routed):
connection = await device(routed)
response = await routed["client"].post("/admin/devices/"+"a"*32+"/revoke",
headers={"X-Admin-Secret":routed["settings"].admin_secret})
assert response.status_code == 200
assert routed["store"].device("a"*32)["revoked"] == 1
with pytest.raises(ConnectionClosed):
await asyncio.wait_for(connection[0].recv(),2)
async def test_delayed_kill_for_old_generation_does_not_disconnect_new_session(routed):
old = await device(routed)
routed["store"].unbind("ou_a")
await routed["app"].state.lifecycle.flush()
with pytest.raises(ConnectionClosed):
await asyncio.wait_for(old[0].recv(),2)
routed["store"].bind("a"*32,"ou_a")
fresh = await device(routed)
with routed["store"].db:
routed["store"].db.execute("UPDATE work_binding_events_v2 SET delivered=0")
await routed["app"].state.lifecycle.flush()
task = asyncio.create_task(routed["app"].state.broker.dispatch("ou_a","hello","fresh-task"))
packet = await instruction(routed,fresh)
await reply(routed,fresh,packet,"fresh-answer")
assert (await task)["result"] == "fresh-answer"
async def test_lost_disconnect_notification_still_rejects_old_reply(routed):
connection = await device(routed)
task = asyncio.create_task(routed["app"].state.broker.dispatch("ou_a","slow","lost-disconnect"))
packet = await instruction(routed,connection)
routed["store"].unbind("ou_a") # Deliberately do not flush PubSub notification.
await reply(routed,connection,packet,"must-not-arrive")
with pytest.raises(ConnectionClosed):
await asyncio.wait_for(connection[0].recv(),2)
routed["app"].state.lifecycle.cancel_waiters()
assert (await task)["status"] == "unbound"
async def test_idle_connection_rechecks_database_even_when_pubsub_flush_fails(routed,monkeypatch):
connection = await device(routed)
async def unavailable():
return False
monkeypatch.setattr(routed["app"].state.lifecycle,"flush",unavailable)
routed["store"].unbind("ou_a")
# No client cooperation or kill publication: router's 8-second control check
# must close the old session from authoritative database state.
with pytest.raises(ConnectionClosed):
await asyncio.wait_for(connection[0].recv(),12)
async def test_control_plane_outage_does_not_forward_new_task(routed,monkeypatch):
from fastapi import HTTPException
connection = await device(routed)
async def unavailable(*args,**kwargs):
raise HTTPException(503,"synthetic_control_outage")
monkeypatch.setattr(routed["app"].state.lifecycle,"admission",unavailable)
task = asyncio.create_task(routed["app"].state.broker.dispatch("ou_a","do not deliver","control-outage"))
try:
with pytest.raises(ConnectionClosed):
await asyncio.wait_for(connection[0].recv(),3)
finally:
task.cancel()
with pytest.raises(asyncio.CancelledError):
await task
assert routed["store"].lifecycle.task("control-outage")["status"] == "unknown"
async def test_dispatch_cleanup_releases_waiter_despite_database_failure(routed,monkeypatch):
connection = await device(routed)
broker = routed["app"].state.broker
task = asyncio.create_task(broker.dispatch("ou_a","pending","cleanup-task"))
await instruction(routed,connection)
def fail(*args,**kwargs):
raise OSError("synthetic cleanup database failure")
monkeypatch.setattr(routed["store"].lifecycle,"transition",fail)
task.cancel()
with pytest.raises(asyncio.CancelledError):
await task
assert "cleanup-task" not in broker.pending
assert not await routed["redis"].exists("hn:inflight:"+"a"*32+":cleanup-task")
async def test_router_releases_connection_slot_despite_redis_cleanup_failure(routed,monkeypatch):
import relay.router as module
connection = await device(routed)
async def fail(*args,**kwargs):
raise ConnectionError("synthetic cleanup Redis failure")
monkeypatch.setattr(module,"compare_delete",fail)
await connection[0].close()
async with httpx.AsyncClient(transport=httpx.ASGITransport(routed["router"]),base_url="http://router") as client:
for _ in range(100):
result = await client.get("/ready")
if result.json()["connections"] == 0:
break
await asyncio.sleep(0.01)
assert result.json()["connections"] == 0
async def test_platform_reply_unknown_not_replayed_on_duplicate_event(routed,monkeypatch):
attempts = []
async def ambiguous(message,text):
attempts.append(message)
raise httpx.ReadTimeout("synthetic platform timeout")
monkeypatch.setattr(routed["platform"],"reply",ambiguous)
connection = await device(routed)
data = event(routed,"om-reply-unknown","hello")
await send_event(routed,data)
packet = await instruction(routed,connection)
await reply(routed,connection,packet,"answer")
for _ in range(100):
if routed["store"].lifecycle.task(packet["turn_id"])["status"] == "reply_unknown":
break
await asyncio.sleep(0.01)
assert routed["store"].lifecycle.task(packet["turn_id"])["status"] == "reply_unknown"
await send_event(routed,data)
await asyncio.sleep(0.05)
assert attempts == ["om-reply-unknown"]
async def test_binding_api_validates_signature_before_consuming_code(env):
state = env["store"].lifecycle
env["store"].unbind("ou_a")
state.issue_code("ou_a","fake-binding-once")
host = "a"*32
proof = sign({"host_id":host,"code":"fake-binding-once","ts":int(time.time())},env["keys"][host])
invalid = {**proof,"signature":"bad"}
assert (await env["client"].post("/device/bind",json={"host_id":host,"proof":invalid})).status_code == 403
response = await env["client"].post("/device/bind",json={"host_id":host,"proof":proof})
assert response.status_code == 200 and response.json()["binding_generation"] == 3
assert (await env["client"].post("/device/bind",json={"host_id":host,"proof":proof})).status_code == 404
async def test_dead_worker_tasks_unknown_and_never_replayed(env):
state = env["store"].lifecycle
row = env["store"].device("a"*32)
state.create_task("abandoned-task","ou_a","a"*32,row["binding_generation"],"session","dead-worker",time.time()+300,"chat","om-abandoned")
await env["app"].state.lifecycle.maintain()
assert state.task("abandoned-task")["status"] == "unknown"
assert not state.complete("abandoned-task","a"*32,"session","dead-worker")
assert not state.claim_reply("abandoned-task","ou_a","om-abandoned")
with pytest.raises(ValueError):
state.create_task("abandoned-task","ou_a","a"*32,row["binding_generation"],"new","new-worker",time.time()+300)
async def test_task_unknown_after_deadline_and_replying_after_crash(env):
state = env["store"].lifecycle
gen = env["store"].device("a"*32)["binding_generation"]
state.create_task("expired-task","ou_a","a"*32,gen,"s","missing",time.time()-1)
state.create_task("reply-crash","ou_a","a"*32,gen,"s","missing",time.time()+300)
state.transition("reply-crash","inflight","replying")
await env["app"].state.lifecycle.maintain()
assert state.task("expired-task")["status"] == "unknown"
assert state.task("reply-crash")["status"] == "reply_unknown"
async def test_wrong_session_and_old_generation_cannot_complete_task(env):
state = env["store"].lifecycle
gen = env["store"].device("a"*32)["binding_generation"]
state.create_task("old-generation","ou_a","a"*32,gen,"session","worker",time.time()+300)
assert not state.complete("old-generation","a"*32,"other-session","worker")
env["store"].unbind("ou_a")
env["store"].bind("a"*32,"ou_a")
assert not state.complete("old-generation","a"*32,"session","worker")
async def test_admission_denies_dead_tasks_without_exposing_credentials(env):
gen = env["store"].device("a"*32)["binding_generation"]
env["store"].lifecycle.create_task("task-check","ou_a","a"*32,gen,"s","dead",time.time()+300)
response = await env["client"].post("/internal/devices/"+"a"*32+"/admission",
headers={"X-Router-Admission-Secret":env["settings"].router_admission_secret},
json={"nonce":secrets.token_hex(16),"turn_id":"task-check","session_id":"s"})
assert response.status_code == 403 and "token" not in response.text.lower()