feat: export core Hermes skills

2026-07-15 02:45:56 +00:00
parent a028b63eda
commit 54711fee2a
308 changed files with 41310 additions and 1 deletions
@@ -0,0 +1,64 @@
+# Notification Silent Failure — WeChat 846609 WebSocket Disconnection
+
+## Incident: 2026-07-09
+
+### Timeline
+- 01:27 UTC — Gateway errcode 846609 first appears (WebSocket not subscribed)
+- 01:28 — 职业卫生监督 workflow final_review completes, notification fails silently
+- 01:30 — WebSocket error attempt 6
+- 02:12 — 舜珙血压计 workflow final_review completes, notification fails silently
+- 02:15 — WebSocket error attempt 7
+- 02:18 — Doro sends "hi", WebSocket recovers (inbound works before outbound stabilizes)
+
+### Root Cause
+`final_review` calls `_send_wecom(extra, 'doro', msg)` in a subprocess. When the WeCom WebSocket is disconnected (846609), the send fails but:
+1. The uwf thread still ends successfully (status=end)
+2. The queue-runner marks the contract as `delivered` in tracker
+3. No retry mechanism exists for failed notifications
+4. No alarm fires for silent notification failures
+
+### Diagnostic Commands
+
+```bash
+# 1. Find delivered contracts that may have missed notifications
+python3 -c "
+import json
+t = json.load(open('$HOME/.hermes/data/contract-tracker.json'))
+for c in t['contracts']:
+    if c.get('status') == 'delivered':
+        print(f\"  {c['original_filename']} delivered_at={c.get('delivered_at','?')}\")
+"
+
+# 2. Check for 846609 errors in the time window
+grep '846609' ~/.hermes/logs/gateway.log | grep "$(date +%Y-%m-%d)"
+
+# 3. Check WebSocket disconnection periods
+grep 'WebSocket error\|websocket closed' ~/.hermes/logs/gateway.log | grep "$(date +%Y-%m-%d)"
+
+# 4. Verify if notification was actually sent (look for successful send around delivered_at)
+# A successful notification looks like:
+#   INFO gateway.platforms.base: [Wecom] Sending response (XXX chars) to doro
+# WITHOUT a subsequent 846609 error in the same second
+
+# 5. Check which threads completed during outage
+grep 'DONE.*status=end' /tmp/contract-queue/queue.log | grep "TIME_RANGE"
+```
+
+### Remediation
+```bash
+# Manually re-send notification for affected contracts
+python3 ~/.hermes/scripts/wecom_dm.py --to doro --text "合同审查完成通知（补发）:
+
+文件名: 【修】XXX.docx
+顾问单位: XXX
+修订摘要: X处插入、Y处删除
+主要修订: ...
+
+已上传至Nextcloud任务交付目录。
+（因企微连接中断未能实时送达，现补发）"
+```
+
+### Prevention (not yet implemented)
+- `final_review` should check send result and retry 3x with backoff
+- Queue-runner should distinguish "delivered + notified" from "delivered + notification failed"
+- Watchdog could audit: for each `delivered` record older than 30 min, verify gateway.log has a matching successful send