65 lines
2.6 KiB
Markdown
65 lines
2.6 KiB
Markdown
# Notification Silent Failure — WeChat 846609 WebSocket Disconnection
|
|
|
|
## Incident: 2026-07-09
|
|
|
|
### Timeline
|
|
- 01:27 UTC — Gateway errcode 846609 first appears (WebSocket not subscribed)
|
|
- 01:28 — 职业卫生监督 workflow final_review completes, notification fails silently
|
|
- 01:30 — WebSocket error attempt 6
|
|
- 02:12 — 舜珙血压计 workflow final_review completes, notification fails silently
|
|
- 02:15 — WebSocket error attempt 7
|
|
- 02:18 — Doro sends "hi", WebSocket recovers (inbound works before outbound stabilizes)
|
|
|
|
### Root Cause
|
|
`final_review` calls `_send_wecom(extra, 'doro', msg)` in a subprocess. When the WeCom WebSocket is disconnected (846609), the send fails but:
|
|
1. The uwf thread still ends successfully (status=end)
|
|
2. The queue-runner marks the contract as `delivered` in tracker
|
|
3. No retry mechanism exists for failed notifications
|
|
4. No alarm fires for silent notification failures
|
|
|
|
### Diagnostic Commands
|
|
|
|
```bash
|
|
# 1. Find delivered contracts that may have missed notifications
|
|
python3 -c "
|
|
import json
|
|
t = json.load(open('$HOME/.hermes/data/contract-tracker.json'))
|
|
for c in t['contracts']:
|
|
if c.get('status') == 'delivered':
|
|
print(f\" {c['original_filename']} delivered_at={c.get('delivered_at','?')}\")
|
|
"
|
|
|
|
# 2. Check for 846609 errors in the time window
|
|
grep '846609' ~/.hermes/logs/gateway.log | grep "$(date +%Y-%m-%d)"
|
|
|
|
# 3. Check WebSocket disconnection periods
|
|
grep 'WebSocket error\|websocket closed' ~/.hermes/logs/gateway.log | grep "$(date +%Y-%m-%d)"
|
|
|
|
# 4. Verify if notification was actually sent (look for successful send around delivered_at)
|
|
# A successful notification looks like:
|
|
# INFO gateway.platforms.base: [Wecom] Sending response (XXX chars) to doro
|
|
# WITHOUT a subsequent 846609 error in the same second
|
|
|
|
# 5. Check which threads completed during outage
|
|
grep 'DONE.*status=end' /tmp/contract-queue/queue.log | grep "TIME_RANGE"
|
|
```
|
|
|
|
### Remediation
|
|
```bash
|
|
# Manually re-send notification for affected contracts
|
|
python3 ~/.hermes/scripts/wecom_dm.py --to doro --text "合同审查完成通知（补发）:
|
|
|
|
文件名: 【修】XXX.docx
|
|
顾问单位: XXX
|
|
修订摘要: X处插入、Y处删除
|
|
主要修订: ...
|
|
|
|
已上传至Nextcloud任务交付目录。
|
|
（因企微连接中断未能实时送达，现补发）"
|
|
```
|
|
|
|
### Prevention (not yet implemented)
|
|
- `final_review` should check send result and retry 3x with backoff
|
|
- Queue-runner should distinguish "delivered + notified" from "delivered + notification failed"
|
|
- Watchdog could audit: for each `delivered` record older than 30 min, verify gateway.log has a matching successful send
|