# Notification Silent Failure — WeChat 846609 WebSocket Disconnection

## Incident: 2026-07-09

### Timeline
- 01:27 UTC — Gateway errcode 846609 first appears (WebSocket not subscribed)
- 01:28 — 职业卫生监督 workflow final_review completes, notification fails silently
- 01:30 — WebSocket error attempt 6
- 02:12 — 舜珙血压计 workflow final_review completes, notification fails silently
- 02:15 — WebSocket error attempt 7
- 02:18 — Doro sends "hi", WebSocket recovers (inbound works before outbound stabilizes)

### Root Cause
`final_review` calls `_send_wecom(extra, 'doro', msg)` in a subprocess. When the WeCom WebSocket is disconnected (846609), the send fails but:
1. The uwf thread still ends successfully (status=end)
2. The queue-runner marks the contract as `delivered` in tracker
3. No retry mechanism exists for failed notifications
4. No alarm fires for silent notification failures

### Diagnostic Commands

```bash
# 1. Find delivered contracts that may have missed notifications
python3 -c "
import json
t = json.load(open('$HOME/.hermes/data/contract-tracker.json'))
for c in t['contracts']:
    if c.get('status') == 'delivered':
        print(f\"  {c['original_filename']} delivered_at={c.get('delivered_at','?')}\")
"

# 2. Check for 846609 errors in the time window
grep '846609' ~/.hermes/logs/gateway.log | grep "$(date +%Y-%m-%d)"

# 3. Check WebSocket disconnection periods
grep 'WebSocket error\|websocket closed' ~/.hermes/logs/gateway.log | grep "$(date +%Y-%m-%d)"

# 4. Verify if notification was actually sent (look for successful send around delivered_at)
# A successful notification looks like:
#   INFO gateway.platforms.base: [Wecom] Sending response (XXX chars) to doro
# WITHOUT a subsequent 846609 error in the same second

# 5. Check which threads completed during outage
grep 'DONE.*status=end' /tmp/contract-queue/queue.log | grep "TIME_RANGE"
```

### Remediation
```bash
# Manually re-send notification for affected contracts
python3 ~/.hermes/scripts/wecom_dm.py --to doro --text "合同审查完成通知（补发）:

文件名: 【修】XXX.docx
顾问单位: XXX
修订摘要: X处插入、Y处删除
主要修订: ...

已上传至Nextcloud任务交付目录。
（因企微连接中断未能实时送达，现补发）"
```

### Prevention (not yet implemented)
- `final_review` should check send result and retry 3x with backoff
- Queue-runner should distinguish "delivered + notified" from "delivered + notification failed"
- Watchdog could audit: for each `delivered` record older than 30 min, verify gateway.log has a matching successful send