hermes-skills/skills/legal/contract-pass-workflow/references/notification-silent-failure-846609.md

Notification Silent Failure — WeChat 846609 WebSocket Disconnection

Incident: 2026-07-09

Timeline

• 01:27 UTC — Gateway errcode 846609 first appears (WebSocket not subscribed)
• 01:28 — 职业卫生监督 workflow final_review completes, notification fails silently
• 01:30 — WebSocket error attempt 6
• 02:12 — 舜珙血压计 workflow final_review completes, notification fails silently
• 02:15 — WebSocket error attempt 7
• 02:18 — Doro sends "hi", WebSocket recovers (inbound works before outbound stabilizes)

Root Cause

final_review calls _send_wecom(extra, 'doro', msg) in a subprocess. When the WeCom WebSocket is disconnected (846609), the send fails but:

1. The uwf thread still ends successfully (status=end)
2. The queue-runner marks the contract as delivered in tracker
3. No retry mechanism exists for failed notifications
4. No alarm fires for silent notification failures

Diagnostic Commands

# 1. Find delivered contracts that may have missed notifications
python3 -c "
import json
t = json.load(open('$HOME/.hermes/data/contract-tracker.json'))
for c in t['contracts']:
    if c.get('status') == 'delivered':
        print(f\"  {c['original_filename']} delivered_at={c.get('delivered_at','?')}\")
"

# 2. Check for 846609 errors in the time window
grep '846609' ~/.hermes/logs/gateway.log | grep "$(date +%Y-%m-%d)"

# 3. Check WebSocket disconnection periods
grep 'WebSocket error\|websocket closed' ~/.hermes/logs/gateway.log | grep "$(date +%Y-%m-%d)"

# 4. Verify if notification was actually sent (look for successful send around delivered_at)
# A successful notification looks like:
#   INFO gateway.platforms.base: [Wecom] Sending response (XXX chars) to doro
# WITHOUT a subsequent 846609 error in the same second

# 5. Check which threads completed during outage
grep 'DONE.*status=end' /tmp/contract-queue/queue.log | grep "TIME_RANGE"

Remediation

# Manually re-send notification for affected contracts
python3 ~/.hermes/scripts/wecom_dm.py --to doro --text "合同审查完成通知（补发）:

文件名: 【修】XXX.docx
顾问单位: XXX
修订摘要: X处插入、Y处删除
主要修订: ...

已上传至Nextcloud任务交付目录。
（因企微连接中断未能实时送达，现补发）"

Prevention (not yet implemented)

• final_review should check send result and retry 3x with backoff
• Queue-runner should distinguish "delivered + notified" from "delivered + notification failed"
• Watchdog could audit: for each delivered record older than 30 min, verify gateway.log has a matching successful send