Watchdog "suspended + delivered" Deadlock (2026-07-08)

Symptom

Doro reports not receiving delivery notification for completed contracts

tail watchdog.log shows repeated lines every 20 minutes:

BLOCK resume <thread_id>: tracker shows <filename> already delivered
runner 退出但有 worker/卡住thread在处理，暂不重启（避免撞车）

Files ARE in 任务交付/ directory (deliverer succeeded), but notification was never sent
Queue runner has exited, watchdog won't restart it

Root Cause Chain

Workflow reaches final_review step (responsible for quality check + notification)
final_review encounters HTTP 500 / API failure → thread suspends
But deliverer already succeeded earlier → tracker shows status=delivered
Queue-runner sees status=suspended (not end) → marks as "needs inspection", does NOT archive to done/
Queue-runner continues to next file, eventually exits
Watchdog checks: finds suspended thread → tries to resume → checks tracker → sees "already delivered" → BLOCK
File still in queue/ (not in done/) → watchdog thinks work remains → won't restart runner for other files
Infinite loop: every 20 min watchdog ticks, hits same BLOCK, same "暂不重启"

Impact Scope (2026-07-08 batch)

All 5 contracts from the 16:00 batch were affected (all hit HTTP 500 at final_review):

【修】华新慢病支持中心运维合同(1)_2.docx
赵巷合同1.doc
疾控中心（卫监所）实习人员宿舍改造工程.docx
2026年爱在党群·沟通有方家长沟通之道青春健康教育项目合作协议.docx
合同.docx

Resolution (止血 SOP)

Step 1: Cancel suspended threads

/home/maggie/.hermes/node/bin/uwf thread cancel <thread_id>
# Repeat for all suspended threads

Step 2: Move stuck files to done/

cd /tmp/contract-queue
mv "filename1.docx" done/
mv "filename2.doc" done/
# Move ALL files that are in tracker as delivered/completed

Step 3: Verify queue is clear

ls /tmp/contract-queue/*.doc* 2>/dev/null  # Should be empty

Step 4: Do pass for delivered files

Since files are already in 任务交付/ and tracker shows delivered, proceed with normal pass flow (update tracker to completed + write xlsx).

Prevention

The watchdog should detect "suspended + already delivered" as a terminal state and auto-archive to done/ (currently it only BLOCKs resume without archiving)
The final_review step should have retry logic for transient HTTP 500 errors
Queue-runner should archive files to done/ even when thread status=suspended IF tracker shows delivered (the work is done, only notification failed)

Diagnosis Commands

# Check for deadlock pattern
tail -20 /tmp/contract-queue/watchdog.log | grep "BLOCK resume"

# Check queue-runner status
ps aux | grep contract-queue-runner | grep -v grep

# Check what's stuck in queue
ls /tmp/contract-queue/*.doc* 2>/dev/null

# Check tracker for delivered status
python3 -c "
import json
t = json.load(open('$HOME/.hermes/data/contract-tracker.json'))
for c in t['contracts']:
    if c.get('status') == 'delivered':
        print(f\"  {c['original_filename']} → delivered\")
"

3.2 KiB Raw Permalink Blame History