# Watchdog "suspended + delivered" Deadlock (2026-07-08)

## Symptom
- Doro reports not receiving delivery notification for completed contracts
- `tail watchdog.log` shows repeated lines every 20 minutes:
  ```
  BLOCK resume <thread_id>: tracker shows <filename> already delivered
  runner 退出但有 worker/卡住thread在处理，暂不重启（避免撞车）
  ```
- Files ARE in 任务交付/ directory (deliverer succeeded), but notification was never sent
- Queue runner has exited, watchdog won't restart it

## Root Cause Chain
1. Workflow reaches `final_review` step (responsible for quality check + notification)
2. `final_review` encounters HTTP 500 / API failure → thread suspends
3. But `deliverer` already succeeded earlier → tracker shows `status=delivered`
4. Queue-runner sees `status=suspended` (not `end`) → marks as "needs inspection", does NOT archive to done/
5. Queue-runner continues to next file, eventually exits
6. Watchdog checks: finds suspended thread → tries to resume → checks tracker → sees "already delivered" → BLOCK
7. File still in queue/ (not in done/) → watchdog thinks work remains → won't restart runner for other files
8. **Infinite loop**: every 20 min watchdog ticks, hits same BLOCK, same "暂不重启"

## Impact Scope (2026-07-08 batch)
All 5 contracts from the 16:00 batch were affected (all hit HTTP 500 at final_review):
- 【修】华新慢病支持中心运维合同(1)_2.docx
- 赵巷合同1.doc  
- 疾控中心（卫监所）实习人员宿舍改造工程.docx
- 2026年爱在党群·沟通有方家长沟通之道青春健康教育项目合作协议.docx
- 合同.docx

## Resolution (止血 SOP)

### Step 1: Cancel suspended threads
```bash
/home/maggie/.hermes/node/bin/uwf thread cancel <thread_id>
# Repeat for all suspended threads
```

### Step 2: Move stuck files to done/
```bash
cd /tmp/contract-queue
mv "filename1.docx" done/
mv "filename2.doc" done/
# Move ALL files that are in tracker as delivered/completed
```

### Step 3: Verify queue is clear
```bash
ls /tmp/contract-queue/*.doc* 2>/dev/null  # Should be empty
```

### Step 4: Do pass for delivered files
Since files are already in 任务交付/ and tracker shows delivered, proceed with normal pass flow (update tracker to completed + write xlsx).

## Prevention
- The watchdog should detect "suspended + already delivered" as a terminal state and auto-archive to done/ (currently it only BLOCKs resume without archiving)
- The final_review step should have retry logic for transient HTTP 500 errors
- Queue-runner should archive files to done/ even when thread status=suspended IF tracker shows delivered (the work is done, only notification failed)

## Diagnosis Commands
```bash
# Check for deadlock pattern
tail -20 /tmp/contract-queue/watchdog.log | grep "BLOCK resume"

# Check queue-runner status
ps aux | grep contract-queue-runner | grep -v grep

# Check what's stuck in queue
ls /tmp/contract-queue/*.doc* 2>/dev/null

# Check tracker for delivered status
python3 -c "
import json
t = json.load(open('$HOME/.hermes/data/contract-tracker.json'))
for c in t['contracts']:
    if c.get('status') == 'delivered':
        print(f\"  {c['original_filename']} → delivered\")
"
```