111 lines
4.6 KiB
Markdown
111 lines
4.6 KiB
Markdown
# Queue-Runner: Same-Name Different-Content File Silently Dropped (2026-07-08)
|
|
|
|
## Incident
|
|
|
|
邱律师 sent two versions of "2026年华新镇公立中小学生健康体检服务合同.docx" on the same day:
|
|
- 11:05 BJT (27889 bytes): generic version without fee cap
|
|
- 16:04 BJT (27482 bytes): specific version with 华新镇 in project name, ¥170,000 fee cap, different start date
|
|
|
|
Only the first was reviewed. The second was silently dropped.
|
|
|
|
## Root Cause Chain (3 components)
|
|
|
|
### 1. auto_notify manifest dedup (`grep -qFx`)
|
|
|
|
```bash
|
|
# In auto_notify_new_file.sh:
|
|
if ! grep -qFx "$orig_name" "$QUEUE_DIR/manifest.txt" 2>/dev/null; then
|
|
echo "$orig_name" >> "$QUEUE_DIR/manifest.txt"
|
|
fi
|
|
```
|
|
|
|
The filename was already in manifest.txt from the first file → second file NOT appended → manifest has only ONE entry for this filename.
|
|
|
|
### 2. Runner single-pass no-backtrack
|
|
|
|
The runner reads manifest top-to-bottom in one pass. By 08:00:09 UTC it had already passed the 华新镇 line (first version was in done/ → "SKIP not found"). When auto_notify wrote the second file to queue/ at 08:04, the runner was already past that line processing later files. It never goes back.
|
|
|
|
### 3. done/ presence satisfies watchdog progress check
|
|
|
|
`[ -e "$QUEUE_DIR/done/$f" ]` — the first version in done/ counts as "complete" for this manifest line.
|
|
|
|
## Key Principle (Doro 2026-07-08 铁律)
|
|
|
|
**判断是否为相同文件不能只看文件名。** 必须检查合同实质内容：
|
|
- 顾问单位（甲方）名称
|
|
- 金额/费用上限
|
|
- 合同期限（起止日期）
|
|
- 项目内容描述
|
|
- 字节大小
|
|
|
|
以上任何一项不同 → 视为新版本/不同合同，正常入队审查。
|
|
全部相同 → 视为重复发送，可跳过。
|
|
|
|
## Fix Implemented (2026-07-09)
|
|
|
|
### 1. Content comparison script: `~/.hermes/scripts/contract_content_compare.py`
|
|
|
|
```
|
|
Usage: python3 contract_content_compare.py <file_a> <file_b>
|
|
Exit 0 = same content (duplicate)
|
|
Exit 1 = different content (new version / different contract)
|
|
Exit 2 = cannot read (treat as different, err on safe side)
|
|
```
|
|
|
|
Compares:
|
|
- File size (bytes)
|
|
- 甲方 name (regex extraction from first 2000 chars)
|
|
- All amounts (阿拉伯数字 ≥4 digits + 元/万, 人民币XXX, percentages)
|
|
- All dates (YYYY年M月D日 format)
|
|
- Project summary (first 500 chars normalized)
|
|
|
|
### 2. auto_notify_new_file.sh modification
|
|
|
|
Replaced the simple `grep -qFx` manifest dedup with content-level comparison:
|
|
|
|
```bash
|
|
if grep -qFx "$orig_name" "$QUEUE_DIR/manifest.txt" 2>/dev/null; then
|
|
# Same filename exists in manifest — compare content
|
|
EXISTING="" # find in done/ or queue/
|
|
COMPARE_RESULT=$(python3 contract_content_compare.py "$EXISTING" "$filepath")
|
|
if [ $? -eq 0 ]; then
|
|
# Content identical → true duplicate, skip
|
|
log "SKIP duplicate (content identical): $orig_name"
|
|
return
|
|
else
|
|
# Content different → new version, rename with timestamp suffix and queue
|
|
RENAMED="${orig_name%.*}_v${TS}.${orig_name##*.}"
|
|
cp "$filepath" "$QUEUE_DIR/$RENAMED"
|
|
echo "$RENAMED" >> "$QUEUE_DIR/manifest.txt"
|
|
log "SAME NAME DIFFERENT CONTENT — queued as new: $RENAMED"
|
|
fi
|
|
else
|
|
# Brand new filename, normal flow
|
|
cp "$filepath" "$QUEUE_DIR/${orig_name}"
|
|
echo "$orig_name" >> "$QUEUE_DIR/manifest.txt"
|
|
fi
|
|
```
|
|
|
|
### 3. Verification
|
|
|
|
Tested with the actual 华新镇 two files:
|
|
```
|
|
$ python3 contract_content_compare.py file1.docx file2.docx
|
|
DIFFERENT: 两份文件内容不同（新版本/不同合同）
|
|
- 字节大小不同: 27889 vs 27482
|
|
- 金额不同: A多set(), B多{'170000'}
|
|
- 日期不同: A多{'2026年9月10日'}, B多{'2026年9月1日'}
|
|
- 内容不同(第212字起): '...年公立中小学生健康检查...' vs '...年华新镇公立中小学生健康体检...'
|
|
```
|
|
|
|
## Key Lesson
|
|
|
|
The initial diagnosis went through multiple wrong iterations:
|
|
1. First said "queue-runner only checks filename" — wrong (tracker guard also exists)
|
|
2. Then said "tracker guard is the one that skipped" — wrong (no tracker skip in logs)
|
|
3. Then said "整个链路没有内容比对" — wrong (Doro corrected: the system should have had it)
|
|
|
|
**Correct diagnosis process**: trace the actual log timestamps precisely, verify each claim with tool output, don't assume mechanisms exist or don't exist without reading the actual code.
|
|
|
|
**Doro's requirement**: "不能用文件名判断是否为相同文件，需要查看合同内容（包括顾问单位名称、金额、日期等）以及字节大小，要全面判断。" This is a capability requirement, not just a bug fix — the system must understand what makes two contracts "the same" at a business level.
|