Queue-Runner: Same-Name Different-Content File Silently Dropped (2026-07-08)

Incident

邱律师 sent two versions of "2026年华新镇公立中小学生健康体检服务合同.docx" on the same day:

11:05 BJT (27889 bytes): generic version without fee cap
16:04 BJT (27482 bytes): specific version with 华新镇 in project name, ¥170,000 fee cap, different start date

Only the first was reviewed. The second was silently dropped.

Root Cause Chain (3 components)

1. auto_notify manifest dedup (`grep -qFx`)

# In auto_notify_new_file.sh:
if ! grep -qFx "$orig_name" "$QUEUE_DIR/manifest.txt" 2>/dev/null; then
    echo "$orig_name" >> "$QUEUE_DIR/manifest.txt"
fi

The filename was already in manifest.txt from the first file → second file NOT appended → manifest has only ONE entry for this filename.

2. Runner single-pass no-backtrack

The runner reads manifest top-to-bottom in one pass. By 08:00:09 UTC it had already passed the 华新镇 line (first version was in done/ → "SKIP not found"). When auto_notify wrote the second file to queue/ at 08:04, the runner was already past that line processing later files. It never goes back.

3. done/ presence satisfies watchdog progress check

[ -e "$QUEUE_DIR/done/$f" ] — the first version in done/ counts as "complete" for this manifest line.

Key Principle (Doro 2026-07-08 铁律)

判断是否为相同文件不能只看文件名。 必须检查合同实质内容：

顾问单位（甲方）名称
金额/费用上限
合同期限（起止日期）
项目内容描述
字节大小

以上任何一项不同 → 视为新版本/不同合同，正常入队审查。全部相同 → 视为重复发送，可跳过。

Fix Implemented (2026-07-09)

1. Content comparison script: `~/.hermes/scripts/contract_content_compare.py`

Usage: python3 contract_content_compare.py <file_a> <file_b>
Exit 0 = same content (duplicate)
Exit 1 = different content (new version / different contract)
Exit 2 = cannot read (treat as different, err on safe side)

Compares:

File size (bytes)
甲方 name (regex extraction from first 2000 chars)
All amounts (阿拉伯数字 ≥4 digits + 元/万, 人民币XXX, percentages)
All dates (YYYY年M月D日 format)
Project summary (first 500 chars normalized)

2. auto_notify_new_file.sh modification

Replaced the simple grep -qFx manifest dedup with content-level comparison:

if grep -qFx "$orig_name" "$QUEUE_DIR/manifest.txt" 2>/dev/null; then
    # Same filename exists in manifest — compare content
    EXISTING=""  # find in done/ or queue/
    COMPARE_RESULT=$(python3 contract_content_compare.py "$EXISTING" "$filepath")
    if [ $? -eq 0 ]; then
        # Content identical → true duplicate, skip
        log "SKIP duplicate (content identical): $orig_name"
        return
    else
        # Content different → new version, rename with timestamp suffix and queue
        RENAMED="${orig_name%.*}_v${TS}.${orig_name##*.}"
        cp "$filepath" "$QUEUE_DIR/$RENAMED"
        echo "$RENAMED" >> "$QUEUE_DIR/manifest.txt"
        log "SAME NAME DIFFERENT CONTENT — queued as new: $RENAMED"
    fi
else
    # Brand new filename, normal flow
    cp "$filepath" "$QUEUE_DIR/${orig_name}"
    echo "$orig_name" >> "$QUEUE_DIR/manifest.txt"
fi

3. Verification

Tested with the actual 华新镇 two files:

$ python3 contract_content_compare.py file1.docx file2.docx
DIFFERENT: 两份文件内容不同（新版本/不同合同）
  - 字节大小不同: 27889 vs 27482
  - 金额不同: A多set(), B多{'170000'}
  - 日期不同: A多{'2026年9月10日'}, B多{'2026年9月1日'}
  - 内容不同(第212字起): '...年公立中小学生健康检查...' vs '...年华新镇公立中小学生健康体检...'

Key Lesson

The initial diagnosis went through multiple wrong iterations:

First said "queue-runner only checks filename" — wrong (tracker guard also exists)
Then said "tracker guard is the one that skipped" — wrong (no tracker skip in logs)
Then said "整个链路没有内容比对" — wrong (Doro corrected: the system should have had it)

Correct diagnosis process: trace the actual log timestamps precisely, verify each claim with tool output, don't assume mechanisms exist or don't exist without reading the actual code.

Doro's requirement: "不能用文件名判断是否为相同文件，需要查看合同内容（包括顾问单位名称、金额、日期等）以及字节大小，要全面判断。" This is a capability requirement, not just a bug fix — the system must understand what makes two contracts "the same" at a business level.

4.6 KiB Raw Permalink Blame History