feat: export core Hermes skills

2026-07-15 02:45:56 +00:00
parent a028b63eda
commit 54711fee2a
308 changed files with 41310 additions and 1 deletions
@@ -0,0 +1,106 @@
+# PDF OCR Troubleshooting for Scanned Chinese Contracts
+
+**Lesson learned**: 2026-06-30 — 通州金鹰 (23MB lease + 1.4MB property contract)
+
+## Problem Pattern: Mixed OCR Results
+
+Scanned PDFs of Chinese contracts often produce:
+- Some pages: clean OCR text via `ocrmypdf` + `pdftotext`/`pymupdf`
+- Other pages: completely empty output (0 chars)
+- Other pages: garbled character soup (`\u0000` null bytes, random symbols)
+
+## Diagnosis
+
+```python
+from PIL import Image
+import numpy as np
+
+for i in range(1, N+1):
+    img = Image.open(f'pages/page-{i}.jpg').convert('L')
+    arr = np.array(img)
+    mean = arr.mean()  # ~160-170 for scanned contracts
+    std = arr.std()    # ~30-37 for text-heavy pages
+    # If tesseract returns 0 chars but mean/std look normal → needs preprocessing
+```
+
+## Fix 1: Binary Threshold Conversion
+
+Standard tesseract OCR produces empty output on some scanned pages. The fix:
+
+```python
+from PIL import Image
+import numpy as np
+
+for i in empty_pages:
+    img = Image.open(f'pages/page-{i}.jpg').convert('L')
+    arr = np.array(img)
+    threshold = 140  # Key value: too high = lose text, too low = noise
+    arr = (arr < threshold).astype(np.uint8) * 255
+    bw = Image.fromarray(arr)
+    bw.save(f'pages/page-{i}_bw.png')
+    # Then: tesseract pages/page-{i}_bw.png stdout -l chi_sim
+```
+
+**Why this works**: Some scanners produce pages where the background is very slightly off-white (mean ~168 vs pure white 255). Tesseract's adaptive thresholding fails on these. Hard binary threshold at 140 separates text (dark, <140) from background (>140) cleanly.
+
+## Fix 2: Vision Service Timeout
+
+`vision_analyze` times out on large images (>200KB at 300dpi). Always compress:
+
+```python
+from PIL import Image
+img = Image.open(f'pages/page-{i}.jpg')
+img = img.resize((img.width // 3, img.height // 3), Image.LANCZOS)
+img.save(f'pages/page-{i}_small.jpg', quality=60)
+# Target: 80-95KB per page at 850x1100 pixels
+```
+
+## Fix 3: ocrmypdf Text Layer Extraction Failure
+
+Even after `ocrmypdf`, the embedded text layer may be garbled when extracted via `pdftotext` or `pymupdf`. This is a known issue with `ocrmypdf` + tesseract for Chinese text.
+
+**Solution**: Don't rely on `ocrmypdf`'s text layer. Instead:
+1. Use `ocrmypdf` to produce the OCR'd PDF (for archiving)
+2. Separately extract pages as images: `pdftoppm -jpeg -r 300 input.pdf pages/page`
+3. Run tesseract directly on each image
+4. For empty pages, apply binary threshold (Fix 1) and retry
+
+## Workflow: Complete OCR Pipeline
+
+```bash
+# 1. Extract pages as images
+pdftoppm -jpeg -r 300 "contract.pdf" pages/page
+
+# 2. Standard tesseract on all pages
+for f in pages/page-*.jpg; do
+    tesseract "$f" "${f%.jpg}" -l chi_sim 2>/dev/null
+done
+
+# 3. Identify empty pages (0 chars)
+for f in pages/page-*.txt; do
+    chars=$(wc -c < "$f")
+    if [ "$chars" -lt 10 ]; then
+        echo "EMPTY: $f"
+    fi
+done
+
+# 4. Binary threshold + retry on empty pages (Python)
+# 5. Vision for remaining empty pages (compress to <100KB first)
+# 6. Concatenate all page texts into final OCR file
+```
+
+## Known OCR Garble Patterns in Chinese Contracts
+
+| OCR Output | Likely Value | Context |
+|---|---|---|
+| "于65" / "也65" / "了芋65" | 765 | Area in ㎡ |
+| "巧" / "葬" | 15 / 30 | Working days |
+| "101" | 10 | Percentage |
+| "0.1‰%o" | 0.1‰ | Daily penalty rate |
+| "直通市赴" | 南通市通州区 | City name |
+| "驳玉年" | 2025年 | Year |
+| "瑞殉年" | 2025年 | Year |
+| "嫂万元整" | 肆万元整 | Amount in Chinese |
+| "103" | 10 | Working days |
+
+**Rule**: Never guess OCR values. Mark as "⚠️待核实原件" in the Excel and list in 需客户核实 section.