# PDF OCR Troubleshooting for Scanned Chinese Contracts

**Lesson learned**: 2026-06-30 — 通州金鹰 (23MB lease + 1.4MB property contract)

## Problem Pattern: Mixed OCR Results

Scanned PDFs of Chinese contracts often produce:
- Some pages: clean OCR text via `ocrmypdf` + `pdftotext`/`pymupdf`
- Other pages: completely empty output (0 chars)
- Other pages: garbled character soup (`\u0000` null bytes, random symbols)

## Diagnosis

```python
from PIL import Image
import numpy as np

for i in range(1, N+1):
    img = Image.open(f'pages/page-{i}.jpg').convert('L')
    arr = np.array(img)
    mean = arr.mean()  # ~160-170 for scanned contracts
    std = arr.std()    # ~30-37 for text-heavy pages
    # If tesseract returns 0 chars but mean/std look normal → needs preprocessing
```

## Fix 1: Binary Threshold Conversion

Standard tesseract OCR produces empty output on some scanned pages. The fix:

```python
from PIL import Image
import numpy as np

for i in empty_pages:
    img = Image.open(f'pages/page-{i}.jpg').convert('L')
    arr = np.array(img)
    threshold = 140  # Key value: too high = lose text, too low = noise
    arr = (arr < threshold).astype(np.uint8) * 255
    bw = Image.fromarray(arr)
    bw.save(f'pages/page-{i}_bw.png')
    # Then: tesseract pages/page-{i}_bw.png stdout -l chi_sim
```

**Why this works**: Some scanners produce pages where the background is very slightly off-white (mean ~168 vs pure white 255). Tesseract's adaptive thresholding fails on these. Hard binary threshold at 140 separates text (dark, <140) from background (>140) cleanly.

## Fix 2: Vision Service Timeout

`vision_analyze` times out on large images (>200KB at 300dpi). Always compress:

```python
from PIL import Image
img = Image.open(f'pages/page-{i}.jpg')
img = img.resize((img.width // 3, img.height // 3), Image.LANCZOS)
img.save(f'pages/page-{i}_small.jpg', quality=60)
# Target: 80-95KB per page at 850x1100 pixels
```

## Fix 3: ocrmypdf Text Layer Extraction Failure

Even after `ocrmypdf`, the embedded text layer may be garbled when extracted via `pdftotext` or `pymupdf`. This is a known issue with `ocrmypdf` + tesseract for Chinese text.

**Solution**: Don't rely on `ocrmypdf`'s text layer. Instead:
1. Use `ocrmypdf` to produce the OCR'd PDF (for archiving)
2. Separately extract pages as images: `pdftoppm -jpeg -r 300 input.pdf pages/page`
3. Run tesseract directly on each image
4. For empty pages, apply binary threshold (Fix 1) and retry

## Workflow: Complete OCR Pipeline

```bash
# 1. Extract pages as images
pdftoppm -jpeg -r 300 "contract.pdf" pages/page

# 2. Standard tesseract on all pages
for f in pages/page-*.jpg; do
    tesseract "$f" "${f%.jpg}" -l chi_sim 2>/dev/null
done

# 3. Identify empty pages (0 chars)
for f in pages/page-*.txt; do
    chars=$(wc -c < "$f")
    if [ "$chars" -lt 10 ]; then
        echo "EMPTY: $f"
    fi
done

# 4. Binary threshold + retry on empty pages (Python)
# 5. Vision for remaining empty pages (compress to <100KB first)
# 6. Concatenate all page texts into final OCR file
```

## Known OCR Garble Patterns in Chinese Contracts

| OCR Output | Likely Value | Context |
|---|---|---|
| "于65" / "也65" / "了芋65" | 765 | Area in ㎡ |
| "巧" / "葬" | 15 / 30 | Working days |
| "101" | 10 | Percentage |
| "0.1‰%o" | 0.1‰ | Daily penalty rate |
| "直通市赴" | 南通市通州区 | City name |
| "驳玉年" | 2025年 | Year |
| "瑞殉年" | 2025年 | Year |
| "嫂万元整" | 肆万元整 | Amount in Chinese |
| "103" | 10 | Working days |

**Rule**: Never guess OCR values. Mark as "⚠️待核实原件" in the Excel and list in 需客户核实 section.