- Daily data entry, now review only
- 12h → 2–3hDaily data entry, now review only
- Orders billed identically to humans
- 71.7%Orders billed identically to humans
- Cost and latency per order
- $0.008 · 23sCost and latency per order
The challenge
Operators spent twelve hours a day reading handwritten waybills — bilingual Arabic and English, photographed on phones, in inconsistent layouts — and typing them into the platform field by field.
The obvious approach failed. A single prompt handling the whole label produced a subtle, dangerous error: correct digits assigned to the wrong field. The values looked plausible, so nothing downstream flagged them, and the field that drives delivery pricing was among the ones that drifted.
The approach
I re-architected it as five single-responsibility agents on a fixed LangGraph state graph — region localization, OCR, location resolution, amounts and verification — with parallel fan-out and a bounded verification/escalation loop.
The graph is deliberately fixed rather than a dynamic supervisor, so repeat runs of the same order stay comparable across versions. That comparability is what makes the evaluation harness meaningful.
Visual grounding happens before OCR: a localizer agent returns box coordinates only, then Python handles rotation, cropping and upscaling deterministically.
Location resolution uses constrained selection over free-text generation — the agent picks an ID from a retrieved candidate list rather than writing a destination name.
The result
A twelve-hour daily workload reduced to two or three hours of review, with operators checking rather than typing.
71.7% of orders billed identically to human entry, measured in shadow mode across 2,700+ orders scored field by field, with per-agent traces isolating whether an error came from reading or from resolution.
Cost and latency engineered down to $0.008 and 23 seconds per order through model routing — a fast model reads, a stronger model verifies. One targeted fix cut expensive verification escalations from 78% to 20% of orders.
Engineering notes
One agent per responsibility makes a class of bug impossible
Cross-field contamination wasn't fixed by better prompting — it was designed out. When no single agent can see two fields at once, it cannot swap them. Structural fixes beat probabilistic ones.
Constrained selection beats free-text generation
For location resolution the agent picks an ID from a retrieved candidate list. A hallucinated destination becomes impossible on the exact field that determines delivery pricing.
Visual grounding before OCR fixed an invisible failure
Downsampling made handwriting unreadable before the model ever saw it. Localizing first, then cropping and upscaling in Python, addressed a resolution problem no amount of prompt tuning would have touched.
A failure taxonomy is what makes automation trustworthy
Separating provider errors from bad model output means an outage marks entries retryable rather than storing empty predictions as successes — the difference between a bad hour and corrupted data.
Shadow mode is the only honest benchmark
Running predictions alongside human operators and scoring field-by-field gives a real number rather than a demo. LLM-as-judge verification, where a stronger model challenges each field and disagreements escalate to a wider crop, does the rest.
Built with
- Python (FastAPI)
- LangGraph
- Google Gemini (Flash + Pro)
- Pillow
- PostgreSQL + pgvector
- Laravel
- Docker