LLM Receipt Parser
Vision-LLM extraction with human-in-the-loop review
A receipt-digitization app built around the idea that LLM extraction is only half the product: the other half is helping a human trust and correct it. Cross-provider model fallback, locale-aware prompt engineering, and a review UI with per-field provenance.
The Problem
Vision LLMs read receipts impressively, and wrongly, in ways users can't see. Naive 'photo in, JSON out' pipelines fail on ambiguous dates, merged line items and silent hallucinations.
The Solution
The backend sends photos through OpenRouter with a fallback chain (gpt-4o-mini → Gemini Flash → Qwen-VL), Zod-validates every response, and degrades to a manual-entry skeleton rather than an error page when all models fail. The prompt handles locale-aware date disambiguation (₹/GST/+91 signals DD/MM), enforces the items + tax + tip = total invariant, and returns calibrated confidence with per-field warnings. The review UI shows edited/new badges against the model's original extraction with live-recomputed totals.
Architecture
- React + Vite frontend; Express backend
- OpenRouter cross-provider fallback chain with Zod validation of every model response
- Deterministic-engine prompt design with a warnings-over-silence policy
- 18 vitest cases over the parsing/fallback layer
Challenges & How I Solved Them
Failing safely
Any model failure falls through the chain; total failure returns a low-confidence skeleton so users always land in manual entry, never on an error screen.
My Contributions
- Entire system, public on GitHub with a README defending every design decision
Impact
- Public, reviewable demonstration of production-grade LLM UX patterns
Lessons Learned
Ship the model's uncertainty to the user: confidence scores, warnings and provenance badges turn 'AI magic' into a tool people trust.