Product problem
Receipt extraction is useful only when downstream systems can distinguish validated data from a model guess. Receipty treats the model as an untrusted extractor and makes uncertainty visible through typed outcomes instead of silently accepting malformed or unsupported fields.
Engineering approach
The FastAPI service sends schema-constrained requests, validates returned data with Pydantic, and applies deterministic locale-aware parsing. The same service path powers a labelled extraction harness. Indexed receipts can then be searched through keyword, dense, hybrid, or hybrid-rerank retrieval before the question-answering endpoint returns cited source IDs or a not-found response.
Retrieval results
Four strategies were scored on the 64 answerable questions (corpus of 55 receipts, k = 5):
| Strategy | recall@5 | MRR |
|---|---|---|
| Keyword | 1.00 | 0.99 |
| Dense (pgvector) | 1.00 | 0.99 |
| Hybrid | 1.00 | 0.99 |
| Hybrid + rerank | 1.00 | 0.99 |
All four strategies score the same, so this suite cannot show one strategy beating another. It confirms retrieval works on this corpus; harder queries are needed to separate the strategies.
Bounded agent
POST /v1/agent runs a LangGraph tool-calling loop over four application-owned tools. Reads run immediately. Writes pause with interrupt() until a human approves them through /v1/agent/resume. Step, time and token budgets stop a loop that keeps requesting tools, and a provider failure ends the run with the completed steps kept. The same four tools are exposed over an MCP stdio server. The HTTP agent enforces the approval pause; direct MCP tool calls do not inherit that approval workflow.
Observability and safety
Added trace logging for model requests, retrieval, and tool calls, including duration, token usage, and provider cost. Redaction handles supported patterns for sensitive data in logs and traces. Latency is p50 1.7 s and p95 3.4 s at about $0.0005 per receipt (60 images, $0.0298 in total). 178 tests and deterministic evaluation checks run in CI behind an 80% branch-coverage gate.
Validation and limits
The extraction suite contains 60 curated images: 55 receipts and five non-receipts. Receipt classification, total, and date checks passed on all 60 examples, but that result describes this limited dataset rather than universal accuracy. The 69-question retrieval suite contains 64 answerable questions and five not-found cases; recall and MRR apply only to the 64 questions with gold sources. Retrieval evaluation measures search quality, not answer correctness. Schema-valid output does not establish visual grounding, and returned answer citations are not constrained to the retrieved source set.


