All work

Independent project · LLM systems

Receipty — Evaluated LLM Extraction & RAG

A FastAPI receipt pipeline that treats model output as untrusted data, measures extraction quality, and grounds answers in indexed source receipts.

Role
Creator & AI Application Engineer
Period
2026
Platforms
API · Backend
Receipty app artwork showing a structured receipt inside a blue frame
Contribution

Built the extraction, validation, retrieval, grounded-answer, evaluation, and CI paths end to end.

p95 3.4 s

extraction latency, ~$0.0005 per receipt

60/60

receipt, total, and date checks

69 queries

labelled retrieval suite

178 tests

in CI behind an 80% branch-coverage gate

Product problem

Receipt extraction is useful only when downstream systems can distinguish validated data from a model guess. Receipty treats the model as an untrusted extractor and makes uncertainty visible through typed outcomes instead of silently accepting malformed or unsupported fields.

Engineering approach

The FastAPI service sends schema-constrained requests, validates returned data with Pydantic, and applies deterministic locale-aware parsing. The same service path powers a labelled extraction harness. Indexed receipts can then be searched through keyword, dense, hybrid, or hybrid-rerank retrieval before the question-answering endpoint returns cited source IDs or a not-found response.

Retrieval results

Four strategies were scored on the 64 answerable questions (corpus of 55 receipts, k = 5):

Strategy recall@5 MRR
Keyword 1.00 0.99
Dense (pgvector) 1.00 0.99
Hybrid 1.00 0.99
Hybrid + rerank 1.00 0.99

All four strategies score the same, so this suite cannot show one strategy beating another. It confirms retrieval works on this corpus; harder queries are needed to separate the strategies.

Bounded agent

POST /v1/agent runs a LangGraph tool-calling loop over four application-owned tools. Reads run immediately. Writes pause with interrupt() until a human approves them through /v1/agent/resume. Step, time and token budgets stop a loop that keeps requesting tools, and a provider failure ends the run with the completed steps kept. The same four tools are exposed over an MCP stdio server. The HTTP agent enforces the approval pause; direct MCP tool calls do not inherit that approval workflow.

Observability and safety

Added trace logging for model requests, retrieval, and tool calls, including duration, token usage, and provider cost. Redaction handles supported patterns for sensitive data in logs and traces. Latency is p50 1.7 s and p95 3.4 s at about $0.0005 per receipt (60 images, $0.0298 in total). 178 tests and deterministic evaluation checks run in CI behind an 80% branch-coverage gate.

Validation and limits

The extraction suite contains 60 curated images: 55 receipts and five non-receipts. Receipt classification, total, and date checks passed on all 60 examples, but that result describes this limited dataset rather than universal accuracy. The 69-question retrieval suite contains 64 answerable questions and five not-found cases; recall and MRR apply only to the 64 questions with gold sources. Retrieval evaluation measures search quality, not answer correctness. Schema-valid output does not establish visual grounding, and returned answer citations are not constrained to the retrieved source set.

Engineering challenges

  1. Prevent a vision model from silently turning uncertain receipt fields into confident application data.
  2. Compare retrieval strategies on labelled questions without confusing search quality with answer quality.
  3. Keep evaluation results reproducible across prompt, model, code, latency, token, and cost changes.

Key decisions

  1. Constrained extraction with strict JSON schemas, Pydantic validation, locale-aware parsing, and explicit success, review, and failure outcomes.
  2. Combined keyword and pgvector retrieval with lexical reranking, source citations, and a deliberate not-found response.
  3. Recorded prompt and commit hashes with model settings and operational measurements, then enforced deterministic quality checks in CI.