My AI Receipt Sorter Shifted Amounts by 2,000 Euros, So I Took the Numbers Away From It

I built a receipt sorter for a small business I run the infrastructure for. The idea is boring and useful: receipts come in by email and from a scanner, something figures out which company they belong to, gives them a sane file name, files them into a folder tree, and at the end of the month tells you which credit card transactions have no receipt yet.

The first version let a small, cheap LLM read everything, including the credit card statements. It was fast, cheap and confidently wrong. Amounts got shifted between lines. The reconciliation was off by up to about −2,000 €. On a document whose entire purpose is "the numbers are correct", that's not a rounding issue, that's useless.

TL;DR: Never let a language model be the source of truth for numbers you'll reconcile against. Card statements are now parsed deterministically, line by line, with pdfplumber, and only accepted if the parsed lines sum to the statement total. The LLM only reads individual receipts, and anything it claims (amount, date) must appear verbatim in the PDF's extracted text, or the receipt goes to a review folder.

The pipeline

mail inbox ──┐
             ├─> extract text ─> classify (company, type) ─> verify ─> file ─> match against statement
scan inbox ──┘                                                  │
                                                                └─> _review/  (anything unsure)

Filing layout on the storage box:

<Company>/<YYYY>/<MM>/<CreditCard|Invoices|Other>/<date>_<vendor>_<amount>EUR_<number>.pdf

• CreditCard: the statement itself, a reconciliation CSV next to it, and every receipt that's been matched to a card transaction. Receipts start in Invoices/ and move to CreditCard/ automatically once they're matched.
• Other: contracts, official letters, bank statements. Only the date is checked here, and there's no amount in the file name, because "the amount" of a contract is not a thing.
• _review/: anything the pipeline isn't sure about. A human looks at it.

The scan inbox is just a folder: files in its root get processed and moved to processed/<timestamp>_<name>, so nothing is processed twice and nothing disappears.

Why the LLM failed on statements

A receipt is one document with one total. A credit card statement is a table with dozens of rows, each with a date, a merchant, maybe a foreign currency amount, and a euro amount. When you hand that to a small model and ask for JSON, it has to keep rows aligned across a long, dense layout. Sometimes it didn't, and amounts ended up shifted against their rows.

The output still looked plausible. That's the dangerous part. A JSON array of transactions with real merchant names and real-looking amounts doesn't scream "wrong". Only the reconciliation did.

The lesson I took: an LLM is great at "what is this document and who is it for". It's the wrong tool for "reproduce these 40 numbers exactly", especially when the layout is regular enough that a parser can do it perfectly.

Deterministic statement parsing

Statements from one card issuer always have the same layout. That's a gift. pdfplumber gives you the text line by line, and a regex does the rest. Simplified (the real regex is tuned to one issuer's format):

import re
from decimal import Decimal
import pdfplumber

LINE = re.compile(
    r"^(?P<date>\d{2}\.\d{2}\.\d{2,4})\s+"
    r"(?P<text>.+?)\s+"
    r"(?P<amount>-?\d{1,3}(?:\.\d{3})*,\d{2})$"
)

def de_amount(s: str) -> Decimal:
    return Decimal(s.replace(".", "").replace(",", "."))

def parse_statement(path: str):
    rows, total = [], None
    with pdfplumber.open(path) as pdf:
        for page in pdf.pages:
            for line in (page.extract_text() or "").splitlines():
                line = line.strip()
                if line.startswith("New charges"):        # the statement's own total line
                    total = de_amount(line.split()[-1])
                    continue
                m = LINE.match(line)
                if m:
                    rows.append((m["date"], m["text"], de_amount(m["amount"])))
    return rows, total

rows, total = parse_statement("statement.pdf")
if total is None or sum(a for _, _, a in rows) != total:
    raise ValueError("statement does not add up -> _review/")

The important line is the last one. The statement prints its own total. If my parsed rows don't add up to that total to the cent, the statement is rejected and lands in review. No "close enough". Two things protect you here: the regex reads date, text and amount from the same physical line, so a row can't borrow its neighbor's amount, and the sum check catches anything that makes the total drift, like a line the regex didn't recognize.

Use Decimal, not float. You're comparing money to the cent; float will eventually bite you on a sum of 40 values.

The LLM still has a job: single receipts

For individual receipts, invoices from random vendors in random layouts, a parser per vendor is hopeless. That's where the model earns its keep: it reads the PDF (or image) and returns company, vendor, date, amount, invoice number, document type.

But it doesn't get the last word on the numbers. Every amount and date it returns must be found verbatim in the text extracted from the PDF. I extract text twice, with pdfplumber and with pdftotext -layout, because they fail on different PDFs:

import subprocess

def pdf_text(path: str) -> str:
    with pdfplumber.open(path) as pdf:
        a = "\n".join(p.extract_text() or "" for p in pdf.pages)
    b = subprocess.run(["pdftotext", "-layout", path, "-"],
                       capture_output=True, text=True).stdout
    return a + "\n" + b

def verified(llm: dict, text: str) -> bool:
    amount_str = f"{llm['amount']:,.2f}".replace(",", "X").replace(".", ",").replace("X", ".")
    return amount_str in text and llm["date_as_printed"] in text

(Real code normalizes a few more formats, with and without thousands separators, but the principle is: if the model says 1.234,56 and that string isn't in the document, the model made it up.)

Gotcha: scans have no text layer

A photo or a scan without OCR has no text to verify against. For those, the model reads the image twice, independently, and both readings must agree on amount and date. If they differ, the receipt goes to _review/. It's a cheap way to catch the cases where the model is guessing: guesses tend not to repeat exactly, correct readings do.

That's not as strong as verbatim verification, and I know it. But the alternative (an OCR stack on the server) is a project of its own. If you're dealing with lots of scanned PDFs, giving the AI proper vision tools matters more than the model you pick.

Gotcha: password-protected PDFs

Some vendors send invoices as PDFs with an open password. The pipeline can't read them, so they go to _review/ with _password-protected in the name. Don't let them silently fail, and definitely don't let the model "read" an encrypted file and return something.

Matching receipts against the statement

Matching rules, deliberately strict:

• Amount to the cent. For foreign currency transactions, match on the original currency amount from the statement, not the converted euro amount, because the exchange rate on the receipt never matches the card's.
• Date window −5 to +12 days. Receipts are often dated before the card charge (pre-authorization) and card charges often post days after the invoice date. The window is asymmetric on purpose.

from datetime import timedelta

def matches(tx, rcpt) -> bool:
    amount_ok = (rcpt.amount == tx.orig_amount) if tx.orig_currency else (rcpt.amount == tx.amount)
    delta = tx.date - rcpt.date
    return amount_ok and timedelta(days=-5) <= delta <= timedelta(days=12)

Every statement gets a reconciliation CSV next to it (semicolon-separated, UTF-8 with BOM so Excel opens it without mangling umlauts). Each transaction either shows the matched receipt file or RECEIPT MISSING. That missing list is the actual product. It's what the bookkeeper wants.

Duplicates

Vendors love sending the same invoice twice: as an attachment and rendered in the mail body. Or a reminder with the same invoice attached again. Same invoice number + same amount = same receipt, filed once. The second copy is logged and skipped. (The HTML-body case is its own story; I render those mails to PDF with headless Chromium.)

Running it

Cron every 10 minutes, wrapped in flock so a slow run never overlaps the next:

*/10 * * * * /usr/bin/flock -n /run/receipt-sorter.lock /opt/receipt-sorter/venv/bin/python /opt/receipt-sorter/sorter.py >> /var/log/receipt-sorter.log 2>&1

State lives in SQLite: documents, transactions, processed mails. That's what makes re-runs harmless. A receipt that's already in the DB isn't filed again; a statement that's already parsed isn't parsed again. If you've never been bitten by a cron job doing its side effect twice, read the idempotency bug that uploaded one file 153 times first. Log rotation via logrotate, because a job running 144 times a day will fill a disk eventually.

What I'd tell anyone building this

• Split "understand" from "count". LLMs classify and extract fuzzy fields well. Exact numbers that must reconcile come from a parser, or are verified against the source text.
• Every number needs a check that doesn't involve the model. Statement total, verbatim match, double reading. Pick at least one per document type.
• A review folder is a feature. The system's job is not to handle 100%. It's to handle the easy 90% correctly and put the rest in front of a human, clearly labeled.
• Small models are fine once they're boxed in like this. The verification layer matters more than the model size. Watch the bill anyway; background AI loops get expensive fast.

"Amount and date always have to be right" was the one hard requirement I got. The first version met it most of the time. That's the same as not meeting it.

Related posts

• My AI Agent Invented a CLI Flag, and I Called the Error "Cosmetic"
• My AI Agent Only Saw 71% of Its Own Memory
• “I Barely Use It” — Where $100 of AI API Credit Actually Went