Receipt
Detection
System.

The receipt detection system is an end-to-end ML product: a fine-tuned object detection model capable of localising receipts within arbitrary photographs, wrapped in a production-grade web application with authentication and a CDN-served frontend.

The core problem is simple — given a photo that may or may not contain a receipt, draw a tight bounding box around any receipt present. The receipts appear at every angle, under variable lighting, partially obscured, crumpled, or on cluttered surfaces. A general-purpose detector struggles; a purpose-trained one does not.

This writeup covers every decision: where the data came from, how I labelled it, why YOLO was chosen, how I prevented overfitting with negative samples, and how the entire system is deployed on AWS.

Good models are made of good data. I combined two sources to create a dataset that is both comprehensive in receipt variety and robust against false positives.

The Voxel51 Consolidated Receipt Dataset is a curated collection of real-world receipt images spanning diverse lighting conditions, receipt types (grocery, restaurant, retail), and photographic angles. I manually labeled 800 images from this dataset, which initially formed the positive class for training.

After training the first version of the model, I observed clear signs of overfitting along with a high rate of false positives. The model frequently misclassified non-receipt objects—particularly text-heavy or rectangular surfaces—as receipts. To address this imbalance, I introduced 800 non-receipt images sampled randomly from the COCO 2017 dataset. These images were included with empty annotation files, explicitly teaching the model that such scenes do not contain receipts. This significantly reduced false positives and improved generalization.

In a later evaluation phase, I identified another limitation: the model struggled to detect receipts captured at angles. To improve robustness to orientation, I augmented the dataset by rotating 800 receipt images by 45 degrees to the left and right, generating an additional 1,600 images.

Through this iterative process—initial labeling, dataset balancing with negative samples, and targeted augmentation—the final dataset was expanded to a total of 3,000 images, resulting in a more generalizable and reliable receipt detection model.

Every receipt image was labelled by hand using LabelImg, an open-source graphical annotation tool. I drew tight bounding boxes in YOLO format — each annotation saved as a .txt file containing the normalised centre coordinates, width, and height of each receipt.

Manual labelling is slow, but it matters. Automated pipelines applied to noisy web-scraped data tend to produce annotations with loose or incorrect boxes. With receipts in particular, the boundary between the receipt and the background can be subtle — especially on white tablecloths or desks. Hand-labelling forces you to make that judgment call precisely for every example.

The COCO negative images receive empty annotation files — a YOLO convention meaning "no objects of interest in this image." This is important: if you simply omit the annotation file, Ultralytics may skip the image during training. An empty file is an explicit instruction.

The YOLO (You Only Look Once) family has become the default choice for real-time object detection. Version 8, released by Ultralytics, brings a cleaner codebase, improved anchor-free detection heads, and excellent transfer learning support. I chose the small variant deliberately.

This is a single-class detection problem with relatively consistent object morphology — receipts are rectangular, white or off-white, with dense text. There is no need for a large backbone. YOLO offers a strong accuracy/speed tradeoff: it's fast enough for real-time inference on a Fargate container with no GPU, yet capable enough to handle the geometric variation in the dataset.

Transfer learning is key here. Starting from pretrained weights means the backbone already understands low-level visual features: edges, textures, corners, and gradients. Fine-tuning then redirects this prior knowledge toward the specific task of receipt localisation, converging faster and with less data than training from scratch.

Ultralytics' built-in augmentation pipeline — mosaic composition, random flipping, HSV colour jitter — provides additional regularisation without requiring a separate preprocessing step, keeping the training code compact and reproducible.

The model is one piece of a larger system. The full production stack handles authentication, secure image upload, model inference, and result delivery — all serverless or containerised, with no infrastructure to manage directly.

The architecture follows a clean separation of concerns across three layers: authentication, inference, and content delivery.

After 50 epochs of fine-tuning on a 3,000-image dataset, the model achieves strong, production-ready detection performance. Evaluated on the held-out validation set, it correctly localises receipts with high accuracy across the full range of IoU thresholds — not just the lenient 50% cutoff.

The confusion matrix tells the clearest story. Of 225 ground-truth receipt instances, 199 were correctly detected. Only 26 were missed (false negatives), and just 6 background images triggered a spurious detection — a false positive rate low enough to be essentially invisible in real-world use.

The asymmetry between false negatives (26) and false positives (6) directly explains the recall-precision gap seen in the metrics. The model has learned to be conservative about background — a direct consequence of the 800 COCO hard negatives introduced during training. The missed receipts are concentrated in heavily occluded or extreme-angle cases, which rotational augmentation improved but did not fully resolve.

The near-identical mAP@50 (90.5%) and mAP@50–95 (89.5%) is particularly significant. Most detectors lose 15–20 points across that IoU sweep; a 1-point drop indicates the model is predicting very tight, accurate bounding boxes — not just approximately correct regions.