Card scanner accuracy: method and numbers

SpellControl's scanner identifies a card from a phone camera frame. This page says how its accuracy is measured, shows the numbers, and says what they do not tell you.

The queries are synthetic, and real-camera numbers are pending. Every query below is a Scryfall card image that was damaged by a program to look like a bad camera shot. None is a photograph. Do not read any figure here as the share of real scans that succeed. A run on photographs taken with real phones has not been done yet, and this page will say so until it has.

The method, the card list and the code are in the repository under backend/src/scanner/benchmark/, so anyone can re-run it. The figures were produced on 2026-11-02 on one laptop (darwin-arm64, Node 22.23.2) against an index of 55,725 cards.

What is tested

The phone finds the card, straightens it, cuts out the art window and sends it to the server, which names the card. The server has two ways of recognizing it, and the scanner uses them together.

Top-1 counts a query when the first answer is the right card. Top-3 counts it when the right card is among the first three answers. For the production pipeline, an answer the scanner is sure of is one card, a pick list is up to five cards, and no answer counts as wrong.

The cards

200 cards, drawn from the scanner's index with a seeded random generator (seed 20261102) and published by name and Scryfall id in corpus.json. The draw is random across everything in the index, so the list reflects the index and not a hand-picked set of easy cards. The first 20 of the list are the subset that runs on every change to the code.

The synthetic queries

Each card is queried seven ways. Every degradation uses a random generator seeded from the card and the condition, so the same card always gets the same damage. The program is degrade.ts.

Results

Top-1, 200 cards per row.

Top-1 accuracy by condition and method
Condition Hash only Embedding only Production pipeline
Clean (control) 200 of 200 (100.0%) 200 of 200 (100.0%) 200 of 200 (100.0%)
Perspective 168 of 200 (84.0%) 200 of 200 (100.0%) 198 of 200 (99.0%)
Blur 200 of 200 (100.0%) 197 of 200 (98.5%) 199 of 200 (99.5%)
Glare 124 of 200 (62.0%) 199 of 200 (99.5%) 184 of 200 (92.0%)
Noise and compression 200 of 200 (100.0%) 199 of 200 (99.5%) 200 of 200 (100.0%)
Occlusion 157 of 200 (78.5%) 199 of 200 (99.5%) 195 of 200 (97.5%)
Combined 86 of 200 (43.0%) 192 of 200 (96.0%) 152 of 200 (76.0%)

Top-3, same queries.

Top-3 accuracy by condition and method
Condition Hash only Embedding only Production pipeline
Clean (control) 200 of 200 (100.0%) 200 of 200 (100.0%) 200 of 200 (100.0%)
Perspective 183 of 200 (91.5%) 200 of 200 (100.0%) 198 of 200 (99.0%)
Blur 200 of 200 (100.0%) 200 of 200 (100.0%) 199 of 200 (99.5%)
Glare 148 of 200 (74.0%) 200 of 200 (100.0%) 184 of 200 (92.0%)
Noise and compression 200 of 200 (100.0%) 200 of 200 (100.0%) 200 of 200 (100.0%)
Occlusion 176 of 200 (88.0%) 200 of 200 (100.0%) 195 of 200 (97.5%)
Combined 111 of 200 (55.5%) 194 of 200 (97.0%) 152 of 200 (76.0%)

How the production pipeline answered, in cards out of 200.

How the production pipeline answered
Condition Confident Confident and wrong Asks you to pick No answer
Clean (control) 200 0 0 0
Perspective 198 0 1 1
Blur 200 1 0 0
Glare 184 0 9 7
Noise and compression 200 0 0 0
Occlusion 195 1 2 3
Combined 128 1 41 31

What the numbers say

Speed

Wall time per query on the benchmark laptop, with nothing else asked of the server. It is not a production latency: it leaves out the phone's work, the upload and the network.

Time per query on one laptop
Method Median 95th percentile Slowest
Hash only 3.2 ms 13.4 ms 236.6 ms
Embedding only 102.7 ms 330.6 ms 825.4 ms
Production pipeline 34 ms 214.6 ms 530.3 ms

The pipeline is faster than the embedding alone because a clear hash match skips the neural network.

What this does not show

Re-run it

From backend/, npm run scanner:bench -- fetch downloads the art for the listed cards from Scryfall, and npm run scanner:bench -- run scores all three methods (the last two need the large embedding file from Git LFS). The hash method on the 20-card subset also runs on every code change and fails if its accuracy drops. The production pipeline on that subset runs on demand in the repository's Scanner Benchmark workflow, because the embedding file is not available in the everyday checks. The figures on this page are stored in results.json, and a test fails if this page and that file disagree.

Open SpellControl

Free, no account required.