Card scanner accuracy: method and numbers
SpellControl's scanner identifies a card from a phone camera frame. This page says how its accuracy is measured, shows the numbers, and says what they do not tell you.
The queries are synthetic, and real-camera numbers are pending. Every query below is a Scryfall card image that was damaged by a program to look like a bad camera shot. None is a photograph. Do not read any figure here as the share of real scans that succeed. A run on photographs taken with real phones has not been done yet, and this page will say so until it has.
The method, the card list and the code are in the repository under
backend/src/scanner/benchmark/, so anyone can re-run it. The figures were
produced on 2026-11-02 on one laptop (darwin-arm64, Node 22.23.2) against an index of 55,725
cards.
What is tested
The phone finds the card, straightens it, cuts out the art window and sends it to the server, which names the card. The server has two ways of recognizing it, and the scanner uses them together.
- Hash only. A 64-bit perceptual hash of the picture, compared with the hash of every card in the index. Fast and small; it is also what the scanner falls back to when the phone is offline.
- Embedding only. A neural network (MobileCLIP2-S0) turns the picture into 512 numbers, compared with the stored numbers for every card in the index.
- Production pipeline. The server's real matcher, called the way the API calls it: the hash picks 50 candidates, the embedding ranks them, and the score decides whether the scanner answers, asks you to pick from a short list, or gives no answer.
Top-1 counts a query when the first answer is the right card. Top-3 counts it when the right card is among the first three answers. For the production pipeline, an answer the scanner is sure of is one card, a pick list is up to five cards, and no answer counts as wrong.
The cards
200 cards, drawn from the scanner's index with a seeded random generator (seed 20261102) and
published by name and Scryfall id in corpus.json. The draw is random across
everything in the index, so the list reflects the index and not a hand-picked set of easy
cards. The first 20 of the list are the subset that runs on every change to the code.
The synthetic queries
Each card is queried seven ways. Every degradation uses a random generator seeded from the
card and the condition, so the same card always gets the same damage. The program is
degrade.ts.
- Clean. The Scryfall art image unchanged. The scanner's index was built from these same images, so this row is a lookup of a known picture and says nothing about cameras. It is there to prove the plumbing works.
- Perspective. A projective warp that moves each corner by up to 6% of the frame, then a JPEG save at quality 90 (the quality the app uploads at). It imitates the error left after the card is straightened.
- Blur. Gaussian blur of 4 pixels on an image about 626 pixels wide, for a shot out of focus or moved.
- Glare. A soft white highlight at a random spot, up to 85% white and about a third of the card's width across, like a light on a sleeve or foil.
- Noise and compression. Random sensor noise plus a hard JPEG save at quality 35, like a dim room and a squeezed upload.
- Occlusion. A dark strip hiding 20% of the picture along one edge, like a thumb or a binder ring.
- Combined. A milder dose of all of the above at once.
Results
Top-1, 200 cards per row.
| Condition | Hash only | Embedding only | Production pipeline |
|---|---|---|---|
| Clean (control) | 200 of 200 (100.0%) | 200 of 200 (100.0%) | 200 of 200 (100.0%) |
| Perspective | 168 of 200 (84.0%) | 200 of 200 (100.0%) | 198 of 200 (99.0%) |
| Blur | 200 of 200 (100.0%) | 197 of 200 (98.5%) | 199 of 200 (99.5%) |
| Glare | 124 of 200 (62.0%) | 199 of 200 (99.5%) | 184 of 200 (92.0%) |
| Noise and compression | 200 of 200 (100.0%) | 199 of 200 (99.5%) | 200 of 200 (100.0%) |
| Occlusion | 157 of 200 (78.5%) | 199 of 200 (99.5%) | 195 of 200 (97.5%) |
| Combined | 86 of 200 (43.0%) | 192 of 200 (96.0%) | 152 of 200 (76.0%) |
Top-3, same queries.
| Condition | Hash only | Embedding only | Production pipeline |
|---|---|---|---|
| Clean (control) | 200 of 200 (100.0%) | 200 of 200 (100.0%) | 200 of 200 (100.0%) |
| Perspective | 183 of 200 (91.5%) | 200 of 200 (100.0%) | 198 of 200 (99.0%) |
| Blur | 200 of 200 (100.0%) | 200 of 200 (100.0%) | 199 of 200 (99.5%) |
| Glare | 148 of 200 (74.0%) | 200 of 200 (100.0%) | 184 of 200 (92.0%) |
| Noise and compression | 200 of 200 (100.0%) | 200 of 200 (100.0%) | 200 of 200 (100.0%) |
| Occlusion | 176 of 200 (88.0%) | 200 of 200 (100.0%) | 195 of 200 (97.5%) |
| Combined | 111 of 200 (55.5%) | 194 of 200 (97.0%) | 152 of 200 (76.0%) |
How the production pipeline answered, in cards out of 200.
| Condition | Confident | Confident and wrong | Asks you to pick | No answer |
|---|---|---|---|---|
| Clean (control) | 200 | 0 | 0 | 0 |
| Perspective | 198 | 0 | 1 | 1 |
| Blur | 200 | 1 | 0 | 0 |
| Glare | 184 | 0 | 9 | 7 |
| Noise and compression | 200 | 0 | 0 | 0 |
| Occlusion | 195 | 1 | 2 | 3 |
| Combined | 128 | 1 | 41 | 31 |
What the numbers say
- The hash alone is fragile. It gets the right card first on 124 of 200 glare queries and 86 of 200 combined ones. It is unaffected by blur and noise, which it throws away by design, and weak against anything that changes the layout of light and dark.
- The embedding is much more robust. Its worst row is combined, at 192 of 200. Searched across the whole index with no hash shortlist, it is not what the scanner does, but it shows the model holds up where the hash does not.
- The production pipeline trades some answers for caution. Its worst row is also combined: 152 of 200, with 41 pick lists and 31 scans with no answer. The embedding alone gets 192 of 200 on the same queries. The pipeline declines to guess below its score thresholds, and a decline costs a re-scan, not a wrong card. Its hash shortlist can also drop the right card before the embedding sees it.
- Wrong confident answers were rare. Across all 1,400 queries the pipeline was sure and wrong 3 times. That is the number that matters most, because a confident wrong answer adds the wrong card to a collection without asking.
Speed
Wall time per query on the benchmark laptop, with nothing else asked of the server. It is not a production latency: it leaves out the phone's work, the upload and the network.
| Method | Median | 95th percentile | Slowest |
|---|---|---|---|
| Hash only | 3.2 ms | 13.4 ms | 236.6 ms |
| Embedding only | 102.7 ms | 330.6 ms | 825.4 ms |
| Production pipeline | 34 ms | 214.6 ms | 530.3 ms |
The pipeline is faster than the embedding alone because a clear hash match skips the neural network.
What this does not show
- No real photographs. Real cameras add lens distortion, rolling shutter, exposure and white-balance shifts, sleeve and foil texture, and a crop that frames the art a little differently from Scryfall's. The synthetic damage covers a few of these crudely and the rest not at all. Real scans could do better or worse.
- The art is the same art. The index and the queries come from the same Scryfall images, so even the damaged queries start from a perfect copy of the reference.
- Printings with the same art. The index holds one entry per artwork, and a query counts as right only for the exact entry. Picking among reprints with identical art is done later, from the card's info line, and is not scored here.
- Small sample. With 200 cards, a gap of a few cards between two rows is within chance. The 20-card subset that gates changes is smaller still.
- One machine. Other processors and image libraries can move a card or two, and timings differ more than accuracy does.
- No comparison with other scanners. Nothing here measures any other app.
Re-run it
From backend/, npm run scanner:bench -- fetch downloads the art
for the listed cards from Scryfall, and npm run scanner:bench -- run scores all
three methods (the last two need the large embedding file from Git LFS). The hash method on
the 20-card subset also runs on every code change and fails if its accuracy drops. The
production pipeline on that subset runs on demand in the repository's
Scanner Benchmark workflow, because the embedding file is not available in the
everyday checks. The figures on this page are stored in results.json, and a
test fails if this page and that file disagree.
Free, no account required.