How a photo becomes a solved Sudoku
You upload a photo of a newspaper puzzle or a screenshot of an app, and a second later the solution appears. Here is what happens in between, and how we make sure a misread digit does not turn into a wrong answer.
1. Find the grid
The first try is classic computer vision with OpenCV: find the largest four-sided outline and check that it really is a Sudoku. A Sudoku has its lines at ten evenly spaced positions in each direction, so after straightening the candidate we look for exactly that. A decorative frame around the puzzle, or a single 3×3 box, fails the check; in that case we fit the 9×9 line lattice around what was found.
Image size matters more than you would think. A 12-megapixel photo of a laptop screen picks up moiré patterns, while a small screenshot has 1-pixel lines that a filter can erase. So the detector tries the image at full resolution and at 1280 pixels wide, and keeps the first grid with regular lines.
OpenCV finds the grid in roughly nine of ten photos. For the rest, such as glare across the lines or a puzzle in a plastic sleeve, a small neural network predicts the four corners directly. It outputs a heatmap per corner rather than coordinates, which turned out to be far more precise: its median corner error on our test photos is about 1% of the grid’s side.
2. Straighten and clean up
The four corners give a perspective transform, which maps the grid to a flat 450×450 image, cut from the full-resolution photo so that small digits stay sharp. The image is then converted to pure black and white with an adaptive threshold, which compares each pixel with its neighbourhood. That removes shadows, the yellow tint of newsprint, and the colour of an app’s theme.
3. Read the digits
Each of the 81 cells is cut out and resized to 48×48 pixels, and a convolutional network decides whether it is blank or holds a digit from 1 to 9. The network is tiny, with 38,000 parameters, and reads all 81 cells in about 8 milliseconds on our two-core server. It was trained on our own labeled screenshots and photos, with corners deliberately moved by up to 1.5% so that it copes with slightly misplaced grids.
Small fonts get special treatment: when the digits in a grid are smaller than anything seen in training, the cells are zoomed in first.
4. The solution is the test
A Sudoku you can publish has exactly one solution. That gives us a free check on the reading: if the digits we read have exactly one solution, the reading is almost certainly right. If they have none, a digit was misread; if they have many, a digit was probably missed.
In both cases the solver does not give up. The network gives each cell a ranked list of candidates, so we search for the smallest change, at most two cells, that leads to exactly one solution, trying the network’s second guesses first. If the OpenCV grid still does not solve, the corner network gets its turn on the same photo.
The solver itself is a backtracking search over bitmasks that always fills the cell with the fewest legal digits first. A typical puzzle takes well under a millisecond.
5. How we keep it honest
- A new digit model is released only if it reads every labeled image exactly. An image that once failed and was fixed stays fixed.
- A separate test set of photos and screenshots is never used for training or for picking models. It is only measured.
- The page shows the live solve rate over all distinct uploaded images, failures included.
- When a digit is wrong you can fix it and solve again. Corrections become training data.
It still struggles with handwritten digits, heavy blur, and grids that are cut off at the edge of the photo. If one of your puzzles fails, uploading it helps: it is exactly the kind of image the next model needs.
The code is open source on GitHub.