I Built an AI That Solves Block Blast Puzzles — Here's What Actually Happened

I kept losing Block Blast runs to the same mistake i.e placing a piece that looked fine but killed my board two moves later. So I decided to build a tool that could look at a screenshot, read the board and suggest the best move. The solver algorithm? Done in a day. Evaluate placement permutations, score based lines clearance, board openness, future flexibility. Getting the computer to actually see the board from a screenshot? That took weeks. #The Gap Between Algorithm and Reality# My solver needed a clean 8×8 grid and three shapes as input. But users would give it a phone screenshot that could be messy, variable, and unpredictable. I needed something to bridge that gap, so I went with YOLO for object detection. Train it to spot board and piece cells, map them to grid coordinates, and feed that into the solver. Simple plan but painful execution. #The Theme Problem# Block Blast has multiple visual themes. My first model, trained on one theme, hit 95% accuracy. I felt great. Tested it on a different theme, confidence score dropped to 40%. The model had learned to match specific colors, not board structure. Switch from vibrant blocks on a dark background to pastels on a light background and it fell apart completely. I had to collect screenshots across every theme, augment training data with brightness and contrast variations and retrain multiple times before the model could generalize. Lesson learned the hard way: your training data needs to represent the full range of what users will actually throw at you. #Shape Detection Was Worse# The three available pieces sit at the bottom of the screen and they're tiny compared to the board. YOLO would detect most cells in a shape but randomly miss one or two. In a 5-cell L-shape, missing one cell gives you a completely different shape and a completely wrong solution. After days of debugging, I found the root cause: the small physical size of piece cells meant YOLO was operating right at its confidence threshold. Some cells scraped above it, some fell below. Essentially random. I built a spatial parsing layer on top. Once the model detected some cells in a shape region, the parser would infer the bounding box, create a local grid and map cells into it. This compensated for missed detections but introduced its own edge cases around grid alignment and rounding errors. #When It Clicked# After three weeks of iterating, I had something that consistently read boards, detected shapes and suggested moves that were genuinely better than my intuition. The first time it analyzed a board I was stuck on and found a three-line clear I'd completely missed, I knew it was worth shipping. I polished the interface and launched it as the [AI Block Blast Solver](https://www.aiblockblastsolver.com) . #What I'd Tell Other Devs# If you're building a tool that reads game state from screenshots, here what I wish someone had told me: ##Screen capture is the hard part## Your algorithm might be elegant but extracting reliable input from messy real-world screenshots is where you'll spend most of your time. ##Don't trust accuracy metrics in isolation## 95% per-cell means nothing when you need every cell correct simultaneously. Test end-to-end accuracy, not component accuracy. ##YOLO isn't magic## For grid-based games, combining basic CV techniques with YOLO works better than YOLO alone. Use structure to your advantage. ##Ship early## Real user screenshots reveal edge cases you'd never find testing on your own device. I waited too long chasing perfection before getting real-world feedback. The gap between "YOLO tutorial on YouTube" and "production-ready detection on unpredictable inputs" is enormous. But closing that gap is where the interesting engineering lives.