🎮 Building an AI Solver for a "Simple" Puzzle Game Was Much Harder Than Expected

When most people look at Magic Sort or similar color sorting games, they assume they're straightforward.
"Just detect the colors and find the shortest path."
That sounds simple...
Until the game starts introducing mechanics like:

✅ Hidden tube segments
✅ Locked boxes
✅ Colored keys
✅ Curtain-covered tubes
✅ Tag tubes

At that point it stops being a simple sorting puzzle and starts behaving more like a dynamic state machine.

🤔 The Biggest Challenge Wasn't the Solver

Most discussions focus on search algorithms.
In reality, the harder problem is understanding the board correctly before solving even begins.
A modern puzzle screenshot contains far more than colored liquids.

The solver has to understand:

• 🧩 Tube positions

• 🎨 Liquid colors

• 🔑 Keys inside tubes

• 📦 Locked box groups

• 🎭 Curtain groups

• 🏷️ Tag tubes

• 🧪 Covered tubes

• ❓ Hidden segments

Missing even one of these can completely change the solution.

🤖 Why Computer Vision Matters

Instead of relying on simple pixel scanning, this project uses a custom YOLO model to reconstruct the visible board from a screenshot.
The computer vision step is only the beginning. Once the board is reconstructed, a heuristic search explores valid move sequences while respecting the game's mechanics.

The interesting part is that the magic sort solver never assumes information it cannot know.
When the game reveals a hidden segment or unlocks a new group of tubes, it pauses, lets the player update the board and then continues searching with the new information.
That feels much closer to how a human actually solves these puzzles.

⚙️ Lessons That Stood Out

A few things I found particularly interesting:

🎯 1. Detection is never perfect
Even good object detection models occasionally miss:

• Keys

• Hidden segments

• Special objects

• Covered tubes

Allowing users to correct the detected board is often more valuable than trying to reach 100% detection accuracy.

🧠 2. Game rules are more important than algorithms

A search algorithm alone isn't enough.

The solver also needs to understand rules like:

• Which key unlocks which box

• When curtain groups reveal

• How ice clusters behave

• When fog stones disappear

• How refill tubes work

• When hidden information becomes available

Without those rules, even an excellent search algorithm produces bad solutions.

🎨 3. UX matters just as much

One thing I appreciated was the focus on the editing workflow.
Instead of forcing another screenshot upload every time detection isn't perfect, users can quickly adjust the board and continue solving.
That small design decision probably saves more time than improving detection accuracy by another 2–3%.

📚 If You're Curious...

The project includes a couple of guides that explain the concepts without exposing implementation details.

🔹 How the AI Solver Works

https://aimagicsortsolver.com/guides/how-ai-solver-works

🔹 Magic Sort Special Objects Guide

https://aimagicsortsolver.com/guides/magic-sort-special-objects-guide

They're interesting reads if you're building anything that combines computer vision with game logic.

💡 Final Thought

Projects like this remind me that many "simple" mobile games are anything but simple.
Once hidden information and dynamic mechanics enter the picture, you're no longer solving a static puzzle.

You're building a system that has to:

• 👀 Understand incomplete information

• 🧩 Apply changing game rules

• 🤝 Work alongside the player

• ⚡ Adapt every time the game reveals something new

That combination makes the problem much more interesting than simply finding the shortest sequence of moves.