A study finds single points of failure in seven models’ refusal behavior

No model launch has appeared today that warrants displacing yesterday’s Kimi K3 news. The honest alternative is to revisit a less recent and more durable question: when a model refuses a harmful request, has it lost the ability to answer, or has it learned a mechanism that decides not to express the answer?

An Apple-affiliated preprint by three researchers, originally posted on May 8 and added to the company’s research archive in July, offers a disturbing and concrete answer. Across seven open-weight models—five sizes of Qwen3 and two of Llama 3.1, ranging from 1.7 billion to 70 billion parameters—the authors identified a neuron whose intervention is sufficient to suppress much of the models’ refusal behavior on harmful requests.

“Neuron” here does not mean a cell or a self-contained piece of knowledge. It is one scalar coordinate in the activation of a transformer feed-forward layer: a small part of an internal computation whose effects are later combined with thousands of others. The study compares its activation on harmful and harmless instructions, uses gradients to rank candidates, tests the top five on a validation set, and retains the combination that most reduces refusals. It then evaluates that choice, without further tuning, on a separate set.

That sequence matters because the phrase “one neuron” can mislead. The researchers do not delete one weight and watch the model fail spontaneously. They pin the activation of a selected coordinate at every token position, both while processing the request and while generating the response. Their more precise variant also runs an initial pass to measure the strength of the signal and adapts the intervention to the context. This is sustained, surgical control over execution, not the digital equivalent of a component breaking by accident.

The result remains striking. According to the two automated judges used by the authors, single-neuron interventions average roughly 90% attack success on JailbreakBench, a held-out set of adversarial requests. The context-sensitive method preserves general performance much better than always forcing a constant value: the reported average absolute change is −0.6 points on MMLU and −0.1 on GSM8K. Those two tests approximate academic knowledge and mathematical problem solving; they do not establish that every model capability is preserved or that a deployed application would behave the same way.

A switch is not the whole system

The most useful reading is not that “safety lives in one neuron.” The paper itself separates two phenomena. Refusal neurons appear to act as a gate that helps decide whether certain information is expressed. Other concept neurons participate in representing content; the study tests this second route only for one self-harm category. The fact that a gate has a leverage point does not mean everything behind it is stored at that point.

Nor does the paper establish universal brittleness. It covers only two open model families, neither of which is Kimi K3 or a proprietary model served through an API. The method requires white-box access to activations and gradients, plus an initial search over harmful examples. A user of a closed service normally lacks that access. Someone who controls open weights can alter inference, but could already remove safeguards through fine-tuning or other, more expensive techniques. The authors themselves argue that their procedure does not substantially change that risk profile.

The advance is in diagnostic resolution. Earlier work showed that refusal can concentrate in a direction in the model’s internal space or in small parameter subsets. This preprint narrows the causal intervention to one coordinate in one layer and obtains an effect comparable to removing a direction across every layer. At the same time, the study finds that several different neurons may each produce the effect: locating one lever is not the same as finding a sole cause.

The evidence still has open limits. This is a preprint with no stated peer review or independent replication. The authors select intervention strength through an empirical sweep, automated judges can confuse harmful compliance with incoherent output, and the capability tests are narrow. A causal effect under an extreme intervention also does not, by itself, explain how the activation arose in training or how much it contributes under ordinary conditions.

My reading is that the work weakens a comfortable assumption: aligning a model’s output is equivalent to removing dangerous capability from its internals. Refusal may look robust in conversation while depending on narrow internal mechanisms. That does not make alignment useless; it changes what it should be asked to withstand. A model maker can search for single points of failure, train for redundancy, and repeat evaluations after quantization or fine-tuning. A system deployer can add independent controls around tools, data, and permissions, so an improper response does not automatically become an action.

The implication for open weights is not that they should be closed, but that a license or model card cannot preserve safeguards through every modification. If a community can inspect and alter a model, it can also measure whether refusal mechanisms survive conversions, fine-tunes, and new inference harnesses. The most valuable contribution of this preprint is a sharper instrument for asking that question; it is not yet a general answer about model safety.

Sources

• Hamid Kazemi, Atoosa Chegini, and Maria Safi, A Single Neuron Is Sufficient to Bypass Safety Alignment in Large Language Models, preprint, May 8, 2026.
• Apple Machine Learning Research, study page, published July 2026.
• Andy Arditi et al., Refusal in Language Models Is Mediated by a Single Direction, June 17, 2024.
• Zhaoxin Wang et al., SafeNeuron: Neuron-Level Safety Alignment for Large Language Models, February 12, 2026.