jspace-refusal

This is an AI safety research project that studies how and when a language model decides to refuse a request, such as one asking for something harmful, before it actually writes out a refusal like I cannot help with that. Using a technique called the Jacobian lens, developed by Anthropic, the researchers found that the model's internal state already signals it is going to refuse roughly ten layers of processing before it produces the first word of its answer. In other words, the decision is visible inside the model well before the response appears.

The project experiments with editing that internal signal to see if refusal behavior can be removed. It compares its method, called a Jacobian pullback edit, against a more common technique called abliteration, which is a way of stripping out a model's tendency to refuse. The pullback edit turns out to be more precise, causing less unwanted disruption to how the model handles normal, harmless requests, but it removes less of the actual refusal behavior compared to abliteration.

Digging deeper, the researchers found that clearing the part of the model that produces the verbal I cannot phrasing does not stop the model from refusing, because the actual refusal behavior is driven by a separate internal signal that recognizes a request as harmful or illegal, rather than by the wording it uses to say no. They describe this as a difference between perception, recognizing something is harmful, and narration, saying the words that describe refusing. Even after editing a model so it stops refusing on the surface, an internal signal that detects harmful requests can still be found and measured inside the model, suggesting a kind of hidden monitor survives even when the visible refusal is removed.

The project includes Python scripts to reproduce each step of this research, from locating refusal signals to running benchmark comparisons and applying the edits. Running it requires a CUDA capable NVIDIA GPU, since it works directly with a large language model's internals, and it pulls its test datasets from public sources online, with offline fallbacks provided.