The Refusal Gap

Safety evaluation scores whether a model refused. It never scores whether the refusal left the person more dangerous than before.

A two-axis benchmark for the Lexsi Labs AI Safety & Alignment Hackathon, built on thirty years of threat-assessment research rather than on knowledge-recall quizzes.

The instrument is pointed at the wrong layer

The dominant way to evaluate whether a language model is dangerous is to ask it a set of multiple-choice questions about hazardous subject matter and count how many it gets right. A lower score is reported as a safer model.

Three things are wrong with this, and they compound.

It is a deterministic test of a non-deterministic system. A recall quiz, scored once, on a stochastic generator, used as a proxy for a human outcome with a base rate close to zero. Every link in that chain leaks. The threat-assessment field reached this conclusion decades ago and abandoned actuarial prediction for structured professional judgement, precisely because low base rates defeat prediction.

It measures recall, not conduct. Knowing a fact and walking someone through a procedure are different capabilities, and the second is the one that matters. A model can score zero on the quiz and still be an effective collaborator; it can score well and help nobody.

It is silent on the thing the model actually did to the person. This is the gap the project is named for. A conversation is not a single request to be permitted or denied — it is an exchange with someone in a state, and the model's reply is an intervention in that state. The evaluation records whether the hazardous string appeared. It records nothing about whether the reply pushed the person further along a trajectory or off it.

Refusal is not a neutral act. It is an intervention with its own direction, and nobody is measuring which way it points.

What the literature says happens when you shut someone down

Three independent bodies of work converge here, and none of them are AI research.

Reactance. Brehm's 1966 theory, replicated for six decades: restricting someone's freedom of action increases the attractiveness of the restricted option and motivates them to restore the freedom. The effect is strongest when the restriction reads as arbitrary or illegitimate — which is exactly how a boilerplate refusal reads to someone who believes their interest is legitimate. The refusal does not merely fail to help. It adds motivational energy to the thing it refused.

Exclusion escalates. In school and workplace threat assessment, the practitioner consensus is to keep a person of concern engaged rather than excluded, because exclusion removes visibility at the same moment it increases grievance and reduces what the person has left to lose.

Leakage. People on a path toward targeted violence frequently communicate something about it beforehand — to a friend, a forum, a confidant. The practitioner literature treats this as the central interception opportunity, and locates the prevention failure not in the person's secrecy but in the recipient not being positioned to act.

Put together: the person asking an alarming question is often leaking rather than operating, and the standard response is to close the channel and tell them to seek help elsewhere. For someone who already believes the legitimate world has no place for them, that lands as one more door closing.

The pathway is a sequence of behaviours, and movement along it is what practitioners treat as signal. Current model evaluation instruments exactly one point on it — the moment a request arrives — and asks a single question about that moment. Leakage, which the threat-assessment literature treats as the principal interception opportunity, branches out of the pathway toward whoever the person trusts, and is not measured by any model benchmark.

The mechanism a refusal actually triggers

The claim is not that models should supply hazardous content. It is that how a model declines determines whether the channel stays open, and the channel is the only thing that makes leakage visible.

The two responses are identical on the axis that current benchmarks score: neither emits the hazardous content. They differ on an axis nothing measures. A refusal that reads as a judgement on the person removes the observer and, by reactance, raises the motivation it meant to suppress.

Two axes, one of them unmeasured

The project's central claim is that safety behaviour lives in a two-dimensional space and existing instruments collapse it to one dimension. Collapse it, and a model that withholds perfectly while being the worst possible interlocutor is indistinguishable from one that withholds perfectly and keeps the person in the room.

Projected onto the horizontal axis alone, the top-right and bottom-right quadrants are the same model. The bottom-right is the one the field is currently optimising toward, and the threat-assessment literature says it is the one that raises risk for the person most likely to be reachable.

Method

Built on the Lexsi Labs open-source stack against a Cohere Labs Aya model, inside the hackathon's two-week build window.

Stage	Tool	What it produces
Test-set construction	CuratorKIT	Paired dialogues that differ in the asker's state and register but not in what is asked, each with a provenance manifest so the set is auditable rather than asserted.
Baseline on both axes	AuditKIT	Hazard-withholding score and an interlocutor-quality rubric scored by LLM judge, with the run fingerprinted for reproducibility.
Locate the behaviour	CircuitKIT · SafeTune	Whether refusal and engagement are separable components at all, or one entangled direction. Determines whether the gap is fixable by steering or needs training.
Correct and re-measure	AlignTune	Preference training on pairs where the chosen response withholds the same content without passing a verdict. Then the full battery again.

What gets reported

Hazard withholding — must not degrade. A fix that trades safety for warmth is not a fix.
Interlocutor quality — the new axis: did the response stay engaged and honest, did it avoid rendering a verdict on the person, did it leave a route back.
Over-refusal on legitimate work — measured on benign requests, which is the safe and publishable side of this question.
General capability — must hold, or the result is an artefact of a damaged model.
Disparity across register — the same question asked bluntly and asked with social hedging. If the outcomes differ, the system's safety behaviour is conditioned on communication style.

That last one is the measurement with a regulatory hook. EU AI Act deployer obligations require attention to whether a system's behaviour produces different outcomes for different users; the compliance templates currently contain no method for detecting it in safety behaviour specifically. The conformity standards are still being written.

Design constraints held deliberately

Behaviour, never person-type. The calibration conditions on observable conversational behaviour, in the manner of the structured instruments' proximal items. Nothing in the design produces a list of people, and that property is load-bearing rather than incidental.
No prediction of individuals. The benchmark scores a model's conduct against documented escalation patterns. It does not claim a probability that any person will act — there is no outcome-linked corpus that would license such a claim, and the base rate defeats it.
Measured on benign work. False positives are demonstrated with legitimate requests. The converse is argued structurally from the published literature, not demonstrated.

What is not established. That an engaged decline reduces real-world harm compared with a refusal is the hypothesis under test, not a finding. The same absence of outcome-linked data that defeats a probability-of-harm benchmark also prevents anyone — including this project — from showing that the alternative saves anyone. What can be shown is that the two axes come apart, that current instruments see only one of them, and that the literature on reactance and exclusion gives a specific mechanism for why the unmeasured axis matters. Treat the rest as motivation.

References

The threat-assessment canon this is built on, with what each contributes. Verification status is marked: these are being checked against primary sources before submission, and specific figures are the most likely thing to be wrong.

Brehm, J. W. — A Theory of Psychological Reactance (1966).Why: the mechanism by which a refusal adds motivation rather than removing it. Strongest when the restriction reads as arbitrary. · verifying

Calhoun, F. & Weston, S. — Contemporary Threat Management (2003).Why: origin of the pathway model used in figure 1. Movement along the pathway is behavioural, therefore observable. · verifying

Meloy, J. R. & O'Toole, M. E. — "The concept of leakage in threat assessment", Behavioral Sciences and the Law (2011).Why: leakage as the principal interception opportunity, and the prevention failure located in the recipient rather than the subject. · verifying

Fein, R. & Vossekuil, B. — Exceptional Case Study Project, US Secret Service (1990s).Why: the anti-profile doctrine. There is no useful profile, so assess behaviour and situation. Directly constrains the calibration design. · verifying

Borum, R., Fein, R., Vossekuil, B. & Berglund, J. — "Threat assessment: defining an approach for evaluating risk of targeted violence" (1999).Why: why the field abandoned actuarial prediction for structured judgement. The low-base-rate argument, made first and better. · verifying

Meloy, J. R. — TRAP-18, Terrorist Radicalization Assessment Protocol.Why: supplies behavioural axes — pathway, fixation, leakage, last resort — that are observable in a conversation and are not person-type attributes. · verifying

Gill, P., Horgan, J. & Deckert, P. — "Bombing alone", Journal of Forensic Sciences (2014).Why: prevalence of prior communication in lone-actor cases. The widely-quoted figure is frequently misstated; not cited until checked. · figure unverified

Horgan, J. — Walking Away from Terrorism (2009); Bjørgo, T. & Horgan, J. (eds) — Leaving Terrorism Behind (2008).Why: exit is driven by disillusionment and the availability of an alternative role, not by counter-argument or raised cost. · verifying

Maruna, S. — Making Good (2001); Laub, J. & Sampson, R.Why: the desistance literature. Trajectories change when a person acquires something incompatible with the old one. · verifying

Al-Attar, Z. — published work on autism in relation to extremism risk assessment.Why: a practitioner source on how risk instruments misread atypical presentation, and on why the association runs opposite to the stereotype. · verifying

Cooke, et al. (2025) — synthesis of 57 studies on violence and autistic people.Why: the literature is overwhelmingly about autistic people as victims. Any instrument that flags atypical communication inherits this problem. · verifying

Peterson, J. & Densley, J. — The Violence Project; NIJ public-mass-shooting data.Why: crisis rather than diagnosis as the proximal factor, and the prevalence of identifiable crisis and isolation. Source of the leakage rate used in the design. · figure being re-checked

UK Home Office — Channel referral statistics; US CVE programme evaluations.Why: the twenty-year record of this intervention model implemented badly. High referral volume, small fraction needing support, concentration on atypical populations. · verifying

Milton, D. — the double empathy problem (2012).Why: locates communication failure as bidirectional, which is why "this user communicates oddly" is not evidence about the user. · verified in prior work

Working bibliography with reading order: refs/threat-assessment-canon.md in the project repository.

nam-shub →Lexsi Labs ↗back to meow-meow.io

safety research · build in progress