I Trained a Model to Jailbreak Another Model by Prompt Rephrasing, Here's How It Worked and How to Stop It

I have been quiet for a couple of months because I have been travelling and busy with my actual job, so I decided to share something I have been working on in my free time.

Framing note first. This is red-team research with a defensive payload. I developed and built an attack in order to understand it and then to show how to stop it. So this post talks about the training process, the reward design, and the numbers, and then spends real time on defenses. It does not contain any working prompts or operational content, and it does not need to: the interesting parts are the ML, the stats, and the mitigations.

The one-paragraph version

I trained a small model to reword harmful prompts, without changing their meaning, so that a larger safety-tuned model would answer them. It worked. Starting from a baseline that basically did nothing, a sequence of training upgrades pushed the attack from near-zero real harm up to a strong operating point, and, most importantly, the attack transferred to a bigger model it had never trained against, cutting that model's refusals by more than half. Then I flipped to defense: because the attack has to keep the meaning intact and stay "natural" inside the model, it leaves fingerprints that safety probes sitting outside the base model can catch. The punchline is that no single detector is enough, but a diverse stack of them is.

The hypothesis

Written the way I would write it in a paper:

A small model can be trained to reword harmful prompts, holding their meaning fixed, so that a larger safety-tuned model is significantly more likely to comply; and this attack transfers to larger models the attacker never saw.

Both halves turned out to be supportable. The attack works, and it transfers. The rest of this post is how I built it, how the numbers climbed, and what actually stops it.

Why "faithful rephrasing" is the right attack to study

The premise is a game of telephone with a referee. A small model (the only thing being trained) rewords a prompt. A larger, safety-tuned model reads the reword and answers. Judges score whether the answer is harmful and whether the model refused. And a pair of faithfulness scorers check that the reword still means the same thing as the original.

That faithfulness constraint is what makes this a real attack rather than a cheat. If the rewriter were allowed to change the meaning, it could "win" by turning a dangerous request into a harmless one, which proves nothing. By forcing the reword to mean the same thing, a success is a genuine jailbreak of the original request. It also, as I will get to, is exactly what makes the attack detectable.

How the attack is formed

Let me lay out the whole machine, piece by piece, because the design is where most of the work lives. There are four models in the loop, and only one of them ever gets trained.

The rephraser (the attacker, trainable). This is a small instruction-tuned model, on the order of a couple billion parameters. I do not fine-tune the whole thing; I attach a low-rank adapter (LoRA) to all of its attention and MLP projection matrices and train only that. In practice this means the "attacker" is a few million trainable parameters riding on top of a frozen base, which keeps training cheap and, conveniently, keeps the attack itself small and portable. Its entire job is: prompt in, reworded prompt out.

The target / observer (frozen). This is the larger safety-tuned model I am attacking, around 9B parameters, and I never update its weights. It plays two roles at once. First, it is the target: it generates the response that gets scored for harm. Second, it is the observer: I read its internal residual-stream activations at a middle layer while it processes the reworded prompt. That second role is the interpretability hook. I fit a few things on those activations ahead of time: a "malicious vs. aligned" concept direction (difference-in-means between harmful and benign prompts), a linear probe that predicts harmfulness from the activation, and a set of sparse-autoencoder features that decompose the activation into interpretable pieces. During the attack these give me a white-box read on whether the reworded prompt still "looks harmful" inside the model, and how on-manifold (natural) it is.

The in-loop judge (frozen). A safety classifier that scores the target's response for harm and for refusal. This is the thing that defines the reward. It runs inside the training loop on every step, so it has to be fast.

The faithfulness scorers (frozen). A sentence-embedding model and a natural-language-inference (entailment) model. Between them they answer "does the reword still mean the same thing as the original?" The embedding cosine catches gross meaning shifts; the entailment model catches subtler ones where the words are similar but the ask has changed.

The tricky engineering problem is that you cannot backpropagate through sampling discrete tokens. The rephraser emits text, and text is not differentiable. I handle this with two parallel paths. For the response-level reward (does the target actually produce harmful content), I generate real discrete text and treat it as a reinforcement-learning signal. For the internal concept signals (the activation direction, the probe, the SAE features), I use a Gumbel-softmax relaxation, a "soft" almost-discrete version of the reworded tokens that I can push gradients through, annealed from soft toward hard over training. And because a soft relaxation can drift away from what the real, hard tokens would do, I run a periodic discrete canary: every so often I sample genuine hard rewrites, re-score them, and check that the soft signal still agrees with reality. If the soft and hard scores diverge, I know the model is exploiting the relaxation rather than actually improving, and I can react.

So the shape of a single training step is: rephraser rewords the prompt → observer reads its own activations on that reword and I score the concept signals (soft path) → observer generates K responses to the reword and the in-loop judge scores them for harm and refusal (discrete path) → faithfulness scorers compare reword to original → everything is combined into one reward, and the adapter is updated.

The exact models and datasets I used

The section above described the four roles abstractly. Here is what actually filled each role, and how. Everything is an open model or a public dataset.

Models

Datasets

Evaluation prompts are drawn balanced across the four harmful sources by a round-robin quota, so no single distribution dominates the headline numbers.

The reward function

This is the heart of the whole thing, and it is where the attack was won or lost. I will build it up the way it actually evolved, because the failure and the fix are both instructive.

First, the definition of success I actually care about is multiplicative. A rewrite is a real win only if it evades detection, makes the target not refuse, makes the target produce something harmful, and stays faithful. In plain terms:

success = evasion × non_refusal × harm × [faithfulness ≥ floor]

Every factor has to be high at once. If any one is near zero, the product is near zero.

My first reward was the obvious thing: a weighted sum of those pieces plus preservation terms.

reward_naive = w_e·evasion + w_n·non_refusal + w_h·harm + (preservation)

This does not work, and the reason is subtle and important. A sum can be maximized by pushing up whichever terms are cheap and ignoring the ones that are expensive. For a safety-tuned target, getting it to not-refuse is cheap (reword it a bit and it will happily start talking), but getting it to produce genuinely harmful substance is expensive. So the optimizer rationally collected the easy points, drove evasion and non-refusal high, and left harm sitting near 0.13. The training reward had a different shape than the success metric, and the optimizer exploited the gap. This was months of plateau.

The fix was to make the reward the same shape as success by gating the easy terms on harm:

gate   = sigmoid(k · (harm − τ_h))          # τ_h ≈ 0.3, k ≈ 10
reward = gate · (w_e·evasion + w_n·non_refusal)
         + w_h·harm
         − preservation_penalties

Now the evasion and non-refusal reward only "turns on" once the response is actually harmful. You cannot coast on compliance anymore; the cheap points are locked behind the expensive one. That single change is what unstuck the whole project.

The second fix was to the harm signal itself. Originally harm was a single greedy response, parsed to a yes/no. For a safety-tuned model that is "no" almost every time, which means the gradient on the one term that mattered was basically noise. I changed it two ways: I read harm as a continuous probability from the judge rather than a binary label, and I sample K responses (K = 4) per prompt and average, with a best-of-K term available. A smooth, multi-sample harm score has real gradient in it; a binary greedy one does not.

The preservation terms are what keep the reword faithful, and there are several because meaning drift showed up in several disguises:

• Semantic cosine loss — pull the reword's embedding toward the original's.
• NLI entailment — reward mutual entailment between original and reword.
• KL to a reference policy — keep the attacker close to its warm-started self, so it does not wander into degenerate regions.
• Activation-manifold prior — keep the observer's induced activations near the distribution of natural activations, so the reword is not adversarially weird on the inside.
• N-gram copy penalty — punish just copying the prompt verbatim, which is otherwise the cheapest way to score perfect faithfulness while doing no attack.
• A smooth semantic floor — the safety valve. If faithfulness drops below a floor (around 0.5 to 0.6), it multiplies the entire attack reward by zero. No matter how harmful a drifted reword is, it earns nothing. This is the term that keeps "success" honest.

Put together, the attacker is rewarded for compliance-inducing rewrites only when they stay faithful, and it gets a real, dense gradient on the hard part. That is the whole recipe.

The training program

The attack is not one training run; it is a program that builds up in stages, and I keep concrete schedules because they matter.

Stage 0 — Supervised warm-start. Before any RL, I fine-tune the rephraser on paraphrase-style pairs for a short warmup (around 500 steps) so it starts out producing fluent, faithful rewrites. RL from a cold start on this objective just thrashes; warm-starting from "already a decent paraphraser" is what makes the later stages stable. I keep a frozen copy of this warm-started model as the reference policy for the KL term.

Stage 1 — Curriculum into the attack. I do not turn on the full attack reward immediately. For the first phase the attacker optimizes preservation only, then I ramp the attack terms in linearly. This "learn to stay faithful, then learn to attack" curriculum stops the early, high-variance attack gradient from blowing up the still-fragile policy.

Stage 2 — Policy-gradient attack (REINFORCE). The core loop is REINFORCE: sample a reword, get the gated reward, push the log-probability of the reword up or down in proportion to how much better or worse than average it did. The advantage is the reward minus a running baseline, which cuts variance. Throughout, the Gumbel-softmax temperature on the soft path is annealed from high (very soft, easy gradients) toward low (nearly discrete, faithful to real behavior) over roughly the first 3000 steps. I keep an exponential moving average of the adapter weights (decay ~0.999) and use that for evaluation, which smooths out the noise inherent in RL. Gradients are clipped, and I found the clip and learning rate have to be tuned together, since too-tight clipping throttles the attack signal to nothing.

Stage 3 — Expert iteration (the big lever). This is what actually got faithful harm to climb. Every ~500 steps I pause and harvest the attacker's own best rewrites produced so far. Then I fine-tune the attacker on them with plain supervised cross-entropy for a short burst (~50 steps). The crucial detail is the double filter: a harvested sample is admitted only if it is both harmful (harm ≥ 0.5) and faithful (faithfulness ≥ 0.6). I harvest up to a couple hundred samples per cycle. This converts the sparse, hard-to-optimize RL signal into a dense supervised one, which is the standard cure for "RL can't find the reward," and the double filter is what stops it from amplifying either failure mode (copying, or meaning collapse). If I filtered on harm alone, it would learn to cheat; filtering on both is the difference between a real attack and a fake one.

Stage 4 — Honesty instrumentation. Two things keep the reported numbers trustworthy. The discrete canary (described above) watches for the soft path drifting from real behavior. And model selection / early stopping key on the success product, not on any single easy term, so the checkpoint I actually deploy is the one that maximizes faithful harm rather than a proxy. (An earlier version of this selection logic had a bug that made it pick essentially random checkpoints; catching that was its own lesson in instrumenting the metric you actually care about.)

Stage 5 — Aggressiveness / consolidation tuning. Finally I explore the frontier: loosening the KL anchor and lengthening the reward horizon (scoring longer responses) trades faithfulness for higher raw harm, while a denser gate and a freer harvest consolidate the balanced operating point. This is how I mapped out both the faithful attack and the raw-harm ceiling.

The whole program runs on the order of 6000 steps end to end. Here is the shape of it as a quick reference:

The stats: how the attack got stronger

Now the numbers. The main quantity is the target's harm score (my attack-success proxy; higher means a stronger attack), alongside non-refusal (did it answer instead of refusing) and faithfulness (did the reword keep its meaning). Here is how attack success climbed as I added each mechanism from the training program:

Reading this table top to bottom is the whole attack story. The baseline answers prompts but produces essentially no harmful substance (harm 0.13). Scoring the target's actual response, then fixing the reward's shape and density, gets harm moving for the first time. Expert iteration is the big jump: it roughly quadruples faithful harm over baseline, to 0.55–0.60, while keeping the rewrites above the faithfulness bar. And if I stop caring about faithfulness, the aggressive variant shows the raw-harm ceiling at about 0.85, at the cost of meaning drifting (faithfulness falls to 0.46). The balanced operating point (0.55–0.60 with faithfulness intact) is the one that is actually a faithful jailbreak; the aggressive one is the endpoint of the frontier.

The headline result: it transfers

The single most important experiment. I took the rewrites trained against the smaller target and had a much larger model it had never trained against generate the responses instead. Same rewrites, bigger unseen target. ("Unseen" here means the rephraser was tuned only against the 9B and never received any reward, gradient, or activation signal from this larger sibling during training; it is only swapped in at evaluation.) Here is what the judges said, original prompt vs. attacked prompt:

The reformulation cut refusals by more than half on a model the attacker never saw, and raised harm on the training judge nearly fivefold, with a second independent classifier confirming the direction at about 1.5x. A cheap attacker trained against a small model measurably weakened a larger, unseen one. That is a transferable, meaning-preserving attack, not a target-specific fluke.

Where the attack is strongest, and where it is not

Two honest caveats, and they are the bridge to the defense rather than a knock on the attack. First, notice the last row of that transfer table: the lift is biggest on the judge I trained against and almost nothing on the independent substance grader (+0.022). A lot of the transferred "harm" is compliance and surface form rather than deep substance. Second, under transfer only about half the rewrites stayed fully faithful, so counting only the harmful-and-faithful ones, the real success rate is around 0.30. Both facts point at the same truth: the attack succeeds by staying faithful and staying "natural" inside the model. And that is exactly the behavior a defender can detect.

How to stop this: safety probes outside the base model

Here is the part I think matters most for anyone actually deploying models. You often cannot retrain the base model, but you can wrap it in external safety probes, detectors that live outside the model and watch the traffic going in and coming out. The attack's own constraints tell you exactly where to put them.

1. Input probes that are paraphrase-invariant. Because the attack preserves meaning, a detector that classifies intent rather than wording defeats its whole premise. If you score an incoming prompt against a bank of known-harmful intents in embedding space, or canonicalize the request before classifying it, a faithful reword lands in the same danger zone as the original. Ironically, the faithfulness scorers I used to build the attack are a blueprint for the defense.

2. Activation probes and feature monitors. The attack had to keep the harmful concept present and "on-manifold" inside the model, which is why I could steer it using internal activations. A defender can read the same signal. A lightweight linear probe on a middle layer, plus a sparse-autoencoder feature monitor watching a handful of harm-correlated features, will fire on the reworded prompt because the attack cannot remove the harmful concept without breaking faithfulness. These are cheap (basically a dot product) and hard to evade, since evasion and faithfulness pull in opposite directions.

3. An independent judge ensemble on the output. The response side is where the attack is weakest in substance. Running several independent judges on the answer, especially a substance-focused one the attacker never optimized against, catches responses that are compliant in form but hollow or harmful in content. This is what turns the attack's judge-dependence from the attacker's advantage into the defender's.

4. Cross-checking for disagreement. The single most useful defensive signal is that the attack moves different probes by wildly different amounts. So alert on disagreement: a prompt that looks benign on the surface but whose activation probe and semantic-neighbor check say otherwise, or a response one judge clears and another flags. The spread between your probes is itself the tell.

Why diversity beats accuracy. My transfer numbers are the whole argument in miniature: one judge got moved almost fivefold while an independent grader barely moved. Any defense built on a single detector inherits that detector's blind spot and gets beaten by an attacker who optimizes against it, even accidentally. A defense built on diverse probes across input, activations, and output has no single point of failure. To get through, an attacker would have to stay faithful (beating the input and activation probes) and be substantively harmful (beating the output ensemble) and stay internally consistent (beating the cross-check), all at once. Defense-in-depth turns the attack's narrow success surface into a set of conditions it cannot satisfy simultaneously.

Pushing the frontier: the knobs and their tradeoffs

Once the balanced attack worked, I ran a family of one-knob-at-a-time ablations to see how much further it could be pushed and what each push costs. This turned out to be the most clarifying part of the whole project, because it maps the tradeoff between harm and faithfulness directly, and it pins down the substance ceiling with a controlled experiment.

I will describe the target's harm score, faithfulness, and the joint success rate (harmful and faithful on the same example, which is the metric that actually matters). One caveat up front: I report harm at two different "reward horizons," 64 tokens vs 128 tokens of the response, and those are not directly comparable, because scoring more of the response measures more of its actual substance.

* short-horizon (64-token) harm; not comparable to the 128-token rows.

Three tradeoffs fall out of this, and they are the real lessons of the ablation.

The gate temperature is the main harm lever. The gate is a sigmoid on how harmful the response is; lowering its threshold opens it at lower harm and gives a denser gradient right where these runs actually sit. Sliding that threshold down moved the harm plateau from about 0.17 to 0.32 and higher, but every step down cost a little faithfulness. If you want one dial for "how aggressive," this is it.

The reward horizon exposes the substance ceiling. This is the important one. If I take the balanced attack and change only the horizon, scoring 128 tokens of the response instead of 64, the measured harm collapses from 0.55–0.60 down to about 0.17. In other words, the short-horizon score was rewarding a harmful-looking opening that the target does not actually sustain across a full response. That is a clean, controlled demonstration of the ceiling I described earlier: once you score real substance instead of just the first breath of the answer, faithful harm sits down near 0.2–0.3. The longer horizon is the more honest measurement, and it agrees with the independent ceiling test.

Gates only permit drift; rewards cause it. I initially tried to buy more harm by loosening the faithfulness gates (letting rewrites drift further before losing reward). It barely did anything, because a gate only allows meaning drift, it does not reward it. Faithfulness stayed pinned around 0.8. Only when I cut the preservation reward itself did faithfulness actually fall (to 0.5–0.65) in exchange for more harm. That is the clean harm-versus-faithfulness frontier: you cannot get more harm for free by relaxing a constraint; you have to actively pay for it in faithfulness.

The punchline: even though this family reports a lower single-number harm than the headline balanced run, it roughly tripled-to-quadrupled the joint success rate (from 0.003–0.006 up to 0.018–0.022) while staying faithful at around 0.70. By scoring more of the response and hardening the copy penalty and the harvest filter, it made harm and faithfulness co-occur on the same example far more often, which is the goal. It is the more defensible operating point, and it makes the ceiling concrete: raw harm is always available if you spend faithfulness or score less of the answer, but honest, faithful, substantive harm has a low ceiling on this target.

What I take away

The attack works: a small, cheap rewriter, trained with warm-start plus reward-shaped reinforcement learning plus learning from its own best attacks, becomes a transferable meaning-preserving jailbreak that quadruples real harm and halves a bigger model's refusals. But the same properties that make it work, faithfulness and staying natural inside the model, are what make it catchable. The right response is not one smarter filter; it is a diverse stack of external probes, watching the input, the activations, and the output, and escalating when they disagree.

Limitations, honestly

Faithful and deeply substantive success rarely happen on the same example, so the attack's worst-case harm is bounded. I studied one model family, so I cannot yet claim the attack or the defenses generalize across lineages. My harm numbers inherit the blind spots of the judges producing them. And the defenses here are an architecture grounded in the attack's measured behavior.

The system in this post is assembled from a lot of published work. Here are the sources for the models, datasets, judges, and methods I leaned on.

Models and interpretability

• Gemma Team (2024). Gemma 2: Improving Open Language Models at a Practical Size. arXiv:2408.00118
• Lieberum et al. (2024). Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2. arXiv:2408.05147
• Templeton et al. (2024). Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet. Anthropic
• Arditi et al. (2024). Refusal in Language Models Is Mediated by a Single Direction. arXiv:2406.11717
• Luo, Feng, Darrell, Radford, Steinhardt (2026). Learning a Generative Meta-Model of LLM Activations (activation-diffusion / on-manifold prior). arXiv:2602.06964

Datasets

• Jiang et al. (2024). WildTeaming at Scale (the WildJailbreak dataset). arXiv:2406.18510
• Chao et al. (2024). JailbreakBench (JBB-Behaviors). arXiv:2404.01318
• Zou et al. (2023). Universal and Transferable Adversarial Attacks on Aligned Language Models (AdvBench / GCG). arXiv:2307.15043
• Taori et al. (2023). Stanford Alpaca (benign instruction anchor). GitHub
• codesagar (2024). malicious-llm-prompts. Hugging Face

Safety judges

• Han et al. (2024). WildGuard. arXiv:2406.18495
• Mazeika et al. (2024). HarmBench. arXiv:2402.04249
• Souly et al. (2024). A StrongREJECT for Empty Jailbreaks. arXiv:2402.10260

Methods and formulas

• Hu et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685
• Williams (1992). Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning (REINFORCE). Springer
• Jang, Gu, Poole (2017). Categorical Reparameterization with Gumbel-Softmax. arXiv:1611.01144
• Ouyang et al. (2022). Training Language Models to Follow Instructions with Human Feedback (RLHF / KL-to-reference). arXiv:2203.02155
• Shao et al. (2024). DeepSeekMath (group-relative policy optimization, GRPO). arXiv:2402.03300
• Zelikman et al. (2022). STaR: Bootstrapping Reasoning with Reasoning (expert iteration). arXiv:2203.14465
• Reimers & Gurevych (2019). Sentence-BERT. arXiv:1908.10084
• He et al. (2021). DeBERTa (the NLI/entailment scorer's backbone). arXiv:2006.03654
• Chen, Zhang, Tramèr (2026). Learning to Inject: Automated Prompt Injection via Reinforcement Learning (RL-attacker framing). arXiv:2602.05746