Improving Activation Steering

Gated Cropped Attention-Delta Steering fixes KV-cache contamination in multi-turn dialogue

Improving Activation Steering

This page describes Prompt–Activation Duality: Improving Activation Steering via Attention-Level Interventions, submitted to NeurIPS 2026.

Activation steering controls language model behavior by adding a direction in the residual stream at inference time — a lightweight, reversible alternative to fine-tuning. But standard residual-stream steering has a hidden failure mode in stateful, multi-turn dialogue: steered token states get stored in the KV cache and repeatedly reused, turning a local perturbation into cumulative coherence degradation across conversation turns.

We identify this failure mode as KV-cache contamination and show that coherence deteriorates across turns even when single-turn behavior looks strong. Crucially, prompt-only control remains stable under the same protocol — so the problem is not long context alone. The intervention is entering the computation at the wrong site.

The diagram below shows how the contamination loop builds up across turns, and why prompt-only control avoids it.

Standard residual-stream steering states written to cache re-read by attention Turn 1 +Δ in residual stream Turn 2 +Δ in residual stream Turn 3 +Δ in residual stream KV cache sys prompt turn-1 steered states turn-2 steered states turn-3 steered states Each turn re-reads steered entries — the perturbation compounds and coherence drifts Prompt-only control (same multi-turn protocol) Turn 1 control via sys prompt Turn 2 control via sys prompt Turn 3 control via sys prompt KV cache system prompt turn-1 clean states turn-2 clean states turn-3 clean states Cached states stay clean — coherence remains stable across turns

Figure: The KV-cache contamination loop under residual-stream steering (top), absent under prompt-only control (bottom).

Coherence comparison across turns Method overview

To address this, we propose Gated Cropped Attention-Delta steering (GCAD), which extracts steering signals from system-prompt contributions to self-attention and applies them with token-level gating. Rather than injecting a large residual-stream perturbation after attention and MLP computation have been combined, GCAD introduces smaller attention-level perturbations that subsequent layers can transform and integrate — following the same pathways through which system prompts already exercise behavioral control.

The diagram below contrasts the two intervention sites inside a transformer layer.

Standard residual steering one large Δ, injected post-block GCAD: gated, cropped, attention-level small gated Δ along the prompt pathway hidden state h Self-attention MLP + residual add steering vector α·v (large Δ) added after attn + MLP outputs are combined later layers resist the perturbation KV cache steered states stored and reused hidden state h Self-attention system-prompt contribution to attention (cropped) small Δattn g token-level gate MLP MLP and later layers transform and integrate the small delta + later layers no downstream resistance KV cache states follow the prompt pathway

Figure: Where each intervention enters the computation — post-block residual injection vs. GCAD's token-gated, attention-level delta.

On persona-steering experiments with Qwen2.5-7B-Instruct:

  • Average coherence drift: improved from −18.6 to −1.9 (standard steering vs. GCAD)
  • Turn-10 trait expression: raised from 78.0% to 93.1%
  • Trait control is preserved while long-horizon stability improves substantially
Trait expression comparison Combined results

Mechanistic analysis shows that GCAD produces smoother perturbation trajectories that better align with downstream computation, while standard residual-stream steering elicits downstream resistance. The results suggest that activation steering becomes reliably usable in production settings only when interventions follow the prompt-mediated pathways that models already use for behavioral control.

Next Project

KINA Benchmark

 KINA Benchmark