From the DeepSeek-V4-Flash-0731 interpretability project

Experiment 13: the H6 ablation test

Why a causal test

Experiments 1, 4b, and 12 establish that H6 correlates with verse/prose format. But correlation is not causation. It is possible H6 fires on prose without actually doing anything useful, the way a smoke alarm might blink whenever the oven is on without being wired into the alarm system. Experiment 13 asks the direct question: does H6 causally contribute to how the model processes prose versus verse, or is it just along for the ride?

The 6 H6 anchor experts

These are the six experts we plan to knock out, one or several at a time:

Two ways to knock an expert out

1. Contribution knockout: The gate still picks the expert (so the model "tries" to use it), but we zero out its output. This measures how much that expert directly contributes to the model's computation.
2. Route-mask compensation: We stop the gate from picking the expert at all, so the gate has to pick the next-best expert instead. This measures whether the model can compensate when its preferred expert is unavailable.

The test corpus

We built a 2x2 design that crosses format (verse vs. prose) with content (religious vs. secular). That lets us test whether the effect is specific to religious text or general to all text.

What we will measure

The primary outcome is the change in negative log-likelihood (NLL) when the H6 anchors are knocked out. NLL is a standard measure of how surprised the model is by the next token: lower is better, and a rising NLL means the model is getting worse at predicting the text. If H6 is causal, knocking it out should raise NLL more on prose than on verse. In plain terms: removing H6 should hurt the model more when it is reading prose.

Controls

• 20 matched control experts, randomly selected but matched for frequency and REAP rank (how strongly and how often they are chosen). Knocking these out gives us a baseline for comparison.
• A dose series: knock out 1, then 2, 3, 4, 5, then all 6 anchors, to see whether the damage scales with how many we remove.
• The secular verse/prose cells test whether the effect is specific to religion or general to format.

The hook that does it

The ablation is implemented in a hook (exp13_ablation_hook.py, 14.6KB) deployed to both TP2 nodes. It injects into the model's forward pass to zero out or mask the H6 anchor experts. We respect the read-only constraint of the study: the hook modifies the forward pass during observation only, never the model's stored weights.