From the DeepSeek-V4-Flash-0731 interpretability project

Forward plan: next experimental phase

Quick orientation for anyone new to the project. DeepSeek-V4-Flash-0731 is a mixture of experts language model: it has 256 small sub-networks (we call them experts) at each of its 43 layers, and for each token (roughly each word) a router picks 6 experts to do the work. We feed the model text and record which experts it picks. Our main measurement, REAP, is the router's gate weight for an expert times that expert's activation norm, a single number summarizing how much each expert contributed.

The finding we are now trying to nail down is H6: a group of 6 experts at layer 42 that fire strongly on prose (flowing, paragraph-shaped text) and go nearly silent on verse (short lineated lines, like poetry or scripture). We also found e164, expert 164 at layer 42, which fires on digit characters. We once thought e164 detected scripture, but that was a mistake in our own data pipeline, which had stripped verse numbers from the Bible, leaving it with no digits. The forward plan below is about proving the H6 finding causally and ruling out the remaining alternative explanations.

1. Rules we follow

Every experiment in this project follows four rules. They exist because we got things wrong before, and these rules are how we keep from repeating it.

• Read-only. We never change the model's weights, remove experts, or compress it. We only feed text in and watch what happens. This keeps the model intact and our observations comparable across experiments.
• Control digit density. Because e164 fires on digits, every experiment must track how many digits are in the text and hold that number steady when comparing groups. This is the control that exposed our original error.
• Fail-closed invariants. Every observation is checked against hard rules (for example, the total number of expert firings must equal the number of tokens times the number of experts picked per token, and there must be no NaN or Inf values). If a check fails, the record is rejected before it enters the dataset.
• External review before publication. Claims go to outside reviewers before we publish them. The e164 correction came from an external review.

2. Exp 13: the H6 ablation

This is the most important next experiment. An ablation means temporarily switching some experts off and seeing what changes. Here we switch the 6 H6 experts off and measure how much worse the model gets at predicting the next token, using negative log likelihood (NLL), a standard measure of prediction error where lower is better. If the error rises sharply on prose but stays flat on verse, that is direct evidence H6 is what drives the prose-versus-verse difference, not merely correlated with it. See the Exp 13 page for the full design.

3. Priority order and what each one answers

1. Exp 13, the ablation. Highest priority, because it is the only experiment that can turn our correlation into a causal claim. Until we run it, we know H6 fires differently on prose and verse, but we do not know that H6 causes the difference.
2. Exp 11, the context dump. We look at the surrounding text at the exact moments when tradition-specific experts (ones that fire on only one religion) switch on, to learn what actually triggers them.
3. Exp 3, the secular null. Text in the same formats and styles as our religious texts but with no religious content. This answers whether anything we have seen is really about religion or only about ordinary English prose and verse.
4. Theology observation. Text about named figures (Jesus, Lucifer, Moloch, Saturn) and about Judaism, to see whether specific theological content changes routing.

4. New experiments we are proposing

• Format-edit factorial. Take the same text and present it two ways, as verse (lineated) and as prose (flowing), and do this for both religious and secular text. A factorial design tests two variables at once, here format and religious content, across all four combinations. If format drives H6 and religion does not, the prose cells fire and the verse cells do not, regardless of whether the content is religious.
• Secular factorial. The same format comparison but with no religious content at all. This is the cleanest test of whether H6 is purely a format detector with no hidden dependence on religious subject matter.
• Lens robustness. We re-run our J-space lens probes (the method that peeks at the model's intermediate predictions layer by layer) split by digit density, to check whether digits are distorting those results the way they distorted e164.

5. Visualization plan

• Interactive H6 explorer. A tool where you pick a passage and see which H6 experts fire on it and where, so the pattern is visible without reading raw numbers.
• Per-layer routing animations. Show how expert selection changes as a token passes through the 43-layer stack, so you can see at which layer the prose-versus-verse split emerges.
• MSG versus KJV side-by-side. The same Bible passage in the Message paraphrase (which reads like prose) and the King James Version (which is verse-formatted), shown together, so you can see how the format change alone alters the routing.

6. The story we are telling

The arc, in plain terms: we asked whether a language model routes religious text differently from other text. The answer is no. But along the way we found something we did not expect, a set of experts that detect whether text is laid out as verse or as prose, and a single expert that detects digits. The forward plan is about proving the verse-versus-prose finding causally, ruling out the remaining confounds, and being honest about the error we made and corrected along the way.