From the DeepSeek-V4-Flash-0731 interpretability project

Experiment 1: the 12-translation Bible test

The question we were trying to answer

H6 is our name for a group of six experts that fire heavily on prose and go quiet on verse. Before this experiment we had only seen that pattern on one corpus of Bible text. We wanted to know whether the pattern holds across translations, or whether it was just an accident of whichever translation we happened to feed the model.

So we picked 30 passages spanning Genesis through Romans and rendered each one in 12 English translations, from the King James Version (1611) to the Christian Standard Bible (2017). That gives 360 records and 613,344 tokens of text in total. We stripped every verse number first, so there are zero digits in any of the text. That matters because we already knew digits confuse the picture (see Experiment 12), and we wanted to remove that variable.

The 12 translations

They span four centuries and several translation philosophies, from ultra-literal word-for-word to loose paraphrase:

What H6 did on each translation

The table below shows how often H6 fired, measured in "firing rate per million tokens." That number means: for every million tokens of text we feed the model, how many times did the gate pick one of the six H6 experts? A low number means H6 stayed quiet. A high number means H6 was active.

Two integrity checks passed. First, the "invariant check," a running audit that confirms our expert-frequency numbers add up correctly, reported 0 violations across all 360 records. Second, our digit-detecting expert e164 fired only 1.6 times per million tokens across the whole corpus, with a single fire total. That confirms the digit confound was controlled: H6 is not reacting to digits here.

The Message outlier

One translation jumps off the page. The Message (MSG) fires at 1,033/M, which is 6 to 60 times higher than any other translation. This is not a broken translation or a failure of our test. It is the strongest piece of evidence we have for the format hypothesis.

The Message is a paraphrase. Its translator, Eugene Peterson, rewrote the Bible as conversational prose, abandoning the short line-by-line verse layout every other translation keeps. From the model's perspective, MSG reads as prose, not verse. So H6 fires on it the same way it fires on any prose.

Eight of the 30 passages show H6 = 0 across all 12 translations, including MSG. Those are the most tightly verse-structured passages, the ones where even The Message keeps enough verse-like formatting that H6 stays quiet.

Charts

What we learned

1. H6 stays near zero on verse-formatted text. KJV and ESV fired at a perfect 0.0 across 99,765 tokens combined. The other 10 verse translations fired at only 17 to 171 per million, which is negligible compared to the 163,284/M we see on prose commentary.
2. The Message confirms the format hypothesis. It fired at 1,033/M, not because it is a bad translation, but because it is formatted as prose.
3. Translation invariance holds for verse. The KJV (1611) and ESV (2001) both fired at 0.0. The era, the register, and the translation philosophy do not matter. Only the format matters.
4. The digit confound was controlled. e164 fired at 1.6/M. All 360 records had 0.0000% digits.

What this means

1. H6 is a format detector. The Message result is decisive. Abandon verse format and H6 fires. Preserve verse lineation and H6 stays near zero.
2. Taken with Exp 4b and Exp 12: H6 is confirmed across 933 records and 6,207,903 tokens. It is the strongest routing effect we found in the whole study.