From the DeepSeek-V4-Flash-0731 interpretability project
We wanted to know whether a mixture of experts language model routes religious text differently from secular text. DeepSeek-V4-Flash-0731 has 256 small expert networks at each of its 43 layers, and for every token it reads, a gate picks just 6 of them to do the work. So the question was simple: does that gate send Bible verses to different experts than it sends, say, a Wikipedia article? Are there "Bible experts" or "Qur'an experts" hiding in there?
No. The model does not route by religion. What we found instead is a large group of experts we call H6 that acts as a verse-versus-prose detector. These experts fire strongly when the model reads flowing prose and go nearly silent when it reads text broken into verse lines. This holds regardless of which religion the text comes from, which translation it is, or which century it was written in.
Our first headline finding was that expert number 164 at layer 42 was a "not-memorized-scripture detector." It fired at zero on the Bible and high on everything else. We were excited. We were also wrong. An external review by Claude Opus 5 caught the confound: our pipeline had stripped verse numbers from the Bible, making it the only text in our collection with zero digits. Every other text contained digits. So expert 164 was not detecting scripture at all. It was detecting digits. Experiment 12 settled this: the expert fired 1,453 times harder on text containing digits than on the same text with digits removed.
What does survive scrutiny is H6: a cluster of about 13 experts that fire at up to 163,284 times per million tokens on prose and at essentially zero on verse. This is a format detector. It responds to whether text is laid out in short verse lines or written in flowing paragraphs. It does not care what the text is about. Here is the evidence:
Of the 20 most active experts at layer 42, 15 are shared between the Bible corpus and the Christian literature corpus. All nine religious traditions in our study share a common routing backbone. No tradition has a uniquely isolated set of experts that only it uses.
We used a technique called the logit lens to peek at the model's running guess at each layer. Across all traditions, the model's next-word predictions stabilize around layers 30 to 35. Earlier layers carry more input-level leverage: when we measured how much a tiny nudge to the input changes the output at each layer (the bounded Jacobian), layer 0 was about 2.5 to 3.5 times more sensitive than later layers. No tradition behaves qualitatively differently from the others.
We hypothesized that texts which use fewer experts would be more predictable (H3). This was wrong. The correlation goes the other way: it is positive, at +0.47. The gap we had seen between two texts, one using 65 effective experts and the other 111, was an artifact of how we chunked the text into windows. We retract H3.
Every record had to pass fail-closed integrity checks before we accepted it. The expert selection frequencies had to add up to exactly the sequence length times the number of experts picked per token, with no NaN or infinity values and correct array shapes. We had zero violations across all 3,682 records. We also verified our interception code line by line against the model's actual gate logic, so we are confident we recorded what the model really does.
| Hypothesis | Status | Evidence |
|---|---|---|
| H1: Expert 164 is a digit detector | confirmed | Exp 12: 1,453x ratio between text with and without digits |
| H2: Routing follows surface form, not content | confirmed | Exp 1: the same passages in 12 translations route the same way |
| H3: Routing concentration tracks predictability | retracted | Correlation is +0.47, the opposite of what we predicted |
| H6: There is a verse-versus-prose format axis | confirmed | Exp 1, 4b, and 12: 933 records, 6,207,903 tokens |