From the DeepSeek-V4-Flash-0731 interpretability project
Experiment 4: the quotation switch test
The setup
A lot of Christian writing is a mix: a paragraph of commentary, then a quoted Bible verse, then more commentary. This gave us a natural experiment. If H6 fires on these documents, is it reacting to the verse quotes embedded inside, or to the prose commentary around them?
We took 30 windows of about 16,000 tokens each from Christian books that quote the King James Bible. Every window contains both quoted verse and surrounding commentary. The question is which part H6 is reacting to.
Two kinds of text in the sample
| Group | n | Description | Digit density |
|---|---|---|---|
| KJV files (verse format) | 5 | Bible text with chapter:verse numbering | 2.3–6.1% |
| Commentary books (prose) | 24 | Sermons, theology with embedded Bible quotes | 0.0–1.0% |
Pilot results (Exp 4a)
Our first pass compared the two kinds of text at the window level. "H6 composite (M)" is the combined firing rate of all six H6 experts, per million tokens. The "vs. Bible verse" column shows how many times higher that rate is than the rate on pure Bible verse.
| Group | n | Tokens | H6 composite (M) | vs. Bible verse |
|---|---|---|---|---|
| Pure Bible verse (core data) | 1,189 | 1,045,776 | 4.1 | baseline |
| KJV files (verse+apparatus) | 5 | 81,920 | 8,664 | 2,109× |
| Commentary books (prose+quotes) | 24 | 367,406 | 44,822 | 10,911× |
The pattern is stark. Pure Bible verse fires H6 at about 4/M. The KJV files, which are verse text plus the chapter and verse apparatus around it, fire at 8,664/M. The commentary books fire at 44,822/M, roughly 11,000 times higher than pure verse. H6 is reacting to the prose, not the verse.
Pilot charts
| Chart | What it shows |
|---|---|
| H6 comparison | H6 composite rates across 4 groups (log scale) |
| Per-anchor | Per-anchor H6 rates: verse vs commentary |
| Digit density scatter | Digit density vs H6, colored by text type |
Pilot findings
- H6 fires on the prose context, not the verse content. Commentary with quotes fires at 44,822/M, the same as pure commentary. The verse quotes inside do not suppress H6.
- All 6 anchors fire more on commentary than on KJV files. No single expert breaks the pattern.
Where the pilot was weak
- Digit density was a confound. The KJV files had 2.3 to 6.1% digits (from chapter:verse numbering) while the commentary had 0 to 1%.
- The sample was small: 30 windows, only 5 KJV files.
- We did not annotate exactly which spans inside each window were verse quotes. The pilot compared whole windows, not quote spans.
Exp 4b: the scaled version (351 records)
Exp 4b fixes all three of those limitations. It uses 351 records and 5,156,659 tokens, with 41,426 tokens of explicitly annotated verse quotes drawn from 18 sources. Now we know, token by token, which parts are verse and which are prose.
| Metric | Value |
|---|---|
| Records | 351 |
| Tokens | 5,156,659 |
| H6 rate (commentary prose) | 163,284/M |
| Quote fraction vs H6 correlation | r = −0.13 |
| Invariant violations | 0 |
The −0.13 correlation between quote fraction and H6 means: the more verse a document quotes, the slightly less H6 fires, but the effect is weak. H6 fires on the prose context regardless of how much verse is quoted inside it. This confirms the pilot finding at scale. H6 is a prose detector, not a verse detector and not a "non-scripture" detector.
