From the DeepSeek-V4-Flash-0731 interpretability project
We gathered texts from 9 religious traditions plus a large collection of Christian literature. Here's the full inventory:
| Text | Passages | Tokens | Digit density | Where it came from |
|---|---|---|---|---|
| KJV Bible | 1,189 | 1,045,776 | 0.0000% | Project Gutenberg, verse numbers stripped |
| Christian literature (wave 1) | 1,267 | 20,409,440 | ~1.1% | Project Gutenberg, 3,705 books across 32 topics |
| Christian literature (wave 2) | 2,295 | ~37M | varies | Project Gutenberg, extended topic set |
| Qur'an (Pickthall translation) | 115 | 258,122 | varies | Project Gutenberg |
| Book of Mormon | 30 | 342,614 | 0.000% | Project Gutenberg |
| Bhagavad Gita | 18 | 29,690 | low | Project Gutenberg |
| Tao Te Ching | 81 | 13,852 | low | Project Gutenberg |
| Dhammapada | 26 | 16,203 | low | Project Gutenberg |
| Analects (Confucius) | 20 | 42,041 | low | Project Gutenberg |
| Upanishads | 3 | 21,669 | low | Project Gutenberg (too few for strong claims) |
| Total | 2,749 | 22,179,407 | ||
The "digit density" column matters a lot, as it turned out. The Bible is the only text at exactly 0%, because we stripped verse numbers from it. That created the confound that led to our retracted "scripture detector" claim.
For targeted experiments, we built custom corpora:
| Experiment | Passages | Tokens | Status | What's in it |
|---|---|---|---|---|
| Exp 1: 12 Bible translations | 360 | 613,344 | done | Same 30 passages in 12 English translations (KJV, WEB, ASV, YLT, BBE + NIV, ESV, NLT, NRSV, NASB, CSB, MSG). All with digits stripped. |
| Exp 4b: Commentary with quotes | 351 | 5,156,659 | done | 351 chunks of Christian commentary that contain embedded Bible quotes. We annotated where the quotes are so we could compare routing on quotes vs. routing on commentary. |
| Exp 12: Digit pairs | 222 | 437,900 | done | 111 pairs of the same text, one with digits and one with digits removed. 5 categories: dates, lists, statistics, scripture citations, misc numeric. |
| Exp 13: Ablation corpus | 480 | 418,695 | staged | 4 groups: verse-religious, prose-religious, verse-secular, prose-secular. 120 passages each. For the causal ablation test. |
| Source | How much | Notes |
|---|---|---|
| Wikipedia theology articles | 7,508 articles | Started from 6 seed topics, followed categories 3 levels deep |
| Project Gutenberg theology books | 3,827 books | 40+ topic searches; about 400M tokens total |
| Project Gutenberg Christian books | 3,705 books | 32 topics spanning patristics through modern |
| Secular sources (for Exp 13) | 5 works | Shakespeare sonnets, Whitman, Dickinson, Darwin, Thoreau |
Each piece of text becomes two files: the input (what we fed the model) and the observation (what we recorded).
{"sample_id": "bible_gen_001", "category": "bible",
"sample_index": 0, "seqlen": 880, "token_ids": [...],
"text": "In the beginning God created..."}
{"category": "bible", "sample_index": 0, "seqlen": 880,
"observation": {"layers": {"0": {"expert_frequencies": [12, 0, 3, ...],
"gate_weights": [...], "activation_norms": [...]}}}}
The expert_frequencies array has 256 numbers, one per expert, counting how many tokens selected that expert at that layer.
| Dataset | What's in it | Link |
|---|---|---|
| Consolidated (45 files) | Everything: observations, J-lens data, manifests, analysis, code | 0xSero/deepseek-v4-flash-reap |
| Raw observations (11 files) | Streaming parquet shards from the observation runs | 0xSero/deepseek-v4-flash-religious-reap-observations |