From the DeepSeek-V4-Flash-0731 interpretability project
These scripts download the religious and secular texts we fed to the model. They are scrapers: they visit Project Gutenberg and Wikipedia, download books and articles, and save them as raw text files.
| Script | What it does | Output |
|---|---|---|
scrape_christian.py | Downloads Christian literature books from Project Gutenberg | corpus/christian/raw/*.txt |
scrape_theology_books.py | Downloads theology books from Gutenberg across 40-plus topics | corpus/theology/raw/book_*.txt (3,827 books) |
scrape_wikipedia.py | Downloads Wikipedia articles about theology using Wikipedia's API | corpus/theology/raw/wiki_*.txt (7,508 articles) |
build_exp1_translations.py | Fetches 5 public-domain English Bible translations from bible-api.com for Experiment 1 | corpus/samples/exp1_translations.jsonl (150 records) |
build_exp1_copyrighted.py | Fetches 7 copyrighted Bible translations from Bible Gateway, also for Experiment 1 | corpus/samples/exp1_all_translations.jsonl (360 records) |
Raw downloaded text is not ready for the model. These scripts clean it up: they split long books into chunks short enough for the model to read in one pass (at most 16,384 tokens), strip verse numbers from the Bible, and convert the text into the token format the model expects. They also build the special paired corpora we needed for individual experiments.
| Script | What it does | Output |
|---|---|---|
prepare_corpus.py | Splits the 8 core religious texts into model-sized chunks and strips verse numbers from the King James Bible. This is the step that accidentally created the digit confound. | corpus/samples/{{bible,quran,...}}.jsonl |
prepare_christian_corpus.py | Does the same chunking and tokenizing for the Gutenberg Christian literature corpus | corpus/samples/christian.jsonl |
prepare_theology_corpus.py | Does the same for the theology corpus | corpus/samples/theology.jsonl |
select_theology.py | Picks a balanced subset of the theology corpus, setting quotas per topic so no single topic dominates | corpus/samples/theology_sel.jsonl (1,090) |
build_exp12_digit_minimal_pairs.py | Builds 111 pairs of text for Experiment 12: the same passage twice, once with digits and once with digits removed, across 5 categories (dates, lists, statistics, scripture citations, misc numeric) | corpus/samples/exp12_digit_minimal_pairs.jsonl (222) |
build_exp4b_v2.py | Builds 351 chunks of Christian commentary that contain embedded Bible quotes, and marks exactly where each quote begins and ends so we can compare routing on the quote versus the surrounding prose | corpus/samples/exp4b_quotation_switch.jsonl |
build_exp13_ablation_corpus.py | Builds the corpus for Experiment 13: 4 groups of text (verse-religious, prose-religious, verse-secular, prose-secular), 120 passages each | corpus/samples/exp13_ablation_corpus.jsonl (480) |
These are the scripts that actually run the model and record which experts it picks. The model is too large for one GPU, so it runs split across two machines working together (we call this "TP2," for tensor parallelism across 2 devices). The main script never modifies the model. It only reads.
| Script | What it does | Notes |
|---|---|---|
observe_religious.py | The heart of the study. This is the script that loads the model, feeds it text, and records which experts the gate selects for every token at every layer. It never lets the model generate output. It also runs the integrity checks on every record before accepting it. | We verified this script line by line against the model's own gate code to make sure we record what the model really does. |
run_full_observation.sh | The launch script that starts the model on both GPU machines and waits for them to connect to each other before observation begins | Machine passwords are passed through environment variables only, never written into the script |
chain_next_runs.sh | A supervisor script that runs experiments one after another, detects when a run has stalled, and relaunches it | Keeps a log at /tmp/chain_runs.log |
exp13_ablation_hook.py | A hook for Experiment 13 that temporarily zeroes out the H6 experts to see how much that hurts the model's performance. This is the one experiment that intervenes rather than just observing. | Deployed to both machines, not yet run |
Once observation is done, these scripts crunch the recorded data: they count how often each expert was selected, compute REAP scores, compare which experts different texts share, and generate the charts and reports you see in this wiki.
| Script | What it does | Output |
|---|---|---|
analyze_experts.py | The main analysis script. It aggregates how often each expert was selected and its REAP score, computes how much overlap there is between the expert sets used by different texts (using a similarity measure called Jaccard), and counts how many distinct experts each text effectively uses. | core_agg.json, christian_agg.json, analysis/*.csv |
analyze_exp1_all.py | Analyzes Experiment 1: how H6 behaves across all 12 Bible translations, comparing public-domain and copyrighted translations, and checking whether routing is the same across translations | Console output + EXP1_RESULTS.md |
analyze_exp4.py | Analyzes the Experiment 4 pilot: H6 firing rates broken down by group and by individual anchor expert | Exp 4 pilot analysis |
analyze_exp4b.py | Analyzes Experiment 4b: H6 rates on the commentary corpus, and whether the fraction of quoted text in each sample correlates with H6 activity | EXP4B_RESULTS.md |
analyze_exp12.py | Analyzes Experiment 12: how H6 and expert 164 behave with and without digits, broken down by category and by individual text pair | EXP12_RESULTS.md |
chart_exp1_all.py | Generates 3 charts visualizing the expanded Experiment 1 results | chart_exp1_all_*.png |
chart_exp12.py | Generates 3 charts visualizing the Experiment 12 results | chart_exp12_*.png |
generate_report.py | Produces an HTML report that re-checks all the integrity invariants on the finished data | report.html |
build_wiki_site.py | Generates this wiki from the data and stats above | site/*.html |
These scripts run the two lens techniques described on the J-space lens page: the logit lens (what word would the model guess at each layer) and the bounded Jacobian (how much does nudging the input change the output).
| Script | What it does |
|---|---|
run_jlens.py / jlens_dsv4.py | Runs both the logit lens and the bounded Jacobian on the religious corpus and saves the per-layer, per-position results |
run_jlens_tp2.sh | The launch wrapper that deploys the lens scripts across both GPU machines |
jlens_viewer.py | Builds the interactive HTML viewer that lets you browse the lens results by tradition, sample, and position |
These scripts handle the plumbing: copying code and data to GitHub and HuggingFace, uploading records as they are produced, and chaining experiments together so they run unattended.
| Script | What it does |
|---|---|
sync_to_github.sh | Pushes code and data to GitHub, but throttles itself to run at most once every 4 hours so it does not spam the repo |
upload_hf_project.py | Uploads the consolidated dataset (45 files) to HuggingFace in one go |
stream_to_hf.py | Uploads observation records to HuggingFace as they are produced, so the dataset stays current during long runs |
chain_exp1_exp12.sh | Runs Experiment 1 and then Experiment 12 back to back without needing a person to start the second one |
The model runs inside a Docker container so that its environment is identical every time. Here is the base image and the key settings:
FROM ghcr.io/anemll/dspark-vllm-gx10:0.1.1 ENV TOKENIZER_ID=deepseek-ai/DeepSeek-V4-Flash-0731 ENV TOKENIZER_REV=9e165c30e2704aec5d9d593cce3eebd58bbef1cb ENV CKPT_DIR=/ckpt
The container image is tagged local/dspark-vllm-gx10:proven-0731. The two GPUs communicate over a plain network socket rather than InfiniBand, because InfiniBand is disabled inside the container.