From the DeepSeek-V4-Flash-0731 interpretability project

Master index

This project studies how DeepSeek-V4-Flash-0731, a mixture of experts language model, routes text. In a mixture-of-experts model, a router picks a few small sub-networks (called experts) to handle each token (roughly each word) rather than running the whole network every time. We feed the model religious and secular texts and record which experts it picks and how strongly. Our headline metric is REAP, the router's gate weight for an expert multiplied by that expert's activation norm, which tells us how much each expert actually contributed.

Everything below is organized so you can find a specific artifact and understand what it is without having read the rest of the wiki first.

1. Data streams

Each row is one batch of observations we recorded from the model. A record is one text passage we fed in; a token is roughly one word or word-part the model processed. The REAP observations rows are the core dataset: for each passage, which experts fired and how strongly. The experiment rows are targeted follow-up tests built from smaller, controlled text sets.

2. Analysis artifacts

These are the files our scripts produce from the raw observations. Most come from analyze_experts.py, which tallies how often each expert fires, on which texts, and with what REAP score. A Jaccard matrix (row 3) is a similarity table: it asks how much the top experts for one text overlap with the top experts for another, where 1 means identical and 0 means no overlap. Bootstrap confidence intervals (row 7) are a way of estimating how stable a number is by resampling the data thousands of times.

3. Corpus registry

These are the text files we feed to the model. Digit density is the percentage of tokens that are numerals, a column we added after discovering that one expert (e164) fires on digits and that our Bible pipeline accidentally strips all digits, which misled us early on. Most texts come from Project Gutenberg, a free archive of public-domain books.

4. Publication artifacts

Where everything is published: the wiki pages, the interactive viewer for the J-space lens (our method for peeking at the model's intermediate predictions layer by layer), and the two public data repositories.