From the DeepSeek-V4-Flash-0731 interpretability project

J-space lens (logit lens and bounded Jacobian)

1. What the two lenses do

"J-space" is our name for the set of intermediate states the model passes through as it processes a token. At each of its 43 layers, the model holds an internal representation of what it has read so far. We used two techniques to inspect those intermediate states, rather than waiting for the final output:

• Logit lens. A language model's final step is to convert its internal state into a ranked list of possible next words. The logit lens takes the model's intermediate state at any layer and runs that same final step on it early. This lets us see what word the model would guess next if it had to stop at layer 5, or layer 20, or layer 35, instead of going all the way to the end. We record the top 10 candidate words and their probabilities at every position, at every layer. Think of it as asking the model to show its hand at each step of its thinking.
• Bounded Jacobian. The Jacobian is a mathematical object that tells you how much the output changes when you nudge the input. Computing it exactly is expensive, so we approximate it: we nudge the input by a tiny amount (0.01) in 16 random directions, re-run the model, and measure how much the output shifts. The size of that shift tells us how sensitive the model is to small input changes at each layer. A bigger shift means that layer is more responsive to what came in.

2. How sensitive each layer is (bounded Jacobian norms)

What we found (with a caveat): Layer 0's influence on the output is about 2.5 to 3.5 times that of later layers, across all traditions. The model is most responsive to its input at the very first layer, then settles down. The caveat: we only nudged in 16 directions, so these numbers are approximate bounds, not exact sensitivity measurements. The differences between traditions are modest and do not line up with how the traditions clustered when we ranked experts by REAP score. That suggests the lens is picking up on surface-level text statistics, not on theological content.

3. What the model guesses at each layer (logit lens)

Watching the top candidate word at each layer, we see the model's predictions stabilize around layers 30 to 35 across all traditions. Before that, the guesses jump around. After that, they mostly settle. No tradition looks qualitatively different from the others in this respect. You can explore this per position and per layer in the interactive viewer.

4. The interactive viewer

The J-space viewer is a self-contained HTML file (about 120KB) that lets you browse all 80 samples. Pick a tradition, a sample, and a position in the text, and you can see the model's running next-word guess at each of its 43 layers, plus how sensitive each layer was to the input. It is the closest thing to watching the model think in real time.

5. A logit lens claim we retracted

6. Caveats and open problems

• Our bounded Jacobian uses only 16 random nudge directions, so it is an approximation, not an exact measurement of input sensitivity.
• The differences between traditions on the lens do not match the differences we found by ranking experts with REAP. The lens may be reacting to surface-level text statistics (like how the text is formatted) rather than to the content.
• We have not yet checked whether digit density drives the lens differences, the way it drove the expert 164 confound.