DSv4-Flash REAP Wiki

Wiki pages

On this page

Resources

External review

From the DeepSeek-V4-Flash-0731 interpretability project

Contents

Experiment 12: the digit minimal-pair test

This is the experiment that settled our original mistake. We took the same text in two versions, one with its digits kept and one with digits removed, and compared them. H6 does not care about digits (1.02x ratio). e164, the expert we once thought was a scripture detector, is actually a digit detector (1,453x ratio). The two effects are completely separate.

Why we ran this

Early in the study we thought we had found an expert (e164, expert number 164 at layer 42) that detected "memorized scripture." It fired at zero on Bible text and high on everything else. We were wrong. The real explanation was digits. Our pipeline had stripped verse numbers from the Bible, so the Bible was the only text in our corpus with 0% digits. e164 was not detecting scripture. It was detecting digits.

Experiment 12 is the clean test of that. We built 111 matched pairs of text across 5 categories. Each pair shares the same source text. The "with_digits" version keeps the original numerals. The "digit_stripped" version has the digits removed. Digit density is the only systematic difference between the two versions of each pair.

CategoryPairsDescription
dates_and_years23Historical texts with dates and chronologies
misc_numeric22Prose with embedded numbers
numbered_list22Numbered enumerations
scripture_citation22Bible verse references (e.g. "John 3:16")
tabular_statistics22Statistical data in tabular format
Total111222 records (111 x 2 conditions)

Invariant check: 0 violations across 222 records.

Results

H6 (the verse/prose axis) does not care about digits

ConditionTokensH6 firesH6 Rate (M)
with_digits221,57630,337136,915
digit_stripped216,32429,042134,252
Ratio1.02×

H6 fires at nearly identical rates whether the digits are present or not. The ratio is 1.02x, meaning essentially no difference. H6 is not a digit detector.

e164 (the layer-42 expert) is confirmed as a digit detector

ConditionTokense164 firese164 Rate (M)
with_digits221,5768,93140,307
digit_stripped216,324627.7
Ratio1,453×

e164 fires 1,453 times harder on text with digits. On digit-stripped text it fires at only 27.7/M, barely above noise. e164 is a digit detector.

Per-category breakdown

H6 by category and condition

Categorywith_digits (M)stripped (M)ratio
dates_and_years176,979167,8371.05×
misc_numeric152,956142,0971.08×
numbered_list201,314184,1011.09×
scripture_citation63,13866,4750.95×
tabular_statistics83,87393,1930.90×

e164 by category and condition

Categorywith_digits (M)stripped (M)ratio
dates_and_years27,12942.6637×
misc_numeric41,3020.0∞
numbered_list37,1010.0∞
scripture_citation49,26473.7669×
tabular_statistics47,93427.91,721×

Charts

ChartWhat it shows
H6 vs e164Grouped bar: H6 (digit-independent) vs e164 (digit detector), log scale
H6 by categoryH6 rates per category, no systematic digit effect
e164 by categorye164 rates with massive with_digits vs near-zero stripped

What we learned

  1. H6 is digit-independent (1.02x ratio). It genuinely detects text format, not digit density.
  2. e164 is a digit detector (1,453x ratio). What we originally called a "scripture detector" was entirely a digit-density confound.
  3. The two routing axes are separate. H6 tracks verse vs. prose format. e164 tracks whether digits are present. They are different phenomena that we originally conflated.
  4. Per-pair comparison: 81 pairs show with_digits higher than stripped, 24 show the reverse, 6 are equal. For H6, that asymmetry is too small to indicate a real digit effect.
  5. Digit density correlation r = -0.22 for H6. A weak negative value, no meaningful relationship.

What this means

  1. The digit confound is fully resolved. e164 = digits. H6 = verse/prose format. Two separate axes, two separate explanations.
  2. Three independent tests now confirm H6: Exp 1 (translation-invariant), Exp 4b (fires on prose, not verse quotes), and Exp 12 (digit-independent). The H6 verse/prose axis is the strongest and most robust routing effect in the study.
Categories: DSv4-Flash interpretability | MoE routing