From the DeepSeek-V4-Flash-0731 interpretability project

Operations (live)

1. What is running now

The two big observation runs are finished. The only thing still actively running is a small automation job that wakes up every 30 minutes to check on things, deploy the next queued experiment, and sync to GitHub. Here is the status of each pipeline:

2. What is queued to run next

These are the experiments and tasks lined up to run. The completed ones are marked done; the rest are waiting for a GPU slot.

1. ./finalize_christian.sh: merges the wave-2 Christian data, re-checks the integrity invariants, and rebuilds the summary tables. done
2. Exp 12 (digit pairs): complete, 222 records.
3. Exp 1 (12 translations): complete, 360 records.
4. Exp 4b (quotation switch): complete, 351 records.
5. Exp 13 (ablation): staged. The hook that zeroes out the H6 experts is deployed to both machines, and the corpus is ready. This is the experiment that will tell us whether H6 actually matters for the model's performance, not just whether it fires.
6. Exp 11: a re-run of the context dump with a larger token budget, so we can see more of what the model was reading when each expert fired.
7. Theology observation: 1,090 records of theology text, estimated at about 16 hours of GPU time.

3. What the automation does every 30 minutes

This is the checklist the 30-minute cron job follows, in order. It is designed to need no human intervention for routine operation:

1. tail /tmp/chain_runs.log            # check run status
2. pgrep -f 'chain_next|obs_christian_watch|jlens_watch|obs_theology_watch'
                                       # check if any supervisor died; restart if so
3. bash sync_to_github.sh              # push to GitHub (throttled to once per 4h)
4. If "ALL RUNS COMPLETE": finalize the data and deploy the next queued experiment
5. On completed observation: pull the data via scp, re-verify the integrity
   invariants, run analyze_experts.py, update this wiki, and sync

4. Rules we follow

These are the constraints we imposed on ourselves to keep the study clean and safe. They matter because the whole point of the project is to observe the model without changing it:

• We never modify the model. No weight edits, no pruning, no quantization. We only watch. The one exception is Exp 13, which temporarily zeroes out experts to test causality, and even that does not change the saved weights.
• One machine is off limits. spark-2822 (100.83.190.2) is out of scope and never touched.
• Passwords stay in environment variables. The de5c machine password is passed through the SSHPASS environment variable only. It is never printed, logged, or committed to git.
• No parallel scraping. We never run multiple scrapers at once against the same site. We wait 0.8 to 1.5 seconds between Gutenberg requests and 3 seconds between Wikipedia requests. This is why: early on we ran 6 Wikipedia scrapers in parallel and got rate-limited (HTTP 429).
• English text only. The core corpora are public domain. The 7 copyrighted Bible translations in Exp 1 are included because we had explicit authorization to use them.
• Every record is checked before it is accepted. The fail-closed integrity invariants run on every record. If any check fails, the record is thrown out and the run stops.

5. What went wrong, and how we fixed it