Overview

The purpose of this repo is replicating parts of this paper: https://arxiv.org/pdf/2511.08579

Specifically, I want to re-run the first and second experiments, providing pre-trained adapter matrices to all models to see if the poor performance is just a result of not being able to recognise latents cross-model.

Experiment 1 - Activation Patching (Experiment 2 in the paper):

High-Level Plan:

1. Download the CounterFact dataset.
2. Download copies of Qwen3 8B and Llama 3.1 8B.
3. Set up experiment without activation patching (just one prompt).
4. Add in activation patching.
5. Try to replicate results.
6. Add in pre-trained adaptor matrices and see how much this changes results.
7. Run for differnet prompts from the paper.

Experiment 2 - Neuron Characterisation (Experiment 1 in the paper):

Can I make better use of the transluce's code? https://github.com/TransluceAI/observatory

High-level plan:

1. Fine-tune a simulator model, as in the paper. Target = Llama-3.1-8B. Sumulator also tuned from Llama-3.1-8B. ...Can I just use the simulator from Choi et al. instead?
2. Pre-train adapter matrices for Llama-3.1-8b, Qwen 8B and possible Gemma 9B as expliainer models. (Llama's will probably just be the identitity matrix, but still worth putting all models on level field. Can run evals with and without).
3. Fine tune all 3 models as explainers and evaluate results.

Next steps:

• Read Choi et al. to get details on process. See how much GPU RAM I need (will I have to run two 8B models at once?).
• Set up environment.
• Download Simulator, Llama, Qwen.
• Write fine-tuning pipeline.
• Run fine-tuning pipeline.
• Repeat for Gemma.