参数忠实性研究工具

🚀 Quick Start Reproduction

1. Set up the environment and install packages.

   conda activate pf
   pip install -r requirements.txt
   python -m spacy download en_core_web_sm
   python -c "import nltk; nltk.download('punkt'); nltk.download('punkt_tab')"
   

Tips: If you are using Qizhi (http://qz.cfff.fudan.edu.cn/), you can directly use the environment we make by conda activate pf.

1. Set your Huggingface Token. Make sure you have the access to use Llama-3.2-3B.

   export HF_TOKEN=hf_xxxxxx 
   

Tips: Apply for your access to visit Llama-3.2-3B in https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct. If you are using Qizhi (http://qz.cfff.fudan.edu.cn/), you can ignore this part because we have download the model on the server.

1. Run the CoT & noCoT baseline before unlearning.

   PRE=1 bash repro/run_all.sh
   

The CoT & noCoT results will be saved in ./final_cot.

1. Run the unlearning experiment.

   bash repro/run_all.sh
   
   # If you want to run them in parallel, instead, run them separately
   python -m repro.run_repro --short_model Phi-3 --dataset openbook --lr 1e-4
   python -m repro.run_repro --short_model Phi-3 --dataset sqa --lr 5e-5
   python -m repro.run_repro --short_model LLaMA-3-3B --dataset openbook --lr 3e-5
   python -m repro.run_repro --short_model LLaMA-3-3B --dataset sqa --lr 3e-5
   

Unlearn results will be saved in ./final_results.

1. Mistake injection and Gemini-3-flash baseline

   set GEMINI_API_KEY=your_google_api_key
   jupyter notebook "Adding mistakes repro.ipynb"
   python mistakes_repro.py --short_model Phi-3 --dataset openbook
   python mistakes_repro.py --short_model Phi-3 --dataset sqa
   python mistakes_repro.py --short_model LLaMA-3-3B --dataset openbook
   python mistakes_repro.py --short_model LLaMA-3-3B --dataset sqa
   

The notebook will write injected mistakes to ./mistake_results. Evaluation results will be written to ./mistake_stats.

1. Compute metrics.

   python repro/compute_metrics.py
   

The results will be save in repro/metrics_summary.csv and repro/metrics_summary.json.

Phase 2 · Extension

• [x] Design $\lambda$ sweep range & run ablation
• [x] Run SimNPO instead of NPO under the same setup
• [ ] Explore more new alternative methods under the same setup
• [ ] Analyze extension results
• [ ] Integrate results from both extension tracks
• [ ] Final report completed

---