README_FOR_CODE.md

Installation for python scripts

Install python environment, requires python >= 3.11.4

Pyenv (we need tkinter hence the extra steps)

brew install pyenv
brew install pyenv-virtualenv
brew install tcl-tk

pyenv install 3.11.4
pyenv virtualenv 3.11 cot
pyenv local cot

Install requirements

make env

Install pre-commmit hooks

make hooks

Checks

To run linting / type checks

make check

To run tests

pytest tests

Downloading large files

We track large files with git lfs. To install

brew install git-lfs

To download the files (git pull should download the lfs files automatically)

git pull

We've set it up to automatically track .json files in the project directory. To manually track more files, run

git lfs track "path/to/file"

See here for more details

Running an experiment

Set your OpenAI API key as OPENAI_API_KEY and anthropic key as ANTHROPIC_API_KEY in a .env file.

To generate examples e.g. this will compare 20 samples for each task in bbh for sycophancy

python stage_one.py --exp_dir experiments/dummy_run --models "['text-davinci-003']" --formatters "['ZeroShotCOTUnbiasedFormatter', 'ZeroShotCOTSycophancyFormatter']" --repeats_per_question 1 --batch 10 --example_cap 20

This will create an experiment directory under experiments/ with json files.

Viewing accuracy

To run analysis

python analysis.py accuracy --exp_dir experiments/dummy_run

Viewing experiment samples

python viewer.py --exp_dir experiments/dummy_run

Tip: You can pass in --n_compare 2 to compare 2 samples side by sde

Streamlit viewer

There is a nicer streamlit viewer that can be run with

streamlit run streamlit_viewer.py experiments/dummy_run

Note that it currently only works for stage one tasks