Reverse engineered from my environment. Happy to update the repo if anything is missing.
conda create -n alignability Python=3.10
conda activate alignability
# adjust for cuda version
conda install pytorch pytorch-cuda=12.1 -c pytorch -c nvidia
pip install -r requirements.txt
# eflomal
git clone https://github.com/robertostling/eflomal.git
cd eflomal
python -m pip install .
# fast_align for symmetrising alignments
git clone https://github.com/clab/fast_align.git
mkdir fast_align/build
cd fast_align/build
cmake .. && make
- prep_tok_files.py: Knows how to read FLORES, OPUS-100, and cc-aligned data from HF, as well as multi-cc-aligned data from MTData Downloads. Multi-CC-Aligned is not actually the same dataset as CC-Aligned. Confusing, I know. And Multi-CC-Aligned is not available on HF, so we can't stream it unfortunately. The data prep script outputs tokenised files in "fast_align" format, i.e., up to n sentence pairs with source and target side separated by " ||| ". n is set to 300k atm.
- the bash scripts: Call data prep script and the aligner CLI, as well as alignability.py and read_and_correlate.py
- alignability.py: Calculate/read and collect the actual alignability score(s), i.e., eflomal and proportion of 1-1 alignments
- cla_utils: Helper code for
crosslingual_alignment.py. Adapted but significantly adjusted from https://github.com/AlexJonesNLP/XLAnalysis5K
To calculate JSD, you also need https://github.com/tomlimi/entangled_in_scripts