Script for building parallel paragraph corpus

[laurejt]

Associated Issue(s): resolves #41

Changes in this PR

• Adds one-off program for building the MTO parallel paragraph corpus

Notes

• This script does not use the MTO webscrape, only the data in the side-by-side translations spreadsheet.
• The input CSVs (exported from the side-by-side translations spreadsheet) are available here
• The language mapping for the articles is hard-coded.
• This script only extracts the parallel texts (i.e., rows) that correspond to paragraphs. So, it ignores section titles.

Reviewer Checklist

• Check that build_paragraph.py runs locally using the MTO-parallel-data files (expected input is a directory). Confirm that the local output matches mto-parallel-pars.jsonl
• Confirm that the output JSONL has the fields specified in data-design.md
• Output JSONL's contents look reasonable
• Confirm that the hard-coded language mapping is correct

[tanhaow]

I tested the code and the output looks good. This branch is ready to merge.

[tanhaow]

Here the ID starts with 0, but the ID in build_sentence.py starts with 1. We may want to unify them.

[laurejt]

Thanks for flagging this, I must have missed this when reviewing build_sentence, I did expect IDs to start at 0 since it's really serving as a index rather than a meaningfully unique identifier.

I find that 1-indexing always leads to trouble in 0-indexed languages.