It would be good to have standard train/dev/test splits (and folds)

[fcbond]

It would be good to have standard train/dev/test splits (and folds)

Because the corpora, especially the Sherlock Holmes stories have widely varying sentence lengths, we should split on number of words, not sentences. Ideally, splits should happen only on the first sentence in a paragraph, or a header.

• Create a script for splitting the corpora #47
• Incorporate the splits into the output package #48
• Add the splits to a table in the database #49
• Make the splits align across languages #50