Multiple languages detection

[C-Loftus]

Hi there,

Thank you for your work on langram it seems very cool and promising and thank you for responding at odilia-app/odilia#21 (comment). I wanted to potentially use langram as a replacement for lingua-rs in this PR odilia-app/ssip-client-async#22 and wanted to clarify a few things

• Do you have any recommendations for how I could adapt this library to use ISO language codes? (If that is possible)
  • For SSIP on Linux, you need to tell it which language to speech for the given text based on either a 2/3 letter language code
  • I saw that One language can be written in multiple scripts, so it will be detected as a different ScriptLanguage which makes sense, I guess I'm just not sure how to map it to ISO without enumerating all of them and mapping them
• Would it be possible to provide any short example programs in this repo to show idiomatic API usage?
  • I suppose I was curious as well if you are using this library for other projects you are working on and if there are projects I could look at
• Does this library support detecting multiple languages in the same block of text and returning the indices where they start / stop?
  • i.e. 你好 Hello -> Chinese: 1-2, English 3-7

My background context

I am trying to get automatic language detection for speech dispatcher, which is the speech synthesis / tts server for Linux. This is relied upon by those who are blind or use a screen reader. So in other words, I'm writing a library which would take in a sentence that could be multilingual, have it tell me which languages each part of it is spoken in.

I am essentially looking for a solution that can disambiguate very small amounts of text at a time if possible. i.e. if a blind user were learning a language they could use my library to have their TTS server speech multiple languages intelligently.

Lingua did not work for me since it failed to disambiguate languages in short sentences, even in trivial cases like 你好 Hello

[RoDmitry]

• ISO 639-3 and ISO 15924 are implemented. You just call into_str, and will get something like engLatn. Or you could use into_parts to get them separately, or Language::from and then into_str to get only ISO 639-3 code.

• I will add examples later. One example from here:

static LANGRAM_MODELS: LazyLock<ModelsStorage> = LazyLock::new(|| {
    ModelsStorage::preloaded::<ahash::RandomState>(ScriptLanguage::all().collect())
});
let config = DetectorConfig::with_languages(languages.into());
let detector = Detector::new(config, &LANGRAM_MODELS);
texts
    .par_iter()
    .map(|text| detector.detect_top_one(text, 0.0))
    .collect()

But detector has other methods

• Multiple languages detection is not yet implemented. There is an implementation in Lingua, but I didn't like it. You can experiment with it.
  You might think that probabilities return multiple, but it's kind of wrong, because small words (languages) will be filtered out by alphabet_detector, and you will not even see them in the output.
  But I would do something different, I would do something like probabilities to get separated words from alphabet_detector (not from fulltext) and try to detect them separately using Langram detector. I plan to try it, but did not have time.

[RoDmitry]

There is also another issue which makes me skeptical about multiple languages detection: the word detection accuracy is low (look here, 3rd column from right). For English it's just 44%. That's why I plan on increasing it first.

[C-Loftus]

Thank you very much that all makes sense.

Multiple languages detection is not yet implemented. There is an implementation in Lingua, but I didn't like it. You can experiment with it.
You might think that probabilities return multiple, but it's kind of wrong, because small words (languages) will be filtered out by alphabet_detector, and you will not even see them in the output.

Ah that makes sense, in that case I assume that may be same the issue I was running to here: pemistahl/lingua-rs#464 And yup unfortunately as you say, that issue sort of makes the multilanguage detector from Lingua unusable for me. For a screen reader use case, it is possible to have lots of short sentences with multiple languages.

I plan to try it, but did not have time.

Sounds good! If you do end up implementing it, I will definitely try it out. But of course no expectation or rush whatever you choose.
Feel free to either close this issue now or leave it open; whatever is best.