Assessment of Semantic Similarity

[arubehn]

While implementing three methods for automated morpheme segmentation that have not been published before (Benden 2005, Bordag 2008, Kirschenbaum 2013), I noticed that it is not possible to exactly mirror the workflow of the latter two, since they make use of word frequencies in the corpus the method is applied to. This is mostly done to assess semantic similarity by the means of co-occurrence distributions - a measurement which is obviously not available for wordlists.

Since including this semantic information seems to be a crucial part of both workflows, I suggest that we find a way to emulate that. The most obvious choice in my opinion would be simply using pre-trained word embeddings (e.g. GloVe or Word2Vec), since they have proven to be quite useful for a wide range of NLP tasks and are conceptually the same as the distributional vectors obtained by the models in the study. An alternative approach could be quantifying the similarity between two concepts by the means of colexification, essentially exploiting CLICS.

Furthermore, Bordag (2008) employs a compound splitter that uses the frequencies of the individual constituents as well. I have no idea so far how this can be emulated in the context of (small) wordlists.

Feel free to discuss and drop your ideas! (especially @LinguList of course)

[LinguList]

We want to add vector semantics anyway to the NoRaRe package norare.clld.org, so the point where these vectors should be accessed in the workflow is from there or from workflows that load the wordlist and then also load semantic similarity scores in any form (they may also come from CLICS, etc., which is of course a bit circular, but I do not think it is a big problem). So word vector similarities would be linked to Concepticon/NoRaRe.

[LinguList]

But dummy similarities should also work at the same time, it is just some additional info that we suppose is provided along with a method, but if semantic vectors are not good, a method can still work.

[arubehn]

Alright, thank you, will set up the architecture so that it can be connected to several endpoints assessing semantic similarity. The problem is that the papers seem to underspecify how exactly they cluster together semantically similar words, so I guess I will have to try around what works reasonably and maybe apply some heuristics.

[LinguList]

I think what we must do as a next step is that we discuss the call structure of the functions and also how we load data. As you use lingpy anyway, my suggestion is to keep this in the lingpy environment, so we don't need to use cldf-constructs as you introduced there using pycldf and nelex data, but rather use wordlist format, and we use pyedictor to convert CLDF datasets to wordlists.

In this way, we also make sure the methods resemble the call structure we had before.

But no need to look into any of this now, first I need to set up the package, and also add tests, which are also still lacking, as far as I can see.

[arubehn]

That makes sense, I wasn't aware of that option since I am not that familiar with LingPy and the surrounding ecosystem yet. Let's maybe discuss the structure around the beginning of next week, whenever it suits you.

[LinguList]

I try and find time to make a pull request with a modified structure that should make that clear. In any case, we'll discuss this asynchronously via github.

[arubehn]

That would be great, just keep me posted! In the meantime, I will try to implement as much from Bordag (2008) as possible.

[LinguList]

German.tsv.zip

This TSV file is a good test set. You can just load it with lingpy when making demos for the algorithms. To get a list of morphemes (without their meaning), just do:

wl = Wordlist("German.tsv")
words = [wl[idx, "tokens"] for idx in wl]

Segments are indicated by a + in the notation, and all algorithms must work on tuples / lists, not on strings in Python.

[LinguList]

I think it is best to make a demo for each implementation, where the segmentation is compared.

To get unsegmented strings:

wl = Wordlist("German.tsv")
words = [wl[idx, "tokens"] for idx in wl]
test = [[t for t in w if t != "+"] for w in words]

To evaluate, the best is probably to be strict.

[LinguList]

So you can use a function that checks if splits are identical or if they aren't to get a rough assessment on how it works.

[arubehn]

Thanks! As for the evaluation, I think a precision-recall-fscore scoring scheme would be suitable, and I will implement that in time. The methods are designed to take lists as input, some of them might "by accident" also work on strings, but the intended workflow is with lists/tuples.
Regarding the package structure - I will just wait for your pull request as a starting point, and then add new code according to your suggested structure. For now, I will just preemptively add the algorithms as scripts in the top level directory.

[LinguList]

Yes, and please add examples in the init function that take the new file, so we have something that we can look at directly.