Monosegmental Morphemes

[LinguList]

If I want to read a word from string now, the following situation will yield an unwanted result:

>>> Word.from_string("p a t + ia")
[['p', 'a', 't'], ['i', 'a']]

Instead, what I would expect would be:

>>> Word([x.split() for x in "p a t + ia".split(" + ")])
[['p', 'a', 't'], ['ia']]

A simple solution would be to add another subclass and to specify words then with specific morphemes. But it seems also odd to have the following situation now with the Morpheme class:

>>> Morpheme("pat")
['p', 'a', 't']
>>> str(Morpheme("pat"))
'p a t'

I remember we discussed this, but I wonder if there's a way to maybe pass an argument to the from_string method to allow for a Morpheme that would contain one segment consisting of multiple characters?

[xrotwang]

No, I think there's no way around this. Arguably, it's rather a bug that + is recognized as morpheme separator in Word.from_string. We cannot really allow abc and a b c as morpheme a b c in the same from_string method, because then the one-multi-char-segment morpheme ia can not be distinguished from the multisegment morpheme ia - because the only way to distinguish the two representations is the presence of the morpheme separator ...

[xrotwang]

We could have multiple factory methods to distinguish: from_segmented_string or similar.

[LinguList]

I see that we have two different morpheme representations of which we also assume they are frequently used: abc + def (or abcdef) and a b c + d e f. The former in IGT, where you don't really care that the segment is a valid sound, and in lingpy/edictor/lexibank-cldf where you care that the segment represents a meaningful sound and must therefore allow for a separator different from the empty string.

[xrotwang]

I think that's why I was hoping we could switch to the unambiguous "nested lists" representation. If we had different factory methods "from_text" (parses stuff as it appears in text, e.g. whitespace separates words) and "from_segmented_string" (recognizes + as morpheme separator and expects (whitespace-)segmented morphemes) or similar, it would work. But then the step to doing away with this and expecting input to already be nested lists wouldn't be too big in my opinion.

[LinguList]

The good thing for even now is that the workaround shown above also works and is not really tedious.

[LinguList]

So I think I can perfectly live with different factory methods. This would mean then, however, that Word.from_string("a b c + a b c") would throw an error, or assume a morpheme ["a", " ", "b", " ", "c"], right?

[xrotwang]

So I think I can perfectly live with different factory methods. This would mean then, however, that Word.from_string("a b c + a b c") would throw an error, or assume a morpheme ["a", " ", "b", " ", "c"], right?

You mean the behaviour of my proposed from_text method? Not sure. It might raise an exception on encountering the "word" +. But otherwise would result in 7 one-character words, I guess.

[LinguList]

I was still thinking of the original problem that I brought up, and here, I wanted to say that if we leave it as is, the behavior is still unexpected, so it would require a change, right, since the "split on whitespace, if no white space do a list" seems problematic? And if that change consists in adding different methods for initialization from different kinds of strings, I am completely fine.

[xrotwang]

Ok, yes, different methods should be a good - if backwards incompatible - solution. But that's why we went with v0.1 I guess :)

[LinguList]

Yes :-)

[LinguList]

The other solution would be to subclass morphemes differently, based on their purpose, and use different from_string methods.

Right now, we have:

>>> Word.from_string("abc-def", separator="-")
[['a', 'b', 'c', '-', 'd', 'e', 'f']]
>>> TypedSequence.from_string("abc-def", separator="-", type=Segment)
['abc', 'def']
>>> TypedSequence.from_string("abc-def", separator="-", type=Morpheme)
[['a', 'b', 'c'], ['d', 'e', 'f']]

I think for broader use cases, we would like to

1. not overwrite "type" in the Word.from_string method
2. allow to pass the separator to the Word.from_string method

But then, we will anyway need targeted from_string methods, since the strict interpretation that goes back from word to segment does not work.

But what would be nice, in my opinion, would be:

>>> Word.from_X("a b c + ei")
[["a", "b", "c"], ["ei"]]
>>> Word.from_Y("abc-ei", separator="-")
[["a", "b", "c"], ["e", "i"]

This would then also cover the two major use cases of IGT vs. Wordlists, right?

[LinguList]

I think the following code would account for the problems:

class Morpheme(TypedSequence):

    item_type = Segment
    item_separator = None

    @classmethod
    def from_string(cls, s, **kw):
        return cls(list(s))



class SegmentedMorpheme(TypedSequence):

    item_type = Segment
    item_separator = " "

    @classmethod
    def from_string(cls, s, **kw):
        return cls(s.split(cls.item_separator))


class Word(TypedSequence):

    item_type = Morpheme
    item_separator = " + "

    @classmethod
    def from_string(cls, s, **kw):
        kw["type"] = kw.get("type", cls.item_type)
        kw["separator"] = kw.get("separator", cls.item_separator)
        
        return cls([kw["type"].from_string(m) for m in
                    s.split(kw["separator"])], **kw)

This would result in:

In [56]: Word.from_string("abc-def", separator="-")
Out[56]: [['a', 'b', 'c'], ['d', 'e', 'f']]

In [57]: Word.from_string("abc-def", separator="+")
Out[57]: [['a', 'b', 'c', '-', 'd', 'e', 'f']]

In [58]: Word.from_string("abc+def", separator="+")
Out[58]: [['a', 'b', 'c'], ['d', 'e', 'f']]

In [59]: Word.from_string("abc+def", separator="+", type=SegmentedMorpheme)
Out[59]: [['abc'], ['def']]

In [60]: Word.from_string("abc + d e f", separator=" + ", type=SegmentedMorpheme)
Out[60]: [['abc'], ['d', 'e', 'f']]

In [61]: Word.from_string("abc + d e f", separator=" + ", type=Morpheme)

TypeError: Segments must be non-empty strings without whitespace or "+".

[LinguList]

I'd consider this behaviour as expected and desirable in all cases, also allowing us to handle different applications with the same Word class.

[xrotwang]

Hm. That doesn't "feel" right. Internally we want morphemes to be exactly the same things - no matter where they came from, no? And even though that could be bolted on here with tailored __eq__ methods or similar, it would still provide potential for confusion.