Character Combinations for Segment Grouper

[LinguList]

I just experimented with a very simple method to allow for character combinations with SEgmentGrouper. The idea is that we use brackets to combine segments, so we can express what we would express in several lines in one. Later, we use a rule-parser, to make several lines out of these statements. This can be handy for orthography profile creation, and I have tested it in the profiler for JavaScript.

The code in Python is very simple:

def parse_rule(rule, delimiter=" "):

    # first a grouping after brackets
    groups = [[""]]
    in_bracket = False
    for char in rule:
        if char == "[":
            if not groups[-1]:
                groups += [""]
            in_bracket = True
        elif char == "]":
            in_bracket = False
        else:
            if in_bracket:
                if char == delimiter:
                    groups[-1] += [""]
                else:
                    groups[-1][-1] += char
            else:
                if char == delimiter:
                    groups += [[""]]
                else:
                    groups[-1][-1] += char                
    return groups

It can be extended by adding a character for diacritics, that is later replaced.

[xrotwang]

Could you add a couple of examples?

[LinguList]

Yes.

[LinguList]

Suppose, you want to say "I know there are 5 vowels aeiou in my data, and I want to convert all vowels with -n to a nasal vowel. So you would then traditionally write several rule:

an ã
un ũ
... ...

[LinguList]

With the rule-parser (better name can be developed later) for the orthoprofiles, you can write:

[a e i o u] n

And this will yield:

an
en
in
on
un

[LinguList]

But you can also write for the conversion part:

[a e i o u] \u0303

and this yields:

ã
ẽ
ĩ
õ
ũ

[LinguList]

We do not want to have these rules in an uncontrolled form in an orthography profile, or in a segmentgrouper, when doing conversions, but we may want to be able to have them as a preliminary step to guide the creation of a profile.

[xrotwang]

How many of these rules can one have on one line? What does [a b] c [d e] expand to?
And can they be nested?

[LinguList]

list(product(*groupeditems))

[LinguList]

So it is always the cartesian product.

[LinguList]

WAIT, I just figured out we use groups in the JavaScript tool, so you can also write something like:

[1 a b] c [2 d e]

This then translates to:

a c d
a c e
b c d
b c e

But

[1 a b c] c [1 a b c]

is

a c a
b c b
c c c

Experience shows that this functionality is important in many cases.

[LinguList]

But it also shows I should think about this a bit more, as it is not clear how to include this in the workflow...

[LinguList]

So it may be useful to NOT start working on anything in Python yet, but rather make sure that this JavaScript code is somehow explained somewhere, so more people than only me could use it ;-)

[xrotwang]

Yes, in particular this [1 ...] ... [2 ...] syntax isn't obvious at all. I guess the canonical example to lean on here would be regular expressions. So maybe aca|bcb|ccc rather than [1 a b c] c [1 a b c]. Not much of a shortcut in that case, but a lot more transparent.

[LinguList]

Yes. I guess the problem is that we must somehow restrict the power of regular expressions to the typical use cases in linguistics. I'll keep reflecting about this.

[xrotwang]

I wouldn't want to allow full regular expressions, but re-using a subset of regex syntax to describe the rules would seem useful.