Requirements for Indian languages

[vermaprashant1]

TDIL(Technology development for Indian languages) have collated Indian languages requirements concerned with Hindi language variations(different keystrokes and spelling variations) with examples that need to be focus and reflected in string searching recommendation. Kindly guide us for further actions.

[r12a]

@vermaprashant1 could you provide a link to the requirements document you created, so that Addison and i can review it?

[vermaprashant1]

@richard ,please refer draft requirement document
ILanguage-requirement document-character-model.pdf
that covers Hindi as a initial language. The same has been circulated with experts for inputs. This document further extended by taking variations in other Indian languages apart from Hindi.

[vermaprashant1]

@r12a. Can you please share your feedback for this document,

[vermaprashant1]

@richard ,please refer draft requirement document ILanguage-requirement document-character-model.pdf that covers Hindi as a initial language. The same has been circulated with experts for inputs. This document further extended by taking variations in other Indian languages apart from Hindi.

@r12a please send your feedback on the shared document. We are also investigating the different variations and rules for other additional 5 languages and will share soon.

[r12a]

@vermaprashant1 sorry it's taken me so long to get to this. Here are my comments.

[1] I think the document would be much clearer if at the beginning you separated out more cleanly the various ways in which words can be encoded differently, and then after that point out the consequences and proposed advice. I would start with a list of problem cases that would include the following:

1. spelling variants such as the alternation between syllable-final /n/ or nasalisation (eg. the word Hindi) – note that spelling variants occur in most languages, and so it's something any search engine typically has to consider - what other common alternative spellings occur in Hindi besides LA vs LLA (which you mention almost in passing without any examples)? It would be good to have a list of at least the more common ones.
2. the choice of characters to represent nuktas (with a little more detail) – this is a little complicated in Devanagari because normalisation produces different results for different visual combinations, see https://r12a.github.io/scripts/devanagari/#nukta_encoding
3. inappropriate combinations that look the same visually – you don't mention these at all, but it's a significant issue for indic scripts. See examples of this for vowel-sign and independent vowel representation at https://r12a.github.io/scripts/devanagari/#vowelsign_encoding and https://r12a.github.io/scripts/devanagari/#vowelsign_encoding2
4. any combinations of combining characters with a single base that can be typed and stored in an order that causes problems - often this is resolved during normalisation, but there are problematic cases that are not resolved by normalising the text - similar issues are motivating some folks involved with Unicode to produce rendering guidelines for Thai, Khmer and Arabic scripts - these advise reordering of specific sequences of characters so as to produce consistent ordering and ensure that the text renders correctly when displayed. Again, you don't mention any such combinations, and i haven't researched this either yet for Devanagari.
5. Matching needs to decide what to do when format characters appear in text, eg. ZWJ, ZWNJ. In languages like Persian, these can affect the semantics of the text, but i suspect that in Devanagari that is not the case, and they can just be ignored. It's worth checking the full list of invisible characters that may appear in Devanagari text.
6. Graphically similar but semantically different (confusable) code points - i would probably put the OM in this category.

Such an analysis would need to indicate which alternations in sequence are handled by normalisation. Normalisation should be expected as a given, always, before matching, so it's the ones that normalisation doesn't fix that we are particularly interested in.

It would be interesting to explore whether what equivalences need to be made for string matching of identifiers (eg. the HTML/CSS case) vs. full text search. For example, in english spelling differences such as 'internationalization' vs. 'internationalisation' are not seen as equivalent, and maybe the anusvara-conjunct alternate is the same. In full text search, however, searching for one should probably find the other.

[2] Section 2.2.

It is requires by the Unicode to store and interchanged the characters in the same logical order or we can say that order that user typed through the keyboards

The initial sentence gives the impression that the Unicode Standard requires that users type keys on the keyboard in a particular order. What the standard actually says is that the stored order of characters typically corresponds to the order in which they are typed, but there is no expectation at all about how the keyboard should actually function, as long as it produces an appropriate sequencing of characters in the end: combining marks after base characters, virama between conjunct parts, etc.

Given that, i'm not sure what point you want to make in section 2.2. Any decent keyboard should allow the user to produce good Unicode character sequences, and any kbd that doesn't should be avoided.

[3] Are there different concerns for other languages using devanagari? - eg. i'm thinking about the eye-lash RA in Marathi.

[4] It would be very much easier for me to review your document if it was available in HTML, rather than PDF form. I'd be able to make annotations on the document for my reference, and i'd be able to copy-paste examples for exploration without the junk that PDF produces.

hope that helps.

[vermaprashant1]

Dear Richard, Greetings.. Thanks for sharing valuable inputs. As it was a long time back, We have already revised the character model requirements document with additional 5 more languages requirements. These are collected from the various Language Experts. Also we will go through your comments and revise documents accordingly wherever required. I will share it with you soon. Thanks, Prashant

…

On Wed, Feb 9, 2022 at 7:37 AM r12a ***@***.***> wrote: @vermaprashant1 <https://github.com/vermaprashant1> sorry it's taken me so long to get to this. Here are my comments. [1] I think the document would be much clearer if at the beginning you separated out more cleanly the various ways in which words can be encoded differently, and then *after that* point out the consequences and proposed advice. I would start with a list of problem cases that would include the following: 1. spelling variants such as the alternation between syllable-final /n/ or nasalisation (eg. the word Hindi) – note that spelling variants occur in most languages, and so it's something any search engine typically has to consider - what other common alternative spellings occur in Hindi besides LA vs LLA (which you mention almost in passing without any examples)? It would be good to have a list of at least the more common ones. 2. the choice of characters to represent nuktas (with a little more detail) – this is a little complicated in Devanagari because normalisation produces different results for different visual combinations, see https://r12a.github.io/scripts/devanagari/#nukta_encoding 3. inappropriate combinations that look the same visually – you don't mention these at all, but it's a significant issue for indic scripts. See examples of this for vowel-sign and independent vowel representation at https://r12a.github.io/scripts/devanagari/#vowelsign_encoding and https://r12a.github.io/scripts/devanagari/#vowelsign_encoding2 4. any combinations of combining characters with a single base that can be typed and stored in an order that causes problems - often this is resolved during normalisation, but there are problematic cases that are not resolved by normalising the text - similar issues are motivating some folks involved with Unicode to produce rendering guidelines for Thai, Khmer and Arabic scripts - these advise reordering of specific sequences of characters so as to produce consistent ordering and ensure that the text renders correctly when displayed. Again, you don't mention any such combinations, and i haven't researched this either yet for Devanagari. 5. Matching needs to decide what to do when format characters appear in text, eg. ZWJ, ZWNJ. In languages like Persian, these can affect the semantics of the text, but i suspect that in Devanagari that is not the case, and they can just be ignored. It's worth checking the full list of invisible characters that may appear in Devanagari text. 6. Graphically similar but semantically different (confusable) code points - i would probably put the OM in this category. Such an analysis would need to indicate which alternations in sequence are handled by normalisation. Normalisation should be expected as a given, always, before matching, so it's the ones that normalisation doesn't fix that we are particularly interested in. It would be interesting to explore whether what equivalences need to be made for string matching of identifiers (eg. the HTML/CSS case) vs. full text search. For example, in english spelling differences such as 'internationalization' vs. 'internationalisation' are not seen as equivalent, and maybe the anusvara-conjunct alternate is the same. In full text search, however, searching for one should probably find the other. [2] Section 2.2. It is requires by the Unicode to store and interchanged the characters in the same logical order or we can say that order that user typed through the keyboards The initial sentence gives the impression that the Unicode Standard requires that users type keys on the keyboard in a particular order. What the standard actually says is that the stored order of characters *typically* corresponds to the order in which they are typed, but there is no expectation at all about how the keyboard should actually function, as long as it produces an appropriate sequencing of characters in the end: combining marks after base characters, virama between conjunct parts, etc. Given that, i'm not sure what point you want to make in section 2.2. Any decent keyboard should allow the user to produce good Unicode character sequences, and any kbd that doesn't should be avoided. [3] Are there different concerns for other languages using devanagari? - eg. i'm thinking about the eye-lash RA in Marathi. [4] It would be very much easier for me to review your document if it was available in HTML, rather than PDF form. I'd be able to make annotations on the document for my reference, and i'd be able to copy-paste examples for exploration without the junk that PDF produces. hope that helps. — Reply to this email directly, view it on GitHub <#10 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB7B5ESYROMPOCRINGJKHV3U2KC43ANCNFSM4ZJ77KSQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Thanks & Regards, Prashant Verma I Program Manager Web Standardization Initiative(WSI) , MeitY New Delhi Cell : +91-8800521042 Website : *http://tdil.meity.gov.in/WSI/AboutWSI.aspx <http://tdil.mit.gov.in/WSI/AboutWSI.aspx>*

[r12a]

Sorry for the delay. I will look at your revised document. (Please point to an HTML file, if that's possible.)

In the meantime, could you take a look at w3c/iip#119 (comment) for me? Thanks.

[vermaprashant1]

@r12a
please find the revised document that covers 6 Indian languages requirements and variations.

[aphillips]

@vermaprashant1

Hello Prashant,

I was looking over your document today in preparation for taking up some of the text into string-search. I intend to add questions to this thread as I go through your document. For starters, I found this paragraph:

Bengali, is one of the notorious languages with regard to spelling variation. The different spellings of word having same meaning are accepted in the Bengali language, it should be treated as different word although have same meaning. It has 5000+ words which record spelling variations.Typically, spelling variation ranges from 2 to 8 words.Majority of words have 2 variations; some have 3, 4, 8 and more variations. At least there is one word that records 16 spelling variations.Nearly 80% words show two spelling, 7% words show three variations, 7% words show four spellings, and 6% words show more than four variations.

Can you clarify that "different spellings of word... should be treated as different word" means that spelling variations should be treated as if they were different words (not matching)? I know that's what the sentence means, but want to be sure that this was your intention.

Do Bengali users expect document searches not to provide spelling variation matches at all? Or are there features in some programs for users to find such matches?

I will ask other clarifications as I work through the document. Thank you so much for providing this information!!

[r12a]

Can you clarify that "different spellings of word... should be treated as different word" means that spelling variations should be treated as if they were different words (not matching)?

I suspect that the word 'not' is missing between 'should' and 'be'.

[asmusf]

I would like to understand whether any of the alternate spellings or alternate code point sequences involve sequences that are listed as "do not use" in the Unicode standard. (Unfortunately, you'll need to read these from tables in the script chapter, they are not defined in any data files). For syntactic elements, editing tools etc. should probably flag any attempted use of "do not use" sequences.