A problem with `@lang`: Review attributes for languages, and attribute-element name sharing

[ebeshero]

An@lang that may only be used on a <code> element for designating a computer programming language seems a very inefficient use of an attribute that should be available elsewhere.

For example, consulting on a project that works with as-yet unknown or identified language systems, we have need to use esoteric codes defined internally by a scholarly community simply to identify shifts from one unknown language system to another (e.g. "language A" vs. "language B"). Scholars have identified multiple language systems in the document that they have designated with a special lettering system that would not be conformant with @xml:lang because the project codes don't follow BCP 47.

Other TEI attributes related to language are sparse and bespoke for set purposes. So we need a more multipurpose attribute for designating as-yet unknown or unclassified languages in a document.

Related questions:
Why is @lang not a member of teidata.language? (Maybe this is because @lang was introduced prior to 2005 with the <code> element and has not been discussed much since then).

Related to P6:

• If we want every attribute to be convertible to an element in P6, we'll have problems with the current definition of @lang not matching <lang> (which broadly "contains the name of a language mentioned in etymological or other linguistic discussion).

• Which other P5 attributes and elements share the same name but are defined with different scopes / semantics?

(Tagging @HadleighJae since this is relevant to her project.)

[djbpitt]

Possible follow-up consideration: We may want to clarify, during our discussion, why @xml:lang is or is not a useful way of recording the language of the text of an element. If I'm reading the language-identifier spec cited above correctly, it seems to allow "private-use" values (§2.2.7).

[hcayless]

For starters, while I agree @lang on code is under-specified, I'm not sure there's anything here we couldn't solve with @ana. @lang could not have the datatype teidata.language (which == xsd:language) because it's not on the face of it representing a "human" language.

Adding computer language metadata—the problem that @lang on code addresses—is actually a very complex problem, though I realize that's not what you're asking about here. Taking a look at https://highlightjs.readthedocs.io/en/latest/supported-languages.html might elucidate my point a little. This is the set of languages (and the coding scheme) that HighlightJS, a syntax highlighting JS library, has adopted. @lang doesn't even allow a pointer to some sort of taxonomy like this.

I sympathize with the desire for a "not-xml:lang" language attribute, but I'd argue that the solution is either a local customization (@custom-lang or something like that) or use @ana, which is more general purpose, but would do the job and is globally available. I guess I'd also say that you could use xml:lang a bit like we use @when, which gives the (normalized) Gregorian date for whatever date is expressed in the document. You could document the alignment between the custom coding scheme and BCP 47 elsewhere. As @djbpitt notes, it does permit custom values if they are needed.

[ebeshero]

What I'm objecting to is the name of the attribute, @lang, being generic-sounding enough, but restricted to use on <code> with apparently no other purpose imagined for it.

I don't object to its being used on <code>, but being defined exclusively for use there seems problematic. I think it could stand a redefinition to allow it in situations that aren't appropriate for @xml:lang, which might include names of programming languages or scholarly language abbreviations that have been in use within a community for a while but aren't compatible with BCP47 without significant alteration and probably don't need to be changed if the point is to create a TEI document suitable for that community.

For the project that inspired the ticket, we're having to introduce several custom (project-namespaced) attributes via ODD anyway, and @ana may have other uses. (We need 7 or 8 different attributes like @ana by the time we're done here.) But for one of the attributes that indicates a shift from one unknown language to another, it would be nice to be able to just use @lang.

[ebeshero]

This is also, I believe, an example of the kind of problem we'd like to be able to solve in P6, where we've discussed that any attribute node should be expressible as an element. That would not be the case for @lang and <lang> as defined in P5.

[hcayless]

You'll have a number of such name collisions if you decide on that approach. To be clear, I think code/@lang is probably a bit inadequate for the purpose it's meant to serve, and you could certainly deprecate and replace it. But again, this seems like a problem solvable via customization. For the record, @ana can handle multiple values and there are no rules I'm aware of that prevent its use for multiple bits of analysis.

I'm a bit skeptical of the attribute::element equivalence thing unless you mean to add a new namespace for the elements. There is (by and large) a fairly well-understood pattern to the meaning of element content vs. element attributes. And if you have such an equivalence, it will be broken as soon as people want to add structure to "attribute elements", which I bet will happen almost immediately. Not to mention the questions of element scope. It would be a change that would justify a switch to P6! But you might be creating a Python 3-style problem for yourself. To be sure, that seems to have finally resolved, but it was a headache for years. Council might need a new maxim, like "Don't make your users hate you." 😉

[ebeshero]

Council is intending to reduce a lot of accumulated antipathy toward the TEI for being needlessly complicated, @hcayless, but it does mean an entirely new approach to modeling semantics. Questioning things like the difference between @lang and <lang> is a small part of the paradigm shift for P6. The analogy to Python 2 vs. Python 3 is one we've been discussing, and we don't think that's a bad thing. The community will still have P5 for quite a while as it will take some serious time for P6. But we hope P6 will be a lot easier to customize without constantly running into confusing barriers.

[ebeshero]

@hcayless We're formulating some blueprints with flexible modeling of node semantics, and experimenting with UML to help with the thinking process. The hope is that we are able to formulate TEI conceptually in a way that can be modeled in more than XML, though it continues to be expressed most effectively in XML node trees. More on this beyond the horizon of this ticket.

[sydb]

@ebeshero — What is it about the scholarly language abbreviations that you are worrying about is incompatible with BCP 47?

[ebeshero]

@sydb Well, I'm not at all confident that a single capital letter (as in "A" or "B") is acceptable according to BCP 47, but I haven't investigated carefully yet. And I guess I'm raising this ticket because I don't think BCP 47 conformance should have to be necessary in all cases of documenting language, and it might be helpful to allow @lang for more than just programming languages on the <code> element. "A" and "B" aren't known languages by the way, just indications that something is written in a different unknown but describable language.

[ebeshero]

It's not earth-shattering for the project, because we're introducing custom attributes anyway via ODD. But the quest for an appropriate attribute led us to @lang and a certain strange limitation in its application.

[sydb]

My instinct is that introducing another way to indicate human languages does the community a disservice, precisely because “BCP 47 conformance should have to be necessary in all cases of documenting language”. BCP 47 has an escape hatch that makes it extensible, as it were. Thus our current rules for handling unusual languages (or unusual indicators, I suppose) should work perfectly well, and do not even require a mapping if the scholars’ usual abbreviations match the regex [A-Za-z0-9]{1,8}.

• BCP 47 “private use” language tags match the regular expression x-[A-Za-z0-9]{1,8}.
• In #CHSH P5 says “[any such private use language tag] should be documented within the <language> element of the TEI header”. (Personally, I think it should say “must”, not “should”.)
• The content of the <language> may contain a mapping to the scholars’ usual abbreviation, but the Guidelines explicitly do not say how to do that.

So if the usual abbreviations are A, B, & C, passages in those languages are encoded in elements that have an @xml:lang value of "x-A", "x-B", or "x-C", and the <langUsage> might look like:

      <langUsage>
        <language ident="x-A">
          Darren Doyle’s Kryptonian
        </language>
        <language ident="x-B">
          Christine Schreyer’s Kryptonian
        </language>
        <language ident="x-C">
          Tolkien’s Elvish
        </language>
      </langUsage>

If the usual abbreviations contain non-alphanumeric characters or are > 8 chars long, a mapping should probably be provided. The project would need to wrestle with how to best do that; here is one idea:

      <langUsage>
        <language ident="x-ddk">
          Darren Doyle’s Kryptonian; equivalent to <idno type="ProjectLang">Kryptonian_01</idno>.
        </language>
        <language ident="x-csk">
          Christine Schreyer’s Kryptonian; equivalent to <idno type="ProjectLang">Kryptonian_02</idno>
        </language>
        <language ident="x-te">
          Tolkien’s Elvish; equivalent to <idno type="ProjectLang">Elvish_01</idno>
        </language>
      </langUsage>

@hcayless might recommend using @ana on <language> instead. I see nothing wrong with either approach.

And I do not, at first glance, see anything wrong with the singular use of @lang on <code>, although I think it might have been better named @formalLang, @computerLang, @in or some such.