Feature: add character/word/line index attributes to the text geometry

[lojjic]

Custom shaders applied to Text instances could enable some really nice animation effects. For many of these, you'd want to treat each character/glyph independently, and for that you need something in the shader telling you which character is currently being rendered.

Theoretically gl_InstanceID could be used for this, since we use instancing for the glyph quads. And this kinda-sorta works: https://codesandbox.io/s/zealous-water-m8lzq?file=/src/index.js -- but gl_InstanceID is only available in WebGL2, and it appears to be broken in ANGLE implementations when used within functions other than void main. So it's not realistically usable for this right now.

Instead, we could add our own instance attribute, something like attribute float charIndex;, which just holds an incrementing character index. Custom shaders could then make use of that.

I'd probably want to make it an opt-in feature, something like textmesh.includeCharIndexInShader = true, just to avoid creating that extra attribute array if it isn't needed.

[lojjic]

Also maybe: wordIndex, lineIndex ...?

[gsimone]

word and line would be enable per line and per word animation, much like https://greensock.com/splittext/ (which we can consider a good example of what this change would enable)

Am I right assuming this would also allow per-token (char, word, line) changes in the fragment shader?

[lojjic]

Am I right assuming this would also allow per-token (char, word, line) changes in the fragment shader?

You'd have to pass it from vertex to fragment as a varying, but yes. :)

[lojjic]

Down the rabbit hole we go...

I can think of uses for all the following indices, plus a total count for each one:

• charIndex, totalChars
• wordIndex, totalWords
• lineIndex, totalLines
• charInWordIndex, totalCharsInWord
• charInLineIndex, totalCharsInLine
• wordInLineIndex, totalWordsInLine

We'd want to make all of these opt-in and do some smart packing to minimize the number of new glsl attributes we're introducing. I'll have to think more about the API for this.

[gsimone]

we could let users pick ALL of them with flags, so

{
split: { words: true, characters: true, lines: true }
}

do some smart packing

Mind expanding on this? I'm curious 👐

[lojjic]

I like that idea of letting users choose the split(s).

By packing I just mean we don't want to add 12 new attribute float foo declarations or we'll hit the limit (I think it's 16 attributes total in webgl), but we could pack those into a maximum of 3 new attribute vec4 foo declarations, and then unpack them to nicer-looking float variables within the shader.

[gsimone]

Oh yeah, makes a lot of sense, thank you!

[lojjic]

Should the charIndex include whitespaces in its incrementing? Or just increment for visible glyphs? I'm leaning toward just visible glyphs.

[gsimone]

Counting whitespaces would create weird staggers when animating by index, so I'd also lean toward visible

[lojjic]

I've got lots of cleanup to do, but I have an initial POC of the above counts. Here are examples using the various index/total pairs to change color in a fragment shader:

charIndex / totalChars:

charInWordIndex / totalCharsInWord:

charInLineIndex / totalCharsInLine:

wordIndex / totalWords: (looks very similar to charIndex/totalChars in this example but there's a subtle difference)

wordInLineIndex / totalWordsInLine:

lineIndex / totalLines:

I managed to get all the data into a max of 3 uniforms + 2 attributes, which is pretty good. I still want to make them optional.

[gsimone]

Amazing! Let me know when you have a test version ready 😄

[FunMiles]

I was looking for doing a feature that this could address. I have a bunch of 3 letter labels to put around a sphere. I am making them always face the viewer and make adjustments so that on the screen they are always the same size, no matter the scaling applied in the view matrix or modeling matrix.
Right now, I create one Text object per label and apply a transformation.
I was thinking I could get a much better performance by packing all the labels in a single text and have an attribute to have the position on the sphere of each label to do the correct transformations in the vertex shader.
@lojjic can you document how to create custom shaders? I think adding the attribute should not be difficult as the existing method setAttribute of BufferGeometry should do.

The other option I was considering would be instanced rendering, since all my labels all have 3 letters. But that would require WebGL 2 for sure and might be too complex to be worth it, at least at the start.

[lojjic]

@FunMiles I think you're right, this may facilitate that sort of thing. If I'm understanding it correctly, optimization could involve two parts:

1. Moving the rotation/scaling logic from CPU-side JS to GPU-side vertex shader logic
2. That plus combining multiple pieces of text into a single draw call

For 1, take a look at this comment, that may actually be sufficient for your needs already. That's also a good demonstration of how to apply a custom shader, basically just assigning it as the material. I've used Troika's createDerivedMaterial utility for it there, but you could similarly pass any ShaderMaterial or a material with its own onBeforeCompile modifications.

For 2, I think you're right you could render a single Text and then use some attribute to displace individual characters around your sphere. The new char/word index attributes described in this issue may be of some help, but I'm not sure they're even necessary. You could probably just encode those displacements into a new InstancedBufferAttribute, where each of its vectors holds the displacement for one character (the Text's geometry is a simple quad that is instanced for each glyph, so additional InstancedBufferAttributes will be stepped through for each glyph.)

I'm interested to see if you have any luck doing this!