Text rendering improvements

[mosra]

The text rendering library didn't age well and while trying to use it for the upcoming UI library (mosra/magnum-extras#2, not there yet) I got really angry at the shameful state of it. My previous job revolved around text a lot and, while this library was the prototype that I built upon, the actual real-world text rendering needed a lot more flexibility than what's provided here. Putting this braindump here before it fades away completely.

If somebody wants to tackle one of these things, I will be glad to help you and suggest a direction.

Things that are shitty and need to be fixed ASAP

• Introduce lower-level reusable & composable APIs that work on singular glyph runs and pre-allocated memory -- 568a420
  • Re-implement the existing Renderer interfaces on top of these -- 568a420
  • Rework (or ditch / completely replace?) the Renderer interfaces to be more efficient --
    8480912, 3c626b0, 69f2da2, f3d6ab4
    • Make it use some global index buffer. It's crazy to have it repeated in each and every instance -- it's just 0 1 2 0 2 3 4 5 6 4 6 7 ... over and over again. -- partially done in 3c626b0, can provide a custom allocator that reuses existing memory but it's still filled again by each renderer
    • Make it usable outside the "one GL buffer and mesh for each text block" use case -- 3c626b0
    • Rename it with a GL suffix to make room for Vulkan, WebGPU, etc. implementations -- 69f2da2
• Reduce the amount of allocations and braindead use of std::vector every time we need some array with size known in advance -- 568a420, ab8dc07, 2692ff2...
• Reduce the amount of virtual calls and batch things more. Currently each glyph on screen equals one virtual call. Crazy. It also unnecessarily complicates things further down, compared to just getting an array of glyph positions and then iterating over it to put them into final positions. -- ab8dc07

Things that need to be improved/fixed before going further

• Finally implement some proper texture atlas packing, as noted in the ancient Texture atlas creation #2 -- 66bf0b2
• Properly handled initialization/finalization of plugins with dependencies. Current way with static initialize() function is broken by design: initializing HarfBuzz plugin after FreeType will call FreeTypeFont::initialize() second time, finalizing HarfBuzz plugin will call FreeTypeFont::finalize() even though FreeType plugin is still used and thus breaks it. -- mosra/magnum-plugins@217832f
• Write a test for TextureTools::distanceField(), ensure that it doesn't have artifacts on older platforms, ensure that it works even on GLES and WebGL -- done in 209cdbc, 2e4beb3 and 8ba2cac
• Write a test for Text::GlyphCache and Text::DistanceFieldGlyphCache -- 82691fc
• Write tests for all font plugins and converters -- 4ed4e44, mosra/magnum-plugins@be2d715
• API-agnostic implementation of glyph cache so it's usable with non-GL APIs -- 834c5fe
  • Great, but now the class is non-movable due to the virtual destructor. Fix that. -- 43efac8

Making the renderer usable with real-world text and font features

Having automatic font fallback would make the engine better than 90% of competitors.

• Ability to specify glyph IDs or Unicode codepoints to add to the cache in addition to a string -- 631b9b8
• A way for the renderer to report missing glyphs to the glyph cache
• Incremental glyph cache updates -- 6707534
  • Incremental TextureTools::Atlas functionality -- 66bf0b2
  • Image::slice() that operate with PixelStorage parameters so we're able to upload image subdata Part of a larger chunk of work, here it's been worked around
• Incremental TextureTools::distanceField() -- ada0645, abf8a75, a2554cb
• Ability to have multiple fonts in the same glyph cache -- 6707534
• Ability to use the text renderer with more than one font -- 8480912
  • Selecting the proper one based on glyph availability or some user-provided predicate function. This would require larger redesign, but very rewarding when done. Possible use cases are combining Latin and CJK text because there is no single font that contains both nice-looking Latin and CJK characters or for example having different parts of the same text block in different font (think bold, italic).
    • Decide how to handle the glyph availability. Some systems are doing it per character combination (better looking but harder), some per-codepoint (easier but the combinations may not match).
  • Properly handling shaping across font boundaries (HarfBuzz has a "text context" for this) -- ab8dc07, mosra/magnum-plugins@0782cdf
  • Switching fonts based on special Unicode control characters (or, if there are none, using the private plane for it) wouldn't be able to use the shaping context that way, has to be a separate stream
• Ability to use Texture2DArray for the glyph cache instead of Texture2D so we can fit also big alphabets inside. And somehow make it optional or backwards compatible with GLES2 / WebGL 1 that doesn't have them. -- 8168a06
  • Rename the glyph caches with a GL suffix, to make room for Vulkan, WebGPU etc. implementations -- 42342cd
  • Ability to use cubemaps? Could help on ES2 platforms, it's not an array but it's 6x times as large as a 2D texture nobody cares about ES2 nowadays anymore
• Specifying OpenType features for (parts of) the shaped text -- 5009d38, ab8dc07, mosra/magnum-plugins@6bc898c
• Specifying not just font size but also variable OpenType features like interpolating between regular and bold variant
• Ability to create several "font variants" of different size (or OTF variable font features) inside a single AbstractFont instance, to not have to load the same font file several times
  • HB distinguishes between "face" and "font" for this
  • FT has setFontSize, how expensive is it to call it every time? any HB-like feature for this?
  • stb_truetype has nothing like that, can just use a different font size every time
  • have addVariant(size) / removeVariant(id) and then using that variant ID in createShaper()?
  • such variant then also has to be distinguished by the GlyphCache, i.e. a second argument to findFont() (and third to addFont())
  • have a feature flag that exposes this "variant caching", and a generic fallback maintained by the AbstractFont for fonts that don't need it
  • and a feature flag where the font has just one fixed size, for (shitty) bitmap fonts? or keep it like this?
  • also options to adjust line height, letter spacing etc
• Fun! Ability to draw RGBA glyphs (emojis!) along with single-channel glyphs -- some sort of two-pass rendering (slow) or a more complex shader that switches between textures on per-character basis (slow) or ...?
  • Could be done with a combined RGBA+single-channel cache, the landfill algorithm would do a max() / fill more slices for the RGBA images, and just one slice for single-channel
  • Then a per-character mask, where it's either all channels 1, or just one channel 1, in which case a vec4(dot()) of the pixel value is used
  • Part of OpenType since 1.7: https://github.com/googlei18n/color-emoji, the way to do it on Android, in Chrome(OS) and Firefox
  • Supported in FreeType since 2.5: example
  • Another emoji font: http://emojione.com/, https://github.com/Ranks/emojione/tree/master/assets/fonts
  • Monochrome fonts: http://www.webhostinghub.com/glyphs/, Bootstrap Icons etc.
  • There's also SVG fonts now, Noto Color Emoji

Improved text rendering capabilities

• Font view that allows the user to override font size, line advance etc. for particular rendering
• Renderer view that allows rendering a part of the text differently (color, fake bold etc.)
• Shear option for fake italics (easy!)
• Allow better batching
  • Allowing the renderer to specify different alignment etc. for each new block that is being added -- 8480912
  • Two-dimensional line advance to allow rendering table cells etc. in one row. Can we abuse some special ASCII characters to advance both vertically and horizontally? Tab stops?
  • Renderer block that allows to share a single buffer/mesh but modify subparts of it -- as of 8480912 can be done with custom allocators, plus the UI library implements a very flexible interface
• Aligning a text in a bounding box instead of just to cursor to avoid duplicating that functionality in user code, to be used by line breaking also
• Line breaking
  • Shaping first and breaking later for better performance (https://github.com/behdad/harfbuzz/issues/224)
  • Ellipsis for overflown text
  • Support for zero-width space in line breaking algorithm
  • Support for hyphenation point character in line breaking algorithm (ignored when rendering, but special handling when a line wraps on it)
  • Try to be conformant with http://unicode.org/reports/tr14/ (make use of https://github.com/adah1972/libunibreak?)
  • Use ideas from UNSAFE_TO_CONCAT [Was Clarify UNSAFE_TO_BREAK and expose OS2V2Tail::usMaxContext as context.] harfbuzz/harfbuzz#1463 (comment)

Unicode-aware text shaping

• Make HarfBuzz the go-to solution on all platforms instead of trying to emulate it poorly using other libraries. Works on Linux, should "just work" on OSX, I was able to compile it for Windows using CMake. What about Android? iOS? Emscripten? No. Instead broaden the API to make it possible to fully control HarfBuzz from it while still making it possible to use stb_truetype or plain bitmap fonts where advanced shaping isn't needed
• Ability to pass direction, script and language to the shaper -- 03c180d, ab8dc07, mosra/magnum-plugins@9672a9f
• Ability to autodetect these in the shaper (http://unicode.org/reports/tr24/ -- I read through it and implemented it, but most of it faded away already. HarfBuzz somehow exposes this in the API so one could maybe take it as a starting point.) Just let HarfBuzz do that on its own.

Improved text editing capabilities

For the UI.

• Moving cursor left/right in UTF-8 text
  • Make it failproof -- mosra/corrade@d90ac34
• A way to convert cursor position to screen coordinate -- e1c9c4d
  • And back for cursor positioning
  • Handle this properly with combined shapes and ligatures

Plugins

• Plugin using stb_truetype to make the usage easier on platforms that don't "just have" FreeType (stb_truetype font plugin magnum-plugins#12)
• Kerning in MagnumFont
• Kerning in FreeTypeFont
• Kerning in StbTrueTypeFont
• Investigate autohinting possibilities in StbTrueTypeFont to make it less blurry and differing in size
• AnyFont so one can just not care about which plugin is getting used
  • OpenTypeFont alias -- mosra/magnum-plugins@2c36ec4
• Plugin that can decompress WOFF2 fonts and then delegate the TTF elsewhere, such as stb_truetype, which doesn't have WOFF2 support for obvious reasons: https://github.com/google/woff2 (does it support OTF also?)
  • Make a FontConverter plugin with this library also

Tools

• magnum-fontconverter --info, like is already done for images and scenes
  • Glyph count
  • Font features
  • Glyph<->Unicode mapping, listing glyph alternates
• Glyph ID <-> name mapping, https://freetype.org/freetype2/docs/reference/ft2-information_retrieval.html -- 477b310, mosra/magnum-plugins@c8d40d9
• Ability to create a subset of a font (HarfBuzzFontConverter that implements what hb-subset does)
  • https://docs.google.com/document/d/1_vZrt97OorJ0jA1YzJ29LRcGr3YGrNJANdOABjVZGEs/ is a nice overview of its capabilities
• Ability to save a glyph cache, finally (Feature Request: load saved glyph cache texture for use in font #488)
• Generating distance field caches and embedding them in TTF/OTFs in a way compatible with Qt: https://web.archive.org/web/20181124183542/https://blog.qt.io/blog/2018/10/10/introducing-distance-field-generator/
  • Blog post not available anymore, the tool is impossible to google for https://doc.qt.io/qt-6/qtdistancefieldgenerator-index.html (intentionally buried?)
  • Internally it's unfortunately a silly uncompressed image, that's quite awful https://github.com/qt/qttools/blob/d221fb51365280955376bd43dbe4e640fed6c8e2/src/distancefieldgenerator/mainwindow.cpp#L389-L559
  • Do it instead in a way that makes it possible to interpret the data as a serialized SceneData + ImageData instance? (Zero-copy importer plugin APIs #240)
  • Or the other way, embedding glyph info in an image? Such as in https://github.com/skeeto/scratch/blob/master/pngattach/pngattach.c

Shaders and rendering

• Per-character coloring
• Underlined / strikethrough text
  • Shader getting a distance to the underline in vertex data, basically like a SDF
  • What about underline that is outside of the glyph quad? Uh...
  • Double underlines, wiggly lines how?
• Drop shadow etc. (inspiration: https://www.youtube.com/watch?v=q3ROZmdu65o)
  • Could work even with non-SDF text, just a second texture lookup that's offset a bit (assuming it doesn't overlap too much with composed glyphs)
• sRGB and gamma-aware rendering
  • http://www.freetype.org/freetype2/docs/text-rendering-general.html
  • https://bsky.app/profile/mikkomononen.bsky.social/post/3lhdrl43r422x
• Having the glyph cache completely on the GPU, passing just a list of glyph IDs and positions and leaving the rest on the shader
  • Tried that, unfortunately it led to exhausting UBO limits with just about 2K glyphs, or 4K with some extremely complex packing
  • From a memory usage PoV would only make sense with instancing and instanced quads are likely not faster than just generating them CPU-side. Generated quads still need 4x position + 4x glyph ID (+ vertex ID) per glyph, which is only a marginal improvement over storing 4x position + 4x texcoord per glyph.
• Subpixel rendering with the distance field
• Improved distance field rendering with subsampling/mipmaps
  • https://www.reddit.com/r/gamedev/comments/2879jd/just_found_out_about_signed_distance_field_text/cicatot
  • https://github.com/paulhoux/Cinder-Samples/tree/master/TextRendering
• Trying something better/other than current distance-field implementation
  • http://wdobbie.com/post/gpu-text-rendering-with-vector-textures/
  • https://github.com/Chlumsky/msdfgen, https://github.com/nyyManni/msdfgl
  • https://medium.com/@evanwallace/easy-scalable-text-rendering-on-the-gpu-c3f4d782c5ac#.gdsqful7b
  • https://github.com/behdad/glyphy
  • Slug and https://github.com/mightycow/Sluggish
  • https://www.superluminal.eu/2018/10/29/16xaa-font-rendering-using-coverage-masks-part-i/
• Optimizing glyph shapes beyond simple rectangles -- https://twitter.com/EricLengyel/status/828548714578878465 -- might have huge non-negligible impact with large text
• Design/optimize for this use case: https://twitter.com/EricLengyel/status/831773058235060230

Making the renderer usable with BIDI

Specs: http://unicode.org/reports/tr9/

• Direction autodetection based on script (easy) HarfBuzz does this implicitly as of 0782cdf5e0b2f588bdbde6aa139e9fcb556c5bda
• Proper BIDI algorithm implementation is probably not feasible to maintain in the context of Magnum, add a AbstractLayouter interface and expose it via plugins
  • Fribidi
  • https://github.com/Tehreer/SheenBidi
• https://github.com/HOST-Oman/libraqm, does it make sense to try to integrate it as a whole AbstractLayouter?
  • It's just 2K lines that call into HB, FT or BIDI, and the direct dependency on a font makes it rather hard to expose in a composable way (i.e., couldn't layout with this and use stb_truetype for rendering for example).
  • Doesn't support line wrapping yet, which I'd have to implement myself anyway.

[mosra]

New tech just keeps appearing: https://github.com/Chlumsky/msdfgen

[mosra]

Color emoji in OTF fonts: https://github.com/googlei18n/color-emoji, https://github.com/googlei18n/noto-emoji Yay!

[mosra]

Safe-to-break API appearing in HarfBuzz: https://github.com/behdad/harfbuzz/issues/224

[ytain]

Also for inspiration to get more creative with text rendering: https://www.youtube.com/watch?v=q3ROZmdu65o (found from there https://news.ycombinator.com/item?id=10824989 )

Also another one https://github.com/behdad/glyphy - Vector based SDF

[ytain]

Another article on rendering using vectors instead of bitmaps https://medium.com/@evanwallace/easy-scalable-text-rendering-on-the-gpu-c3f4d782c5ac#.gdsqful7b

[mosra]

@ytain wow, that vector rendering is really neat! Knew about Glyphy but forgot to add it. Updated the list, thank you!

[mosra]

stb_truetype font plugin merged: mosra/magnum-plugins#12

[mosra]

More emoji fonts: http://emojione.com/, https://github.com/Ranks/emojione/tree/master/assets/fonts

@mosra I've been using stb_truetype, but will we ever get an AnyFontImporter or something?

[mosra]

@alicemargatroid there's currently not much use of AnyFont-like plugin, because the engine supports mainly just the TTF format.

I think you might want to use the TrueTypeFont plugin alias -- it maps to StbTrueTypeFont, FreeTypeFont or HarfBuzzFont based on which one of them is available, so then you don't need to bother about particular plugin implementation in you app.