IDE dies silently (uncatchable StackOverflowException) when a font fallback face fails to load every glyph

[gm-bug-reporter]

Description

made by claude code

IDE dies silently (uncatchable StackOverflowException) when a font fallback face fails to load every glyph

Summary

If the IDE's glyph-fallback path selects a font face from which FreeType cannot load
any glyph, FontTTFforDPI.UpdateGlyphs and FontTTFforDPI.GetGlyph recurse into
each other without bound. The result is a .NET StackOverflowException, which cannot
be caught, so GameMaker-LTS2026.exe disappears with no error dialog and no chance to
save. There is no bad state in the project — the trigger is entirely outside it.

In my case the offending face was Mongolian Baiti, which a Windows update turned
into a font FreeType cannot rasterise. The IDE never renders Mongolian text; it only
reached that face while looking for a fallback glyph.

Environment

• GameMaker IDE 2026.0.0.16 (LTS2026), runtime 2026.0.0.23
• Windows 11 Pro, build 10.0.26100.8972
• Bundled FreeType: <install>\Vendor\freetype\freetype-x86_64-windows-Release\freetype.dll

What the user sees

The IDE vanishes roughly 15-20 s after a build starts. It happens with both
Create Executable (YYC) and a plain Windows Run (VM). Igor runs out-of-process and
finishes fine, so the build output is produced even though the IDE is gone. Building the
same project from the command line never crashes, because that path does no text
rendering.

Log evidence

C:\ProgramData\GameMakerStudio2-LTS2026\ui.log, every crashing session, always exactly
94 lines:

FontFreeType failed to load glyph='4',  for face='Mongolian Baiti'
FontFreeType failed to load glyph='5',  for face='Mongolian Baiti'
...
FontFreeType failed to load glyph='97', for face='Mongolian Baiti'
Exception in YYCodeEditor.DocumentView,Void DocumentChangedEvent(YYCodeEditor.DocumentView) :
  SharpFont.FreeTypeException: FreeType error: Invalid reference., glyphIndex 1380
   at YoYoStudio.Graphics.FontTTFforDPI.UpdateGlyphs(UInt32[] _requestedChars, Boolean _allowReplacements)
   at YoYoStudio.Graphics.FontTTFforDPI.GetGlyph(UInt32 _chr, Boolean _updateTextureSlotCache, Boolean _allowReplacements)
   at YoYoStudio.Graphics.FontTTFforDPI.UpdateGlyphs(...)
   at YoYoStudio.Graphics.FontTTFforDPI.GetGlyphWidths(String _text)
   at YoYoStudio.TextEditor.TextRenderer.CalculateLineDisplayMetrics(...)
   at YYCodeEditor.DocumentView.DocumentChangedEvent(DocumentView _view)

Glyph indices 4..97 are the 94 printable ASCII characters, i.e. the IDE is building a
full atlas for the fallback face and every single glyph fails.

WER for the process: module coreclr.dll, exception code 0xC00000FD (STACK_OVERFLOW).
The faulting offset moves between crashes, as expected for stack exhaustion.

Standalone reproduction (no IDE involved)

Using the IDE's own FreeType build via P/Invoke — FT_Init_FreeType, FT_New_Face,
FT_Set_Pixel_Sizes(0,16), FT_Load_Glyph:

File	Result
C:\Windows\Fonts\monbaiti.ttf (26100.8972, 579 752 B, v5.54)	FT_Load_Glyph returns 134 (0x86 Invalid_Reference) for all 3432 glyphs
same file, FT_LOAD_NO_HINTING (flag 2)	0 failures
monbaiti.ttf from 26100.1 (289 052 B, v5.53)	0 failures on all 1857 glyphs

FT_New_Face succeeds in every case — the face opens, only glyph loading fails.

Cause on the Windows side: the 26100.8972 file contains cvt and prep but no
fpgm table, so the TrueType bytecode references function definitions that do not
exist and the interpreter aborts. That is arguably a Microsoft bug, but the IDE should
not die because of it.

A sweep of every font in C:\Windows\Fonts with the same code found monbaiti.ttf to be
the only genuinely broken one. (marlett.ttf returns error 6 for indices past its glyph
count and mstmc.ttf is not a font — both are handled correctly by the IDE today.)

Steps to reproduce

1. On Windows 11 build 26100.8972 or later, confirm C:\Windows\Fonts\monbaiti.ttf is
   579 752 bytes (v5.54).
2. Open any project and start a build, or simply cause the code editor / Output window to
   render text containing a character missing from the editor font.
3. The IDE terminates silently. ui.log ends with the 94 lines above.

Replacing that file with the 26100.1 version from the component store
(C:\Windows\WinSxS\amd64_microsoft-windows-f..type-mongolianbaiti_*_10.0.26100.1_*\monbaiti.ttf)
makes the crash disappear completely — verified: zero failed to load glyph lines and a
successful build.

Suggested fixes (any one of these would prevent the crash)

1. Bound the recursion. UpdateGlyphs -> GetGlyph -> UpdateGlyphs must not be able
   to re-enter indefinitely; a depth guard or an explicit "already tried replacements"
   flag would turn this into a rendering glitch instead of process death.
2. Blacklist a failing fallback face. If a face fails to produce a glyph, drop it from
   the fallback chain for the session instead of retrying it per character. The IDE
   already does something similar for faces that fail FT_New_Face (see the existing
   Failed to load font folder ... mstmc.ttf handling) — the same treatment for faces
   that open but cannot rasterise would be enough.
3. Retry without hinting. FT_LOAD_NO_HINTING loads this font perfectly. Falling back
   to an unhinted load when a hinted load fails would fix the rendering as well as the
   crash.
4. Validate faces at discovery time by test-loading one glyph, not just opening the face.

Impact

Silent, unsaved-work-losing termination of the IDE, triggered by a system font the user
never chose to use, on a mainstream Windows build. It is very hard to diagnose from the
user side: there is no dialog, the build still succeeds because Igor is a separate
process, and command-line builds are unaffected.

Related, possibly a separate report

The same install also died with System.StackOverflowException alongside
OddlySpecific.Feather.FeatherModule.HandleParseQueue() in the crash dump (also
coreclr.dll, 0xC00000FD). Disabling Feather removed that one. Two different unbounded
recursions, same fatal outcome.

Note: the "Feather -> Enabled" preference is only persisted on a clean exit, so a user
whose IDE keeps crashing can untick it repeatedly and it will silently keep coming back
in local_settings.json. That is worth fixing on its own.

Steps To Reproduce

Steps to reproduce

1. On Windows 11 build 26100.8972 or later, confirm C:\Windows\Fonts\monbaiti.ttf is
   579 752 bytes (v5.54).
2. Open any project and start a build, or simply cause the code editor / Output window to
   render text containing a character missing from the editor font.
3. The IDE terminates silently. ui.log ends with the 94 lines above.

Replacing that file with the 26100.1 version from the component store
(C:\Windows\WinSxS\amd64_microsoft-windows-f..type-mongolianbaiti_*_10.0.26100.1_*\monbaiti.ttf)
makes the crash disappear completely — verified: zero failed to load glyph lines and a
successful build.

Which version of GameMaker are you reporting this issue for?

IDE v2026.0.0.16 Runtime v2026.0.0.23

Which operating system(s) are you seeing the problem on?

Windows 10.0.26200.0

Attached Files

• yoyo-bug-report.md
• GameMaker-LTS2026.exe.2244.dmp
• GameMaker-LTS2026.exe.7256.dmp
• GameMaker-LTS2026.exe.21996.dmp
• GameMaker-LTS2026.exe.11844.dmp
• GameMaker-LTS2026.exe.11108.dmp
• GameMaker-LTS2026.exe.10444.dmp
• ui.log

f0425285-7627-4a77-9232-a884812d5b06