Skip to content

Test suite fails with UnicodeDecodeError on Windows with non-UTF-8 locale: read_text() calls missing encoding #3244

Description

@ShuQingDollarVoyager

Describe the bug

On Windows systems where the ANSI code page is not UTF-8 (e.g. Chinese-locale Windows, code page 936 / GBK), running the test suite fails with UnicodeDecodeError in two places:

  1. tests/examples/conftest.py:41 - (STORIES_DIR / manifest.toml).read_text() fails during collection (8 collection errors, blocking all of tests/examples):
tests\examples\conftest.py:41: in <module>
    MANIFEST = tomllib.loads((STORIES_DIR / manifest.toml).read_text())
E   UnicodeDecodeError: 'gbk' codec can't decode byte 0x94 in position 441: illegal multibyte sequence
  1. tests/examples/test_story_shape.py:30 - _parse reads story sources with path.read_text() (~70 test failures once collection succeeds):
tests\examples\test_story_shape.py:30: in _parse
    return ast.parse(path.read_text(), filename=str(path))
E   UnicodeDecodeError: 'gbk' codec can't decode byte 0x92 in position 1311: illegal multibyte sequence

Path.read_text() without an encoding argument uses locale.getpreferredencoding(), which is the ANSI code page on Windows (cp936/GBK on Chinese-locale systems, cp1252 on Western-locale systems). The files being read (manifest.toml, story sources) are UTF-8 and contain non-ASCII bytes, so decoding fails. CI does not catch this because windows-latest runners use cp1252, which happens to decode those byte sequences without raising.

Expected behavior

The test suite passes regardless of the host locale. TOML is UTF-8 by specification, and the story sources are UTF-8, so both call sites should pass encoding=utf-8 explicitly.

Environment

  • OS: Windows 10 (Chinese locale, ANSI code page 936/GBK)
  • Python: 3.11.14
  • SDK: main @ a4f4ccd

Additional context

Fix is two one-line changes adding encoding=utf-8; PR to follow.

Disclosure: this bug was found and the fix prepared with AI assistance (Claude Code); the change has been reviewed and verified by me (full suite green on the affected machine: 5574 passed).

Metadata

Metadata

Assignees

No one assigned

    Labels

    P3Nice to haves, rare edge cases

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions