Skip to content

[Bug]: A Shapefile that names no encoding turns its non-UTF-8 text into U+FFFD without a notice #2324

Open

Description

@ro9er117911

Before you start

What happened?

A Shapefile whose .dbf has no language driver (byte 29 is 0x00) and no .cpg is read as UTF-8 (dbf::encoding, crates/data/src/dbf.rs). Its text values that aren't UTF-8 become U+FFFD. RustGIS gives no notice, so the layer looks loaded and its names are gone.

This is how many Shapefiles from Taiwan arrive: Big5 (code page 950) text, written by tools that leave byte 29 at zero and write no .cpg. The same path takes any undeclared legacy code page (Windows-1252, GBK, Shift-JIS).

The user guide documents the default ("Files without encoding metadata keep the UTF-8 default. Unsupported characters become replacement characters."). The read itself says nothing, though, while an unsupported .cpg or an unsupported .dbf code page each give a notice. AGENTS.md ("Never silent") asks a reader that approximates data to say what, how much and where.

Steps to reproduce

  1. Make a Shapefile with Big5 text, LDID 0 and no .cpg. With pyshp 3.1.9:
    import shapefile
    w = shapefile.Writer("big5pts", shapeType=shapefile.POINT, encoding="big5")
    w.field("NAME", "C", 40)
    w.point(121.5645, 25.0340); w.record("臺北101")
    w.point(120.3025, 22.6394); w.record("高雄車站")
    w.close()  # byte 29 of big5pts.dbf is 0x00, and no .cpg is written
  2. Run:
    rustgis-cli run --data big5pts.shp \
      --cmd 'share.exportLayer={"layer":"big5pts","path":"out.geojson"}'
  3. stderr has only rustgis-cli: Exported out.geojson. The properties in out.geojson are:
    [{'NAME': '�O�_101'}, {'NAME': '��������'}]
    
  4. Add big5pts.cpg containing 950 and run it again. The properties are [{'NAME': '臺北101'}, {'NAME': '高雄車站'}], so the bytes are intact and only the encoding is unknown.

The same happens with real downloads from Taiwan's open data portal (all under the Open Government Data License, version 1.0). Each has byte 29 0x00 and no .cpg, and its text decodes cleanly as code page 950. Read with a07f1da8 and exported as above:

Dataset Publisher Text values with non-ASCII bytes Became U+FFFD Encoding notice
嘉義市雨水下水道管線圖資 (the zip) Chiayi City 866 866 none
高雄市全區地籍段界圖 Kaohsiung City 1,955 1,955 none
臺北市共同管道圖資, 幹管.shp Taipei City 7,986 7,986 none

For contrast, 全臺保安林分布概略圖 (APO1200000251031.shp) has byte 29 0x4F and no .cpg. RustGIS reads it correctly (保安林類型=土砂捍止保安林), so only the file that names no encoding is affected.

Also seen in the Chiayi zip, and probably a separate issue: its entry names are code page 950 without the zip's UTF-8 flag, and the layer is named �B��_�u instead of 雨水_線.

Expected behavior

The layer comes with a notice when text was replaced: how many values, and in which fields. The notice can point to the remedy RustGIS already supports, a .cpg naming the code page (for example 950 for Big5). Clean UTF-8 files give no notice, as today.

Which encoding to use when a file names none is a separate question; I'll open a proposal for it rather than change the default here.

How are you running RustGIS?

Command line (rustgis-cli), and the Python package (rustgis 0.7.0 from PyPI), which shows the same values and no notice.

Version

a07f1da8 (main, built from source); crates/data/src/dbf.rs and crates/data/src/io.rs are the same on 55e1ede5. Also 0.7.0.

Operating system, browser and graphics

macOS on Apple Silicon.

Sample data

The script in step 1 makes the sample. The real files are linked in the table above.

Activity

  1. ro9er117911 commented on Nov 17, 2026

    @ro9er117911
    ContributorAuthor

    The same happens with Japan's National Land Numerical Information (国土数値情報). N03 administrative areas, Tokyo, 2023, published by MLIT, has N03-23_13_230101.dbf with byte 29 0x00 and no .cpg. Its text is Shift_JIS (code page 932): all 18,261 values with non-ASCII bytes decode strictly as Shift_JIS (東京都, 千代田区).

    Read with a07f1da8 and exported as in the steps above:

    • 6,177 features. All 18,261 of those values became U+FFFD, in N03_001, N03_003 and N03_004, with no encoding notice.
    • With a .cpg containing 932 beside it, they read as 東京都 and 千代田区, with no U+FFFD.

    With #2325 the read says N03-23_13_230101: 18261 text values aren't valid UTF-8 and were read with U+FFFD in place of the bytes that can't be read, in 'N03_001', 'N03_003', 'N03_004'; no .cpg file names another encoding RustGIS reads (…).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions