Repository navigation
Description
Before you start
- I searched the existing issues and did not find a duplicate. ([Bug]: Shapefiles with .dbf code page 0x59 (Spanish ANSI, Windows-1252) are read as UTF-8 #1079 and fix(data): read the .dbf language drivers dbase doesn't know (#1079) #1082 cover
.dbffiles that do name a code page; [Bug]: A CSV in Latin-1 (Windows-1252) can't be opened ("is not UTF-8 text") #1303 covers CSV; Codebase audit: crash safety, data-loss notices, UI responsiveness & hygiene #2008's Never-silent list doesn't include it. I checked main at55e1ede5, open issues and open pull requests and found no report or fix.) - This is a bug report, not a usage question.
What happened?
A Shapefile whose .dbf has no language driver (byte 29 is 0x00) and no .cpg is read as UTF-8 (dbf::encoding, crates/data/src/dbf.rs). Its text values that aren't UTF-8 become U+FFFD. RustGIS gives no notice, so the layer looks loaded and its names are gone.
This is how many Shapefiles from Taiwan arrive: Big5 (code page 950) text, written by tools that leave byte 29 at zero and write no .cpg. The same path takes any undeclared legacy code page (Windows-1252, GBK, Shift-JIS).
The user guide documents the default ("Files without encoding metadata keep the UTF-8 default. Unsupported characters become replacement characters."). The read itself says nothing, though, while an unsupported .cpg or an unsupported .dbf code page each give a notice. AGENTS.md ("Never silent") asks a reader that approximates data to say what, how much and where.
Steps to reproduce
- Make a Shapefile with Big5 text, LDID 0 and no
.cpg. With pyshp 3.1.9:import shapefile w = shapefile.Writer("big5pts", shapeType=shapefile.POINT, encoding="big5") w.field("NAME", "C", 40) w.point(121.5645, 25.0340); w.record("臺北101") w.point(120.3025, 22.6394); w.record("高雄車站") w.close() # byte 29 of big5pts.dbf is 0x00, and no .cpg is written
- Run:
rustgis-cli run --data big5pts.shp \ --cmd 'share.exportLayer={"layer":"big5pts","path":"out.geojson"}' - stderr has only
rustgis-cli: Exported out.geojson. The properties inout.geojsonare:[{'NAME': '�O�_101'}, {'NAME': '��������'}] - Add
big5pts.cpgcontaining950and run it again. The properties are[{'NAME': '臺北101'}, {'NAME': '高雄車站'}], so the bytes are intact and only the encoding is unknown.
The same happens with real downloads from Taiwan's open data portal (all under the Open Government Data License, version 1.0). Each has byte 29 0x00 and no .cpg, and its text decodes cleanly as code page 950. Read with a07f1da8 and exported as above:
| Dataset | Publisher | Text values with non-ASCII bytes | Became U+FFFD | Encoding notice |
|---|---|---|---|---|
| 嘉義市雨水下水道管線圖資 (the zip) | Chiayi City | 866 | 866 | none |
| 高雄市全區地籍段界圖 | Kaohsiung City | 1,955 | 1,955 | none |
臺北市共同管道圖資, 幹管.shp |
Taipei City | 7,986 | 7,986 | none |
For contrast, 全臺保安林分布概略圖 (APO1200000251031.shp) has byte 29 0x4F and no .cpg. RustGIS reads it correctly (保安林類型=土砂捍止保安林), so only the file that names no encoding is affected.
Also seen in the Chiayi zip, and probably a separate issue: its entry names are code page 950 without the zip's UTF-8 flag, and the layer is named �B��_�u instead of 雨水_線.
Expected behavior
The layer comes with a notice when text was replaced: how many values, and in which fields. The notice can point to the remedy RustGIS already supports, a .cpg naming the code page (for example 950 for Big5). Clean UTF-8 files give no notice, as today.
Which encoding to use when a file names none is a separate question; I'll open a proposal for it rather than change the default here.
How are you running RustGIS?
Command line (rustgis-cli), and the Python package (rustgis 0.7.0 from PyPI), which shows the same values and no notice.
Version
a07f1da8 (main, built from source); crates/data/src/dbf.rs and crates/data/src/io.rs are the same on 55e1ede5. Also 0.7.0.
Operating system, browser and graphics
macOS on Apple Silicon.
Sample data
The script in step 1 makes the sample. The real files are linked in the table above.
Activity
ro9er117911 commentedon Nov 17, 2026
The same happens with Japan's National Land Numerical Information (国土数値情報). N03 administrative areas, Tokyo, 2023, published by MLIT, has
N03-23_13_230101.dbfwith byte 290x00and no.cpg. Its text is Shift_JIS (code page 932): all 18,261 values with non-ASCII bytes decode strictly as Shift_JIS (東京都,千代田区).Read with
a07f1da8and exported as in the steps above:N03_001,N03_003andN03_004, with no encoding notice..cpgcontaining932beside it, they read as東京都and千代田区, with no U+FFFD.With #2325 the read says
N03-23_13_230101: 18261 text values aren't valid UTF-8 and were read with U+FFFD in place of the bytes that can't be read, in 'N03_001', 'N03_003', 'N03_004'; no .cpg file names another encoding RustGIS reads (…).