Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
67 changes: 63 additions & 4 deletions crates/data/src/io.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1804,16 +1804,21 @@ fn out_of_lonlat(g: &Geometry) -> bool {
}

/// The type of a column of text values: Integer or Double when every non-empty value is one,
/// otherwise (or with no values) Text.
/// otherwise (or with no values) Text. A number whose digits a Double doesn't keep, such as a
/// 20-digit identifier, keeps its column Text, so no digit is rounded away ([`double_holds`]).
pub(crate) fn text_type<'a>(vals: impl Iterator<Item = &'a str>) -> FieldType {
let mut ty: Option<FieldType> = None;
// A whole number a Double can't hold: fine in an Integer column, not in a Double one.
let mut whole_past_double = false;
for v in vals.map(str::trim).filter(|s| !s.is_empty()) {
let t = if has_leading_zero(v) {
// Identifiers such as ZIP codes keep their leading zeros.
FieldType::Text
} else if v.parse::<i64>().is_ok() {
} else if let Ok(i) = v.parse::<i64>() {
// A whole number is held when its Double is the same number (in i128: no saturation).
whole_past_double |= (i as f64) as i128 != i128::from(i);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consistency (low confidence, not a data loss): whole numbers are now judged two ways. Those in the i64 range are judged by their exact f64 value, so 2^60 beside 0.5 stays Double. Whole numbers past i64::MAX go to double_holds, which applies the shortest-text digit rule. So 18446744073709551616 (2^64, exact in f64) beside 0.5 makes the column Text, though no digit is lost. This is conservative, and the test covers only the i64 case. Either accept it and say so in the text_type doc comment, or extend the exact-value check to u128-range whole numbers so both paths follow one rule.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left for the release quality pass: both ways keep every digit, so the difference is only which exact values read as Double past the i64 range.

FieldType::Integer
} else if v.parse::<f64>().is_ok_and(f64::is_finite) {
} else if double_holds(v) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Behaviour question (medium-low confidence, not a bug): this is stricter than "more digits than a Double holds". A value such as 0.10000000000000001 (what C's %.17g writes for 0.1) is a valid 17-digit Double, but its digits differ from the shortest round-trip text 0.1, so the whole column now becomes Text where it was Double before. The text is kept, so nothing is lost, but numeric columns from such exports stop working for symbology, sorting and statistics, and the switch is silent. Is that the intended trade-off? If so, the doc comment on text_type should say "digits that differ from the shortest text" rather than "more digits than a Double holds", and a notice (for example, a count of columns kept as Text for this reason) may be worth considering.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch; that wasn't intended. A value now counts as a Double when the Double, written with as many significant digits as the text has, gives the same digits back, so 0.10000000000000001 reads as 0.1 and 20-digit IDs still stay text. The doc comment says so and the test covers the 17-digit case.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug (high confidence): a 20-digit integer ID that ends in zeros still gets rounded silently.

double_holds strips trailing zeros before comparing, so 12345678901234567000 passes the check. The chain:

  • parse::<i64>() fails (the value is past i64::MAX), so the value goes to double_holds.
  • want is 12345678901234567 (17 digits after trimming the trailing 000).
  • The nearest f64 is 12345678901234567168, and {:.16e} prints 1.2345678901234567e19, so the digits match.
  • The check returns true, and parse_source stores Float(12345678901234567168) in a Double column.

So an ID written as ...567000 is read back as ...567168, with no notice. This is the same silent change the PR describes, and it is not covered by the new tests.

For text that has no ., e or E (an integer spelling), the check should be exact: the f64 printed with format!("{x:.0}") must equal the text's digits, with the sign and leading zeros removed and the trailing zeros kept. That still accepts 1e20 spelled out (100000000000000000000) and 2^64, since both are exact. A regression test is needed, for example a CSV column of 12345678901234567000 that must stay Text.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in the latest push: text written as a whole number past i64 is a Double only if its Double is that number exactly (compared in i128), so 12345678901234567000 keeps its column text and 2^64 reads as a number. Both are in the test.

FieldType::Double
} else {
FieldType::Text
Expand All @@ -1823,7 +1828,36 @@ pub(crate) fn text_type<'a>(vals: impl Iterator<Item = &'a str>) -> FieldType {
Some(prev) => widen(prev, Some(t)),
});
}
ty.unwrap_or(FieldType::Text)
match ty {
Some(FieldType::Double) if whole_past_double => FieldType::Text,
ty => ty.unwrap_or(FieldType::Text),
}
}

/// Whether a number's text reads as a finite Double that, written with as many significant digits,
/// gives the same digits back, so no digit is lost: `1.50`, `0.1`, `1e300` and
/// `0.10000000000000001` (0.1 to 17 digits) and `0.00000000023283064365386962890625` (2^-32
/// exactly) are; `12345678901234567891` and `0.12345678901234567891` aren't. No Double's exact
/// value has more than 767 significant digits, so longer text isn't one, and isn't formatted.
fn double_holds(v: &str) -> bool {
/// The digits of a number's mantissa without leading or trailing zeros.
fn digits(s: &str) -> String {
let mantissa = s.split(['e', 'E']).next().unwrap_or_default();
let all: String = mantissa.chars().filter(char::is_ascii_digit).collect();
all.trim_start_matches('0').trim_end_matches('0').to_string()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug (high confidence): a 20-digit identifier written with a trailing .0 or e0 is still rounded silently.

digits strips every trailing zero, including the zeros before the point. So 12345678901234567000.0 (and 12345678901234567000e0, 1234567890123456700e1) reduces to 12345678901234567, which is 17 digits. The nearest Double is 12345678901234567168, and format!("{:.16e}") gives 1.2345678901234567e19, so the digit comparison passes. text_type then returns Double and parse_source stores Float(12345678901234567168.0), with no notice. This is the same loss the PR describes, and the whole-number branch above already rejects the same value without the .0.

This also isn't a fix that the reply to the earlier thread addresses. In standard significant-figure counting, zeros before an explicit decimal point are significant (1000.0 has four), so this text has 20 significant digits. Float writers don't produce this shape either: %.17g and Python's repr use an exponent (1.2345678901234567e+19), and %f writes the exact expansion. So the text is an identifier, which is the case the PR exists to keep.

Suggested fix: keep the zeros before the point when counting the text's digits, and drop only the zeros after it. The comparison side must then compare the formatted digits without trimming, since 10.0 formats as 1.0e1 and would otherwise lose its digit. Both sites change together, so I have not given a suggestion block. Add regression cases for double_holds("12345678901234567000.0"), double_holds("12345678901234567000e0"), and keep double_holds("10.0"), "1.50" and "1e300" true.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same case as the earlier thread on this function: text with a decimal point or an exponent is read as a decimal at the precision of its digits, which is what a floating-point writer gives; an identifier written as a whole number is held exactly. Left for the release quality pass.

}
// A whole number written as one is held only exactly: `12345678901234567000` is a Double's
// 17 digits and zeros, but that Double is 12345678901234567168.
let whole = v.strip_prefix(['-', '+']).unwrap_or(v);
if !whole.is_empty() && whole.bytes().all(|b| b.is_ascii_digit()) {
// A whole Double's exact digits, written out (no finite Double has more than 309).
let digits = whole.trim_start_matches('0');
let digits = if digits.is_empty() { "0" } else { digits };
return digits.len() <= 309 && v.parse::<f64>().is_ok_and(|x| x.is_finite() && format!("{:.0}", x.abs()) == digits);
}
let want = digits(v);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug (high confidence): the same 20-digit identifier still rounds silently when written with a decimal point or exponent.

The whole-number branch above (line 1852) runs the exact check only when the text is all digits. 12345678901234567000.0 contains a ., so it skips that branch and reaches digits(v) here. That strips trailing zeros across the decimal point, leaving 12345678901234567 (17 digits). The nearest f64 is 12345678901234567168, and {:.16e} prints 1.2345678901234567e19, which matches. So double_holds returns true, and Value::parse_source (value.rs:97) stores Float(12345678901234567168.0) in a Double column. No notice is given.

The same path accepts 12345678901234567000e0, 12345678901234567000.00, and similar spellings of that identifier.

Suggested fix: in double_holds, when the fraction is all zeros or the exponent leaves the value whole, judge the value as a whole number and use the exact i128 check. A test for double_holds("12345678901234567000.0") should be added beside the assertion at io.rs:4475.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left for the release quality pass: text with a decimal point or an exponent is read as a decimal at the precision its digits give, and 12345678901234567000.0 is 17 significant digits and zeros, which is what writing a floating-point value gives (such a column was a Double where it came from). An identifier written as a whole number is held exactly since the last commit.

v.parse::<f64>()
.is_ok_and(|x| x.is_finite() && (want.is_empty() || want.len() <= 767 && digits(&format!("{x:.*e}", want.len() - 1)) == want))
}

/// Types a dataset's Text columns from their values (see [`text_type`]), for readers whose
Expand Down Expand Up @@ -4425,6 +4459,31 @@ mod tests {
assert_eq!(to_csv(&back).unwrap(), once, "export/import is stable");
}

#[test]
fn csv_digits_a_number_cannot_hold_stay_text() {
let text = "id,share,mixed,n,x\n12345678901234567891,0.12345678901234567891,9007199254740993,7,1.50\n2,0.5,1.5,8,1e300\n";
let (ds, _) = read_csv(text, "t").unwrap();
let types: Vec<FieldType> = ds.fields.iter().map(|f| f.ty).collect();
assert_eq!(types, [FieldType::Text, FieldType::Text, FieldType::Text, FieldType::Integer, FieldType::Double]);
assert_eq!(ds.value(0, 0), &Value::Text("12345678901234567891".into()));
assert_eq!(ds.value(0, 1), &Value::Text("0.12345678901234567891".into()));
assert_eq!(ds.value(0, 2), &Value::Text("9007199254740993".into()), "a whole number past 2^53 in a decimal column");
assert_eq!(ds.value(1, 4), &Value::Float(1e300));
// A number written to 17 digits, as `%.17g` writes 0.1, is a Double.
let (ds, _) = read_csv("v\n0.10000000000000001\n2.5\n", "t").unwrap();
assert_eq!((ds.fields[0].ty, ds.value(0, 0)), (FieldType::Double, &Value::Float(0.1)));
// So is one written out exactly, however many digits that takes.
assert!(double_holds("0.00000000023283064365386962890625") && !double_holds(&"1".repeat(800)));
// A whole number past i64 with trailing zeros is held only if exactly.
assert!(!double_holds("12345678901234567000") && double_holds("18446744073709551616"));
// At any size, with no saturation: i128::MAX isn't one, 2^200 is.
assert!(!double_holds("170141183460469231731687303715884105727"));
assert!(double_holds("1606938044258990275541962092341162602522202993782792835301376"));
// A whole number beside a decimal: 2^60 is a Double exactly, 10^18 + 100 isn't.
let (ds, _) = read_csv("a,b\n1152921504606846976,1000000000000000100\n0.5,0.5\n", "t").unwrap();
assert_eq!((ds.fields[0].ty, ds.fields[1].ty), (FieldType::Double, FieldType::Text));
}

/// CSV export writes numbers as stored (#513): through `Display` a 0.4 m² area in km² was 0.
#[test]
fn csv_floats_keep_full_precision() {
Expand Down
10 changes: 10 additions & 0 deletions crates/data/src/kml.rs
Original file line number Diff line number Diff line change
Expand Up @@ -616,6 +616,16 @@ mod tests {
<GroundOverlay><Icon><href>map.png</href></Icon></GroundOverlay>
</Document></kml>"##;

#[test]
fn digits_a_number_cannot_hold_stay_text() {
let doc = r#"<kml><Document><Placemark><ExtendedData><Data name="id"><value>12345678901234567891</value></Data>
</ExtendedData><Point><coordinates>1,2</coordinates></Point></Placemark></Document></kml>"#;
let layers = read("ids.kml", doc.as_bytes()).unwrap();
let ds = &layers[0].dataset;
let id = ds.field_index("id").unwrap();
assert_eq!(ds.value(0, id), &Value::Text("12345678901234567891".into()));
}

#[test]
fn reads_placemarks_folders_data_and_styles() {
let layers = read("trip.kml", DOC.as_bytes()).unwrap();
Expand Down
Loading