Repository navigation
fix(data): long digit strings in CSV, KML and GPX stay text instead of being rounded #1924
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
69cbbb1
76abc9e
4986f36
0970ef3
ad3cf82
a6ea9ca
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -1804,16 +1804,21 @@ fn out_of_lonlat(g: &Geometry) -> bool { | |
| } | ||
|
|
||
| /// The type of a column of text values: Integer or Double when every non-empty value is one, | ||
| /// otherwise (or with no values) Text. | ||
| /// otherwise (or with no values) Text. A number whose digits a Double doesn't keep, such as a | ||
| /// 20-digit identifier, keeps its column Text, so no digit is rounded away ([`double_holds`]). | ||
| pub(crate) fn text_type<'a>(vals: impl Iterator<Item = &'a str>) -> FieldType { | ||
| let mut ty: Option<FieldType> = None; | ||
| // A whole number a Double can't hold: fine in an Integer column, not in a Double one. | ||
| let mut whole_past_double = false; | ||
| for v in vals.map(str::trim).filter(|s| !s.is_empty()) { | ||
| let t = if has_leading_zero(v) { | ||
| // Identifiers such as ZIP codes keep their leading zeros. | ||
| FieldType::Text | ||
| } else if v.parse::<i64>().is_ok() { | ||
| } else if let Ok(i) = v.parse::<i64>() { | ||
| // A whole number is held when its Double is the same number (in i128: no saturation). | ||
| whole_past_double |= (i as f64) as i128 != i128::from(i); | ||
| FieldType::Integer | ||
| } else if v.parse::<f64>().is_ok_and(f64::is_finite) { | ||
| } else if double_holds(v) { | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Behaviour question (medium-low confidence, not a bug): this is stricter than "more digits than a Double holds". A value such as
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Good catch; that wasn't intended. A value now counts as a Double when the Double, written with as many significant digits as the text has, gives the same digits back, so
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Bug (high confidence): a 20-digit integer ID that ends in zeros still gets rounded silently.
So an ID written as For text that has no
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Fixed in the latest push: text written as a whole number past i64 is a Double only if its Double is that number exactly (compared in i128), so |
||
| FieldType::Double | ||
| } else { | ||
| FieldType::Text | ||
|
|
@@ -1823,7 +1828,36 @@ pub(crate) fn text_type<'a>(vals: impl Iterator<Item = &'a str>) -> FieldType { | |
| Some(prev) => widen(prev, Some(t)), | ||
| }); | ||
| } | ||
| ty.unwrap_or(FieldType::Text) | ||
| match ty { | ||
| Some(FieldType::Double) if whole_past_double => FieldType::Text, | ||
| ty => ty.unwrap_or(FieldType::Text), | ||
| } | ||
| } | ||
|
|
||
| /// Whether a number's text reads as a finite Double that, written with as many significant digits, | ||
| /// gives the same digits back, so no digit is lost: `1.50`, `0.1`, `1e300` and | ||
| /// `0.10000000000000001` (0.1 to 17 digits) and `0.00000000023283064365386962890625` (2^-32 | ||
| /// exactly) are; `12345678901234567891` and `0.12345678901234567891` aren't. No Double's exact | ||
| /// value has more than 767 significant digits, so longer text isn't one, and isn't formatted. | ||
| fn double_holds(v: &str) -> bool { | ||
| /// The digits of a number's mantissa without leading or trailing zeros. | ||
| fn digits(s: &str) -> String { | ||
| let mantissa = s.split(['e', 'E']).next().unwrap_or_default(); | ||
| let all: String = mantissa.chars().filter(char::is_ascii_digit).collect(); | ||
| all.trim_start_matches('0').trim_end_matches('0').to_string() | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Bug (high confidence): a 20-digit identifier written with a trailing
This also isn't a fix that the reply to the earlier thread addresses. In standard significant-figure counting, zeros before an explicit decimal point are significant ( Suggested fix: keep the zeros before the point when counting the text's digits, and drop only the zeros after it. The comparison side must then compare the formatted digits without trimming, since
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Same case as the earlier thread on this function: text with a decimal point or an exponent is read as a decimal at the precision of its digits, which is what a floating-point writer gives; an identifier written as a whole number is held exactly. Left for the release quality pass. |
||
| } | ||
| // A whole number written as one is held only exactly: `12345678901234567000` is a Double's | ||
| // 17 digits and zeros, but that Double is 12345678901234567168. | ||
| let whole = v.strip_prefix(['-', '+']).unwrap_or(v); | ||
| if !whole.is_empty() && whole.bytes().all(|b| b.is_ascii_digit()) { | ||
| // A whole Double's exact digits, written out (no finite Double has more than 309). | ||
| let digits = whole.trim_start_matches('0'); | ||
| let digits = if digits.is_empty() { "0" } else { digits }; | ||
| return digits.len() <= 309 && v.parse::<f64>().is_ok_and(|x| x.is_finite() && format!("{:.0}", x.abs()) == digits); | ||
| } | ||
| let want = digits(v); | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Bug (high confidence): the same 20-digit identifier still rounds silently when written with a decimal point or exponent. The whole-number branch above (line 1852) runs the exact check only when the text is all digits. The same path accepts Suggested fix: in
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Left for the release quality pass: text with a decimal point or an exponent is read as a decimal at the precision its digits give, and |
||
| v.parse::<f64>() | ||
| .is_ok_and(|x| x.is_finite() && (want.is_empty() || want.len() <= 767 && digits(&format!("{x:.*e}", want.len() - 1)) == want)) | ||
| } | ||
|
|
||
| /// Types a dataset's Text columns from their values (see [`text_type`]), for readers whose | ||
|
|
@@ -4425,6 +4459,31 @@ mod tests { | |
| assert_eq!(to_csv(&back).unwrap(), once, "export/import is stable"); | ||
| } | ||
|
|
||
| #[test] | ||
| fn csv_digits_a_number_cannot_hold_stay_text() { | ||
| let text = "id,share,mixed,n,x\n12345678901234567891,0.12345678901234567891,9007199254740993,7,1.50\n2,0.5,1.5,8,1e300\n"; | ||
| let (ds, _) = read_csv(text, "t").unwrap(); | ||
| let types: Vec<FieldType> = ds.fields.iter().map(|f| f.ty).collect(); | ||
| assert_eq!(types, [FieldType::Text, FieldType::Text, FieldType::Text, FieldType::Integer, FieldType::Double]); | ||
| assert_eq!(ds.value(0, 0), &Value::Text("12345678901234567891".into())); | ||
| assert_eq!(ds.value(0, 1), &Value::Text("0.12345678901234567891".into())); | ||
| assert_eq!(ds.value(0, 2), &Value::Text("9007199254740993".into()), "a whole number past 2^53 in a decimal column"); | ||
| assert_eq!(ds.value(1, 4), &Value::Float(1e300)); | ||
| // A number written to 17 digits, as `%.17g` writes 0.1, is a Double. | ||
| let (ds, _) = read_csv("v\n0.10000000000000001\n2.5\n", "t").unwrap(); | ||
| assert_eq!((ds.fields[0].ty, ds.value(0, 0)), (FieldType::Double, &Value::Float(0.1))); | ||
| // So is one written out exactly, however many digits that takes. | ||
| assert!(double_holds("0.00000000023283064365386962890625") && !double_holds(&"1".repeat(800))); | ||
| // A whole number past i64 with trailing zeros is held only if exactly. | ||
| assert!(!double_holds("12345678901234567000") && double_holds("18446744073709551616")); | ||
| // At any size, with no saturation: i128::MAX isn't one, 2^200 is. | ||
| assert!(!double_holds("170141183460469231731687303715884105727")); | ||
| assert!(double_holds("1606938044258990275541962092341162602522202993782792835301376")); | ||
| // A whole number beside a decimal: 2^60 is a Double exactly, 10^18 + 100 isn't. | ||
| let (ds, _) = read_csv("a,b\n1152921504606846976,1000000000000000100\n0.5,0.5\n", "t").unwrap(); | ||
| assert_eq!((ds.fields[0].ty, ds.fields[1].ty), (FieldType::Double, FieldType::Text)); | ||
| } | ||
|
|
||
| /// CSV export writes numbers as stored (#513): through `Display` a 0.4 m² area in km² was 0. | ||
| #[test] | ||
| fn csv_floats_keep_full_precision() { | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Consistency (low confidence, not a data loss): whole numbers are now judged two ways. Those in the i64 range are judged by their exact f64 value, so
2^60beside0.5stays Double. Whole numbers past i64::MAX go todouble_holds, which applies the shortest-text digit rule. So18446744073709551616(2^64, exact in f64) beside0.5makes the column Text, though no digit is lost. This is conservative, and the test covers only the i64 case. Either accept it and say so in thetext_typedoc comment, or extend the exact-value check tou128-range whole numbers so both paths follow one rule.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Left for the release quality pass: both ways keep every digit, so the difference is only which exact values read as Double past the i64 range.