Fix si_format() producing out-of-range mantissa like "1000.0 k"#17
Open
patchwright wants to merge 1 commit into
Open
Fix si_format() producing out-of-range mantissa like "1000.0 k"#17patchwright wants to merge 1 commit into
patchwright wants to merge 1 commit into
Conversation
split() chooses the SI prefix from the unrounded mantissa, but si_format() rounds the mantissa to `precision` digits afterwards. A value just under 1000 (e.g. 999.999 from si_format(999999)) rounds up to 1000 and is printed with the smaller prefix -> "1000.0 k" instead of "1.0 M". Make split()'s >= 1000 promotion test the value rounded to `precision`, so it carries into the next prefix. Values below the rounding boundary and higher-precision inputs are unaffected. Adds parametrized regression cases; the existing 38 si_format cases are unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
si_format()can print a mantissa of1000with the smaller prefix, which isoutside the
[1, 1000)range each SI prefix represents:Root cause
split()selects the prefix by promoting the mantissa whilevalue >= 1000.0,but it works on the unrounded value.
si_format()then rounds the mantissa toprecisiondigits ("%.1f" % 999.999→"1000.0"). So a value just below 1000rounds up to 1000 after the prefix has already been chosen.
Fix
Make
split()'s promotion test the value rounded toprecision, so a mantissathat will round to 1000 is carried into the next prefix:
Values below the rounding boundary (
si_format(999500, 1) == "999.5 k") andhigher-precision inputs (
si_format(999999, 4) == "999.9990 k") are unaffected.Tests
test_si_format_rounding_rolloverwith the cases above; it fails onmasterand passes with the fix.si_formatcases are unchanged — full suite (112 tests) green.