Normalize integer numerics when performing aggregate queries#993
Open
sgrif wants to merge 1 commit into
Open
Conversation
The AVG function in PG will always convert any integer types to numeric. The scale of that result does not appear to be clearly defined, but appears to be a scale of 16 for small numbers and 0 for larger ones. Unfortunately, since this behavior isn't documented or defined, it's difficult for us to match. That said, normalizing integers in this case will prevent our values from diverging with larger values (e.g. very small fractional parts that a single shrad query would not return), while only causing us to diverge from PG by not including trailing zeroes on small integers (e.g. we will now return `85` instead of `85.0000000000000000`), which I believe is a net win overall
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
This comment has been minimized.
This comment has been minimized.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The AVG function in PG will always convert any integer types to numeric. The scale of that result does not appear to be clearly defined, but appears to be a scale of 16 for small numbers and 0 for larger ones. Unfortunately, since this behavior isn't documented or defined, it's difficult for us to match.
That said, normalizing integers in this case will prevent our values from diverging with larger values (e.g. very small fractional parts that a single shrad query would not return), while only causing us to diverge from PG by not including trailing zeroes on small integers (e.g. we will now return
85instead of85.0000000000000000), which I believe is a net win overall