From: Tom Lane <tgl@sss.pgh.pa.us>
To: Andrew Gierth <andrew@tao11.riddles.org.uk>
Cc: Gilleain Torrance <Gilleain.Torrance@alfasystems.com>
Cc: pgsql-bugs\@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
Subject: Re: Incorrect rounding of double values at max precision
Date: Mon, 21 Oct 2019 22:41:11 -0400
Message-ID: <26443.1571712071@sss.pgh.pa.us> (raw)
In-Reply-To: <875zkhhjs0.fsf@news-spur.riddles.org.uk>
References: <DB3D1D143DDA534BBB39DE713E92343701CBA89718@GBEDBP01.chp.co.uk>
<5045.1571686957@sss.pgh.pa.us>
<875zkhhjs0.fsf@news-spur.riddles.org.uk>
Andrew Gierth <andrew@tao11.riddles.org.uk> writes:
> "Tom" == Tom Lane <tgl@sss.pgh.pa.us> writes:
> Tom> The other thing we could conceivably do is ask sprintf for more
> Tom> digits. But since those extra digit(s) aren't fully precise, I'm
> Tom> afraid that would likewise introduce as many oddities as it fixes.
> Tom> Still, it's somewhat interesting to wonder whether applying the
> Tom> Ryu algorithm would produce better or worse results on average.
> Hmm.
> The Ryu output values will still throw out edge cases similar to the
> above; for example, 502.15::float8 / 10 = 50.214999999999996 whereas
> 502.15::numeric / 10 = 50.215, so rounding the result of that to 2
> digits will give a different result.
Yeah. Worse, casting that to numeric currently gives the "correct"
result:
regression=# select (502.15::float8 / 10)::numeric;
numeric
---------
50.215
(1 row)
while if we changed float8_numeric to apply Ryu, the result would be
50.214999999999996. So that's not great. But there are other cases
where the result would be better than before, such as the OP's example
of 42258656681.38498::float8. I'd like to get my hands around how
many "better" and "worse" cases there would be, but I'm not sure how
to attack that question.
> Perhaps it would make more sense for the float8 to numeric cast to look
> at the requested typmod and use that for the conversion?
As things stand right now, float8_numeric has no idea what the target
typmod is; any typmod-driven rounding happens in a separate function
call afterwards. I don't recall whether the parser's casting
infrastructure could support merging those steps, and I'm not sure
it matters in most cases. Commonly, we don't have a target typmod.
(Still, if we do, having two separate rounding steps isn't nice.)
regards, tom lane
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-bugs@postgresql.org
Cc: tgl@sss.pgh.pa.us, andrew@tao11.riddles.org.uk, Gilleain.Torrance@alfasystems.com, pgsql-bugs@lists.postgresql.org
Subject: Re: Incorrect rounding of double values at max precision
In-Reply-To: <26443.1571712071@sss.pgh.pa.us>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox