From: Tom Lane <tgl@sss.pgh.pa.us>
To: Dean Rasheed <dean.a.rasheed@gmail.com>
Cc: pgsql-hackers@lists.postgresql.org
Subject: Definitional issue: stddev_pop (and related) for 1 input
Date: Thu, 11 Jun 2020 14:06:06 -0400
Message-ID: <353062.1591898766@sss.pgh.pa.us> (raw)
Before v12, stddev_pop() had the following behavior with just a
single input value:
regression=# SELECT stddev_pop('42'::float8);
stddev_pop
------------
0
(1 row)
regression=# SELECT stddev_pop('inf'::float8);
stddev_pop
------------
NaN
(1 row)
regression=# SELECT stddev_pop('nan'::float8);
stddev_pop
------------
NaN
(1 row)
As of v12, though, all three cases produce 0. I am not sure what
to think about that with respect to an infinity input, but I'm
quite sure I don't like it for NaN input.
It looks like the culprit is the introduction of the "Youngs-Cramer"
algorithm in float8_accum: nothing is done to Sxx at the first iteration,
even if the input is inf or NaN. I'd be inclined to force Sxx to NaN
when the first input is NaN, and perhaps also when it's Inf.
Alternatively we could clean up in the finalization routine by noting
that Sx is Inf/NaN, but that seems messier. Thoughts?
(I came across this by noting that the results don't agree with
numeric accumulation, which isn't using Youngs-Cramer.)
regards, tom lane
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: tgl@sss.pgh.pa.us, dean.a.rasheed@gmail.com, pgsql-hackers@lists.postgresql.org
Subject: Re: Definitional issue: stddev_pop (and related) for 1 input
In-Reply-To: <353062.1591898766@sss.pgh.pa.us>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox