pg.ddx.io  pgsql-committers@postgresql.org mailing list archive  
help / color / mirror / Atom feed
pgsql: Fix corner-case loss of precision in numeric pow() calculation
9+ messages / 1 participants
[nested] [flat]

* pgsql: Fix corner-case loss of precision in numeric pow() calculation
@ 2016-05-05 10:19 Dean Rasheed <dean.a.rasheed@gmail.com>
  0 siblings, 0 replies; 9+ messages in thread

From: Dean Rasheed @ 2016-05-05 10:19 UTC (permalink / raw)
  To: pgsql-committers

Fix corner-case loss of precision in numeric pow() calculation

Commit 7d9a4737c268f61fb8800957631f12d3f13be218 greatly improved the
accuracy of the numeric transcendental functions, however it failed to
consider the case where the result from pow() is close to the overflow
threshold, for example 0.12 ^ -2345.6. For such inputs, where the
result has more than 2000 digits before the decimal point, the decimal
result weight estimate was being clamped to 2000, leading to a loss of
precision in the final calculation.

Fix this by replacing the clamping code with an overflow test that
aborts the calculation early if the final result is sure to overflow,
based on the overflow limit in exp_var(). This provides the same
protection against integer overflow in the subsequent result scale
computation as the original clamping code, but it also ensures that
precision is never lost and saves compute cycles in cases that are
sure to overflow.

The new early overflow test works with the initial low-precision
result (expected to be accurate to around 8 significant digits) and
includes a small fuzz factor to ensure that it doesn't kick in for
values that would not overflow exp_var(), so the overall overflow
threshold of pow() is unchanged and consistent for all inputs with
non-integer exponents.

Author: Dean Rasheed
Reviewed-by: Tom Lane
Discussion: http://www.postgresql.org/message-id/CAEZATCUj3U-cQj0jjoia=qgs0SjE3auroxh8swvNKvZWUqegrg@mail.gmail....
See-also: http://www.postgresql.org/message-id/CAEZATCV7w+8iB=07dJ8Q0zihXQT1semcQuTeK+4_rogC_zq5Hw@mail.gmail....

Branch
------
master

Details
-------
http://git.postgresql.org/pg/commitdiff/18a02ad2a506e4425c6dd2ea235039cd5659467d

Modified Files
--------------
src/backend/utils/adt/numeric.c           | 20 ++++++++++++++++----
src/test/regress/expected/numeric_big.out | 14 ++++++++++++++
src/test/regress/sql/numeric_big.sql      | 10 ++++++++++
3 files changed, 40 insertions(+), 4 deletions(-)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers


^ permalink  raw  reply  [nested|flat] 9+ messages in thread

* pgsql: Fix corner-case loss of precision in numeric ln().
@ 2020-03-01 14:54 Dean Rasheed <dean.a.rasheed@gmail.com>
  0 siblings, 0 replies; 9+ messages in thread

From: Dean Rasheed @ 2020-03-01 14:54 UTC (permalink / raw)
  To: pgsql-committers@lists.postgresql.org

Fix corner-case loss of precision in numeric ln().

When deciding on the local rscale to use for the Taylor series
expansion, ln_var() neglected to account for the fact that the result
is subsequently multiplied by a factor of 2^(nsqrt+1), where nsqrt is
the number of square root operations performed in the range reduction
step, which can be as high as 22 for very large inputs. This could
result in a loss of precision, particularly when combined with large
rscale values, for which a large number of Taylor series terms is
required (up to around 400).

Fix by computing a few extra digits in the Taylor series, based on the
weight of the multiplicative factor log10(2^(nsqrt+1)). It remains to
be proven whether or not the other 8 extra digits used for the Taylor
series is appropriate, but this at least deals with the obvious
oversight of failing to account for the effects of the final
multiplication.

Per report from Justin AnyhowStep. Reviewed by Tom Lane.

Discussion: https://postgr.es/m/16280-279f299d9c06e56f@postgresql.org

Branch
------
master

Details
-------
https://git.postgresql.org/pg/commitdiff/43a899f41f46918a0bf442edb091b08c214c68f8

Modified Files
--------------
src/backend/utils/adt/numeric.c           | 10 +++++++++-
src/test/regress/expected/numeric_big.out | 19 +++++++++++++++++++
src/test/regress/sql/numeric_big.sql      | 14 ++++++++++++++
3 files changed, 42 insertions(+), 1 deletion(-)



^ permalink  raw  reply  [nested|flat] 9+ messages in thread

* pgsql: Fix corner-case loss of precision in numeric_power().
@ 2021-10-06 12:29 Dean Rasheed <dean.a.rasheed@gmail.com>
  0 siblings, 0 replies; 9+ messages in thread

From: Dean Rasheed @ 2021-10-06 12:29 UTC (permalink / raw)
  To: pgsql-committers@lists.postgresql.org

Fix corner-case loss of precision in numeric_power().

This fixes a loss of precision that occurs when the first input is
very close to 1, so that its logarithm is very small.

Formerly, during the initial low-precision calculation to estimate the
result weight, the logarithm was computed to a local rscale that was
capped to NUMERIC_MAX_DISPLAY_SCALE (1000). However, the base may be
as close as 1e-16383 to 1, hence its logarithm may be as small as
1e-16383, and so the local rscale needs to be allowed to exceed 16383,
otherwise all precision is lost, leading to a poor choice of rscale
for the full-precision calculation.

Fix this by removing the cap on the local rscale during the initial
low-precision calculation, as we already do in the full-precision
calculation. This doesn't change the fact that the initial calculation
is a low-precision approximation, computing the logarithm to around 8
significant digits, which is very fast, especially when the base is
very close to 1.

Patch by me, reviewed by Alvaro Herrera.

Discussion: https://postgr.es/m/CAEZATCV-Ceu%2BHpRMf416yUe4KKFv%3DtdgXQAe5-7S9tD%3D5E-T1g%40mail.gmail.com

Branch
------
master

Details
-------
https://git.postgresql.org/pg/commitdiff/e54a758d24dab056bb7f50d26c57a3c8761cc44a

Modified Files
--------------
src/backend/utils/adt/numeric.c       | 6 +++++-
src/test/regress/expected/numeric.out | 6 ++++++
src/test/regress/sql/numeric.sql      | 1 +
3 files changed, 12 insertions(+), 1 deletion(-)



^ permalink  raw  reply  [nested|flat] 9+ messages in thread

* pgsql: Fix corner-case loss of precision in numeric_power().
@ 2021-10-06 12:29 Dean Rasheed <dean.a.rasheed@gmail.com>
  0 siblings, 0 replies; 9+ messages in thread

From: Dean Rasheed @ 2021-10-06 12:29 UTC (permalink / raw)
  To: pgsql-committers@lists.postgresql.org

Fix corner-case loss of precision in numeric_power().

This fixes a loss of precision that occurs when the first input is
very close to 1, so that its logarithm is very small.

Formerly, during the initial low-precision calculation to estimate the
result weight, the logarithm was computed to a local rscale that was
capped to NUMERIC_MAX_DISPLAY_SCALE (1000). However, the base may be
as close as 1e-16383 to 1, hence its logarithm may be as small as
1e-16383, and so the local rscale needs to be allowed to exceed 16383,
otherwise all precision is lost, leading to a poor choice of rscale
for the full-precision calculation.

Fix this by removing the cap on the local rscale during the initial
low-precision calculation, as we already do in the full-precision
calculation. This doesn't change the fact that the initial calculation
is a low-precision approximation, computing the logarithm to around 8
significant digits, which is very fast, especially when the base is
very close to 1.

Patch by me, reviewed by Alvaro Herrera.

Discussion: https://postgr.es/m/CAEZATCV-Ceu%2BHpRMf416yUe4KKFv%3DtdgXQAe5-7S9tD%3D5E-T1g%40mail.gmail.com

Branch
------
REL_14_STABLE

Details
-------
https://git.postgresql.org/pg/commitdiff/8e26b868d5cfe9c746578aa3b172b6f24a5a310c

Modified Files
--------------
src/backend/utils/adt/numeric.c       | 6 +++++-
src/test/regress/expected/numeric.out | 6 ++++++
src/test/regress/sql/numeric.sql      | 1 +
3 files changed, 12 insertions(+), 1 deletion(-)



^ permalink  raw  reply  [nested|flat] 9+ messages in thread

* pgsql: Fix corner-case loss of precision in numeric_power().
@ 2021-10-06 12:29 Dean Rasheed <dean.a.rasheed@gmail.com>
  0 siblings, 0 replies; 9+ messages in thread

From: Dean Rasheed @ 2021-10-06 12:29 UTC (permalink / raw)
  To: pgsql-committers@lists.postgresql.org

Fix corner-case loss of precision in numeric_power().

This fixes a loss of precision that occurs when the first input is
very close to 1, so that its logarithm is very small.

Formerly, during the initial low-precision calculation to estimate the
result weight, the logarithm was computed to a local rscale that was
capped to NUMERIC_MAX_DISPLAY_SCALE (1000). However, the base may be
as close as 1e-16383 to 1, hence its logarithm may be as small as
1e-16383, and so the local rscale needs to be allowed to exceed 16383,
otherwise all precision is lost, leading to a poor choice of rscale
for the full-precision calculation.

Fix this by removing the cap on the local rscale during the initial
low-precision calculation, as we already do in the full-precision
calculation. This doesn't change the fact that the initial calculation
is a low-precision approximation, computing the logarithm to around 8
significant digits, which is very fast, especially when the base is
very close to 1.

Patch by me, reviewed by Alvaro Herrera.

Discussion: https://postgr.es/m/CAEZATCV-Ceu%2BHpRMf416yUe4KKFv%3DtdgXQAe5-7S9tD%3D5E-T1g%40mail.gmail.com

Branch
------
REL_13_STABLE

Details
-------
https://git.postgresql.org/pg/commitdiff/9ab94ccb15af288f9ff295b4f04b28d86115b803

Modified Files
--------------
src/backend/utils/adt/numeric.c       | 6 +++++-
src/test/regress/expected/numeric.out | 6 ++++++
src/test/regress/sql/numeric.sql      | 1 +
3 files changed, 12 insertions(+), 1 deletion(-)



^ permalink  raw  reply  [nested|flat] 9+ messages in thread

* pgsql: Fix corner-case loss of precision in numeric_power().
@ 2021-10-06 12:29 Dean Rasheed <dean.a.rasheed@gmail.com>
  0 siblings, 0 replies; 9+ messages in thread

From: Dean Rasheed @ 2021-10-06 12:29 UTC (permalink / raw)
  To: pgsql-committers@lists.postgresql.org

Fix corner-case loss of precision in numeric_power().

This fixes a loss of precision that occurs when the first input is
very close to 1, so that its logarithm is very small.

Formerly, during the initial low-precision calculation to estimate the
result weight, the logarithm was computed to a local rscale that was
capped to NUMERIC_MAX_DISPLAY_SCALE (1000). However, the base may be
as close as 1e-16383 to 1, hence its logarithm may be as small as
1e-16383, and so the local rscale needs to be allowed to exceed 16383,
otherwise all precision is lost, leading to a poor choice of rscale
for the full-precision calculation.

Fix this by removing the cap on the local rscale during the initial
low-precision calculation, as we already do in the full-precision
calculation. This doesn't change the fact that the initial calculation
is a low-precision approximation, computing the logarithm to around 8
significant digits, which is very fast, especially when the base is
very close to 1.

Patch by me, reviewed by Alvaro Herrera.

Discussion: https://postgr.es/m/CAEZATCV-Ceu%2BHpRMf416yUe4KKFv%3DtdgXQAe5-7S9tD%3D5E-T1g%40mail.gmail.com

Branch
------
REL_12_STABLE

Details
-------
https://git.postgresql.org/pg/commitdiff/676218034fa8361cc79b1b4631e220b5a7ec6074

Modified Files
--------------
src/backend/utils/adt/numeric.c       | 6 +++++-
src/test/regress/expected/numeric.out | 6 ++++++
src/test/regress/sql/numeric.sql      | 1 +
3 files changed, 12 insertions(+), 1 deletion(-)



^ permalink  raw  reply  [nested|flat] 9+ messages in thread

* pgsql: Fix corner-case loss of precision in numeric_power().
@ 2021-10-06 12:29 Dean Rasheed <dean.a.rasheed@gmail.com>
  0 siblings, 0 replies; 9+ messages in thread

From: Dean Rasheed @ 2021-10-06 12:29 UTC (permalink / raw)
  To: pgsql-committers@lists.postgresql.org

Fix corner-case loss of precision in numeric_power().

This fixes a loss of precision that occurs when the first input is
very close to 1, so that its logarithm is very small.

Formerly, during the initial low-precision calculation to estimate the
result weight, the logarithm was computed to a local rscale that was
capped to NUMERIC_MAX_DISPLAY_SCALE (1000). However, the base may be
as close as 1e-16383 to 1, hence its logarithm may be as small as
1e-16383, and so the local rscale needs to be allowed to exceed 16383,
otherwise all precision is lost, leading to a poor choice of rscale
for the full-precision calculation.

Fix this by removing the cap on the local rscale during the initial
low-precision calculation, as we already do in the full-precision
calculation. This doesn't change the fact that the initial calculation
is a low-precision approximation, computing the logarithm to around 8
significant digits, which is very fast, especially when the base is
very close to 1.

Patch by me, reviewed by Alvaro Herrera.

Discussion: https://postgr.es/m/CAEZATCV-Ceu%2BHpRMf416yUe4KKFv%3DtdgXQAe5-7S9tD%3D5E-T1g%40mail.gmail.com

Branch
------
REL_11_STABLE

Details
-------
https://git.postgresql.org/pg/commitdiff/b2a0f1673371fd0156685b44b06a818609ed7c95

Modified Files
--------------
src/backend/utils/adt/numeric.c       | 6 +++++-
src/test/regress/expected/numeric.out | 6 ++++++
src/test/regress/sql/numeric.sql      | 1 +
3 files changed, 12 insertions(+), 1 deletion(-)



^ permalink  raw  reply  [nested|flat] 9+ messages in thread

* pgsql: Fix corner-case loss of precision in numeric_power().
@ 2021-10-06 12:29 Dean Rasheed <dean.a.rasheed@gmail.com>
  0 siblings, 0 replies; 9+ messages in thread

From: Dean Rasheed @ 2021-10-06 12:29 UTC (permalink / raw)
  To: pgsql-committers@lists.postgresql.org

Fix corner-case loss of precision in numeric_power().

This fixes a loss of precision that occurs when the first input is
very close to 1, so that its logarithm is very small.

Formerly, during the initial low-precision calculation to estimate the
result weight, the logarithm was computed to a local rscale that was
capped to NUMERIC_MAX_DISPLAY_SCALE (1000). However, the base may be
as close as 1e-16383 to 1, hence its logarithm may be as small as
1e-16383, and so the local rscale needs to be allowed to exceed 16383,
otherwise all precision is lost, leading to a poor choice of rscale
for the full-precision calculation.

Fix this by removing the cap on the local rscale during the initial
low-precision calculation, as we already do in the full-precision
calculation. This doesn't change the fact that the initial calculation
is a low-precision approximation, computing the logarithm to around 8
significant digits, which is very fast, especially when the base is
very close to 1.

Patch by me, reviewed by Alvaro Herrera.

Discussion: https://postgr.es/m/CAEZATCV-Ceu%2BHpRMf416yUe4KKFv%3DtdgXQAe5-7S9tD%3D5E-T1g%40mail.gmail.com

Branch
------
REL_10_STABLE

Details
-------
https://git.postgresql.org/pg/commitdiff/4853baaaca4a3469247027f7dc86b516e35f7c0a

Modified Files
--------------
src/backend/utils/adt/numeric.c       | 6 +++++-
src/test/regress/expected/numeric.out | 6 ++++++
src/test/regress/sql/numeric.sql      | 1 +
3 files changed, 12 insertions(+), 1 deletion(-)



^ permalink  raw  reply  [nested|flat] 9+ messages in thread

* pgsql: Fix corner-case loss of precision in numeric_power().
@ 2021-10-06 12:29 Dean Rasheed <dean.a.rasheed@gmail.com>
  0 siblings, 0 replies; 9+ messages in thread

From: Dean Rasheed @ 2021-10-06 12:29 UTC (permalink / raw)
  To: pgsql-committers@lists.postgresql.org

Fix corner-case loss of precision in numeric_power().

This fixes a loss of precision that occurs when the first input is
very close to 1, so that its logarithm is very small.

Formerly, during the initial low-precision calculation to estimate the
result weight, the logarithm was computed to a local rscale that was
capped to NUMERIC_MAX_DISPLAY_SCALE (1000). However, the base may be
as close as 1e-16383 to 1, hence its logarithm may be as small as
1e-16383, and so the local rscale needs to be allowed to exceed 16383,
otherwise all precision is lost, leading to a poor choice of rscale
for the full-precision calculation.

Fix this by removing the cap on the local rscale during the initial
low-precision calculation, as we already do in the full-precision
calculation. This doesn't change the fact that the initial calculation
is a low-precision approximation, computing the logarithm to around 8
significant digits, which is very fast, especially when the base is
very close to 1.

Patch by me, reviewed by Alvaro Herrera.

Discussion: https://postgr.es/m/CAEZATCV-Ceu%2BHpRMf416yUe4KKFv%3DtdgXQAe5-7S9tD%3D5E-T1g%40mail.gmail.com

Branch
------
REL9_6_STABLE

Details
-------
https://git.postgresql.org/pg/commitdiff/0de8f9bc866a6a54ef891f07890d35455e5e6a00

Modified Files
--------------
src/backend/utils/adt/numeric.c       | 6 +++++-
src/test/regress/expected/numeric.out | 6 ++++++
src/test/regress/sql/numeric.sql      | 1 +
3 files changed, 12 insertions(+), 1 deletion(-)



^ permalink  raw  reply  [nested|flat] 9+ messages in thread


end of thread, other threads:[~2021-10-06 12:29 UTC | newest]

Thread overview: 9+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2016-05-05 10:19 pgsql: Fix corner-case loss of precision in numeric pow() calculation Dean Rasheed <dean.a.rasheed@gmail.com>
2020-03-01 14:54 pgsql: Fix corner-case loss of precision in numeric ln(). Dean Rasheed <dean.a.rasheed@gmail.com>
2021-10-06 12:29 pgsql: Fix corner-case loss of precision in numeric_power(). Dean Rasheed <dean.a.rasheed@gmail.com>
2021-10-06 12:29 pgsql: Fix corner-case loss of precision in numeric_power(). Dean Rasheed <dean.a.rasheed@gmail.com>
2021-10-06 12:29 pgsql: Fix corner-case loss of precision in numeric_power(). Dean Rasheed <dean.a.rasheed@gmail.com>
2021-10-06 12:29 pgsql: Fix corner-case loss of precision in numeric_power(). Dean Rasheed <dean.a.rasheed@gmail.com>
2021-10-06 12:29 pgsql: Fix corner-case loss of precision in numeric_power(). Dean Rasheed <dean.a.rasheed@gmail.com>
2021-10-06 12:29 pgsql: Fix corner-case loss of precision in numeric_power(). Dean Rasheed <dean.a.rasheed@gmail.com>
2021-10-06 12:29 pgsql: Fix corner-case loss of precision in numeric_power(). Dean Rasheed <dean.a.rasheed@gmail.com>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox