pg.ddx.io pgsql-committers@postgresql.org mailing list archivehelp / color / mirror / Atom feed
pgsql: Fix corner-case loss of precision in numeric pow() calculation 9+ messages / 1 participants [nested] [flat]
* pgsql: Fix corner-case loss of precision in numeric pow() calculation @ 2016-05-05 10:19 Dean Rasheed <dean.a.rasheed@gmail.com> 0 siblings, 0 replies; 9+ messages in thread From: Dean Rasheed @ 2016-05-05 10:19 UTC (permalink / raw) To: pgsql-committers Fix corner-case loss of precision in numeric pow() calculation Commit 7d9a4737c268f61fb8800957631f12d3f13be218 greatly improved the accuracy of the numeric transcendental functions, however it failed to consider the case where the result from pow() is close to the overflow threshold, for example 0.12 ^ -2345.6. For such inputs, where the result has more than 2000 digits before the decimal point, the decimal result weight estimate was being clamped to 2000, leading to a loss of precision in the final calculation. Fix this by replacing the clamping code with an overflow test that aborts the calculation early if the final result is sure to overflow, based on the overflow limit in exp_var(). This provides the same protection against integer overflow in the subsequent result scale computation as the original clamping code, but it also ensures that precision is never lost and saves compute cycles in cases that are sure to overflow. The new early overflow test works with the initial low-precision result (expected to be accurate to around 8 significant digits) and includes a small fuzz factor to ensure that it doesn't kick in for values that would not overflow exp_var(), so the overall overflow threshold of pow() is unchanged and consistent for all inputs with non-integer exponents. Author: Dean Rasheed Reviewed-by: Tom Lane Discussion: http://www.postgresql.org/message-id/CAEZATCUj3U-cQj0jjoia=qgs0SjE3auroxh8swvNKvZWUqegrg@mail.gmail.... See-also: http://www.postgresql.org/message-id/CAEZATCV7w+8iB=07dJ8Q0zihXQT1semcQuTeK+4_rogC_zq5Hw@mail.gmail.... Branch ------ master Details ------- http://git.postgresql.org/pg/commitdiff/18a02ad2a506e4425c6dd2ea235039cd5659467d Modified Files -------------- src/backend/utils/adt/numeric.c | 20 ++++++++++++++++---- src/test/regress/expected/numeric_big.out | 14 ++++++++++++++ src/test/regress/sql/numeric_big.sql | 10 ++++++++++ 3 files changed, 40 insertions(+), 4 deletions(-) -- Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-committers ^ permalink raw reply [nested|flat] 9+ messages in thread
* pgsql: Fix corner-case loss of precision in numeric ln(). @ 2020-03-01 14:54 Dean Rasheed <dean.a.rasheed@gmail.com> 0 siblings, 0 replies; 9+ messages in thread From: Dean Rasheed @ 2020-03-01 14:54 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix corner-case loss of precision in numeric ln(). When deciding on the local rscale to use for the Taylor series expansion, ln_var() neglected to account for the fact that the result is subsequently multiplied by a factor of 2^(nsqrt+1), where nsqrt is the number of square root operations performed in the range reduction step, which can be as high as 22 for very large inputs. This could result in a loss of precision, particularly when combined with large rscale values, for which a large number of Taylor series terms is required (up to around 400). Fix by computing a few extra digits in the Taylor series, based on the weight of the multiplicative factor log10(2^(nsqrt+1)). It remains to be proven whether or not the other 8 extra digits used for the Taylor series is appropriate, but this at least deals with the obvious oversight of failing to account for the effects of the final multiplication. Per report from Justin AnyhowStep. Reviewed by Tom Lane. Discussion: https://postgr.es/m/16280-279f299d9c06e56f@postgresql.org Branch ------ master Details ------- https://git.postgresql.org/pg/commitdiff/43a899f41f46918a0bf442edb091b08c214c68f8 Modified Files -------------- src/backend/utils/adt/numeric.c | 10 +++++++++- src/test/regress/expected/numeric_big.out | 19 +++++++++++++++++++ src/test/regress/sql/numeric_big.sql | 14 ++++++++++++++ 3 files changed, 42 insertions(+), 1 deletion(-) ^ permalink raw reply [nested|flat] 9+ messages in thread
* pgsql: Fix corner-case loss of precision in numeric_power(). @ 2021-10-06 12:29 Dean Rasheed <dean.a.rasheed@gmail.com> 0 siblings, 0 replies; 9+ messages in thread From: Dean Rasheed @ 2021-10-06 12:29 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix corner-case loss of precision in numeric_power(). This fixes a loss of precision that occurs when the first input is very close to 1, so that its logarithm is very small. Formerly, during the initial low-precision calculation to estimate the result weight, the logarithm was computed to a local rscale that was capped to NUMERIC_MAX_DISPLAY_SCALE (1000). However, the base may be as close as 1e-16383 to 1, hence its logarithm may be as small as 1e-16383, and so the local rscale needs to be allowed to exceed 16383, otherwise all precision is lost, leading to a poor choice of rscale for the full-precision calculation. Fix this by removing the cap on the local rscale during the initial low-precision calculation, as we already do in the full-precision calculation. This doesn't change the fact that the initial calculation is a low-precision approximation, computing the logarithm to around 8 significant digits, which is very fast, especially when the base is very close to 1. Patch by me, reviewed by Alvaro Herrera. Discussion: https://postgr.es/m/CAEZATCV-Ceu%2BHpRMf416yUe4KKFv%3DtdgXQAe5-7S9tD%3D5E-T1g%40mail.gmail.com Branch ------ REL_13_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/9ab94ccb15af288f9ff295b4f04b28d86115b803 Modified Files -------------- src/backend/utils/adt/numeric.c | 6 +++++- src/test/regress/expected/numeric.out | 6 ++++++ src/test/regress/sql/numeric.sql | 1 + 3 files changed, 12 insertions(+), 1 deletion(-) ^ permalink raw reply [nested|flat] 9+ messages in thread
* pgsql: Fix corner-case loss of precision in numeric_power(). @ 2021-10-06 12:29 Dean Rasheed <dean.a.rasheed@gmail.com> 0 siblings, 0 replies; 9+ messages in thread From: Dean Rasheed @ 2021-10-06 12:29 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix corner-case loss of precision in numeric_power(). This fixes a loss of precision that occurs when the first input is very close to 1, so that its logarithm is very small. Formerly, during the initial low-precision calculation to estimate the result weight, the logarithm was computed to a local rscale that was capped to NUMERIC_MAX_DISPLAY_SCALE (1000). However, the base may be as close as 1e-16383 to 1, hence its logarithm may be as small as 1e-16383, and so the local rscale needs to be allowed to exceed 16383, otherwise all precision is lost, leading to a poor choice of rscale for the full-precision calculation. Fix this by removing the cap on the local rscale during the initial low-precision calculation, as we already do in the full-precision calculation. This doesn't change the fact that the initial calculation is a low-precision approximation, computing the logarithm to around 8 significant digits, which is very fast, especially when the base is very close to 1. Patch by me, reviewed by Alvaro Herrera. Discussion: https://postgr.es/m/CAEZATCV-Ceu%2BHpRMf416yUe4KKFv%3DtdgXQAe5-7S9tD%3D5E-T1g%40mail.gmail.com Branch ------ REL_11_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/b2a0f1673371fd0156685b44b06a818609ed7c95 Modified Files -------------- src/backend/utils/adt/numeric.c | 6 +++++- src/test/regress/expected/numeric.out | 6 ++++++ src/test/regress/sql/numeric.sql | 1 + 3 files changed, 12 insertions(+), 1 deletion(-) ^ permalink raw reply [nested|flat] 9+ messages in thread
* pgsql: Fix corner-case loss of precision in numeric_power(). @ 2021-10-06 12:29 Dean Rasheed <dean.a.rasheed@gmail.com> 0 siblings, 0 replies; 9+ messages in thread From: Dean Rasheed @ 2021-10-06 12:29 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix corner-case loss of precision in numeric_power(). This fixes a loss of precision that occurs when the first input is very close to 1, so that its logarithm is very small. Formerly, during the initial low-precision calculation to estimate the result weight, the logarithm was computed to a local rscale that was capped to NUMERIC_MAX_DISPLAY_SCALE (1000). However, the base may be as close as 1e-16383 to 1, hence its logarithm may be as small as 1e-16383, and so the local rscale needs to be allowed to exceed 16383, otherwise all precision is lost, leading to a poor choice of rscale for the full-precision calculation. Fix this by removing the cap on the local rscale during the initial low-precision calculation, as we already do in the full-precision calculation. This doesn't change the fact that the initial calculation is a low-precision approximation, computing the logarithm to around 8 significant digits, which is very fast, especially when the base is very close to 1. Patch by me, reviewed by Alvaro Herrera. Discussion: https://postgr.es/m/CAEZATCV-Ceu%2BHpRMf416yUe4KKFv%3DtdgXQAe5-7S9tD%3D5E-T1g%40mail.gmail.com Branch ------ master Details ------- https://git.postgresql.org/pg/commitdiff/e54a758d24dab056bb7f50d26c57a3c8761cc44a Modified Files -------------- src/backend/utils/adt/numeric.c | 6 +++++- src/test/regress/expected/numeric.out | 6 ++++++ src/test/regress/sql/numeric.sql | 1 + 3 files changed, 12 insertions(+), 1 deletion(-) ^ permalink raw reply [nested|flat] 9+ messages in thread
* pgsql: Fix corner-case loss of precision in numeric_power(). @ 2021-10-06 12:29 Dean Rasheed <dean.a.rasheed@gmail.com> 0 siblings, 0 replies; 9+ messages in thread From: Dean Rasheed @ 2021-10-06 12:29 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix corner-case loss of precision in numeric_power(). This fixes a loss of precision that occurs when the first input is very close to 1, so that its logarithm is very small. Formerly, during the initial low-precision calculation to estimate the result weight, the logarithm was computed to a local rscale that was capped to NUMERIC_MAX_DISPLAY_SCALE (1000). However, the base may be as close as 1e-16383 to 1, hence its logarithm may be as small as 1e-16383, and so the local rscale needs to be allowed to exceed 16383, otherwise all precision is lost, leading to a poor choice of rscale for the full-precision calculation. Fix this by removing the cap on the local rscale during the initial low-precision calculation, as we already do in the full-precision calculation. This doesn't change the fact that the initial calculation is a low-precision approximation, computing the logarithm to around 8 significant digits, which is very fast, especially when the base is very close to 1. Patch by me, reviewed by Alvaro Herrera. Discussion: https://postgr.es/m/CAEZATCV-Ceu%2BHpRMf416yUe4KKFv%3DtdgXQAe5-7S9tD%3D5E-T1g%40mail.gmail.com Branch ------ REL_14_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/8e26b868d5cfe9c746578aa3b172b6f24a5a310c Modified Files -------------- src/backend/utils/adt/numeric.c | 6 +++++- src/test/regress/expected/numeric.out | 6 ++++++ src/test/regress/sql/numeric.sql | 1 + 3 files changed, 12 insertions(+), 1 deletion(-) ^ permalink raw reply [nested|flat] 9+ messages in thread
* pgsql: Fix corner-case loss of precision in numeric_power(). @ 2021-10-06 12:29 Dean Rasheed <dean.a.rasheed@gmail.com> 0 siblings, 0 replies; 9+ messages in thread From: Dean Rasheed @ 2021-10-06 12:29 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix corner-case loss of precision in numeric_power(). This fixes a loss of precision that occurs when the first input is very close to 1, so that its logarithm is very small. Formerly, during the initial low-precision calculation to estimate the result weight, the logarithm was computed to a local rscale that was capped to NUMERIC_MAX_DISPLAY_SCALE (1000). However, the base may be as close as 1e-16383 to 1, hence its logarithm may be as small as 1e-16383, and so the local rscale needs to be allowed to exceed 16383, otherwise all precision is lost, leading to a poor choice of rscale for the full-precision calculation. Fix this by removing the cap on the local rscale during the initial low-precision calculation, as we already do in the full-precision calculation. This doesn't change the fact that the initial calculation is a low-precision approximation, computing the logarithm to around 8 significant digits, which is very fast, especially when the base is very close to 1. Patch by me, reviewed by Alvaro Herrera. Discussion: https://postgr.es/m/CAEZATCV-Ceu%2BHpRMf416yUe4KKFv%3DtdgXQAe5-7S9tD%3D5E-T1g%40mail.gmail.com Branch ------ REL_12_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/676218034fa8361cc79b1b4631e220b5a7ec6074 Modified Files -------------- src/backend/utils/adt/numeric.c | 6 +++++- src/test/regress/expected/numeric.out | 6 ++++++ src/test/regress/sql/numeric.sql | 1 + 3 files changed, 12 insertions(+), 1 deletion(-) ^ permalink raw reply [nested|flat] 9+ messages in thread
* pgsql: Fix corner-case loss of precision in numeric_power(). @ 2021-10-06 12:29 Dean Rasheed <dean.a.rasheed@gmail.com> 0 siblings, 0 replies; 9+ messages in thread From: Dean Rasheed @ 2021-10-06 12:29 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix corner-case loss of precision in numeric_power(). This fixes a loss of precision that occurs when the first input is very close to 1, so that its logarithm is very small. Formerly, during the initial low-precision calculation to estimate the result weight, the logarithm was computed to a local rscale that was capped to NUMERIC_MAX_DISPLAY_SCALE (1000). However, the base may be as close as 1e-16383 to 1, hence its logarithm may be as small as 1e-16383, and so the local rscale needs to be allowed to exceed 16383, otherwise all precision is lost, leading to a poor choice of rscale for the full-precision calculation. Fix this by removing the cap on the local rscale during the initial low-precision calculation, as we already do in the full-precision calculation. This doesn't change the fact that the initial calculation is a low-precision approximation, computing the logarithm to around 8 significant digits, which is very fast, especially when the base is very close to 1. Patch by me, reviewed by Alvaro Herrera. Discussion: https://postgr.es/m/CAEZATCV-Ceu%2BHpRMf416yUe4KKFv%3DtdgXQAe5-7S9tD%3D5E-T1g%40mail.gmail.com Branch ------ REL_10_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/4853baaaca4a3469247027f7dc86b516e35f7c0a Modified Files -------------- src/backend/utils/adt/numeric.c | 6 +++++- src/test/regress/expected/numeric.out | 6 ++++++ src/test/regress/sql/numeric.sql | 1 + 3 files changed, 12 insertions(+), 1 deletion(-) ^ permalink raw reply [nested|flat] 9+ messages in thread
* pgsql: Fix corner-case loss of precision in numeric_power(). @ 2021-10-06 12:29 Dean Rasheed <dean.a.rasheed@gmail.com> 0 siblings, 0 replies; 9+ messages in thread From: Dean Rasheed @ 2021-10-06 12:29 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix corner-case loss of precision in numeric_power(). This fixes a loss of precision that occurs when the first input is very close to 1, so that its logarithm is very small. Formerly, during the initial low-precision calculation to estimate the result weight, the logarithm was computed to a local rscale that was capped to NUMERIC_MAX_DISPLAY_SCALE (1000). However, the base may be as close as 1e-16383 to 1, hence its logarithm may be as small as 1e-16383, and so the local rscale needs to be allowed to exceed 16383, otherwise all precision is lost, leading to a poor choice of rscale for the full-precision calculation. Fix this by removing the cap on the local rscale during the initial low-precision calculation, as we already do in the full-precision calculation. This doesn't change the fact that the initial calculation is a low-precision approximation, computing the logarithm to around 8 significant digits, which is very fast, especially when the base is very close to 1. Patch by me, reviewed by Alvaro Herrera. Discussion: https://postgr.es/m/CAEZATCV-Ceu%2BHpRMf416yUe4KKFv%3DtdgXQAe5-7S9tD%3D5E-T1g%40mail.gmail.com Branch ------ REL9_6_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/0de8f9bc866a6a54ef891f07890d35455e5e6a00 Modified Files -------------- src/backend/utils/adt/numeric.c | 6 +++++- src/test/regress/expected/numeric.out | 6 ++++++ src/test/regress/sql/numeric.sql | 1 + 3 files changed, 12 insertions(+), 1 deletion(-) ^ permalink raw reply [nested|flat] 9+ messages in thread
end of thread, other threads:[~2021-10-06 12:29 UTC | newest] Thread overview: 9+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2016-05-05 10:19 pgsql: Fix corner-case loss of precision in numeric pow() calculation Dean Rasheed <dean.a.rasheed@gmail.com> 2020-03-01 14:54 pgsql: Fix corner-case loss of precision in numeric ln(). Dean Rasheed <dean.a.rasheed@gmail.com> 2021-10-06 12:29 pgsql: Fix corner-case loss of precision in numeric_power(). Dean Rasheed <dean.a.rasheed@gmail.com> 2021-10-06 12:29 pgsql: Fix corner-case loss of precision in numeric_power(). Dean Rasheed <dean.a.rasheed@gmail.com> 2021-10-06 12:29 pgsql: Fix corner-case loss of precision in numeric_power(). Dean Rasheed <dean.a.rasheed@gmail.com> 2021-10-06 12:29 pgsql: Fix corner-case loss of precision in numeric_power(). Dean Rasheed <dean.a.rasheed@gmail.com> 2021-10-06 12:29 pgsql: Fix corner-case loss of precision in numeric_power(). Dean Rasheed <dean.a.rasheed@gmail.com> 2021-10-06 12:29 pgsql: Fix corner-case loss of precision in numeric_power(). Dean Rasheed <dean.a.rasheed@gmail.com> 2021-10-06 12:29 pgsql: Fix corner-case loss of precision in numeric_power(). Dean Rasheed <dean.a.rasheed@gmail.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions for how to clone and mirror all data and code used for this inbox