public inbox for [email protected]help / color / mirror / Atom feed
pgsql: Improve overflow/underflow handling in regr_r2(). 2+ messages / 1 participants [nested] [flat]
* pgsql: Improve overflow/underflow handling in regr_r2(). @ 2026-06-03 08:03 Dean Rasheed <[email protected]> 0 siblings, 0 replies; 2+ messages in thread From: Dean Rasheed @ 2026-06-03 08:03 UTC (permalink / raw) To: [email protected] Improve overflow/underflow handling in regr_r2(). Commit 6498287696d improved corr()'s final function to cope with overflow/underflow in the final calculation, and clamped its result to [-1, 1] in case of roundoff error. Improve regr_r2() in a similar way, clamping its result to [0, 1]. Arguably this is a bug fix, but given the lack of prior complaints, refrain from back-patching, as we did with 6498287696d. Reported-by: Chengpeng Yan <[email protected]> Author: Chengpeng Yan <[email protected]> Reviewed-by: Dean Rasheed <[email protected]> Reviewed-by: Tom Lane <[email protected]> Discussion: https://postgr.es/m/[email protected] Branch ------ master Details ------- https://git.postgresql.org/pg/commitdiff/d58ec50e0faae364fa59cdb88eba7e23b55ab583 Modified Files -------------- src/backend/utils/adt/float.c | 37 ++++++++++++++++++-- src/test/regress/expected/aggregates.out | 58 +++++++++++++++++++++----------- src/test/regress/sql/aggregates.sql | 19 ++++++++--- 3 files changed, 87 insertions(+), 27 deletions(-) ^ permalink raw reply [nested|flat] 2+ messages in thread
* pgsql: Improve overflow/underflow handling in regr_intercept(). @ 2026-06-03 08:22 Dean Rasheed <[email protected]> 0 siblings, 0 replies; 2+ messages in thread From: Dean Rasheed @ 2026-06-03 08:22 UTC (permalink / raw) To: [email protected] Improve overflow/underflow handling in regr_intercept(). As with corr() and regr_r2(), improve regr_intercept()'s final function to cope with overflow/underflow in the final calculation. Here, instead of using sqrt(), we use frexp() and ldexp() to recover, if an overflow or underflow is detected, so that the multiplication and division steps operate on normalised mantissas, and cannot overflow or underflow. As with 6498287696d, and the previous commit improving regr_r2(), this is arguably a bug fix, but given the lack of prior complaints, refrain from back-patching. Reported-by: Tom Lane <[email protected]> Author: Dean Rasheed <[email protected]> Reviewed-by: Chengpeng Yan <[email protected]> Discussion: https://postgr.es/m/[email protected] Branch ------ master Details ------- https://git.postgresql.org/pg/commitdiff/eb8e76e130fd8bd42982d597f5a66f08b13380c0 Modified Files -------------- src/backend/utils/adt/float.c | 39 ++++++++++++++++++++++++++++++-- src/test/regress/expected/aggregates.out | 12 ++++++++++ src/test/regress/sql/aggregates.sql | 2 ++ 3 files changed, 51 insertions(+), 2 deletions(-) ^ permalink raw reply [nested|flat] 2+ messages in thread
end of thread, other threads:[~2026-06-03 08:22 UTC | newest] Thread overview: 2+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2026-06-03 08:03 pgsql: Improve overflow/underflow handling in regr_r2(). Dean Rasheed <[email protected]> 2026-06-03 08:22 pgsql: Improve overflow/underflow handling in regr_intercept(). Dean Rasheed <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox