postgres.git / summary / log / commit / refs

commit    54f82c4aae768037fe14d909b2b535cfbe89f900
Author:   David Rowley <drowley@postgresql.org>
Date:     Sun Jan 04 07:34:01 2026 +0000
First released in: 14.21, 15.16, 16.12, 17.8, 18.2

    Fix selectivity estimation integer overflow in contrib/intarray
    
    This fixes a poorly written integer comparison function which was
    performing subtraction in an attempt to return a negative value when
    a < b and a positive value when a > b, and 0 when the values were equal.
    Unfortunately that didn't always work correctly due to two's complement
    having the INT_MIN 1 further from zero than INT_MAX.  This could result
    in an overflow and cause the comparison function to return an incorrect
    result, which would result in the binary search failing to find the
    value being searched for.
    
    This could cause poor selectivity estimates when the statistics stored
    the value of INT_MAX (2147483647) and the value being searched for was
    large enough to result in the binary search doing a comparison with that
    INT_MAX value.
    
    Author: Chao Li <li.evan.chao@gmail.com>
    Reviewed-by: David Rowley <dgrowleyml@gmail.com>
    Discussion: https://postgr.es/m/CAEoWx2ng1Ot5LoKbVU-Dh---dFTUZWJRH8wv2chBu29fnNDMaQ@mail.gmail.com
    Backpatch-through: 14


 contrib/intarray/_int_selfuncs.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/contrib/intarray/_int_selfuncs.c b/contrib/intarray/_int_selfuncs.c
index 17dc6768ea5..43146feb18f 100644
--- a/contrib/intarray/_int_selfuncs.c
+++ b/contrib/intarray/_int_selfuncs.c
@@ -327,7 +327,12 @@ static int
 compare_val_int4(const void *a, const void *b)
 {
 	int32		key = *(int32 *) a;
-	const Datum *t = (const Datum *) b;
+	int32		value = DatumGetInt32(*(const Datum *) b);
 
-	return key - DatumGetInt32(*t);
+	if (key < value)
+		return -1;
+	else if (key > value)
+		return 1;
+	else
+		return 0;
 }


[parent: 3e77e944e730]