Date: Fri, 9 Feb 2024 10:24:33 -0600
From: Nathan Bossart <nathandbossart@gmail.com>
To: Mats Kindahl <mats@timescale.com>
Cc: Tom Lane <tgl@sss.pgh.pa.us>, Andres Freund <andres@anarazel.de>,
	Thomas Munro <thomas.munro@gmail.com>,
	Heikki Linnakangas <hlinnaka@iki.fi>,
	pgsql-hackers@lists.postgresql.org
Subject: Re: glibc qsort() vulnerability
Message-ID: <20240209162433.GA663211@nathanxps13>
References: 
 <CA+hUKGLbC4O56rtFM69Tx6StT3PcVio9K2GisjZp=z6O20Q3sw@mail.gmail.com>
 <CA+hUKGJaOM5ULeXboxV+fGkU-pYHotL5A10hYxdu8_kwRcVEgQ@mail.gmail.com>
 <20240208025620.GC445153@nathanxps13>
 <CA+14424k0MbdkJuSSLrr1==PYK+oL5Gtq7siTsMgCs+KcCrEvA@mail.gmail.com>
 <20240208183835.GA503311@nathanxps13>
 <1074897.1707417842@sss.pgh.pa.us>
 <20240208195954.vlpoii4ftoow2of4@awork3.anarazel.de>
 <20240208200737.GA504276@nathanxps13>
 <1242426.1707424769@sss.pgh.pa.us>
 <CA+14425sEcy-KzCE1ztpO=XrHZ5u_5tR2UNsw6874dVbAShpzQ@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: 
 <CA+14425sEcy-KzCE1ztpO=XrHZ5u_5tR2UNsw6874dVbAShpzQ@mail.gmail.com>
Archived-At: 
 <https://www.postgresql.org/message-id/20240209162433.GA663211%40nathanxps13>
Precedence: bulk

On Fri, Feb 09, 2024 at 08:52:26AM +0100, Mats Kindahl wrote:
> Here is a new version introducing pg_cmp_s32 and friends and use them
> instead of the INT_CMP macro introduced before. It also moves the
> definitions to common/int.h and adds that as an include to all locations
> using these functions.

Thanks for the new version of the patch.

> Note that for integers with sizes less than sizeof(int), C standard
> conversions will convert the values to "int" before doing the arithmetic,
> so no casting is *necessary*. I did not force the 16-bit functions to
> return -1 or 1 and have updated the comment accordingly.

It might not be necessary, but this is one of those places where I would
add casting anyway to make it abundantly clear what we are expecting to
happen and why it is safe.

> The types "int" and "size_t" are treated as s32 and u32 respectively since
> that seems to be the case for most of the code, even if strictly not
> correct (size_t can be an unsigned long int for some architecture).

Why is it safe to do this?

> -	return ((const SPLITCOST *) a)->cost - ((const SPLITCOST *) b)->cost;
> +	return INT_CMP(((const SPLITCOST *) a)->cost, ((const SPLITCOST *) b)->cost);

The patch still contains several calls to INT_CMP.

> +/*------------------------------------------------------------------------
> + * Comparison routines for integers
> + *------------------------------------------------------------------------
> + */

I'd suggest separating this part out to a 0001 patch to make it easier to
review.  The 0002 patch could take care of converting the existing qsort
comparators.

> +static inline int
> +pg_cmp_s16(int16 a, int16 b)
> +{
> +	return a - b;
> +}
> +
> +static inline int
> +pg_cmp_u16(uint16 a, uint16 b)
> +{
> +	return a - b;
> +}
> +
> +static inline int
> +pg_cmp_s32(int32 a, int32 b)
> +{
> +	return (a > b) - (a < b);
> +}
> +
> +static inline int
> +pg_cmp_u32(uint32 a, uint32 b)
> +{
> +	return (a > b) - (a < b);
> +}
> +
> +static inline int
> +pg_cmp_s64(int64 a, int64 b)
> +{
> +	return (a > b) - (a < b);
> +}
> +
> +static inline int
> +pg_cmp_u64(uint64 a, uint64 b)
> +{
> +	return (a > b) - (a < b);
> +}

As suggested above, IMHO we should be rather liberal with the casting to
ensure it is abundantly clear what is happening here.

-- 
Nathan Bossart
Amazon Web Services: https://aws.amazon.com